Compare commits

..
224 Commits
Author SHA1 Message Date
peter d66174ed16 test(arch): a unit's HTTP layer does no database work beyond its pin
The fence is switched on unit by unit: each route file's database work (SQL built, session data
calls, model attributes set, unit tables imported) is pinned at today's count and may only shrink.
A core route may not reach into a unit's tables. Every model has an owner.
2026-10-18 23:01:12 +02:00
peter c85ae42b31 release: 0.108.0 (feature/e40-job-history) 2026-10-18 21:48:23 +02:00
peter cb9a89682a chore(release): 0.108.0 — one Activity page for runs, admin actions, the server log and quota; every background run has a history 2026-10-18 21:40:52 +02:00
peter b842e4e871 fix(ui): a looping pulse stops for reduced motion; the e2e waits for skeleton bones, not for any pulse
- Seven looping pulses (a running job's dot, the scheduler's progress bar, a queued download, the
  download tray, a playlist upload, a channel's full-history action, the setup wizard's test mail)
  ran regardless of prefers-reduced-motion, which docs/DESIGN.md promises to respect: each is
  motion-safe now, held by a contract test.
- The a11y suite's "settled" wait took every aria-hidden pulse for a loading skeleton, so the
  Scheduler legend's running dot, which pulses for good, kept it waiting until it timed out. A
  skeleton bone now says it is one (data-skeleton) and the wait looks for that.
2026-10-18 21:40:52 +02:00
peter 5082636f15 fix(activity): a run nothing here has open reads as interrupted, a rollback sums a stopped run's spend
- The scheduler keeps one registry of the runs this process has open, for the interval jobs and the
  ad-hoc runs alike. A row the table still says is `running` but no run here has open (its outcome
  failed to write) reads as interrupted, as the next start's sweep will close it: no pulsing dot
  beside "No run recorded yet".
- Migration 0109's downgrade sums a still-running run's quota events into its spend before the link
  to them is dropped, as the startup sweep would.
- A test holds that a dialog closing onto the same address leaves an open log line open.
2026-10-18 21:21:19 +02:00
peter 13b7577d9f fix(activity): an ad-hoc run's row is one run's, a rollback keeps a run whole, Back reads one entry's filters
- The channel-backfill row says "running" from the runs this process has open, not from the table:
  a row whose outcome failed to write no longer reads as running until a restart. The run still
  going leads the row, so a newer run's outcome never sits under the older one's channel.
- Migration 0109's downgrade carries every run's trigger, spend and start in the audit row, and the
  upgrade restores them: a manual run whose admin was deleted stays manual.
- Back/Forward is read by the page's one popstate listener, both halves in one render; the server
  log adopts its half during that render, so no read goes out with one entry's filters and the
  other's range.
- The server log's failed facets use the other tabs' words (the menus may be short); the accounts
  message shows only when the menu really lacks them.
2026-10-18 17:32:22 +02:00
peter fad656988a fix(activity): runs keep their true outcome and spend, the page reaches every kept row, a closing dialog leaves the server log alone
Backend
- A run's outcome write sits outside the job's own try: a database blip after a run that
  succeeded no longer records it as failed (the row waits for the next start's sweep).
- An interrupted run keeps what it spent (the sweep sums its quota rows, as finish does).
- The Scheduler's ad-hoc row says running while any backfill runs, not only the newest.
- recent_runs reads each job's last runs off its index instead of ranking the whole table.
- An interval change is stored with its audit row in one transaction, then rescheduled.
- One API instance per database is the stated contract (.env.example, jobruns).
- Turning full history on for a channel already complete records no run.
- Migration 0109 round-trips interrupted and ad-hoc runs; the run words' DB CHECKs are
  test-held to the enums; the account filter is one helper for the four activity reads.
- An audit row says how many of its request's log lines are still kept.

Frontend
- An "All" period reaches every row still kept (runs and admin actions outlive 90 days).
- Back/Forward is followed in place by the server log: a closing dialog no longer turns
  Live off; a pivot still opens a fresh log.
- Runs moved over from the audit log read their one-liner in the reader's language.
- Filter choices or accounts that failed to load say so beside the filters, with a retry.
- An old server-log link keeps its account, text and range; a remembered retired page
  opens its tab.
- A request whose lines are gone says so instead of linking to an empty list.
- The log lines and the history rows share one row frame, and the server log one list
  frame with the other tabs; an empty list is announced; each day's quota is readable.
2026-10-18 15:38:32 +02:00
peter 3021be5166 fix(activity): a run's quota reads with its unit, its outcome keeps room at a large text size, a token's audit rows read in Hungarian
The run list showed the quota spend as a bare figure and, at the largest text size on a 1024-wide
screen, its fixed columns left the outcome column no width at all. The outcome now keeps 10rem and
the trigger column gives way; the audit list's summary keeps the same floor. A run that stored no
log line says so instead of linking to an empty list. An API token revoke and a token's file work
rendered the server's English line under the Hungarian UI; both are rebuilt in the reader's
language, and the test holds the server's wording.
2026-10-18 04:22:52 +02:00
peter c288ba147b fix(activity): an old audit/log link opens its tab, times read relative, one name for admin actions
Seen in the browser, not in the page's own tests: the navigation provider rewrites a retired page id
to `activity` before the page's chunk reads the URL, so `?page=audit` landed on the Runs tab. The
retired-page map now carries the params that open the successor's part (`at`), and the provider
writes them as it rewrites the page — on load and on Back alike.

A list time is relative with the exact second on hover (the DESIGN copy rule); a run moved over from
the audit log, whose start and end are one instant, shows no length instead of an invented "1 sec".
The audit log is called "admin actions" everywhere it is named (the clear button and its confirm, the
Configuration group, the cleanup job, which also prunes the run history its description left out).
2026-10-18 04:00:38 +02:00
peter fe6308d844 refactor(activity): the window's hours stay private to its URL state 2026-10-18 03:48:30 +02:00
peter 24448e7c6f feat(activity): one Activity page — runs, admin actions, server log and quota under one filter bar
The audit log, the server log and the admin quota view were three pages (the quota one a Stats tab),
each with its own filters, and the background runs had no list at all. Now one page ("Napló" /
"Activity") reads all four, each still its own table on the server: a shared bar (text, a period of
1 h to 90 days defaulting to 7, an account) and each tab's own filters, the whole view in the URL.
The old page ids open their tab.

Server side: the audit log and the new run list (`/api/admin/runs`) filter and page like the server
log, through one keyset helper (`routes/_keyset`) the log now shares; an audit row keeps the request
it was made in (migration 0111), so its log lines are a pivot away, as a run's are (`lrun`). The
quota breakdown takes since/until and one account. The Scheduler rows get a history button to their
job's runs.

Also fixed on the way: re-saving the value in force on the Configuration page staged its override
without committing it (the commit sat inside the audit branch), so the save was lost.
2026-10-18 03:45:18 +02:00
peter d13483a593 feat(config): job intervals live in sysconfig, every setting says when it applies, the unread SponsorBlock default goes
An interval override was a row of its own table beside the one store every other admin knob uses; it
is now the `job_interval_<job>` sysconfig key (migration 0110 copies them over and drops the table).
The Scheduler page stays its one editor: the Configuration page and the setup wizard neither list nor
write those keys, since a write there would skip the reschedule.

Each ConfigSpec now states `applies` (required, no default). Every key is `live`, held by a scan that
finds no read at import time; a `restart` key is refused until the page can say so. A second scan
requires a reader for every key on the Configuration page — `sponsorblock_default` had none since it
was added, so it leaves the registry, the translations and the store.
2026-10-18 02:23:15 +02:00
peter 3bab2e7dc7 docs(scheduler): the snapshot docstring names the ad-hoc runs it now carries 2026-10-18 01:29:46 +02:00
peter 1b1e3cdcf2 feat(scheduler): each row shows its last three runs, an interrupted run as such, and the ad-hoc runs beside the jobs
A row's one dot said how the LAST run went, and an interrupted run fell through to "idle" — a grey dot
that read as "no run recorded". The row now leads with its last three runs, newest first, each named
for a screen reader and in its tooltip (how it went, when, who started it); an interrupted run is a
ring, set apart from a skipped run and a failure, and its line says the server restarted under it.

A channel's full-history backfill gets its own "Ad-hoc runs" block: its runs, its channel and what
it fetched, no interval and no run button — it needs its target. AD_HOC_JOBS names every such job;
the backend-id contract holds their names in both languages, and run_ad_hoc refuses any other id.
2026-10-18 01:20:53 +02:00
peter 27c2fcbd2e feat(scheduler): a channel's "get full history" backfill is a recorded run, and a row knows its last three runs
The one-channel deep backfill ran as a FastAPI BackgroundTask: what it fetched, what it spent and
how it failed reached only the log. It is now a one-shot run on the scheduler (`channel_backfill`,
trigger ad_hoc, actor = the user, target = the channel) with its quota tagged to the run. Its
per-channel exclusion with the scheduled backfill and the two-run cap are unchanged, and a run
they turn away says why in its result instead of returning nothing. A failure is no longer
swallowed: the run records it.

The scheduler snapshot carries each job's last three runs (one windowed query), so the page can
show an interrupted or skipped run instead of the latest outcome alone.
2026-10-18 01:05:24 +02:00
peter 911527a548 fix(scheduler): the dashboard no longer says a job has not run "this session" — runs are kept across restarts 2026-10-17 23:22:13 +02:00
peter 8fd91b7b15 fix(scheduler): keep run history 180 days, as long as the audit log kept the runs it moved over 2026-10-17 23:15:17 +02:00
peter 45c0ee7d58 feat(scheduler): every background run is a durable row
What a job last did lived in the API process's memory and was gone at
every restart; the audit log kept a run only when it was manual, failed
or changed something. Now each run is a job_runs row written at its
start and its end in its own session: who started it, its outcome as
data, what it spent of the YouTube quota (quota_events.job_run_id) and
the lines it logged (log_events.run_id, filterable in the log console).

- a run a stopped process left open is marked interrupted at startup
- "run now" is a one-shot job on the scheduler, which now always starts
  (interval jobs only with SCHEDULER_ENABLED); the manual actor is reset
  after the run, since pool threads are reused
- the dashboard's last run comes from job_runs, so it survives a deploy
- runs leave the audit log: migration 0109 moves the old job_run rows
  over (and back on downgrade); the audit page's run folding goes
- job_run_retention_days (default 30) prunes old runs in audit_gc
2026-10-17 23:08:55 +02:00
peter 4b1f6d7ec2 release: 0.107.3 (improvement/dev-tooling) 2026-10-17 21:34:53 +02:00
peter 51bf780100 chore(release): 0.107.3 — text on glass holds 4.5:1; the e2e suite runs axe, the gate lints every shell script 2026-10-17 21:21:37 +02:00
peter 3ada34073b fix(theme): text on glass holds 4.5:1 — light --muted kept at 6:1 on the page, tonal fills opaque
Pixel-measured over every scheme x mode x backdrop: the item count fell to 3.8:1 (dark too: its 8%
fill let the toolbar glass and the thumbnails under it through), Channels' tabs to 4.1:1 and the
current Files row to 4.4:1. Light --muted moves 15% toward --fg in every scheme; the badges, the item
count and the current rail row paint opaque bg-tint* fills mixed into the surface. The token contract
now asks 6:1 of --muted on the bare page and checks those three sites use the tints.
2026-10-17 20:06:54 +02:00
peter d74f2a9ca1 fix(deps): source-map-js 1.2.2 — the indexed-source-map denial-of-service advisory (dev tooling only) 2026-10-17 17:51:10 +02:00
peter a8dddc8c24 test(e2e): axe-core's WCAG A + AA rules over every page and the player (chromium only); contrast stays with the visual check 2026-10-17 17:48:53 +02:00
peter 460bdec406 chore(tooling): drop scripts/lint-changed.sh — the shared gate runner's Stop hook lints the turn's files, shellcheck included 2026-10-17 17:22:35 +02:00
peter 66181d1394 chore(gate): shellcheck joins the gate — every tracked shell script, the host binary held to a pin 2026-10-17 15:31:34 +02:00
peter 0a1fef28ac docs(claude): the YouTube write-error helper, the overlay and z-scale rule, and px-frozen sizes join the canonical patterns 2026-10-17 14:47:57 +02:00
peter fe572d9207 docs: a copy-only change runs the Conformance check's step 4 alone, in both languages 2026-10-17 14:27:45 +02:00
peter c8247c0dde release: 0.107.2 (improvement/gate-runner) 2026-10-17 04:37:30 +02:00
peter 722c37a6d0 chore(release): 0.107.2 — the gate lists and runs single lanes, the backend tests use every core 2026-10-17 04:30:33 +02:00
peter 5ba5679239 test: tests that assumed an idle machine wait for what they assert, or carry a budget for their work
Under a busy machine (the gate's lanes side by side, 12 busy cores) these went red at random: a press landed while the button was still busy, an answer was settled before it was asked for, a page grew past one wait's budget, a lazy chunk was transformed inside a 1 s wait, focus was taken a frame after the press. They now wait for the condition they depend on, preload the chunks they open, and the tests that do heavy work (whole-repo scans, the type-aware linter, module resets) carry an explicit budget naming that work. The whole suite four times under 12 busy-loop processes: red 4/4 before, 0/4 after.
2026-10-17 01:08:15 +02:00
peter 926e5e9899 chore(gate): check.sh can list its lanes and run one of them
--list prints the lane names, --only "<lane>" runs that one lane, so a runner can schedule the lanes itself and a red lane can be re-run on its own. An --only run reports that lane, never the gate, and an unknown name exits 2 instead of passing with nothing run; the contract test holds both.
2026-10-16 02:53:30 +02:00
peter 9251918e08 chore(test): the DB lane runs on every core — each pytest-xdist worker gets its own database 2026-10-16 01:13:31 +02:00
peter 2b6d968833 release: 0.107.1 (improvement/gate-timing) 2026-10-16 00:45:45 +02:00
peter 95e6b2c3d9 chore(release): 0.107.1 — a failed download's explanation says press Retry 2026-10-16 00:33:59 +02:00
peter 80c55fbf1b merge bug/download-retry-copy into improvement/gate-timing 2026-10-16 00:33:38 +02:00
peter 011f269152 chore(gate): every lane reports how long it took, and a run can log the times for comparison 2026-10-16 00:13:47 +02:00
peter dcb244dd04 fix(downloads): a failed download's explanation says press Retry, not retry it from a list that is gone 2026-10-15 22:56:40 +02:00
peter 08a9f4e255 release: 0.107.0 (feature/e39b-machine-access) 2026-10-15 06:20:35 +02:00
peter 1673134c36 chore(release): 0.107.0 — API tokens for your machines, share a whole folder by link 2026-10-15 06:09:23 +02:00
peter 27427eef32 fix(files): ending every session and making a token lock the user without holding up rows that refer to it 2026-10-15 06:05:51 +02:00
peter a248ca6c13 fix(files): the token list is fresh after the sessions end, a copy again reads Copied again, the Copied button says so 2026-10-15 05:47:00 +02:00
peter 620f84dba4 fix(files): a token made as every session ends goes with them, and a file being shared holds its job 2026-10-15 05:47:00 +02:00
peter 2efa7d5a75 fix(files): an upload is said once in any order, copy buttons speak, a link list that failed says so
- The queue holds an upload it sees land done while this tab's upload still waits for its answer, so
  the dialog's notice is the only one whichever comes first; another device's upload follows after.
- One `useCopyFeedback` for every Copy beside a value (token, share link, set-password link): Copied
  for a moment, a notice a screen reader hears, a warning when the clipboard refuses; the token
  dialog's two Copy buttons are named by what they copy.
- The share dialog's link list shows loading, and a load error with Retry, never "no links" — and no
  New link beside a list it has not seen.
- The settings texts say that signing out elsewhere or a password change revokes the API tokens.
- DESIGN.md logs the folder-link decisions; the design lab keeps only the chosen renderings of the
  Links, Tokens and Files rounds.
2026-10-15 04:58:17 +02:00
peter 827df7494c fix(files): ending every session ends every API token, and the share links hold their folder
- A password reset or change, "log out everywhere" and a revoked untrusted password revoke all of the
  account's API tokens too (audited): one minted from a stolen session no longer outlives the answer.
- A bearer token or link grant with a non-ASCII character is refused (401/403), not a 500.
- Making a token's grants or a folder link holds the folder, so a racing delete cannot fail the
  foreign key; a link's view count is one atomic UPDATE.
- A folder link's card counts exactly the files its page lists (one `links.folder_files`).
- A file with no extension is served without a trailing dot.
- 0106's downgrade names a Hub folder so it never collides with one of the user's folders.
- The speed baseline reads share tokens through the app (the database keeps them hashed/encrypted).
- Tests: the token scope on the real session stack, the folder poster, the Hub refused for a link,
  0106's downgrade data step, 0107 without a key; docs/comments follow (three folder kinds, folder
  links, key change, the cleanup job's token warnings).
2026-10-15 04:58:16 +02:00
peter 86712b6b0e feat(files): share a folder by link — its page lists what is in it, plays a video above the list
A /watch link may now open one of its owner's own folders (migration 0108: folder_id beside job_id,
exactly one set; a folder's links go with it). The page shows what is in the folder when it is
opened, newest first: a video or a song plays in the same player a file link has, above the list,
each keeping its own resume point; a document only downloads, never opens as a page. A file moved
out stops opening through the link at its next request. Password, expiry and stream-only work as on
a file link; the unfurl names the folder and counts its files, never their names. Source folders
and the Hub cannot be shared.

The folder menu gets "Copy link" and "Share with a link…" (the Share dialog's link half); the rail
marks a folder a live link opens.

Two fixes in the Share dialog on the way: revoking the last link no longer mints a new one at once,
and the new-link form goes back to downloadable after making a link.
2026-10-15 02:23:24 +02:00
peter 90567718b2 feat(files): share-link tokens are kept only hashed and encrypted, and every link change is audited
A /watch link's token is the credential, and it sat in the database as written. Migration 0107
replaces the column with the token's SHA-256 (the lookup) and a Fernet-encrypted copy (so the owner
can copy the same link again); every existing address keeps working, and the downgrade decrypts
them back. Without a copy this key can read, a link still opens but its address cannot be shown:
"Copy link" then takes the newest link that has one or makes a new one, and the Share dialog says
so in that row instead of a broken URL. Making, changing and revoking a link are audited, never
with its token or password.
2026-10-15 01:02:29 +02:00
peter c31d7f9d65 fix(files): an upload is announced once, as an upload, and a finish shows in Files at once
An upload that came back done was announced twice: by the upload dialog ("File uploaded") and
again by the tray, which saw a job turn done ("Downloaded: …"). The dialog now marks its job as
announced, so the queue keeps it for the tray but does not say it again. An upload the worker
converted first is still the tray's to announce, now as "Uploaded: …".

A finished job's notice offered "Open" while Files still listed the folder without it (its own
poll runs every 15 s): the tray now refreshes the library the moment it sees a finish.
2026-10-14 23:54:19 +02:00
peter db56f25387 feat(files): API tokens in Settings, the Hub folder in Files, a notice before a token runs out
Settings gains an API tokens tab: a row per token (name and prefix; each
folder with its rights; last use and expiry, "expires soon" marked),
revoked from its menu behind a confirm. A new token is the Hub exchange in
one click, or folders and rights picked per folder (no Write where an
upload cannot land); expiry 30/90 days, 1 year or never (warned). The
token shows once in the same dialog, with Copy and a ready curl line.

The Hub folder closes the Sources in the Files rail, takes uploads and
moved-in files. A token about to run out warns its user once, a week
before (the Files GC's sixth pass); the notice opens the tokens tab.
2026-10-14 23:33:34 +02:00
peter 4f58d15d3f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 21:19:14 +02:00
peter 2c66ce1cf4 release: 0.106.2 (improvement/thumbs-and-apt-refresh) 2026-10-14 21:17:39 +02:00
peter ce45445d17 chore(release): 0.106.2 — pictures that fail to load retry instead of breaking 2026-10-14 21:06:18 +02:00
peter 383d394e32 fix(ui): a picture that fails to load shows the card's own frame, then retries
A bare <img> sent a Referer that Google's image CDNs refuse and had no retry and no fallback, so a
Home shelf card showed the browser's broken-image mark. ui/RemoteImage gathers what Avatar, the
channel banner and the download rows had each hand-rolled: no Referer, up to three retries past the
browser's negative cache, the caller's fallback (by default the card's empty frame) instead of the
broken mark, the failure remembered by URL. Every picture now goes through it; a lint rule refuses
a bare <img>.
2026-10-14 20:51:08 +02:00
peter 06e8291896 build(image): rebuild the apt-get upgrade layer when APT_REFRESH changes
A build cache kept the upgrade layer for as long as nothing above it changed, so a publish shipped
curl and librsvg a release behind fixes already in the Debian archive (the CVE gate refused it).
The release build passes the day; empty, the default for a self-hosted build, leaves the cache alone.
2026-10-14 20:51:07 +02:00
peter db6b60029f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 20:17:05 +02:00
peter 02134799cc release: 0.106.1 (bug/plex-settings-outside-click) 2026-10-14 20:03:38 +02:00
peter c8bc843517 chore(release): 0.106.1 — the Plex player survives a click outside its Settings panel 2026-10-14 19:44:25 +02:00
peter 759a7536b9 fix(history): an overlay opened while a back() is in flight waits to push its entry
The Plex player's ⋯ → Settings closed the menu (its entry went back) and opened the panel a task
later, before that back() had landed. The panel's push was the entry the landing back() took, so
the books counted one entry more than the history held, and closing the panel by a click outside
it stepped back over the player's own entry: the player closed.

A push now waits while a back() of ours is in flight; the popstate that lands the last one
reconciles again.
2026-10-14 19:43:51 +02:00
peter b2d34e6e6c feat(files): API tokens — a machine reaches the library folder by folder
A user can make a token for a machine of theirs (`/api/tokens`, signed-in only): `sfl_` + 32
random bytes + a checksum, only its SHA-256 kept, shown once. Sent as a Bearer header it is
honoured on six `/api/files` routes only — folders, list, file, hash, upload, delete — and there
only in the folders its grants name, with the rights they give (list / read / write / delete).
One check, `FilesCaller.may`, in every route; a folder or file out of reach answers like a
missing one; the files shared with the user are never in reach. A token beats a cookie on the
same request.

The Hub preset makes the user's one Hub folder (a new folder kind) and grants it every right.
Refused tokens are slowed down per client, a runaway machine per token; last use is stamped at
most once a minute; making and revoking a token is audited, and a token's uploads and deletes
name it in their audit rows.
2026-10-14 18:40:23 +02:00
peter c335d54929 release: 0.106.0 (feature/e39-files) 2026-10-13 03:34:56 +02:00
peter 7188ad7d71 chore(release): 0.106.0 — the Files library 2026-10-13 03:24:14 +02:00
peter ebbae6794c fix(files): review round 3, the page — a search waits for the server, the viewer keeps its file, a drop ends the drag
- past the list's cap a search waits in a skeleton, never showing the newest files' 'no match' for a
  file the server is about to find; a sort keeps its stand-in rows
- the viewer keeps the file it was opened on when a server answer replaces the rows under it
- a drop another zone took still clears the page's drop overlay
- 'The first N items' only for a server sort; the past-the-end notice no longer claims the file exists
- a failed server search is tested to say so
2026-10-13 03:15:59 +02:00
peter c9baf68b16 fix(files): review round 3, the server — a fetched locked re-read, a finish that follows its file, an eviction that counts again
- set_retention and delete_own re-read the folder under its lock and use that row (an unfetched ORM
  result never refreshes the object); a folder gone meanwhile is refused (409), and the route audits
  the retention actually replaced
- a finishing download is stamped only while it is still in the folder (and keep) it was read with:
  a move or a folder delete landing meanwhile sends the read round again, so no file is done with no
  expiry
- the eviction pass counts the disk before each candidate and skips one asked for since it began
- docs: source builds get their own volume-copy snippet
- race tests assert the lock wait; one built on a committing delete never raced
2026-10-13 03:15:59 +02:00
peter 5bb101e493 fix(files): review round 2, the page — the viewer's Escape, the dialog's drop, the capped list's honesty
- Escape closes the file viewer again: its layer sits above its dialog's, so it takes the Escape.
- The page's drop guard leaves a drag the open Upload dialog's zone accepted alone (it ran last and
  refused that drop).
- A capped list's server sort or search keeps the rows on screen until its answer is in (no skeleton
  under the pressed header), says "The first N items" when that answer is cut too, and an arrival is
  decided on the rows shown: one past the end of a cut list is said to be so, not gone.
- The search an arrival clears is taken at once.
- Tests: confirm names asserted per confirm, a fitting text pages under the arrows, a deleted file
  leaves the tray's finished list.
2026-10-13 02:32:50 +02:00
peter 54a6bbf07f fix(files): review round 2, the server — locks the GC and folders read under, the mount check, the old mount name
- GC: the expiry warning marks each file conditionally on the expiry it saw, one commit each (a file
  deleted meanwhile no longer fails the run; one re-timed keeps its fresh expiry unwarned); eviction
  counts the disk again when a candidate was freed meanwhile, so it never evicts for space already free.
- Folders: a retention change locks the folder and its files, and a finishing file reads the retention
  under a share lock, so neither misses the other; a folder delete locks the files it takes home (a
  file moved out meanwhile stays put, one deleted meanwhile no longer fails the delete).
- Upload: the folder is checked before the probe and held only once the job names it.
- Dedup enqueue locks the cached asset, so a release meanwhile yields a fresh one instead of a 500.
- Mount check reads the kernel's mount table instead of comparing devices (the vfs storage driver gave
  a mounted volume the device of /), and only inside a container; the API also logs a fresh, empty
  volume in the library's place.
- Compose files fall back to DOWNLOAD_HOST_PATH; the upgrade notes cover the from-source pull.
- Folder routes call owned_or_404 directly.
2026-10-13 02:32:49 +02:00
peter 8b2260c9a8 fix(files): review round 1, the page — honest states, the keyboard, the tray's words
- A library past the list's cap is searched and sorted on the server; the count says "the newest".
- Each folder and each search opens on its first page (DataTable pageScope, useCardPager scope).
- A file dropped where the page takes none never reaches the browser (which would unload the app).
- The viewer's arrows wait while a layer opened from it is on top, and scroll a wide text line;
  a text shows its first 5000 lines and says where the rest is.
- Confirms name the file (no more "your downloads"); the asking menu rows end in "…".
- The tray names a running step beside its bar, shows a pause mark for paused work, polls slowly
  while only paused work waits, and forgets a finished file that was deleted.
- An unchanged rename sends nothing; a paste opens the download dialog only with nothing on top;
  the link box never replaces what was typed.
- Honest states: folders that failed to load, a file no longer there, the storage figure; the rail
  says "nothing downloaded" only when there is nothing.
- Focus after deleting the open folder and when the format form opens; the format name's label.
- ui/Meter (one danger line for both quota bars) and ui/RenameField in the platform layer.
- Copy fixes, stale comments and the speculative hub key; a contract test for the list's
  parameters; e2e: the Files page and the paste intake.
2026-10-13 00:14:09 +02:00
peter 815083bcc4 fix(files): review round 1, the server — the tray's window, the GC's locks, a file's origin, the upgrade path
- The queue keeps a done job by when it became done (ready_at), not by its last write: renaming,
  keeping or moving an old file, or the GC's expiry warning, no longer brings it back as just finished.
- GC expiry and eviction re-read each file under the asset→job locks the user routes take, and commit
  one file at a time: a delete, a re-download or a keep made while the GC runs is honoured, never a
  second release that frees a file another user still holds.
- A file records the origin it was placed under (download_jobs.source_key, migration 0105): an edit
  whose parent expired still goes back to its folder, not to Uploads.
- A folder a file is being put in is held (FOR SHARE) until it lands; a delete of it waits and takes
  that file home too, instead of failing on the foreign key.
- The hash route reads at most two files at once (429 past that) and holds no connection meanwhile.
- 0104's backfill dates old files by their request date only: no notice prod wrote names its job.
- The old DOWNLOAD_ROOT still finds the files; a files root that is not a mounted volume stops the
  worker and is logged by the API; the volume-copy recipe looks the project-prefixed names up.
- Stale comments about the /downloads mount and the Downloads page.
2026-10-13 00:14:09 +02:00
peter 96509fe356 fix(files): a file is added when it is done, not when it was asked for
A download that failed for days and then went through sat days down the library, under its
request date, and read as missing. A job now records when it became done (`ready_at`), stamped
where the expiry already is (`folders.stamp_done`, `finalize_if_running`) and on an insert that is
already done; a check constraint holds every path to done to it. The library sorts and dates its
"Added" by it; the queue keeps the request order.

The migration dates a file from before by its "ready to watch" notice where the inbox still holds
one, else by its request date, where it already sat.
2026-10-12 22:19:37 +02:00
peter 132fb56428 fix(downloads): a site the downloader no longer understands says so, and offers a retry
yt-dlp asks for a bug report on an extraction failure it did not expect: the site answered in a way its extractor does not read (measured on a TikTok link: fails on every attempt under yt-dlp 2026.07.04, downloads under 2026.08.19). That failure read as "Something went wrong" with no retry; it is now its own category, site_changed, with a Retry in the failure window. A contract test holds every failure category to its copy in both languages.
2026-10-12 20:54:47 +02:00
peter f6bce23a94 fix(files): the Storage tab drops its user picker — every user already has a row and a menu
The list holds every user, files or not, so the picker above it only repeated a row's Edit quota. The empty text names what the list holds: users.
2026-10-12 20:06:30 +02:00
peter fb8fb980e5 feat(files): a file that is not media opens in a viewer — an image, a PDF, text as text
- GET /api/files/{id}/preview: an allow-list on the stored extension (downloads/preview.py) picks
  image / PDF / text; served inline under `CSP: sandbox; frame-ancestors 'self'`, the type from the
  table; HTML and SVG arrive as plain text; a text is its first 1 MiB (X-Preview-Truncated), sent
  as UTF-8 whatever it was written in (BOM, UTF-8, else cp1250); a NUL means binary: 415
- the list says which preview each file gets (asset.preview); FE/BE contract on kinds and header
- the viewer (shared dialog shell): ← / → through the list's such files in its order, Save, open in
  a tab, the file's menu; one stage height for every kind; a no-preview card offers the file
- the name, the card's picture and the menu's Preview open it; an image up to 8 MB is its own
  thumbnail
2026-10-12 15:28:31 +02:00
peter 6707550513 fix(files): a steady list header, every user on the Storage tab, the name gets the row's room
- The Files list starts at the scroller's top: the padding above it let the sticky header ride up
  12px on the first scroll, then stop.
- Users -> Storage lists every user with their quota, not only those already holding a file — a
  quota is set before anyone downloads.
- On a wide Files list only the "…" holds a place; Copy link, Share and Save come up over the date
  while the row is hovered or holds the keyboard (a card keeps them in its line). Keeps is narrower.
2026-10-11 20:33:48 +02:00
peter 078194731b feat(files): the Downloads page goes — the tray, a Storage tab and a Downloads settings tab take its place
- The Downloads module, its rail icon and badge are gone. A bookmark, the remembered page and an old
  history entry that name it land on Files (urlState.resolvePage); `g d` goes to Files too.
- Users page: a Storage tab — the disk in numbers, every user's use against their own limit with
  their file count, every user's downloads; aligned tables, one menu per row (edit / reset quota,
  open source, copy, the owner's quota). The admin storage rows carry quota + file count, the admin
  job list its owner.
- Settings: a Downloads tab — the download formats in one list (duplicate a built-in, edit or delete
  your own) and the site sign-ins, moved from Account.
- "View" on an added download, a clip or a converting upload opens the tray on that row (unfolded,
  back if closed; a finished one opens as a file; under a player it opens without taking focus).
- The download dialog opens on the format used last.
- DataTable: `pager={false}` for a short list, `cardRow` to give a lone "…" its own card line.
- Welcome and README speak of Files; a new Files screenshot.
2026-10-11 19:37:13 +02:00
peter 1c4f0042d2 feat(files): download from anywhere — a link box, a pasted link, the palette; the running downloads in a tray
- Files toolbar link box (paste button, Enter); a lone http(s) link pasted onto any page opens the one
  app-wide download dialog, which now names the link; the palette's "Download from a link…" focuses
  the box on its history entry.
- The download tray, bottom right on every page: what waits, runs, is paused or failed (reason +
  Retry, the queue's right-click / "…" menu), and what finished in this tab as "Done · Open" until
  closed; folds to a pill (per-account pref); closed, it stays shut — across a reload — until the
  work changes or something finishes. Back to top steps up over it, the page end keeps clear.
- A finish is announced once: "Downloaded: …", whether a phone plays it, Copy link + Open (toasts
  and notices may carry a second action).
- The inbox's ready notice opens the file in its folder: download_done carries job_id + folder_id.
- GET /api/downloads is the queue only (in flight, failed, finished in the last 10 minutes), with
  each row's source key.
2026-10-11 16:56:13 +02:00
peter d22c64f4dd feat(files): folders managed in place, a grid per folder, drop to upload, open on one file
- Own folders are made, renamed and deleted from the rail (right click, long press,
  Shift+F10) or the "…" beside an open folder's name; any folder's retention is a
  submenu (7/30/90 days, until deleted). Deleting confirms where its files go.
- List or grid per folder, sharing one sort; the grid is the one media card.
- A file dropped anywhere on the page opens Upload with it, bound for the folder
  on screen.
- A link (?item=) or the upload notice opens Files on the file: its page turned
  to, the keyboard on it, a ring round it.
- An in-place name field left by Enter or Escape hands the focus back to what
  replaced it (the row, the entry, the "…") instead of dropping it on <body>.
- The table's saved page size is per account.
2026-10-11 15:59:12 +02:00
peter 85aa917b38 feat(files): the Files page — the library in one aligned list, its folders in a rail
Files is the one library: every done file, the ones shared with me, or one
folder's, picked in a side rail (sources, own folders, the storage meter) and
mirrored to ?folder=. One table with every kind of value in its own column;
Copy link, Share and Save in sight, every action on a right click: play,
cut, rename in the row, move to an own folder or back home, keep, details,
re-download, delete.

The Downloads page keeps only the queue until the tray replaces it. The old
/api/downloads library routes are gone: list shared, upload, the details
PATCH, delete, the file and the usage now live under /api/files, and a
listed file says which source it came from. The job's "expired" flag is gone
with the state it described.
2026-10-11 05:34:25 +02:00
peter 1a1665b8ec feat(files): the Files library API — folders, moving and keeping files, upload into a folder, the file's hash
/api/files is the library's home: list a user's done files (by folder, searched, sorted, or the
ones shared with them), rename / move / keep / delete a file, make, rename and delete own folders
(a deleted folder's files go back to their source folders), and set how long any folder keeps a
file — done files shift by the difference, never closer than the GC's warning window.

An upload can name its folder (an own folder or Uploads), checked before the body is read and again
before the file lands; it is hashed while it streams, and any file can be hashed on first ask. The
download says the hash (Repr-Digest) once it is known; a new file at an asset's path forgets it.

Uploads, deletes, keeps, moves and folder changes are audited (ids only). The fileless "expired"
rows of the old retention are deleted by migration 0103: an expired file now leaves the library.

The shared route helpers moved to routes/files.py; the old /api/downloads library routes delegate
to the new ones until the client moves over.
2026-10-11 04:42:55 +02:00
peter bf99444ded feat(files): every file sits in a library folder, and keeps its own expiry
The Files library starts here. A user's every download, clip and upload now
lands in a source folder made per origin (YouTube, Plex clips, uploads, one per
other site), decided once in downloads/folders.py at every place a job is made.
How long a file stays moves from the shared asset to the user's job: stamped
from its folder when it is done, never for a file the user keeps. The GC warns
each user once, then takes the expired file out of their library; the cache cap
never evicts a kept file, and an evicted one leaves every holder's library.

The media mount is /files now (FILES_ROOT, FILES_HOST_PATH, volume
siftlode_files); docs/self-hosting.md says how to carry an old volume over.
Migration 0102 places every existing job and carries each done file's expiry
over from its asset.

safe_abs_path takes a missing path and answers None, which drops one baselined
type error.
2026-10-11 03:30:34 +02:00
peter 07e0990ecf design(files): the one-library round in the lab — aligned list columns, the tray, the done notice
The Files page replaces the Downloads page: list rows with one column per value kind and a
right-click menu, an app-wide download tray, a done notice that copies the link. Picked
variants are logged in DESIGN.md.
2026-10-11 02:59:51 +02:00
peter e2ed2bf874 merge: chore/e2e-retry-connect into E39 (rides the next release) 2026-10-11 02:30:16 +02:00
peter b9eb8458b3 test(e2e): a localhost connect the machine dropped is tried again on the test side
On a Windows host a burst of test traffic can drop a loopback connect now and then (every port,
any program — measured on the dev desktop). The browser resends it and only waits; Node (libuv)
sends a loopback SYN once and fails at ~300 ms with "connect ETIMEDOUT". `e2e/retryConnect.ts` wraps
Playwright's request context so that one error is tried again (0.5 / 1 / 2 / 4 / 8 s, the system's
own retry waits) while every other failure goes straight to the test; global.setup's health check,
login and /api/me and speed.measure's share probe go through it. Unit-tested; e2e 61/61.
2026-10-11 00:45:13 +02:00
peter 891507dc75 release: 0.105.0 (feature/e38-ad-hoc-clips) 2026-10-10 23:38:13 +02:00
peter e6894ac0fc release: 0.105.0 — ad-hoc clips 2026-10-10 23:13:22 +02:00
peter 663a7e8012 fix(player): the Plex player's fills hold on its chrome: an amber head on a white played part
A light theme's accent as a fill on the always-dark chrome was ~2.8:1 (Slate), under the 3:1 a
control needs. A native checkbox and slider tint with `accent-chosen` (the accent elsewhere, the
media amber under `media-marks`), and so does the skip button's countdown. The progress bar plays
white with an amber head: amber there marks a place, as the intro / credits markers do, and an amber
played part made a marker ahead of the head read as already played (owner's choice of three).
2026-10-10 22:56:09 +02:00
peter 93192dadd8 fix(player): a chosen tab's underline and an open or pressed icon read the scope's chosen colour
The underline named `--color-chosen`, which is resolved once on :root, so it stayed the accent under
`media-marks` (seen in the browser: fuchsia on Slate light); it reads `--chosen` where it is used
now. An icon button pressed or with its menu open tinted its glyph with the accent; it takes the
chosen colour — the accent everywhere but the player chrome, where it is the media amber.
2026-10-10 22:26:22 +02:00
peter 5b48018e92 fix(downloads): a produced file is held to the space left by its real size
yt-dlp's `max_filesize` only stops a plain HTTP file that announces its size: YouTube's fragmented
formats never meet it — measured on localdev, a 718 MB download ran to the end with 1 MB left and
put its owner four times over the cap. The download now re-reads what is left once its file is made
(after the phone-safe pass) and refuses one larger than that as the owner's refusal; an editor clip,
which nothing checked, is held the same way, as a player clip already was.
2026-10-10 22:10:56 +02:00
peter 8312c6d592 fix(player): what is chosen on the player chrome is marked amber, not written in the accent
A light theme's accent as text on the always-dark chrome was 2.84:1 (Slate). `--chosen` is the
accent unless a scope names its own; `media-marks` (was media-focus) points it, with the focus
ring, at the media amber. A chosen track stays white with an amber tick (MenuCheck reads
`text-chosen`); a chosen settings tab or toggle is white with an amber underline (MEDIA_CHOSEN).
The theme contract test holds the token, the tick and the player components free of accent text.
2026-10-10 21:56:47 +02:00
peter e1d325b942 fix(downloads): a download larger than the space left says so, not "processing failed"
yt-dlp refuses a file announced over `max_filesize` by logging one line and returning with no file
(measured, and pinned in test_formats against a local server): the worker found no output and
failed the job as `postprocess`. `formats.watching_size_cap` now sees that line through our yt-dlp
logger, and the download (and, should it ever happen, a YouTube clip's range fetch) fails as the
owner's refusal with DOWNLOAD_TOO_BIG_MESSAGE, classified `quota_exceeded`. The download's quota
copy says it does not fit in what is left — true at a full cap too — instead of "Storage full".
A pre-existing type error the change touched is fixed (baseline down by one).
2026-10-10 21:56:47 +02:00
peter 977ba244fa fix(clip): a time's error line sits in the flow, the remove button off the cut, and every player surface rings amber
- useTimecodeField holds a time field's draft for its caller, which places TimecodeError (role
  alert) in its own flow under the fields' row: it no longer lies over the crop row taking its
  clicks, runs out of the editor's card, or joins the field's <label> name.
- The editor's remove-cut buttons sit in a row under the track at their cut (trackLeft), off the
  cut's grab area, where a press drags it; RangeTrack's adornment slot is gone.
- media-focus re-points --focus to the media amber on the Plex player, the YouTube stage and every
  bg-media-chrome surface; a contract test holds each such class string to media-focus or
  media-scope, the field's focus edge to the focus token, and --color-focus to the inline theme.
- An upload's quota headline says the converted file does not fit, not "Storage full".
- DESIGN.md: the tokens paragraph says what is re-pointed where; the decision row measures all
  three colours on the chrome.
2026-10-10 21:29:55 +02:00
peter 89d18b46a6 fix(downloads): a converted upload too big says so, and a suspended owner's waiting rebuild is not "preparing"
- A converted upload over the space left (not zero) fails with CONVERTED_TOO_BIG_MESSAGE, not
  "storage is full"; it classifies as quota_exceeded by the phrase it shares with the clip's.
- The public watch page says "preparing" only for a job a worker will take: a suspended owner's
  queued job waits for as long as the suspension lasts, so the link says the video is gone, as
  for a paused rebuild.
- The worker's claim docstring names the restart case: a running job put back in the queue waits.
2026-10-10 21:29:55 +02:00
peter c27cfdcb9c fix(clip): typed times belong to their value, and every mark on the player chrome can be seen
TimecodeField drops a draft when the value moves under it (another segment selected, a nudge), so
leaving the field never commits it to another cut; it is invalid only once offered, with a line
under it saying how to type a time. On the always-dark chrome media-scope points every focus ring
at --media-mark and errors at the new --media-danger (a light theme's accent and danger were under
3:1 there). A track whose dragged handle unmounts ends the drag on the pointer's release instead of
locking up; a remove refused while clips are queued leaves no focus to take later; each remove-cut
button names its cut and is a 24px target. DESIGN.md records the media tokens; stale comments fixed.
2026-10-10 20:28:52 +02:00
peter a867145bcb fix(downloads): a suspended user's queued jobs wait, and a clip too big for the space left says so
A suspension now blocks the user's downloads as quota.resolve already advised: the worker leaves
their queued jobs unclaimed until it is lifted (the confirm says so). A cut clip larger than the
remaining storage is refused with its own message (not "storage is full"), and a clip's quota
refusal reads as a clip's in both languages. The cancel poll compares to JobStatus.RUNNING.
2026-10-10 20:28:52 +02:00
peter 4d4eea2c54 fix(clip): marks on the player chrome keep their amber in a light theme, and a removed cut hands the focus on
- A media amber (`--media-mark`), the same in both themes, for what marks a place on the always-dark
  chrome or over a video: the clip strip's playhead and focused end, the crop frame's ring, Plex's
  intro / credits markers. A light theme's warn was a dark amber there, under 3:1 (contract-tested).
- The editor's remove-cut button rings inside itself (the track clipped its ring), and once it goes
  with its cut the focus moves to the cut beside it, or to Split here, not to <body>.
2026-10-10 15:57:02 +02:00
peter c489b60c8b fix(clip): clip mode keeps its range, its focus and its keys through the player's own moves
- A mark on the very end seeks a moment short of it; while a clip is marked the end of the item
  neither closes the player nor steps the queue, and Plex's auto-skip waits.
- Plex: Escape and Back peel what is over the picture (info, up-next, the countdown) before the
  clip; Back under a dialog is the dialog's; no range opens while the length is unknown.
- YouTube: the focus goes back to the scissors the user can see, else the player; the panel keeps
  its name, Loop and focus when the view is maximised or restored under it.
- A typed time stays in its field until taken, then shows where it landed; unreadable stays marked
  invalid; a decimal comma is read (ui/TimecodeField, the editor's fields too).
- RangeTrack: a drag is its pointer's, a cancelled drag reports final, an adornment sits beside the
  slider, a caller-chosen focus ring (amber on the clip strip). CropFrame: a drag is its pointer's.
- The editor's remove-cut is a real button at the cut's foot; a segment's keep toggle sits over the
  cut's hit area.
2026-10-10 15:35:21 +02:00
peter b4bf71ea64 fix(clip): a clip is held to its owner's storage cap, and its limits and failures say the server's truth
- The cut file is measured against what its owner has left, for a YouTube range (whose fetch never
  reads yt-dlp's size ceiling) and for a Plex clip (which fetches nothing), before the fetch and
  again before it is placed; a refusal leaves the asset pending for a Retry.
- A silent Plex file is cut silent; a file the probe cannot read says so.
- A second job on a failed or renamed clip gets the same YouTube diagnosis and the fetched title.
- The "too long" refusal names the server's cap; a contract test holds the client's copies of the
  clip limits to clip.py; the shipped player clients are pinned against the installed yt-dlp.
- One HDR transfer set and one rating_key lookup; job status words from JobStatus; the display-name
  guard sees subscript writes and the clip route; stale comments and types follow the clip job.
2026-10-10 15:35:12 +02:00
peter 72df11f2c0 fix(ui): a small crop frame takes smaller handles, so it can still be grabbed to move
The handles reach 10 px into the frame from every side; a frame at its minimum on a small picture
(the Downloads editor at 800 px: 41 x 23) was then nearly all handle - 74 of its 920 px still moved
it. Below 64 px on its shorter side the frame takes 12 px corners and 8 px edges: 262 px move it.
2026-10-10 13:54:04 +02:00
peter 0a16bc40ae fix(ui): a crop frame grabbed with the pointer takes the focus, so the arrows go on from there
The press is held back from the page (no text selection, no play / pause underneath), which also
kept it from focusing: after a drag the arrows still went to whatever had the focus before, and
Escape did not reach the frame. The press now focuses the frame or the corner it grabbed (an edge
focuses its frame).
2026-10-10 13:38:49 +02:00
peter 520b0afb90 feat(clip): crop a clip to a frame drawn on the picture, and give the clip panel the room it needs
An optional crop in clip mode: a Crop toggle puts a frame on the picture (free, 16:9, 9:16 or 1:1;
the whole picture stays the default). The frame is sent in fractions of the box it was drawn in,
with that box's shape; the worker fits the real picture into the box by its display aspect (sample
aspect and rotation aware), drops the part over the bars, and crops first in the filter chain. A
frame covering the whole box is no crop at all, and one off the picture is refused by name.

The crop frame is a ui primitive (CropFrame) shared with the video editor, keyboard-operable.
YouTube: the wide side panel shuts while clipping and comes back after; maximised, the clip panel
docks in the stage. Plex: the picture sits above the clip panel so the frame lies on it.
2026-10-10 12:52:59 +02:00
peter e301632765 fix(clip): a clip refusal speaks the user's language, and a choice in the clip form no longer sends it
The backend refuses a clip with a structured reason, but the client showed every one as the generic
"couldn't make the clip": each reason now has its own message in both languages, held to the
backend's reasons in both directions by the contract test. A segmented toggle's and the priority
stepper's buttons had no type, so inside a form one of them submitted it.
2026-10-10 12:52:12 +02:00
peter 50157cc2fd feat(player): clip mode - mark a range of what is playing and cut it into a download
Both players get a scissors in their control row (Plex: the control row; YouTube: the actions and
the docked bar), `c`, and `[` / `]` to mark at the playhead. The clip panel lives in the player, so
the watched video is the preview: an overview of the whole item and a zoomed strip around the
selection whose ends are sliders (arrows step a tenth, Shift a second), Start / End fields with
nudges and "to the playhead", the length, a looping Play selection, a name, Cut. Escape closes the
panel before the player and the focus goes back to the scissors. A cut is a toast with a Downloads
link plus the usual Ready notification.

- ui/RangeTrack + RulerTicks: the shared, domain-free time track (regions, keyboard-steppable
  handles with a 1.5rem hit area, a playhead, a background slot, over a window of the item). The
  Downloads editor's cuts moved onto it, so they are keyboard-steppable now.
- lib/clipRange holds the marking rules (1 s to 10 min, the server's), lib/timecode the editing
  timecode the editor's private tc/parseTc became.
- keymap: a symbol typed through AltGr answers its bare-symbol row - on the HU layout `[` is
  AltGr+F, which arrives as Ctrl+Alt; letter and Mod rows still never match Ctrl+Alt.
- The YouTube stage sits in a wrapper with the panel, so the maximised stage's blur escape matches
  at any depth.
- media-scope: fields on the always-dark player chrome read white in a light theme too.
2026-10-10 03:58:27 +02:00
peter f221c1962a fix(downloads): a YouTube clip goes with the user's own cookie and is diagnosed like a download
A clip fetched its range signed out, so a video behind YouTube's per-video "not a bot" wall could
never be clipped while its download went through with the user's cookie - and the failure copy told
the user to add a cookie the clip would not use. The clip asset is private by construction
(`clip.asset_ref` names its user), so `usable_clip_cookie` attaches the user's YouTube cookie on that
identity, and a failed range fetch now carries the download catalog's diagnosis
(`youtube.bot_wall` / `youtube.cookie_rejected`) instead of a bare category.
2026-10-10 03:58:27 +02:00
peter d466ae915b fix(downloads): a YouTube clip outside the catalog takes its title from the fetch
It was queued as "Clip (clip)" and stayed so: nothing renamed it once yt-dlp had answered.
2026-10-10 02:43:13 +02:00
peter 14598ec320 feat(downloads): ad-hoc clips cut straight from a YouTube video or a Plex item
POST /api/downloads/clip takes a source (a YouTube id or a Plex rating key), a range of at most
ten minutes and, for Plex, the audio track being heard. The clip becomes a per-user asset in the
library like an editor clip, shareable as soon as it is ready. A YouTube clip fetches only its
range (yt-dlp download_ranges: 3-7 s whatever the video's length); a Plex clip reads the library
file in place. Both then go through one re-encode: frame-exact at the marks, H.264 at most 1080p
with HDR tonemapped to SDR, stereo AAC, no chapter track, .mp4.

Job kinds are a JobKind StrEnum now, and Re-download refuses every kind but a download.
2026-10-10 02:31:38 +02:00
peter b16cf7668c fix(downloads): YouTube downloads reach their profile's height again
The primary player client, android_vr, was still offered 1080p and 2160p, but every fetch of
those formats died with HTTP 403: it now needs a stream PO token the bgutil sidecar cannot mint.
The fallback set then served 360p only, so every download finished at 360p without failing.
tv_simply is offered the full ladder and its fetch completes (measured: 1080p, and a localdev
download through the pipeline reached the default profile's 720p). yt-dlp 2026.8.19 marks
the clients that need such a token, so it stops fetching into the 403.
2026-10-10 02:08:43 +02:00
peter 3438aeb695 fix(player): the player's own focus grabs leave a tabbed-to control alone
Turning ready, the pointer leaving the video, the window regaining focus, a wheel over it and
leaving fullscreen each put the focus back on the card unconditionally, so a viewer who had
tabbed to a control right after opening started over from the first button on the next Tab.
They now take it back only when it sits in the iframe or off the card.
2026-10-10 01:14:31 +02:00
peter c003a48244 release: 0.104.0 (improvement/adhoc-batch-oct) 2026-10-10 00:24:27 +02:00
peter 8a2334e198 release: 0.104.0 notes and version 2026-10-10 00:15:23 +02:00
peter 1f010328df fix(plex): plain text survives a decimal reference too long to parse
int() refuses a decimal run past 4300 digits, so one such reference in a summary raised out of the sync and the text migration. Any number past seven significant digits is beyond U+10FFFF, so it becomes U+FFFD like any out-of-range reference; leading zeros are dropped first. The rails test's comment now says what it guards: both rails agree with the rollup.
2026-10-10 00:03:08 +02:00
peter 77638f7a0b fix(player): a link that opens the cut text is scrolled back into view
Focus had scrolled the clamped box itself; opened, the link dropped by that much, out of a scrolling panel's sight. DESIGN states where Close sits in the stacked layout; a stale test comment goes.
2026-10-09 23:46:40 +02:00
peter 2cff4ad954 fix(feed): an override yields only to a refetch that set out after its write was answered
Send order let a refetch read before an unanswered mark committed and drop it. A reset's position 0 is written again after a refetch that set out before the server took the reset. The player-session refetch also waits for the player's closing checkpoint, which the player now tracks (progressFlushes); refetched first, the list put the older position back.
2026-10-09 23:46:39 +02:00
peter 8776586ca3 fix(plex): a matched episode and a collection member offer Details in their menu
Their titles open the item's page, a pointer-only door; the card menu is the keyboard's.
2026-10-09 23:46:39 +02:00
peter 92a17aba83 fix(plex): a reshuffle survives a refresh landing mid-deal; the ↻ spins for a deal only
The ↻ intent was spent as its fetch set out, so a watched toggle's resync cancelling that fetch re-ran it as a re-read of the old draw. It now lasts until the latest deal answers. The button spun on every re-read of the same titles; it now follows the deal. Comments that said the Surprise draw is never refetched are brought in line.
2026-10-09 23:46:39 +02:00
peter fc53bc03bc fix(plex): the rails' Restart comes from the one rollup; plain text decodes whole entity names only
Both Continue-watching rails built a show's card state inline; they now take it from rollup.show_state, so a show part watched with nothing to resume offers Play alone there too. A named reference counts only when its whole name is an HTML5 entity: html.unescape still decoded the legacy "&not" inside "&notice;". Tests cover the grid's in-progress-but-not-resumable case through its SQL and both rails.
2026-10-09 23:46:39 +02:00
peter 6df257d7df fix(plex): Restart only where an episode has a resume point; the Surprise draw is re-read, not redrawn
Show cards carry resumable (ShowState, from the same rollup) so a part-watched
show whose next episode is untouched offers Play, not a Continue and a Restart
that do the same thing; plexStarted is the one rule for cards and menus. The
Surprise rail's key sits under the discover root: whatever refreshes the rails
re-reads the draw (keep=) with its watch state now, kept past the 5-minute gc,
and only the reshuffle deals. Closing the player refreshes the Discover rails.
A show card's play acts on the last click only, and only where it was made.
2026-10-09 22:01:28 +02:00
peter 5572499b56 fix(player): the shut panel keeps the dialog's name; Tab onto a cut link opens the text
With the side panel shut the title heading was gone, and the dialog it named
went unnamed: the strip carries an sr-only copy. A link below a clamp's cut
stayed tabbable out of sight; reaching it now opens the text, measured from the
text's top since focus scrolls the clamped box. DESIGN describes the modal as
built, with log rows for its chrome and the wheel rule.
2026-10-09 22:01:27 +02:00
peter 76738f8ea9 fix(plex): plain text decodes only well-formed character references
html.unescape also decodes the legacy forms without a semicolon, so prose like
"AT&notice" came out as "AT¬ice". The show and collection sync paths are
now tested too.
2026-10-09 22:01:27 +02:00
peter 568bf93510 fix(feed): an optimistic write outlives only the server answers older than it
A reset wrote a position mask over the cached list, so a replay's saved
position stayed hidden: a reopen resumed at 0 and its first checkpoint wrote
over the progress. The reset now writes 0 into the cache, where the player's
closing patch writes the next one. Overrides carry their write order, and a
landing refetch drops only those written before it set out. The player
session ends after the modal unmounts, so a clip auto-marked watched as it
closes still refetches the list.
2026-10-09 22:01:27 +02:00
peter 942154cf49 fix(plex): a Surprise draw stays put until its reshuffle
The rail's query was always stale, so Back from a card's player or page remounted it into a new
draw, as did a return to the window: the card just looked at was gone. It now stands until the
reshuffle (or another scope, language or filter) asks for a new one.
2026-10-09 01:29:12 +02:00
peter 3eb87e7108 feat(plex): every show card plays the episode it would put on
A show card's picture opened the show page everywhere but Home, whose shelves carried the
resolved `next`. Now every Plex picture plays: a show asks `GET /api/plex/show/{id}/next`
(the in-progress episode, else the first unwatched, else S1·E1 — the rule Home and the show
page's Resume follow) on the click, fresh each time. A started show shows Continue + Restart;
its title, an ⓘ and the menu's "Open show" row open its page; one with no episodes opens there.

The question is asked per show, never carried on every card: three probes a show would cost a
grid page up to 180 queries. The hover preview asks it once it opens and names the episode
("Next: S6·E16 — Title"). Related shows on a show page play the same way.
2026-10-09 01:22:00 +02:00
peter 851b97b09d fix(ui): a card's hover chrome answers keyboard focus, not any focus
Back from a channel page hands the card its focus again after a mouse trip;
revealed on focus-within, its hover row stayed up under no pointer until a
reload (owner). The row, the Play layer, the Continue + Restart pair and the
global .reveal rule now show on hover or on :focus-visible on the card or
inside it, so Tab and the arrow keys still reveal it.
2026-10-09 00:44:46 +02:00
peter ee94d1949a fix(feed): a video marked watched in the player stays out of an Unwatched view
Marked watched in the player (by hand, or by the player at its end), a card
left an Unwatched list at once, then came back as "new" when the player
closed, until a reload. The player's closing position patch writes into the
cached feed; the feed took the bumped dataUpdatedAt for fresh server data,
dropped its optimistic overrides, and nothing refetched. The overrides now
clear only when a refetch lands, and a player session that changed a state
refetches the list and its count by the view's filters on close, once its
writes have landed.
2026-10-09 00:44:46 +02:00
peter 4f076b5d45 feat(feed): a card's menu filters the feed by its channel, and back
Under "Open the channel" a video card's menu (the right-click on its
channel name included) adds the channel to the feed's channel filter, or,
when it is already in, takes it out. Channels OR together, so it widens a
filter that names others, as "Show in feed" does. Not on a channel page,
whose view is that channel already, nor in a live YouTube search or on Home.
2026-10-09 00:44:46 +02:00
peter ece74a42ab feat(player): a roomier YouTube player, its panel shut on demand
- The full-height previous/next strips beside the card are gone: the panel's
  transport row and Shift+arrows step on their own, and the video gains the
  7rem the strips' gutters took.
- Close is one icon in the card's top-right corner (the wide panel's head,
  the stacked card's title row).
- The panel's description is cut to three lines with More / Less
  (ClampedText now takes the linked text as children).
- Wide, the panel shuts to a 2.75rem strip (close, open, previous, next) for
  a bigger video; it stays shut while the queue steps on, and a video opened
  from a card starts it open. Focus moves to the counterpart toggle.
- The wheel has one meaning per place: over a panel (or the stacked card)
  with something to scroll it only scrolls, never falling through to the
  volume at the end; over the video, or a panel that fits, it is the volume.
- A long channel name is cut on the panel's line instead of running the
  20rem panel into a sideways scroll.
2026-10-09 00:27:29 +02:00
peter 600d5c39da feat(ui): a button over a picture answers the pointer with an accent ring
The media variant's hover darkened its body from 60 to 75% black, which no
one could see on a poster; a rail's scroll arrow over a card could not be told
from the card's own Play under it. Its hover is now a 2px accent ring over an
80% body (owner pick B), on every card control, Plex and YouTube, and the rail
arrows. DESIGN.md B3 names it beside the edged pills, with its log row.
2026-10-09 00:27:28 +02:00
peter 4c7f73af9d feat(plex): a title opens its page, the picture plays; an episode links up
A card title lit like a link but played the item. Now the picture plays and
the title opens the page of what it names: a movie's or an episode's info
page, a show's page, and for an episode standing as its show (Recently
watched) the show. Every poster and episode card carries it (grid, rails,
person, collection, season, related, Home), a card with no page keeps a plain
title, and the season page gains the menu's Details row too.

The episode page had no way up: the detail now serves the show's and the
season's rating keys, and on the page the show title opens the show and
"S1·E25" the season (Back returns to where the page was).
2026-10-09 00:06:18 +02:00
peter 5f42ff61c8 feat(plex): a Surprise reshuffle deals the previous draw last
The rail draws three unseen titles per kind under the sidebar filters; on a
small filtered pool every reshuffle dealt the same few again. The redraw now
sends the draw on screen (exclude=), the server deals the rest of the pool
first and tops up from the excluded titles only when the pool is short, so a
small pool still fills the rail. (The two discover.py baseline entries went
with the rewritten draw.)
2026-10-09 00:06:11 +02:00
peter 6dd82ef5e2 fix(plex): summaries and taglines are stored as plain text
A Plex agent can hand a summary over as HTML (<p> paragraphs, &aacute;
entities); the info page and the hover card showed the tags. One cleaner,
app/plex/plaintext.py, now runs on every summary and tagline the sync, the
Hungarian overlay and the rich-detail fetch store, and migration 0101 cleans
the rows written before through the same function. Paragraph breaks survive
as newlines, which the info page and the hover card now keep.
2026-10-09 00:06:02 +02:00
peter 75f4efde98 release: 0.103.0 (bug/yt-fullscreen-button) 2026-10-08 22:50:52 +02:00
peter c0059a0c4b release: 0.103.0 notes within the item length cap 2026-10-08 22:36:15 +02:00
peter a29cadf03e release: 0.103.0 notes and version 2026-10-08 22:30:56 +02:00
peter 83886494ce fix(ui): a membership menu re-homes focus a frame later, once its new rows are on the list
Rows that just mounted join the menu's navigation list a render after the status effect ran: a row
focused that early was unknown to it, and the arrows did nothing from it (seen live after a keyboard
Retry). The retry test now arrows on from the row it lands on.
2026-10-08 22:16:47 +02:00
peter 4eaa55385f fix(plex): a collection kept after a failed refresh holds its seed title
When Plex made the collection and only the read after it failed, the kept row had no members: the
title showed unticked in the collection it had just been added to, the collection empty, and the
name stayed in the field for a retry that could only be refused as taken. The seed's row now names
the collection (child_count 1), and the panel clears the field on that refusal as on success. Also
held: the panel's lines sit beside its list, and a keyboard Retry lands on the first row.
2026-10-08 22:12:43 +02:00
peter 7541b1b016 fix(ui): membership menus keep their lines beside the list and focus in the panel
The loading / failed-load lines and their Retry sat inside role=menu, which may own its items only:
the three surfaces now open with `listInside` and the rows sit in a PopoverList. A keyboard Retry
unmounted the focused button and left focus on <body> with the menu open, the arrows dead: focus is
put back on the first row, else the field, else the new Retry. The channel tag editor said a failed
tag load as "no tags yet"; it reads the catalog query's state now. The playlist panel's name and the
tag editor's error state are held by tests.
2026-10-08 22:12:43 +02:00
peter 24ad17fc7d test: hold the show collection's type, an unknown title's 404 and Brave's fullscreen stand-in 2026-10-08 21:18:48 +02:00
peter 48bd038c5f fix(plex): a collection created on Plex keeps its editable row when the refresh read fails
The rollback dropped the new row: the next sync recreated it read-only, out of every 'add to' list, and the duplicate-name check let a retry make a second same-named collection on Plex.
2026-10-08 21:18:48 +02:00
peter 0fce732497 fix(plex): a playlist row's menu adds its title to a collection too
The row menu called the title's action list without the anchor the panel opens beside, and the optional parameter dropped the row silently. The anchor is now required, so every host menu decides.
2026-10-08 21:18:48 +02:00
peter 7e94da72b9 fix(ui): one membership menu for playlists, collections and tags
Three hand-copied panels had drifted: Add to playlist said a failed load as 'no playlists yet'. ui/MembershipMenu now holds them: a loading line and an error with Retry instead of a blank or an empty list, the up/down arrows step from the name field into rows that arrived after the menu opened, and the panels opened from a menu carry their own accessible name. A write that landed flips its tick at once, so a second tap before the refetch undoes it instead of repeating it; a collection edit also refreshes a show's own page.
2026-10-08 21:18:48 +02:00
peter fadb4f17a9 fix(plex): subtitles draw only the player's own track, lifted in the player's own pixels
A direct-played file's own subtitle stream (WebKit lists an mp4's mov_text) was switched to hidden and drawn under any choice, Off included: only the player's <track> is kept hidden and read, other subtitle tracks are switched off. The lift was measured in screen pixels and applied inside the x1.25 windowed player, and re-measured off a box carrying its own (or a mid-slide) transform: an untransformed outer box is measured, the distance divided by the scale, and a cue that just appeared is placed, not slid from where the last one was left.
2026-10-08 21:18:48 +02:00
peter 57a81c5643 fix(plex): subtitles survive hls.js and a remounted track
hls.js emptied the cues of every text track on the element when a manifest
loaded or the media attached, our own <track> included, so a VTT that loaded
first showed nothing for the session; its subtitle and caption controllers are
no longer built. A remounted default <track> was switched to showing by the
browser after the hide had run, drawing the cues natively under ours; the hide
now follows every added track and mode change.
2026-10-08 19:35:47 +02:00
peter dae0b80f6a feat(plex): add a movie or show to a collection from its card menu
"Add to collection…" opens a panel beside the card: the title's library's
manual collections tick it in or out, and a name + Enter creates a new one with
the title in it. It used to take the Collections page, a new collection, its
name and a title search, then another search per title. A lean endpoint lists
the collections with whether each holds the title. Admin only, as on the
title's page.

Also: a collection seeded with a show is made as a show collection (Plex type
2). Measured: as a movie one it came out empty and refused the show after.
2026-10-08 03:09:55 +02:00
peter 75093458c9 feat(plex): subtitles rise above the control bar while it shows
On an iPad the bar covered the text whenever it was up: the browser places a
native cue by its line, which nothing animates. The track now stays loaded as
hidden and PlexSubtitles draws the active cues, measured against the bar and
lifted above it in 200 ms, back down when it hides. The viewer's size, colour,
background and outline settings style the text directly. Also fixes a test
typing slip from the search pill change.
2026-10-08 02:59:13 +02:00
peter 5db6dc8ee6 docs(design): log the touch header search 2026-10-08 02:52:39 +02:00
peter 45a9ceae25 fix(ui): the header search has room on a tablet
On touch the 44 px key cap and Go filled a 44 px pill edge to edge and the
placeholder was cut. The key cap is left out on touch (⌘K still works), the
pill grows to 52 px so a ring shows round Go, the box's floor counts the
controls beside the text, and Go drops to its icon by the column's width,
not the screen's.
2026-10-08 02:52:32 +02:00
peter 5249c41edc fix(player): a fullscreen button the browser cannot honour is not shown
Brave on iPadOS switches element fullscreen off, patches requestFullscreen with
a video-only stand-in and fakes fullscreenEnabled, so the Plex fullscreen button
and the YouTube docked bar's did nothing. Support is now read off
Element.prototype and the buttons are left out without it (owner). A stand-in
that returns a boolean no longer throws. A test's bare matchMedia stub no
longer outlives its describe.
2026-10-08 02:50:51 +02:00
peter c3664826fe fix(ui): a caption never opens under a finger
On an iPad a tap's synthetic mouseenter opened the Tooltip; WebKit then took the
tap as a hover and swallowed its click, so every rail button needed two taps,
and with no mouseleave from a finger the caption stayed up. Tooltip now opens
for a mouse and a key's focus only (owner: no captions on touch).
2026-10-08 02:45:48 +02:00
peter c9b8fe15a3 fix(ui): a scroller with no edge to fade carries no mask
iPad WebKit kept a scrolled container's mask stuck to its content: at the
bottom of a scrolled page the last strip, a table's pager, sat under a dark
band. With nothing to fade there is no mask; isolation keeps its layer.
2026-10-08 02:43:58 +02:00
peter 1044eaf8a9 fix(plex): the player holds the page still and covers it whole; the row drops the countdown
On an iPad a drag on the video scrolled the page behind the player, and in Brave
dvh did not follow the toolbar, so a strip of the page showed under it. The
page is now locked without overscroll on a black root while the player is open,
and the player is sized from its fixed box. The time row keeps the position,
the length and the end clock; the countdown is gone (owner).
2026-10-08 02:42:31 +02:00
peter ca949d22de fix(player): the channel line wraps instead of squeezing the meta into a column
On a narrow card (a tablet, a zoomed window) the unwrapped row of name, meta and
actions left the meta a one-word column. The row now wraps: the meta keeps at
least 12rem, and the actions move down together.
2026-10-08 02:25:58 +02:00
peter 20178f0ec4 release: 0.102.0 (improvement/e37-player) 2026-10-08 01:58:01 +02:00
peter 49a9319f84 fix(player): the click layer clears only the fullscreen button's column, not a full-width band
Measured: across the full width YouTube needs only its seek bar (up to 96 px
from the bottom); the 140 px is the fullscreen button's, at the right end. A
full-width 140 px band handed clicks on the picture to the iframe, which then
takes the keyboard focus and our shortcuts with it. The layer is now two boxes,
hidden from assistive tech (a mouse surface; the keys are its shortcuts), and
the tests render the big view too, where the docked bar sits under the iframe.
2026-10-08 01:42:39 +02:00
peter 70d56ab939 fix(player): YouTube's fullscreen button stays clickable on a short player
The click layer over the embed left the bottom 22 % of the stage free. YouTube's
new embed keeps its fullscreen button 80-128 px above the iframe's bottom, so on
a stage under ~580 px (a small window, a browser zoomed to 150 %) the layer lay
over it and a click paused instead. The cutouts now have px floors (56 top, 140
bottom) and are measured against the iframe's own box.
2026-10-08 01:36:36 +02:00
peter 3865689c26 chore(release): 0.102.0 2026-10-08 01:33:39 +02:00
peter c32429ba69 fix(plex): an episode with no number is named by its title on every card
The cards, the up-next card and the mark-next label printed a question
mark where Plex has no number; the player's header already showed the
title alone. One helper (episodeNumbers/episodeName) for every place.
2026-10-08 01:16:30 +02:00
peter a504550196 docs: the superseded fallback answers its own session; who uses beacon() 2026-10-08 01:16:30 +02:00
peter 5095047c81 test(player): pin the stop URL's ids, the beacon's settle, the kept point's edges
The stop-URL contract passed with the two ids swapped and read the backend
from the working directory; nothing called a real beacon's onSettled or its
library pass-through; the kept-point rule was held by its constants only.
2026-10-08 01:16:30 +02:00
peter 4e70f168b1 fix(player): a layer that opens by itself takes no history entry
The auto-skip countdown and the up-next card pushed an entry without a
user gesture (Chrome's Back button skips those), and one pushed under the
open shortcuts sheet sat above it, so Back closed the hidden layer.
2026-10-08 01:16:30 +02:00
peter 7f17f488ab fix(nav): a sub-view step closes the overlays standing over it first
Leaving the Plex player while one of its layers was up (the info panel when
the film ended, the countdown under the back arrow) landed as a user Back:
the overlay listener spent it closing the top layer and the player stayed.
useHistorySubview's open/back now go through closeOverlaysThenNavigate,
like setPage. The RECLOSE comment no longer cites the shared Plex entry.
2026-10-08 01:16:30 +02:00
peter d1704a5519 revert: the Modal Escape guard for a fullscreen own-content video
Chrome does not hand the page the Escape it spends on leaving fullscreen (checked with a real
keypress: the first Esc leaves fullscreen, the second closes the player). The guard defended a case
that does not happen, under a comment saying it does. This reverts 121ebeeb.
2026-10-07 23:59:32 +02:00
peter 121ebeebc1 fix(ui): an Escape while a dialog's own video is fullscreen does not close the dialog
The page gets the Escape keydown with the video still the fullscreen element. When that element is
the dialog's own content (the library player), closing the dialog took the player away with the
fullscreen. A dialog opened over something else that is fullscreen still closes.
2026-10-07 23:57:04 +02:00
peter ee5290ba01 refactor(plex): the item progress body extends the shared checkpoint
ItemProgressIn repeated ProgressIn's config, fields and bounds; it now adds only `final`. The
threshold comment points where the constants live now, and plexProgressBeacon's comment is back
on it.
2026-10-07 23:16:38 +02:00
peter dd6a666ad1 test(player): hold the session stop and the kept-point rule to the server
- The client's resume thresholds equal the server's (read from both sources).
- The session stop's URL matches the route the server serves it on.
- The player stops its server session by its item and its own id on an unload and on close,
  and keeps it when the page only goes into the back/forward cache.
2026-10-07 23:16:38 +02:00
peter 392099770a fix(plex): an episode Plex has no number for is named by its title alone
The OS media title printed the missing number as "Enull", the player and info headers as a bare
"E". One helper for the three lines.
2026-10-07 23:16:38 +02:00
peter 2aa3ac6e85 fix(player): the browser Back peels each layer over the Plex video, one press at a time
All the layers shared one history entry, so the first Back spent it on one layer and the second
left the player with the stats readout (or the countdown) still up. Each layer now holds its own
entry; Back closes the one opened last.
2026-10-07 23:16:38 +02:00
peter 891dc628f3 fix(library): a listing poll can no longer put back an older resume point
A poll already out when the player closed answered with the point the server held before the
closing beacon, over the row's patch; Continue then opened there and wrote it back. The close now
cancels the listings' fetches before patching and reads them again once the beacon has landed.
2026-10-07 23:16:25 +02:00
peter 0a35f2a3b1 fix(player): a stale session answer can no longer stop the playback that won
- The engine tracks the request out per item: A, then B, then back to A sent a second request
  for A beside the one still out.
- A hardware transcode superseded during its fallback answered with the NEWER session; the losing
  request's player then stopped it as stale. It now answers with its own (dead) session.
- A replaced hls.js instance's late manifest no longer marks the player ready or plays the
  outgoing media.
- A pause given after the answer stands: the manifest and the first canplay read the live intent.
- The debounced A/V-offset restart is dropped on an item change instead of restarting the
  previous item under the next one's title.
2026-10-07 23:16:25 +02:00
peter d25a132495 fix(ui): a dialog whose own video goes fullscreen no longer crashes the app
The portal host followed the fullscreen element into it. When that element is the dialog's own
content (the library player's <video>, by its f key or its own button), the move is into the host's
own descendant, which the DOM refuses: HierarchyRequestError in a layout effect, and the crash
screen replaced the app. The host now stays where it is then — the content already shows.
2026-10-07 23:16:25 +02:00
peter 61d3a1a189 chore(db): drop plex_items.markers, a column nothing writes or reads
Markers are read live from Plex for the item the player opens; the cached copy was never filled.
2026-10-07 02:23:37 +02:00
peter d482a5e8e2 chore(dev): the dev proxy answers a failed connect with 502, as the prod proxy does
Vite's own answer was a bodiless 500, which the app reads as a real server fault and shows the blocking error dialog; a 502 is the passing connection loss it is.
2026-10-07 02:23:37 +02:00
peter 2064373ac2 fix(history): a step back never leaves the app for an older page
Chrome keeps 50 entries a tab, and when the list is full a new entry drops the
oldest one the page left without a user gesture, which can be the page's own
entry under a sub-view or an overlay. A programmatic back() then left the
document; once it landed on an older entry whose restored Plex player started
playing on its own. Every programmatic back() now asks the Navigation API first
and, with no entry of this document below, settles in place: the sub-view turns
to its root, an overlay's entry drops its mark, the channel page and the YouTube
search clear.
2026-10-07 02:23:37 +02:00
peter 0cda5b6dd4 fix(player): only the credits that end an item move on to the next one
An item can carry several credits blocks with mid- and post-credit scenes between
them, and Plex flags the one that really ends it. The player left at the first
block. Now the Plex-final block moves on (by the auto-skip or the Skip button), an
earlier block is only skipped to its end by the button, and with no final flag the
last block ends the item by the button alone.
2026-10-07 01:48:10 +02:00
peter f3c94641a7 fix(player): a library item played from the start keeps its point
A cached file that autoplays from 0:00 fires its one playing event before the resume hook
subscribes, so the hook never counted it as played and reported nothing - not on pause, not on
close. An element already playing when the hook subscribes now counts as played.
2026-10-07 00:36:28 +02:00
peter 59b27467ab feat(player): the OS media controls, the basic keys and resume for the library and shared-link players
Plex, library and /watch tell the operating system what plays (title, line, picture) and take its
play / pause / skip / seek through one hook (lib/mediaSession); the Plex player drives it with its own
seek and clock, so the lock screen shows the film's position, not the stream's. The YouTube player is
left alone: its iframe already publishes all of it, and the OS routes to the frame that plays.

The library and /watch players get the basic keys (lib/mediaKeys, the keymap's mediaPlayer rows) and
open where the viewer stopped (lib/resume): the library keeps the point on the server, per user and
library item (library_states, POST /api/downloads/{id}/progress, listed back on both listings), the
shared link in this browser. A started library row continues, its menu can start it over, and its
picture carries the cards' progress bar. The resume rule (too early / near the end = from the start)
is one function shared with the YouTube route.

The library dialog opens focused on its panel (Modal initialFocus "player"): on the ✕, Space closed
the film. While the media element itself has the focus, Space and the arrows stay its own controls'
— the browser acts on them whatever a listener does, and one Space played and paused.
2026-10-06 23:55:41 +02:00
peter 02feaa9a1c refactor(player): one panel at a time over the Plex video, and one Back cascade
The ⋯ menu, the tracks menu, the tuning panel, the info overlay and the up-next card were a flag
each, so two could stand on top of each other, and Back/Escape were written out three times. They
are one `panel` value now; the stats readout stays a separate toggle beside it. One `back()` peels
the countdown, the panel, then the stats, then leaves — for the arrow, ⌫, Escape and the mouse
Back alike. The mouse Back now also closes the ⋯ menu instead of leaving the player.
2026-10-06 23:04:35 +02:00
peter c1058f1d5a refactor(player): the Plex session machine is a tested engine, one request per item in flight
PlexPlayer's session logic (generation, pending play intent, clock ownership, session offset, reclaim
budget, listener teardown) moves into lib/hlsEngine.ts, a plain class with hls.js behind lib/hlsPort.ts
and its own unit tests.

- A seek, track change or reclaim asked for while a session request is out replaces the queued
  target; it is sent once the in-flight answer settles. A burst of 20 presses past the buffer was 20
  requests and 20 ffmpeg spawns; it is now 2-3, and plays again ~2 s sooner.
- The player stops its server session when it leaves the item, closes or the page unloads
  (POST /api/plex/stream/{rk}/session/{sid}/stop, identity-aware). A left transcode used to run to
  the end of the film first.
- A restart asked for after a new session attached but before it played no longer reads the emptied
  media as the viewer's pause.
2026-10-06 22:31:48 +02:00
peter 23f16b69df release: 0.101.0 (improvement/e36-delivery) 2026-10-06 21:14:35 +02:00
peter c813814c3c chore(release): 0.101.0 2026-10-06 21:04:19 +02:00
peter ed6facb703 fix(downloads): a conversion's commit, a stopped upload, the fast join and a clip's copy hold up
- The fast join gives each ffconcat entry its own media-only format whitelist, so a library file
  can no longer be opened as a nested concat list (the list's whitelist was handed down to it).
- A file the guarded probe cannot read is a download-only file, not a guess from yt-dlp's info.
- A conversion's or an ingest's commit that raises is re-read in a fresh session before any file
  is removed: a commit that landed keeps the file its row now names.
- An original that cannot be removed after conversion is logged; the upload stays ready.
- An upload whose conversion is cancelled or deleted midway is freed once nothing holds it; a
  paused one is kept.
- A shared link says "preparing" only while its video's job is queued or running.
- The admin storage view counts every file on disk ("Files on disk"), not only ready ones.
- A clip's failure title, unknown failure and cancel dialog, and a clip's and an upload's delete
  dialog speak of what the job made; an upload's quota refusal explains that the original is kept.
- /watch announces "preparing" as an alert and moves focus to the new Retry after an action.
2026-10-06 21:02:07 +02:00
peter d44d8d6dda fix(gate): knip sees the watch entry, the build must emit watch.html, and two hand copies are held
- The raw-source contract tests read files from disk (rawSources.testkit)
  instead of import.meta.glob, which knip counted as importing every src file —
  its unused-files lane could never name an orphan. knip now names
  src/watchMain.tsx as an entry (registered in docs/gates.md).
- vite build fails when index.html or watch.html is not emitted.
- The watch-entry import walk follows dynamic import() too.
- audit_metrics.py's plan-code patterns are held equal to the guard's.
- CrashBoundary is domain-free: it moves into components/ui.
2026-10-06 03:19:33 +02:00
peter b20e9a3947 fix(player): the stacked layout's minimum width no longer clips the wide one
The 36rem floor guards the stacked layout's width/height feedback loop; the
wide side-panel card has no such loop, and on a short wide window the floor
made the video taller than the card, cutting off YouTube's control bar. The
floor now applies to the stacked layout only; e2e covers a 1400x380 window.
2026-10-06 03:19:33 +02:00
peter 6073a006b8 fix(downloads): an upload's cancel, progress, failure and notice speak of an upload
- Cancelling an upload asks "Delete upload?" and says the uploaded file is
  removed from the server with it.
- The phone-safe conversion's real ffmpeg % shows as a bar ("Converting for
  phones"), not an indeterminate pulse.
- A failed upload is titled "The upload isn't ready", with its own reasons
  (convert_failed, retryable; upload_missing) instead of download copy.
- The inbox says "Upload ready to share" for a converted upload.
2026-10-06 03:19:32 +02:00
peter dba4758c59 fix(watch): a link whose video is being made again says so, in the page's language
The public watch route answers video_preparing for an asset that is pending or
downloading, and the page shows "being prepared — try again" with a Retry
instead of "this link is gone". watch.html ships one lang; the page now sets
documentElement.lang to the language its strings were picked in.
2026-10-06 03:19:32 +02:00
peter 68cfc7e25d fix(downloads): an upload's conversion is guarded, counted, named and cleaned up
- Every ffmpeg/ffprobe over a library file names the demuxers it may use
  (edit.INPUT_GUARD; a downloaded thumbnail edit.THUMB_GUARD): a playlist-type
  demuxer would read local paths named inside a user's upload.
- A re-encode ends on an even frame (edit.EVEN_FRAME): an odd 4:4:4/RGB source
  failed its conversion; a clip stores the frame its file has.
- An upload counts against its owner's cap from the moment it lands (queued, or
  kept after a failed conversion) — the footprint, the GC's cap and the admin
  view read one ON_DISK predicate — and a conversion that outgrows the cap is
  refused, keeping the original.
- The converted name keeps _a{id} (a dotted bare name once lost it and two
  uploads shared a file); a converted file placed but not committed is removed,
  and the orphan sweep enters .uploads; the LRU cap never evicts an upload.
- Ingest judges the extension the file is stored under; a failed conversion is
  convert_failed / upload_missing, reports the "converting" phase, and notifies
  as an upload. Job statuses in routes/downloads use JobStatus.
2026-10-06 03:19:32 +02:00
peter 00886d92fe fix(player): the YouTube player keeps its size on a short window and when maximised
Two layout bugs since the player grew a side panel, both reported from prod.

Maximised (F) on a wide window, the video was a thin line across the screen: the side-panel layout
aligns the stage `self-start`, and Chrome applies a flex item's self-alignment to an absolutely
positioned box too, so the fixed stage took its content's height — 0 px. The maximised stage now
stretches between its insets whatever the layout asks of it.

On a short window under 1280 px (a laptop at 100%, a 1080p screen at 175%) only the side strips were
left: the card narrows so the chrome under the video fits, but that chrome grows as the card narrows
(its text wraps), and the loop ran the card down to 2 px. The video now never goes narrower than
36rem to make room; past that the card scrolls.

Measured before/after (localdev, real YouTube iframe): 1097×520 open 0×0 → 610×343; 1920×911 maximised
1920×0 → 1920×911; 1440×760 and 1366×650 maximised fill the window; leaving maximise restores the card.
Held by an e2e test at both sizes (red on the old code: 0 px wide, and 911 ≠ 0), green on Chrome,
WebKit and iPad.
2026-10-05 23:40:13 +02:00
peter e7d2eb8cf6 test(downloads): what a share link serves passes the phone-safe rule
A link serves an asset once it is ready, so the contract is pinned on both sides. The code: every
function in app/ that marks something ready must run the rule (`_ensure_phone_safe`, or the
upload's own `playable.plan`) — an AST scan names any that does not, and fails blind if it stops
finding the four paths known today (download, editor clip, upload conversion, upload kept as is).
The bytes: the files an iPhone + iPad refused (HEVC tagged hev1, AV1, Matroska, MP3 inside an
.mp4) and the ones other browsers cannot be trusted with (VP9 + Opus, 10-bit H.264, Opus in Ogg)
go through the upload route, the worker's conversion and the public file route; what the link
hands out must be served as mp4 / m4a / mp3 and pass the rule.

Seen red: a planted ready-marking function in service.py is named; with the conversion step
short-circuited, all seven files fail (on the rule, or on being served as quicktime / webm / ogg).
2026-10-05 21:05:22 +02:00
peter e886890d6e perf(watch): a shared link loads the player, not the app
A /watch link is opened by whoever it was sent to — signed out, often on a phone on mobile data —
and it booted through main.tsx: every app provider, the app's i18n (every locale of every area),
the flag sprites, ~392 KB gzip before the player could start. It now has its own HTML entry,
watch.html → watchMain.tsx, which imports WatchPage and what it renders; the server hands every
/watch/ path that page (the OG block injected as before), and vite dev does the same.

Measured on the build: /watch JS 290 → 87 KB gzip, CSS 104 → 19 KB; the app's own bytes unchanged.
Two leaks had to go for that: lib/format imported the app's i18n for one "just now" string (the
relative-time formatters now take `t`, like the file's other t-formatters), and the boundary's
crash copy came from the same i18n (CrashBoundary takes its copy; ErrorBoundary is the app's
wrapper, the watch entry gives it the page's own strings). The page's popover tree went too: it
renders no popover.

Held by: a contract test walking the entry's runtime imports, naming the path to anything only the
app may load (red on a planted i18n import in lib/format), and holding watch.html's head equal to
index.html's; a DB-lane test that every /watch/ path gets the player's page with the link's OG tags
and app paths get the app (red on the old routing). Screenshots at 375 px, EN and HU, video / audio /
password / gone / wrong password / unlocked: identical to the old entry's, bar the video's buffered bar.
2026-10-05 20:51:39 +02:00
peter 5561d9a379 feat(downloads): every ready file is phone-safe — uploads and clips too
A share link is opened on whatever the friend holds. Measured on an iPhone and an iPad: AV1,
Matroska, HEVC tagged hev1 and MP3 inside an .mp4 fail there; Chrome and Firefox add HEVC and
10-bit / 4:4:4 H.264. The old check judged the codec NAME only and ran on downloads only, so a
10-bit H.264 passed, an upload went out exactly as it came (an iPhone's HEVC .mov, a webm, an mkv)
and a fast trim kept its source's codecs.

One rule now, `downloads/playable.py`: video is H.264 8-bit 4:2:0 + AAC in .mp4, audio-only is
.mp3 or AAC .m4a; `plan` says what must change, `build_cmd` re-encodes only the failing stream (or
remuxes). The worker applies it to a download, an editor clip and — new — an upload: media that
fails the rule is queued (asset pending, job queued) and the worker converts it, keeping only the
converted file (owner's choice). Until then the original sits at rel_path, so Retry starts from it
and a delete removes it; an upload that lost its last holder mid-conversion is reaped by the GC.
Every libx264 run (edit, concat, conversion) now pins `-pix_fmt yuv420p` — without it a 10-bit
source came out High 10. The download cache signature moves to v5 so a cached 10-bit asset
re-derives. The upload toast says when the file is still converting (HU/EN).

Tests: the plan for every measured class; real ffmpeg turning a 10-bit Matroska + Opus and an
mp3-in-mp4 into files the rule accepts (red without the pixel-format pin); the queued upload's
delete, cancel-then-GC, failed conversion and real worker conversion (the cleanup tests red
without their rules); the toast (red without the branch). Live on localdev: 9 fixture uploads
converted, every served file probed H.264 High yuv420p + AAC, or mp3 / m4a.
2026-10-05 03:16:27 +02:00
peter 2768ab5bdd fix(gate): the plan-code guard catches decision codes, and the purge takes them
A plan's decision ids (`D1`, `D4`, `D6-A`) cited in 11 comments and test titles named a document
no reader of this repo has, and the guard had no pattern for them. It has now (mirrored in
audit_metrics), with cases for each form the citations took and for the look-alikes it must leave
alone (a JPEG marker, "3D"); the 11 lines say their why without the id. Case test seen red without
the pattern, the scan red on a planted `D9:`.
2026-10-05 03:16:13 +02:00
peter 8c79590165 release: 0.100.0 (improvement/e35-tooling) 2026-10-05 01:42:39 +02:00
peter eef3ac9c5c chore(release): 0.100.0 2026-10-05 01:19:51 +02:00
peter 4ba7c966d7 fix(nav): the dead-page rewrite does not restore a spent focus stamp
It spread the event's snapshot, which still carried the sfNav stamp lib/history's own listener had
just taken off the entry, so a later landing there (an overlay closing) refocused a stale item. It
now spreads the live entry (test red on the snapshot). Also: format.test's comment named a test
image that never runs vitest; it is the gate host's locale.
2026-10-05 01:13:06 +02:00
peter 513dd8175a fix(auth): becoming admin through ADMIN_EMAILS drops the password, on any sign-in
The promotion revoke guessed from today's SMTP state whether the password was trustworthy, and ran
only when the account was adopted. Both guesses failed: a row verified while SMTP was off stays
"verified" after SMTP is configured, and a later sign-in matched by Google identity promoted the
account with no revoke at all. Either way a stranger's password and session could become admin.

Now one rule, where the promotion happens: when ADMIN_EMAILS turns a non-admin account into an
admin and it holds a password, the password goes and every session ends; the owner, signed in
through Google at that moment, sets a new one in Settings. An account that already is admin keeps
its password. Tests: the activated squat, the SMTP case, the later sign-in (all three red without
the rule) and an admin's own sign-in.
2026-10-05 01:13:06 +02:00
peter c4298b0a73 docs(comments): no WebSocket and no CI in the present tense
The chat cut removed the only WebSocket and CI went before it, but comments still leaned on both:
`?account=` was justified by the socket, job polling by an in-process WS registry, the e2e flows as
"proxy-safe (none use the WebSocket)", and pyrightconfig named a CI container the engine contract
test never checked. Each now says what holds today (the wallet gate, no push channel, the two
contract test halves, the image build's `npm ci`, the gate).
2026-10-05 00:50:36 +02:00
peter 5ca2a0e6ee fix(gate): the plan-code guard catches bare sprint and phase codes, and the purge takes the P ones
The sprint alternative matched only `(S6)` or `S3e2`, so the commonest purged form (`# S6:`,
`S3's`, `"""S6 —`, `S1 justified`) and every phase code (`P1`) would come back unflagged; the phase
codes had in fact survived the purge in 16 lines (Plex client sections, the upload's header shape),
now reworded around them.

One pattern now covers epic, roadmap, sprint and phase codes; the named exceptions are TV episode
notation, a fixture's season or episode title (`title="S1"`) and ruff's `E4/E7/E9`, and a ruff rule
like `S110` does not match. The case test lists the purged forms and no longer claims every form.
audit_metrics copies the definition (ASCII, as JavaScript's \b is) and reports it as
plan_code_lines / plan_code_files.
2026-10-05 00:50:36 +02:00
peter 5257d57196 fix(nav): Back onto a removed page also rewrites the entry to the feed
The fallback rendered the feed but left the history entry and the address bar on the dead page, so
a Feed click was a no-op (the same-page guard), filter mirroring kept `?page=messages` and the nav
diagnostics logged a page that does not exist. The entry is now replaced with the feed's, keeping
every other URL key (test red without it).
2026-10-05 00:50:35 +02:00
peter 58ad95263c fix(auth): a Google sign-in that makes an account admin drops a password nobody proved
The never-activated rule left one order of events open: on an instance without SMTP, a stranger
who registered an ADMIN_EMAILS address (or a sibling spelling) and was then activated by another
admin kept their password and live session, and the owner's first Google sign-in raised that row
to admin, the stranger's cookie with it. Activation vouches for access, not for who chose the
password, and without SMTP nothing else ever proved the mailbox.

When ADMIN_EMAILS promotes an adopted account that holds a password on an instance without SMTP,
the password goes and the session epoch is bumped. With SMTP the verified flag is a proven mailbox,
and an account the sign-in does not promote keeps its password, so the "your password was cleared"
detour stays off the default path. Tests: the activated squat (red without the rule; the squatter's
session is signed out) and the SMTP case that keeps the password.
2026-10-05 00:50:35 +02:00
peter 96ad1ef306 docs(comments): correct comments the cuts left stale
The registration language is read today (preferences.language, the mail language); the Modal and
the players order against popovers through layerStack, not the FloatingTree; the upload's auth
dependency is current_user; the module of access tests covers onboarding and access, not only the
canonical email. The dev proxy's WebSocket upgrade served the removed messages socket and goes; the
test compose header says a pure-logic run needs SIFTLODE_TEST_REQUIRE_DB=0; the speed measure no
longer reports an unreachable database as "no share link".
2026-10-05 00:19:50 +02:00
peter d1bbe4524c fix(gate): close the gaps in the plan-code guard, the button lists and the tooling
- planCodes: the quote and slash lookarounds let the commonest form through (a docstring or test
  title opening with a code, a slash pair); the two real exceptions (an episode titled "E1", ruff's
  E4/E7/E9) are now named and cut first, with cases that fail on the old pattern. audit_metrics
  counts by the same definition.
- button-primitive: both file lists live in the rule body, out of sight of the snapshot and the
  register; each is now pinned to its exact set of ui/ files (red when either is widened).
- check.sh --brief shows a short log whole and keeps the line before each hit, so ruff's rule and
  eslint's file path survive the excerpt.
- lint-changed runs prettier over every changed frontend file, as the gate does, not only code.
- CLAUDE.md and gates.md no longer claim the basedpyright baseline was found harmless: it was never
  triaged, and the reportOptional* entries are the likeliest real bugs.
2026-10-05 00:19:50 +02:00
peter eecc195b7e fix(migrations): dropping the demo also takes its reset job's interval-change audit rows
0098 deleted the reset job's run rows but not the scheduler_interval rows that changing its interval
wrote; with the job's label gone they rendered as a raw id. The delete now takes every job-targeted
row for it, and a test seeds both kinds below 0098 and upgrades (red on the old filter). The test
has its own file so the removed-features guard can name it.
2026-10-05 00:19:50 +02:00
peter 2f901ad643 fix(nav): Back onto a page this build no longer has lands on the feed
A history entry pushed while the Messages page existed still names it after a reload onto the new
build; popstate took sfPage unchecked and the shell's module lookup threw. It is now validated with
isPage, like every initial-load path already was.
2026-10-05 00:19:50 +02:00
peter 1f357b0c21 fix(auth): Google adoption revokes the password of a never-activated account
Without SMTP, registration marks the address verified with no proof, and an ADMIN_EMAILS /
ALLOWED_EMAILS address gets no invite row to gate it. So a stranger who registered such an address
(or a sibling spelling) before its owner kept their own password when the owner's first Google
sign-in adopted the row, activated it and made it admin, and could then sign in with it.

The adoption now treats a never-activated row like an unverified one: the password goes,
password_revoked_at is stamped and the session epoch bumped. An account already let in keeps its
password. The helper is renamed to say what it now judges (revoke_untrusted_preset_password).

test_google_callback.py drives the callback's account-matching branches through the real route,
none of which a test ran before: the squat (red on the old rule), adoption by mailbox, the
sub-collision and unverified-address refusals, the refused sign-in recorded in Google's locale, and
the link flow's conflict, mismatch and attach answers.
2026-10-05 00:19:50 +02:00
peter fb962682eb chore(blame): skip the plan-code purge in git blame
It touched 671 files' comments and nothing else; blame should keep naming the commit that wrote each line.
2026-10-05 00:19:50 +02:00
peter 4e71723a07 test(comments): no plan code lands in the code again
planCodes.contract.test.ts scans every tracked text file and every new one not yet added (comments,
docstrings, test titles, lint messages alike) for an epic or roadmap code (`E28`, `E9.5`, `R14`) and
for a sprint or review round cited on its own (`(S6)`, `S3e2`, `round 4`, `deep-review`). Not
`E2E`, not a ruff rule family (`E4/E7/E9`), not a TV episode (`S2 · E5`, a test episode titled
"E1"), not `S3h` (the media-card decision docs/DESIGN.md is headed by). Markdown and the Alembic
migrations keep their history and are not scanned.

One sweep test rather than an ESLint rule plus a ruff one: the codes sat in Python, TS, CSS, shell,
TOML, YAML and lint messages, and no single linter reads all of them. Seen red on a planted
backend/app file (all three plan-code lines named, its allowed line not).

audit_metrics.py counts epic codes by the same definition (it used to miss `E9.5` and `R6`), and
CLAUDE.md's "comment the why" rule names the test.
2026-10-05 00:19:39 +02:00
peter 4aaaaa7860 chore(comments): drop epic, sprint and review-round ids from the code
The code cited its plan instead of saying why: `E28 review J4`, `(R6 S3b)`, `S1 justified…`,
`deep-review round 5`. That plan lives in the private ops repo, so for every reader of this one the
citation was noise, and a sentence whose subject was the plan step ("round 2 showed…") hid its
lesson behind an id. Each one is gone and the "why" kept: a bare citation is deleted, a sentence
carrying a lesson is reworded around it ("a later attempt showed…").

Measured (`scripts/audit_metrics.py`, product code): 652 lines in 270 files carried an epic code by
the definition the next commit holds (507 / 241 by the old, two-digit-only one) -> 0. Across all
code, tests, scripts and configs: ~1100 epic-code lines plus ~320 sprint/round ids.

Behaviour unchanged, checked mechanically: every changed .py file has the same AST once docstrings
are blanked; css/sh/toml/yml/json files are identical once comments are stripped; the only
non-comment changes in TS/JS are test titles, one description string in a contract table, and five
ESLint messages, whose text is all the gates snapshot diff holds. Not touched: Markdown (docs and
the changelog keep their history), Alembic migrations, TV episode notation (`S2 · E5`), DESIGN.md
ids (`C1`, `B4`, `S3h`), tracker ids (`SEC-16`, `DX-17`).

One stale claim fixed on the way: test_hls_reaper's docstring said the player checkpoints "every
10 s whether or not it is playing", which plex/stream.py calls wrong; it now says what the code does.
2026-10-05 00:19:39 +02:00
peter c73d8be094 docs(welcome): retake the landing screenshots on today's UI
The seven gallery shots dated from 2026-07 and still showed the old
labelled sidebar with the removed Messages item. Retaken at the same size
(1200x675 @2x, dark) from the localdev stack with the E2E account (no
personal account name on screen), filled for the shoot with a copy of a
real account's subscriptions, playlists and finished downloads, removed
again afterwards.
2026-10-05 00:19:39 +02:00
peter 30545d0d88 fix(gate): --brief leaves the test database's own log out of an excerpt
The DB lane provokes constraint errors on purpose, and their ERROR: lines
from the testdb container came first in a red pytest excerpt, ahead of the
failing test (seen in E35 S3). The excerpt now skips the testdb-N lines.
2026-10-05 00:19:39 +02:00
peter ac702192c9 docs(claude): drop the canonical-pattern rows a lint rule already holds
Buttons, text fields and native controls, glass surfaces, the native
<select> and network calls are each refused by a lint rule whose message
names the primitive to use (button-primitive, input-primitive,
surface-role, no-restricted-syntax, no-restricted-globals). Their table
rows repeated that, so a change had to be made twice. One paragraph now
says they are lint-held and lists the primitives. The rows nothing
mechanical holds stay, as do ORM and current time, now test-held.
2026-10-05 00:19:39 +02:00
peter 24c47d12bd refactor(backend): select() and an inline now everywhere, held by a test
CLAUDE.md's canonical table names SQLAlchemy 2.0 select() and an inline
datetime.now(timezone.utc), and both had drifted: five db.query(...) calls
remained in routes/auth.py and routes/me.py (seven before the demo went),
and four private now helpers stood in sync/videos.py, utils.py (imported
as _now by explore and maintenance), scheduler.py (_now_iso) and the e2e
seed. All are the canonical form now.

Ruff cannot ban a method call on an instance or a function by name, so
tests/test_canonical_patterns.py reads every app module's AST and fails on
a .query(...) call or a now/_now/now_utc helper — seen red on a planted file.
2026-10-05 00:19:39 +02:00
peter ab76166878 docs(gates): basedpyright keeps new-errors-only, without a burn-down mandate
E34 read the standard-mode findings and none was a real bug, so working the
baseline down as work of its own buys nothing. The lane is unchanged: it
fails on a new error, and on a run that tightened the baseline without the
commit. CLAUDE.md and the register row say so.
2026-10-05 00:19:39 +02:00
peter a42f346a32 test(gates): the snapshot holds a config per scope, not a path per file
gates.snapshot.json listed ~800 file paths under their config, so every
new or deleted frontend file regenerated it (31% of commits, E34 S2) and
the regen diff buried the gate changes it exists to show. Now a scope is
named by path (scopeOf: src, src tests, ui, lib tests, …), the snapshot
holds one effective config per scope (2345 → 1456 lines), and a new test
fails when a file's config is not its scope's.

Not narrowed: a new file in a known place changes nothing (it gets its
scope's config, checked), a glob that moves one file into another config
fails (seen: an override on Feed.tsx → "src: src/components/Feed.tsx"), a
scope whose rules change is a snapshot diff, and a file no config lints
is still a register miss.
2026-10-05 00:19:39 +02:00
peter f2fe0fc0df build(gate)!: coverage leaves the gate and runs on demand
Both lanes printed a coverage total with no threshold (owner decision E32
D4). No real hit as a lane: the one time the number found anything (5 bugs,
E32) it was read by hand, and since then it was a line nobody acted on. It
was also the most expensive part of the vitest lane, and the run that
flaked in E35 S1. The vitest lane runs without --coverage, the pytest lane
runs plain pytest; docs/development.md gives both on-demand commands, and
the register row says why.
2026-10-05 00:19:39 +02:00
peter ca707fd390 build(eslint): focus-safe-disabled folds into button-primitive
Two rules looked at the same <button>: button-primitive refused a raw one
outside the control primitives, focus-safe-disabled refused a raw
`disabled` one everywhere but FocusSafeButton and AnchoredPopover. They are
one rule now, with the same reach: outside the primitives a raw button is
refused (a disabled one with the focus message); inside a primitive and in
a test only a raw `<button disabled>` is, and the two native-disabled owners
are named in the rule rather than in a config override. One override block,
one rule entry and two register rows go; the snapshot loses a config group.
focusSafe.contract.test.ts pins every case, the primitive and test ones new.
2026-10-05 00:19:39 +02:00
peter 2eb20ca227 build(eslint): one px-frozen box selector for box utilities and grid minimums
BOX_PX and GRID_MIN_PX were two esquery patterns with two messages for one
rule (DESIGN.md: a px box does not scale with --font-scale). They are one
alternation now, with one message that names both fixes. Nothing it refused
before passes now; pxBox.contract.test.ts pins both shapes unchanged.
2026-10-05 00:19:38 +02:00
peter adaa7dc23c build(eslint): drop the Z_RAW and EDGED_LABEL selectors
Both came out of the E28 claim audit (0521149c named z-levels, e13d93a3
static labels as flat Badges) after the audit itself had fixed the sites.
No real hit since: 0 matches in every lint run from E28 to E34 (S2/S3
measurement). Each still cost a probe test (zLayer.contract, the edged-label
half of hoverAndLink.contract) and a place in the gate snapshot. The design
rules stay in DESIGN.md (Tokens: no ad-hoc z-index; Static vs pressable),
with a decision-log row saying the selectors went and why.
2026-10-05 00:19:38 +02:00
peter 91f6a77fa3 build(eslint): drop the three React Compiler rules that cannot fire on 7.1.1
react-hooks/exhaustive-effect-dependencies, memoized-effect-dependencies and
rule-suppression were switched on in E32 "for the plugin version that wires
them up". Read in the plugin source and probed (E32 review): the first's
validation is compiled off, nothing emits the second's category, and lint
mode hands the third no suppression rules. 0 hits by construction — a rule
that cannot report reads as coverage it does not give. The comment keeps
their names and why, so a plugin upgrade can re-check them.
2026-10-05 00:19:38 +02:00
peter 36653886d5 build(frontend)!: knip drops --production; plain knip stays
The production pass (7ed2ac4e, E32) asked "is this export used by what
ships?". No real hit: its only findings were 37 exports a test imports,
answered with 36 `@internal` JSDoc tags (E34 S2) — test-only exports, not
defects. Since then it has only ever asked for another tag. The tags go
with it (36, no longer read by anything), and so do the production-only
`!…!` exclusions in knip.jsonc and their register rows. Plain knip (unused
files, exports and dependencies) keeps running in the gate.
2026-10-05 00:19:38 +02:00
peter 7e681e112e test(gates)!: the size/complexity ratchet leaves the gate; audit_metrics reports it
sourceSize.contract.test.ts, its test kit and backend/tests/test_source_size.py
held 13 FE / 9 BE file ceilings and 64 FE / 32 BE function ceilings. No real
hit (E34 S2/S4): its history is its own birth (751e1ae5), two rounds of
fixes to its own logic (e432dc27, d1df76e8), the snapshot move (182ec311)
and ceiling edits inside feature fixes (5806adda, df4a79a1, and the S2/S3
cuts here). It caught no defect, went red 8 times on its own limits and drew
5 review findings about itself. The numbers stay visible: audit_metrics
reports file size and complexity, and an epic compares before and after.
docs/gates.md: the two size rows go; the C901 row says where it is read now.
2026-10-05 00:19:38 +02:00
peter 3642397f63 fix(tooling): audit_metrics counts the demo footprint by the banned word
Its pattern also matched "demote" and the Hungarian "Demokratikus", so
the footprint read 5 after the demo was gone (and 56, not 53, in the S1
baseline). It now uses the removedFeatures contract test's pattern.
2026-10-05 00:19:38 +02:00
peter 6d8b7be6d8 feat!: remove the shared demo account
The demo account — one shared user that whitelisted emails entered from
the login page, reset on a schedule — goes end to end (owner decision,
E34): demo.py and the reset job, users.is_demo and demo_whitelist
(migration 0098 deletes the demo user, its audit rows, the reset job's
override and notices, then drops the column and table; the downgrade
restores the empty schema), the /auth/demo entry, the admin whitelist and
reset, the Demo tab, the banner, the login page's "Try the demo", and
every is_demo branch in routes, userscope, scheduler and the UI.

require_human only ever refused the demo account, so its 87 uses become
current_user (owner-approved); require_write_scope keeps its own check.
The login page's "or" divider now shows only beside the Google button:
without the demo entry under it, an instance without Google would have
said "or" to nothing.

A contract test (removedFeatures) fails on the word anywhere in the
backend, frontend, e2e or scripts — seen red on a planted hit. Size
ceilings tightened for the modules that shrank; admin.py is back under
the limit.
2026-10-05 00:19:38 +02:00
peter cca3315836 feat!: remove direct messaging (chat)
E2EE direct messages, the chat dock, the Messages module, the message
WebSocket and its realtime registry, and the messages / message_keys
tables (migration 0097; the downgrade restores the empty schema). Prod
held two welcome notes, no message between people and one key — never
used (owner decision, E34). WebSocket now has no owner module: the lint
bans it everywhere. Privacy policy updated: it no longer promises E2EE
for a feature that is gone.
2026-10-05 00:19:38 +02:00
peter 7f08c510bf feat(auth)!: remove the public access-request endpoint
POST /auth/request-access had no caller left: the login page's form was
gone, and registration and a refused Google sign-in already land in the
same admin approval. A third public write path with no protection of
its own (owner decision, E34).

Its one guard moves to where every path passes: upsert_pending_invite no
longer opens a pending row for an address the env already lets in. The
registration path lacked it — registering b.ob@gmail.com beside an
env-granted bob@gmail.com opened a pending row, and is_allowed answers
from the invite rows once any exist, so the real owner was locked out.
The tests that drove the endpoint now drive registration or the shared
function.
2026-10-05 00:19:19 +02:00
peter a499f00209 refactor(compose): localdev is an override of docker-compose.yml, not a copy 2026-10-04 20:09:58 +02:00
peter c2bf8fb7f6 feat(tooling): audit_metrics.py — size, complexity, gate weight and a speed baseline 2026-10-04 20:09:58 +02:00
peter 11d35910fd feat(tooling): lint-changed.sh — fast lint lanes on the uncommitted files, usable as a Stop hook 2026-10-04 20:09:58 +02:00
peter 27f48b7a8b feat(gate): check.sh --brief prints a red lane's first error lines only 2026-10-04 20:09:58 +02:00
peter bc856ab2af test: warm cold dynamic imports before the timed cases
Under a full coverage run on every core, a test file's first cold import
(0.6-0.8 s alone) outran the 5 s case timeout and the half-done import
failed every later case in the file: the vitest lane went red in 2 of 5
runs with no defect. A beforeAll warm-up in the 4 files of that pattern;
5 of 5 full runs green after.
2026-10-04 20:09:58 +02:00
1074 changed files with 50871 additions and 18260 deletions
+11 -9
View File
@@ -28,7 +28,7 @@ ALLOWED_EMAILS=
# Admin emails (subset of the above) get the admin role.
ADMIN_EMAILS=
# Logging (E22). LOG_LEVEL is the BOOTSTRAP level — the level a process starts at and the only one
# Logging. LOG_LEVEL is the BOOTSTRAP level — the level a process starts at and the only one
# the worker/download child read; the API's running level is the admin Configuration page's
# `log_level` knob, which takes effect without a restart. TRACE | DEBUG | INFO | WARNING | ERROR, in
# any letter case; any other value stops the boot with an error. LOG_FORMAT is "json" (one object per
@@ -68,18 +68,20 @@ SMTP_FROM=
# --- Scheduler ---
# The background scheduler (subscription sync, backfill, enrichment) runs inside the app.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. If you ever run
# more than one instance against the same database, keep SCHEDULER_ENABLED=true on exactly one
# of them and false on the rest, to avoid double quota use and write races.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. Run ONE app instance
# per database: a starting instance closes every run still open as interrupted (it assumes the
# previous process is gone), and "is this job already running" is held in the process. A second
# instance would cut the first one's runs short and could run the same job twice.
# SCHEDULER_ENABLED=false keeps the recurring jobs off; "run now" still works.
SCHEDULER_ENABLED=true
# --- Download center ---
# The Download Center adds a `worker` container (runs the yt-dlp/ffmpeg job loop) and a small
# --- Files library & downloads ---
# Downloading adds a `worker` container (runs the yt-dlp/ffmpeg job loop) and a small
# `bgutil-pot` sidecar (mints YouTube tokens) — both come up automatically with docker compose.
# Downloaded media defaults to a Docker-managed named volume. Set DOWNLOAD_HOST_PATH to a host
# The Files library defaults to a Docker-managed named volume. Set FILES_HOST_PATH to a host
# directory instead — e.g. one your Plex server can read — to keep the Plex-style tree there.
# The path must be writable by the container user (uid of `appuser`, 1000): chown 1000:1000 <dir>.
# DOWNLOAD_HOST_PATH=/mnt/media/youtube
# FILES_HOST_PATH=/mnt/media/youtube
# --- Plex integration (optional) ---
# The optional Plex module browses/plays your Plex library, playing the LOCAL physical file
@@ -89,6 +91,6 @@ SCHEDULER_ENABLED=true
# to /plex-media inside the container (the local side of the admin "path map", whose default is
# /data=/plex-media). Leave unset if you don't use Plex.
# PLEX_MEDIA_HOST_PATH=/mnt/media
# Optional: scratch dir for on-the-fly HLS remux segments. Defaults to DOWNLOAD_ROOT; point it at
# Optional: scratch dir for on-the-fly HLS remux segments. Defaults to FILES_ROOT; point it at
# fast local storage if your download volume is slow.
# PLEX_HLS_DIR=/var/tmp/plex-hls
+4 -1
View File
@@ -2,8 +2,11 @@
# whole-repo reformatting does not bury the real authorship of each line.
# Configure once per clone: git config blame.ignoreRevsFile .git-blame-ignore-revs
#
# Prettier one-time repo-wide format pass (R2 S1b, 2026-07-21)
# Prettier one-time repo-wide format pass (2026-07-21)
51c36c2071728d2cd7ae580b7bfdf21f6867a347
# ruff format one-time backend + scripts pass (line-length 100, 2026-08-23)
b1832afd180866b922521148603db447020c57a4
# Plan-code purge from comments, one-time (2026-10-04)
4aaaaa786017278460e0b4f94cc0d3374e0e37e2
+3 -2
View File
@@ -29,9 +29,9 @@ frontend/e2e/.auth/
*.sql.gz
backups/
# Download center: locally downloaded media (the DOWNLOAD_ROOT bind mount at the repo root).
# Download center: locally downloaded media (the FILES_ROOT bind mount at the repo root).
# Anchored with a leading slash so it does NOT also ignore the backend/app/downloads package.
/downloads/
/files/
# OS / editor
.DS_Store
@@ -42,3 +42,4 @@ Thumbs.db
# TypeScript incremental build artifacts — machine-local, absolute-path-bearing, and no gate lane
# reads or regenerates them, so a committed one is stale from the commit that adds it.
*.tsbuildinfo
frontend/e2e/.speed/
+1 -1
View File
@@ -1,4 +1,4 @@
# The CVE gate's allowlist (E32). The publish gate scans the built image and the committed tree with
# The CVE gate's allowlist. The publish gate scans the built image and the committed tree with
# Trivy and fails on any HIGH or CRITICAL finding that has a fix available. A finding that cannot be
# fixed yet is let through ONLY by an entry here, and every entry carries both:
# statement — why it is acceptable for now (what is exposed, what is not)
+137
View File
@@ -7,6 +7,143 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.108.0 — 2026-10-18
### New
- Activity: one admin page for background runs, admin actions, the server log and quota, with one time range, account and search. A run opens to its outcome and its log lines, an admin action to its request's lines; old audit and log links open their tab.
- Scheduler: each job shows its last three runs (a run cut short by a restart is marked) and a button opens all of them; turning on a channel's full history is now a run of its own, listed under Ad-hoc runs.
- Configuration: how long the run history is kept (Activity › Run history, 180 days by default). Stats keeps your own numbers; the system quota view moved to Activity › Quota.
### Fixed
- Configuration: saving a setting with the value already in force now keeps it, instead of dropping the change.
- Configuration: the unused SponsorBlock default setting is gone.
## 0.107.3 — 2026-10-17
### New
- Development: the end-to-end suite runs the WCAG A and AA accessibility checks over every page and the player, and every shell script is linted in the gate.
### Fixed
- Readability: muted text in the light themes is a shade darker, and the item count, the labels and the selected Files folder sit on a solid tint, so text on the frosted-glass bars stays readable (4.5:1) in every scheme.
## 0.107.2 — 2026-10-17
### New
- Development: the gate script can list its checks (scripts/check.sh --list) and run just one (--only), and the backend tests run on every CPU core, each worker with its own database.
## 0.107.1 — 2026-10-16
### Fixed
- Downloads: a failed download's explanation now says to press Retry, instead of pointing to a download list that no longer exists.
## 0.107.0 — 2026-10-15
### New
- API tokens: Settings › API tokens gives one machine of yours a key to the Files folders you pick — or, in one click, to a Hub folder for swapping files with it. Signing out your other sessions or changing your password revokes every token.
- Share by link: a link can now open a whole folder of yours — its page lists what is in it and plays a video above the list.
### Fixed
- Uploads: a finished upload is announced once, and shows in Files at once.
## 0.106.2 — 2026-10-14
### Fixed
- Pictures: a thumbnail, poster or channel picture that fails to load is retried, and meanwhile the card shows its empty frame instead of a broken-image mark.
## 0.106.1 — 2026-10-14
### Fixed
- Plex player: clicking outside the Settings panel closes the panel again instead of closing the player.
## 0.106.0 — 2026-10-13
### New
- Files: one library for everything you keep — downloads, clips and your own uploads of any file — in a folder per site plus folders of your own, each keeping its files as long as you choose. Documents, pictures and text open in a viewer.
- Downloads: paste a link anywhere on a page to download it, and the tray shows what is running and what is done. The Downloads page is gone: your files are in Files, and your download settings are under Settings › Downloads.
### Fixed
- Self-hosting: the library is now mounted at /files (FILES_HOST_PATH). Old settings keep working; if you used the default volume, copy it over once — see the upgrade notes.
## 0.105.0 — 2026-10-10
### New
- Clips: mark a range of what you're watching — a YouTube video or a Plex item — set its ends to the tenth, crop it to a frame drawn on the picture, and cut it straight into your downloads.
### Fixed
- Downloads: a download, a clip or an editor clip that doesn't fit in the storage you have left says so, and is never stored over your cap; YouTube downloads reach their format's height again.
- Player: focus rings, the chosen track or tab, sliders and the progress bar's head are amber on the dark player in every theme, so they show in a light theme too.
## 0.104.0 — 2026-10-10
### New
- YouTube: a card's menu filters the feed to its channel and back, and the player gives the video more room — its side panel shuts on demand.
- Plex: a card's picture plays it and its title opens its page; a show card puts on the episode it would continue or start, and an episode links up to its show.
### Fixed
- Plex: summaries written as HTML read as clean text, and a Surprise draw stays put until you reshuffle. A video marked watched in the player stays out of an Unwatched feed. Plus smaller keyboard and focus fixes.
## 0.103.0 — 2026-10-08
### New
- Plex: “Add to collection…” in a movie's or show's menu ticks the title into or out of your collections, or names a new one with it inside — one step instead of a trip to the Collections page.
### Fixed
- Plex player: subtitles rise above the control bar while it shows and slide back when it hides, so the bar never covers them — on an iPad too.
- iPad: the header search fits the touch screen, tooltips no longer stick, the Plex player page no longer scrolls or shows a blue band, and the fullscreen button stays hidden where the browser cannot really go fullscreen.
- The playlist, collection and tag menus say when their list is loading or failed to load (with a retry) instead of showing it as empty, and work from the keyboard. Plus smaller fixes in the YouTube player's title line.
## 0.102.0 — 2026-10-08
### New
- The library and shared-link players answer the keyboard's and the system's media keys, and resume where you stopped.
### Fixed
- Plex player: the browser Back closes the panels over the video one at a time, and only the credits at the very end move on to the next episode. Plus smaller fixes around resume points and stopping a stream.
## 0.101.0 — 2026-10-06
### New
- A shared link plays on any phone, iPhone and iPad included: downloads, clips and uploads are kept in a format every phone plays, and an upload in another format is converted after it arrives.
- A shared link opens much faster: its page no longer loads the whole app.
### Fixed
- The YouTube player keeps its size on a short window and when maximised — before, it could shrink to a thin strip. Plus smaller fixes around uploads and clips.
## 0.100.0 — 2026-10-05
### New
- Direct messages are gone: the Messages page, the chat dock and their encryption keys were removed with their stored messages, and the privacy policy no longer describes end-to-end encrypted chat.
- The shared demo account is gone, with its sign-in entry and its whitelist.
### Fixed
- Sign-in security without email: when a Google sign-in takes over an account someone else registered first, or makes it an admin through the admin list, its password is cleared and its sessions end. The owner sets a new one in Settings.
- Pressing Back onto a page that no longer exists opens the feed instead of breaking the app.
- Behind the scenes: tidier checks and comments across the code.
## 0.99.2 — 2026-10-04
### Fixed
+24 -18
View File
@@ -5,23 +5,23 @@ be developed that way. This file is the rules of the road: read it before you to
loaded into every agent session, so keep it **short and true**. A rule here that has gone stale is
worse than no rule — if you change how the code works, change this file in the same commit.
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a download center). It is a
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a files library with downloads). It is a
**modular monolith**: one Postgres, one FastAPI process (uvicorn + APScheduler), one worker process
(yt-dlp/ffmpeg), one PO-token sidecar. React 19 (with the React Compiler) + react-query frontend, with a
router-free navigation provider.
## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip (both modes) + vitest + vite build + pytest + pip-audit + npm audit**, then a coverage report (no threshold). It runs
The local gate is **ruff (`check` + `format --check`) + shellcheck + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip + vitest + vite build + pytest + pip-audit + npm audit**. It runs
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
green **before** any merge, release, or handing work to review.
- Full gate: `bash scripts/check.sh` (`lint` for the static lanes only); the private ops dispatcher's `siftlode check` runs the same script. Single lanes: [`docs/development.md`](docs/development.md) § The gate.
- **There is no CI** (owner, 2026-10-04): this gate, run here, is the only one — green before every hand-over, and `release` refuses to merge without it.
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every file and the gate/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: a new code file or any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` and burn down epic by epic. Fixed some? Regenerate with `--writebaseline`.
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every scope and the gate/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` (no burn-down mandate, but none was ever triaged: a `reportOptional*` entry in code you touch may be a real bug). A run that fixed some rewrites it — commit that.
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
## Workflow
@@ -55,17 +55,19 @@ how it gets built. Concretely:
- **Challenge unsound requests.** If a request is technically ill-advised and you can back it with real
arguments, push back or propose an alternative — don't comply blindly.
- **Verify UI in the browser before calling it done.** tsc/unit/DOM tests miss visual and interaction
bugs; do a real visual check of any UI change, and run DESIGN.md's four-step Conformance check.
bugs; do a real visual check of any UI change, and run DESIGN.md's Conformance check (a copy-only
change runs its step 4 alone: every state the new text renders in, in both languages).
- **Be terse everywhere else.** Outside those decision points, lead with the answer or decision. During
implementation and debugging especially: no restating the same point reworded, no filler.
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → module
Frontend: `main.tsx` provider tree → `App.tsx` shell (a shared `/watch` link boots from its own
entry, `watch.html` → `watchMain.tsx`, held app-free by a contract test) → `moduleRegistry` route-chunks → module
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
boundary is one-way and **lint-enforced**, not just described: a platform component may not import a
@@ -90,22 +92,19 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` and `jobruns` own their session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE button | `ui/Button` (a `SIZES` rung) · `IconButton` · `Pressable` (a clickable card/row) · `Chip` · `DragHandle` | a raw `<button>` or `FocusSafeButton` outside the control primitives (lint-enforced) |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE text field / checkbox / slider | `ui/Input` · `Textarea` · `InnerInput` in a `fieldClass` frame · `Checkbox`/`CheckRow` · `Slider` | a raw `<input>`/`<textarea>` anywhere but those primitives (lint-enforced) |
| FE glass surface | a role: `SURFACE.panel` / `card` / `menu` / `popover` / `tooltip` (`ui/surfaces.ts`) | a bare `glass`/`glass-card`/`glass-menu` beside a hand-picked radius (lint-enforced) |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE control | `SegmentedToggle`; a long or user's list: `FilterSelect` | a native `<select>` (lint-enforced) |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect | a bare `catch {}` or `notify()` with no error attached |
| FE floating or full-screen layer | `ui/Overlay` (portals to `<body>`) or the popover engine (`AnchoredPopover` portals itself); stacking by the z-index scale (`z-menu` … `z-popover`, docs/DESIGN.md "Layers") | one rendered in place inside a virtualized row or a masked/transformed scroller — clipped or painted over; a magic `z-[n]` |
| FE player on a native media element | `lib/mediaSession` (`useMediaSession`) + `lib/mediaKeys` + `lib/resume` (`useResume`), the element held as STATE | a private `navigator.mediaSession` / `keydown` / position-saving effect per player |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
@@ -114,6 +113,12 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
| FE popover / scroll-fade ref | destructure it: `const { setReference } = pop.refs`, then `ref={setReference}` | `ref={pop.refs.setReference}` / `ref={fade.ref}` — the compiler then treats the whole object as a ref and skips |
| FE async handler with cleanup | `try { … } catch { … }`, then the cleanup after it (steps that `return` early go in their own function); a conditional, loop or `?.` the `try` needs goes before it or into a module-level helper | `try/finally`, or a `?:` / `??` / loop / `?.` inside `try` — the React Compiler skips the whole component (and `react-hooks/todo` misses most of these) |
Not repeated here, because a lint rule refuses the wrong one and its message names the right one:
buttons (`ui/Button`, `IconButton`, `Pressable`, `Chip`, `DragHandle`), text fields and native
controls (`ui/Input`, `Checkbox`, `Slider`, …), glass surfaces (`SURFACE.*`), a native `<select>`
(`SegmentedToggle` / `FilterSelect`), a picture (`ui/RemoteImage`, never a bare `<img>`), a box or font size (the spacing scale or rem,
never a px-frozen `w-[Npx]` / `text-[Npx]`: the text-size slider scales the root font), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
## House rules that bite (learned from real bugs)
- **A rule stated in a comment must also be enforced in code.** The worst bugs here came from a comment
@@ -125,7 +130,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
"no results". Reads degrade honestly; writes are conditional (`UPDATE … WHERE status='running'`).
- **i18n parity.** HU/EN, zero key drift. Every user-facing string is translated.
- **Comment the "why", keep it current.** This codebase uses comments as its architecture doc; a stale
"why" is a liability, so update it when the behavior changes.
"why" is a liability, so update it when the behavior changes. Say the why itself, never the plan
step that produced it: no epic, sprint or review-round ids in the code (`planCodes.contract.test.ts`).
---
+13 -7
View File
@@ -39,24 +39,30 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
WORKDIR /app
# Create the app user BEFORE the copies so files land already-owned via COPY --chown (a trailing
# `chown -R /app` would rewrite every file into a duplicate layer). /downloads must be appuser-owned
# `chown -R /app` would rewrite every file into a duplicate layer). /files must be appuser-owned
# so a fresh named volume mounted there inherits that ownership (Docker copies the image dir's
# ownership into a new empty volume) — the worker/API can then write without a manual chown. A bind
# mount to a host path instead needs that path writable by this uid (see docs/self-hosting.md).
RUN adduser --disabled-password --gecos "" appuser \
&& mkdir -p /downloads \
&& chown appuser /app /downloads
&& mkdir -p /files \
&& chown appuser /app /files
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode (E6 S1).
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode.
# intel-media-va-driver (iHD): the VA driver ffmpeg's h264_vaapi needs — libva2 ships in the base but
# no driver, so without this `vainfo`/`h264_vaapi` fail and transcode silently falls back to libx264.
# Harmless on hosts with no /dev/dri (the driver just goes unused). Deno: the JavaScript runtime yt-dlp
# uses to solve YouTube's "n" signature challenge for the web player clients (required alongside the PO
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# `apt-get upgrade` (E32): the base image is a snapshot, and the Debian security fixes released since it
# `apt-get upgrade`: the base image is a snapshot, and the Debian security fixes released since it
# was cut are only in the archive — the first Trivy scan found one (pcre2) fixed upstream but not in it.
# APT_REFRESH: the upgrade only runs when this layer is rebuilt, and a build cache keeps it for as long
# as nothing above it changes — a publish shipped an image whose curl and librsvg were a release behind
# fixes already in the archive. The release build passes the day here, so the layer is rebuilt at
# least daily; empty (the default, a self-hosted build) leaves the cache alone.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
ARG APT_REFRESH=
RUN echo "apt refresh: ${APT_REFRESH:-cached}" \
&& apt-get update \
&& apt-get upgrade -y --no-install-recommends \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
@@ -81,7 +87,7 @@ RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nCo
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones the gate's pip-audit scanned — not whatever
# floats up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# pip itself is removed once it has installed the lock (E32): nothing runs it at runtime, and it carries
# pip itself is removed once it has installed the lock: nothing runs it at runtime, and it carries
# its own VENDORED copies of urllib3, msgpack and setuptools, which Trivy reports as the image's own —
# an attack surface and a CVE stream for a tool the running app never calls.
COPY backend/requirements.lock .
+5 -3
View File
@@ -22,9 +22,11 @@ blocker and SponsorBlock keep working.
tags to slice the feed by.
- **Playlists with two-way YouTube sync** — build them locally, keep them in sync in both directions.
- **In-app player** with resume, plus keyboard/scroll controls.
- **Download Center** — save videos to the server with yt-dlp in a Plex-friendly layout (format
presets, per-user storage quota), trim / crop / split & join them in a built-in editor, then save
to your device, share with another user, or hand out a public watch link.
- **Files** — one library for everything you keep: paste a link anywhere in the app to download it
with yt-dlp (format presets, per-user storage quota, the running downloads in a tray), upload your
own files, sort them into folders, trim / crop / split & join videos in a built-in editor, play them
where you left off, then save to your device, share with another user, or hand out a public watch
link that plays on any phone.
- **Plex library** *(optional)* — point Siftlode at your own Plex server to browse and play its
movies and shows alongside your feed, with watch state kept in sync.
- **Multi-user** with per-user private state, a shared catalog, and a fair daily API-quota guard.
+1 -1
View File
@@ -1 +1 @@
0.99.2
0.108.0
-214
View File
@@ -437,14 +437,6 @@
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 12,
"endColumn": 13,
"lineCount": 10
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
@@ -638,41 +630,7 @@
}
}
],
"./app/routes/admin_logs.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 15,
"endColumn": 47,
"lineCount": 1
}
}
],
"./app/routes/audit.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 17,
"endColumn": 98,
"lineCount": 1
}
},
{
"code": "reportAssignmentType",
"range": {
"startColumn": 17,
"endColumn": 98,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 22,
"endColumn": 97,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
@@ -735,38 +693,6 @@
}
],
"./app/routes/downloads.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 29,
"endColumn": 33,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 37,
"endColumn": 47,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 29,
"endColumn": 33,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 40,
"endColumn": 50,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
@@ -806,112 +732,6 @@
"endColumn": 54,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 29,
"endColumn": 33,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 40,
"endColumn": 50,
"lineCount": 1
}
}
],
"./app/routes/messages.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 11,
"endColumn": 43,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 23,
"endColumn": 27,
"lineCount": 1
}
},
{
"code": "reportCallIssue",
"range": {
"startColumn": 13,
"endColumn": 5,
"lineCount": 7
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 15,
"lineCount": 5
}
},
{
"code": "reportReturnType",
"range": {
"startColumn": 15,
"endColumn": 72,
"lineCount": 1
}
},
{
"code": "reportCallIssue",
"range": {
"startColumn": 30,
"endColumn": 48,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 41,
"endColumn": 44,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 36,
"endColumn": 38,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 71,
"endColumn": 73,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 30,
"endColumn": 39,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 27,
"endColumn": 29,
"lineCount": 1
}
}
],
"./app/routes/playlists.py": [
@@ -1232,22 +1052,6 @@
"endColumn": 48,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 39,
"endColumn": 45,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 38,
"endColumn": 43,
"lineCount": 1
}
}
],
"./app/routes/plex/playback.py": [
@@ -1318,16 +1122,6 @@
}
}
],
"./app/routes/public.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 57,
"endColumn": 71,
"lineCount": 1
}
}
],
"./app/routes/scheduler.py": [
{
"code": "reportArgumentType",
@@ -1553,14 +1347,6 @@
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 38,
"endColumn": 43,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
@@ -6,7 +6,7 @@ Create Date: 2026-07-07
Some sources (a direct media URL like a reddit v.redd.it HLS manifest) carry no thumbnail, so the
download shows a blank image box. The worker now extracts a representative frame with ffmpeg and
saves it as a poster sidecar; `poster_path` (relative to download_root) records it so the card /
saves it as a poster sidecar; `poster_path` (relative to files_root) records it so the card /
watch page / link preview can fall back to it. Nullable — only set when there was no thumbnail.
"""
@@ -0,0 +1,69 @@
"""messages: drop direct messaging (the owner's decision, 2026-10-04)
Chat — end-to-end encrypted direct messages plus the server's welcome note — was never used: prod held
two welcome notes, no message between people, and one key. Its tables go with its code. The rows are
not migrated anywhere; the nightly backup holds them.
The downgrade restores the schema as it stood (0026 + 0027), empty: a message can't be brought back
from a migration, and the E2EE key blobs were only ever readable by their owner's browser anyway.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0097_drop_messages"
down_revision: str | None = "0096_videos_sort_indexes"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.drop_table("message_keys")
op.drop_index("ix_messages_created_at", table_name="messages")
op.drop_index("ix_messages_recipient_id", table_name="messages")
op.drop_index("ix_messages_sender_id", table_name="messages")
op.drop_table("messages")
def downgrade() -> None:
op.create_table(
"messages",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"sender_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=True
),
sa.Column(
"recipient_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("body", sa.Text(), nullable=True),
sa.Column("read_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("kind", sa.String(length=16), nullable=False, server_default="user"),
sa.Column("ciphertext", sa.Text(), nullable=True),
sa.Column("iv", sa.String(length=32), nullable=True),
)
op.create_index("ix_messages_sender_id", "messages", ["sender_id"])
op.create_index("ix_messages_recipient_id", "messages", ["recipient_id"])
op.create_index("ix_messages_created_at", "messages", ["created_at"])
op.create_table(
"message_keys",
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
primary_key=True,
),
sa.Column("public_key", sa.Text(), nullable=False),
sa.Column("wrapped_private_key", sa.Text(), nullable=False),
sa.Column("salt", sa.String(length=64), nullable=False),
sa.Column("wrap_iv", sa.String(length=32), nullable=False),
sa.Column("key_check", sa.Text(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
@@ -0,0 +1,55 @@
"""demo: drop the shared demo account (the owner's decision, 2026-10-04)
The demo account — one shared user that whitelisted emails entered from the login page, reset on a
schedule — goes with its code. Prod held the demo user and 10 whitelist rows.
The upgrade deletes the demo user first (its playlists, states and subscriptions cascade; audit and
quota rows keep their history with the actor nulled, as for any deleted account), then the traces
that only the demo code could render: its audit rows (whitelist edits, manual resets, and every row
about the reset job — its runs and its interval changes), the reset job's interval override and its
completion notices. Nothing is migrated anywhere; the nightly backup holds the rows.
The downgrade restores the schema as it stood (0014), empty: the demo user is created lazily on its
first sign-in, so an empty `demo_whitelist` is the whole of its state.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0098_drop_demo_account"
down_revision: str | None = "0097_drop_messages"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.execute("DELETE FROM users WHERE is_demo")
op.execute(
"DELETE FROM audit_log WHERE action IN ('demo_add', 'demo_remove', 'demo_reset')"
" OR (target_type = 'job' AND target_id = 'demo_reset')"
)
op.execute("DELETE FROM scheduler_settings WHERE job_id = 'demo_reset'")
op.execute("DELETE FROM notifications WHERE data->>'job_id' = 'demo_reset'")
op.drop_index("ix_demo_whitelist_email", table_name="demo_whitelist")
op.drop_table("demo_whitelist")
op.drop_index("ix_users_is_demo", table_name="users")
op.drop_column("users", "is_demo")
def downgrade() -> None:
op.add_column(
"users",
sa.Column("is_demo", sa.Boolean(), nullable=False, server_default=sa.false()),
)
op.create_index("ix_users_is_demo", "users", ["is_demo"])
op.create_table(
"demo_whitelist",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("email", sa.String(length=320), nullable=False),
sa.Column("note", sa.Text(), nullable=True),
sa.Column("added_by", sa.String(length=320), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index("ix_demo_whitelist_email", "demo_whitelist", ["email"], unique=True)
@@ -0,0 +1,42 @@
"""library: a resume point per user and library item
The library player opened every item at 0:00. `library_states` keeps where a user stopped, per job
(their own download, or one shared with them) — the same shape as `video_states`' position, without a
status: the library has no "watched". It starts empty; the downgrade drops it.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0099_library_states"
down_revision: str | None = "0098_drop_demo_account"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"library_states",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"job_id",
sa.Integer(),
sa.ForeignKey("download_jobs.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("position_seconds", sa.Integer(), nullable=False),
sa.Column("progress_updated_at", sa.DateTime(timezone=True), nullable=False),
sa.UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),
)
op.create_index("ix_library_states_job_id", "library_states", ["job_id"])
def downgrade() -> None:
op.drop_index("ix_library_states_job_id", table_name="library_states")
op.drop_table("library_states")
@@ -0,0 +1,22 @@
"""plex_items: drop `markers`, a column nothing writes or reads
Intro/credits markers are read live from Plex for the item the player opens (with Plex's `final` flag
on the block that ends it); this cached copy was never filled — every row is NULL. The downgrade
restores the empty column.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0100_drop_plex_item_markers"
down_revision: str | None = "0099_library_states"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.drop_column("plex_items", "markers")
def downgrade() -> None:
op.add_column("plex_items", sa.Column("markers", sa.JSON(), nullable=True))
@@ -0,0 +1,46 @@
"""Plex summaries and taglines as plain text
Revision ID: 0101_plex_text_plain
Revises: 0100_drop_plex_item_markers
Create Date: 2026-10-08
Some Plex agents write a summary as HTML (`<p>…</p>`, `&aacute;`), and the app showed the tags as
text. The sync now stores every summary and tagline through `plain_text` (app/plex/plaintext.py);
this cleans the rows it wrote before, through the same function, so nothing waits for a full resync.
"""
import sqlalchemy as sa
from alembic import op
from app.plex.plaintext import plain_text
revision: str = "0101_plex_text_plain"
down_revision: str | None = "0100_drop_plex_item_markers"
branch_labels: str | None = None
depends_on: str | None = None
_COLUMNS = {
"plex_items": ("summary", "summary_hu", "tagline", "tagline_hu"),
"plex_shows": ("summary", "summary_hu", "tagline", "tagline_hu"),
"plex_collections": ("summary",),
}
def upgrade() -> None:
conn = op.get_bind()
for table, columns in _COLUMNS.items():
for col in columns:
rows = conn.execute(
sa.text(f"SELECT id, {col} FROM {table} WHERE {col} ~ '[<&]'")
).fetchall()
updates = [
{"id": rid, "v": clean}
for rid, value in rows
if (clean := plain_text(value)) != value
]
if updates:
conn.execute(sa.text(f"UPDATE {table} SET {col} = :v WHERE id = :id"), updates)
def downgrade() -> None:
pass # the markup is not wanted back; the next sync would clean it again
@@ -0,0 +1,193 @@
"""The Files library: folders, and a file's retention on the user's job
Revision ID: 0102_file_folders
Revises: 0101_plex_text_plain
Create Date: 2026-10-11
Every download, clip and upload of a user now sits in a folder of their Files library
(`app.downloads.folders`): a source folder made per origin (YouTube, Plex, uploads, one per other
site) or one of their own. How long a file stays moves from the shared asset to each user's job
(`download_jobs.expires_at`, stamped from the folder's `retention_days`), so one user keeping a file
no longer depends on another's copy; the asset's own `expires_at` / `gc_notified` go.
This places every existing job in its source folder and carries each done job's expiry over from
the asset it holds (an upload keeps none). The source-key rule is copied here, frozen, from
`folders.source_key_for` as it is today — a migration must not follow the code's later changes.
"""
from urllib.parse import urlsplit
import sqlalchemy as sa
from alembic import op
revision: str = "0102_file_folders"
down_revision: str | None = "0101_plex_text_plain"
branch_labels: str | None = None
depends_on: str | None = None
_FRONT_DOORS = ("www.", "m.", "mobile.", "web.")
_YOUTUBE_HOSTS = {"youtube.com", "youtu.be", "youtube-nocookie.com", "music.youtube.com"}
_DEFAULT_RETENTION_DAYS = 30 # config.download_retention_days on the day this was written
def _site_key(url: str) -> str:
host = (urlsplit(url).hostname or "").lower().rstrip(".")
for door in _FRONT_DOORS:
if host.startswith(door) and host.count(".") > 1:
host = host[len(door) :]
break
if host in _YOUTUBE_HOSTS:
return "youtube"
return f"site:{host or 'unknown'}"
def _asset_key(kind: str | None, ref: str | None) -> str:
if kind == "youtube":
return "youtube"
if kind == "url" and ref:
return _site_key(ref)
return "upload"
def _key(job, jobs_by_id, assets) -> str:
kind = job.source_kind
if kind == "youtube":
return "youtube"
if kind == "upload":
return "upload"
if kind == "clip":
spec = job.edit_spec or {}
return "plex" if spec.get("source") == "plex" else "youtube"
if kind == "url":
return _site_key(job.source_ref)
if kind == "edit":
parent = jobs_by_id.get(job.source_job_id)
if parent is not None and parent.id != job.id:
return _key(parent, jobs_by_id, assets)
asset = assets.get(job.source_asset_id)
if asset is not None:
return _asset_key(asset.source_kind, asset.source_ref)
return "upload"
def upgrade() -> None:
op.create_table(
"file_folders",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("kind", sa.String(16), nullable=False),
sa.Column("source_key", sa.String(260)), # "site:" + a host (<= 253)
sa.Column("name", sa.String(255)),
sa.Column("retention_days", sa.Integer()),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
sa.CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
op.create_index("ix_file_folders_user_id", "file_folders", ["user_id"])
op.create_index(
"uq_file_folder_own_name",
"file_folders",
["user_id", sa.text("lower(name)")],
unique=True,
postgresql_where=sa.text("name IS NOT NULL"),
)
op.add_column(
"download_jobs",
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="RESTRICT"),
),
)
op.add_column(
"download_jobs",
sa.Column("kept", sa.Boolean(), server_default="false", nullable=False),
)
op.add_column("download_jobs", sa.Column("expires_at", sa.DateTime(timezone=True)))
op.add_column(
"download_jobs",
sa.Column("expiry_warned", sa.Boolean(), server_default="false", nullable=False),
)
op.create_index("ix_download_jobs_folder_id", "download_jobs", ["folder_id"])
op.create_index("ix_download_jobs_expires_at", "download_jobs", ["expires_at"])
conn = op.get_bind()
raw = conn.execute(
sa.text("SELECT value FROM system_config WHERE key = 'download_retention_days'")
).scalar()
retention = int(raw) if raw not in (None, "") else _DEFAULT_RETENTION_DAYS
jobs = conn.execute(
sa.text(
"SELECT j.id, j.user_id, j.source_kind, j.source_ref, j.edit_spec, j.source_job_id,"
" j.source_asset_id, j.status, j.asset_id, a.expires_at AS asset_expires_at"
" FROM download_jobs j LEFT JOIN media_assets a ON a.id = j.asset_id"
)
).fetchall()
jobs_by_id = {j.id: j for j in jobs}
assets = {
a.id: a
for a in conn.execute(sa.text("SELECT id, source_kind, source_ref FROM media_assets"))
}
folder_ids: dict[tuple[int, str], int] = {}
for job in jobs:
key = _key(job, jobs_by_id, assets)
slot = (job.user_id, key)
if slot not in folder_ids:
folder_ids[slot] = conn.execute(
sa.text(
"INSERT INTO file_folders (user_id, kind, source_key, retention_days)"
" VALUES (:u, 'source', :k, :r) RETURNING id"
),
{"u": job.user_id, "k": key, "r": None if key == "upload" else retention},
).scalar_one()
# A done file keeps the expiry it had (its asset's); an upload had none and keeps none.
expires = job.asset_expires_at if job.status == "done" and key != "upload" else None
conn.execute(
sa.text("UPDATE download_jobs SET folder_id = :f, expires_at = :e WHERE id = :id"),
{"f": folder_ids[slot], "e": expires, "id": job.id},
)
op.drop_column("media_assets", "expires_at")
op.drop_column("media_assets", "gc_notified")
def downgrade() -> None:
op.add_column("media_assets", sa.Column("expires_at", sa.DateTime(timezone=True)))
op.add_column(
"media_assets",
sa.Column("gc_notified", sa.Boolean(), server_default="false", nullable=False),
)
# The asset goes when its last holder would: the latest of its holders' expiries (a holder that
# never expires keeps the asset unexpired).
op.execute(
"UPDATE media_assets a SET expires_at = sub.e FROM ("
" SELECT asset_id, CASE WHEN bool_or(expires_at IS NULL) THEN NULL ELSE max(expires_at) END"
" AS e FROM download_jobs WHERE asset_id IS NOT NULL AND status = 'done' GROUP BY asset_id"
") sub WHERE a.id = sub.asset_id AND a.source_kind <> 'upload'"
)
op.drop_index("ix_download_jobs_expires_at", table_name="download_jobs")
op.drop_index("ix_download_jobs_folder_id", table_name="download_jobs")
op.drop_column("download_jobs", "expiry_warned")
op.drop_column("download_jobs", "expires_at")
op.drop_column("download_jobs", "kept")
op.drop_column("download_jobs", "folder_id")
op.drop_index("uq_file_folder_own_name", table_name="file_folders")
op.drop_index("ix_file_folders_user_id", table_name="file_folders")
op.drop_table("file_folders")
@@ -0,0 +1,32 @@
"""The Files library: a file's SHA-256, and the expired rows retired
Revision ID: 0103_files_sha256
Revises: 0102_file_folders
Create Date: 2026-10-11
`media_assets.sha256` holds the file's hash once it is known (an upload is hashed while it streams
in, any other file on the first ask), so a machine fetching a file can check what it got.
An expired file now leaves its user's library (`downloads.gc`). Before, an expired download stayed
as a fileless "expired · download again" row (a done job with no asset). Those rows hold nothing on
disk and the library no longer shows them, so they are deleted here rather than left behind
unseen; their shares and public links go with them (ON DELETE CASCADE), and those already served
nothing. The downgrade cannot bring them back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0103_files_sha256"
down_revision: str | None = "0102_file_folders"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("media_assets", sa.Column("sha256", sa.String(64)))
op.execute("DELETE FROM download_jobs WHERE status = 'done' AND asset_id IS NULL")
def downgrade() -> None:
op.drop_column("media_assets", "sha256")
@@ -0,0 +1,38 @@
"""A file's "Added" is when it became done
Revision ID: 0104_job_ready_at
Revises: 0103_files_sha256
Create Date: 2026-10-12
`download_jobs.ready_at` is when a job became done, and the Files library sorts and dates by it: a
download that failed for days and then went through used to sit days down the list, under its
request date. Every path to done stamps it, held by `ck_download_jobs_ready_at`.
A done job from before has no record of that moment (its "ready to watch" notice never named the
job, so it cannot be matched back), so it takes its request date, which is where it already sat.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0104_job_ready_at"
down_revision: str | None = "0103_files_sha256"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("ready_at", sa.DateTime(timezone=True)))
op.execute(
"""
UPDATE download_jobs SET ready_at = created_at WHERE status = 'done'
"""
)
op.create_check_constraint(
"ck_download_jobs_ready_at", "download_jobs", "status <> 'done' OR ready_at IS NOT NULL"
)
def downgrade() -> None:
op.drop_constraint("ck_download_jobs_ready_at", "download_jobs", type_="check")
op.drop_column("download_jobs", "ready_at")
@@ -0,0 +1,37 @@
"""A file remembers where it came from
Revision ID: 0105_job_source_key
Revises: 0104_job_ready_at
Create Date: 2026-10-12
`download_jobs.source_key` is the origin a file came from — the source folder it may go back to —
recorded when it is placed. An edit used to work its origin out through its parent each time, and
once the parent was deleted or expired it fell back to "upload": moved into an own folder, it could
no longer go back to its folder, and deleting that folder sent it to Uploads to stay forever.
A file in a source folder takes that folder's key. One already in an own folder is left NULL and
keeps being worked out (`folders.source_key_for`), as before.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0105_job_source_key"
down_revision: str | None = "0104_job_ready_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("source_key", sa.String(260)))
op.execute(
"""
UPDATE download_jobs j SET source_key = f.source_key
FROM file_folders f
WHERE f.id = j.folder_id AND f.kind = 'source'
"""
)
def downgrade() -> None:
op.drop_column("download_jobs", "source_key")
+110
View File
@@ -0,0 +1,110 @@
"""Machine access to the Files library: API tokens bound to folders, and the Hub folder
Revision ID: 0106_api_tokens
Revises: 0105_job_source_key
Create Date: 2026-10-14
`api_tokens` is a user's token for a machine (a script, the hub): only its SHA-256 is kept, plus the
first characters to recognise it by, and whether its user was told it is about to run out. `api_token_grants` says what it may do in which folder — list,
read, write, delete — so a token reaches nothing it was not given, folder by folder.
A third folder kind, `hub`: the one folder a user exchanges files with their hub machine through.
Like a source folder its name is the client's to render; like an own folder a file may be moved in.
One per user.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0106_api_tokens"
down_revision: str | None = "0105_job_source_key"
branch_labels: str | None = None
depends_on: str | None = None
_KIND_OLD = (
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
)
_KIND_NEW = _KIND_OLD + " OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)"
def upgrade() -> None:
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_NEW)
op.create_index(
"uq_file_folder_hub",
"file_folders",
["user_id"],
unique=True,
postgresql_where=sa.text("kind = 'hub'"),
)
op.create_table(
"api_tokens",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("name", sa.String(80), nullable=False),
sa.Column("prefix", sa.String(16), nullable=False),
sa.Column("token_hash", sa.String(64), nullable=False, unique=True),
sa.Column("expires_at", sa.DateTime(timezone=True)),
sa.Column("last_used_at", sa.DateTime(timezone=True)),
sa.Column("expiry_warned", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_table(
"api_token_grants",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"token_id",
sa.Integer(),
sa.ForeignKey("api_tokens.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("can_list", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_read", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_write", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_delete", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
sa.CheckConstraint(
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
),
)
def downgrade() -> None:
op.drop_table("api_token_grants")
op.drop_table("api_tokens")
op.drop_index("uq_file_folder_hub", table_name="file_folders")
# A Hub folder may hold files (their foreign key RESTRICTs a delete): it becomes an own folder,
# named "Hub <id>" — or "Hub <id> (2)", … when its user already has a folder of that name,
# which `uq_file_folder_own_name` (per user, case-blind) would refuse.
conn = op.get_bind()
hubs = conn.execute(sa.text("SELECT id, user_id FROM file_folders WHERE kind = 'hub'")).all()
for folder_id, user_id in hubs:
name, n = f"Hub {folder_id}", 1
while conn.execute(
sa.text("SELECT 1 FROM file_folders WHERE user_id = :u AND lower(name) = lower(:n)"),
{"u": user_id, "n": name},
).first():
n += 1
name = f"Hub {folder_id} ({n})"
conn.execute(
sa.text("UPDATE file_folders SET kind = 'own', name = :n WHERE id = :i"),
{"n": name, "i": folder_id},
)
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_OLD)
@@ -0,0 +1,66 @@
"""Share-link tokens no longer stored in the clear
Revision ID: 0107_link_token_at_rest
Revises: 0106_api_tokens
Create Date: 2026-10-15
A `/watch` link's token is the credential itself, and it sat in `download_links.token` as written,
so a copy of the database was every shared file. Now a link is found by the token's SHA-256
(`token_hash`), and the token is kept only Fernet-encrypted (`token_enc`, TOKEN_ENCRYPTION_KEY) so its
owner can copy the same link again. Without a key the copy is not kept: the link still opens, its
address just cannot be shown again (`app/downloads/links.py`).
Every existing link is converted here, so the addresses already sent keep working. The downgrade
writes the tokens back from their encrypted copies; a link with none left (no key, or a key since
changed) gets a fresh random token, which is the same as revoking it.
"""
import hashlib
import secrets
import sqlalchemy as sa
from alembic import op
from app import security
revision: str = "0107_link_token_at_rest"
down_revision: str | None = "0106_api_tokens"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_links", sa.Column("token_hash", sa.String(64)))
op.add_column("download_links", sa.Column("token_enc", sa.Text()))
conn = op.get_bind()
sealing = security.can_encrypt()
for link_id, token in conn.execute(sa.text("SELECT id, token FROM download_links")).fetchall():
conn.execute(
sa.text("UPDATE download_links SET token_hash = :h, token_enc = :e WHERE id = :id"),
{
"id": link_id,
"h": hashlib.sha256(token.encode()).hexdigest(),
"e": security.encrypt(token) if sealing else None,
},
)
op.alter_column("download_links", "token_hash", nullable=False)
op.create_unique_constraint("uq_download_link_token_hash", "download_links", ["token_hash"])
op.drop_constraint("uq_download_link_token", "download_links", type_="unique")
op.drop_column("download_links", "token")
def downgrade() -> None:
op.add_column("download_links", sa.Column("token", sa.String(64)))
conn = op.get_bind()
rows = conn.execute(sa.text("SELECT id, token_enc FROM download_links")).fetchall()
for link_id, enc in rows:
token = security.decrypt_optional(enc) or secrets.token_urlsafe(24)
conn.execute(
sa.text("UPDATE download_links SET token = :t WHERE id = :id"),
{"id": link_id, "t": token},
)
op.alter_column("download_links", "token", nullable=False)
op.create_unique_constraint("uq_download_link_token", "download_links", ["token"])
op.drop_constraint("uq_download_link_token_hash", "download_links", type_="unique")
op.drop_column("download_links", "token_enc")
op.drop_column("download_links", "token_hash")
@@ -0,0 +1,40 @@
"""A share link may open a folder
Revision ID: 0108_folder_links
Revises: 0107_link_token_at_rest
Create Date: 2026-10-15
A `/watch` link opened one file. Now it may open one of its owner's own folders instead: the link's
page lists what is in the folder when it is opened. `folder_id` sits beside `job_id`, exactly one of
the two set; a folder's links go with the folder.
The downgrade has nowhere to keep a folder link, so it deletes them: they stop opening.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0108_folder_links"
down_revision: str | None = "0107_link_token_at_rest"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"download_links",
sa.Column("folder_id", sa.Integer(), sa.ForeignKey("file_folders.id", ondelete="CASCADE")),
)
op.create_index("ix_download_links_folder_id", "download_links", ["folder_id"])
op.alter_column("download_links", "job_id", nullable=True)
op.create_check_constraint(
"ck_download_link_target", "download_links", "(job_id IS NULL) <> (folder_id IS NULL)"
)
def downgrade() -> None:
op.execute("DELETE FROM download_links WHERE folder_id IS NOT NULL")
op.drop_constraint("ck_download_link_target", "download_links", type_="check")
op.alter_column("download_links", "job_id", nullable=False)
op.drop_index("ix_download_links_folder_id", table_name="download_links")
op.drop_column("download_links", "folder_id")
+160
View File
@@ -0,0 +1,160 @@
"""Background runs get a durable history
Revision ID: 0109_job_runs
Revises: 0108_folder_links
Create Date: 2026-10-17
What a scheduler job last did lived in the API process's memory, gone at every restart; the audit
log kept a row for a run only when it was manual, failed, or changed something. Now every run is a
`job_runs` row, written when it starts and when it ends. A run's quota spend points at it
(`quota_events.job_run_id`), and so do the lines it logged (`log_events.run_id`, no foreign key: the
log writer runs behind the event).
Runs are not admin actions, so the audit log stops carrying them: the existing `job_run` audit rows
MOVE into `job_runs` (start = end = when the row was written, the only time it kept), and leave the
audit log. The downgrade moves every run back into the audit log as such a row (one still running as
interrupted, what the audit row has no column for in its `after`), then drops the table.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0109_job_runs"
down_revision: str | None = "0108_folder_links"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"job_runs",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("job", sa.String(40), nullable=False),
sa.Column("trigger", sa.String(16), nullable=False),
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="SET NULL")),
sa.Column("target", sa.String(128)),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True)),
sa.Column("summary", sa.Text()),
sa.Column("result", postgresql.JSONB()),
sa.Column("error", sa.Text()),
sa.Column("failure", postgresql.JSONB()),
sa.Column("quota_units", sa.Integer(), nullable=False, server_default="0"),
sa.CheckConstraint(
"status IN ('running', 'ok', 'error', 'skipped', 'interrupted')",
name="ck_job_runs_status",
),
sa.CheckConstraint(
"trigger IN ('scheduled', 'manual', 'ad_hoc')", name="ck_job_runs_trigger"
),
sa.CheckConstraint(
"(status = 'running') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
op.create_index("ix_job_runs_job_started", "job_runs", ["job", "started_at"])
op.create_index("ix_job_runs_actor_id", "job_runs", ["actor_id"])
op.create_index("ix_job_runs_started_at", "job_runs", ["started_at"])
op.add_column(
"quota_events",
sa.Column("job_run_id", sa.BigInteger(), sa.ForeignKey("job_runs.id", ondelete="SET NULL")),
)
op.create_index("ix_quota_events_job_run_id", "quota_events", ["job_run_id"])
op.add_column("log_events", sa.Column("run_id", sa.BigInteger()))
op.create_index(
"ix_log_events_run",
"log_events",
["run_id"],
postgresql_where=sa.text("run_id IS NOT NULL"),
)
# The audit trail's runs move over: the three statuses the scheduler wrote, and `interrupted`,
# which a downgrade writes back (as it does a run's trigger, target, spend and start, which the
# audit row has no column for: its actor alone cannot say "manual" once that admin is deleted),
# so a rollback and a redeploy keep every run as it was. A status the run table does
# not know would fail the CHECK.
op.execute(
"""
INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at, finished_at,
summary, result, error, failure, quota_units)
SELECT target_id,
coalesce(after::jsonb ->> 'trigger',
CASE WHEN actor_id IS NULL THEN 'scheduled' ELSE 'manual' END),
actor_id,
after::jsonb ->> 'target',
after::jsonb ->> 'status',
coalesce((after::jsonb ->> 'started_at')::timestamptz, created_at),
created_at,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb -> 'result' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb -> 'result' END,
coalesce((after::jsonb ->> 'quota_units')::int, 0)
FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
ORDER BY id
"""
)
op.execute(
"""
DELETE FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
"""
)
def downgrade() -> None:
# A run still `running` is one the rollback stopped: it goes back as interrupted. What the audit
# row's own columns cannot say rides in `after` (its trigger, target, spend and start), so the
# upgrade restores them.
op.execute(
"""
INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after, created_at)
SELECT actor_id,
'job_run',
'job',
job,
left(job || ': ' || st
|| coalesce(' — ' || CASE WHEN st = 'error' THEN error ELSE summary END, ''),
255),
(jsonb_build_object(
'status', st,
'summary', CASE WHEN st = 'error' THEN error ELSE summary END)
|| CASE
WHEN st = 'error' AND failure IS NOT NULL
THEN jsonb_build_object('result', failure)
WHEN st <> 'error' AND result IS NOT NULL
THEN jsonb_build_object('result', result)
ELSE '{}'::jsonb
END
|| jsonb_build_object('trigger', trigger, 'quota_units', units,
'started_at', started_at)
|| CASE WHEN target IS NOT NULL
THEN jsonb_build_object('target', target) ELSE '{}'::jsonb END)::json,
coalesce(finished_at, started_at)
FROM (SELECT *, CASE WHEN status = 'running' THEN 'interrupted' ELSE status END AS st,
-- A run stopped mid-way has not summed its spend yet: sum it as the startup
-- sweep would, before the link to it is dropped.
CASE WHEN status = 'running'
THEN (SELECT coalesce(sum(q.units), 0) FROM quota_events q
WHERE q.job_run_id = job_runs.id)
ELSE quota_units END AS units
FROM job_runs) AS r
ORDER BY id
"""
)
op.drop_index("ix_log_events_run", table_name="log_events")
op.drop_column("log_events", "run_id")
op.drop_index("ix_quota_events_job_run_id", table_name="quota_events")
op.drop_column("quota_events", "job_run_id")
op.drop_index("ix_job_runs_started_at", table_name="job_runs")
op.drop_index("ix_job_runs_actor_id", table_name="job_runs")
op.drop_index("ix_job_runs_job_started", table_name="job_runs")
op.drop_table("job_runs")
@@ -0,0 +1,79 @@
"""Job intervals move into sysconfig; the unread SponsorBlock default goes
Revision ID: 0110_intervals_in_sysconfig
Revises: 0109_job_runs
Create Date: 2026-10-18
An admin's interval override lived in its own table, `scheduler_settings`, beside the one store every
other admin knob uses. Each override becomes the `job_interval_<job>` sysconfig key and the table
goes. Only the jobs that still exist are copied: a row for any other id never did anything.
The `sponsorblock_default` setting was offered on the Configuration page but nothing ever read it, so
its key is gone and its stored value with it — the downgrade cannot bring that value back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0110_intervals_in_sysconfig"
down_revision: str | None = "0109_job_runs"
branch_labels: str | None = None
depends_on: str | None = None
# The interval jobs as of this revision (a migration does not read the live code's list).
_JOBS = (
"rss_poll",
"enrich",
"backfill",
"autotag",
"shorts",
"subscriptions",
"playlist_sync",
"maintenance",
"explore_cleanup",
"download_gc",
"plex_sync",
"plex_watch_sync",
"plex_watch_reconcile",
"tmdb_enrich",
"people_enrich",
"audit_gc",
"log_gc",
)
_PREFIX = "job_interval_"
def upgrade() -> None:
op.execute(
sa.text(
"INSERT INTO system_config (key, value, encrypted)"
" SELECT :prefix || job_id, interval_minutes::text, false FROM scheduler_settings"
" WHERE interval_minutes IS NOT NULL AND job_id IN :jobs"
" ON CONFLICT (key) DO NOTHING"
).bindparams(sa.bindparam("jobs", expanding=True), prefix=_PREFIX, jobs=list(_JOBS))
)
op.drop_table("scheduler_settings")
op.execute("DELETE FROM system_config WHERE key = 'sponsorblock_default'")
def downgrade() -> None:
op.create_table(
"scheduler_settings",
sa.Column("job_id", sa.String(40), primary_key=True),
sa.Column("interval_minutes", sa.Integer(), nullable=False),
)
op.execute(
sa.text(
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
" SELECT substr(key, :start), value::int FROM system_config WHERE key IN :keys"
).bindparams(
sa.bindparam("keys", expanding=True),
start=len(_PREFIX) + 1,
keys=[_PREFIX + j for j in _JOBS],
)
)
op.execute(
sa.text("DELETE FROM system_config WHERE key IN :keys").bindparams(
sa.bindparam("keys", expanding=True), keys=[_PREFIX + j for j in _JOBS]
)
)
@@ -0,0 +1,28 @@
"""An audit row keeps the request that made it
Revision ID: 0111_audit_request_id
Revises: 0110_intervals_in_sysconfig
Create Date: 2026-10-18
The activity page opens an admin action's log lines from its row: `audit_log.request_id` is the
request the change was made in, the same id `log_events.request_id` carries. Older rows have none —
the request that made them was never recorded.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0111_audit_request_id"
down_revision: str | None = "0110_intervals_in_sysconfig"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("audit_log", sa.Column("request_id", sa.String(64)))
op.create_index("ix_audit_log_request_id", "audit_log", ["request_id"])
def downgrade() -> None:
op.drop_index("ix_audit_log_request_id", table_name="audit_log")
op.drop_column("audit_log", "request_id")
+10 -7
View File
@@ -1,7 +1,7 @@
# Backend conventions
The rules below are already followed across `backend/app` — they lived only in a few module
docstrings (`demo.py`, `userscope.py`, `routes/_common.py`) until this file collected them (E3 S3,
docstrings (`userscope.py`, `routes/_common.py`) until this file collected them (E3 S3,
B-core-20). When you add code, match these so the next reader doesn't have to guess "which pattern do
I copy".
@@ -20,8 +20,10 @@ Three layers, by *who* runs the code — not by feature:
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline;
`issues/bundle.py` is the one definition of a bug report's files, for the zip export and the tracker.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, `logs` (the one logging
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth` (with `apitokens`, a machine's
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`, `jobruns` (every
background run's durable row, its words in `runstatus`),
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core
@@ -35,7 +37,7 @@ pre-abstract for a single caller; don't leave shared logic duplicated once there
## Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
(only `quota` and `jobruns` own their session)"** — so read it that way, not as an exception to it. The commit belongs
to the **operation owner**; anything called *inside* another transaction only stages its change
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
@@ -44,7 +46,7 @@ mutation it records, so an audit entry can never exist without its change (or vi
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
`jobruns`' run rows — the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
attribute — see `service.decr_ref_count`.
@@ -66,8 +68,9 @@ mutation it records, so an audit entry can never exist without its change (or vi
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
startup, not user-tunable at runtime.
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
slated to fold into `sysconfig` — do not add more like it.)
goes in `SPECS`; don't invent a fourth store. Each spec states when a change takes effect
(`applies`, required): `live` keys are read where they are used, never at import (test-held). The
scheduler's job intervals are `SPECS` keys too, edited on the Scheduler page, not the Configuration page.
## The gate
+273
View File
@@ -0,0 +1,273 @@
"""Machine access to the Files library: API tokens, and what each one may do where.
A user makes a token for a machine of theirs — their hub, a script — and it reaches their library as
`Authorization: Bearer sfl_…`. Only on `/api/files`: the routes there take `files_caller` instead of
`current_user`, and nothing else does (a test holds the route table to that), so a token can never
manage the account, the tokens, or anything outside the library. And only in the folders its grants
name, with the rights they give: list, read, write, delete. Every route asks the one question,
`FilesCaller.may(folder_id, right)`; a signed-in user may everything in their own folders.
The token is `sfl_` + 32 random bytes in base62 + a CRC32 of them: a mistyped or cut-off token is
refused without a database read, and a secret scanner can tell one by its shape. Only its SHA-256 is
kept (`security.hash_token`) — the user sees it once, when it is made — with its first characters to
tell tokens apart by. It is looked up by that hash on every request: no signed claims, so a deleted
token stops working at once. A token about to run out warns its user once, a week before
(`warn_expiring`), so a machine does not lose its key unannounced.
"""
import secrets
import zlib
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from fastapi import Depends, Request
from sqlalchemy import delete, or_, select, update
from sqlalchemy.orm import Session
from app import audit, debugsession, logs
from app.audit import AuditAction
from app.auth import current_user
from app.db import get_db
from app.models import ApiToken, ApiTokenGrant, User
from app.notifications import create_notification
from app.ratelimit import RateLimiter, client_key
from app.refusal import refused
from app.security import hash_token
from app.utils import iso
log = logs.get_logger(__name__)
PREFIX = "sfl_"
_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
_BODY_BYTES = 32
_BODY_LEN = 43 # 62**43 > 2**256: every 32-byte value fits
_CHECK_LEN = 6 # 62**6 > 2**32: every CRC32 fits
TOKEN_LEN = len(PREFIX) + _BODY_LEN + _CHECK_LEN
# What a token is shown by once it is made: enough to tell a user's tokens apart, far too little to
# narrow a guess (8 of its 43 random characters).
SHOWN_LEN = len(PREFIX) + 8
# A machine that runs away (a loop in a script) is held back per token; a caller trying tokens is
# held back per client, counting only the refusals.
_calls = RateLimiter(600, 60)
_bad = RateLimiter(20, 300)
# How stale `last_used_at` may get before a request writes it again: a busy script costs one write a
# minute, not one per request.
_TOUCH_EVERY = timedelta(minutes=1)
# How long before a token runs out its user is told (`warn_expiring`); the token list marks it
# "expires soon" for the same span.
EXPIRY_WARNING = timedelta(days=7)
class Right(StrEnum):
LIST = "list"
READ = "read"
WRITE = "write"
DELETE = "delete"
# Each right's column on a grant. A dict over the enum, so a new right without a column fails here.
GRANT_COLUMNS = {
Right.LIST: "can_list",
Right.READ: "can_read",
Right.WRITE: "can_write",
Right.DELETE: "can_delete",
}
def _b62(value: int, width: int) -> str:
out = []
for _ in range(width):
value, digit = divmod(value, 62)
out.append(_ALPHABET[digit])
return "".join(reversed(out))
def _check(body: str) -> str:
return _b62(zlib.crc32(body.encode()), _CHECK_LEN)
def mint() -> str:
"""A new token, in full. The caller stores its hash and shows it to the user, once."""
body = _b62(int.from_bytes(secrets.token_bytes(_BODY_BYTES)), _BODY_LEN)
return PREFIX + body + _check(body)
def well_formed(raw: str) -> bool:
"""Whether `raw` has a token's shape and its checksum adds up."""
if len(raw) != TOKEN_LEN or not raw.startswith(PREFIX):
return False
body, check = raw[len(PREFIX) : -_CHECK_LEN], raw[-_CHECK_LEN:]
# The check too: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it.
if any(c not in _ALPHABET for c in body + check):
return False
return secrets.compare_digest(_check(body), check)
def grant_rights(grant: ApiTokenGrant) -> frozenset[Right]:
return frozenset(r for r, col in GRANT_COLUMNS.items() if getattr(grant, col))
@dataclass(frozen=True)
class FilesCaller:
"""Who is asking the Files library, and where they may do what. `grants` is None for a signed-in
user (everything in their own folders); for a token, folder id → its rights there."""
user: User
token_id: int | None = None
token_prefix: str | None = None
grants: Mapping[int, frozenset[Right]] | None = None
def may(self, folder_id: int | None, right: Right) -> bool:
if self.grants is None:
return True
return folder_id is not None and right in self.grants.get(folder_id, frozenset())
def folders_with(self, right: Right) -> set[int] | None:
"""The folders `right` is granted in; None = any of the user's own."""
if self.grants is None:
return None
return {f for f, rights in self.grants.items() if right in rights}
@property
def via(self) -> str | None:
"""How an audit row says a token did it, None when the user did."""
return None if self.token_prefix is None else f"api token {self.token_prefix}…"
def bearer_token(request: Request) -> str | None:
"""The token of an `Authorization: Bearer` header; None when the request carries none (another
scheme counts as none — it is not ours to judge)."""
scheme, _, value = request.headers.get("authorization", "").partition(" ")
if scheme.lower() != "bearer":
return None
return value.strip()
def session_caller(request: Request, db: Session = Depends(get_db)) -> User | None:
"""The signed-in user, or None when the request brings a token instead: a token wins over a
cookie, so a machine's request is judged by its grants alone, whatever else it carries."""
if bearer_token(request) is not None:
return None
return current_user(request, db)
def files_caller(
request: Request,
db: Session = Depends(get_db),
user: User | None = Depends(session_caller),
) -> FilesCaller:
"""The `/api/files` caller: a signed-in user, or a valid token's user with its grants."""
if user is not None:
return FilesCaller(user=user)
return _token_caller(request, db, bearer_token(request) or "")
def _refuse_token(key: str, reason: str) -> Exception:
_bad.allow(key)
log.info(
"a request brought an API token that does not hold",
extra=logs.event("apitoken.refused", reason=reason),
)
return refused(401, "not_authenticated")
def _token_caller(request: Request, db: Session, raw: str) -> FilesCaller:
key = client_key(request)
if _bad.exhausted(key):
raise refused(429, "api_token_rate_limited")
if not well_formed(raw):
raise _refuse_token(key, "malformed")
token = db.execute(
select(ApiToken).where(ApiToken.token_hash == hash_token(raw))
).scalar_one_or_none()
if token is None:
raise _refuse_token(key, "unknown")
now = datetime.now(timezone.utc)
if token.expires_at is not None and token.expires_at <= now:
raise _refuse_token(key, "expired")
user = db.get(User, token.user_id)
if user is None or not user.is_active or user.is_suspended:
raise _refuse_token(key, "account")
if not _calls.allow(str(token.id)):
raise refused(429, "api_token_rate_limited")
grants = {
g.folder_id: grant_rights(g)
for g in db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == token.id))
.scalars()
.all()
}
caller = FilesCaller(user=user, token_id=token.id, token_prefix=token.prefix, grants=grants)
db.execute(
update(ApiToken)
.where(
ApiToken.id == token.id,
or_(ApiToken.last_used_at.is_(None), ApiToken.last_used_at < now - _TOUCH_EVERY),
)
.values(last_used_at=now)
.execution_options(synchronize_session=False)
)
# Committed here, ahead of the route: a read-only route never commits, and a write route must
# not carry this along into a rollback of its own.
db.commit()
logs.set_user(user.id, *debugsession.level_for(user, None))
request.state.user_id = user.id
return caller
def revoke_all(db: Session, user_id: int) -> int:
"""Revoke every token of `user_id`, each with its audit row (NOT committed). A token is a
credential like a session, so whatever ends every session of an account — a password reset or
change, "log out everywhere", a revoked untrusted password — ends its tokens too: one minted
from a stolen session must not outlive the owner's answer to that theft. Returns how many."""
gone = (
db.execute(delete(ApiToken).where(ApiToken.user_id == user_id).returning(ApiToken.id))
.scalars()
.all()
)
for token_id in gone:
audit.record(
db,
user_id,
AuditAction.API_TOKEN_REVOKE,
target_type="api_token",
target_id=token_id,
summary="revoked with every session of the account",
)
return len(gone)
def expiring(token: ApiToken, now: datetime) -> bool:
"""Whether `token` runs out within the warning span (and has not yet)."""
return token.expires_at is not None and now < token.expires_at <= now + EXPIRY_WARNING
def warn_expiring(db: Session) -> int:
"""Tell each user whose token runs out within `EXPIRY_WARNING` — once per token: the mark and the
notice commit together, and a token made again (a new row) warns afresh. Returns how many."""
now = datetime.now(timezone.utc)
hits = db.execute(
update(ApiToken)
.where(
ApiToken.expiry_warned.is_(False),
ApiToken.expires_at.is_not(None),
ApiToken.expires_at > now,
ApiToken.expires_at <= now + EXPIRY_WARNING,
)
.values(expiry_warned=True)
.returning(ApiToken.id, ApiToken.user_id, ApiToken.name, ApiToken.expires_at)
.execution_options(synchronize_session=False)
).all()
for token_id, user_id, name, expires_at in hits:
create_notification(
db,
user_id,
"api_token_expiring",
title=f"API token {name} expires soon", # English fallback; the client names it from data
data={"token_id": token_id, "name": name, "expires_at": iso(expires_at)},
commit=False,
)
db.commit()
return len(hits)
+31 -13
View File
@@ -1,9 +1,9 @@
"""Admin/scheduler audit trail: an append-only record of who changed what and when.
"""Admin audit trail: an append-only record of who changed what and when.
Design (locked): log at ACTION / job-run granularity, NEVER per-row — a scheduler run that
touches thousands of videos is ONE audit row summarising the outcome. Automatic job runs are
only logged when they actually changed something (or errored); all admin actions and all manual
runs are always logged. Secrets are never stored (config changes log the key name only).
Design (locked): log at ACTION granularity, NEVER per-row — an action that touches thousands of
rows is ONE audit row summarising the outcome. Background runs are not actions: they have their
own history (app/jobruns.py). Secrets are never stored (config changes log the key name only).
Each row keeps the request that made it, so the request's log lines are one click away.
`record()` only db.add()s the row — the caller commits it in the SAME transaction as the
mutation it describes, so the trail can't drift from reality (and a rolled-back mutation drops
@@ -15,6 +15,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from sqlalchemy.orm import Session
from app import logs
from app.models import AuditLog
@@ -27,7 +28,7 @@ class AuditAction:
USER_SUSPEND = "user_suspend"
USER_UNSUSPEND = "user_unsuspend"
USER_DELETE = "user_delete"
# E9.5 S5: an admin let a stuck registrant in, or re-sent their confirmation link. Both are
# An admin let a stuck registrant in, or re-sent their confirmation link. Both are
# recorded because both are the admin acting ON someone's access, not merely reading a list.
USER_ACTIVATE = "user_activate"
USER_VERIFY_RESEND = "user_verify_resend"
@@ -36,12 +37,9 @@ class AuditAction:
INVITE_DENY = "invite_deny"
INVITE_ADD = "invite_add"
INVITE_DELETE = "invite_delete"
# E9.5 S6: the admin chose to tell someone their address is whitelisted. Adding the address is
# The admin chose to tell someone their address is whitelisted. Adding the address is
# silent, so this is the act that actually reaches a person — and the one worth a record.
INVITE_SEND = "invite_send"
DEMO_ADD = "demo_add"
DEMO_REMOVE = "demo_remove"
DEMO_RESET = "demo_reset"
DISCOVERY_PURGE = "discovery_purge"
# Config
CONFIG_SET = "config_set"
@@ -51,16 +49,35 @@ class AuditAction:
MAINTENANCE_TUNE = "maintenance_tune"
SYNC_PAUSE = "sync_pause"
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
# The log itself was cleared (a single row survives the clear as tamper-evidence)
AUDIT_CLEAR = "audit_clear"
# E16: a user sent a debug session's issue to the tracker — the one act that takes their log
# A user sent a debug session's issue to the tracker — the one act that takes their log
# off this instance, so it is recorded (target = the session, summary = the issue's URL).
ISSUE_CREATE = "issue_create"
# The Files library (`routes.files`): what a user put in, took out, kept or reorganised. Ids
# only — a file's or folder's name is the user's own text, not the admin's to read here.
FILE_UPLOAD = "file_upload"
FILE_DELETE = "file_delete"
FILE_KEEP = "file_keep"
FILE_UNKEEP = "file_unkeep"
FILE_MOVE = "file_move"
FILE_FOLDER_CREATE = "file_folder_create"
FILE_FOLDER_RENAME = "file_folder_rename"
FILE_FOLDER_RETENTION = "file_folder_retention"
FILE_FOLDER_DELETE = "file_folder_delete"
# A machine's token to the library (`routes.tokens`): made (its folders and rights) or revoked.
# What a token then did is the file rows above, their summary naming it.
API_TOKEN_CREATE = "api_token_create"
API_TOKEN_REVOKE = "api_token_revoke"
# A public `/watch` link to a file or a folder (`routes.downloads`): made, its settings changed,
# revoked.
# Never its token or password — only whether it has one.
DOWNLOAD_LINK_CREATE = "download_link_create"
DOWNLOAD_LINK_UPDATE = "download_link_update"
DOWNLOAD_LINK_REVOKE = "download_link_revoke"
def record(
@@ -86,6 +103,7 @@ def record(
summary=(summary or None) and summary[:255],
before=before,
after=after,
request_id=logs.current_request_id(),
)
)
+10 -47
View File
@@ -24,8 +24,6 @@ def operator_contact() -> str | None:
return next(iter(sorted(settings.admin_email_set)), None)
DEMO_GOOGLE_SUB = "__demo__"
DEMO_EMAIL = "demo@siftlode.local"
MAX_SESSION_ACCOUNTS = 6
# OAuth scope vocabulary — the atoms + the request bundles, kept together as core. has_*_scope check
@@ -69,7 +67,7 @@ def has_write_scope(user: User) -> bool:
def env_allowed_canonicals() -> set[str]:
"""The env bootstrap allow-list, in CANONICAL form — the only shape it may be compared in.
S3 canonicalised the Invite branch of `is_allowed` and left this one matching the literal string,
Canonicalising only the Invite branch of `is_allowed` left this one matching the literal string,
which is the 2026-09-03 lockout with a different source of truth: an operator who writes
`first.last@gmail.com` into ALLOWED_EMAILS is refused when Google answers with the undotted
primary, and a `+tag`ged entry can never match at all, because a Google account cannot contain a
@@ -129,7 +127,7 @@ def can_activate(db: Session, user: User) -> bool:
Lives in core so the admin surface and the /api/me badge count answer from ONE definition; a
button that appears and a number that counts must never disagree about who is stuck.
"""
if user.is_demo or user.is_suspended or user.is_active:
if user.is_suspended or user.is_active:
return False
return is_allowed(db, user.email)
@@ -147,7 +145,6 @@ def count_pending_activations(db: Session) -> int:
rows = (
db.execute(
select(User).where(
User.is_demo.is_(False),
User.is_suspended.is_(False),
User.is_active.is_(False),
)
@@ -188,24 +185,6 @@ def find_user_by_email(db: Session, email: str) -> User | None:
return candidates[0] if len(candidates) == 1 else None
def get_or_create_demo_user(db: Session) -> User:
"""The one shared demo user, created on first use. No OAuth token / YouTube scope, so
has_read_scope/has_write_scope are False and every YouTube-touching path is closed to it."""
user = db.query(User).filter(User.is_demo.is_(True)).one_or_none()
if user is None:
user = User(
google_sub=DEMO_GOOGLE_SUB,
email=DEMO_EMAIL,
display_name="Demo",
role="user",
is_demo=True,
preferences={"language": "en"},
)
db.add(user)
db.commit()
return user
def revoke_google_token(token: str | None) -> None:
"""Best-effort revocation of a Google OAuth grant at Google's endpoint. Revoking the refresh
token tears down the whole grant, so a later sign-in starts from a clean consent (no scopes
@@ -278,8 +257,7 @@ ACTIVE_ACCOUNT_HEADER = "x-siftlode-account"
def resolved_user_id(conn: HTTPConnection, *, strict: bool = True) -> tuple[int | None, bool]:
"""Which account this connection acts as, and whether that's the session's default account.
Takes any HTTPConnection — an HTTP Request OR a WebSocket — so the WS push channel shares this
exact per-tab resolution instead of re-implementing it.
Takes an HTTPConnection rather than a Request: it reads only the session and the headers.
The signed cookie holds the *wallet* (`account_ids`, every account signed into this browser)
plus a default `user_id`. A caller may name its account with the X-Siftlode-Account header (or
@@ -297,8 +275,8 @@ def resolved_user_id(conn: HTTPConnection, *, strict: bool = True) -> tuple[int
"""
default_id = conn.session.get("user_id")
wallet = conn.session.get("account_ids") or []
# Header for normal XHR; ?account= for contexts that can't set headers (WebSocket, and a
# plain <a> file download). Both are wallet-gated below.
# Header for normal XHR; ?account= for a context that can't set headers (a plain <a> file
# download). Both are wallet-gated below.
hdr = conn.headers.get(ACTIVE_ACCOUNT_HEADER) or conn.query_params.get("account")
if hdr:
try:
@@ -362,12 +340,12 @@ def _session_user(request: Request, db: Session, user_id: int | None, is_default
if default_id and default_id not in accounts:
accounts.append(default_id)
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
# Attribute this request's log lines to the account (E22). Here, not in the request middleware:
# Attribute this request's log lines to the account. Here, not in the request middleware:
# this is the one place a session becomes an account, and the middleware runs outside both the
# session middleware and this task's context. `request.state` is shared across middleware layers
# (it lives on the ASGI scope), so the access-log line can read it back. The user's consented
# diagnostics level rides along: their requests are logged — and stored — in that detail (S5).
# So does a debug session the request names (E16): only the caller's own open one counts, and a
# diagnostics level rides along: their requests are logged — and stored — in that detail.
# So does a debug session the request names: only the caller's own open one counts, and a
# header naming anything else is ignored rather than refused — a stale tab must keep working.
session = debugsession.from_header(db, user, request.headers.get(debugsession.HEADER))
level, until = debugsession.level_for(user, session)
@@ -403,25 +381,10 @@ def optional_current_user(request: Request, db: Session = Depends(get_db)) -> Us
return None
def require_human(user: User = Depends(current_user)) -> User:
"""Reject the shared demo account from actions that need a real Google/YouTube identity
or spend the shared quota. Most YouTube paths are already closed to it implicitly (it has
no token, so has_read_scope/has_write_scope are False); this makes the boundary explicit
where there's no scope check to lean on."""
if user.is_demo:
# Structured, because `req` leaves 403 to the caller and `notifyActionError` only shows a
# STRUCTURED detail — a plain string here reached the demo user as their caller's generic
# "couldn't do that", with no hint that the demo account is the reason (round 6). Affordances
# this gate sits behind are not all UI-gated on `is_demo` (the comfort shelf is not).
raise HTTPException(status_code=403, detail={"code": "demo"})
return user
def require_write_scope(user: User = Depends(require_human)) -> User:
def require_write_scope(user: User = Depends(current_user)) -> User:
"""Dependency: require the YouTube write scope (subscribe/unsubscribe, playlist export/edit).
Single source of truth for the write gate — replaces the inline `if not has_write_scope(user)`
checks that had drifted into three different 403 messages across channels/playlists. Layers on
`require_human` because writing to YouTube needs a real, non-demo identity."""
checks that had drifted into three different 403 messages across channels/playlists."""
if not has_write_scope(user):
raise refused(403, "youtube_editing_off")
return user
+6 -6
View File
@@ -59,9 +59,9 @@ DEFAULT_MAX_BODY_BYTES = 1024 * 1024
# a say.
#
# Both listed routes now satisfy that. The screenshot route always did (`capped_body` streams it,
# and the owner check resolves first). The upload route did NOT until P1: it declared
# and the owner check resolves first). The upload route did NOT at first: it declared
# `file: UploadFile = File(...)`, so an anonymous caller could make the process write the whole
# per-file allowance to temp before `require_human` refused them — this middleware bounded that
# per-file allowance to temp before the auth dependency refused them — this middleware bounded that
# number but could not stop the writing. It now reads its own body from a dependency declared after
# the auth one, so its in-handler cap measures bytes as they arrive, like everything else here.
#
@@ -73,12 +73,12 @@ DEFAULT_MAX_BODY_BYTES = 1024 * 1024
# `POST /api/diagnostics/events` is absent: its `capped_body` limit is 512 KiB, already below the
# default, so listing it could only widen what it accepts.
ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# Ingesting a local file. The body IS the file since P1 — no multipart envelope to allow for —
# Ingesting a local file. The body IS the file now — no multipart envelope to allow for —
# so this ceiling and the route's own cap count exactly the same bytes. `sysconfig`'s
# `download_upload_max_bytes` is bounded by this same constant (see sysconfig.SPECS), so an
# admin can only ever LOWER the route's cap below the edge, never raise it past one.
#
# WHICH LAYER ANSWERS, measured rather than assumed (deep-review round 1):
# WHICH LAYER ANSWERS, measured rather than assumed:
# * knob LOWERED (an override below the default) — the route's cap is the smaller number, so
# `capped_stream_to` trips first and answers 422 `upload_too_large` carrying the admin's own
# figure, after draining the rest of the body so the client can read it.
@@ -92,8 +92,8 @@ ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# So this 413 is a refusal a real user meets on a default install, not a theoretical edge, and
# it has to be as readable as the route's own: the client localizes `too_large` (`errors.tooLarge`)
# exactly as it does `upload_too_large`, and the upload dialog refuses an over-cap file before
# sending it at all, off the `upload_max_bytes` that `GET /api/downloads/usage` reports.
("POST", "/api/downloads/upload"): settings.download_upload_max_bytes,
# sending it at all, off the `upload_max_bytes` that `GET /api/files/usage` reports.
("POST", "/api/files/upload"): settings.download_upload_max_bytes,
("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
}
+1 -1
View File
@@ -1,4 +1,4 @@
"""Alphabetical order in the reader's language (E30 S3).
"""Alphabetical order in the reader's language.
Why not the database default: the Postgres image is Alpine, and musl's `en_US.utf8` collation is BYTE
order — every accented initial sorted after "Z" ("Zorró < Életrajzi < Ötödik"), in every A→Z list the
+30 -23
View File
@@ -1,7 +1,7 @@
from pathlib import Path
from typing import Literal, get_args, get_origin
from pydantic import Field, ValidationInfo, field_validator, model_validator
from pydantic import AliasChoices, Field, ValidationInfo, field_validator, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
# Shipped placeholder; production must override it. Used by the startup guard below.
@@ -146,8 +146,6 @@ class Settings(BaseSettings):
autotag_interval_minutes: int = 30
subscriptions_resync_minutes: int = 360
playlist_sync_minutes: int = 360
# How often the shared demo account is auto-reset to a clean baseline (communal sandbox).
demo_reset_minutes: int = 720
# --- Maintenance / validation job ---
# Runs once a day by default (admin-tunable via the Scheduler dashboard). Grace periods
# are in days because that's the granularity that matters; the rolling re-validation
@@ -173,9 +171,13 @@ class Settings(BaseSettings):
feed_default_hidden_live: str = "live,upcoming"
# --- Download center (yt-dlp) ---
# Root directory (inside the container) for downloaded media — a Docker volume/bind mount.
# Root directory (inside the container) for the files library — a Docker volume/bind mount.
# Env-only (a mount path can't live in the DB). The worker lays a Plex-style tree under it.
download_root: str = "/downloads"
# DOWNLOAD_ROOT is its old name: an install still on the old compose file (mounted at
# /downloads, DOWNLOAD_ROOT=/downloads) keeps finding its files after it pulls a new image.
files_root: str = Field(
default="/files", validation_alias=AliasChoices("FILES_ROOT", "DOWNLOAD_ROOT", "files_root")
)
# Whether THIS process runs the download worker loop. On for the dedicated worker
# container, off for the API (which keeps the scheduler). Mirrors scheduler_enabled.
worker_enabled: bool = False
@@ -183,15 +185,18 @@ class Settings(BaseSettings):
# for YouTube Proof-of-Origin tokens (bypasses bot-detection + unlocks high-quality
# formats). Empty disables it (yt-dlp still works, but may hit "confirm you're not a bot").
download_pot_base_url: str = "http://bgutil-pot:4416"
# yt-dlp player clients (comma-separated). PRIMARY = android_vr: high-quality DASH, no PO
# token / JS runtime needed, reliable for normal use. On a bot/DRM/format failure (YouTube's
# per-client responses are flaky, and android_vr can get bot-flagged under heavy load) the
# worker retries with the FALLBACK set — the POT-capable web clients (web needs the token +
# Deno) plus android for audio. Both admin-tunable when YouTube shifts what works.
# yt-dlp player clients (comma-separated). PRIMARY = tv_simply: measured 2026-10-10 as the one
# client that is offered 1080p+ AND finishes the fetch, its stream token minted by the bgutil
# sidecar. android_vr (the primary until then) was still offered 2160p, but every fetch died
# with HTTP 403 — it now needs a stream PO token the sidecar cannot mint — and the fallback
# below served 360p only, so every download came out at 360p without failing. On a
# bot/DRM/format failure the worker retries with the FALLBACK set — the POT-capable web clients
# (web needs the token + Deno) plus android, whose 360p is the last resort. Both admin-tunable
# (env) when YouTube shifts what works.
# A SIGNED-IN download (the user added YouTube cookies) keeps only the cookie-capable clients
# of these two sets — yt-dlp drops the rest, so android_vr / android / ios never run there.
# See adapters.YouTubeAdapter._COOKIE_CAPABLE.
download_player_clients: str = "android_vr"
download_player_clients: str = "tv_simply"
download_player_clients_fallback: str = "web_safari,web,android"
@staticmethod
@@ -244,7 +249,7 @@ class Settings(BaseSettings):
download_default_max_bytes: int = 5_368_709_120 # 5 GiB
download_default_max_concurrent: int = 2
download_default_max_jobs: int = 50
# Per-file cap for local uploads (E7). Unlike a download (whose size is unknown until the worker
# Per-file cap for local uploads. Unlike a download (whose size is unknown until the worker
# finishes), an upload's size is known at ingest, so this bounds a single file directly; the
# user's overall byte footprint is still governed by download_default_max_bytes.
# ge=1: this number is also the EDGE ceiling (`bodylimit.ROUTE_MAX_BODY_BYTES`), where 0 is a
@@ -263,12 +268,14 @@ class Settings(BaseSettings):
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
quota_retention_days: int = 365
# Background-run history (job_runs): days kept (0 = forever). The audit log's 180, which is how
# long a run lived there before it moved here: ~1.8k rows a day at the default intervals
# (measured 2026-10-17), ~320k at the limit. The same daily cleanup job prunes it.
job_run_retention_days: int = 180
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: DownloadLayout = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
sponsorblock_default: bool = False
# --- Diagnostics / logging (E22) ---
# --- Diagnostics / logging ---
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
# startup and on change without a restart. This is what a process logs at before it can reach
# the database — and the only knob the worker/runner children read, since they are short-lived.
@@ -301,7 +308,7 @@ class Settings(BaseSettings):
plex_path_map: str = ""
# Which Plex library section keys are exposed (comma-separated). Empty = all enabled sections.
plex_libraries: str = ""
# HU-overlay mapping (E5 S5): comma-separated `huSectionKey:enSectionKey` pairs. Each named HU
# HU-overlay mapping: comma-separated `huSectionKey:enSectionKey` pairs. Each named HU
# section is synced as a non-browsable overlay whose Hungarian metadata is written onto the EN
# twin section's rows (matched by guid). Empty = no overlays. E.g. "3:4,5:1".
plex_overlay_map: str = ""
@@ -311,7 +318,7 @@ class Settings(BaseSettings):
plex_sync_interval_min: int = 360
# How often the incremental Plex→Siftlode watch-state reconcile runs, in minutes (Phase C):
# a cheap history + on-deck pull plus a re-push of any states that failed their immediate push.
# 15 min (S2): with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# 15 min: with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# Plex-authoritative pull), a faster incremental pull keeps Siftlode close to Plex-side changes.
plex_watch_sync_interval_min: int = 15
# How often the FULL watch-state reconcile runs, in minutes (Phase C). A whole-section rescan
@@ -326,7 +333,7 @@ class Settings(BaseSettings):
# each one is a whole segmented copy of the film on the scratch volume, written faster than
# realtime. One account holds at most two regardless (plex/stream.py `_MAX_SESSIONS_PER_USER`).
plex_max_sessions: int = 8
# --- Transcode (playable="transcode") re-encode tuning (E6 S1) ---
# --- Transcode (playable="transcode") re-encode tuning ---
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
@@ -344,12 +351,12 @@ class Settings(BaseSettings):
plex_transcode_crf: int = 20
plex_transcode_max_height: int = 1080
# Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
# dir under download_root. On localdev the download_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's download_root is fast
# dir under files_root. On localdev the files_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's files_root is fast
# local disk, so the default is fine there.
plex_hls_dir: str = ""
# --- TMDB enrichment (E5 S2; optional — enriches the Plex catalog from TMDB) ---
# --- TMDB enrichment (optional — enriches the Plex catalog from TMDB) ---
# TMDB API key (v3 api_key, 32 hex chars). Server-side only; stored encrypted when set via the
# admin Config page. Empty → the TMDB enrichment job no-ops. Feeds keywords / crew / budget /
# revenue / TMDb score / recommendations / backdrops onto the mirrored Plex rows, matched by the
@@ -359,7 +366,7 @@ class Settings(BaseSettings):
# Metadata language for TMDB fetches (keyword/overview locale). Stored EN by default.
tmdb_language: str = "en-US"
# --- Bug reports (E16): where a debug session's issue goes. All three empty = no tracker; the
# --- Bug reports: where a debug session's issue goes. All three empty = no tracker; the
# composer still exports a zip. The repository must be PRIVATE — an issue carries the reporter's
# log — and app/issues/gitea.py refuses a public one. Runtime knobs (sysconfig, admin UI). ---
issues_gitea_url: str = ""
@@ -369,7 +376,7 @@ class Settings(BaseSettings):
tmdb_enrich_interval_min: int = 120
# Titles fetched per TMDB enrichment run (TMDB is fast + generously rate-limited).
tmdb_enrich_batch: int = 100
# How often the people job (E30 S6) fetches a batch of person bios from TMDB and portraits through Plex,
# How often the people job fetches a batch of person bios from TMDB and portraits through Plex,
# in minutes, and how many of each per run: ~18k linked people at 300 per 10 min is about ten hours
# once, then only the newcomers.
people_enrich_interval_min: int = 10
+2 -2
View File
@@ -1,4 +1,4 @@
"""Debug sessions (E16): Start → reproduce → Stop, and the evidence a bug report is built from.
"""Debug sessions: Start → reproduce → Stop, and the evidence a bug report is built from.
While a session is open its user's browser names it in `X-Debug-Session` on every request. The auth
dependency accepts the header only for the caller's OWN open session (anything else is ignored, never
@@ -269,7 +269,7 @@ def set_delivery(db: Session, session: DebugSession, state: dict, token: str) ->
every step. It is stamped from `token` here instead, and only after the ROW confirms the claim
is still ours. Without that check the write is a full-column overwrite with no WHERE: a send
that lost its lease came back and stomped the take-over send's committed links, then put its
own dead token back on an idle session (deep-review round 4). `release_send` has always made
own dead token back on an idle session. `release_send` has always made
the same check; this is the other half of it."""
if claim_owner(db, session) != token:
raise SendClaimLost(session.id)
-82
View File
@@ -1,82 +0,0 @@
"""Demo-account service: reset the shared demo sandbox to a clean, seeded baseline.
Shared by the admin's manual reset endpoint (routes/admin.py) and the scheduled demo-reset job
(scheduler.py). Lives in the service layer, not routes, so the scheduler can call it without an
`app.routes -> app.scheduler` import cycle (the reason its import used to be lazy).
"""
from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from app.models import Playlist, PlaylistItem, Subscription, User, Video, VideoState
def _seed_demo_playlists(db: Session, demo: User) -> int:
"""Seed a few sample playlists from the shared catalog so a freshly-reset demo isn't
empty — gives visitors something to play with. Best-effort: skips silently if the
catalog is too small."""
recent = (
db.execute(
select(Video.id)
.where(Video.is_short.is_(False), Video.live_status == "none")
.order_by(Video.published_at.desc().nullslast())
.limit(60)
)
.scalars()
.all()
)
if not recent:
return 0
samples = [
("Fresh picks", recent[0:10]),
("Quick watch", recent[10:18]),
("Saved for later", recent[18:26]),
]
created = 0
for pos, (name, vids) in enumerate(samples):
if not vids:
continue
pl = Playlist(user_id=demo.id, name=name, position=pos)
db.add(pl)
db.flush()
for ipos, vid in enumerate(vids):
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=ipos))
created += 1
return created
def _seed_demo_subscriptions(db: Session, demo: User, n: int = 14) -> int:
"""Seed the demo with a handful of catalog channels (the ones with the most videos) so its
Channel manager isn't empty — gives visitors something to explore and matches the landing-page
screenshot. Idempotent: clears the demo's subscriptions first, then re-adds the top N."""
db.execute(delete(Subscription).where(Subscription.user_id == demo.id))
top = (
db.execute(
select(Video.channel_id)
.where(Video.channel_id.is_not(None))
.group_by(Video.channel_id)
.order_by(func.count().desc())
.limit(n)
)
.scalars()
.all()
)
for channel_id in top:
db.add(Subscription(user_id=demo.id, channel_id=channel_id))
return len(top)
def reset_demo_state(db: Session, demo: User) -> int:
"""Wipe the demo account's per-user state (watch/save/hide states, playlists, preferences)
back to a clean baseline and re-seed a few sample playlists + channel subscriptions. Returns
the seeded playlist count. Shared by the admin's manual reset button and the scheduled demo
reset (see scheduler)."""
db.execute(delete(VideoState).where(VideoState.user_id == demo.id))
# Playlist items cascade on playlist delete (ondelete="CASCADE").
db.execute(delete(Playlist).where(Playlist.user_id == demo.id))
demo.preferences = {"language": "en"}
db.commit()
seeded = _seed_demo_playlists(db, demo)
_seed_demo_subscriptions(db, demo)
db.commit()
return seeded
+2 -3
View File
@@ -1,4 +1,4 @@
"""A user's diagnostics level (E22 S5): what they consented to, and for how long.
"""A user's diagnostics level: what they consented to, and for how long.
Three levels, named in the UI Alap / Részletes / Teljes:
@@ -10,7 +10,6 @@ Three levels, named in the UI Alap / Részletes / Teljes:
A raised level always lapses after `ELEVATION_HOURS`: the expiry is computed HERE at consent time,
never taken from the client, and an expired level reads as INFO without anything having to clear it.
The shared demo account cannot raise its level — its log would mix every visitor's session.
Core layer: imports `models` and `sysconfig`.
"""
@@ -27,7 +26,7 @@ DiagnosticsLevel = Literal["INFO", "DEBUG", "TRACE"]
LEVELS: tuple[str, ...] = get_args(DiagnosticsLevel)
BASE: DiagnosticsLevel = "INFO"
ELEVATION_HOURS = 24
# How long a debug session records at most (E16, app/debugsession.py). Here, beside the other
# How long a debug session records at most (app/debugsession.py). Here, beside the other
# consent window, because the start dialog's consent text states it and reads it from `state`.
SESSION_MINUTES = 60
+1 -1
View File
@@ -1,4 +1,4 @@
"""E24 — the read-model behind the Home surface ("Mit nézzek most?").
"""The read-model behind the Home surface ("Mit nézzek most?").
The anti-paralysis answer layer: one hero suggestion, a cross-source resume rail, and the user's own
comfort shelf, all served from our own Postgres with no upstream call at serve time (the same rule
+1 -1
View File
@@ -1,4 +1,4 @@
"""The time budget — E24's replacement for "what mood are you in?".
"""The time budget — the replacement for "what mood are you in?".
`random-gondolatok` 15/D asked for a mood questionnaire before the recommendations. The same file's
later BA-interview note (#23) says the mood half is not what makes this work: the friction is
+5 -5
View File
@@ -24,7 +24,7 @@ from sqlalchemy.orm import Session
from app.discover import cards, queries
from app.models import Channel, ComfortPin, PlexItem, PlexShow, Playlist, User, Video
from app.plex import rollup
from app.plex.cards import leaf_card, show_card
from app.plex.cards import SHOW_NEW, leaf_card, show_card
from app.plex.lookups import by_rating_key
# kind -> the ComfortPin column that holds it. The client's vocabulary and the schema's, in one place.
@@ -46,8 +46,8 @@ def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, obje
translation belongs here rather than in five call sites.
The COLUMN half of every answer comes from `PIN_COLUMNS`; only the LOOKUP differs per kind
(a direct `db.get`, a rating-key translation, an int parse plus an ownership check). Before the
E24 review the branches spelled the column names out again and `PIN_COLUMNS` had no readers at
(a direct `db.get`, a rating-key translation, an int parse plus an ownership check). Before a
review the branches spelled the column names out again and `PIN_COLUMNS` had no readers at
all — a constant that called itself the one vocabulary while nothing spoke it."""
col = PIN_COLUMNS.get(kind)
if col is None:
@@ -203,8 +203,8 @@ def _shelf_card(db: Session, user: User, pin: ComfortPin, batch: _Batch, lang: s
sh = db.get(PlexShow, pin.plex_show_id)
if sh is None:
return None
status = rollup.state_map(db, user.id, [sh.id]).get(sh.id, "new")
card = cards.from_plex(show_card(sh, status, lang=lang))
state = rollup.state_map(db, user.id, [sh.id]).get(sh.id, SHOW_NEW)
card = cards.from_plex(show_card(sh, state, lang=lang))
ep = queries.next_episode(db, user.id, sh.id, lang=lang)
nxt = cards.from_plex(ep) if ep else None
+7 -3
View File
@@ -256,16 +256,20 @@ def plex_in_progress(
if show_libs:
prog = rollup.agg_subquery(db, uid, include_resumed=True).subquery()
sq = apply_show_state_filter(
select(PlexShow, prog.c.resumed_at).join(prog, prog.c.show_id == PlexShow.id),
select(PlexShow, prog.c.resumed_at, prog.c.total, prog.c.watched, prog.c.inprog).join(
prog, prog.c.show_id == PlexShow.id
),
prog.c.total,
prog.c.watched,
prog.c.inprog,
"in_progress",
).where(PlexShow.library_id.in_(show_libs))
for sh, resumed_at in db.execute(
for sh, resumed_at, total, watched, inprog in db.execute(
sq.order_by(prog.c.resumed_at.desc().nullslast()).limit(limit)
).all():
out.append((resumed_at, show_card(sh, "in_progress", lang=lang)))
out.append(
(resumed_at, show_card(sh, rollup.show_state(total, watched, inprog), lang=lang))
)
return out
+5 -4
View File
@@ -106,7 +106,8 @@ class YouTubeAdapter(SourceAdapter):
# `player_client=android_vr` yt-dlp logs `Skipping client "android_vr" since it does not support
# cookies`, drops it, and the request that is left with no client fails with "Requested format is
# not available" — which is already a retriable marker above, so the cost is a wasted attempt
# rather than a hard failure. Our primary set is android_vr ALONE, so without this filter every
# rather than a hard failure. Our primary set is tv_simply ALONE, which cannot carry cookies either
# (measured on yt-dlp 2026.08.19: `SUPPORTS_COOKIES` False), so without this filter every
# signed-in download would burn its first attempt that way before reaching the POT-backed web
# fallback the cookie exists for. The list is yt-dlp's own INNERTUBE_CLIENTS[...]
# ["SUPPORTS_COOKIES"], pinned by test_adapters against the installed yt-dlp, so a version bump
@@ -156,7 +157,7 @@ class YouTubeAdapter(SourceAdapter):
class FacebookAdapter(SourceAdapter):
key = "facebook"
# Measured (E11.5, yt-dlp 2026.07.04): with the user's cookie, some reels answer a plain request
# Measured (yt-dlp 2026.07.04): with the user's cookie, some reels answer a plain request
# with "Cannot parse data" and parse only when the request looks like Chrome. Impersonation is a
# SECOND attempt behind that one error, not the default: the links that already download with the
# cookie keep their proven path, and only the ones stuck at the wall go through curl_cffi. Without
@@ -166,7 +167,7 @@ class FacebookAdapter(SourceAdapter):
_POST_PAGE_MAX_BYTES = 16 * 1024 * 1024
def resolve_url(self, url: str, ydl_opts: dict) -> str:
# Measured (E11.5 S4): given a group post's link, yt-dlp either finds no formats or — worse —
# Measured: given a group post's link, yt-dlp either finds no formats or — worse —
# downloads ANOTHER video from the page's feed data. The post's own video is the one Video node
# in the page scripts that carry the post id; its watch link downloads (plain first, then the
# Chrome retry above). Needs the cookie: signed out, the page is the login wall.
@@ -176,7 +177,7 @@ class FacebookAdapter(SourceAdapter):
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
page = raw.decode("utf-8", "replace")
# Measured (E11.5 review): a refused or expired cookie gets the login page (~450 KB, a login
# Measured: a refused or expired cookie gets the login page (~450 KB, a login
# form, no post id at all), which is not a post without a video — the user's cookie is the cause.
if _quoted(post_id) not in page:
raise StatedFailure(POST_NOT_SHOWN_MESSAGE)
+1 -1
View File
@@ -89,7 +89,7 @@ def run() -> dict:
if a.poster_path is None:
# Record a self-hosted poster for every download (the reliable og:image source):
# ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
rel = editmod.ensure_poster(a, settings.download_root)
rel = editmod.ensure_poster(a, settings.files_root)
if rel:
a.poster_path = rel
filled_poster += 1
+289
View File
@@ -0,0 +1,289 @@
"""Ad-hoc clips: a marked range of a YouTube video or a Plex item, cut into a new per-user asset.
A clip never needs the whole source first. A YouTube clip fetches only its range (yt-dlp's
`download_ranges`; measured: 3-7 s whether the video runs four minutes or an hour, and the fetched
range starts on the marked frame), and a Plex clip reads the library file in place. Either way the
range then goes through ONE re-encode built here, so every clip has the same shape whatever its
source: frame-exact at both ends, H.264 at most 1080p (an HDR source tonemapped to SDR — kept as
10-bit PQ it played washed out), stereo AAC, in an .mp4 that plays on any phone.
A clip may also crop the picture to a frame the user drew over the playing video. The frame is
given in fractions of the box it was drawn in, with that box's aspect: the YouTube iframe never
tells its picture's size, so the frame there is drawn over the player's 16:9 box (a short's 9:16),
and the cut maps it onto the real picture (`crop_filter`) — what lies over a letterbox bar is dropped.
The spec is stored on the job (`DownloadJob.edit_spec`) and is the clip's whole identity: its
signature is the asset's cache key, so marking the same range again is an instant cache hit, and a
different range or audio track is a different clip.
"""
import hashlib
import json
import math
import re
from pathlib import Path
from app.downloads.edit import AAC_ARGS, EVEN_FRAME, INPUT_GUARD, X264_ARGS
from app.plex.sync import HDR_TRANSFERS
SOURCES = ("youtube", "plex")
# Owner's cap: a clip is a moment, not a way to export a whole film to a public link.
MAX_CLIP_S = 600
MIN_CLIP_S = 1
MAX_HEIGHT = 1080
# What yt-dlp fetches the range in (formats.normalize fills the rest): the clip's own ceiling, so the
# re-encode never upscales and never throws away a better source than it keeps.
YOUTUBE_FETCH_SPEC = {"mode": "av", "max_height": MAX_HEIGHT, "container": "mp4"}
# Bump when the spec → output-bytes mapping changes, so a cached clip re-derives.
_SIG_VERSION = 1
_YOUTUBE_ID = re.compile(r"[A-Za-z0-9_-]{11}")
_RATING_KEY = re.compile(r"[0-9]{1,32}")
_MAX_AUDIO_TRACK = 63
# HDR → SDR as measured on the owner's 4K HDR files (2026-10-10): linearise, tonemap (hable),
# convert the primaries and transfer to BT.709 and back to 8-bit 4:2:0.
_TONEMAP = (
"zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable:desat=0,"
"zscale=t=bt709:m=bt709:r=tv,format=yuv420p"
)
_SCALE = f"scale=-2:'min({MAX_HEIGHT},ih)'"
# A crop frame is at least this share of its box on each side (the player's frame holds the same
# floor), and what is left of it on the real picture at least _MIN_VISIBLE of it.
MIN_CROP = 0.1
_MIN_VISIBLE = 0.05
# The box aspects a frame may be drawn in: a 9:16 short to a 21:9 film, with room either side.
_CROP_ASPECTS = (0.25, 4.0)
# Float slack on the frame's far edges (x + w, y + h), which the client rounds to 4 decimals.
_EDGE_SLACK = 1e-3
class ClipError(Exception):
"""A clip request that cannot be cut; `reason` is an i18n-mappable key, `params` the numbers its
message names (the cap itself, so the client never says a stale one)."""
def __init__(self, reason: str, **params: float):
self.reason = reason
self.params = params
super().__init__(reason)
def normalize(
source: str,
ref: str,
start_s: float,
end_s: float,
audio_track: int | None = None,
crop: dict | None = None,
) -> dict:
"""The validated, canonical clip spec. Times are rounded to the millisecond and a crop to four
decimals, so the same marks always give the same cache key. Raises ClipError."""
if source not in SOURCES:
raise ClipError("bad_source")
pattern = _YOUTUBE_ID if source == "youtube" else _RATING_KEY
if not isinstance(ref, str) or not pattern.fullmatch(ref):
raise ClipError("bad_source")
if not (math.isfinite(start_s) and math.isfinite(end_s)) or start_s < 0:
raise ClipError("bad_range")
start, end = round(start_s, 3), round(end_s, 3)
if end - start < MIN_CLIP_S:
raise ClipError("too_short")
if end - start > MAX_CLIP_S:
raise ClipError("too_long", limit_s=MAX_CLIP_S)
spec: dict = {"source": source, "ref": ref, "start_s": start, "end_s": end}
if source == "plex" and audio_track is not None:
if not 0 <= audio_track <= _MAX_AUDIO_TRACK:
raise ClipError("bad_audio")
spec["audio_track"] = audio_track
frame = _crop(crop)
if frame is not None:
spec["crop"] = frame
return spec
def _crop(raw: dict | None) -> dict | None:
"""A crop frame, canonical — or None for no crop, which a frame over the whole box also is (the
same clip as one cut without a frame, so the same cache key)."""
if raw is None:
return None
try:
x, y, w, h, aspect = (float(raw[k]) for k in ("x", "y", "w", "h", "aspect"))
except (KeyError, TypeError, ValueError) as e:
raise ClipError("bad_crop") from e
if not all(math.isfinite(v) for v in (x, y, w, h, aspect)):
raise ClipError("bad_crop")
lo, hi = _CROP_ASPECTS
if not lo <= aspect <= hi or x < 0 or y < 0:
raise ClipError("bad_crop")
if w < MIN_CROP - _EDGE_SLACK or h < MIN_CROP - _EDGE_SLACK:
raise ClipError("bad_crop")
if x + w > 1 + _EDGE_SLACK or y + h > 1 + _EDGE_SLACK:
raise ClipError("bad_crop")
frame = {"x": x, "y": y, "w": min(w, 1 - x), "h": min(h, 1 - y), "aspect": aspect}
frame = {k: round(v, 4) for k, v in frame.items()}
if (frame["x"], frame["y"], frame["w"], frame["h"]) == (0, 0, 1, 1):
return None
return frame
def duration(spec: dict) -> float:
return spec["end_s"] - spec["start_s"]
def sig(spec: dict) -> str:
"""The clip's cache signature: the spec itself, versioned."""
body = json.dumps({**spec, "_v": _SIG_VERSION}, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(body.encode()).hexdigest()[:24]
def asset_ref(user_id: int, spec: dict) -> str:
"""The asset's `source_ref`. The user id makes a clip private to whoever cut it, like an edit:
never dedup-shared, never cache-served to someone else."""
return f"{user_id}:{spec['source']}:{spec['ref']}"
def reference_url(spec: dict) -> str | None:
"""Where a YouTube clip came from: its video at the clip's start. A Plex clip has no public page."""
if spec.get("source") != "youtube" or not spec.get("ref"):
return None
return f"https://www.youtube.com/watch?v={spec['ref']}&t={int(spec.get('start_s') or 0)}s"
def is_hdr(probe: dict) -> bool:
"""Whether the first video stream of an ffprobe `-show_streams` JSON carries an HDR transfer."""
for s in probe.get("streams") or []:
if s.get("codec_type") == "video":
# PQ / HLG: an 8-bit SDR encode must tonemap them (the one set the Plex stream uses).
return s.get("color_transfer") in HDR_TRANSFERS
return False
def display_aspect(probe: dict) -> float | None:
"""The picture's shape as a player shows it, from an ffprobe `-show_streams` JSON: the first video
stream's width × its sample aspect ratio over its height, turned a quarter when the stream says
it is rotated (ffmpeg turns such a stream upright before any filter of ours, so a crop is cut
from the upright picture). None when the stream has no size."""
for s in probe.get("streams") or []:
if s.get("codec_type") != "video":
continue
w, h = s.get("width"), s.get("height")
if not isinstance(w, int) or not isinstance(h, int) or w <= 0 or h <= 0:
return None
dar = w * (_ratio(s.get("sample_aspect_ratio")) or 1.0) / h
return 1 / dar if _quarter_turned(s) else dar
return None
def _ratio(v: object) -> float | None:
"""'32:27' → 1.185…; '0:1', 'N/A', a missing or malformed value → None."""
if not isinstance(v, str) or ":" not in v:
return None
num, _, den = v.partition(":")
try:
n, d = float(num), float(den)
except ValueError: # silent-ok: a malformed ratio reads as none given; the caller takes 1:1
return None
return n / d if n > 0 and d > 0 else None
def _quarter_turned(stream: dict) -> bool:
rotation: object = (stream.get("tags") or {}).get("rotate")
for sd in stream.get("side_data_list") or []:
if "rotation" in sd:
rotation = sd["rotation"]
try:
return round(float(str(rotation))) % 180 == 90
except ValueError: # silent-ok: no (None) or a malformed rotation: the stream is upright
return False
def crop_filter(crop: dict, dar: float) -> str | None:
"""The ffmpeg `crop` for a frame drawn in a box of `crop["aspect"]`, over a picture of display
aspect `dar` fitted into that box as a player fits it (whole, centred, bars on the short sides).
The frame's part over a bar is dropped: "" when what is left is the whole picture, None when too
little of the frame lies on the picture at all.
In fractions of the input's own width and height, so a sample aspect ratio other than 1:1 (an
anamorphic DVD) needs nothing of its own, and floored to even pixels: an even 4:2:0 frame, and a
window that never runs past the picture's far edge."""
box = crop["aspect"]
pic_w, pic_h = (1.0, box / dar) if dar > box else (dar / box, 1.0)
def span(start: float, size: float, extent: float) -> tuple[float, float]:
origin = (1 - extent) / 2
a = min(1.0, max(0.0, (start - origin) / extent))
b = min(1.0, max(0.0, (start + size - origin) / extent))
return a, b - a
x, w = span(crop["x"], crop["w"], pic_w)
y, h = span(crop["y"], crop["h"], pic_h)
if w < _MIN_VISIBLE or h < _MIN_VISIBLE:
return None
if w > 1 - 1e-4 and h > 1 - 1e-4:
return ""
def px(axis: str, f: float) -> str:
return f"trunc({axis}*{f:.6f}/2)*2"
return f"crop={px('iw', w)}:{px('ih', h)}:{px('iw', x)}:{px('ih', y)}"
def audio_track_count(probe: dict) -> int:
return sum(1 for s in probe.get("streams") or [] if s.get("codec_type") == "audio")
def build_cmd(
src: Path,
dest: Path,
start_s: float,
dur_s: float,
*,
audio_track: int | None,
hdr: bool,
threads: int,
crop: str = "",
) -> list[str]:
"""ffmpeg for one clip: an input seek (decodes from the keyframe before `start_s` and drops the
frames up to it, so the clip starts on the marked frame), the first video stream, the chosen audio
track (the first when none was chosen; none at all is fine), no subtitles, data or chapters.
A `crop` (crop_filter) comes first: the tonemap and the scale then work on the frame it keeps."""
vf = ",".join([*([crop] if crop else []), *([_TONEMAP] if hdr else []), _SCALE, EVEN_FRAME])
audio = f"0:a:{audio_track}" if audio_track is not None else "0:a:0?"
return [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
"-ss",
f"{start_s:.3f}",
*INPUT_GUARD,
"-i",
str(src),
"-t",
f"{dur_s:.3f}",
"-map",
"0:v:0",
"-map",
audio,
"-sn",
"-dn",
# A source's chapters would ride into the .mp4 as a data track of their own (an mkv's
# chapter list came out as `bin_data`); a clip of a moment has no chapters to keep.
"-map_chapters",
"-1",
"-vf",
vf,
*X264_ARGS,
"-threads",
str(threads),
*AAC_ARGS,
"-ac",
"2",
"-movflags",
"+faststart",
str(dest),
]
+33 -2
View File
@@ -30,6 +30,7 @@ from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs
from app.downloads import clip as clipmod
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential, MediaAsset
@@ -249,8 +250,8 @@ def usable_cookie(
If (2) fails the asset is the SHARED (unsalted) one — attaching the cookie would poison the
global cache with a private-group video, so we attach nothing. An unreadable blob (rotated key →
InvalidToken) also yields None; the download then fails honestly on the login wall (S3's
`login_required`)."""
InvalidToken) also yields None; the download then fails honestly on the login wall (the
`login_required` category)."""
host_key = host_key_for_url(url)
if host_key is None:
return None
@@ -259,6 +260,28 @@ def usable_cookie(
return None
if not is_authed_asset(asset, spec, user_id, url):
return None # shared asset — never attach a cookie to a non-private identity
return _readable(cred, user_id, host_key)
def usable_clip_cookie(
db: Session, user_id: int, url: str | None, asset: MediaAsset, clip_spec: dict
) -> str | None:
"""`usable_cookie` for an ad-hoc clip's range fetch. The second gate is the clip's own identity:
a clip asset is private by construction (`clip.asset_ref` names its user, see downloads/clip),
so a cookie goes with it exactly when that asset is THIS user's clip of `clip_spec`. Without it a
video YouTube gates signed out could never be clipped, while its download goes through."""
host_key = host_key_for_url(url)
if host_key is None:
return None
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset.source_kind != "clip" or asset.source_ref != clipmod.asset_ref(user_id, clip_spec):
return None
return _readable(cred, user_id, host_key)
def _readable(cred: DownloadCredential, user_id: int, host_key: str) -> str | None:
raw = decrypt_optional(cred.cookies_enc)
if not raw:
log.warning(
@@ -284,6 +307,14 @@ def resolve_cookiefile(
return _write_cookiefile(raw, staging) if raw else None
def resolve_clip_cookiefile(
db: Session, user_id: int, url: str | None, asset: MediaAsset, clip_spec: dict, staging: Path
) -> Path | None:
"""`resolve_cookiefile` for an ad-hoc clip (`usable_clip_cookie`)."""
raw = usable_clip_cookie(db, user_id, url, asset, clip_spec)
return _write_cookiefile(raw, staging) if raw else None
def _write_cookiefile(raw: str, staging: Path) -> Path:
"""Write the cookie blob to a private 0600 file under `staging/.auth/` and return its path.
+75 -117
View File
@@ -188,6 +188,51 @@ def clip_duration(spec: dict, source_duration: int | None) -> int | None:
# --- ffmpeg: trim / crop --------------------------------------------------------------------
# Every libx264 / aac re-encode in this package uses these, so a re-encoded file is always the
# phone-safe shape (`playable`): `-pix_fmt yuv420p` pins 8-bit 4:2:0 — without it libx264 keeps a
# 10-bit or 4:4:4 source's pixel format and writes H.264 High 10 / High 4:4:4, which browsers decode
# only patchily (the one safe H.264 is 8-bit 4:2:0).
X264_ARGS = ["-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p"]
AAC_ARGS = ["-c:a", "aac", "-b:a", "192k"]
# The last step of every video filter chain a libx264 re-encode runs: 4:2:0 halves the chroma, so
# libx264 refuses an odd width or height ("width not divisible by 2") — measured on an odd source
# that is not 4:2:0 already (a 4:4:4 or RGB screen capture, 1365x767). A crop of a 4:2:0 source is
# rounded to its chroma grid by ffmpeg itself; this holds every other path to the encoder to the same
# even frame. Rounding down to even costs at most one pixel row/column.
EVEN_FRAME = "scale=trunc(iw/2)*2:trunc(ih/2)*2"
# The demuxers ffmpeg/ffprobe may use to open a library file — single-file containers only. Without
# it ffmpeg picks the demuxer from the file's CONTENT and NAME, and an upload's name is the user's:
# a playlist-type demuxer (HLS, DASH, concat) opens whatever local paths its entries name, so a
# crafted upload could copy another user's clip or a Plex file into its own converted asset. A file
# no demuxer here can open fails the probe, and an unprobeable upload is a download-only 'file'.
MEDIA_DEMUXERS = (
"mov", # mp4, m4a, mov, 3gp
"matroska", # mkv, webm
"mpegts",
"mpeg",
"avi",
"asf",
"flv",
"ogg",
"mp3",
"aac",
"ac3",
"eac3",
"flac",
"wav",
"w64",
"aiff",
"caf",
"amr",
)
INPUT_GUARD = ["-format_whitelist", ",".join(MEDIA_DEMUXERS)]
# A downloaded thumbnail is the remote site's bytes under the suffix yt-dlp gave it (`.png`, `.webp`,
# `.jpg` — `pipeline._THUMB_SUFFIXES`): only the single-image demuxers those open, so a "thumbnail"
# that is really a playlist cannot pull a local file into the poster. Anything else fails, and the
# poster falls back to a cut frame (`ensure_poster`).
THUMB_GUARD = ["-format_whitelist", "png_pipe,jpeg_pipe,webp_pipe"]
def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]:
"""ffmpeg command for a trim/crop edit. `-progress pipe:1` streams machine-readable progress.
@@ -214,7 +259,7 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
]
if start:
cmd += ["-ss", f"{start:.3f}"]
cmd += ["-i", str(src)]
cmd += [*INPUT_GUARD, "-i", str(src)]
if end is not None:
dur = end - start
if dur > 0:
@@ -222,21 +267,9 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
if reencode:
# Input -ss + re-encode is frame-accurate (decodes from the prior keyframe, discards up to
# the exact cut). A crop adds the filter; an accurate trim re-encodes with no filter.
if crop:
cmd += ["-vf", f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}"]
cmd += [
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"20",
"-c:a",
"aac",
"-b:a",
"192k",
]
# the exact cut). A crop adds the filter; an accurate trim only evens the frame.
cropf = f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}," if crop else ""
cmd += ["-vf", cropf + EVEN_FRAME, *X264_ARGS, *AAC_ARGS]
else:
cmd += ["-c", "copy", "-avoid_negative_ts", "make_zero"]
@@ -284,11 +317,14 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
parts, labels = [], []
for i, s in enumerate(segs):
st, en = s["start_s"], s["end_s"]
parts.append(f"[0:v]trim=start={st}:end={en},setpts=PTS-STARTPTS{cropf}[v{i}]")
parts.append(
f"[0:v]trim=start={st}:end={en},setpts=PTS-STARTPTS{cropf},{EVEN_FRAME}[v{i}]"
)
parts.append(f"[0:a]atrim=start={st}:end={en},asetpts=PTS-STARTPTS[a{i}]")
labels.append(f"[v{i}][a{i}]")
parts.append(f"{''.join(labels)}concat=n={len(segs)}:v=1:a=1[v][a]")
cmd = head + [
*INPUT_GUARD,
"-i",
str(src),
"-filter_complex",
@@ -297,26 +333,23 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
"[v]",
"-map",
"[a]",
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"20",
"-c:a",
"aac",
"-b:a",
"192k",
*X264_ARGS,
*AAC_ARGS,
]
else:
listp = staging / "concat.txt"
lines = []
lines = ["ffconcat version 1.0"]
for s in segs:
lines.append(f"file '{_concat_quote(src)}'")
# ffmpeg hands the list's whitelist (which must name `concat` to read the list) down to
# the file it names, so a library file whose bytes are a concat list would be read as one
# and open the paths inside it. Each entry's own whitelist replaces the inherited one.
lines.append(f"option format_whitelist {INPUT_GUARD[1]}")
lines.append(f"inpoint {s['start_s']}")
lines.append(f"outpoint {s['end_s']}")
prep[listp] = "\n".join(lines) + "\n"
cmd = head + ["-f", "concat", "-safe", "0", "-i", str(listp), "-c", "copy"]
guard = ["-format_whitelist", "concat," + INPUT_GUARD[1]]
cmd = head + ["-f", "concat", "-safe", "0", *guard, "-i", str(listp), "-c", "copy"]
if out_ext in ("mp4", "m4a", "mov"):
cmd += ["-movflags", "+faststart"]
@@ -474,14 +507,7 @@ def run_ffmpeg(
raise StatedFailure(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
# --- ffmpeg: universal browser playability --------------------------------------------------
# A downloaded file must play in EVERY browser, whatever the source shipped. Universal = H.264
# video + AAC/MP3 audio in an mp4 container: H.264 is the only video codec all browsers decode
# (VP9/AV1 fail on Safari/iOS; HEVC fails on Chrome/Firefox), and mp4/AAC is the safe pairing.
# Anything else is fixed post-download: re-encode the offending stream(s), or just remux when the
# codecs are already fine but the container isn't.
_BROWSER_SAFE_VCODEC = ("h264",)
_BROWSER_SAFE_ACODEC = ("aac", "mp3")
# --- ffprobe ---------------------------------------------------------------------------------
def run_ffprobe_json(
@@ -493,10 +519,10 @@ def run_ffprobe_json(
) -> dict | None:
"""One ffprobe call → its parsed JSON dict, or None on any failure (timeout, non-zero exit,
missing binary, unparseable output). The SINGLE ffprobe-JSON-invocation point in the downloads
package: the browser-playability codec probe here and the shared media probe (`probe.probe_media`,
used by both the download worker's finalize and the upload ingest) both go through it, so the
package: the shared media probe (`probe.probe_json`: the stored facts and the phone-safe rule) goes
through it, so the
subprocess flags/timeout/error handling live in one place instead of drifting copies."""
args = ["ffprobe", "-v", "error"]
args = ["ffprobe", "-v", "error", *INPUT_GUARD]
if show_streams:
args.append("-show_streams")
if show_format:
@@ -517,35 +543,6 @@ def run_ffprobe_json(
return None
def _codecs_from_streams(streams: list) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) lowercased from an ffprobe JSON stream list — the first of each kind.
Parsed from the JSON `streams` list, NOT ffprobe's flat `default=nw=1` output: there the fields
arrive in ffprobe's own struct order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line
"remember the last codec_type" parser attributes every name to the PREVIOUS stream. Measured on real
downloads: an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download. Reading each
stream dict by its own `codec_type` is immune to that ordering."""
vcodec = acodec = None
for s in streams:
name = (s.get("codec_name") or "").lower() or None
ctype = (s.get("codec_type") or "").lower()
if ctype == "video" and vcodec is None:
vcodec = name
elif ctype == "audio" and acodec is None:
acodec = name
return vcodec, acodec
def probe_codecs(path: Path) -> tuple[str | None, str | None]:
"""Return (video_codec, audio_codec) lowercased via one ffprobe call; None for a missing
stream or if ffprobe fails (a failed probe is treated as 'leave it alone' upstream)."""
data = run_ffprobe_json(path, entries="stream=codec_type,codec_name")
if data is None:
return None, None
return _codecs_from_streams(data.get("streams") or [])
def probe_duration(path: Path) -> int:
"""Media duration in whole seconds via ffprobe, or 0 if unknown — a progress denominator for a
re-encode when the yt-dlp info dict carried no `duration` (some URL/live extractions)."""
@@ -555,6 +552,7 @@ def probe_duration(path: Path) -> int:
"ffprobe",
"-v",
"error",
*INPUT_GUARD,
"-show_entries",
"format=duration",
"-of",
@@ -577,49 +575,6 @@ def probe_duration(path: Path) -> int:
return 0
def browser_transcode_flags(
vcodec: str | None, acodec: str | None, is_mp4: bool
) -> tuple[bool, bool, bool]:
"""Decide what (if anything) makes a file universally browser-playable. Returns
(needs_change, reencode_video, reencode_audio). needs_change=False => already H.264 + AAC/MP3
in mp4. A None codec (no such stream, or a failed probe) is never re-encoded."""
rv = bool(vcodec) and vcodec not in _BROWSER_SAFE_VCODEC
ra = bool(acodec) and acodec not in _BROWSER_SAFE_ACODEC
return (rv or ra or not is_mp4), rv, ra
def build_browser_safe_cmd(
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool, threads: int | None = None
) -> list[str]:
"""ffmpeg command that lands `src` as a browser-safe mp4 (faststart): re-encode only the
stream(s) that need it (libx264 / aac), stream-copy the rest, always into mp4. `-progress
pipe:1` streams machine-readable progress for run_ffmpeg. `threads` caps the encoder so a
re-encode can't claim every core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
"-i",
str(src),
]
cmd += (
["-c:v", "libx264", "-preset", "veryfast", "-crf", "20"]
if reencode_video
else ["-c:v", "copy"]
)
cmd += ["-c:a", "aac", "-b:a", "192k"] if reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
cmd += ["-movflags", "+faststart", str(dest)]
return cmd
# --- filmstrip (editor scrub track) ---------------------------------------------------------
_SB_COLS = 12
@@ -655,6 +610,7 @@ def build_storyboard_cmd(src: Path, dest: Path, duration_s: int | None, geom: di
"-loglevel",
"error",
"-nostdin",
*INPUT_GUARD,
"-i",
str(src),
"-frames:v",
@@ -678,6 +634,7 @@ def build_poster_cmd(src: Path, dest: Path, at_s: float) -> list[str]:
"-nostdin",
"-ss",
f"{max(0.0, at_s):.3f}",
*INPUT_GUARD,
"-i",
str(src),
"-frames:v",
@@ -699,6 +656,7 @@ def build_thumbnail_jpg_cmd(src: Path, dest: Path) -> list[str]:
"-loglevel",
"error",
"-nostdin",
*THUMB_GUARD,
"-i",
str(src),
"-frames:v",
@@ -734,7 +692,7 @@ def _step_failure(err: BaseException) -> str:
def _is_jpeg(path: Path) -> bool:
"""True if the file starts with the JPEG SOI marker (FF D8 FF)."""
"""True if the file starts with the JPEG SOI marker (0xFF 0xD8 0xFF)."""
try:
with path.open("rb") as f:
return f.read(3) == b"\xff\xd8\xff"
@@ -774,7 +732,7 @@ def thumbnail_to_jpg(src: Path | None, dest: Path, timeout_s: int = 30) -> Path
return dest if dest.exists() else None
def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
def ensure_poster(asset, files_root: str, timeout_s: int = 30) -> str | None:
"""Generate a poster frame for a ready asset that has no thumbnail (e.g. a direct media URL
like a reddit HLS manifest, which yt-dlp can't get a thumbnail for). Written as the media's
`<base>.jpg` sidecar — so Plex uses it too and `delete_asset_files` cleans it up — and returned
@@ -782,7 +740,7 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
short clip) dodges black intro frames."""
if not asset.rel_path:
return None
root = Path(download_root)
root = Path(files_root)
src = root / asset.rel_path
if not src.exists():
return None
@@ -798,7 +756,7 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
return rel if dest.exists() else None
def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict | None:
def ensure_storyboard(asset, files_root: str, timeout_s: int = 90) -> dict | None:
"""Return the filmstrip meta for a ready source asset, generating the sprite on first use.
Best-effort + time-bounded: for a very long/high-res source the one-pass decode can be slow,
@@ -810,7 +768,7 @@ def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict |
if not asset.rel_path:
return None
geom = storyboard_geometry(asset.duration_s)
root = Path(download_root)
root = Path(files_root)
rel = f".storyboards/asset-{asset.id}.jpg"
dest = root / rel
+39 -5
View File
@@ -8,7 +8,7 @@ explanation (and decides whether a retry makes sense). The raw text stays availa
"details" disclosure. This is the same string-matching approach the worker already trusts for
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
**The failure catalog** (`_CATALOG`, E11.5). A category is generic: "Sign-in required" cannot tell a
**The failure catalog** (`_CATALOG`). A category is generic: "Sign-in required" cannot tell a
Facebook user that the cookie they already gave was refused, and yt-dlp words several different
Facebook walls identically (`Cannot parse data`). A catalog entry is scoped to ONE host and may also
depend on whether a cookie was attached and on the kind of link, and names a `detail` code the UI has
@@ -37,9 +37,10 @@ from app.downloads.credentials import host_key_for_url
# Worker-authored strings that this module owns because each is coupled to `_RULES`.
#
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
# rule drifting apart is exactly how a category silently becomes unreachable.
# QUOTA_FULL_MESSAGE and the three *_TOO_BIG_MESSAGEs are raised by the worker and must classify
# back to `quota_exceeded`; their rules below match a fragment of each, and test_download_errors pins
# each one, because a message and its rule drifting apart is exactly how a
# category silently becomes unreachable.
#
# NETWORK_REDACTED_MESSAGE REPLACES a message after classification (`user_error_text`). It is written
# to the shared asset too, and a second job that inherits that failed asset re-diagnoses the stored
@@ -49,10 +50,24 @@ from app.downloads.credentials import host_key_for_url
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
# classifies as `unknown` — it is a refusal, not a diagnosis, and no category would say more.
#
# A source adapter's verdicts on a link (`SourceAdapter.resolve_url`, E11.5 S4) are adapters.py's
# A source adapter's verdicts on a link (`SourceAdapter.resolve_url`) are adapters.py's
# constants, imported above; each has a rule below that gives its category back, and a catalog entry
# that gives the site's sentence.
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
# A download, a cut clip, or an upload's converted file, held to the space left, which is not zero:
# "full" would be untrue. All three classify by the one phrase they share.
DOWNLOAD_TOO_BIG_MESSAGE = (
"The video is larger than the download storage you have left — free up space, then try again."
)
CLIP_TOO_BIG_MESSAGE = (
"The clip is larger than the download storage you have left — "
"free up space or cut a shorter clip, then try again."
)
CONVERTED_TOO_BIG_MESSAGE = (
"The converted file is larger than the download storage you have left — "
"free up space, then try again."
)
UPLOAD_MISSING_MESSAGE = "The uploaded file is missing."
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
@@ -61,6 +76,7 @@ _RULES: tuple[tuple[str, str], ...] = (
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
# first — it is our wording, so it can never be a false match on an upstream message.
("download storage is full", "quota_exceeded"),
("larger than the download storage you have left", "quota_exceeded"),
# Our own redaction of a transport failure (see NETWORK_REDACTED_MESSAGE) — our wording too.
(NETWORK_REDACTED_MESSAGE.lower(), "network"),
# Our own verdicts on a post link (adapters.py) — our wording too: its page holds no single video,
@@ -98,6 +114,8 @@ _RULES: tuple[tuple[str, str], ...] = (
# An edit's source download went away — our own wording, and MORE specific than the generic
# "no longer available" below, so it must be matched first.
("source download is no longer available", "source_missing"),
# An upload's original went away before its conversion ran — our wording (UPLOAD_MISSING_MESSAGE).
(UPLOAD_MISSING_MESSAGE.lower().rstrip("."), "upload_missing"),
("source file is missing", "source_missing"),
# Private / deleted / removed / terminated.
("private video", "unavailable"),
@@ -126,6 +144,13 @@ _RULES: tuple[tuple[str, str], ...] = (
("postprocessing", "postprocess"),
("produced no output file", "postprocess"),
("no output file", "postprocess"),
# yt-dlp's own verdict that its extractor did not understand the site's answer: it appends this
# request for a bug report to an UNEXPECTED extraction failure, never to an expected one (a private,
# removed or geo-blocked video says so in words matched above). Measured 2026-10-12: a TikTok link
# failed with it on every attempt under yt-dlp 2026.07.04 and downloads under 2026.08.19 — the
# site changed and a newer yt-dlp is the cure. After the media-format and post-processing rules,
# whose texts can carry the same request; before the network ones, which are about the transport.
("please report this issue", "site_changed"),
# Network / transport.
("unable to download", "network"),
("http error", "network"),
@@ -263,6 +288,15 @@ def classify(msg: str | None) -> str:
return "unknown"
def classify_conversion(msg: str | None) -> str:
"""The category of an upload's failed conversion. The generic ffmpeg categories speak of a
download ("the video downloaded but…"), which an upload never was: a failed encode or an
unrecognised failure of one is `convert_failed`; a category of its own (a full quota, a missing
original) stays as it is."""
code = classify(msg)
return "convert_failed" if code in ("postprocess", "unknown") else code
def diagnose(
msg: str | None, *, url: str | None = None, cookie_attached: bool = False
) -> Diagnosis:
+312
View File
@@ -0,0 +1,312 @@
"""The folders of the Files library — where each of a user's files sits.
A folder is a row, never a directory: on disk a file stays where the download layout put it (one
shared asset can be held by several users), so a rename or a move is a column update. Three kinds:
- a SOURCE folder is made automatically, one per origin a user has files from (YouTube, Plex clips,
uploads, one per other site). Its `source_key` names the origin; the client renders its name
from the key, in the reader's language.
- an OWN folder is the user's, with the name they gave it.
- the HUB folder (one per user, made when they first give a machine a token for it, `hub_folder`) is
where they swap files with that machine. Its name, like a source folder's, is the client's; a file
may be moved in like into an own folder, and it keeps a file until it is deleted.
Every file of a user is in exactly one folder. The folder a new file lands in is decided here
(`source_key_for` + `source_folder`), at every place a job is created, so the rule lives once. A file
moves only into one of the user's own folders, their Hub folder, or back to its own source folder
(`move`): a YouTube file in the TikTok folder would be a lie the folder's name tells.
Retention is the folder's: a source folder keeps a file `download_retention_days` (the value when the
folder was made; the user can change it), Uploads, own folders and the Hub keep it until it is deleted. A
done file carries its own `expires_at`, stamped by `done_expiry` when it finishes — unless the user
kept it — and its `ready_at`, the moment it finished (`stamp_done`).
"""
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from urllib.parse import urlsplit
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobStatus
from app.models import DownloadJob, FileFolder, MediaAsset
class FolderKind(StrEnum):
SOURCE = "source"
OWN = "own"
HUB = "hub"
class SourceKey(StrEnum):
"""The fixed origins. Any other site is `site:<host>` (see `site_key`)."""
YOUTUBE = "youtube"
PLEX = "plex"
UPLOAD = "upload"
SITE_PREFIX = "site:"
# A host's leading label that only says "the same site, another front door".
_FRONT_DOORS = ("www.", "m.", "mobile.", "web.")
_YOUTUBE_HOSTS = {"youtube.com", "youtu.be", "youtube-nocookie.com", "music.youtube.com"}
def site_key(url: str) -> str:
"""The source key of a page on some site: `site:<host>` without its front door (www., m.),
or the YouTube key for a YouTube address that arrived as a plain URL."""
host = (urlsplit(url).hostname or "").lower().rstrip(".")
for door in _FRONT_DOORS:
if host.startswith(door) and host.count(".") > 1:
host = host[len(door) :]
break
if host in _YOUTUBE_HOSTS:
return SourceKey.YOUTUBE
return f"{SITE_PREFIX}{host or 'unknown'}"
def source_key_for(db: Session, job: DownloadJob) -> str:
"""The origin a job's file came from — the source folder it belongs in.
The one recorded when it was placed, else: youtube → YouTube; a clip → the clip's source
(YouTube or Plex); an upload → Uploads; a URL → its site; an edit → wherever its parent's file
came from (its parent job, or, once that is deleted, the parent's asset)."""
if job.source_key:
return job.source_key
kind = job.source_kind
if kind == "youtube":
return SourceKey.YOUTUBE
if kind == "upload":
return SourceKey.UPLOAD
if kind == "clip":
return (
SourceKey.PLEX if (job.edit_spec or {}).get("source") == "plex" else SourceKey.YOUTUBE
)
if kind == "url":
return site_key(job.source_ref)
if kind == "edit":
parent = db.get(DownloadJob, job.source_job_id) if job.source_job_id else None
if parent is not None and parent.id != job.id:
return source_key_for(db, parent)
asset = db.get(MediaAsset, job.source_asset_id) if job.source_asset_id else None
if asset is not None:
if asset.source_kind == "youtube":
return SourceKey.YOUTUBE
if asset.source_kind == "upload":
return SourceKey.UPLOAD
if asset.source_kind == "url":
return site_key(asset.source_ref)
return SourceKey.UPLOAD
def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
"""The user's source folder for `key`, made on first use. A race on the unique (user, key) is
settled by re-selecting, inside a SAVEPOINT so the caller's own pending writes survive."""
stmt = select(FileFolder).where(FileFolder.user_id == user_id, FileFolder.source_key == key)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
retention = (
None if key == SourceKey.UPLOAD else sysconfig.get_int(db, "download_retention_days")
)
folder = FileFolder(
user_id=user_id, kind=FolderKind.SOURCE, source_key=key, retention_days=retention
)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: the expected outcome of the race the savepoint exists for — another request made
# the same folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def hub_folder(db: Session, user_id: int) -> FileFolder:
"""The user's Hub folder, made on first use; a race on its one-per-user index is settled like
`source_folder`'s."""
stmt = select(FileFolder).where(
FileFolder.user_id == user_id, FileFolder.kind == FolderKind.HUB
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
folder = FileFolder(user_id=user_id, kind=FolderKind.HUB)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: another request made the Hub folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def place(db: Session, job: DownloadJob) -> None:
"""Put a new job's file in its source folder, and record that origin on it. Called wherever a
job is created; the job must carry its `user_id`, `source_kind` and source refs already."""
key = source_key_for(db, job)
job.source_key = key
job.folder_id = source_folder(db, job.user_id, key).id
def done_expiry(db: Session, folder_id: int | None, kept: bool) -> datetime | None:
"""When a file that finishes now goes: its folder's retention from now, or never (kept, or a
folder that keeps files)."""
if kept or folder_id is None:
return None
# Read under a share lock, held to the caller's commit: a retention change (`set_retention` locks
# the folder) waits for this file and restamps it too, or this read waits for the change and
# takes the new value — never the old value stamped on a file the change did not see.
days = db.execute(
select(FileFolder.retention_days)
.where(FileFolder.id == folder_id)
.with_for_update(read=True)
).scalar()
return None if days is None else datetime.now(timezone.utc) + timedelta(days=days)
def stamp_done(db: Session, job: DownloadJob) -> None:
"""Stamp a job that just became done (in this session): when it did, and its expiry."""
job.ready_at = datetime.now(timezone.utc)
job.expires_at = done_expiry(db, job.folder_id, job.kept)
job.expiry_warned = False
class FolderError(Exception):
"""A folder change the rules refuse; `reason` is the refusal the route answers with."""
def __init__(self, reason: str) -> None:
super().__init__(reason)
self.reason = reason
def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The user's folder `folder_id` that a file is about to be put in — held (FOR SHARE) until the
caller's transaction ends, so a delete of the folder waits for the file and then takes it home
with the rest (`delete_own` locks it FOR UPDATE), never failing on its foreign key. `hold=False`
only looks (a check ahead of a long wait, which must not hold a lock through it). Another user's
folder answers like a missing one."""
stmt = select(FileFolder).where(FileFolder.id == folder_id)
if hold:
stmt = stmt.with_for_update(read=True)
folder = db.execute(stmt.execution_options(populate_existing=True)).scalar_one_or_none()
if folder is None or folder.user_id != user_id:
raise FolderError("file_folder_gone")
return folder
def upload_target(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The folder an upload asked to land in, if it may: one of the user's own folders, their Hub
folder, or their Uploads folder. A file put in a site's folder would claim an origin it does not
have."""
folder = own_folder(db, user_id, folder_id, hold=hold)
if folder.kind == FolderKind.SOURCE and folder.source_key != SourceKey.UPLOAD:
raise FolderError("file_folder_not_its_source")
return folder
def restamp(db: Session, job: DownloadJob) -> None:
"""Start a file's clock again from now: it moved to another folder, or its user stopped keeping
it. A file not done yet gets its expiry when it finishes (`stamp_done`), so it has none now."""
done = job.status == JobStatus.DONE
job.expires_at = done_expiry(db, job.folder_id, job.kept) if done else None
job.expiry_warned = False
def set_kept(db: Session, job: DownloadJob, kept: bool) -> None:
"""Keep a file until its user deletes it, or hand it back to its folder's retention."""
job.kept = kept
restamp(db, job)
def move(db: Session, job: DownloadJob, folder: FileFolder) -> None:
"""Put a file in `folder` — one of its user's own folders, their Hub folder, or the source folder
it came from.
The caller has checked that `folder` is the job's user's."""
if folder.kind == FolderKind.SOURCE and folder.source_key != source_key_for(db, job):
raise FolderError("file_folder_not_its_source")
if folder.id == job.folder_id:
return
job.folder_id = folder.id
restamp(db, job)
def _hold_for_change(db: Session, folder: FileFolder) -> None:
"""Lock `folder` FOR UPDATE and read it afresh into the same object: a change waits out a file
being put in it or stamped from it right now (`own_folder`, `done_expiry`), and works from what
another change that landed first left, not from what the caller read before the wait. The row is
consumed on purpose: an ORM result refreshes the object only as its rows are fetched."""
fresh = db.execute(
select(FileFolder)
.where(FileFolder.id == folder.id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if fresh is None:
raise FolderError("file_folder_gone")
def set_retention(db: Session, folder: FileFolder, days: int | None) -> int | None:
"""Change how long `folder` keeps a file (None = until deleted). The done files already in it
move by the difference — each keeps the age it has — but none ends up closer than the GC's
warning window, so a shortened retention still warns before it takes anything. Returns the
retention it replaced (equal to `days` when nothing changed)."""
_hold_for_change(db, folder) # so the file arriving now is among the ones restamped below
old = folder.retention_days
if old == days:
return old
folder.retention_days = days
now = datetime.now(timezone.utc)
floor = now + timedelta(days=sysconfig.get_int(db, "download_gc_grace_days"))
# Locked as they are read: one moved out or deleted meanwhile is no longer this folder's to time.
jobs = db.execute(
select(DownloadJob)
.where(
DownloadJob.folder_id == folder.id,
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
)
.with_for_update()
.execution_options(populate_existing=True)
).scalars()
for job in jobs:
if days is None:
job.expires_at = None
elif old is None or job.expires_at is None:
job.expires_at = max(now + timedelta(days=days), floor)
else:
job.expires_at = max(job.expires_at + timedelta(days=days - old), floor)
job.expiry_warned = False
return old
def delete_own(db: Session, folder: FileFolder) -> int:
"""Delete one of a user's own folders. Its files are not deleted: each goes back to the source
folder it came from, its clock starting there now. Returns how many moved."""
if folder.kind != FolderKind.OWN:
raise FolderError("file_folder_not_own")
_hold_for_change(db, folder) # so the file arriving now is taken home too
# Locked as they are read: a file moved out meanwhile stays where it went, and one deleted
# meanwhile is not written back (its gone row would fail the delete).
jobs = (
db.execute(
select(DownloadJob)
.where(DownloadJob.folder_id == folder.id)
.with_for_update()
.execution_options(populate_existing=True)
)
.scalars()
.all()
)
for job in jobs:
job.folder_id = source_folder(db, job.user_id, source_key_for(db, job)).id
restamp(db, job)
db.flush() # the files leave before the folder goes: its foreign key RESTRICTs the delete
db.delete(folder)
return len(jobs)
+44 -14
View File
@@ -13,7 +13,7 @@ spec shape (all keys optional; defaults + browser-playability clamp applied by `
vcodec always coerced to null → H.264 selected (see `normalize`); no opt-out
embed_subs bool embed_chapters bool embed_thumbnail bool sponsorblock bool
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser (E7); the
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser; the
`container`/`vcodec` keys survive only so an old stored profile deserializes without error.
"""
@@ -105,6 +105,28 @@ def log_safe(text: object) -> str:
return _PATH_RE.sub("…", text)
# yt-dlp's HTTP downloader, refusing a file announced larger than `max_filesize`, logs this line and
# RETURNS — no exception and no file (measured; test_formats holds it against the pinned yt-dlp).
_OVER_CAP_LINE = "File is larger than max-filesize"
_OVER_CAP: contextvars.ContextVar[list[bool] | None] = contextvars.ContextVar(
"formats_over_cap", default=None
)
@contextlib.contextmanager
def watching_size_cap() -> Iterator[list[bool]]:
"""While yt-dlp works inside it: a one-slot flag, set when it refused a file over the size cap
(`build_ydl_opts(max_bytes=…)`). It then returns as if it had downloaded, and all the caller
would find is no file — a processing fault, to anyone reading it, for what is the owner's
storage. A context variable, like `hiding`: yt-dlp's logger calls run on the extraction's thread."""
flag = [False]
token = _OVER_CAP.set(flag)
try:
yield flag
finally:
_OVER_CAP.reset(token)
class _YdlLog:
"""yt-dlp's logger protocol, pointed at ours.
@@ -124,6 +146,8 @@ class _YdlLog:
is the one line whose level should say so."""
def _emit(self, level: int, msg: str) -> None:
if _OVER_CAP_LINE in msg and (over := _OVER_CAP.get()) is not None:
over[0] = True
# The line goes as an ARGUMENT, not as the format string. yt-dlp writes progress as
# `[download] 53.2% of 12.00MiB`, and while stdlib logging only applies `%` formatting to
# a record that has args, this way the question never arises — and it is the shape ruff's
@@ -182,7 +206,9 @@ _DEFAULTS = {
# v4: EVERY download is clamped to the mp4/H.264/AAC compat profile (no more mkv/webm/VP9/AV1/Opus
# opt-out), so any asset cached under an old non-compat spec — or a stale non-H.264 mp4 that slipped
# the v3 net — must re-derive through the current browser-safe guarantee.
_SIG_VERSION = 4
# v5: the phone-safe rule (`playable`) also judges the pixel format, so a cached 10-bit / 4:4:4 H.264
# asset (which v4 let through as "H.264") re-derives as 8-bit 4:2:0 instead of being reused.
_SIG_VERSION = 5
# Audio-only downloads version on their own, so a change to the audio bytes doesn't re-download every
# cached video. v5: an extracted .mp3/.m4a is stored as itself instead of remuxed into an .mp4 that
# iPad's WebKit refuses to play, so a cached remuxed audio asset must re-derive.
@@ -217,14 +243,15 @@ def normalize(spec: dict) -> dict:
# Audio-only files can't embed subtitles/thumbnails as video; keep the flags meaningful.
if s["mode"] == "a":
s["embed_subs"] = False
# Browser-playability guarantee (E7): the Download module exists to download → share → PLAY
# Browser-playability guarantee: the Download module exists to download → share → PLAY
# in-browser, so every output must play in EVERY browser, iOS/Safari/WebKit included. The only
# universally-safe combo is mp4 + H.264 + AAC/MP3, so we CLAMP the spec to it HERE — the single
# point every path (picker, stored profile, raw API) flows through — instead of only hiding
# options in the UI. Non-mp4 containers (mkv doesn't play at all; webm/VP9/AV1 fail iOS) and any
# explicit vcodec are coerced to the compat profile; Opus audio (Safari-flaky as a bare file)
# falls back to AAC/m4a. A source that exposes no H.264 is still caught by the worker's
# browser-safe re-encode (`_ensure_browser_playable`), so playback is guaranteed end-to-end.
# universally-safe combo is mp4 + H.264 + AAC (an .mp3/.m4a for audio), so we CLAMP the spec to
# it HERE — the single point every path (picker, stored profile, raw API) flows through — instead
# of only hiding options in the UI. Non-mp4 containers (mkv plays nowhere on iOS; AV1 fails there,
# and webm/VP9 plays only on recent devices) and any explicit vcodec are coerced to the compat
# profile; Opus audio falls back to AAC/m4a. A source that exposes no H.264 (or a 10-bit one) is
# still caught by the worker's phone-safe step (`downloads.playable`), so playback is guaranteed
# end-to-end.
s["container"] = "mp4"
s["vcodec"] = None
if s["audio_format"] not in ("m4a", "mp3"):
@@ -274,12 +301,15 @@ def build_ydl_opts(
`extractor_args` with the player-client set + bgutil PO-token base_url); it is merged onto the
generic options below. Everything here is source-agnostic.
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
`cookiefile` is the acting user's per-host cookies.txt path, attached ONLY by the worker
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
merge so a source adapter can never clobber it.
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp refuses a format larger
than it rather than filling the disk and only then being caught by the quota accounting."""
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp's HTTP downloader refuses
a file announced larger than it rather than filling the disk and only then being caught by the
quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
pipeline also holds every produced file to what is left by its real size."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
@@ -329,7 +359,7 @@ def build_ydl_opts(
if max_bytes and max_bytes > 0:
opts["max_filesize"] = max_bytes
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
# The acting user's per-host cookie. AFTER the adapter merge on purpose: an adapter must not
# be able to override the authenticated session the worker resolved for this specific user.
if cookiefile:
opts["cookiefile"] = cookiefile
@@ -351,7 +381,7 @@ def build_ydl_opts(
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there — so H.264 is selected at the
# SELECTION level, not merely preferred: a sort key can't conjure an H.264 stream a source
# doesn't expose as video-only. A source that ships only non-H.264 is caught after download by
# the worker's browser-safe re-encode (`_ensure_browser_playable`).
# the worker's phone-safe step (`downloads.playable`).
if s["mode"] == "v":
# Video-only: keep it audio-free. Prefer an H.264 video-only stream, then any video-only,
# and only fall to a progressive (audio-bearing) `best` as a last resort.
+232 -96
View File
@@ -1,82 +1,167 @@
"""Download retention garbage collector (a scheduler job).
Five passes each run:
1. Pre-expiry warning — for ready assets expiring within the grace window, notify each holder
once (gc_notified flag) so a download isn't silently deleted.
2. Expiry deletion — delete assets past their TTL (files + row). The FK SET NULL nulls the
holders' jobs' asset_id, so those jobs surface as "expired" (re-downloadable) in the UI.
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed
ready assets until back under it.
Six passes each run:
1. Pre-expiry warning — a done file whose `expires_at` (its folder's retention, `folders`) falls
within the grace window, and that its user did not keep, warns its user once (`expiry_warned`).
2. Expiry — such a file past its `expires_at` leaves the library: the job is deleted and its hold
released, so the shared file goes with its last holder.
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed ready
assets (never an upload, never one a user kept) until back under it; each holder's file leaves
their library, with a notice.
4. Orphan reap — delete errored, fileless, unreferenced asset rows (they carry no TTL, so
passes 1-3 skip them; benign row-count hygiene).
5. Orphan FILE sweep — delete on-disk files/dirs no surviving asset owns (cruft left when a
re-download renames an asset, e.g. old accented channel dirs), skipping system trees and
anything newer than an hour so an in-flight download is never touched.
re-download renames an asset, e.g. old accented channel dirs, or an upload's stranded file).
It skips the dot-prefixed system trees (all but `.uploads`) and anything newer than an hour,
so an in-flight download is never touched.
6. Token expiry warning — a machine's token for the library about to run out warns its user once
(`apitokens.warn_expiring`): the library's other clock.
Runs in the API process (which mounts DOWNLOAD_ROOT). Pure disk/DB work — no YouTube quota.
Runs in the API process (which mounts FILES_ROOT). Pure disk/DB work — no YouTube quota.
"""
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
from sqlalchemy import func, select
from sqlalchemy import func, or_, select, update
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app import apitokens, logs, sysconfig
from app.config import settings
from app.downloads import storage
from app.downloads import quota, service, storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, MediaAsset
from app.notifications import create_notification
log = logs.get_logger(__name__)
# Dot-prefixed top-level trees that are NOT part of the Plex/flat asset layout — the orphan sweep
# never touches them (user clips, Plex HLS/img caches, in-progress staging, editor storyboards).
_SYSTEM_TREE = {".edits", ".plex-hls", ".plex-img-cache", ".staging", ".storyboards"}
# The one dot-prefixed top-level tree the orphan sweep enters. The others are not part of the asset
# layout (user clips, Plex HLS/img caches, in-progress staging, editor storyboards) and are skipped.
_UPLOAD_TREE = ".uploads"
_ORPHAN_MIN_AGE_S = 3600 # only sweep files older than this, so an in-flight download is never hit
def _holders(db: Session, asset_id: int) -> list[int]:
return list(
db.execute(
select(DownloadJob.user_id).where(DownloadJob.asset_id == asset_id).distinct()
).scalars()
def _notify(db: Session, job: DownloadJob, asset: MediaAsset | None, ntype: str) -> None:
title = job.display_name or (asset.title if asset else None) or "download"
create_notification(
db,
user_id=job.user_id,
type=ntype,
title=title, # English fallback; the client renders a translated message from data
data={
"kind": ntype,
"title": title,
"video_id": asset.source_ref if asset else None,
"job_id": job.id,
},
commit=False,
)
def _notify_holders(db: Session, asset: MediaAsset, ntype: str) -> None:
title = asset.title or "download"
for uid in _holders(db, asset.id):
create_notification(
db,
user_id=uid,
type=ntype,
title=title, # English fallback; the client renders a translated message from data
data={"kind": ntype, "title": asset.title, "video_id": asset.source_ref},
commit=False,
def _relock_expired(
db: Session, job_id: int, asset_id: int | None, now: datetime
) -> DownloadJob | None:
"""Re-read an expired job under the lock order the user routes take (`own_job_for_update`: the
asset, then the job) and say whether it is still to go. Between the pass's SELECT and here a
user may have deleted or re-downloaded it (dropping it again would release its asset a second
time, freeing a file another job still holds) or kept or moved it (it is no longer expired)."""
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
still = (
job is not None
and job.asset_id == asset_id
and job.status == JobStatus.DONE
and not job.kept
and job.expires_at is not None
and job.expires_at <= now
)
return job if still else None
def _mark_warned(db: Session, job_id: int, expires_at: datetime) -> DownloadJob | None:
"""Mark one file warned if it still goes when the pass saw it go — not deleted, kept, moved or
re-timed since (each of those sets a fresh expiry, and its own warning to come) — and return it.
The UPDATE's row lock holds it so until the commit."""
hit = db.execute(
update(DownloadJob)
.where(
DownloadJob.id == job_id,
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expiry_warned.is_(False),
DownloadJob.expires_at == expires_at,
)
.values(expiry_warned=True)
.returning(DownloadJob.id)
.execution_options(synchronize_session=False)
).scalar_one_or_none()
return db.get(DownloadJob, job_id) if hit is not None else None
def _delete_asset(db: Session, asset: MediaAsset, ntype: str) -> None:
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
_notify_holders(db, asset, ntype)
db.delete(asset) # jobs.asset_id -> NULL via FK ON DELETE SET NULL
def _on_disk(db: Session) -> int:
"""The bytes the cache holds now (every asset whose file is on disk)."""
return (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).scalar()
or 0
)
def _sweep_orphan_files(db: Session, download_root: str) -> int:
"""Delete media/sidecar files (and now-empty dirs) under DOWNLOAD_ROOT that no `ready` asset
def _relock_evictable(db: Session, asset_id: int) -> tuple[MediaAsset | None, list[DownloadJob]]:
"""Re-read an eviction candidate and its holders under their locks (the asset, then its jobs —
the order the user routes take): between the pass's SELECT and here a holder may have kept it,
or every holder deleted it."""
asset = db.execute(
select(MediaAsset)
.where(MediaAsset.id == asset_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
holders = (
db.execute(
select(DownloadJob)
.where(DownloadJob.asset_id == asset_id)
.with_for_update()
.execution_options(populate_existing=True)
)
.scalars()
.all()
)
return asset, list(holders)
def _drop_job(db: Session, job: DownloadJob, ntype: str) -> None:
"""Take a file out of its user's library: tell them, release the job's hold on the asset (the
file goes with the last holder), delete the row (its shares and links cascade)."""
_notify(db, job, db.get(MediaAsset, job.asset_id) if job.asset_id else None, ntype)
service.release_asset(db, job)
db.delete(job)
def _sweep_orphan_files(db: Session, files_root: str) -> int:
"""Delete media/sidecar files (and now-empty dirs) under FILES_ROOT that no `ready` asset
owns — cruft left when a re-download changes an asset's on-disk name (e.g. an old accented
channel dir, or a poster from a superseded naming scheme). Skips the dot-prefixed system trees
and only removes files older than `_ORPHAN_MIN_AGE_S`, so a just-completed or in-flight download
is never deleted, and only files NOT owned by a surviving asset."""
root = Path(download_root)
channel dir, or a poster from a superseded naming scheme) — or, under `.uploads`, that no upload
row names at all. Skips the other dot-prefixed system trees and only removes files older than
`_ORPHAN_MIN_AGE_S`, so a just-completed or in-flight download is never deleted, and only files
NOT owned by a surviving asset."""
root = Path(files_root)
if not root.is_dir():
return 0
keep: set[Path] = set()
# An upload's file is on disk from ingest on, whatever its state (a queued conversion's original,
# a failed one kept for Retry), so every upload row keeps its file — not only a ready one.
for rel in db.execute(
select(MediaAsset.rel_path).where(
MediaAsset.status == "ready", MediaAsset.rel_path.is_not(None)
or_(MediaAsset.status == "ready", MediaAsset.source_kind == "upload"),
MediaAsset.rel_path.is_not(None),
)
).scalars():
media = (root / rel).resolve()
@@ -99,7 +184,10 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
unreclaimed = 0
for entry in root.iterdir():
if entry.name.startswith("."): # .edits / .plex-* / .staging / .storyboards + any dotfile
# .edits / .plex-* / .staging / .storyboards + any dotfile — but `.uploads` is swept: a
# conversion stopped between placing the new file and the commit that points at it (or the
# unlink of the original after it) leaves a file no row names, and nothing else finds it.
if entry.name.startswith(".") and entry.name != _UPLOAD_TREE:
continue
files = [entry] if entry.is_file() else [p for p in entry.rglob("*") if p.is_file()]
for f in files:
@@ -143,74 +231,95 @@ def run_download_gc(db: Session) -> dict:
grace_days = sysconfig.get_int(db, "download_gc_grace_days")
warned = expired = evicted = reaped = 0
# 1. Pre-expiry warnings.
# 1. Pre-expiry warnings — one per file, the first run inside the grace window. Each mark is
# conditional on the expiry the SELECT saw and committed on its own: a file deleted since is
# passed over (a write to its gone row would fail the whole run), and one kept, moved or re-timed
# since keeps its fresh expiry unwarned.
warn_before = now + timedelta(days=grace_days)
soon = (
db.execute(
select(MediaAsset).where(
MediaAsset.status == "ready",
MediaAsset.gc_notified.is_(False),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at > now,
MediaAsset.expires_at <= warn_before,
)
soon = db.execute(
select(DownloadJob.id, DownloadJob.expires_at).where(
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expiry_warned.is_(False),
DownloadJob.expires_at.is_not(None),
DownloadJob.expires_at > now,
DownloadJob.expires_at <= warn_before,
)
.scalars()
.all()
)
for asset in soon:
_notify_holders(db, asset, "download_expiring")
asset.gc_notified = True
warned += 1
if soon:
).all()
db.rollback() # the pass-1 snapshot holds no locks; each mark takes its own below
for job_id, expires_at in soon:
job = _mark_warned(db, job_id, expires_at)
if job is not None:
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
_notify(db, job, asset, "download_expiring")
warned += 1
db.commit()
# 2. Expiry deletion (ready assets past TTL, and any errored leftovers with a TTL).
dead = (
db.execute(
select(MediaAsset).where(
MediaAsset.status.in_(("ready", "error")),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at <= now,
)
# 2. Expiry — a file past its folder's retention leaves the library (warned in pass 1).
# Each file is re-checked under its locks and committed on its own: the release unlinks the
# file at once, so it must never act on what the SELECT saw a while ago.
dead = db.execute(
select(DownloadJob.id, DownloadJob.asset_id).where(
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expires_at.is_not(None),
DownloadJob.expires_at <= now,
)
.scalars()
.all()
)
for asset in dead:
_delete_asset(db, asset, "download_expired")
expired += 1
if dead:
).all()
db.rollback() # the pass-2 snapshot holds no locks; each file takes its own below
for job_id, asset_id in dead:
job = _relock_expired(db, job_id, asset_id, now)
if job is not None:
_drop_job(db, job, "download_expired")
expired += 1
db.commit()
# 3. LRU eviction over the total cache cap.
# 3. LRU eviction over the total cache cap: the disk's safety net. Never an upload (no source to
# fetch again) and never a file anyone kept; an evicted file leaves every holder's library.
cap = sysconfig.get_int(db, "download_total_max_bytes")
if cap > 0:
total = (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(
MediaAsset.status == "ready"
)
).scalar()
or 0
)
if total > cap:
candidates = (
if _on_disk(db) > cap:
kept_assets = select(DownloadJob.asset_id).where(
DownloadJob.kept.is_(True), DownloadJob.asset_id.is_not(None)
)
candidate_ids = (
db.execute(
select(MediaAsset)
.where(MediaAsset.status == "ready")
select(MediaAsset.id)
.where(
MediaAsset.status == "ready",
MediaAsset.source_kind != "upload",
MediaAsset.id.not_in(kept_assets),
)
.order_by(MediaAsset.last_access_at.asc().nulls_first())
)
.scalars()
.all()
)
for asset in candidates:
if total <= cap:
db.rollback()
for asset_id in candidate_ids:
# Counted afresh before each: a file deleted since (this one, an upload, a candidate
# further down) has freed its space, and no other user's file goes for that space.
if _on_disk(db) <= cap:
break
total -= asset.size_bytes or 0
_delete_asset(db, asset, "download_evicted")
# Locked and re-read like an expiry: since the SELECT a holder may have kept it,
# every holder deleted it, or someone asked for it again (a fresh cache hit).
asset, holders = _relock_evictable(db, asset_id)
if (
asset is None
or asset.status != "ready"
or any(j.kept for j in holders)
or (asset.last_access_at is not None and asset.last_access_at > now)
):
db.commit()
continue
for job in holders:
_notify(db, job, asset, "download_evicted")
db.delete(job)
if asset.rel_path:
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
evicted += 1
db.commit()
db.commit()
# 4. Reap orphaned errored rows (no file, no holder). get_or_create_asset resets a still-
# referenced errored row to pending on re-enqueue, so these only accumulate once every
@@ -232,9 +341,35 @@ def run_download_gc(db: Session) -> dict:
if orphans:
db.commit()
# 4b. Reap unheld uploads. An upload carries no expires_at, is never evicted and never re-enqueued,
# so the passes above never free one that lost its last holder outside `release_asset` — a
# conversion that finished after its job was cancelled, or one cancelled mid-run (the asset
# was `downloading` then). Its file, ready or still the unconverted original, is at `rel_path`.
unheld = (
db.execute(
select(MediaAsset).where(
MediaAsset.source_kind == "upload",
MediaAsset.ref_count == 0,
MediaAsset.status != "downloading",
)
)
.scalars()
.all()
)
for asset in unheld:
if asset.rel_path: # no holder is left to tell, so no notice — just the file and the row
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
reaped += 1
if unheld:
db.commit()
# 5. Orphan FILE sweep — remove on-disk cruft no surviving asset owns (old accented dirs, stale
# posters). Runs last so the keep-set reflects the assets that survived passes 2-4.
swept = _sweep_orphan_files(db, settings.download_root)
swept = _sweep_orphan_files(db, settings.files_root)
# 6. A machine's token about to run out.
tokens_warned = apitokens.warn_expiring(db)
result = {
"warned": warned,
@@ -242,6 +377,7 @@ def run_download_gc(db: Session) -> dict:
"evicted": evicted,
"reaped": reaped,
"swept": swept,
"tokens_warned": tokens_warned,
}
log.info("download_gc %s", result)
return result
+2 -2
View File
@@ -68,8 +68,8 @@ def _reset_asset_pending(asset_id: int) -> None:
def _staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"asset-{asset_id}"
return Path(settings.files_root) / ".staging" / f"asset-{asset_id}"
def _edit_staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"edit-{asset_id}"
return Path(settings.files_root) / ".staging" / f"edit-{asset_id}"
+75 -11
View File
@@ -1,10 +1,15 @@
"""Public "share by link" helpers for the download center.
A DownloadLink is a capability URL: the unguessable `token` grants anyone read access to one
download's file on the login-free `/watch/{token}` page. Password-protected links additionally
require a short-lived signed **grant** on the file request — because a `<video src>` can't send a
header, the password is exchanged once at `/unlock` for an HMAC grant appended to the file URL,
so the raw password never rides in the media URL (or server logs).
A DownloadLink is a capability URL: the unguessable token grants anyone read access to one
download's file, or to the files in one of its owner's own folders (`folder_files`), on the
login-free `/watch/{token}` page. So the token is never stored as written:
the link is found by its SHA-256, and the owner's copy is Fernet-encrypted — a database copy alone
opens nothing. Without that copy (no TOKEN_ENCRYPTION_KEY, or a key since changed) the link still
opens; only its address cannot be shown again, and the owner makes a new link to share.
Password-protected links additionally require a short-lived signed **grant** on the file request —
because a `<video src>` can't send a header, the password is exchanged once at `/unlock` for an HMAC
grant appended to the file URL, so the raw password never rides in the media URL (or server logs).
"""
import hashlib
@@ -12,16 +17,71 @@ import hmac
import secrets
from datetime import datetime, timezone
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import security
from app.config import settings
from app.models import DownloadLink
from app.downloads import storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_GRANT_TTL = 6 * 3600 # a watch session's grant is good for 6h, then the viewer re-unlocks
# The most files a folder link lists: as many as the owner's own Files list shows (`routes.files.
# MAX_LIST`; a test holds the two equal — this layer may not import that one).
FOLDER_MAX = 500
def new_token() -> str:
return secrets.token_urlsafe(24) # ~32 chars, 192 bits
def seal(link: DownloadLink, token: str) -> None:
"""Store `token` on `link` the only ways it is kept: its hash, and its encrypted copy."""
link.token_hash = security.hash_token(token)
link.token_enc = security.encrypt(token) if security.can_encrypt() else None
def find(db: Session, token: str) -> DownloadLink | None:
"""The link a `/watch` token names, by its hash."""
return db.execute(
select(DownloadLink).where(DownloadLink.token_hash == security.hash_token(token))
).scalar_one_or_none()
def recover(link: DownloadLink) -> str | None:
"""The link's token from its encrypted copy, or None when there is none this key can read."""
return security.decrypt_optional(link.token_enc)
def in_folder(folder: FileFolder) -> tuple:
"""Which jobs a link to `folder` reaches: the done files in it NOW, the folder owner's. A file
moved out stops opening through the link at its next request; one moved in starts."""
return (
DownloadJob.folder_id == folder.id,
DownloadJob.user_id == folder.user_id,
DownloadJob.status == JobStatus.DONE,
)
def folder_files(db: Session, folder: FileFolder) -> list[tuple[DownloadJob, MediaAsset]]:
"""What a link to `folder` shows — its page lists exactly these, and its link card counts them:
each file `in_folder` whose asset is ready and whose bytes are on disk (so the list never offers
a dead row), newest first, at most `FOLDER_MAX`."""
rows = db.execute(
select(DownloadJob, MediaAsset)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(*in_folder(folder), MediaAsset.status == "ready")
.order_by(DownloadJob.ready_at.desc().nulls_last(), DownloadJob.id.desc())
.limit(FOLDER_MAX)
).all()
return [
(job, asset)
for job, asset in rows
if storage.safe_abs_path(settings.files_root, asset.rel_path) is not None
]
def is_expired(link: DownloadLink) -> bool:
return link.expires_at is not None and link.expires_at <= datetime.now(timezone.utc)
@@ -55,18 +115,22 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
return False
if exp < datetime.now(timezone.utc).timestamp():
return False
return hmac.compare_digest(sig, _sign(f"{token}.{exp}.{password_version}"))
# As bytes: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it, and
# `sig` is whatever the URL carried.
return hmac.compare_digest(sig.encode(), _sign(f"{token}.{exp}.{password_version}").encode())
# --- serialization -------------------------------------------------------------------------
def owner_view(link: DownloadLink, base_url: str = "") -> dict:
"""What the file owner sees when managing their links."""
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
"""What the file owner sees when managing their links. `token` is the one just made (it is in
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
token = token or recover(link)
return {
"id": link.id,
"token": link.token,
"url": f"{base_url}/watch/{link.token}",
"token": token,
"url": f"{base_url}/watch/{token}" if token else None,
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
+37 -10
View File
@@ -11,13 +11,12 @@ import html
import re
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import links as linksmod
from app.downloads import storage
from app.models import DownloadJob, DownloadLink, MediaAsset
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
@@ -74,22 +73,25 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
def _watch_tags(token: str, db: Session) -> str | None:
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
return None
url = f"{settings.app_base}/watch/{token}"
folder = link.folder_id is not None
if link.password_hash:
# Password-gated: don't leak the title/thumbnail in the unfurl.
return "\n ".join(
[
"<title>Siftlode</title>",
_tag("og:title", "Password-protected video"),
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
_tag("og:description", "Shared via Siftlode"),
_tag("og:type", "video.other"),
_tag("og:type", "website" if folder else "video.other"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
]
)
if folder:
return _folder_tags(db, link, url)
job = db.get(DownloadJob, link.job_id)
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if job is None or asset is None or asset.status != "ready":
@@ -135,7 +137,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
if image_dims is None and image.endswith("poster.jpg") and asset.poster_path:
# safe_abs_path returns None if the poster file is missing (gc'd) or escapes the root —
# guard it, else _jpeg_size(None) would raise an uncaught AttributeError on this public page.
poster_abs = storage.safe_abs_path(settings.download_root, asset.poster_path)
poster_abs = storage.safe_abs_path(settings.files_root, asset.poster_path)
image_dims = _jpeg_size(poster_abs) if poster_abs else None
if image_dims:
tags.append(_tag("og:image:width", str(image_dims[0])))
@@ -147,13 +149,38 @@ def _watch_tags(token: str, db: Session) -> str | None:
return "\n ".join(tags)
def render_watch_html(token: str, index_html: Path, db: Session) -> str | None:
"""Return index.html with the OG block replaced for this watch token, or None to serve the
generic page unchanged (invalid/expired/not-ready link, or a template without the markers)."""
def _folder_tags(db: Session, link: DownloadLink, url: str) -> str | None:
"""A folder link's card: the folder's name and how many files it holds — never their names,
which the page shows only to whoever opens it."""
folder = db.get(FileFolder, link.folder_id)
if folder is None or not folder.name:
return None
count = len(linksmod.folder_files(db, folder)) # the page's own list, so the two agree
title = folder.name.strip()
about = f"{count} file{'' if count == 1 else 's'} · Shared via Siftlode"
return "\n ".join(
[
f"<title>{html.escape(title)}</title>",
_tag("og:title", title),
_tag("og:description", about),
_tag("og:type", "website"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
_tag("twitter:title", title, attr="name"),
_tag("twitter:description", about, attr="name"),
_tag("twitter:card", "summary", attr="name"),
]
)
def render_watch_html(token: str, page_html: Path, db: Session) -> str | None:
"""Return the watch page (watch.html) with the OG block replaced for this token, or None to
serve the generic page unchanged (invalid/expired/not-ready link, or a template without the
markers)."""
tags = _watch_tags(token, db)
if tags is None:
return None
text = index_html.read_text(encoding="utf-8")
text = page_html.read_text(encoding="utf-8")
if "<!--OG:START-->" not in text:
return None
# A function replacement avoids re.sub interpreting backslashes/group refs in `tags`.
+546 -82
View File
@@ -3,17 +3,20 @@
import re
import shutil
import time
from datetime import date, datetime, timedelta, timezone
from datetime import date, datetime, timezone
from pathlib import Path
import yt_dlp
from yt_dlp.utils import download_range_func
from app import logs, sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads import clip as clipmod
from app.downloads import edit as editmod
from app.downloads import errors as dlerrors
from app.downloads import adapters, credentials, formats, mediakind, probe, quota, service, storage
from app.downloads import adapters, credentials, formats, mediakind, playable, probe, quota
from app.downloads import service, storage
from app.downloads import urlguard
from app.downloads.jobdb import (
_edit_staging_dir,
@@ -25,7 +28,9 @@ from app.downloads.jobdb import (
)
from app.downloads.stated import StatedFailure
from app.downloads.status import JobStatus
from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset
from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset, PlexItem
from app.plex import lookups as plex_lookups
from app.plex import paths as plex_paths
from app.notifications import create_notification
from app.titles import normalize_title
@@ -36,6 +41,15 @@ class _Aborted(Exception):
"""Raised inside the progress hook when the job was paused/canceled mid-download."""
class _ClipFetchFailed(Exception):
"""A clip's YouTube range fetch failed: its message and the download catalog's diagnosis, so a
clip on YouTube's wall says what a download of the same video would."""
def __init__(self, msg: str, diagnosis: dlerrors.Diagnosis):
super().__init__(msg)
self.diagnosis = diagnosis
class _OwnerRefused(Exception):
"""A refusal that belongs to THIS job's owner, not to the source.
@@ -58,6 +72,10 @@ _DB_WRITE_INTERVAL_S = 1.0
_CANCEL_POLL_INTERVAL_S = 1.0
_THUMB_SUFFIXES = (".jpg", ".jpeg", ".png", ".webp")
# The job phase while a file is brought into the phone-safe shape (`_ensure_phone_safe`) for a
# download or an upload. Its own word, not yt-dlp's "processing": it reports ffmpeg's real progress,
# so the queue row shows a % for it, where yt-dlp's post-steps only get an indeterminate pulse.
CONVERTING = "converting"
# yt-dlp's working files in the staging dir. None of them is ever the finished media, but several
# CAN be the biggest file there (a video-only per-format stream, an abandoned fragment run), so the
# staging fallback must exclude them by name — picking one would store e.g. a silent video-only
@@ -73,20 +91,28 @@ _YTDLP_TEMP_RE = re.compile(
def _notify_download_done(db, user_id: int | None, title: str | None, job_id, *, kind: str) -> None:
"""Post a best-effort 'ready' notice to the owner's inbox — downloads/edits used to finish
silently while scheduler jobs and playlist-push already notified (R11 feedback symmetry). A
silently while scheduler jobs and playlist-push already notified (feedback symmetry). A
notification failure must never mask the completed job, so it is isolated + rolled back (mirrors
scheduler._notify_done). `kind` ("download"/"edit") lets the inbox pick accurate copy; the
scheduler._notify_done). `kind` ("download"/"edit"/"upload") lets the inbox pick accurate copy; the
English title here is only a fallback for an unknown type. `user_id`/`title` are captured by the
caller BEFORE its commit, so reading them here needs no post-commit refresh of an expired row."""
caller BEFORE its commit; only the folder is read here, once, for the notice's "Open"."""
if user_id is None:
return
try:
# The notice's "Open" lands on the file in its folder: both ride along (a folder the file
# has left since is the page's to notice — it then looks among every file).
job = db.get(DownloadJob, job_id)
create_notification(
db,
user_id=user_id,
type="download_done",
title=f"Ready to watch: {title}" if title else "Ready to watch",
data={"title": title, "kind": kind},
data={
"title": title,
"kind": kind,
"job_id": job_id,
"folder_id": job.folder_id if job else None,
},
)
except Exception:
db.rollback()
@@ -208,7 +234,7 @@ def _cancel_poll(job_id: int, interval_s: float = _CANCEL_POLL_INTERVAL_S, *, no
def cancelled() -> bool:
if not state["cancelled"] and allow():
state["cancelled"] = _job_status(job_id) not in ("running", None)
state["cancelled"] = _job_status(job_id) not in (JobStatus.RUNNING, None)
return state["cancelled"]
return cancelled
@@ -219,7 +245,7 @@ def _ffmpeg_progress_writer(job_id: int, phase: str, *, now=time.monotonic):
when the whole-percent value changed (the bar shows integers), plus a separate once-per-interval
heartbeat while the percent is stuck (so a slow re-encode isn't mistaken for a stall)."""
# Progress writes have their OWN 1/interval budget so a heartbeat can never delay a real write (the
# round-1 concern). The heartbeat, in turn, is gated on `last_touch` — the last DB write OR beat —
# original concern). The heartbeat, in turn, is gated on `last_touch` — the last DB write OR beat —
# so a progress write (which already refreshes updated_at) suppresses a redundant heartbeat that
# same interval, keeping active downloads at ~1 write/interval, not two.
allow_write = _rate_limiter(_DB_WRITE_INTERVAL_S, now=now)
@@ -328,7 +354,7 @@ def _media_info(info: dict) -> dict:
def _extract_with_fallback(
job_id, asset_id, url, spec, staging, adapter, cookiefile=None, max_bytes=None
job_id, asset_id, url, spec, staging, adapter, cookiefile=None, max_bytes=None, section=None
):
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
@@ -341,10 +367,17 @@ def _extract_with_fallback(
(adapters.POST_*_MESSAGE) instead of letting yt-dlp guess, and a failed page read fails the
download as itself.
`cookiefile` (S6) is the acting user's per-host cookies.txt, already gated by
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated.
`cookiefile` is the acting user's per-host cookies.txt, already gated by
`credentials.resolve_cookiefile` (a clip: `resolve_clip_cookiefile`); passed to every attempt so
the fallback stays authenticated.
`max_bytes` (SEC-13) is the owner's remaining storage budget, applied to every attempt."""
`max_bytes` (SEC-13) is the owner's remaining storage budget, applied to every attempt. A
`section` fetch goes through yt-dlp's ffmpeg downloader, which does not read it: a clip holds
its own output to the cap (`_process_clip`).
`section` = (start_s, end_s) fetches only that range (an ad-hoc clip, see downloads/clip): the
range is stream-copied, and its first frame is the marked one (measured), so the clip's own
re-encode starts at 0 instead of re-fetching keyframes at the cut."""
client_sets = adapter.client_sets(authed=cookiefile is not None)
outtmpl = str(staging / "%(id)s.%(ext)s")
# Resolving may read a page first — network time the stall watchdog must not see as a stall.
@@ -370,6 +403,9 @@ def _extract_with_fallback(
max_bytes=max_bytes,
)
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
if section is not None:
opts["download_ranges"] = download_range_func([], [section])
opts["force_keyframes_at_cuts"] = False
_touch_job(job_id) # stall-watchdog heartbeat: extraction writes no progress, so a slow /
# multi-client-set fallback would otherwise cross the stall timeout with a stale updated_at.
# (Per-ATTEMPT only; a single extract_info that itself runs past the stall timeout is still
@@ -406,58 +442,58 @@ def _extract_with_fallback(
raise last_exc # unreachable (loop either returns or raises)
def _download_settings() -> tuple[str, int]:
"""Layout + retention days from the DB config (admin-overridable on the Configuration page),
falling back to the env defaults. Read here, not from `settings`, so an admin edit takes effect."""
def _download_layout() -> str:
"""The on-disk layout from the DB config (admin-overridable on the Configuration page), falling
back to the env default. Read here, not from `settings`, so an admin edit takes effect. (How long
a file stays is its folder's, stamped on the job when it is done — `folders.done_expiry`.)"""
with SessionLocal() as db:
return (
sysconfig.get_str(db, "download_layout"),
sysconfig.get_int(db, "download_retention_days"),
)
return sysconfig.get_str(db, "download_layout")
def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict) -> Path:
"""Make an mp4-compat-profile download play in every browser: H.264 + AAC/MP3 in mp4. If the
download (whatever the source shipped) isn't already that, re-encode the offending stream(s) —
or just remux when the codecs are fine but the container isn't. **Scope:** only the mp4 compat
profile is normalized; an explicit `vcodec` or a non-mp4 custom profile is the user opting out
of the guarantee and is left as-is (so a `webm` profile can still ship VP9/Opus). Returns the
file to store (converted, or the original) and corrects `info`'s codec fields to match disk.
def _ensure_phone_safe(
job_id: int,
produced: Path,
out_dir: Path,
phase: str,
total_s: int | None = None,
*,
keep_source: bool = False,
) -> Path:
"""Bring a file that is about to become a ready library asset into the phone-safe shape
(`playable`), writing any converted copy into `out_dir`. Returns the file to store — `produced`
itself when it already is that shape, or isn't media at all — and removes `produced` once a
converted copy exists, unless `keep_source` (an upload's original, which must outlive the
conversion until the database points at the new file). The job shows `phase` with real ffmpeg
progress while it converts.
An audio download (mode "a") is never touched: yt-dlp already extracted it to .mp3 / .m4a, which
every browser plays as audio/mpeg / audio/mp4. Remuxing it into .mp4 served an mp3 as video/mp4,
which iPad's WebKit refuses to play (prod job 268)."""
if spec.get("mode") == "a" or spec.get("container") != "mp4" or spec.get("vcodec"):
Every ready path calls this — a download, an upload's worker conversion and an editor clip — so
a share link can only ever serve the one shape every phone and browser plays."""
p = playable.plan(probe.probe_json(produced), produced.suffix)
if p is None or not p.needs_change:
return produced
vcodec, acodec = editmod.probe_codecs(produced)
is_mp4 = produced.suffix.lower() == ".mp4"
needs, reenc_v, reenc_a = editmod.browser_transcode_flags(vcodec, acodec, is_mp4)
if not needs:
return produced
dest = produced.with_name(produced.stem + ".browser.mp4")
action = "re-encode" if (reenc_v or reenc_a) else "remux"
dest = out_dir / f"{produced.stem}.safe.{p.ext}"
action = "re-encode" if (p.reencode_video or p.reencode_audio) else "remux"
log.info(
"job %s: %s (v=%s a=%s mp4=%s) -> browser-safe mp4", job_id, action, vcodec, acodec, is_mp4
"job %s: %s (video=%s audio=%s) -> phone-safe .%s",
job_id,
action,
p.reencode_video,
p.reencode_audio,
p.ext,
extra=logs.event("download.phone_safe", job_id=job_id, action=action, ext=p.ext),
)
_set_job(job_id, phase="processing", progress=0, speed_bps=None, eta_s=None)
# Prefer the info duration; probe the file when the extractor didn't provide one, so a long
# re-encode still shows real progress instead of a frozen 0%.
total_s = int(info["duration"]) if info.get("duration") else editmod.probe_duration(produced)
_set_job(job_id, phase=phase, progress=0, speed_bps=None, eta_s=None)
# A re-encode with no known length would hold a frozen 0%; probe the file for a denominator.
editmod.run_ffmpeg(
editmod.build_browser_safe_cmd(
produced, dest, reenc_v, reenc_a, threads=editmod.encode_thread_cap()
),
total_s=total_s,
on_progress=_ffmpeg_progress_writer(job_id, "processing"),
playable.build_cmd(produced, dest, p, threads=editmod.encode_thread_cap()),
total_s=total_s or editmod.probe_duration(produced),
on_progress=_ffmpeg_progress_writer(job_id, phase),
should_cancel=_cancel_poll(job_id),
)
produced.unlink(missing_ok=True)
# Keep the stored asset metadata truthful about what's now on disk.
if reenc_v:
info["vcodec"] = "h264"
if reenc_a:
info["acodec"] = "aac"
if not dest.exists():
raise StatedFailure("ffmpeg produced no output file")
if not keep_source:
produced.unlink(missing_ok=True)
return dest
@@ -483,7 +519,7 @@ def _download(
urlguard.check_url(url)
except urlguard.BlockedUrl as e:
raise StatedFailure(f"Refusing to download from this address ({e.reason}).") from e
# S6: attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
# Attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
# this asset is that user's private, salted identity (the double gate lives in credentials).
remaining_bytes: int | None = None
with SessionLocal() as cdb:
@@ -495,8 +531,8 @@ def _download(
# SEC-13: what this user has left. `check_enqueue` only compared the footprint as it
# stood when the job was QUEUED — a download's size is unknown until it finishes, so a
# single job could previously blow through the cap by any amount. Handing the remainder
# to yt-dlp as `max_filesize` makes the format selector refuse an oversized file up
# front instead.
# to yt-dlp as `max_filesize` makes it refuse an oversized file (its HTTP downloader
# checks the announced size) instead.
remaining_bytes = quota.remaining_bytes(cdb, user_id)
# …and zero is a REFUSAL, not "no limit". Earlier jobs of this user may have filled the cap
# while this one sat in the queue, and 0 is not an expressible yt-dlp `max_filesize`, so
@@ -505,8 +541,8 @@ def _download(
# free space, which is what the `quota_exceeded` category tells them.
if remaining_bytes == 0:
raise _OwnerRefused(dlerrors.QUOTA_FULL_MESSAGE)
info = _media_info(
_extract_with_fallback(
with formats.watching_size_cap() as over_cap:
fetched = _extract_with_fallback(
job_id,
asset_id,
url,
@@ -516,13 +552,30 @@ def _download(
cookiefile=str(cookiefile) if cookiefile else None,
max_bytes=remaining_bytes,
)
)
info = _media_info(dict(fetched))
# yt-dlp refused a file larger than what is left: it returns with no file, which would
# otherwise read as a processing fault. The owner's storage, so the owner's refusal.
if over_cap[0]:
raise _OwnerRefused(dlerrors.DOWNLOAD_TOO_BIG_MESSAGE)
# Re-resolve now that yt-dlp's info reveals the true source (a bare/redirected url only
# identifies post-extraction) so the .nfo uniqueid type + uploader_url agree on one source.
adapter = adapters.resolve_adapter(source_kind, source_ref, info)
produced = _produced_file(info, staging)
produced = _ensure_browser_playable(job_id, produced, spec, info)
produced = _ensure_phone_safe(
job_id,
produced,
staging,
CONVERTING,
int(info["duration"]) if info.get("duration") else None,
)
# Held to the cap by its real size, as a clip is: yt-dlp's `max_filesize` only stops a plain
# HTTP file that announces its size, and YouTube's fragmented formats never meet it
# (measured: 718 MB fetched with 1 MB left). Read again — other jobs may have landed.
with SessionLocal() as qdb:
room = quota.remaining_bytes(qdb, user_id)
if room is not None and produced.stat().st_size > room:
raise _OwnerRefused(dlerrors.DOWNLOAD_TOO_BIG_MESSAGE)
ext = produced.suffix.lstrip(".").lower()
# Codecs/dims/duration come from a PROBE of the produced file — the ground truth — NOT yt-dlp's
# `info` dict, whose fields are absent for some sources (Facebook among them): deriving
@@ -541,19 +594,18 @@ def _download(
thumbnail_url=info.get("thumbnail"),
)
layout, retention_days = _download_settings()
layout = _download_layout()
rel = storage.rel_path(meta, ext, layout, asset_id=asset_id)
thumb = _find_thumbnail(staging, produced)
# Convert the raw thumbnail (webp/png/jpeg) to a real JPEG ourselves — robustly — for the
# poster + og:image sidecar, now that yt-dlp's fatal thumbnail convertor is gone. On failure
# this is None and write_sidecars writes no poster, so ensure_poster cuts a frame instead.
poster_jpg = editmod.thumbnail_to_jpg(thumb, staging / "poster.jpg")
storage.place_file(produced, settings.download_root, rel)
nfo_ok = storage.write_sidecars(settings.download_root, rel, meta, poster_jpg)
storage.place_file(produced, settings.files_root, rel)
nfo_ok = storage.write_sidecars(settings.files_root, rel, meta, poster_jpg)
final = storage.abs_path(settings.download_root, rel)
final = storage.abs_path(settings.files_root, rel)
size = final.stat().st_size if final.exists() else None
expires = datetime.now(timezone.utc) + timedelta(days=retention_days)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
@@ -579,12 +631,11 @@ def _download(
# (og:image) source — a remote thumbnail, esp. Facebook's signed CDN URL, expires
# and can't be relied on cross-origin. write_sidecars already wrote <base>.jpg from
# the thumbnail; ensure_poster returns that, or cuts a frame if there was none.
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
asset.source_webpage_url = info.get("webpage_url")
asset.nfo_written = nfo_ok
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = expires
job = db.get(DownloadJob, job_id)
# Capture the owner before commit expires the row (avoids a refresh SELECT just to notify).
owner_id = job.user_id if job else None
@@ -722,9 +773,19 @@ def _log_download_failed(job_id: int, msg: str, code: str | None, exc: BaseExcep
)
def _fail_edit(job_id: int, asset_id: int, msg: str, exc: BaseException | None = None) -> None:
"""Fail an edit job, and say so ONCE — every way an edit fails comes through here, so this is
where its line and its event code live. `exc` is the exception when there was one.
def _fail_edit(
job_id: int,
asset_id: int,
msg: str,
exc: BaseException | None = None,
*,
what: str = "edit",
diagnosis: dlerrors.Diagnosis | None = None,
) -> None:
"""Fail a local ffmpeg job — an edit, a clip, or an upload's conversion (`what="upload"`) — and
say so ONCE: every way one fails comes through here, so this is where its line and its event code
(`<what>.job.failed`) live. `exc` is the exception when there was one; `diagnosis` replaces the
plain category when the failure was a fetch the download catalog can name (a clip's).
Its traceback goes with the line unless the message already tells it all (`_failure_stack`).
Logged after the write, as on the download error path."""
@@ -739,19 +800,25 @@ def _fail_edit(job_id: int, asset_id: int, msg: str, exc: BaseException | None =
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
error_code=diagnosis.code
if diagnosis
else dlerrors.classify_conversion(msg)
if what == "upload"
else dlerrors.classify(msg),
error_detail=diagnosis.detail if diagnosis else None,
phase=None,
)
db.commit()
# `msg` is the user's copy; ffmpeg's stderr tail in it can name the source file, whose path is
# built from the channel and the title.
log.warning(
"edit job %s failed: %s",
"%s job %s failed: %s",
what,
job_id,
formats.log_safe(msg),
extra={
**logs.event(
"edit.job.failed",
f"{what}.job.failed",
job_id=job_id,
asset_id=asset_id,
**({"error_type": type(exc).__name__} if exc is not None else {}),
@@ -777,7 +844,7 @@ def _process_edit(job_id: int, asset_id: int) -> None:
if not src_ready or not src_rel:
_fail_edit(job_id, asset_id, "The source download is no longer available.")
return
src_path = storage.abs_path(settings.download_root, src_rel)
src_path = storage.abs_path(settings.files_root, src_rel)
if not src_path.exists():
_fail_edit(job_id, asset_id, "The source file is missing.")
return
@@ -811,13 +878,24 @@ def _process_edit(job_id: int, asset_id: int) -> None:
)
if not dest_staging.exists():
raise StatedFailure("ffmpeg produced no output file")
# A fast trim stream-copies the source's codecs and container, so the clip is only as
# playable as its source: hold it to the same rule as every other ready file.
dest_staging = _ensure_phone_safe(job_id, dest_staging, staging, "editing", total_s)
out_ext = dest_staging.suffix.lstrip(".").lower() or out_ext
# New storage on the owner's footprint, held to what is left by its real size, as a
# download and a player clip are.
if user_id is not None:
with SessionLocal() as qdb:
room = quota.remaining_bytes(qdb, user_id)
if room is not None and dest_staging.stat().st_size > room:
raise StatedFailure(dlerrors.CLIP_TOO_BIG_MESSAGE)
rel = storage.edit_rel_path(user_id, asset_id, out_ext)
storage.place_file(dest_staging, settings.download_root, rel)
final = storage.abs_path(settings.download_root, rel)
storage.place_file(dest_staging, settings.files_root, rel)
final = storage.abs_path(settings.files_root, rel)
size = final.stat().st_size if final.exists() else None
_, retention_days = _download_settings()
expires = datetime.now(timezone.utc) + timedelta(days=retention_days)
# The clip's real frame: a re-encode evens an odd crop or source (`editmod.EVEN_FRAME`).
_, _, out_w, out_h, _ = probe.probe_media(final)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
@@ -827,14 +905,15 @@ def _process_edit(job_id: int, asset_id: int) -> None:
asset.size_bytes = size
asset.container = out_ext
asset.duration_s = total_s or asset.duration_s
if crop:
if out_w and out_h:
asset.width, asset.height = out_w, out_h
elif crop:
asset.width = edit_spec["crop"]["w"]
asset.height = edit_spec["crop"]["h"]
else:
asset.width, asset.height = src_w, src_h
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = expires
job = db.get(DownloadJob, job_id)
# Capture owner + title before commit expires the rows (avoids a refresh SELECT).
owner_id = job.user_id if job else None
@@ -867,3 +946,388 @@ def _process_edit(job_id: int, asset_id: int) -> None:
_fail_edit(job_id, asset_id, str(exc)[:500], exc)
finally:
shutil.rmtree(staging, ignore_errors=True)
def _clip_room(user_id: int) -> int | None:
"""What the clip's owner may still store (`quota.remaining_bytes`), refusing at zero — see
`_download`: zero is a refusal, not "no limit"."""
with SessionLocal() as db:
remaining = quota.remaining_bytes(db, user_id)
if remaining == 0:
raise _OwnerRefused(dlerrors.QUOTA_FULL_MESSAGE)
return remaining
def _clip_source(
job_id: int, asset_id: int, spec: dict, staging: Path, remaining: int | None
) -> tuple[Path, float, str | None]:
"""The file a clip is cut from, where in it the clip starts, and the source's title when only
the fetch could tell it. A YouTube clip fetches ONLY its range into `staging` (so it starts at
0 there); a Plex clip reads the library file in place."""
if spec["source"] == "youtube":
url = service.source_url("youtube", spec["ref"])
cookiefile: Path | None = None
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
asset = db.get(MediaAsset, asset_id)
# Signed in when the user has a YouTube cookie, as their downloads are: YouTube gates
# single videos signed out ("not a bot"), and the clip is theirs alone.
if job and asset:
cookiefile = credentials.resolve_clip_cookiefile(
db, job.user_id, url, asset, spec, staging
)
_set_job(job_id, phase="downloading", progress=0)
try:
with formats.watching_size_cap() as over_cap:
info = _extract_with_fallback(
job_id,
asset_id,
url,
formats.normalize(clipmod.YOUTUBE_FETCH_SPEC),
staging,
adapters.resolve_adapter("youtube", spec["ref"]),
cookiefile=str(cookiefile) if cookiefile else None,
max_bytes=remaining,
section=(spec["start_s"], spec["end_s"]),
)
# Should yt-dlp's own cap ever refuse the range (see `_download`): the owner's storage.
if over_cap[0]:
raise _OwnerRefused(dlerrors.CLIP_TOO_BIG_MESSAGE)
except (_Aborted, _OwnerRefused):
raise
except Exception as exc: # re-raised below with the download catalog's diagnosis
msg = str(exc)[:500]
raise _ClipFetchFailed(
msg, dlerrors.diagnose(msg, url=url, cookie_attached=cookiefile is not None)
) from exc
finally:
# Never leave a plaintext session cookie on disk past the fetch (see `_download`).
if cookiefile is not None:
cookiefile.unlink(missing_ok=True)
info = _media_info(dict(info))
title = normalize_title(adapters.resolve_adapter("youtube", spec["ref"]).title(info))
return _produced_file(info, staging), 0.0, title
with SessionLocal() as db:
item = plex_lookups.by_rating_key(db, PlexItem, spec["ref"])
src = plex_paths.local_media_path(db, item.file_path) if item else None
if src is None:
raise StatedFailure("The library file is not reachable from here.")
return src, spec["start_s"], None
def _clip_crop(spec: dict, data: dict) -> str:
"""The clip's crop filter, from its frame and the picture the file really holds ("" for none)."""
frame = spec.get("crop")
if not frame:
return ""
dar = clipmod.display_aspect(data)
if dar is None:
raise StatedFailure("The picture's size could not be read, so the crop cannot be placed.")
crop = clipmod.crop_filter(frame, dar)
if crop is None:
raise StatedFailure("The crop frame lies outside the picture.")
return crop
def _process_clip(job_id: int, asset_id: int) -> None:
"""Cut an ad-hoc clip (downloads/clip): fetch or open its source, ONE re-encode to the clip
shape, then into the user's library beside their edits."""
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
if job is None:
return
spec = dict(job.edit_spec or {})
user_id = job.user_id
staging = _edit_staging_dir(asset_id)
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True, exist_ok=True)
# Refused before the fetch and the encode when the owner's storage is already full.
remaining = _clip_room(user_id)
src, start, fetched_title = _clip_source(job_id, asset_id, spec, staging, remaining)
data = probe.probe_json(src)
if not data:
raise StatedFailure("The source file could not be read.")
audio_track = spec.get("audio_track")
tracks = clipmod.audio_track_count(data)
if tracks == 0:
# A silent source: the player still names its track 0, and the clip is silent too.
audio_track = None
elif audio_track is not None and audio_track >= tracks:
raise StatedFailure("The chosen audio track is not in the file.")
crop = _clip_crop(spec, data)
total_s = clipmod.duration(spec)
out = staging / "clip.mp4"
_set_job(job_id, phase="editing", progress=0, speed_bps=None, eta_s=None)
editmod.run_ffmpeg(
clipmod.build_cmd(
src,
out,
start,
total_s,
audio_track=audio_track,
hdr=clipmod.is_hdr(data),
threads=editmod.encode_thread_cap(),
crop=crop,
),
total_s,
on_progress=_ffmpeg_progress_writer(job_id, "editing"),
should_cancel=_cancel_poll(job_id),
)
if not out.exists():
raise StatedFailure("ffmpeg produced no output file")
# The phone-safe shape by construction; still held to the rule every ready file meets.
out = _ensure_phone_safe(job_id, out, staging, "editing", round(total_s))
# Held to the cap by its real size: a range fetch passes yt-dlp's `max_filesize` to its
# ffmpeg downloader, which never reads it, and a Plex clip fetches nothing at all. Read
# again, since other jobs of this owner may have landed during the encode.
remaining = _clip_room(user_id)
if remaining is not None and out.stat().st_size > remaining:
raise _OwnerRefused(dlerrors.CLIP_TOO_BIG_MESSAGE)
ext = out.suffix.lstrip(".").lower() or "mp4"
rel = storage.edit_rel_path(user_id, asset_id, ext)
storage.place_file(out, settings.files_root, rel)
final = storage.abs_path(settings.files_root, rel)
vcodec, acodec, width, height, _ = probe.probe_media(final)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "ready"
asset.rel_path = rel
asset.size_bytes = final.stat().st_size if final.exists() else None
asset.container = ext
asset.vcodec, asset.acodec, asset.width, asset.height = (
vcodec,
acodec,
width,
height,
)
asset.duration_s = round(total_s)
asset.media_kind = mediakind.derive(vcodec, acodec)
# The clip's own frame as its poster: the link preview shows the moment, and a Plex
# item has no public thumbnail at all.
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
fields: dict = {"status": JobStatus.DONE, "progress": 100, "phase": None}
# A video outside the catalog was queued as "Clip (clip)": the fetch has its title now.
placeholder = service.clip_title(None, None)
if asset and fetched_title and asset.title == placeholder:
named = service.clip_title(fetched_title, None)
asset.title = named
if job_row := db.get(DownloadJob, job_id):
if job_row.display_name == placeholder:
fields["display_name"] = named[:DISPLAY_TEXT_MAX]
title = asset.title if asset else None
finalized = service.finalize_if_running(db, job_id, **fields)
db.commit()
if finalized:
_notify_download_done(db, user_id, title, job_id, kind="edit")
log.info(
"clip job %s done: asset %s",
job_id,
asset_id,
extra=logs.event(
"clip.job.done", job_id=job_id, asset_id=asset_id, source=spec["source"]
),
)
except (_Aborted, editmod.EditAborted):
# Paused/canceled mid-fetch or mid-encode: free the asset for a later retry.
_reset_asset_pending(asset_id)
log.info(
"clip job %s aborted (pause/cancel)",
job_id,
extra=logs.event("clip.job.aborted", job_id=job_id),
)
except _OwnerRefused as exc:
# Quota full: the job fails, the asset goes back to pending for a Retry once there is room
# (the download's handling; see `_download`).
msg = str(exc)[:500]
with SessionLocal() as db:
service.finalize_if_running(
db,
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
error_detail=None,
phase=None,
)
db.commit()
_reset_asset_pending(asset_id)
log.info(
"clip job %s refused for its owner: %s",
job_id,
formats.log_safe(msg),
extra=logs.event("clip.job.refused", job_id=job_id),
)
except _ClipFetchFailed as exc:
# silent-ok: not silent — `_fail_edit` is the report, with the fetch's diagnosis.
_fail_edit(job_id, asset_id, str(exc), exc, what="clip", diagnosis=exc.diagnosis)
except Exception as exc: # noqa: BLE001 — surface the failure, keep the worker alive
# silent-ok: not silent — `_fail_edit` is the report, and it takes `exc` for its stack.
_fail_edit(job_id, asset_id, str(exc)[:500], exc, what="clip")
finally:
shutil.rmtree(staging, ignore_errors=True)
def _check_converted_fits(owner_id: int | None, old_size: int, new_size: int) -> None:
"""Refuse a converted upload that would push its owner over the storage cap: the upload was
charged at its original size, and a re-encode can come out larger. The original stays, counted,
for the owner to delete or keep."""
if owner_id is None:
return
with SessionLocal() as db:
if not quota.fits_after_conversion(db, owner_id, old_size, new_size):
raise StatedFailure(dlerrors.CONVERTED_TOO_BIG_MESSAGE)
def _unplace(placed: Path | None) -> None:
"""Remove a converted upload that was placed at its new rel but never committed: the row still
names the original, so nothing else would ever find this file."""
if placed is not None:
placed.unlink(missing_ok=True)
def _conversion_landed(asset_id: int, new_rel: str) -> bool | None:
"""After the commit that makes a converted upload ready RAISED: did it land anyway? A commit can
reach the server and lose its answer (the connection drops after COMMIT). True/False by what the
row now says; None when the row cannot be read either."""
try:
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
return asset is not None and asset.status == "ready" and asset.rel_path == new_rel
except Exception: # noqa: BLE001 — "cannot tell" is the answer, the caller acts on it
# silent-ok: the caller re-raises the commit's own error, which `_fail_edit` reports.
return None
def _drop_original(job_id: int, src: Path) -> None:
"""Remove an upload's original once the converted file is committed in its place. Best effort:
the job is done and its file is ready, so a failure here must not fail it — the original is
named by no row now, and the GC's `.uploads` sweep takes it."""
try:
src.unlink(missing_ok=True)
except OSError as exc:
log.warning(
"upload job %s: original not removed, left to the GC sweep: %s",
job_id,
exc,
extra=logs.event("upload.original.unlink_failed", job_id=job_id),
)
def _free_unheld_upload(asset_id: int) -> bool:
"""Free an upload no job holds any more — cancelled or deleted while its conversion ran, when
`release_asset` had to leave it because the conversion still read its file. Once that has
stopped, free it as the release would have; left alone, its file would sit on disk counted by no
owner's footprint until the GC's upload pass. False when a job still holds it (paused)."""
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset is None or asset.ref_count != 0:
return False
if asset.rel_path:
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
db.commit()
return True
def _convert_upload(job_id: int, asset_id: int) -> None:
"""Convert a queued upload into the phone-safe shape (`playable`) and make it ready.
The original sits at the asset's `rel_path` (see `upload`); the converted file replaces it there
under the new extension, and only then is the original removed — so a failed or stopped
conversion leaves the original in place for Retry, and a delete still finds it by `rel_path`.
A converted file placed but never committed is removed again on the way out. When the commit's
own outcome is unknown neither file is removed: the row names one of them, and the GC's `.uploads`
sweep takes the one it does not — as it does an original whose unlink after the commit failed."""
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
old_rel = asset.rel_path if asset else None
old_ext = asset.container if asset else None
old_size = (asset.size_bytes or 0) if asset else 0
duration = asset.duration_s if asset else None
job = db.get(DownloadJob, job_id)
owner_id = job.user_id if job else None
src = storage.safe_abs_path(settings.files_root, old_rel) if old_rel else None
if old_rel is None or src is None or not src.exists():
_fail_edit(job_id, asset_id, dlerrors.UPLOAD_MISSING_MESSAGE, what="upload")
return
staging = _staging_dir(asset_id)
placed: Path | None = None # the converted file while it is at its new rel, uncommitted
try:
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True, exist_ok=True)
safe = _ensure_phone_safe(job_id, src, staging, CONVERTING, duration, keep_source=True)
ext = safe.suffix.lstrip(".").lower()
new_rel = storage.with_ext(old_rel, old_ext, ext)
if safe != src:
_check_converted_fits(owner_id, old_size, safe.stat().st_size)
storage.place_file(safe, settings.files_root, new_rel)
if new_rel != old_rel:
placed = storage.abs_path(settings.files_root, new_rel)
final = storage.abs_path(settings.files_root, new_rel)
vcodec, acodec, width, height, dur = probe.probe_media(final)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "ready"
asset.rel_path = new_rel
asset.container = ext
asset.size_bytes = final.stat().st_size
asset.vcodec, asset.acodec = vcodec, acodec
asset.width, asset.height = width or asset.width, height or asset.height
asset.duration_s = dur or asset.duration_s
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
title = asset.title if asset else None
finalized = service.finalize_if_running(
db, job_id, status=JobStatus.DONE, progress=100, phase=None
)
try:
db.commit()
except Exception:
landed = _conversion_landed(asset_id, new_rel)
if landed is False:
raise # rolled back: the original is still the asset's file — fail as any step
# Landed, or cannot tell: the converted file may be the asset's file now, so it is
# never removed blind (the sweep takes whichever file no row names).
placed = None
if landed is None:
raise
db.rollback()
placed = None
if finalized:
_notify_download_done(db, owner_id, title, job_id, kind="upload")
if new_rel != old_rel:
_drop_original(job_id, src)
if not finalized:
_free_unheld_upload(asset_id) # finished after its job was cancelled or deleted
if finalized:
log.info(
"upload job %s converted: asset %s",
job_id,
asset_id,
extra=logs.event("upload.job.done", job_id=job_id, asset_id=asset_id, ext=ext),
)
except editmod.EditAborted:
# Paused/canceled mid-encode: the original is untouched. A paused job keeps it for its
# resume; one cancelled or deleted holds it no more, so it is freed now.
_unplace(placed)
if not _free_unheld_upload(asset_id):
_reset_asset_pending(asset_id)
log.info(
"upload job %s conversion aborted (pause/cancel)",
job_id,
extra=logs.event("upload.job.aborted", job_id=job_id),
)
except Exception as exc: # noqa: BLE001 — surface the failure, keep the worker alive
# silent-ok: not silent — `_fail_edit` is the report, and it takes `exc` for its stack.
_unplace(placed)
_fail_edit(job_id, asset_id, str(exc)[:500], exc, what="upload")
finally:
shutil.rmtree(staging, ignore_errors=True)
+103
View File
@@ -0,0 +1,103 @@
"""The phone-safe rule: what a library file must be so it plays wherever a share link lands.
A shared `/watch` link is opened by whoever got it, on whatever they hold: an iPhone's WebKit, an
Android Chrome, a desktop Firefox. The one shape every one of them plays:
* video → H.264, 8-bit 4:2:0 (`yuv420p`, or its full-range twin `yuvj420p`), AAC audio, in .mp4
* audio only → MP3 in .mp3, or AAC in .m4a
Measured on the owner's iPhone + iPad (2026-10-05): AV1, Matroska, HEVC tagged `hev1` and MP3
inside an .mp4 all fail there. VP9/WebM, HEVC `hvc1` and 10-bit H.264 played on those devices, but
Chrome and Firefox do not decode HEVC and browsers decode 10-bit / 4:4:4 H.264 only patchily, so the
rule is the shape every browser plays, not the shape one device plays.
Every file that becomes a ready library asset passes through here — a download, an upload (the
worker converts it, see `upload`), and an editor clip — so a link can only ever serve this shape.
`plan` says what (if anything) must change; `build_cmd` is the one ffmpeg command that changes it:
re-encode only the stream that fails, copy the rest, or just remux when only the container is wrong.
"""
from dataclasses import dataclass
from pathlib import Path
from app.downloads import edit as editmod
from app.downloads import probe
_SAFE_VCODEC = "h264"
_SAFE_PIX_FMTS = frozenset({"yuv420p", "yuvj420p"})
_VIDEO_EXT = "mp4"
# An audio-only file: the codec, and the extension it plays in. Anything else becomes AAC in .m4a.
_AUDIO_HOME = {"mp3": "mp3", "aac": "m4a"}
_AUDIO_FALLBACK_EXT = "m4a"
@dataclass(frozen=True)
class Plan:
"""What makes one file phone-safe. `ext` is the extension the safe file has (no dot)."""
ext: str
has_video: bool
reencode_video: bool
reencode_audio: bool
needs_change: bool
def plan(data: dict | None, ext: str) -> Plan | None:
"""The phone-safe plan for a file whose `probe.probe_json` is `data` and whose extension is
`ext`; None when it is not media at all (an unreadable probe, a still image, no audio or video
stream) — such a file is a download-only 'file' and never plays inline, so there is nothing to
make safe."""
if data is None:
return None
video, audio = probe.primary_streams(data)
ext = ext.lower().lstrip(".")
if video is not None:
rv = (video.get("codec_name") or "").lower() != _SAFE_VCODEC or (
video.get("pix_fmt") or ""
).lower() not in _SAFE_PIX_FMTS
ra = audio is not None and (audio.get("codec_name") or "").lower() != "aac"
return Plan(_VIDEO_EXT, True, rv, ra, rv or ra or ext != _VIDEO_EXT)
if audio is not None:
home = _AUDIO_HOME.get((audio.get("codec_name") or "").lower())
target = home or _AUDIO_FALLBACK_EXT
return Plan(target, False, False, home is None, home is None or ext != target)
return None
def build_cmd(src: Path, dest: Path, p: Plan, threads: int | None = None) -> list[str]:
"""The ffmpeg command that writes `src` to `dest` (whose suffix is `p.ext`) in the phone-safe
shape: the failing stream(s) re-encoded with the package's shared encoder settings, the rest
stream-copied. An audio-only result drops any cover-art stream. `-progress pipe:1`
streams machine-readable progress; `threads` caps the encoder so a re-encode cannot claim every
core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
*editmod.INPUT_GUARD,
"-i",
str(src),
]
# Exactly the two streams `plan` judged: `V` is a video stream that is not cover art. Nothing
# else rides along — a subtitle or data track the .mp4 muxer cannot take would fail the run.
if p.has_video:
cmd += ["-map", "0:V:0", "-map", "0:a:0?"]
if p.reencode_video:
cmd += ["-vf", editmod.EVEN_FRAME, *editmod.X264_ARGS]
else:
cmd += ["-c:v", "copy"]
else:
cmd += ["-map", "0:a:0"]
cmd += editmod.AAC_ARGS if p.reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
if p.ext in ("mp4", "m4a"):
cmd += ["-movflags", "+faststart"]
cmd += [str(dest)]
return cmd
+107
View File
@@ -0,0 +1,107 @@
"""What a library file previews as — the single home of the preview allow-list.
A generic file (`media_kind == "file"`; video and audio have their players) previews by its stored
`container`, the upload's cleaned extension, and ONLY through this table: an image is shown as an
image, a PDF in the browser's own viewer, and a text format as plain text. Anything else has no
preview. The served Content-Type comes from here too, never from a guess on the name, and HTML / SVG
sit in the TEXT list on purpose: rendered on this origin, an uploaded page would run with the
reader's session. The route adds `CSP: sandbox` on top, so even a mislabelled file (a "png" that is
really a page) gets no script and no origin.
The serializer hands the kind to the client (`asset.preview`), so the library and the server can
never disagree on which files open a preview.
"""
import codecs
from typing import Literal
from app.downloads import mediakind
PreviewKind = Literal["image", "pdf", "text"]
IMAGE_TYPES: dict[str, str] = {
"png": "image/png",
"jpg": "image/jpeg",
"jpeg": "image/jpeg",
"gif": "image/gif",
"webp": "image/webp",
"avif": "image/avif",
"bmp": "image/bmp",
}
PDF_TYPE = "application/pdf"
# Read as text: notes, data, subtitles, config and source. HTML and SVG belong here (see above).
TEXT_EXTS: frozenset[str] = frozenset(
{
"txt", "text", "md", "markdown", "log", "csv", "tsv", "json", "jsonl", "ndjson", "xml",
"yaml", "yml", "toml", "ini", "conf", "cfg", "properties", "env", "srt", "vtt", "ass", "ssa",
"sub", "nfo", "cue", "m3u", "m3u8", "html", "htm", "xhtml", "svg", "css", "scss", "less",
"js", "mjs", "cjs", "ts", "tsx", "jsx", "py", "rb", "php", "pl", "lua", "go", "rs", "java",
"kt", "kts", "swift", "c", "h", "cc", "cpp", "hpp", "cs", "sh", "bash", "zsh", "fish", "ps1",
"bat", "cmd", "sql", "r", "dart", "diff", "patch", "tex", "rst", "adoc", "gradle", "dockerfile",
}
) # fmt: skip
TEXT_TYPE = "text/plain; charset=utf-8"
# A text preview is the file's head: enough to read, never a whole log loaded into a tab.
TEXT_PREVIEW_BYTES = 1 << 20
# A NUL byte in the first this-many bytes means the "text" is not text.
BINARY_SNIFF_BYTES = 8192
def kind(media_kind: str | None, container: str | None) -> PreviewKind | None:
"""The preview a library file gets, or None (a media file plays instead; others download)."""
if media_kind != mediakind.FILE:
return None
ext = (container or "").lower()
if ext in IMAGE_TYPES:
return "image"
if ext == "pdf":
return "pdf"
if ext in TEXT_EXTS:
return "text"
return None
def media_type(preview: PreviewKind, container: str | None) -> str:
"""The Content-Type a preview is served with: from this table, never guessed from the name."""
if preview == "image":
return IMAGE_TYPES[(container or "").lower()]
if preview == "pdf":
return PDF_TYPE
return TEXT_TYPE
# A text that is not UTF-8 is read as Central European Windows: what an old Hungarian subtitle or
# note is written in (its ő and ű are not in Latin-1).
FALLBACK_ENCODING = "cp1250"
_BOMS = (
(codecs.BOM_UTF8, "utf-8-sig"),
(codecs.BOM_UTF16_LE, "utf-16"),
(codecs.BOM_UTF16_BE, "utf-16"),
)
def looks_binary(head: bytes) -> bool:
"""Whether the start of a "text" file is binary after all (a NUL in it)."""
return b"\x00" in head[:BINARY_SNIFF_BYTES]
def text_utf8(head: bytes) -> bytes | None:
"""A text file's head as UTF-8, the one charset the preview is served in; None when it is not text.
A byte-order mark names its encoding (UTF-16 is full of NULs, so it is read before the binary
check); else UTF-8 is kept as it is — a character cut at the size cap included, the reader shows
that one as a replacement mark — and anything else is read as `FALLBACK_ENCODING`."""
for bom, encoding in _BOMS:
if head.startswith(bom):
return head.decode(encoding, errors="replace").encode()
if looks_binary(head):
return None
try:
head.decode("utf-8")
except UnicodeDecodeError as e: # silent-ok: not UTF-8 is an answer — read it as the fallback
if not (e.reason == "unexpected end of data" and e.end == len(head)):
return head.decode(FALLBACK_ENCODING, errors="replace").encode()
return head
+52 -25
View File
@@ -34,6 +34,28 @@ _IMAGE_FORMATS = {
}
def primary_streams(data: dict) -> tuple[dict | None, dict | None]:
"""(video stream, audio stream) — the first of each kind — from a NON-None ffprobe JSON dict, by
the same reading the stored facts use: a still image / image-sequence container has neither (it
is a generic 'file'), and a video stream flagged `attached_pic` (an mp3/m4a's cover art) is not a
video track. The phone-safe rule (`playable`) judges exactly these two streams."""
fmt_names = {
n.strip() for n in ((data.get("format") or {}).get("format_name") or "").split(",")
}
if fmt_names & _IMAGE_FORMATS:
return None, None
video = audio = None
for stream in data.get("streams", []):
kind = stream.get("codec_type")
if kind == "video" and video is None:
if (stream.get("disposition") or {}).get("attached_pic"):
continue # cover art, not a video track
video = stream
elif kind == "audio" and audio is None:
audio = stream
return video, audio
def _facts_from_probe(
data: dict,
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
@@ -45,19 +67,11 @@ def _facts_from_probe(
fmt_names = {n.strip() for n in (fmt.get("format_name") or "").split(",")}
if fmt_names & _IMAGE_FORMATS:
return None, None, None, None, None
vcodec = acodec = None
width = height = None
for stream in data.get("streams", []):
kind = stream.get("codec_type")
if kind == "video" and vcodec is None:
if (stream.get("disposition") or {}).get("attached_pic"):
continue # cover art, not a video track
vcodec = stream.get("codec_name")
width = stream.get("width")
height = stream.get("height")
elif kind == "audio" and acodec is None:
acodec = stream.get("codec_name")
video, audio = primary_streams(data)
vcodec = video.get("codec_name") if video else None
width = video.get("width") if video else None
height = video.get("height") if video else None
acodec = audio.get("codec_name") if audio else None
duration = None
try:
@@ -71,16 +85,29 @@ def _facts_from_probe(
return vcodec, acodec, width, height, duration
def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) from one ffprobe call; all None if ffprobe fails
or the file isn't playable media (an unprobeable file — or a still image — is treated as a generic
non-media file, so it becomes a download-only 'file' rather than a broken 'video')."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
def probe_json(path: Path) -> dict | None:
"""The full ffprobe JSON (streams + format) of `path`, or None when ffprobe could not read it —
the one call both the stored facts (`facts`) and the phone-safe rule read from."""
return editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
def facts(
data: dict | None,
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) of a `probe_json` result; all None when it is None
(an unprobeable file is treated as a generic non-media file, so it becomes a download-only 'file'
rather than a broken 'video')."""
if data is None:
return None, None, None, None, None
return _facts_from_probe(data)
def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) from one ffprobe call; all None if ffprobe fails
or the file isn't playable media (a still image is a generic 'file', see `primary_streams`)."""
return facts(probe_json(path))
def resolve_media_facts(
path: Path, info: dict
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
@@ -94,18 +121,18 @@ def resolve_media_facts(
is authoritative — INCLUDING a deliberate None (cover-art-only audio, a still image). A per-field
`probe or info` fallback would let `info`'s stale vcodec resurrect a video track the file doesn't
have (yt-dlp reports a progressive `vcodec` for an audio-mode download whose video it then strips),
mislabelling an audio download as a broken in-app 'video'. Only when ffprobe could NOT read the
file at all do we fall back to `info` (the pre-probe behaviour, a safety net); the cosmetic
dimensions and the duration still gap-fill from `info` in either case."""
mislabelling an audio download as a broken in-app 'video'. When ffprobe could NOT read the file
at all, no codec is reported: the probe runs with the media demuxer whitelist, so a file it
cannot open is one the player and the editor cannot open either — and taking `info`'s codecs
for it would make it an editable 'video' whose bytes are whatever the remote site served. The
cosmetic dimensions and the duration still gap-fill from `info` in either case."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
info_v = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
info_a = info.get("acodec") if info.get("acodec") not in (None, "none") else None
info_w = int(info["width"]) if info.get("width") else None
info_h = int(info["height"]) if info.get("height") else None
info_d = int(info["duration"]) if info.get("duration") else None
if data is None:
# ffprobe could not read the file — fall back to the extractor's info entirely.
return info_v, info_a, info_w, info_h, info_d
# Unreadable to the guarded probe: a download-only 'file' (no codecs → media_kind FILE).
return None, None, info_w, info_h, info_d
pv, pa, pw, ph, pd = _facts_from_probe(data)
# Probe ran → its codecs win outright (a None means "no such track", not "unknown"); info only
# gap-fills the width/height/duration the probe couldn't read.
+39 -10
View File
@@ -1,7 +1,9 @@
"""Per-user download limits + footprint accounting.
A user's *footprint* is the total size of the ready assets their library references (cache hits
count — it's "how much disk you're responsible for", not "how much you personally downloaded").
count — it's "how much disk you're responsible for", not "how much you personally downloaded"). An
upload counts from the moment it lands, ready or not: its original is on disk while its conversion
waits, and stays there when the conversion fails (for Retry) — bytes a ready-only sum never saw.
Limits come from a per-user DownloadQuota row if the admin set one, else the sysconfig defaults.
`unlimited` (e.g. the admin's own account) bypasses the byte cap.
@@ -13,21 +15,30 @@ the footprint as it stood when the job was created and earlier jobs may have fil
That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user each read the same
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
is unknown until it finishes. Edit jobs do not consult it at all (tracked as SEC-19).
is unknown until it finishes. CLIP jobs consult it before their fetch or encode and again with
their output's real size before it is placed. Edit jobs do not consult it at all (tracked as SEC-19).
"""
from dataclasses import dataclass
from sqlalchemy import func, select
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobKind, JobStatus
from app.models import DownloadJob, DownloadQuota, MediaAsset
# Jobs a user is considered to be "holding" (occupy a job slot / footprint). Terminal failures
# (error, canceled) don't count.
_HOLDING = ("queued", "running", "paused", "done")
_RUNNING = ("queued", "running")
_HOLDING = (JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED, JobStatus.DONE)
_RUNNING = (JobStatus.QUEUED, JobStatus.RUNNING)
# An asset whose file is on disk: a ready one, or an upload in any state (its original lands at
# ingest, see the module docstring). Every byte sum — a user's footprint, the GC's total-cap pass,
# the admin's storage view — reads it, so they all count the same disk.
ON_DISK = or_(
MediaAsset.status == "ready",
and_(MediaAsset.source_kind == "upload", MediaAsset.rel_path.is_not(None)),
)
@dataclass
@@ -56,7 +67,8 @@ def resolve(db: Session, user_id: int) -> QuotaLimits:
# "at limit" checks — the worker's claim-window exclusion vs at_concurrency_limit — since a
# 0-limit user is "capped" with zero running jobs, which the SQL exclusion can't see. Keep
# them consistent by never letting the effective limit drop below 1. (To BLOCK a user's
# downloads, suspend them or set max_jobs=0, not max_concurrent=0.)
# downloads, suspend them — the worker leaves a suspended user's queued jobs unclaimed — or
# set max_jobs=0, not max_concurrent=0.)
return QuotaLimits(row.max_bytes, max(1, row.max_concurrent), row.max_jobs, row.unlimited)
return QuotaLimits(
max_bytes=sysconfig.get_int(db, "download_default_max_bytes"),
@@ -67,12 +79,17 @@ def resolve(db: Session, user_id: int) -> QuotaLimits:
def footprint(db: Session, user_id: int) -> int:
"""Total bytes of the distinct ready assets this user's active jobs reference."""
"""Total bytes of the distinct assets this user's jobs hold on disk: the ready ones their active
jobs reference, and every upload of theirs — including one whose conversion failed, which keeps
its original (see the module docstring)."""
held_assets = (
select(DownloadJob.asset_id)
.where(
DownloadJob.user_id == user_id,
DownloadJob.status.in_(_HOLDING),
or_(
DownloadJob.status.in_(_HOLDING),
and_(DownloadJob.job_kind == JobKind.UPLOAD, DownloadJob.status == JobStatus.ERROR),
),
DownloadJob.asset_id.is_not(None),
)
.distinct()
@@ -80,7 +97,7 @@ def footprint(db: Session, user_id: int) -> int:
return int(
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(
MediaAsset.id.in_(held_assets), MediaAsset.status == "ready"
MediaAsset.id.in_(held_assets), ON_DISK
)
).scalar()
or 0
@@ -111,7 +128,7 @@ def at_concurrency_limit(db: Session, user_id: int) -> bool:
lim = resolve(db, user_id)
if lim.unlimited:
return False
running = _count(db, user_id, ("running",))
running = _count(db, user_id, (JobStatus.RUNNING,))
return running >= lim.max_concurrent
@@ -160,6 +177,18 @@ def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, projected)
def fits_after_conversion(db: Session, user_id: int, old_bytes: int, new_bytes: int) -> bool:
"""Whether an upload converted from `old_bytes` (what it is charged at now) to `new_bytes` keeps
its owner within the cap. A re-encode can come out larger than the original, and nothing else
re-checks it once the upload was let in."""
if new_bytes <= old_bytes: # a smaller file never breaks a cap the larger one was let in under
return True
lim = resolve(db, user_id)
if lim.unlimited:
return True
return footprint(db, user_id) - old_bytes + new_bytes <= lim.max_bytes
def usage(db: Session, user_id: int) -> dict:
"""Compact usage snapshot for the UI (footprint + counts vs limits)."""
lim = resolve(db, user_id)
+7 -2
View File
@@ -22,6 +22,7 @@ from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import formats
from app.downloads.status import JobKind
from app.models import DownloadJob
log = logs.get_logger(__name__)
@@ -95,12 +96,16 @@ def _run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id) ->
(`cause=child_exit`)."""
# Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
# real work, and to keep the import graph obvious: runner is a thin entry over the pipeline.
from app.downloads.pipeline import _download, _process_edit
from app.downloads.pipeline import _convert_upload, _download, _process_clip, _process_edit
with formats.hiding(source_ref):
try:
if job_kind == "edit":
if job_kind == JobKind.EDIT:
_process_edit(job_id, asset_id)
elif job_kind == JobKind.UPLOAD:
_convert_upload(job_id, asset_id)
elif job_kind == JobKind.CLIP:
_process_clip(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
except Exception as exc: # noqa: BLE001 — whatever it is, it is reported here or raw
+194 -18
View File
@@ -16,11 +16,22 @@ from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import adapters
from app.downloads import clip as clipmod
from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobStatus
from app.models import DISPLAY_TEXT_MAX, Channel, DownloadJob, MediaAsset, Video
from app.downloads import folders, formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobKind, JobStatus
from app.models import (
DISPLAY_TEXT_MAX,
Channel,
DownloadJob,
MediaAsset,
PlexItem,
PlexShow,
Video,
)
from app.plex import lookups as plex_lookups
from app.plex import paths as plex_paths
class EditError(Exception):
@@ -38,12 +49,15 @@ def source_url(source_kind: str, source_ref: str) -> str:
def reference_url(job, asset) -> str | None:
"""The clean "downloaded from" URL for a job — shared by the private Downloads page and the public
"""The clean "downloaded from" URL for a job — shared by the Files library and the public
watch page so both surfaces always agree. None for edit derivatives (a clip's source is the user's
own earlier download, not a web page); else the canonical page URL yt-dlp recorded; else one derived
from source_kind+source_ref (covers queued/older rows before the worker fills it)."""
if job is None or job.job_kind == "edit":
own earlier download, not a web page) and for a Plex clip (the library is not a public page); a
YouTube clip links its video at the clip's start; else the canonical page URL yt-dlp recorded; else
one derived from source_kind+source_ref (covers queued/older rows before the worker fills it)."""
if job is None or job.job_kind == JobKind.EDIT:
return None
if job.job_kind == JobKind.CLIP:
return clipmod.reference_url(job.edit_spec or {})
if asset is not None and asset.source_webpage_url:
return asset.source_webpage_url
if job.source_kind in ("youtube", "url"):
@@ -51,6 +65,23 @@ def reference_url(job, asset) -> str | None:
return None
def library_left_behind(db: Session, files_root: str) -> bool:
"""True when the library names files and none of the newest of them is under `files_root` — a
fresh, empty volume mounted in the library's place (a compose file changed under an install, its
old volume left behind). The mount check cannot see that: the new volume is a real mount."""
rels = (
db.execute(
select(MediaAsset.rel_path)
.where(MediaAsset.status == "ready", MediaAsset.rel_path.is_not(None))
.order_by(MediaAsset.id.desc())
.limit(20)
)
.scalars()
.all()
)
return bool(rels) and not any(storage.abs_path(files_root, r).exists() for r in rels if r)
def get_or_create_asset(
db: Session, source_kind: str, source_ref: str, format_sig: str
) -> MediaAsset:
@@ -63,7 +94,12 @@ def get_or_create_asset(
MediaAsset.source_ref == source_ref,
MediaAsset.format_sig == format_sig,
)
asset = db.execute(stmt).scalar_one_or_none()
# Locked to the caller's commit: a release or the GC freeing this asset meanwhile (they lock it
# too) either waits for the new hold, or has deleted the row by the time this read returns — and
# the caller then gets a fresh one, never a job pointing at a row that is gone.
asset = db.execute(
stmt.with_for_update().execution_options(populate_existing=True)
).scalar_one_or_none()
if asset is not None:
# A previously-FAILED shared asset must be re-attempted for the new job about to hold it —
# reset it to pending (+clear the error) so the worker actually re-downloads instead of
@@ -180,16 +216,18 @@ def enqueue(
profile_snapshot=spec,
display_name=display_name
or (asset.title[:DISPLAY_TEXT_MAX] if asset.status == "ready" and asset.title else None),
status="queued",
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
# Cache hit: the file already exists — no worker round-trip needed.
if asset.status == "ready":
job.status = "done"
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
@@ -197,7 +235,7 @@ def enqueue(
return job
def _clip_title(source_title: str | None, display_name: str | None) -> str:
def clip_title(source_title: str | None, display_name: str | None) -> str:
# Clipped to DISPLAY_TEXT_MAX, not to a literal: this feeds `DownloadJob.display_name` through
# `enqueue_edit`, the same column `display_text` guards. A machine-made title, so it is clipped
# rather than run through `display_text` — blank here means "no title", not None.
@@ -239,7 +277,7 @@ def enqueue_edit(
ref = f"{user_id}:{src_asset.id}"[:512]
asset = get_or_create_asset(db, "edit", ref, sig)
if not asset.title:
asset.title = _clip_title(src_asset.title, display_name)
asset.title = clip_title(src_asset.title, display_name)
asset.uploader = src_asset.uploader
asset.thumbnail_url = src_asset.thumbnail_url
asset.upload_date = src_asset.upload_date
@@ -251,7 +289,7 @@ def enqueue_edit(
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind="edit",
job_kind=JobKind.EDIT,
source_kind="edit",
source_ref=ref,
source_job_id=source_job.id,
@@ -259,15 +297,106 @@ def enqueue_edit(
edit_spec=spec,
profile_snapshot={},
display_name=display_name or asset.title[:DISPLAY_TEXT_MAX],
status="queued",
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
if asset.status == "ready": # identical clip already produced for this user → instant
job.status = "done"
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
return job
def _clip_source_facts(db: Session, spec: dict) -> dict:
"""Title, uploader, thumbnail and length of what a clip is cut from, read from our own catalog.
Raises ClipError when a Plex item is unknown or its file cannot be read here — said at enqueue,
rather than as a job that fails in the queue a minute later."""
if spec["source"] == "youtube":
video = db.get(Video, spec["ref"])
if video is None: # not in the catalog: titled "Clip" until yt-dlp names it (worker)
return {"title": None, "uploader": None, "thumbnail_url": None, "length_s": None}
channel = db.get(Channel, video.channel_id)
return {
"title": video.title,
"uploader": channel.title if channel else None,
"thumbnail_url": video.thumbnail_url,
"length_s": video.duration_seconds,
}
item = plex_lookups.by_rating_key(db, PlexItem, spec["ref"])
if item is None:
raise clipmod.ClipError("not_found")
if plex_paths.local_media_path(db, item.file_path) is None:
raise clipmod.ClipError("media_missing")
title = item.title
show = db.get(PlexShow, item.show_id) if item.show_id else None
if show is not None:
episode = (
f"S{item.season_number}·E{item.episode_number}"
if item.season_number is not None and item.episode_number is not None
else None
)
title = " — ".join(p for p in (show.title, episode, item.title) if p)
return {
"title": title,
"uploader": None,
# Plex art is served through our own authenticated routes, never a public URL; the worker
# cuts the clip's poster from the clip itself.
"thumbnail_url": None,
"length_s": item.duration_s,
}
def enqueue_clip(
db: Session, user_id: int, spec: dict, display_name: str | None = None
) -> DownloadJob:
"""Queue an ad-hoc clip (`clipmod.normalize`d spec) for `user_id`: a NEW per-user asset cut
straight from the source — a YouTube range or a Plex library file (see downloads/clip).
Raises ClipError (unknown or unreachable source, a range past its end) or quota.QuotaExceeded."""
facts = _clip_source_facts(db, spec)
# One second of grace: a mark set on the last frame may round past the catalog's whole seconds.
if facts["length_s"] and spec["end_s"] > facts["length_s"] + 1:
raise clipmod.ClipError("past_end")
quota.check_enqueue(db, user_id)
ref = clipmod.asset_ref(user_id, spec)[:512]
asset = get_or_create_asset(db, "clip", ref, clipmod.sig(spec))
if not asset.title:
asset.title = clip_title(facts["title"], display_name)
asset.uploader = facts["uploader"]
asset.thumbnail_url = facts["thumbnail_url"]
asset.duration_s = round(clipmod.duration(spec))
asset.media_kind = mediakind.VIDEO
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind=JobKind.CLIP,
source_kind="clip",
source_ref=ref,
edit_spec=spec,
profile_snapshot={},
display_name=display_name or asset.title[:DISPLAY_TEXT_MAX],
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
if asset.status == "ready": # the same range was already cut for this user → instant
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
@@ -291,6 +420,8 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
if not fields:
return False
db.flush() # asset-then-job lock order: push pending asset writes before we lock the job row
if fields.get("status") == JobStatus.DONE:
return _finalize_done(db, job_id, {**fields, "ready_at": datetime.now(timezone.utc)})
# A Core UPDATE with the guard as a WHERE — same atomic "only while still running" write as the
# rest of the codebase's 2.0 statements, no hand-built SQL string. synchronize_session=False: this
# is a terminal write, so we don't pay to sync the (about-to-be-discarded) ORM identity map. Bound
@@ -304,6 +435,41 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
return result.rowcount > 0
def _finalize_done(db: Session, job_id: int, fields: dict) -> bool:
"""`finalize_if_running` for a job that finished: done is when a file's retention starts, its
folder's unless the user kept it. The expiry is written only while the job is still where it was
read from — a move, a keep, or its folder's delete (which sends its files home) landing while the
folder's retention was read sends the read round again, never stamping a folder's retention on a
file now in another one."""
while True:
held = db.execute(
select(DownloadJob.folder_id, DownloadJob.kept, DownloadJob.status).where(
DownloadJob.id == job_id
)
).one_or_none()
if held is None or held.status != JobStatus.RUNNING:
return False
stamped = {
**fields,
"expires_at": folders.done_expiry(db, held.folder_id, held.kept),
"expiry_warned": False,
}
hit = db.execute(
update(DownloadJob)
.where(
DownloadJob.id == job_id,
DownloadJob.status == JobStatus.RUNNING.value,
DownloadJob.folder_id.is_not_distinct_from(held.folder_id),
DownloadJob.kept == held.kept,
)
.values(**stamped)
.returning(DownloadJob.id)
.execution_options(synchronize_session=False)
).scalar_one_or_none()
if hit is not None:
return True
def release_asset(db: Session, job: DownloadJob) -> None:
"""Drop this job's hold on its asset when it leaves a holding state. Once NO job holds the
asset anymore, delete the file + row immediately — a deleted download should free its disk,
@@ -318,16 +484,24 @@ def release_asset(db: Session, job: DownloadJob) -> None:
re-enqueuing the same (source,format) — get_or_create_asset would reset that row to `pending` and
+1 it, and deleting it under that would FK-null the new job's asset (a lost update on ref_count).
A ready asset is still freed at ref==0 (its file is the point); an orphaned errored row is cheap
(no file) and gets reused+reset by the next enqueue, or reclaimed by a later GC pass."""
(no file) and gets reused+reset by the next enqueue, or reclaimed by a later GC pass.
An UPLOAD is the exception: it is never shared or re-enqueued (its ref is unique), and while it
waits for or failed its phone-safe conversion the original file already sits at `rel_path` — so
it is freed at ref==0 whatever its state, except mid-conversion (`downloading`: the worker child
still reads it; the GC upload pass takes it once the child has stopped)."""
if not job.asset_id or job.status not in HOLDING:
return
asset = db.get(MediaAsset, job.asset_id)
if asset is None:
return
new_count = decr_ref_count(db, asset.id)
if new_count == 0 and asset.status == "ready":
freeable = asset.status == "ready" or (
asset.source_kind == "upload" and asset.status != "downloading"
)
if new_count == 0 and freeable:
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
@@ -456,4 +630,6 @@ def redownload(db: Session, job: DownloadJob) -> None:
# name would survive the rebuild).
job.display_name = None
job.queue_pos = next_queue_pos(db) # tail of the queue, not its stale original slot
job.expires_at = None # stamped again when the rebuild is done
job.expiry_warned = False
db.commit()
+21
View File
@@ -33,6 +33,27 @@ HOLDING = frozenset(
{JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED, JobStatus.DONE, JobStatus.ERROR}
)
class JobKind(StrEnum):
"""What a job produces, which decides the worker path that runs it and what the library may do
with the result (only a DOWNLOAD has a web source to fetch again)."""
DOWNLOAD = "download" # yt-dlp fetches a source into a shared, deduplicated asset
EDIT = "edit" # a per-user trim/crop of a file already in the library
UPLOAD = "upload" # a file the user sent; the worker only converts it
CLIP = (
"clip" # a per-user range of a YouTube video or a Plex item, cut straight from the source
)
# A job kind with no web source to fetch again: Re-download refuses these.
NOT_REFETCHABLE = frozenset({JobKind.EDIT, JobKind.UPLOAD, JobKind.CLIP})
# States a route sets to STOP a running job. The worker's supervisor must step in (kill the child, free
# the asset, leave the route's status intact) when it observes one of these on a job it is running.
STOPPED_BY_ROUTE = frozenset({JobStatus.PAUSED, JobStatus.CANCELED})
# A job that will not move again on its own: the queue lists it only briefly after it got here.
FINISHED = frozenset({JobStatus.DONE, JobStatus.CANCELED})
+93 -22
View File
@@ -11,6 +11,7 @@ a Plex library using a metadata agent picks up rich info. Channel = "show", vide
date-based — the community-standard mapping for YouTube-in-Plex.
"""
import os
import re
import shutil
import unicodedata
@@ -67,6 +68,57 @@ _FOLD_MAP = str.maketrans(
)
def _mount_points(mountinfo: str) -> list[str]:
"""The mount points a /proc/<pid>/mountinfo table lists (its fifth field, octal escapes undone)."""
points = []
for line in mountinfo.splitlines():
fields = line.split(" ")
if len(fields) > 4:
points.append(re.sub(r"\\([0-7]{3})", lambda m: chr(int(m.group(1), 8)), fields[4]))
return points
def _in_container() -> bool:
return Path("/.dockerenv").exists() or Path("/run/.containerenv").exists()
def _read_mountinfo() -> str | None:
try:
return Path("/proc/self/mountinfo").read_text()
# silent-ok: no mount table to read (not Linux) — the caller then cannot tell, and refuses nothing.
except OSError:
return None
def files_root_problem(
root: str, mountinfo: str | None = None, in_container: bool | None = None
) -> str | None:
"""Why `root` cannot be the files library's volume, or None when it can. In a container, a root
that is no mount of its own (nor inside one) is the image's empty directory: the library would
answer 404 for every file, and what a download writes would go when the container does — the
state an install is in when its compose file still mounts the library somewhere else.
Asked of the kernel's mount table, not by comparing devices: a storage driver that keeps the
container's root on the host's own disk (vfs, the fallback under Docker-in-LXC on older ZFS) gives
a mounted volume the same device as `/`. Outside a container a plain directory is a fine root."""
path = Path(root)
if not path.is_dir():
return f"{root} does not exist"
if in_container if in_container is not None else _in_container():
table = mountinfo if mountinfo is not None else _read_mountinfo()
if table is not None:
real = os.path.realpath(root)
mounted = any(
m != "/" and (real == m or real.startswith(m.rstrip("/") + "/"))
for m in _mount_points(table)
)
if not mounted:
return f"{root} is not a mounted volume (it is the container's own empty directory)"
if not os.access(path, os.W_OK):
return f"{root} is not writable"
return None
def _ascii_fold(text: str) -> str:
"""Transliterate to pure ASCII: apply the explicit map, NFKD-split accented letters and drop
the combining marks (á→a, ő→o), then strip any leftover non-ASCII (non-Latin scripts that have
@@ -122,7 +174,7 @@ def display_filename(name: str) -> str:
def rel_path(meta: MediaMeta, ext: str, layout: str = "plex", asset_id: int | None = None) -> str:
"""Path of the media file relative to DOWNLOAD_ROOT (forward slashes).
"""Path of the media file relative to FILES_ROOT (forward slashes).
`asset_id` disambiguates the file. A MediaAsset is unique on (source_kind, source_ref,
format_sig), so the SAME video downloaded in two formats is two distinct assets — but without
@@ -144,7 +196,7 @@ def rel_path(meta: MediaMeta, ext: str, layout: str = "plex", asset_id: int | No
def edit_rel_path(user_id: int, asset_id: int, ext: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a phase-2 editor derivative.
"""On-disk path (relative to FILES_ROOT) for a phase-2 editor derivative.
Kept in a per-user `.edits/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (these are user clips, not canonical episodes; there are no .nfo/poster
@@ -161,20 +213,32 @@ def _clean_upload_ext(filename: str) -> str:
def upload_rel_path(user_id: int, asset_id: int, filename: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a locally-uploaded file.
"""On-disk path (relative to FILES_ROOT) for a locally-uploaded file.
Kept in a per-user `.uploads/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (an upload is arbitrary user content, not a canonical episode; no .nfo/poster
episode sidecars). The name is system-decided and bound to the asset id (`_a{id}` keeps two
same-named uploads apart); the original filename is display metadata only. The extension is
preserved (shell-safe, bounded) so the served file + its Content-Disposition carry a correct
suffix; an extensionless upload stays extensionless."""
stem = sanitize(Path(filename).stem, 80)
suffix; an extensionless upload stays extensionless.
The stem holds no dot, so the last dot of the name is always the extension's: every sibling
derived by swapping it (`with_suffix` — the poster, the converted file) keeps `_a{id}`. A dotted
stem with no extension ('Holiday.2024.ä') once lost it, and two such uploads shared one file."""
stem = sanitize(Path(filename).stem, 80).replace(".", "_")
ext = _clean_upload_ext(filename)
suffix = f".{ext}" if ext else ""
return f".uploads/{user_id}/{stem}_a{asset_id}{suffix}"
def with_ext(rel: str, old_ext: str | None, ext: str) -> str:
"""`rel` with its extension — `old_ext`, the stored `container`, None for a bare name — swapped
for `ext`. Not `Path.with_suffix`: on a bare name whose stem holds a dot (an upload placed before
the stems lost their dots) that would take the stem's tail for the extension, and `_a{id}` with it."""
base = rel[: -len(old_ext) - 1] if old_ext and rel.endswith(f".{old_ext}") else rel
return f"{base}.{ext}"
def upload_ext(filename: str) -> str | None:
"""The stored `container` for an uploaded file: its shell-safe lowercased extension, or None when
the upload has no extension (a bare name). Shares `_clean_upload_ext` with `upload_rel_path` so the
@@ -230,32 +294,39 @@ def served_media_type(media_kind: str | None, container: str | None) -> str | No
def download_filename(display_name: str, container: str | None, path: Path) -> str:
"""The Content-Disposition filename for a served download: the cleaned display name carrying a single
correct extension — the asset container, else the file's own suffix — without doubling it when the
name already ends in that extension. Shared by the private download and the public watch endpoints."""
name already ends in that extension. A file with no extension at all is served without one —
never as `name.` (Windows drops a trailing dot, so the saved file would not be the one named).
Shared by the private download and the public watch endpoints."""
ext = container or path.suffix.lstrip(".")
base = display_filename(display_name)
if not ext:
return base
if base.lower().endswith(f".{ext.lower()}"):
base = base[: -(len(ext) + 1)]
return f"{base}.{ext}"
def abs_path(download_root: str, rel: str) -> Path:
return Path(download_root) / rel
def abs_path(files_root: str, rel: str) -> Path:
return Path(files_root) / rel
def safe_abs_path(download_root: str, rel: str) -> Path | None:
"""The resolved absolute path for `rel`, but ONLY if it stays inside DOWNLOAD_ROOT and exists on
disk — otherwise None. Centralizes the path-traversal + existence guard every file-serving endpoint
shares, so the containment check lives in exactly one place."""
root = Path(download_root).resolve()
path = abs_path(download_root, rel).resolve()
def safe_abs_path(files_root: str, rel: str | None) -> Path | None:
"""The resolved absolute path for `rel`, but ONLY if it stays inside FILES_ROOT and exists on
disk — otherwise None (no `rel` at all included: an asset without a file has nothing to serve).
Centralizes the path-traversal + existence guard every file-serving endpoint shares, so the
containment check lives in exactly one place."""
if not rel:
return None
root = Path(files_root).resolve()
path = abs_path(files_root, rel).resolve()
if root not in path.parents or not path.exists():
return None
return path
def place_file(staging_file: Path, download_root: str, rel: str) -> None:
"""Move a completed staging file into its final location under DOWNLOAD_ROOT."""
dest = abs_path(download_root, rel)
def place_file(staging_file: Path, files_root: str, rel: str) -> None:
"""Move a completed staging file into its final location under FILES_ROOT."""
dest = abs_path(files_root, rel)
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(staging_file), str(dest))
@@ -289,9 +360,9 @@ def _sidecar(media: Path, ext: str) -> Path:
return media.with_suffix(ext)
def write_sidecars(download_root: str, rel: str, meta: MediaMeta, thumb_src: Path | None) -> bool:
def write_sidecars(files_root: str, rel: str, meta: MediaMeta, thumb_src: Path | None) -> bool:
"""Write `<basename>.nfo` + `<basename>.jpg` next to the media file. Returns True if written."""
media = abs_path(download_root, rel)
media = abs_path(files_root, rel)
try:
_sidecar(media, ".nfo").write_text(_nfo_xml(meta), encoding="utf-8")
if thumb_src and thumb_src.exists():
@@ -307,9 +378,9 @@ def write_sidecars(download_root: str, rel: str, meta: MediaMeta, thumb_src: Pat
return False
def delete_asset_files(download_root: str, rel: str) -> None:
def delete_asset_files(files_root: str, rel: str) -> None:
"""Remove the media file and its sidecars; prune now-empty parent dirs (best effort)."""
media = abs_path(download_root, rel)
media = abs_path(files_root, rel)
for p in (media, _sidecar(media, ".nfo"), _sidecar(media, ".jpg")):
try:
p.unlink()
@@ -323,7 +394,7 @@ def delete_asset_files(download_root: str, rel: str) -> None:
extra=logs.event("download.asset.file_unlink_failed", reason=type(exc).__name__),
)
# Prune empty Season/Channel dirs up to (but not including) the root.
root = Path(download_root).resolve()
root = Path(files_root).resolve()
parent = media.parent
while parent != root and root in parent.parents:
try:
+86 -22
View File
@@ -1,15 +1,24 @@
"""Local file ingest for the download center (E7 S5) — the non-yt-dlp path into the media library.
"""Local file ingest for the download center — the non-yt-dlp path into the media library.
A user-uploaded file joins the SAME library as a downloaded one, so it inherits the whole function
set already built (metadata chips, share links, and — for media — in-app play/edit). Unlike a
download it bypasses yt-dlp AND the worker entirely: the file is already on the host, so ingest
probes it, places it on disk, and creates an already-`ready` MediaAsset + `done` DownloadJob in one
step — the same "the file exists, mark it done now" shortcut `service.enqueue` takes on a cache hit.
download it bypasses yt-dlp: the file is already on the host, so ingest probes it and places it on
disk. A file that already is the phone-safe shape (`playable`) — or is not media at all — becomes an
already-`ready` MediaAsset + `done` DownloadJob in one step, the same "the file exists, mark it done
now" shortcut `service.enqueue` takes on a cache hit.
Media that is NOT phone-safe (an iPhone's HEVC .mov, a VP9 .webm, an .mkv) is placed as it came and
queued: the asset stays `pending` and the job `queued`, and the download worker converts it
(`pipeline._convert_upload`) — so a share link never hands a friend a file their phone cannot play.
Only the converted file is kept (the owner's choice: one file per upload). A 10-bit HDR original
becomes 8-bit H.264 with its colour tags kept — still tagged HDR (HLG/PQ, BT.2020), not tone-mapped
to SDR (the owner's choice too). Until then the original sits at the asset's `rel_path`, which is
what a failed conversion's Retry starts from again, and what a delete removes
(`service.release_asset`).
Kinds (the invariant lives in `mediakind`): a probe with a real video stream -> video, audio-only
-> audio, neither -> a generic 'file' (pdf/zip/docx/…) that is download-only and never rendered
inline (see the force-download serving in the file/watch routes). This module is the ingest seam a
future sub-function (e.g. convert) plugs a new adapter into, mirroring how it sidesteps the worker.
inline (see the force-download serving in the file/watch routes).
An upload does NOT get an `expires_at`: it's user-provided and may be irreplaceable (there's no
source to re-download), so it doesn't age out on the retention clock the way a re-fetchable download
@@ -22,9 +31,12 @@ from pathlib import Path
from sqlalchemy.orm import Session
from app import audit
from app.audit import AuditAction
from app.config import settings
from app.downloads import edit as editmod
from app.downloads import mediakind, probe, quota, service, storage
from app.downloads import folders, mediakind, playable, probe, quota, service, storage
from app.downloads.status import JobKind, JobStatus
from app.models import DownloadJob, MediaAsset, display_text
@@ -43,16 +55,35 @@ def ingest_upload(
src_path: Path,
size: int,
display_name: str | None = None,
sha256: str | None = None,
folder_id: int | None = None,
via: str | None = None,
) -> DownloadJob:
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return the `done`
job. `src_path` is MOVED into place on success (the caller stages it to a temp file first).
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return its job —
`done`, or `queued` for the worker's phone-safe conversion (see the module docstring). `src_path`
is MOVED into place on success (the caller stages it to a temp file first). `sha256` is the
hash of those bytes, when the caller took it on the way in; `folder_id` the folder it asked for
(`folders.upload_target`), else its Uploads folder; `via` what the audit row names as the doer
when it was not the user themselves (an API token).
Raises folders.FolderError when that folder is not one an upload may land in (or is gone).
Raises quota.QuotaExceeded (holdings/footprint cap) — checked BEFORE any disk write, so a
rejected upload leaves nothing behind and the caller just cleans up its temp file."""
quota.check_upload(db, user_id, size) # the file's size is known → charge it up front
if folder_id is not None:
# Refused before any work; held only once the job is about to name it (below), not through
# the probe and the poster, which a rename or delete of the folder would wait out.
folders.upload_target(db, user_id, folder_id, hold=False)
vcodec, acodec, width, height, duration = probe.probe_media(src_path)
data = probe.probe_json(src_path)
vcodec, acodec, width, height, duration = probe.facts(data)
kind = mediakind.derive(vcodec, acodec)
# Judged by the extension the file is PLACED under (and the worker re-judges), not the raw one:
# 'clip.mp4~' is stored as .mp4, and judging 'mp4~' queued a conversion with nothing to convert.
ext = storage.upload_ext(filename) or ""
safe = playable.plan(data, ext) if kind != mediakind.FILE else None
convert = safe is not None and safe.needs_change
# An upload never dedups (each file is its own asset): a random ref makes the cache key unique,
# so `format_sig` is a constant marker rather than a format hash.
@@ -64,19 +95,23 @@ def ingest_upload(
db.flush() # need asset.id for the on-disk path
rel = storage.upload_rel_path(user_id, asset.id, filename)
storage.place_file(src_path, settings.download_root, rel)
storage.place_file(src_path, settings.files_root, rel)
# From here the file is on disk under `.uploads/`. If the transaction fails before it commits, the
# asset/job rows roll back but the placed file would be ORPHANED — nothing references it, the
# per-user delete path keys off asset.rel_path (which is gone), and the GC orphan sweep skips the
# dot-prefixed `.uploads/` tree. So roll back the on-disk side too on any failure.
# asset/job rows roll back and nothing names the placed file: the per-user delete path keys off
# asset.rel_path (which is gone), and the GC's `.uploads` sweep reaches it only an hour later. So
# roll back the on-disk side too — unless the commit itself raised after it landed (see below).
asset_id = asset.id
try:
final = storage.abs_path(settings.download_root, rel)
final = storage.abs_path(settings.files_root, rel)
actual_size = final.stat().st_size if final.exists() else size
asset.status = "ready"
asset.status = "pending" if convert else "ready"
asset.rel_path = rel
# The hash names the bytes that arrived: a conversion replaces them, so it is kept only
# when the file stays as it came.
asset.sha256 = None if convert else sha256
asset.size_bytes = actual_size
asset.container = storage.upload_ext(filename)
asset.container = ext or None
asset.vcodec = vcodec
asset.acodec = acodec
asset.width = width
@@ -90,25 +125,44 @@ def ingest_upload(
# A frame poster gives the library row a thumbnail + the share link an og:image. Audio has
# no frame and a generic file no player, so neither gets one. Best-effort (ffmpeg decodes
# server side even for a codec the browser can't play).
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind="upload",
job_kind=JobKind.UPLOAD,
source_kind="upload",
source_ref=ref,
profile_snapshot={},
display_name=display_text(display_name),
status="done",
progress=100,
status=JobStatus.QUEUED if convert else JobStatus.DONE,
progress=0 if convert else 100,
queue_pos=service.next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
if folder_id is not None:
job.folder_id = folders.upload_target(db, user_id, folder_id).id
if not convert:
folders.stamp_done(db, job)
service.incr_ref_count(db, asset.id)
db.flush()
audit.record(
db,
user_id,
AuditAction.FILE_UPLOAD,
target_type="file",
target_id=job.id,
summary=via,
)
db.commit()
except Exception:
db.rollback()
if _row_landed(db, asset_id) is not False:
# The commit reached the server and lost its answer, or the row cannot be read: the file
# may be the asset's now, so it is never removed blind — the sweep takes it if no row
# names it.
raise
# The rows are gone; remove the now-unreferenced file so no bytes leak. Clean the poster by its
# DETERMINISTIC path (rel with a .jpg suffix, the same path ensure_poster writes to) rather than
# ensure_poster's return value: it writes the frame directly to that path and returns None on an
@@ -116,7 +170,17 @@ def ingest_upload(
# return would skip that orphan. unlink is missing_ok, so the poster path is a harmless no-op
# when none was generated (audio/file, or a video whose poster never started).
for leaked in (rel, str(Path(rel).with_suffix(".jpg"))):
storage.abs_path(settings.download_root, leaked).unlink(missing_ok=True)
storage.abs_path(settings.files_root, leaked).unlink(missing_ok=True)
raise
db.refresh(job)
return job
def _row_landed(db: Session, asset_id: int) -> bool | None:
"""After the ingest transaction raised: is the asset row there anyway? A commit can reach the
server and lose its answer (the connection drops after COMMIT). None when it cannot be read."""
try:
return db.get(MediaAsset, asset_id) is not None
except Exception: # noqa: BLE001 — "cannot tell" is the answer, the caller acts on it
# silent-ok: the caller re-raises the ingest's own error.
return None
+2 -2
View File
@@ -197,7 +197,7 @@ def _next_step(verify_link: str | None, has_account: bool, hours: int, lang: str
Three states, because a mail that guesses wrong sends someone looking for a link that isn't
coming — or worse, tells them to sign in when their address is still unconfirmed and sign-in will
refuse them (E9.5 S4 named that state on screen; the mail must not contradict it).
refuse them (the sign-in screen names that state; the mail must not contradict it).
The link branch is the point of the C option: in the common ordering — approve first, the person
signs up later, or signs up and waits — they would otherwise have to find and open TWO separate
@@ -253,7 +253,7 @@ def send_access_invitation(
the closing line offers no deadline — nudging is the one thing this mail must not do.
Its language is the one place the app has no signal to read: the admin typed this address in, so
nobody has ever told us how to address its owner. That is why the send button asks (E9.5 S7).
nobody has ever told us how to address its owner. That is why the send button asks.
"""
admin, contact = _reply_contact(lang)
body = mailtext.text(
+5 -5
View File
@@ -109,7 +109,7 @@ def _tag_visible(user: User):
# The pseudo-category of a selected tag id no Tag row answers (a saved view or the feed state outlives
# a deleted tag). It filters like a category whose tags no channel carries — to nothing, in the SQL and
# in feed_summary's mirror alike — instead of the id being dropped, which showed the WHOLE feed under
# the view's old name (E32 S2). No real category is called this (language/topic/other).
# the view's old name. No real category is called this (language/topic/other).
MISSING_TAG_CATEGORY = "missing"
@@ -118,7 +118,7 @@ def tags_by_category(db: Session, user: User, tag_ids: list[int]) -> dict[str, l
shared by the feed's tag WHERE (filtered_query) and the facet/count summary (feed_summary), so the
two can't disagree on which selected tag belongs to which category. An unknown id lands in
MISSING_TAG_CATEGORY — and so does another user's private tag: resolving it to its real category
told the caller which foreign ids exist (E32 review)."""
told the caller which foreign ids exist."""
if not tag_ids:
return {} # skip the pointless `WHERE id IN ()` round-trip on the no-tag-filter hot path
out: dict[str, list[int]] = {}
@@ -463,7 +463,7 @@ def feed_summary(db: Session, user: User, **params) -> dict:
def feed_years(db: Session, user: User, **params) -> list[dict]:
"""Per-year video counts for the current filter context (item 8 / E8·S2): powers the year control's
"""Per-year video counts for the current filter context: powers the year control's
quick chips + the "more…" per-year-counts dialog. Every OTHER filter (tags, channels, content type,
search, watch state, scope) still applies, but the publish-date narrowing itself is DROPPED —
published_after/before and max_age_days — so the year list is the full set of years available and
@@ -559,7 +559,7 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
return table.get(sort) or table["newest"]
# Relevance ranks only the newest RANK_WINDOW matches (E31): ts_rank reads every match's tsvector, and
# Relevance ranks only the newest RANK_WINDOW matches: ts_rank reads every match's tsvector, and
# a broad word matches most of the library ("a" ranked 193k rows in 6.8 s on prod, against 1.1 s to just
# find them). The rows past the window keep rank 0, so they follow the ranked ones newest first; a word
# that broad has no meaningful relevance order anyway, and a specific one fits the window whole.
@@ -671,7 +671,7 @@ def encode_cursor(keys: list[dict], row, window: datetime | None = None) -> str:
def _cursor_dt(v):
"""A cursor's timestamp: aware or refused. The server writes only aware ones (timestamptz), and
windowed_relevance_rows compares them in Python, where a naive one beside an aware one raises — a
500 for a hand-made cursor instead of the 400 (E31 review 2)."""
500 for a hand-made cursor instead of the 400."""
if v is None:
return None
dt = datetime.fromisoformat(v)
+2 -2
View File
@@ -1,3 +1,3 @@
"""Bug reports built from debug sessions (E16): the files an issue carries, and where they go —
downloaded as a zip, or (S4) sent to a Gitea issue tracker. The session itself — what is recorded
"""Bug reports built from debug sessions: the files an issue carries, and where they go —
downloaded as a zip, or sent to a Gitea issue tracker. The session itself — what is recorded
and tagged — is the core module `app/debugsession.py`."""
+1 -1
View File
@@ -5,7 +5,7 @@ Gitea upload, so the two cannot drift apart.
in their language; the server does not rewrite it);
* `client.log` / `server.log` — the session's evidence (`debugsession.bundle`) minus the rows the
user took out, one JSON object per line, browser rows and server rows apart. `.log`, not
`.jsonl`: Gitea refuses that extension (measured, plans/e16 S0);
`.jsonl`: Gitea refuses that extension (measured);
* the screenshots, in the order they were taken.
Every row is already redacted twice (at capture, and again when stored), and nothing here adds a
+5 -5
View File
@@ -1,5 +1,5 @@
"""The Gitea issue tracker a debug session's issue goes to (E16 S4) — measured against Gitea 1.27.1
before a line of this was written (plans/e16-debug-issue.md, S0).
"""The Gitea issue tracker a debug session's issue goes to — measured against Gitea 1.27.1
before a line of this was written.
Configured by an admin (sysconfig `issues_gitea_url` / `_repo` / `_token`, the token encrypted at
rest). The repository must be PRIVATE: an issue carries its reporter's log — page titles, filter
@@ -79,7 +79,7 @@ def target(db: Session) -> dict | None:
@contextmanager
def _client(settings: tuple[str, str, str]) -> Iterator[tuple[httpx.Client, str]]:
# Takes the settings already read, not the session: reading them HERE opened a transaction one
# line after `deliver` had ended one, and held it across the calls below (deep-review round 4).
# line after `deliver` had ended one, and held it across the calls below.
url, repo, token = settings
if not (url and repo and token):
raise GiteaError("not_configured")
@@ -203,7 +203,7 @@ def deliver(
# call, so nothing holds a snapshot (and an idle connection) across minutes of tracker I/O.
# The settings are read HERE and handed to `_client`, not read inside it: reading them there
# was a SELECT one line after this commit, which reopened the transaction and held it — idle,
# with its snapshot pinned — across the repository check and the create (deep-review round 4
# with its snapshot pinned — across the repository check and the create (a review
# measured it in pg_stat_activity). A comment saying otherwise sat right here.
checkpoint()
with _client(settings) as (client, repo):
@@ -223,7 +223,7 @@ def deliver(
)
# Built ON the state read at the start so nothing already in it is dropped. The claim
# is NOT among those things, whatever this comment used to say: `set_delivery` strips
# any `sending` passed to it and stamps the row's own from `token` (round 4 — removing
# any `sending` passed to it and stamps the row's own from `token` (removing
# the spread left the whole suite green, because the claim never travelled here).
state = {
**state,
+131
View File
@@ -0,0 +1,131 @@
"""The durable history of background runs — the `job_runs` table.
A run is written twice: a `running` row when it starts, its outcome when it ends. Each write is its
OWN short session, committed at once, for the reason `quota.record_usage` gives: the run's own work
may roll back (its `except: db.rollback()`), and the history of what happened must not go with it —
nor may this bookkeeping commit the job's half-done session.
Only the API process runs jobs (the worker never starts the scheduler), and one API process serves a
database (`.env.example`, the Scheduler section: the single-run guard is in its memory too), so at its
startup nothing can still be running: `interrupt_leftovers` closes what a stopped process left open.
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete, func, select, union_all, update
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import JobRun, QuotaEvent
from app.runstatus import RunStatus, RunTrigger
def start(job: str, trigger: RunTrigger, actor_id: int | None, target: str | None = None) -> int:
"""Open a run's row; returns its id."""
with SessionLocal() as s:
row = JobRun(
job=job,
trigger=trigger,
actor_id=actor_id,
target=target,
status=RunStatus.RUNNING,
started_at=datetime.now(timezone.utc),
)
s.add(row)
s.commit()
return row.id
def finish(
run_id: int,
status: RunStatus,
*,
summary: str | None = None,
result: dict | None = None,
error: str | None = None,
failure: dict | None = None,
) -> None:
"""Close a run with its outcome and what it spent. Conditional on the row still running: a row
the startup sweep already called interrupted is not rewritten by a thread that outlived it."""
with SessionLocal() as s:
units = s.scalar(
select(func.coalesce(func.sum(QuotaEvent.units), 0)).where(
QuotaEvent.job_run_id == run_id
)
)
s.execute(
update(JobRun)
.where(JobRun.id == run_id, JobRun.status == RunStatus.RUNNING)
.values(
status=status,
finished_at=datetime.now(timezone.utc),
summary=summary,
result=result,
error=error,
failure=failure,
quota_units=units or 0,
)
)
s.commit()
def interrupt_leftovers() -> int:
"""Every row a previous process left `running` → `interrupted`, with what it spent until then
(its quota rows outlive it, as `finish` sums them). Called once at startup, before the scheduler
can open a new run. Returns how many."""
spent = (
select(func.coalesce(func.sum(QuotaEvent.units), 0))
.where(QuotaEvent.job_run_id == JobRun.id)
.scalar_subquery()
)
with SessionLocal() as s:
res = s.execute(
update(JobRun)
.where(JobRun.status == RunStatus.RUNNING)
.values(
status=RunStatus.INTERRUPTED,
finished_at=datetime.now(timezone.utc),
quota_units=spent,
)
)
s.commit()
return res.rowcount if isinstance(res, CursorResult) else 0
def recent_runs(db: Session, jobs: list[str], per_job: int) -> dict[str, list[JobRun]]:
"""Each of `jobs`' last `per_job` runs, newest first, by job id — one query, each job's part a
short walk of `ix_job_runs_job_started`, so a poll costs the same at any history length (a window
over the whole table grew with every run kept)."""
if not jobs:
return {}
newest = union_all(
*(
select(JobRun.id)
.where(JobRun.job == job)
.order_by(JobRun.started_at.desc(), JobRun.id.desc())
.limit(per_job)
for job in jobs
)
).subquery()
rows = db.execute(
select(JobRun)
.where(JobRun.id.in_(select(newest.c.id)))
.order_by(JobRun.job, JobRun.started_at.desc(), JobRun.id.desc())
).scalars()
out: dict[str, list[JobRun]] = {}
for r in rows:
out.setdefault(r.job, []).append(r)
return out
def gc(db: Session, retention_days: int) -> int:
"""Delete finished runs older than `retention_days` (0 = keep forever). A run still going is
never reaped. The caller commits."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
res = db.execute(
delete(JobRun).where(JobRun.started_at < cutoff, JobRun.status != RunStatus.RUNNING)
)
return res.rowcount if isinstance(res, CursorResult) else 0
+28 -14
View File
@@ -10,8 +10,8 @@ the logs were both unjoinable and leaky. `configure_logging()` replaces all four
Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
ts · level · sev · source · logger · event · msg · attrs
plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
`job_id`) and the app version. That shape is what E22.6's admin console filters on and what a Gitea
issue attachment will carry, so it is deliberately flat and stable.
`job_id`, `run_id`) and the app version. That shape is what the admin log console filters on and
what a Gitea issue attachment will carry, so it is deliberately flat and stable.
Call sites stay ordinary stdlib logging:
@@ -81,15 +81,28 @@ _debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
"log_debug_session", default=None
)
_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
# The background run (app/jobruns.py) this context is doing; quota's spend is tagged with it too.
_run_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_run_id", default=None)
_CONTEXT_VARS = {
"request_id": _request_id,
"user_id": _user_id,
"debug_session_id": _debug_session,
"job_id": _job_id,
"run_id": _run_id,
}
def current_request_id() -> str | None:
"""The HTTP request this context is serving, if any — the request middleware sets it."""
return _request_id.get()
def current_run_id() -> int | None:
"""The background run this context is doing, if any — `bind(run_id=…)` sets it."""
return _run_id.get()
class _RequestScope:
"""What the auth dependency learns about a request, for every line logged after it.
@@ -106,7 +119,7 @@ class _RequestScope:
self.user_id: int | None = None
# The user's consented diagnostics level as a logging number; None at the base level.
self.level: int | None = None
# The user's own open debug session this request named (E16), else None.
# The user's own open debug session this request named, else None.
self.debug_session_id: str | None = None
@@ -122,11 +135,11 @@ def set_user(
debug_session: str | None = None,
) -> None:
"""Attribute the rest of this request's log lines to `user_id`, and log them at the user's
consented diagnostics `level` until `until` (E22 S5; "INFO" = no elevation). Called from the
consented diagnostics `level` until `until` ("INFO" = no elevation). Called from the
auth dependency — the one place the session is actually resolved into an account — rather than
from the request middleware, which runs OUTSIDE the session middleware.
`debug_session` tags every line with the user's open debug session (E16). The caller has
`debug_session` tags every line with the user's open debug session. The caller has
already checked it is theirs and open; a session's level arrives here as `level`/`until`.
The root logger's floor is tracked per elevation: a request without the session's header (an
@@ -334,7 +347,7 @@ _RECORD_STANDARD = frozenset(
class ContextFilter(logging.Filter):
"""Copies the correlation contextvars onto the record. A Filter (not the formatter) so the
values are captured at emit time, before a QueueHandler can move the record to another
thread — the DB sink in S3 depends on that."""
thread — the DB sink depends on that."""
def filter(self, record: logging.LogRecord) -> bool:
for name, var in _CONTEXT_VARS.items():
@@ -355,7 +368,7 @@ class ContextFilter(logging.Filter):
class LevelGate(logging.Filter):
"""The configured level, enforced at the HANDLER, with one exception: the lines of a request
whose user raised their diagnostics level (E22 S5).
whose user raised their diagnostics level.
The root logger's level cannot do this alone: it is one number for the whole process, and the
point is to log ONE user's requests in detail while everyone else's stay at the base level. So
@@ -457,7 +470,7 @@ class TextFormatter(logging.Formatter):
# --- configuration -------------------------------------------------------------------------
# Third-party loggers that are noisy at our own level. Pinned explicitly so the diagnostics knob can
# lower the ROOT level to DEBUG/TRACE without drowning the output in SQL statements and HTTP chatter
# — raising THESE is a deliberate, separate act (E22.6's console).
# — raising THESE is a deliberate, separate act (the admin log console).
_THIRD_PARTY_LEVELS = {
"sqlalchemy": "WARNING",
"sqlalchemy.engine": "WARNING",
@@ -481,7 +494,7 @@ def get_logger(name: str) -> logging.Logger:
`siftlode.plex.sync`.
The names were literals before, and 25 of them covered 34 modules — six different Plex modules
all called themselves `siftlode.plex`, so neither a level pinned per logger nor E22.6's console
all called themselves `siftlode.plex`, so neither a level pinned per logger nor the console's
filter could tell a sync line from a stream line. Derived from `__name__` there is exactly one
name per module, it cannot drift from the file it sits in, and `test_logger_names.py` fails if a
module hand-rolls one again.
@@ -562,12 +575,12 @@ def current_level() -> str:
return logging.getLevelName(_base_level or logging.getLogger().getEffectiveLevel())
# --- per-user elevation (E22 S5) -----------------------------------------------------------
# --- per-user elevation -----------------------------------------------------------
# The base level (`LOG_LEVEL`, then the admin knob). NOTSET until configure_logging runs — a unit
# test, a script — so the gate lets everything through there, as it did before the gate existed.
_base_level = logging.NOTSET
# → (level number, expiry) for every elevation this process has seen and not yet expired: a user's
# own, by user id, and a debug session's (E16), by (user id, session id) — its requests alone carry
# own, by user id, and a debug session's, by (user id, session id) — its requests alone carry
# its level, so it is tracked apart from the user's own requests.
_ElevationKey = int | tuple[int, str]
_elevations: dict[_ElevationKey, tuple[int, datetime]] = {}
@@ -667,7 +680,7 @@ def current_source() -> str:
# --- request logging -----------------------------------------------------------------------
# A client may supply its own request id so a browser log line and a server line join up (E22 S4).
# A client may supply its own request id so a browser log line and a server line join up.
# Accepted only in this shape: it ends up in a log field, and an unbounded header would be a way to
# write arbitrary text into the log.
_REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9._-]{8,64}$")
@@ -688,10 +701,11 @@ _POLLED_ROUTES = frozenset(
{
"/api/me",
"/api/me/notifications/unread_count",
"/api/messages/unread_count",
"/api/downloads",
"/api/downloads/index",
"/api/downloads/usage",
"/api/files",
"/api/files/folders",
"/api/files/usage",
"/api/sync/my-status",
"/api/admin/scheduler",
"/api/admin/downloads",
+5 -4
View File
@@ -1,4 +1,4 @@
"""The `log_events` sink (E22 S3): the stored copy of the log the diagnostics console reads.
"""The `log_events` sink: the stored copy of the log the diagnostics console reads.
stdout stays the complete log (docker keeps 3×50 MB of it); this table keeps the part worth querying
for `log_retention_days`:
@@ -6,7 +6,7 @@ for `log_retention_days`:
* every WARNING and above,
* the `http.request` line of every request at INFO or above (not the DEBUG-level health checks and
static files),
* every line of a request whose user raised their diagnostics level (E22 S5 — `logs.LevelGate`
* every line of a request whose user raised their diagnostics level (`logs.LevelGate`
marks them), which is the detail they consented to have kept,
* nothing a caller marked `extra={logs.PERSIST: False}` — how the request log caps anonymous
traffic per client.
@@ -347,8 +347,8 @@ def gc(db: Session, retention_days: int) -> int:
def serialize(row: LogEvent) -> dict:
"""A stored row as the API sends it — one shape for every reader: the admin console (E22.6) and
a debug session's evidence list (E16)."""
"""A stored row as the API sends it — one shape for every reader: the admin console and
a debug session's evidence list."""
return {
"id": row.id,
"ts": iso(row.ts),
@@ -362,6 +362,7 @@ def serialize(row: LogEvent) -> dict:
"request_id": row.request_id,
"user_id": row.user_id,
"job_id": row.job_id,
"run_id": row.run_id,
"debug_session_id": row.debug_session_id,
"version": row.version,
}
+4 -4
View File
@@ -1,5 +1,5 @@
"""Bilingual text for every outbound mail, and the one resolver that decides which language a
recipient gets (E9.5 S7).
recipient gets.
Why a catalog instead of literals in `email.py`: HU/EN parity is a house rule for every user-facing
string, and a mail is the one user-facing surface the reader CANNOT re-render by switching the UI
@@ -53,7 +53,7 @@ def lang_for_user(user: User | None) -> str:
def lang_for_email(db: Session, email: str) -> str:
"""The language for an address that may not have an account yet: their own account preference,
else the language they filled our access-request form in, else English.
else the language their access request came in (the locale Google reported), else English.
Matched CANONICALLY like every other identity lookup in this epic, and it walks past rows that
carry no usable code instead of stopping at the first one — prod really does hold two accounts
@@ -82,8 +82,8 @@ def lang_for_email(db: Session, email: str) -> str:
# a name, not a translated string, and giving them keys would invite someone to "translate" them.
MAIL_TEXT: dict[str, dict[str, str]] = {
# ── How long a link lasts ───────────────────────────────────────────────────────────────
# Rendered, not interpolated as a raw hour count: the approval mail's token lives seven days
# (E9.5 S6), and "expires in 168 hours" is a unit no reader converts.
# Rendered, not interpolated as a raw hour count: the approval mail's token lives seven days,
# and "expires in 168 hours" is a unit no reader converts.
"duration.hours": {"en": "{n} hours", "hu": "{n} óra"},
"duration.days": {"en": "{n} days", "hu": "{n} nap"},
# ── Shared closing fragments ────────────────────────────────────────────────────────────────
+40 -11
View File
@@ -28,6 +28,8 @@ log = logs.get_logger(__name__)
from app import bodylimit, collation, logsink, state, sysconfig # noqa: E402 — logging is configured first
from app.config import settings # noqa: E402 — logging is configured first
from app.db import SessionLocal, engine # noqa: E402 — logging is configured first
from app.downloads import service as download_service # noqa: E402 — logging is configured first
from app.downloads import storage # noqa: E402 — logging is configured first
from app.plex import stream as plex_stream # noqa: E402 — logging is configured first
from app.plex import subtitles as plex_subtitles # noqa: E402 — logging is configured first
from app.routes import ( # noqa: E402 — logging is configured first
@@ -42,20 +44,22 @@ from app.routes import ( # noqa: E402 — logging is configured first
discover as discover_routes,
downloads,
feed,
files as files_routes,
health,
me,
messages,
notifications,
playlists,
plex as plex_routes,
public as public_routes,
quota,
runs as runs_routes,
saved_views,
scheduler as scheduler_routes,
search as search_routes,
setup as setup_routes,
sync,
tags,
tokens as tokens_routes,
version,
)
from app.scheduler import shutdown_scheduler, start_scheduler # noqa: E402 — logging is configured first
@@ -64,6 +68,10 @@ from app.scheduler import shutdown_scheduler, start_scheduler # noqa: E402 —
@asynccontextmanager
async def lifespan(app: FastAPI):
log.info("Siftlode starting up")
problem = storage.files_root_problem(settings.files_root)
if problem:
# The rest of the app still serves; the files library cannot (the worker refuses to start).
log.critical("The files library is not reachable: %s. See docs/self-hosting.md.", problem)
# The stored copy of the log (log_events). The entrypoint has already migrated the schema.
logsink.start()
# First-run: if the instance isn't configured yet, mint a fresh one-time setup token and print
@@ -79,6 +87,18 @@ async def lifespan(app: FastAPI):
# With the traceback: this catch spans a connection, a query and a coercion at boot,
# and `%s` of the exception does not say which of them raised.
log.exception("could not read the stored log level; staying at %s", current_level())
if not problem:
try:
left = download_service.library_left_behind(db, settings.files_root)
except Exception: # a database not reachable yet: the check is a hint, never a gate
log.exception("could not check the files library against its database")
else:
if left:
log.critical(
"The files library is empty but the database names files: the volume "
"mounted at %s is not the one that holds them. See docs/self-hosting.md.",
settings.files_root,
)
if not state.is_configured(db):
token = state.rotate_setup_token(db)
url = f"{settings.app_base}/setup?token={token}"
@@ -132,7 +152,7 @@ async def _reap_hls_sessions() -> None:
A FINISHED session is reaped on the short rule here, not kept for the full idle window: an open
player checkpoints every 10 s and that route touches the session, so a paused viewer is no longer
invisible to this pass (see `reap_idle`). Keeping every finished session for ten minutes instead
held one whole segmented copy of a film per item on the download volume (round 5)."""
held one whole segmented copy of a film per item on the download volume."""
while True:
await asyncio.sleep(_HLS_REAP_EVERY_S)
try:
@@ -188,8 +208,8 @@ async def _nul_in_text(request: Request, exc: DataError) -> JSONResponse:
NUL in text, so the driver refuses to even send the query — and a `%00` in a query parameter or
an escaped NUL in a JSON body (a playlist's `contains=`, a reorder's key list, a collection
search) was a 500 with a traceback wherever it reached a comparison. Guarding each site did not
hold: round 3 guarded the by-rating_key lookup and missed the rest. Any other DataError is left
alone — re-raised, it stays the 500 it always was."""
hold: an earlier fix guarded the by-rating_key lookup and missed the rest. Any other DataError
is left alone — re-raised, it stays the 500 it always was."""
if "NUL (0x00)" not in str(exc.orig):
raise exc
return JSONResponse(
@@ -268,7 +288,8 @@ async def static_cache_headers(request, call_next):
# Baseline security response headers (SEC-1). A minimal, safe set that needs no per-route tuning:
# nosniff stops MIME-confusion, the Referrer-Policy keeps the (private) app URL from leaking to
# external sites the user follows, `frame-ancestors 'none'` blocks click-jacking by refusing to be
# iframed anywhere (we never embed our own pages; we embed OTHERS, which this does not affect), and
# iframed anywhere (we embed OTHERS, which this does not affect; the one own answer the app frames, a
# library file's preview, sets its own `frame-ancestors 'self'` — these are defaults, `setdefault`), and
# Permissions-Policy turns off powerful features the app never uses. `X-Frame-Options: DENY` duplicates
# the frame-ancestors intent for legacy engines that never implemented CSP frame-ancestors (belt and
# suspenders — modern browsers use the CSP directive and ignore the header). A full Content-Security-Policy
@@ -311,19 +332,21 @@ app.include_router(me.router)
app.include_router(diagnostics_routes.router)
app.include_router(debug_sessions_routes.router)
app.include_router(notifications.router)
app.include_router(messages.router)
app.include_router(channels.router)
app.include_router(playlists.router)
app.include_router(saved_views.router)
app.include_router(plex_routes.router)
app.include_router(downloads.router)
app.include_router(downloads.admin_router)
app.include_router(files_routes.router)
app.include_router(tokens_routes.router)
app.include_router(public_routes.router)
app.include_router(admin.router)
app.include_router(config_routes.router)
app.include_router(scheduler_routes.router)
app.include_router(audit_routes.router)
app.include_router(admin_logs_routes.router)
app.include_router(runs_routes.router)
app.include_router(quota.router)
app.include_router(version.router)
app.include_router(setup_routes.router)
@@ -341,6 +364,8 @@ STATIC_DIR = Path(__file__).parent / "static_spa"
# stay cached but forces revalidation (cheap 304 via the FileResponse ETag) on every load. The
# hashed bundles under /assets can cache forever (a new build changes their filename).
INDEX_HTML = STATIC_DIR / "index.html"
# A public share link's own page: the player alone, without the app's bundle (frontend/watch.html).
WATCH_HTML = STATIC_DIR / "watch.html"
INDEX_HEADERS = {"Cache-Control": "no-cache"}
app.mount(
"/assets",
@@ -356,16 +381,16 @@ async def index() -> FileResponse:
@app.get("/watch/{token}")
async def watch_page(token: str):
"""Serve the SPA for a public share link, but with per-video Open Graph tags injected so the
"""Serve the public player page for a share link, with per-video Open Graph tags injected so the
link unfurls richly (title/channel/thumbnail) in Messenger/social crawlers. A real browser
ignores the extra tags and hydrates WatchPage as usual; invalid/expired/password links fall
back to the generic card."""
ignores the extra tags and boots WatchPage; invalid/expired/password links fall back to the
generic card."""
from app.downloads import og
with SessionLocal() as db:
rendered = og.render_watch_html(token, INDEX_HTML, db)
rendered = og.render_watch_html(token, WATCH_HTML, db)
if rendered is None:
return FileResponse(INDEX_HTML, headers=INDEX_HEADERS)
return FileResponse(WATCH_HTML, headers=INDEX_HEADERS)
return HTMLResponse(rendered, headers=INDEX_HEADERS)
@@ -383,4 +408,8 @@ async def spa_fallback(full_path: str) -> FileResponse:
# Real SPA-root assets (welcome images, favicon, robots.txt) rarely change and have
# stable names, so a moderate cache is safe and satisfies the cache-lifetime audit.
return FileResponse(candidate, headers={"Cache-Control": "public, max-age=604800"})
# A /watch/ path the route above does not match (a trailing segment, no token) is still the
# player's: WatchPage reads the token off the path and says when a link is gone.
if full_path.startswith("watch/"):
return FileResponse(WATCH_HTML, headers=INDEX_HEADERS)
return FileResponse(INDEX_HTML, headers=INDEX_HEADERS)
+271 -151
View File
@@ -1,4 +1,4 @@
from datetime import date, datetime
from datetime import date, datetime, timezone
from sqlalchemy import (
BigInteger,
@@ -18,11 +18,13 @@ from sqlalchemy import (
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from app.db import Base
from app.runstatus import RunStatus, RunTrigger, sql_in
from app.utils import canonical_email
# Every user-supplied display string in this schema shares one width, and it lives HERE so the
@@ -38,7 +40,7 @@ def display_text(raw: str | None) -> str | None:
blank -> None.
One definition because there are SIX writers and they had drifted: `POST /api/downloads`,
`POST /api/downloads/edit`, `POST /api/downloads/upload`, the metadata PATCH's two fields, and
`POST /api/downloads/edit`, `POST /api/files/upload`, the library PATCH's two fields, and
the Google sign-in callback. Only the PATCH truncated. Truncating is the right answer rather
than refusing: the name is a label on work the user has already done — a file already
transferred, an account already authenticated — and losing its tail is not worth failing the
@@ -59,7 +61,7 @@ POSITION_MAX_S = 2**31 - 1
class TimestampMixin:
"""Adds a server-defaulted `created_at`. Models that index created_at or order it
differently (QuotaEvent, Notification, Message) keep their own column on purpose."""
differently (QuotaEvent, Notification) keep their own column on purpose."""
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
@@ -101,15 +103,15 @@ class User(Base, TimestampMixin, CanonicalEmailMixin):
# Google sign-in that failed to find it), and a migration must not merge accounts on its own.
email_canonical: Mapped[str] = mapped_column(String(320), index=True, server_default="")
# The address Google asserted for this account, recorded at sign-in / link time. NOT the same
# thing as `email`: since E9.5 S3 a Google sign-in no longer overwrites the address the user
# thing as `email`: a Google sign-in no longer overwrites the address the user
# registered with, so a deliberate `me+siftlode@gmail.com` survives — and this is where the other
# half of the truth lives, for the UI to show when the two differ. NULL until the next Google
# sign-in for accounts linked before this column existed.
google_email: Mapped[str | None] = mapped_column(String(320))
# argon2id hash for email+password login; NULL for Google-only / demo accounts.
# argon2id hash for email+password login; NULL for Google-only accounts.
password_hash: Mapped[str | None] = mapped_column(String(255))
# When SEC-10 dropped a preset password on Google adoption (see the single caller of
# auth.revoke_preset_password_if_unverified; admin approval no longer revokes — E9.5 S1). Without
# auth.revoke_untrusted_preset_password; admin approval no longer revokes). Without
# this trace a revoked account was indistinguishable from a Google-only one, so it failed the
# password-reset eligibility check and could never be recovered — it could neither sign in nor
# reset. Set = "had a password, lost it to the guard" -> reset-eligible. Cleared whenever a
@@ -123,7 +125,8 @@ class User(Base, TimestampMixin, CanonicalEmailMixin):
is_active: Mapped[bool] = mapped_column(Boolean, default=True, server_default="true")
# Admin-imposed access block, distinct from is_active (approval lifecycle). A suspended
# account can't sign in by any method and its live sessions are rejected; the user is told
# why and pointed at the operator's contact. The demo account is never suspendable.
# why and pointed at the operator's contact. The worker leaves its queued downloads unclaimed
# until the block is lifted (`worker._claim_job`).
is_suspended: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Server-side session revocation (SA4): a monotonic counter bumped on password reset, password
# change, or "log out everywhere". A signed session cookie records the epoch it was minted under;
@@ -132,9 +135,9 @@ class User(Base, TimestampMixin, CanonicalEmailMixin):
session_epoch: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# When this account last signed in (Google or password), stamped by the one sign-in path,
# routes/auth._establish_session. NULL = not since the column existed (0085; no backfill is
# possible). The shared demo account is never stamped — its "last login" would mean nobody.
# possible).
last_login_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The diagnostics level this user consented to (E22 S5) and when it lapses back to the base
# The diagnostics level this user consented to and when it lapses back to the base
# level. Both NULL = base. Written only by `PUT /api/me/diagnostics`, which computes the expiry
# itself; read through `app.diagnostics.effective_level`, never directly (an expired level is
# not cleared eagerly — it simply stops counting).
@@ -146,12 +149,6 @@ class User(Base, TimestampMixin, CanonicalEmailMixin):
# The user's own YouTube channel id, resolved lazily (channels.list?mine=true) and
# cached. Used to exclude their own channel from playlist discovery. NULL = unresolved.
yt_channel_id: Mapped[str | None] = mapped_column(String(32))
# The single shared "demo" account: signed into without Google OAuth (via a whitelisted
# email on the login page), has no OAuth token / YouTube scope, and its per-user state is
# shared by everyone who enters this way. Exactly one such row is expected.
is_demo: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
)
# Free-form UI preferences (theme, color scheme, font scale, default filters…).
preferences: Mapped[dict | None] = mapped_column(JSON)
@@ -218,25 +215,12 @@ class Invite(Base, CanonicalEmailMixin):
# The language this person asked us in ("en"/"hu"), captured when the request was made: the page
# their form was on, or the locale Google reported for a refused sign-in. NULL for a row the
# admin whitelisted by hand — nobody has told us anything about that mailbox, which is exactly
# why the invitation send asks (E9.5 S7). Captured rather than derived for the same reason
# why the invitation send asks. Captured rather than derived for the same reason
# `preferences.language` and `google_email` are: the signal exists only in that one moment and
# cannot be backfilled afterwards.
lang: Mapped[str | None] = mapped_column(String(8))
class DemoWhitelist(Base, TimestampMixin):
"""Emails that may enter the shared demo account from the login page (no Google OAuth).
A hidden, admin-curated list of "keys to the same door" — every entry logs into the one
shared demo user. Distinct from Invite (which gates real Google sign-in)."""
__tablename__ = "demo_whitelist"
id: Mapped[int] = mapped_column(primary_key=True)
email: Mapped[str] = mapped_column(String(320), unique=True, index=True)
note: Mapped[str | None] = mapped_column(Text)
added_by: Mapped[str | None] = mapped_column(String(320)) # admin email
class Channel(Base, TimestampMixin):
"""A YouTube channel. Shared across all users (one channel's videos are the same
for everyone), so its expensive metadata is fetched and stored only once."""
@@ -481,6 +465,27 @@ class VideoState(Base, UpdatedAtMixin):
progress_updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class LibraryState(Base):
"""Where a user stopped in a library item (their own download, or one shared with them) — the
library player's resume point. Keyed by the JOB, not the file: a job is the user's library entry,
a re-download keeps it, and deleting the job deletes the row; the shared file under it may outlive
both. A revoked share leaves its recipient's row behind, unread — only a job the user can still
reach is ever read or written — and it is back if the item is shared again. Rows exist only for a
position worth keeping (see `routes._common.kept_position`); no row means "from the start"."""
__tablename__ = "library_states"
__table_args__ = (UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
job_id: Mapped[int] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
position_seconds: Mapped[int] = mapped_column(Integer, nullable=False)
progress_updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
class SearchFind(Base, TimestampMixin):
"""Per-user record of a video the user surfaced via their own live YouTube search.
@@ -559,14 +564,20 @@ class QuotaEvent(Base):
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), index=True
)
# The background run that spent it (app/jobruns.py); NULL = a request's own spend. SET NULL: a
# reaped run leaves the spend in the budget's history.
job_run_id: Mapped[int | None] = mapped_column(
ForeignKey("job_runs.id", ondelete="SET NULL"), index=True
)
class AuditLog(Base):
"""Append-only admin/scheduler audit trail: who did what, when, and (where it applies)
the before->after values. `actor_id` is the acting admin; NULL means the scheduler /
background did it (SET NULL on user delete keeps the trail). Logged at ACTION / job-run
granularity — never per-row — so a scheduler run that touches thousands of videos is ONE
row summarising the outcome. `before`/`after` are small JSON snapshots (secrets excluded)."""
"""Append-only admin audit trail: who did what, when, and (where it applies) the before->after
values. `actor_id` is the acting admin; NULL means the background did it (SET NULL on user
delete keeps the trail). Logged per ACTION, never per row; a background run is not an action
and has its own history (`JobRun`). `before`/`after` are small JSON snapshots (secrets
excluded). `request_id` is the request that made the change, so its log lines are one click
away (NULL for rows written outside a request, and for rows older than the column)."""
__tablename__ = "audit_log"
@@ -583,10 +594,11 @@ class AuditLog(Base):
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), index=True
)
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
class LogEvent(Base):
"""One stored log event (E22 S3) — the `log_events` table the diagnostics console reads. Rows are
"""One stored log event — the `log_events` table the diagnostics console reads. Rows are
written by app/logsink.py's background writer, never inside a request's transaction, already
redacted (the same `logs.record_fields` the stdout line is built from).
@@ -599,12 +611,14 @@ class LogEvent(Base):
__tablename__ = "log_events"
__table_args__ = (
Index("ix_log_events_user_ts", "user_id", "ts"),
# E16: an issue's bundle is `WHERE debug_session_id = …`; nearly every row has none.
# An issue's bundle is `WHERE debug_session_id = …`; nearly every row has none.
Index(
"ix_log_events_debug_session",
"debug_session_id",
postgresql_where="debug_session_id IS NOT NULL",
),
# A run's lines are `WHERE run_id = …`; request lines carry none.
Index("ix_log_events_run", "run_id", postgresql_where="run_id IS NOT NULL"),
)
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
@@ -620,12 +634,16 @@ class LogEvent(Base):
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
user_id: Mapped[int | None] = mapped_column(Integer)
debug_session_id: Mapped[str | None] = mapped_column(String(64))
# A DOWNLOAD job's id (download_jobs) — not a background run, which is `run_id`.
job_id: Mapped[int | None] = mapped_column(Integer)
# The background run (job_runs) that logged it. No foreign key, as `user_id`: the writer runs
# behind the event, and a run reaped meanwhile must not fail the batch its lines travel in.
run_id: Mapped[int | None] = mapped_column(BigInteger)
version: Mapped[str | None] = mapped_column(String(64))
class DebugSession(Base):
"""One recording: Start → reproduce → Stop (E16, app/debugsession.py). While it is open, every
"""One recording: Start → reproduce → Stop (app/debugsession.py). While it is open, every
request of its user that names it (`X-Debug-Session`) is logged at `level` and tagged with `id`,
and the issue it ends in is built from the rows carrying that tag. `until` is the hard cap the
server set at Start; an open session past it is over whether or not the sweep has closed it yet.
@@ -650,12 +668,12 @@ class DebugSession(Base):
# app/debugsession.py `EndReason`: stopped by the user, or past `until`.
ended_reason: Mapped[str | None] = mapped_column(String(16))
issue_url: Mapped[str | None] = mapped_column(String(512))
# The issue being written from it (E16 S3; app/debugsession.py `save_draft` holds its shape).
# The issue being written from it (app/debugsession.py `save_draft` holds its shape).
draft: Mapped[dict | None] = mapped_column(JSONB)
class DebugScreenshot(Base):
"""A screenshot attached to a debug session (E16) — pasted, dropped, or captured from the pill.
"""A screenshot attached to a debug session — pasted, dropped, or captured from the pill.
Stored in the row: bounded per session (app/debugsession.py), and gone with it by cascade."""
__tablename__ = "debug_screenshots"
@@ -685,15 +703,44 @@ class AppState(Base):
setup_token_hash: Mapped[str | None] = mapped_column(String(64))
class SchedulerSetting(Base):
"""Admin override for a scheduler job's run interval (minutes). One row per job id;
absent = use the env/config default. DB-backed so it's editable from the admin UI at
runtime without a redeploy (env stays a fallback)."""
class JobRun(Base):
"""One run of a background job — scheduled, an admin's "run now", or a user's ad-hoc work on one
target — written when it starts and again when it ends (app/jobruns.py), each in its own
committed session, so the history survives the job's own rollback and a restart. A row left
`running` by a process that died is set `interrupted` at the next startup.
__tablename__ = "scheduler_settings"
`quota_units` is what the run spent of the YouTube budget: the `quota_events` rows carrying its
id, summed when it ends. `summary` is the English one-line fallback; `result` / `failure` are the
same outcome as data, which the client names in the reader's language."""
job_id: Mapped[str] = mapped_column(String(40), primary_key=True)
interval_minutes: Mapped[int] = mapped_column(Integer)
__tablename__ = "job_runs"
__table_args__ = (
Index("ix_job_runs_job_started", "job", "started_at"),
CheckConstraint(f"status IN ({sql_in(RunStatus)})", name="ck_job_runs_status"),
CheckConstraint(f"trigger IN ({sql_in(RunTrigger)})", name="ck_job_runs_trigger"),
# Only a run still going has no end.
CheckConstraint(
f"(status = '{RunStatus.RUNNING}') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
job: Mapped[str] = mapped_column(String(40))
trigger: Mapped[str] = mapped_column(String(16))
# Who started it: the admin of a manual run, the user of an ad-hoc one; NULL = the schedule.
actor_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="SET NULL"), index=True
)
# What an ad-hoc run worked on (a channel id); NULL for the recurring jobs.
target: Mapped[str | None] = mapped_column(String(128))
status: Mapped[str] = mapped_column(String(16))
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
summary: Mapped[str | None] = mapped_column(Text)
result: Mapped[dict | None] = mapped_column(JSONB)
error: Mapped[str | None] = mapped_column(Text)
failure: Mapped[dict | None] = mapped_column(JSONB)
quota_units: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class SystemConfig(Base):
@@ -814,44 +861,6 @@ class Notification(Base):
)
class Message(Base):
"""A message in the notification module (phase 2).
Two kinds share this table:
- `kind="user"`: a private, END-TO-END ENCRYPTED direct message between two users. The
server only ever stores `ciphertext` + `iv` (base64); it never holds the plaintext or
the keys, so not even an admin can read it. A conversation is just the set of messages
between two users — no separate thread entity; queries group by the {sender, recipient}
pair.
- `kind="system"`: a server-authored plaintext message (e.g. the welcome, or future admin
announcements). `sender_id` is NULL and `body` holds the text. These are non-private
boilerplate, readable without any E2EE key setup, and surface as a "Siftlode" conversation.
`read_at` is stamped when the recipient opens the thread (NULL = unread, feeds the nav
badge). The recipient FK cascades (and sender, for user messages) so a GDPR account erasure
removes that user's messages outright."""
__tablename__ = "messages"
id: Mapped[int] = mapped_column(primary_key=True)
kind: Mapped[str] = mapped_column(String(16), default="user", server_default="user")
# NULL for system messages (no human sender).
sender_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
)
recipient_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
)
# System messages carry plaintext `body`; user messages carry `ciphertext`+`iv` (base64).
body: Mapped[str | None] = mapped_column(Text)
ciphertext: Mapped[str | None] = mapped_column(Text)
iv: Mapped[str | None] = mapped_column(String(32))
read_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), index=True
)
class MediaAsset(Base, TimestampMixin):
"""A physical downloaded media file — the SHARED cache layer of the download center.
@@ -861,12 +870,13 @@ class MediaAsset(Base, TimestampMixin):
links to this ready asset instantly — no re-download, no cost. Only the SOURCE download is
shared; per-user edited derivatives (phase 2) are never cached here.
`rel_path` is relative to `settings.download_root` (the mount), so the physical filename is
`rel_path` is relative to `settings.files_root` (the mount), so the physical filename is
system-decided and bound to the source id — never renamed (the user's custom name is display
metadata on `download_job`, applied only when downloading to their own machine). The naming
fields (`title`/`uploader`/`upload_date`) are denormalized so the Plex-tree path can be built
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here)
plus `last_access_at`/`expires_at` drive retention + LRU eviction in the GC job."""
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here) frees
the file with its last holder, and `last_access_at` orders the GC's LRU eviction. How long a file
stays is not the asset's: each holder's job carries its own `expires_at` (its library folder's)."""
__tablename__ = "media_assets"
__table_args__ = (
@@ -882,10 +892,10 @@ class MediaAsset(Base, TimestampMixin):
status: Mapped[str] = mapped_column(
String(16), default="pending", server_default="pending", index=True
) # "pending" | "ready" | "error"
rel_path: Mapped[str | None] = mapped_column(Text) # relative to download_root
rel_path: Mapped[str | None] = mapped_column(Text) # relative to files_root
storyboard_path: Mapped[str | None] = mapped_column(Text) # phase-2 filmstrip mosaic
# A locally-generated poster frame (ffmpeg), set only when the source had NO thumbnail so the
# card / watch page / link preview can still show an image. Relative to download_root.
# card / watch page / link preview can still show an image. Relative to files_root.
poster_path: Mapped[str | None] = mapped_column(Text)
size_bytes: Mapped[int | None] = mapped_column(BigInteger)
duration_s: Mapped[int | None] = mapped_column(Integer)
@@ -901,8 +911,6 @@ class MediaAsset(Base, TimestampMixin):
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Set once the GC has warned holders of the upcoming expiry, so it doesn't warn every run.
gc_notified: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Denormalized naming/metadata for the Plex tree + ad-hoc display.
title: Mapped[str | None] = mapped_column(Text)
uploader: Mapped[str | None] = mapped_column(String(255))
@@ -912,13 +920,21 @@ class MediaAsset(Base, TimestampMixin):
upload_date: Mapped[date | None] = mapped_column(Date)
thumbnail_url: Mapped[str | None] = mapped_column(Text)
# yt-dlp's canonical page URL for the source (webpage_url) — the clean "downloaded from"
# reference shown on the Downloads page. Null until the worker extracts it (older/queued rows
# reference shown in the Files library. Null until the worker extracts it (older/queued rows
# fall back to a URL derived from source_kind+source_ref).
source_webpage_url: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
ref_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
last_access_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The file's SHA-256 (hex), so a machine fetching it can check what it got. Hashed while an upload
# streams in, otherwise on the first ask (`routes.files`); NULL = not hashed yet.
sha256: Mapped[str | None] = mapped_column(String(64))
@validates("rel_path")
def _forget_hash(self, _key: str, rel_path: str | None) -> str | None:
# A hash names the bytes at `rel_path`: a new file there (a conversion, a retry) is not them.
self.sha256 = None
return rel_path
class DownloadProfile(Base, TimestampMixin):
@@ -942,12 +958,98 @@ class DownloadProfile(Base, TimestampMixin):
sort_order: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class FileFolder(Base, TimestampMixin):
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
__tablename__ = "file_folders"
__table_args__ = (
UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
Index(
"uq_file_folder_own_name",
"user_id",
func.lower(text("name")),
unique=True,
postgresql_where=text("name IS NOT NULL"),
),
# One Hub folder per user.
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
kind: Mapped[str] = mapped_column(String(16)) # folders.FolderKind
# "site:" + a host name (DNS caps one at 253 characters).
source_key: Mapped[str | None] = mapped_column(String(260))
name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
retention_days: Mapped[int | None] = mapped_column(Integer)
class ApiToken(Base, TimestampMixin):
"""A user's token for a machine to reach their Files library with (`app.apitokens`): sent as
`Authorization: Bearer sfl_…`, honoured on `/api/files` only, and only in the folders its
grants name. Only the SHA-256 of the token is kept — it is shown once, when it is made — plus
its first characters (`prefix`), so the user can tell their tokens apart."""
__tablename__ = "api_tokens"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
name: Mapped[str] = mapped_column(String(80))
prefix: Mapped[str] = mapped_column(String(16))
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) # NULL = never
last_used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The one notice its user gets before it runs out (`apitokens.warn_expiring`).
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class ApiTokenGrant(Base):
"""What one token may do in one folder. A folder with no grant is out of the token's reach."""
__tablename__ = "api_token_grants"
__table_args__ = (
UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
CheckConstraint(
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
token_id: Mapped[int] = mapped_column(ForeignKey("api_tokens.id", ondelete="CASCADE"))
folder_id: Mapped[int] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
can_list: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_read: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_write: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_delete: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
def _ready_on_insert(context) -> datetime | None:
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
done = context.get_current_parameters().get("status") == "done"
return datetime.now(timezone.utc) if done else None
class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
"""A user's request to download one source — the per-user layer over `MediaAsset`.
The worker claims `queued` jobs with ``FOR UPDATE SKIP LOCKED`` and writes live
`progress`/`speed_bps`/`eta_s`/`phase` here (the frontend polls via useLiveQuery — the
worker is a separate process and can't reach the in-process WS registry). `profile_snapshot`
`progress`/`speed_bps`/`eta_s`/`phase` here (the frontend polls via useLiveQuery: there is
no server push channel, and the worker is a separate process anyway). `profile_snapshot`
freezes the spec at enqueue so later edits/deletion of the profile don't change this job.
`display_name` is the user's own name for the file (display + the Content-Disposition on
local download); the on-disk asset is never renamed. `asset_id` links to the shared file
@@ -960,7 +1062,13 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
point at the parent download the clip was cut from; `edit_spec` is the (trim/crop) recipe."""
__tablename__ = "download_jobs"
__table_args__ = (Index("ix_download_jobs_claim", "status", "queue_pos"),)
__table_args__ = (
Index("ix_download_jobs_claim", "status", "queue_pos"),
# Every way a job becomes done stamps when (`folders.stamp_done`, `finalize_if_running`).
CheckConstraint(
"status <> 'done' OR ready_at IS NOT NULL", name="ck_download_jobs_ready_at"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
@@ -969,7 +1077,9 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512))
# "download" (default) or "edit" (a per-user trim/crop derivative of another job).
# "download" (default), "edit" (a per-user trim/crop derivative of another job), "upload" (a
# user's own file, converted by the worker when it is not phone-safe — see `downloads.upload`) or
# "clip" (a range cut straight from a YouTube video or a Plex item — see `downloads.clip`).
job_kind: Mapped[str] = mapped_column(String(16), default="download", server_default="download")
# For edit jobs: the parent download this clip is cut from (SET NULL if the parent is deleted).
source_job_id: Mapped[int | None] = mapped_column(
@@ -978,7 +1088,9 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
source_asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL")
)
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}}.
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
edit_spec: Mapped[dict | None] = mapped_column(JSON)
profile_id: Mapped[int | None] = mapped_column(
ForeignKey("download_profiles.id", ondelete="SET NULL")
@@ -1005,6 +1117,27 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
# The host-specific failure-catalog entry on top of the category (`facebook.login_wall`), or NULL.
error_detail: Mapped[str | None] = mapped_column(String(48)) # see downloads.errors.diagnose
queue_pos: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# The Files library: the folder this file sits in (`downloads.folders.place` at creation; an own
# folder is emptied back into the source folders before it is deleted, hence RESTRICT), whether
# the user keeps it past its folder's retention, and when it goes (stamped once done; NULL =
# never). `expiry_warned` marks the one warning the GC sends before it deletes the file.
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="RESTRICT"), index=True
)
# The origin the file came from (the source folder's key), recorded where it is placed: an edit
# traces its origin through its parent, and that parent may be deleted or expire later — the
# file must still know which source folder it may go back to. NULL only for a file placed before
# the column existed outside a source folder; `folders.source_key_for` works those out.
source_key: Mapped[str | None] = mapped_column(String(260))
kept: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# When the file became done — what the library's "Added" means: a download that failed for days
# and then went through is new to its user, so it sorts as new. `created_at` stays when it was
# asked for (the queue's order). Set again when a file is made anew (a retry, a re-download).
ready_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), default=_ready_on_insert
)
class DownloadShare(Base, TimestampMixin):
@@ -1039,21 +1172,33 @@ class DownloadQuota(Base, UpdatedAtMixin):
class DownloadLink(Base, TimestampMixin):
"""A public capability link to one download — the "share by link" surface.
"""A public capability link to one download, or to one of its owner's own folders — the "share
by link" surface.
Anyone holding the unguessable `token` can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style). Optional per-link controls: `expires_at`, `allow_download`
(stream-only vs downloadable), and a `password_hash` (argon2, like a user password). Revoke =
delete the row. Distinct from `DownloadShare`, which grants a *registered* user access via the
in-app "Shared with me" list."""
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
the folder at the time it is opened. The token itself is never stored as written: a link is
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
grants a *registered* user access via the in-app "Shared with me" list."""
__tablename__ = "download_links"
__table_args__ = (
# A link opens exactly one thing: a file or a folder.
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int] = mapped_column(
job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
token: Mapped[str] = mapped_column(String(64), unique=True)
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
password_hash: Mapped[str | None] = mapped_column(Text)
@@ -1094,29 +1239,6 @@ class DownloadCredential(Base, TimestampMixin, UpdatedAtMixin):
cookies_enc: Mapped[str] = mapped_column(Text) # Fernet ciphertext — never returned/logged
class MessageKey(Base, TimestampMixin):
"""A user's end-to-end-encryption key material for direct messaging (phase 2).
One row per user. `public_key` (base64 SPKI, ECDH P-256) is shared with others so they can
derive the pairwise conversation key. `wrapped_private_key` is the user's private key
encrypted IN THE BROWSER with a key derived (PBKDF2) from a message passphrase the server
never sees — so the server stores only an opaque blob it cannot unwrap. `salt`/`wrap_iv`
parameterise that derivation; `key_check` is a small ciphertext the client decrypts to
verify a passphrase on unlock. The server is purely a key directory + blob store; it can
never read any message."""
__tablename__ = "message_keys"
user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), primary_key=True
)
public_key: Mapped[str] = mapped_column(Text)
wrapped_private_key: Mapped[str] = mapped_column(Text)
salt: Mapped[str] = mapped_column(String(64))
wrap_iv: Mapped[str] = mapped_column(String(32))
key_check: Mapped[str] = mapped_column(Text)
# --- Plex integration (optional) -------------------------------------------------------------
# A dedicated, self-contained catalog mirrored from a locally-reachable Plex server. Playback is
# from the LOCAL physical file (Plex is metadata only); per-user watch state lives in plex_states
@@ -1137,7 +1259,7 @@ class PlexLibrary(Base, TimestampMixin, UpdatedAtMixin):
kind: Mapped[str] = mapped_column(String(16)) # "movie" | "show"
enabled: Mapped[bool] = mapped_column(Boolean, default=True, server_default="true")
synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# HU-overlay support (E5 S5). `metadata_lang` tags the section's metadata language. An OVERLAY
# HU-overlay support. `metadata_lang` tags the section's metadata language. An OVERLAY
# library (overlay_of set) is the Hungarian-metadata twin of another section: it is never browsable
# on its own — its titles' HU fields are written onto the matched EN rows (paired by `guid`).
metadata_lang: Mapped[str] = mapped_column(String(8), default="en", server_default="en")
@@ -1166,7 +1288,7 @@ class PlexShow(Base, TimestampMixin, UpdatedAtMixin):
ForeignKey("plex_libraries.id", ondelete="CASCADE"), index=True
)
# Canonical external id (e.g. "tmdb:12345") — the key that pairs this EN row with its HU-overlay
# twin during sync (E5 S5). Indexed for the guid->row overlay match.
# twin during sync. Indexed for the guid->row overlay match.
guid: Mapped[str | None] = mapped_column(String(64), index=True)
title: Mapped[str] = mapped_column(String(512))
summary: Mapped[str | None] = mapped_column(Text)
@@ -1196,7 +1318,7 @@ class PlexShow(Base, TimestampMixin, UpdatedAtMixin):
) # cast+directors, folded into search_vector
# Set once the show's FULL cast has been rich-fetched by the backfill; NULL = still the listing top-3.
people_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Rich-fetch-only detail fields, persisted by the same backfill that fills the full cast (E5 S5b) so
# Rich-fetch-only detail fields, persisted by the same backfill that fills the full cast so
# the info page serves from this row instead of a live plex.metadata() call. NULL until enriched.
imdb_rating: Mapped[float | None] = mapped_column(
Float
@@ -1206,7 +1328,7 @@ class PlexShow(Base, TimestampMixin, UpdatedAtMixin):
full_cast: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # ordered [{name, role, thumb}]
# --- TMDB enrichment (E5 S2): see the identical block on plex_items for the full rationale. Additive;
# --- TMDB enrichment: see the identical block on plex_items for the full rationale. Additive;
# served from the row; NULL until enriched. (`budget`/`revenue` stay NULL for shows — TV has none.) ---
tmdb_id: Mapped[int | None] = mapped_column(Integer)
tmdb_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
@@ -1224,10 +1346,10 @@ class PlexShow(Base, TimestampMixin, UpdatedAtMixin):
production_countries: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
recommendations: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
backdrops: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
# Hungarian-metadata twin fields, written by the overlay sync (E5 S5). NULL = no HU twin matched,
# Hungarian-metadata twin fields, written by the overlay sync. NULL = no HU twin matched,
# so serving falls back to the EN columns. `people_text` is language-neutral, reused for both vectors.
title_hu: Mapped[str | None] = mapped_column(String(512))
tagline_hu: Mapped[str | None] = mapped_column(Text) # the HU overlay's tagline (E30)
tagline_hu: Mapped[str | None] = mapped_column(Text) # the HU overlay's tagline
summary_hu: Mapped[str | None] = mapped_column(Text)
genres_hu: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
search_vector: Mapped[object | None] = mapped_column(
@@ -1289,7 +1411,7 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
)
kind: Mapped[str] = mapped_column(String(16), index=True) # "movie" | "episode"
# Canonical external id (e.g. "tmdb:12345") — pairs a MOVIE row with its HU-overlay twin during
# sync (E5 S5); indexed for the guid->row match. Episodes carry it too but it's unused for them.
# sync; indexed for the guid->row match. Episodes carry it too but it's unused for them.
guid: Mapped[str | None] = mapped_column(String(64), index=True)
show_id: Mapped[int | None] = mapped_column(
ForeignKey("plex_shows.id", ondelete="CASCADE"), index=True
@@ -1316,7 +1438,7 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
# Browser-playability class from ffprobe: "direct" | "remux" | "transcode" (None = unprobed).
playable: Mapped[str | None] = mapped_column(String(12), index=True)
probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Transcode HW-capability facet (E7 playback-class filter). NULL for non-transcode items; for a
# Transcode HW-capability facet (the playback-class filter). NULL for non-transcode items; for a
# `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
# AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
# breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
@@ -1332,8 +1454,6 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
# bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are
# probed (the rest are SW-only by codec and need no bit-depth); see `_probe_media_batch`.
media_probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Intro/credit markers: [{"type": "intro"|"credits", "start_s": int, "end_s": int}, ...]
markers: Mapped[list | None] = mapped_column(JSON)
added_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
# Filterable / orderable metadata mirrored from the Plex section listing (no extra API calls;
# `rating` = Plex audienceRating, which is ~the IMDb score). The arrays back genre/person/studio
@@ -1352,7 +1472,7 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
people_text: Mapped[str | None] = mapped_column(Text)
# Set once the movie's FULL cast has been rich-fetched by the backfill; NULL = still the listing top-3.
people_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Rich-fetch-only detail fields, persisted by the same backfill that fills the full cast (E5 S5b) so
# Rich-fetch-only detail fields, persisted by the same backfill that fills the full cast so
# the info page serves from this row instead of a live plex.metadata() call. NULL until enriched.
imdb_rating: Mapped[float | None] = mapped_column(
Float
@@ -1362,7 +1482,7 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
full_cast: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # ordered [{name, role, thumb}]
# --- TMDB enrichment (E5 S2): matched by the canonical `guid` ("tmdb:12345"), filled by the TMDB
# --- TMDB enrichment: matched by the canonical `guid` ("tmdb:12345"), filled by the TMDB
# backfill job, SERVED FROM THIS ROW (never a live TMDB call). Additive — imdb_rating/rating
# above stay Plex-sourced; `tmdb_rating` is the distinct TMDb community score. NULL until enriched;
# `tmdb_enriched_at` is the "tried" watermark (stamped on a definitive no-match too, so an
@@ -1385,11 +1505,11 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
recommendations: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
backdrops: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # TMDB backdrop file_paths [str] (crossfade pool, S8)
# Hungarian-metadata twin fields, written by the overlay sync (E5 S5). NULL = no HU twin matched
) # TMDB backdrop file_paths [str] (crossfade pool)
# Hungarian-metadata twin fields, written by the overlay sync. NULL = no HU twin matched
# (serving falls back to the EN columns). `people_text` is language-neutral, reused for both vectors.
title_hu: Mapped[str | None] = mapped_column(String(512))
tagline_hu: Mapped[str | None] = mapped_column(Text) # the HU overlay's tagline (E30)
tagline_hu: Mapped[str | None] = mapped_column(Text) # the HU overlay's tagline
summary_hu: Mapped[str | None] = mapped_column(Text)
genres_hu: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
search_vector: Mapped[object | None] = mapped_column(
@@ -1428,15 +1548,15 @@ class PlexPerson(Base, TimestampMixin, UpdatedAtMixin):
id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str] = mapped_column(String(255), unique=True, index=True)
photo_thumb: Mapped[str | None] = mapped_column(Text) # proxied /person-image URL, or None
# TMDB person id, linked by name from a title's TMDB credits (E5 S2 lays the foundation; the /person
# biography fetch + person-page render land in S6). NULL until a TMDB-enriched title names this person.
# TMDB person id, linked by name from a title's TMDB credits (the /person
# biography fetch builds on it). NULL until a TMDB-enriched title names this person.
tmdb_person_id: Mapped[int | None] = mapped_column(Integer)
# --- TMDB person enrichment (E5 S6): fetched from /person/{tmdb_person_id} by the people_enrich job
# (E30 S6), or by the first person-page open the job hasn't reached, then SERVED FROM THE ROW. `tmdb_enriched_at` is the watermark (set even on a
# --- TMDB person enrichment: fetched from /person/{tmdb_person_id} by the people_enrich job,
# or by the first person-page open the job hasn't reached, then SERVED FROM THE ROW. `tmdb_enriched_at` is the watermark (set even on a
# no-op fetch so it isn't retried every open). `known_for` = top combined-credits [{tmdb_id,
# media_type, title}], resolved to library rows by guid at serve time (like recommendations). ---
biography: Mapped[str | None] = mapped_column(Text)
# TMDB's Hungarian biography (E30 S4), from the same request's `translations`; a Hungarian reader is
# TMDB's Hungarian biography, from the same request's `translations`; a Hungarian reader is
# served it, and the instance-language `biography` when TMDB has none. Birthplace, aliases and
# keywords have no Hungarian on TMDB, so they stay in the instance language.
biography_hu: Mapped[str | None] = mapped_column(Text)
@@ -1477,7 +1597,7 @@ class PlexCollection(Base, TimestampMixin, UpdatedAtMixin):
class PlexState(Base, UpdatedAtMixin):
"""Per-user watch state for a Plex item — mirrors VideoState. Lives in Siftlode's own DB so
it works for users WITHOUT a Plex account (the whole point of the access-layer design).
`synced_to_plex` marks whether this state was pushed back to a linked Plex account (P5)."""
`synced_to_plex` marks whether this state was pushed back to a linked Plex account."""
__tablename__ = "plex_states"
__table_args__ = (UniqueConstraint("user_id", "item_id", name="uq_plex_user_item"),)
@@ -1556,7 +1676,7 @@ class PlexPlaylistItem(Base):
class ComfortPin(Base, TimestampMixin):
"""One entry on a user's comfort shelf — the anti-paralysis fallback: "just put on X" (E24).
"""One entry on a user's comfort shelf — the anti-paralysis fallback: "just put on X".
Deliberately NOT a boolean on `VideoState`/`PlexState`, and NOT a polymorphic `(kind, ref)` string
pair. A flag cannot ORDER a shelf and cannot hold a channel or a show — and comfort media is
+1 -1
View File
@@ -10,4 +10,4 @@ from pathlib import Path
from app.config import settings
CACHE_DIR = Path(settings.download_root) / ".plex-img-cache"
CACHE_DIR = Path(settings.files_root) / ".plex-img-cache"
+19 -3
View File
@@ -3,9 +3,24 @@ playlists, and detail views render. Extracted from routes/plex/browse.py so the
lives beside the other Plex read-model modules (detail.py, query.py) rather than inside the
router. Pure functions — no DB, no request state."""
from typing import NamedTuple
from app.models import PlexItem, PlexShow, PlexState
class ShowState(NamedTuple):
"""A show's watch state rolled up from its episodes, as its card carries it. `resumable`: an unwatched
episode has a resume point, the one the card's Continue resumes and its Restart replays from 0. A
show part watched with its next episode untouched has none: Continue and Restart would both play
that episode from its start, so the card offers Play alone."""
status: str
resumable: bool
SHOW_NEW = ShowState("new", False)
# A card's wide backdrop, for a rail's landscape tile or a crossfading hero — but ONLY when the item
# actually has art. `thumb` is always emitted (its endpoint placeholders a missing poster), but an
# `art` URL for an item with no `art_key` would resolve to a broken/blank image on every card that
@@ -25,7 +40,7 @@ def norm_lang(lang: str | None) -> str:
return "hu" if lang == "hu" else "en"
# Every card speaks the reader's language (E30): `lang="hu"` serves the HU overlay twin's title and
# Every card speaks the reader's language: `lang="hu"` serves the HU overlay twin's title and
# summary, falling back to EN where a row has none — the same rule as browse.unified_library's
# `title_col`/`summary_col`. These rails once served English to a Hungarian reader.
def display_title(row: PlexItem | PlexShow, lang: str) -> str:
@@ -54,7 +69,7 @@ def movie_card(it: PlexItem, st: PlexState | None, *, lang: str) -> dict:
}
def show_card(sh: PlexShow, status: str = "new", *, lang: str) -> dict:
def show_card(sh: PlexShow, state: ShowState = SHOW_NEW, *, lang: str) -> dict:
return {
"id": sh.rating_key,
"type": "show",
@@ -66,7 +81,8 @@ def show_card(sh: PlexShow, status: str = "new", *, lang: str) -> dict:
"art": _art_url(sh.rating_key, sh.art_key),
"season_count": sh.child_count,
# Aggregate watch-state across the show's episodes (new | in_progress | watched) → grid badge.
"status": status,
"status": state.status,
"resumable": state.resumable,
}
+16 -12
View File
@@ -2,7 +2,7 @@
Server-side ONLY: uses the admin ``X-Plex-Token`` from sysconfig. Playback never goes through
this client — Siftlode plays the LOCAL physical file (see app.plex.paths). Plex is used purely for
metadata, image proxying, and (P5) optional watch-state write-back. Requests JSON via the
metadata, image proxying, and optional watch-state write-back. Requests JSON via the
``Accept: application/json`` header (Plex serves XML by default).
"""
@@ -65,7 +65,7 @@ class PlexClient:
except ValueError as e:
raise PlexError("Plex returned a non-JSON response") from e
# --- Connectivity + sections (P0) ---
# --- Connectivity + sections ---
def server_info(self) -> dict:
"""Root identity (friendlyName / version) — used by the admin 'Test connection'."""
return self._get("/")
@@ -75,7 +75,7 @@ class PlexClient:
mc = self._get("/library/sections")
return mc.get("Directory", []) or []
# --- Catalog mirror (P1) ---
# --- Catalog mirror ---
def section_items(
self, key: str, item_type: int, start: int = 0, size: int = 200
) -> tuple[list[dict], int | None]:
@@ -88,14 +88,14 @@ class PlexClient:
"type": item_type,
# includeGuids=1 carries each item's Guid array (tmdb/imdb/tvdb ids) in the cheap
# listing, so the sync can capture the canonical guid without a per-item rich fetch —
# the pairing key for the HU overlay (E5 S5).
# the pairing key for the HU overlay.
"includeGuids": 1,
"X-Plex-Container-Start": start,
"X-Plex-Container-Size": size,
},
)
# `totalSize` only: `size` is THIS page's count, and read as the total it made a single
# page look like the whole section — "complete" to a prune or a reconcile (E32 review).
# page look like the whole section — "complete" to a prune or a reconcile.
total = mc.get("totalSize")
return mc.get("Metadata", []) or [], None if total is None else int(total)
@@ -137,12 +137,12 @@ class PlexClient:
mc = self._get(f"/library/collections/{rating_key}/children")
return mc.get("Metadata", []) or []
# --- Collection editing (P2 — write-back to Plex; all shared/global, admin-gated in the routes) ---
# --- Collection editing (write-back to Plex; all shared/global, admin-gated in the routes) ---
@property
def machine_id(self) -> str:
"""Server machineIdentifier — needed in the `uri` of collection add/create calls. Cached.
Without one the uri would read `server:///…`, a malformed write Plex answers in its own way;
it is refused here, by name (E32 S2)."""
it is refused here, by name."""
if not getattr(self, "_machine_id", None):
mid = self._get("/identity").get("machineIdentifier")
if not mid:
@@ -167,12 +167,16 @@ class PlexClient:
keys = ",".join(str(k) for k in rating_keys)
return f"server://{self.machine_id}/com.plexapp.plugins.library/library/metadata/{keys}"
def create_collection(self, section_key: str, title: str, item_rating_key: str) -> dict:
"""Create a (dumb) collection seeded with one item; returns its Metadata (incl. ratingKey)."""
def create_collection(
self, section_key: str, title: str, item_rating_key: str, item_type: int = 1
) -> dict:
"""Create a (dumb) collection seeded with one item; returns its Metadata (incl. ratingKey).
`item_type` is the Plex metadata type of its members — 1 movie, 2 show. Measured: a collection
made as type 1 in a TV section came out EMPTY with a show seed, and refused the show after."""
uri = quote(self._metadata_uri(item_rating_key), safe="")
mc = self._write(
"POST",
f"/library/collections?type=1&smart=0&sectionId={section_key}"
f"/library/collections?type={item_type}&smart=0&sectionId={section_key}"
f"&title={quote(title, safe='')}&uri={uri}",
)
items = mc.get("Metadata", []) or []
@@ -206,7 +210,7 @@ class PlexClient:
items = mc.get("Metadata", []) or []
return items[0] if items else None
# --- Watch-state write-back (P5 Phase B — Siftlode→Plex push; acts on the token's account) ---
# --- Watch-state write-back (Siftlode→Plex push; acts on the token's account) ---
_LIBRARY_IDENTIFIER = "com.plexapp.plugins.library"
def scrobble(self, rating_key: str) -> None:
@@ -243,7 +247,7 @@ class PlexClient:
f"&state={state}&time={int(time_ms)}&duration={int(duration_ms)}",
)
# --- Watch-state read for incremental sync (P5 Phase C) ---
# --- Watch-state read for incremental sync ---
def accounts(self) -> list[dict]:
"""Server accounts (the owner + any managed/home users): id, name. Used to resolve the
owner's accountID so the shared watch-history feed can be filtered to just their views."""
+3 -2
View File
@@ -1,5 +1,5 @@
"""Parse the "info page" rich metadata out of a full Plex metadata dict, and persist the subset that
the detail endpoints serve from the DB row (E5 S5b).
the detail endpoints serve from the DB row.
Shared by the live info-page path (routes.plex.browse) and the background rich re-sync (plex.sync) so
the two never diverge — the same one-parser rule that keeps `people.py`'s cast handling consistent.
@@ -7,6 +7,7 @@ the two never diverge — the same one-parser rule that keeps `people.py`'s cast
(genres/directors/studio/content_rating already land from the cheap listing at `_apply_facet_fields`)."""
from app.plex import people
from app.plex.plaintext import plain_text
def rich_detail(meta: dict) -> dict:
@@ -42,7 +43,7 @@ def rich_detail(meta: dict) -> dict:
"genres": [g.get("tag") for g in (meta.get("Genre") or []) if g.get("tag")][:8],
"directors": [d.get("tag") for d in (meta.get("Director") or []) if d.get("tag")][:4],
"studio": meta.get("studio"),
"tagline": meta.get("tagline"),
"tagline": plain_text(meta.get("tagline")),
"cast": people.cast_from_meta(meta),
}
+1 -1
View File
@@ -1,4 +1,4 @@
"""One Hungarian word per Plex genre (E30).
"""One Hungarian word per Plex genre.
Plex's Hungarian agent returns a MIXED vocabulary: some genres come back translated ("Dráma",
"Vígjáték"), others stay English ("Action", "Adventure", "Biography"), and a title with no HU twin has
+1 -1
View File
@@ -26,7 +26,7 @@ _CT_EXT = {ct: ext for ext, ct in _EXT_CT}
# Target box per variant — Plex resizes to these server-side. The full poster (400px) covers a detail
# page's ≤224px poster at 2× DPR, the full art (1280px) a hero backdrop. A card is much smaller: a grid
# poster is ~144px and a shelf's landscape art ~212px, so the `_sm` pair serves those at 1.33× DPR
# (E31 S4e: cards fetched the 400 poster, 42 KB, and the 1280 art for a 212px picture). The browser
# (cards used to fetch the 400 poster, 42 KB, and the 1280 art for a 212px picture). The browser
# picks between a pair itself (`srcset`, built by the frontend's `lib/plexImage` cardImage, whose
# widths plexImage.contract.test holds to these).
SIZES = {
+7 -6
View File
@@ -1,5 +1,5 @@
"""The HU overlay: a Hungarian Plex section written onto its English twin's rows (E30), split out of
`plex/sync.py` (E32 S2), which runs it once per paired section during the full sync.
"""The HU overlay: a Hungarian Plex section written onto its English twin's rows, split out of
`plex/sync.py`, which runs it once per paired section during the full sync.
The pairing is admin-configured (`plex_overlay_map`, ``huKey:enKey,...``). An overlay section creates
no browsable rows of its own; it only fills the `*_hu` columns of the EN rows it matches by guid.
@@ -17,6 +17,7 @@ from app.models import PlexItem, PlexLibrary, PlexShow
from app.plex import sync as plex_sync
from app.plex.client import PlexClient
from app.plex.genres import hu_genres
from app.plex.plaintext import plain_text
log = logs.get_logger(__name__)
@@ -41,7 +42,7 @@ def sync_overlay(
"""Write an overlay section's Hungarian metadata onto its EN twin's rows, matched by guid. Creates
NO browsable rows — an overlay library is never listed on its own. Only title/summary/tagline/genres
differ by language; the language-neutral people_text feeds both search vectors. A show section's
episodes are overlaid too (E30): the HU section lists every episode with its Hungarian title and
episodes are overlaid too: the HU section lists every episode with its Hungarian title and
summary, and the episode rows live in the EN show library.
This is a full RECONCILE: after a COMPLETE HU listing every guid-bearing EN row's HU fields are
@@ -109,8 +110,8 @@ def _overlay_pass(
for g, meta in hu_by_guid.items():
targets = by_guid[g]
title_hu = (meta.get("title") or "").strip() or None
summary_hu = meta.get("summary")
tagline_hu = (meta.get("tagline") or "").strip() or None
summary_hu = plain_text(meta.get("summary"))
tagline_hu = plain_text(meta.get("tagline"))
genres_hu = hu_genres(plex_sync._tags(meta, "Genre"))
for row in targets:
row.title_hu = title_hu
@@ -141,7 +142,7 @@ def _clear_hu(by_guid: dict, episodes: bool) -> None:
cheap. The genres are the exception: every row gets Hungarian ones, from its HU twin or, lacking
one, from its own English list — otherwise the HU genre list mixed "Drama" (no twin) with "Dráma".
Only after a COMPLETE listing: one a Plex hiccup cut short would clear the HU text of every title
it failed to reach, until the next full sync (E32 S2, the watch reconcile's twin)."""
it failed to reach, until the next full sync (the watch reconcile's twin)."""
for group in by_guid.values():
for row in group:
row.title_hu = row.summary_hu = None
+2 -2
View File
@@ -160,7 +160,7 @@ def people_by_credits(
return [r[0] for r in rows]
# --- Portraits (E30 S6) ---------------------------------------------------------------------------
# --- Portraits ---------------------------------------------------------------------------
# A Plex CDN headshot is a full-size original (~170 KB on average, measured). Plex's photo transcoder
# fits it into this box (~13 KB), which covers the largest place a portrait is drawn (the person page's
# 6rem avatar) at 2x density and a raised text size. The box is part of the cache key, like a poster's:
@@ -226,7 +226,7 @@ _DEAD = frozenset({404, 410})
def warm_portraits(db: Session, limit: int) -> dict:
"""Cache up to `limit` not-yet-cached headshots, the most-credited people first (E30 S6), so a
"""Cache up to `limit` not-yet-cached headshots, the most-credited people first, so a
person's face comes from local disk. Needs Plex (the resize) and the Plex module on; without them
nothing is prefetched — full-size originals for every person would cost gigabytes. A headshot Plex
answers "not found" for is marked and left for PORTRAIT_RETRY; Plex failing in a row ends the run."""
+51
View File
@@ -0,0 +1,51 @@
"""Plex text fields as plain text.
Plex hands a summary over as the agent wrote it, and some agents (a Hungarian TV listing, for one)
write HTML: `<p>…</p>` paragraphs and `&aacute;`-style entities. The app renders summaries as text,
so the tags showed literally on the info page and the hover card, and a card's 300-character cut
could land inside one. Every summary and tagline is stored through `plain_text`, and the migration
that cleaned the existing rows calls it too, so the two cannot disagree.
"""
import html
import re
from html.entities import html5
_BREAK = re.compile(r"<br\s*/?>", re.IGNORECASE)
# A paragraph or block boundary: a blank line between the two. Only a tag name ends the match, so a
# bare "<" in prose ("a < b", "<3") is left alone.
_BLOCK = re.compile(r"</?(?:p|div|li|ul|ol|h[1-6]|blockquote)\b[^>]*>", re.IGNORECASE)
_TAG = re.compile(r"</?[a-zA-Z][^>]*>")
_BLANKS = re.compile(r"[ \t]*\n[ \t]*\n\s*")
# A well-formed character reference only. `html.unescape` alone also decodes the legacy forms that
# have no semicolon, so prose like "AT&notice" or "R&copy" came out as "AT¬ice" and "R©". A named
# one counts only when its whole name is an entity: for "&notice;" `html.unescape` still falls back
# to the legacy "&not" inside it.
_REF = re.compile(r"&(?:#[0-9]+|#[xX][0-9a-fA-F]+|[A-Za-z][A-Za-z0-9]*);")
def _decode(m: re.Match[str]) -> str:
ref = m.group()
if ref[1] != "#":
return html.unescape(ref) if ref[1:] in html5 else ref
if ref[2] in "xX":
return html.unescape(ref)
# `int()` refuses a decimal run past 4300 digits, and any number past seven significant digits is
# beyond U+10FFFF, which the parser turns into U+FFFD anyway.
digits = ref[2:-1].lstrip("0") or "0"
return "�" if len(digits) > 7 else html.unescape(f"&#{digits};")
def plain_text(value: str | None) -> str | None:
"""`value` with its markup gone: line breaks kept as newlines, paragraphs as a blank line,
every other tag dropped and entities decoded. None or an all-blank result stays None."""
if value is None:
return None
if "<" in value or "&" in value:
value = _BREAK.sub("\n", value)
value = _BLOCK.sub("\n\n", value)
value = _TAG.sub("", value)
value = _REF.sub(_decode, value)
value = _BLANKS.sub("\n\n", value)
return value.strip() or None
+10 -10
View File
@@ -31,9 +31,9 @@ PLEX_SORT_KEYS: tuple[str, ...] = (
"title",
)
# --- Language-aware column pickers (E5 S6) ----------------------------------
# --- Language-aware column pickers ----------------------------------
#
# The HU-overlay twin (E5 S5) stores Hungarian metadata in `*_hu` columns on the SAME EN row. When
# The HU-overlay twin stores Hungarian metadata in `*_hu` columns on the SAME EN row. When
# the app language is `hu` we serve those, falling back to the EN column where a row has no HU twin.
# Two flavours of picker, on purpose:
# * DISPLAY / FILTER (`title_col`/`genres_col`): `coalesce(*_hu, *)` — an EN fallback so a row with
@@ -41,8 +41,8 @@ PLEX_SORT_KEYS: tuple[str, ...] = (
# HU summary directly off the loaded row, so no `summary_col` SQL helper is needed.)
# * SEARCH (`vector_col` + `field_recheck`): the raw `*_hu` column coalesced to '' — MIRRORING the
# generated `search_vector_hu` definition (which uses `coalesce(title_hu, '')`, NOT an EN
# fallback), so the field recheck stays a strict subset of the GIN vector prefilter (the S2
# invariant). `people_text` is language-neutral and shared by both vectors.
# fallback), so the field recheck stays a strict subset of the GIN vector prefilter (the
# field-search invariant). `people_text` is language-neutral and shared by both vectors.
def title_col(model, lang: str):
@@ -216,7 +216,7 @@ def meta_filter_conds(model, p: dict, lang: str = "en") -> list:
stds = split_csv(p.get("studios"))
if stds: # OR
conds.append(model.studio.in_(stds))
for kw in split_csv(p.get("keywords")): # AND — TMDB keyword chips from the detail page (E5 S5)
for kw in split_csv(p.get("keywords")): # AND — TMDB keyword chips from the detail page
conds.append(model.keywords.contains([kw]))
csel = split_csv(p.get("collections"))
if (
@@ -232,7 +232,7 @@ def apply_meta_filters(q, model, p: dict, lang: str = "en"):
return q.where(*conds) if conds else q
# --- Playback-class filter (E7) — MOVIE-ONLY (a show is an aggregate with no single class) ---------
# --- Playback-class filter — MOVIE-ONLY (a show is an aggregate with no single class) ---------
# The 4 classes the feed filter + facet expose, derived from the stored `playable` + `transcode_hw`.
# ONE definition (this SQL CASE) so the filter WHERE and the facet counts can't drift: `playable_conds`
# filters on it, `facets` groups by it. transcode_hw IS NOT True groups NULL (unprobed) with software.
@@ -278,7 +278,7 @@ class LibraryFilters:
actors: str | None = None
actor_mode: str = "any" # how multiple actors combine: any (OR, default) | all (AND)
studios: str | None = None
keywords: str | None = None # TMDB keyword chips (E5 S5), CSV; language-neutral
keywords: str | None = None # TMDB keyword chips, CSV; language-neutral
collections: str | None = None # CSV of collection rating_keys
collection_mode: str = "any" # how multiple collections combine: any (OR, default) | all (AND)
playable: str | None = None # CSV of playback classes: direct|remux|transcode_hw|transcode_sw
@@ -311,7 +311,7 @@ class LibraryFilters:
# watch state. The SAME predicate has to be applied in two places — `unified_library` (the grid) and
# `facets.wsq` (so the facet counts reflect exactly the visible set) — and they MUST stay identical:
# a hidden movie kept out of the grid must not inflate the facets either. These helpers are the one
# home for that predicate so the two call sites can't drift (E3 S1; pinned by test_plex_facets_grid_
# home for that predicate so the two call sites can't drift (pinned by test_plex_facets_grid_
# parity). Callers pass the already-joined state expressions, so this module stays model-agnostic:
# * movies — `st` is an aliased PlexState outer-joined on (item_id, user_id).
# * shows — `tot`/`wat`/`inp` are the coalesced totals from `rollup.agg_subquery`.
@@ -367,9 +367,9 @@ def apply_show_state_filter(query, tot, wat, inp, show: str):
def content_libs(db: Session) -> tuple[list[int], list[int]]:
"""The BROWSABLE content libraries, partitioned into (movie_lib_ids, show_lib_ids) — the single
source of the browse scope. Excludes HU overlay libraries (`overlay_of` set): those are never
browsed on their own, they only lend their Hungarian metadata to the EN twin's rows (E5 S5).
browsed on their own, they only lend their Hungarian metadata to the EN twin's rows.
Lives here, in the query layer, rather than in `routes/plex/_shared.py` where it started: E24's
Lives here, in the query layer, rather than in `routes/plex/_shared.py` where it started: the
Home surface needs the same scope, and a domain package importing from `routes/` would invert the
one-way dependency CLAUDE.md draws. It never touched request state, so nothing was lost by the
move."""
+11 -5
View File
@@ -9,14 +9,15 @@ from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.orm import Session, aliased
from app.models import PlexItem, PlexState
from app.plex.cards import ShowState
# Sort sentinel for a rail row whose timestamp is missing, so `key=lambda r: r[0] or NO_TS` reads the
# same at every rail that merges two type-homogeneous lists in Python.
#
# It is NOT a crash fix, whatever the E24 commit that introduced it says. The `(ts is not None, ts)`
# It is NOT a crash fix, whatever the commit that introduced it says. The `(ts is not None, ts)`
# key it replaced cannot raise: tuple comparison finds the first UNEQUAL element via `==`, so two
# `(False, None)` keys compare equal and `None < None` is never reached — verified by execution
# during the E24 epic-end review, over every permutation of null/non-null rows. What the sentinel
# during a review, over every permutation of null/non-null rows. What the sentinel
# buys is one idiom instead of two, and a key that says "missing sorts last" outright rather than
# encoding it in a boolean flag. A BARE `r[0]` would genuinely raise, which is the mutation the
# rails' tests do guard against.
@@ -32,6 +33,11 @@ def show_status(total: int, watched: int, inprog: int) -> str:
return "new"
def show_state(total: int, watched: int, inprog: int) -> ShowState:
"""`show_status` plus whether an episode holds a resume point (the card's `ShowState`)."""
return ShowState(show_status(total, watched, inprog), (inprog or 0) > 0)
def rollup(cards: list[dict]) -> dict:
"""From ORDERED episode cards → the aggregate watch status, the "on deck" episode to Resume (the
last in-progress one, else the first unwatched), the first episode (for Play from the start), and
@@ -92,16 +98,16 @@ def agg_subquery(db: Session, user_id: int, include_resumed: bool = False):
)
def state_map(db: Session, user_id: int, show_ids: list[int]) -> dict[int, str]:
def state_map(db: Session, user_id: int, show_ids: list[int]) -> dict[int, ShowState]:
if not show_ids:
return {}
rows = db.execute(agg_subquery(db, user_id).where(PlexItem.show_id.in_(show_ids))).all()
return {r.show_id: show_status(r.total, r.watched, r.inprog) for r in rows}
return {r.show_id: show_state(r.total, r.watched, r.inprog) for r in rows}
def item_state_map(db: Session, user_id: int, item_ids: list[int]) -> dict[int, PlexState]:
"""A user's PlexState rows for a set of leaf items (movies or episodes), keyed by item_id — the
leaf analogue of `state_map` (which rolls a show's episodes up to a status string). One batched
leaf analogue of `state_map` (which rolls a show's episodes up to a `ShowState`). One batched
fetch, so the caller serializes cards without a per-card query. Empty in → empty dict (no `IN ()`)."""
if not item_ids:
return {}
+44 -24
View File
@@ -15,7 +15,7 @@ HLS muxer does the segmentation (the only reliable way to cut a stream-copy at k
Sessions live in this (single) API process; a periodic reaper kills idle ones and frees the temp
segments. The remux is CPU-light (video copy + a tiny audio transcode), so it runs fine on the
CPU-only prod host; only P3's full transcode is CPU-heavy.
CPU-only prod host; only the full transcode is CPU-heavy.
"""
import os
@@ -35,13 +35,13 @@ from app.config import settings
from app.downloads.edit import encode_thread_cap
from app.models import PlexItem
from app.plex import paths
from app.plex.sync import HW_DECODABLE_CODECS
from app.plex.sync import HDR_TRANSFERS, HW_DECODABLE_CODECS
log = logs.get_logger(__name__)
_HLS_ROOT = (
Path(settings.plex_hls_dir)
if settings.plex_hls_dir
else Path(settings.download_root) / ".plex-hls"
else Path(settings.files_root) / ".plex-hls"
)
_SEG_SECONDS = 6
# Cold-start warm-up. A freshly-spawned ffmpeg has produced only the first segment (or two) by the
@@ -120,8 +120,8 @@ class HlsSession:
# Unguessable per-session id, carried in the URLs the client is handed. A seek-restart
# replaces the viewer's registered session for the item — and the old player's next request
# for `seg_12.ts`, resolved by (viewer, rating_key) alone, was answered 200 with the NEW session's twelfth segment, produced from a different offset
# under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it
# (deep-review round 5). A request naming a session that is no longer registered is a 404,
# under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it.
# A request naming a session that is no longer registered is a 404,
# which the player recovers from by restarting at the playhead.
self.id = secrets.token_urlsafe(9)
self.key = key # the item's rating_key (what the logs name)
@@ -203,7 +203,8 @@ _HW_DECODABLE = HW_DECODABLE_CODECS
# plain scale_vaapi=format=nv12 path (Gen 9.5 VPP refuses it) — it needs the OpenCL tonemap graph
# (_venc_opts tonemap branch) when the OpenCL runtime is present, else software libx264. A 10-bit source
# WITHOUT one of these transfers is SDR, not HDR, and must not be tone-mapped. See _probe_video_kind.
_HDR_TRANSFERS = {"smpte2084", "arib-std-b67"}
# THE single source is plex.sync (a clip's tonemap reads it too).
_HDR_TRANSFERS = HDR_TRANSFERS
# The three source classes that drive the transcode-pipeline choice (see _probe_video_kind).
@@ -413,7 +414,7 @@ def _venc_opts(
def _render_node_present() -> bool:
"""Whether the VAAPI render node is available — i.e. `/dev/dri` was passed into the container.
Present on prod after the iGPU enablement; absent on the CPU-only dev box and in the localdev
Docker. Its own function so the selection is monkeypatchable in tests and one place for S1c's knob."""
Docker. Its own function so the selection is monkeypatchable in tests and the one place to change it."""
return Path(_VAAPI_DEVICE).exists()
@@ -576,7 +577,7 @@ def _ffmpeg_cmd(
entry += ",default:yes"
var.append(entry)
# The name patterns come from the vocabulary functions, with ffmpeg's own placeholders passed
# through them — so the producer and the parser cannot spell the layout differently (round 7).
# through them — so the producer and the parser cannot spell the layout differently.
args += _HLS_TAIL + [
"-master_pl_name",
_MASTER_PL,
@@ -744,8 +745,9 @@ def start_session(
# `replace_only` makes the supersede identity-aware for the runtime fallback: the whole
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
# seek-restart) installed between the fallback's decision and here is never clobbered. When the
# registered session is no longer `replace_only`, we abort and return None (the caller yields to
# whoever is registered now). The INITIAL spawn never gets None — see its call site.
# registered session is no longer `replace_only`, we abort and return None, and the caller
# answers with its own dead session — never the newer one, whose id a stale requester would
# stop. The INITIAL spawn never gets None — see its call site.
# Software-fallback encode tuning — read only on the SW transcode path (the VAAPI path and
# remux ignore preset/crf/max_height, and remux must not touch the DB for them).
preset, crf, max_height = "veryfast", 20, 0
@@ -840,12 +842,13 @@ def start_session(
)
respawned = _spawn(False, False, replace_only=s)
if respawned is None:
# Superseded/culled mid-fallback — yield whoever is registered now (a newer seek-restart), or
# our own now-dead session if the key was cap-culled. Never None here: the POST /session route
# turns a None return into a "Media file not found on disk" 404 — dishonest for a file that
# exists. A (dead) session instead degrades to a retryable 404 ("No active playback session" /
# "Not available yet") the client re-requests.
return current_session(user_id, key) or s
# Superseded/culled mid-fallback — answer with our own now-dead session. Never the newer one
# registered meanwhile: this answer goes to the request that LOST, whose player drops it as
# stale and stops the session it names (`stop_session`), which stopped the playback that won.
# Never None either: the POST /session route turns a None return into a "Media file not found
# on disk" 404 — dishonest for a file that exists. A dead session instead degrades to a
# retryable 404 ("No active playback session" / "Not available yet") the client re-requests.
return s
s = respawned
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
if ready:
@@ -887,7 +890,7 @@ def current_session(user_id: int, key: str) -> HlsSession | None:
# The names ffmpeg writes into a session directory, in ONE place — and "one place" means the argv
# BUILDS its patterns from these same functions (`segment_name("%d", "%v")`), not that a comment
# says so. Getting this wrong is invisible in both directions: a name we failed to anticipate used
# to fall back to "wait and see" (round 6's HIGH), and now that the rule is positive it falls back
# to fall back to "wait and see", and now that the rule is positive it falls back
# to a 404 instead — so the producer and the parser have to be literally the same spelling.
#
# MEASURED (ffmpeg 7.1.5 in the API container — the binary that actually writes these files — and
@@ -901,7 +904,7 @@ def current_session(user_id: int, key: str) -> HlsSession | None:
# SESSION and not to HLS in general: a single-audio session can never write `master.m3u8`,
# `stream_*.m3u8` or any `seg_<variant>_<n>.ts`, and a multi-audio one can never write `index.m3u8`
# or `seg_<n>.ts`. Accepting the union meant a hopeless name of the other shape was waited out for
# the full playlist/segment timeout — the threadpool parking this guard exists to end (round 7).
# the full playlist/segment timeout — the threadpool parking this guard exists to end.
_MASTER_PL = "master.m3u8"
_INDEX_PL = "index.m3u8"
_SEG_NAME_RE = re.compile(r"^seg_(?:([0-9a-z]{1,4})_)?(\d{1,9})\.ts$")
@@ -953,12 +956,12 @@ def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
name belonging to the other shape is as hopeless here as a random string. Reading those as "an
HLS name, so wait and see" left the whole class reachable — measured through the real route
against a live single-audio session, `master.m3u8` and `stream_q7z.m3u8` were 404s after 30.0 s
and `seg_a0_0.ts` / `seg_zzzz_0.ts` after 20.1 s each (deep-review round 7).
and `seg_a0_0.ts` / `seg_zzzz_0.ts` after 20.1 s each.
Saying no at once matters because the route that waits is a SYNC one: it holds a thread off the
~40-slot pool, and a pooled DB connection with it, for the whole timeout. An ordinary seek past
the produced region was enough to park one for 20 s, and forty concurrent requests for a name
that will never exist parked the whole API (deep-review round 5). Round 5's guard read only
that will never exist parked the whole API. An earlier guard read only
single-audio segment names, so both halves of that were still fully reachable: every multi-audio
name fell through, and so did any `.m3u8`/`.vtt` name at all."""
if filename in playlist_names(s.variants):
@@ -1093,15 +1096,15 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
A finished session is also what a PAUSED viewer has: a remux runs far faster than realtime, so
ffmpeg is done a minute or two in and writes ENDLIST; hls.js then stops asking for the playlist,
and with its forward buffer full a paused player fetches nothing at all. With the 30 s rule that
killed a one-minute pause, and playback died ~30 s after resuming (deep-review round 4).
killed a one-minute pause, and playback died ~30 s after resuming.
What makes the short rule safe again is that fetching is no longer the only signal: a player
parked MID-ITEM checkpoints its position every 10 s, and that route calls `touch_session`, so a
viewer who is merely paused says so far more often than this rule asks (round 5). The heartbeat
viewer who is merely paused says so far more often than this rule asks. The heartbeat
stops once the item is finished or before a position is known (`usePlexProgress.flush`) — both
states where reaping is correct, and where the player's 404 → reclaim recovers a viewer who
scrubs back in anyway. Do not restate this as "every 10 s whether or not it is playing": round 6
found that claim in two comments and not in the code. Round 4's blanket
scrubs back in anyway. Do not restate this as "every 10 s whether or not it is playing":
a review found that claim in two comments and not in the code. An earlier blanket
`reap_abandoned` bought the same safety by keeping every finished session for ten minutes — each
one a whole segmented copy of the film on a volume shared with the download center."""
now = time.time()
@@ -1117,6 +1120,23 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
return dropped
def stop_session(user_id: int, key: str, session_id: str) -> bool:
"""Stop this viewer's session for `key` now, because the player that asked for it is gone (left
the item, closed, unloaded the page). Left to `reap_idle`, a transcode ran on to the end of the
film first — measured: ~4 cores for a minute after leaving a resumed 480p film.
Only while `session_id` is still the registered one: the player names the session it played, and
a seek-restart or a second tab may have replaced it meanwhile — that newer session is somebody's
current playback, not this request's to end."""
with _lock:
s = _sessions.get((user_id, key))
if s is None or s.id != session_id:
return False
_kill(s)
_sessions.pop((user_id, key), None)
return True
def touch_session(user_id: int, key: str) -> None:
"""Say this viewer is still there for `key`, without fetching anything. `last_access` used to move
only on an HLS file fetch, which a paused player does not do — so the only way to keep a paused
+16 -13
View File
@@ -12,7 +12,7 @@ on a NON-empty listing — a section that returns zero items is left untouched,
glitch that yields an empty page can't wipe a library (the metadata is cheap to re-sync; an accidental
mass-delete is the only real risk). Library-level reconcile (dropping a whole removed section, the
dead `enabled` column) is a separate follow-up. Cast + intro/credit markers are fetched lazily on the
item-detail/player path (P2), not here.
item-detail/player path, not here.
"""
from datetime import datetime, timezone
@@ -26,6 +26,7 @@ from app.plex import detail, overlay, people
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.genres import hu_genres
from app.plex.lookups import by_rating_key
from app.plex.plaintext import plain_text
log = logs.get_logger(__name__)
# Codecs/containers a browser can play in a native <video> without any transcoding.
@@ -63,9 +64,11 @@ def classify_playable(vcodec: str | None, acodec: str | None, container: str | N
# outside it (AV1, MPEG-4 ASP/XviD…) is software-only even on the iGPU (it can't decode it to feed its
# own encoder).
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"})
# Color-transfer values marking an HDR source (PQ / HLG). A 10-bit/HDR source breaks the VAAPI
# scale=nv12 path on Gen 9.5 (see plex.stream) → software transcode, so it counts as NOT HW-capable.
_HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
# Color-transfer values marking an HDR source (PQ / HLG). THE single source of this set: plex.stream
# (the tonemap choice) and downloads.clip (a clip's tonemap) import it here, so a transfer added for
# one is tonemapped by all. A 10-bit/HDR source breaks the VAAPI scale=nv12 path on Gen 9.5 (see
# plex.stream) → software transcode, so it counts as NOT HW-capable.
HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
def transcode_hw_capable(
@@ -106,7 +109,7 @@ def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
# whole library would drown the sync's own lines.
except (TypeError, ValueError):
depth = None
hdr = str(s.get("colorTrc") or "").lower() in _HDR_TRANSFERS
hdr = str(s.get("colorTrc") or "").lower() in HDR_TRANSFERS
return depth, hdr
return None, False
@@ -150,7 +153,7 @@ _GUID_PREFERENCE = ("tmdb", "imdb", "tvdb")
def _canonical_guid(meta: dict) -> str | None:
"""The canonical external id used to pair an EN title with its HU-overlay twin (E5 S5). Reads the
"""The canonical external id used to pair an EN title with its HU-overlay twin. Reads the
`Guid` array (present only with includeGuids=1 in the listing), preferring tmdb > imdb > tvdb,
normalised to '<provider>:<id>' (e.g. 'tmdb:12345'). None when no known provider id is present."""
found: dict[str, str] = {}
@@ -419,7 +422,7 @@ def _paginate(plex: PlexClient, key: str, item_type: int, progress: dict | None
# A truthy `total` we reached = a trustworthy complete listing. `total` unknown/zero-with-rows
# (a Plex that doesn't report totalSize) is treated as NOT complete, so prune stays conservative.
# `total` itself is kept for a caller to whom a server that SAID 0 is news it can trust (the
# watch reconcile); an empty mirror or overlay must not read it so (E32 S2).
# watch reconcile); an empty mirror or overlay must not read it so.
progress["complete"] = bool(total) and start >= total
progress["total"] = total
@@ -447,7 +450,7 @@ def _apply_facet_fields(row, meta: dict) -> None:
# full cast (up to 24, grown by the rich backfill/info-page enrich), silently breaking the
# person-page cast filter for names past the top ~20. The watermark gate below (`people_enriched_at
# is None`) normally protects this, but it is NOT enough on its own: a migration that resets the
# watermark re-opens the regression (0065 nulled it, which is exactly how the S5b fix got undone in
# watermark re-opens the regression (0065 nulled it, which is exactly how a fix got undone in
# prod). So guard STRUCTURALLY too, by LENGTH against the listing cap: a stored cast LONGER than the
# cap can only be an enriched fuller list, so keep it (robust to any watermark reset). A stored cast
# at or below the cap is itself listing-sourced, so take the fresh listing — which tracks Plex cast
@@ -468,7 +471,7 @@ def _apply_item(row: PlexItem, lib_id: int, kind: str, meta: dict) -> None:
# carries none (a server ignoring includeGuids mustn't wipe the overlay's only match key).
row.guid = _canonical_guid(meta) or row.guid
row.title = (meta.get("title") or "").strip() or "Untitled"
row.summary = meta.get("summary")
row.summary = plain_text(meta.get("summary"))
row.year = meta.get("year")
row.duration_s = _dur_s(meta)
row.thumb_key = meta.get("thumb")
@@ -576,7 +579,7 @@ def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) ->
_canonical_guid(meta) or sh.guid
) # EN<->HU pairing key; keep prior if listing has none
sh.title = (meta.get("title") or "").strip() or "Untitled"
sh.summary = meta.get("summary")
sh.summary = plain_text(meta.get("summary"))
sh.year = meta.get("year")
sh.thumb_key = meta.get("thumb")
sh.art_key = meta.get("art")
@@ -659,7 +662,7 @@ def _upsert_collection(
db.add(col)
col.library_id = lib_id
col.title = (meta.get("title") or "").strip() or "Untitled"
col.summary = meta.get("summary")
col.summary = plain_text(meta.get("summary"))
col.thumb_key = meta.get("thumb")
col.child_count = meta.get("childCount")
col.smart = str(meta.get("smart")) == "1"
@@ -694,7 +697,7 @@ def resync_collection(db: Session, plex: PlexClient, lib: PlexLibrary, rk: str)
def _believed_children(plex: PlexClient, rk: str, meta: dict) -> list[dict] | None:
"""A collection's members, or None when the answer is not believed: no members from a collection
whose own listing counts some (`childCount`) is a glitch, not a collection emptied — the same
empty-answer rule as the collection listing's (E32 review). Rebuilding on it showed the
empty-answer rule as the collection listing's. Rebuilding on it showed the
collection empty until the next sync."""
children = plex.collection_children(rk)
count = str(meta.get("childCount") or "0")
@@ -745,7 +748,7 @@ def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: di
if not cols and existing:
# The catalog's own rule (see the module doc): an empty listing while we hold rows is a Plex
# glitch, not a library emptied of collections. Reconciling on it dropped every collection
# (its user-set `editable` flag with it) and every member's keys (E32 S2).
# (its user-set `editable` flag with it) and every member's keys.
log.warning(
"Plex listed no collections for a library that has some; keeping them this run",
extra=logs.event("plex.sync.collections_empty", library=lib.plex_key),
+11 -11
View File
@@ -1,4 +1,4 @@
"""TMDB enrichment for the mirrored Plex catalog (E5 S2).
"""TMDB enrichment for the mirrored Plex catalog.
A thin synchronous TMDB v3 client plus the parse/persist of the enrichment fields onto plex_items /
plex_shows, matched by the stored canonical `guid` ("tmdb:12345", or imdb/tvdb resolved via /find).
@@ -134,7 +134,7 @@ class TmdbClient:
return cast("list[dict] | None", self._get("/configuration/countries"))
def fetch_person(self, person_id: int) -> dict | None:
"""One person's bio + combined credits (E5 S6) + the bio's translations (E30 S4), in a single
"""One person's bio + combined credits + the bio's translations, in a single
append_to_response request."""
return self._get(
f"/person/{person_id}", {"append_to_response": "combined_credits,translations"}
@@ -226,7 +226,7 @@ def parse_tmdb(payload: dict, media_type: str) -> dict:
crew.sort(key=lambda c: priority.get(c["job"], 99))
crew = crew[:12]
# name -> tmdb person id, for the plex_people foundation (S6 fetches bios off these ids).
# name -> tmdb person id, for the plex_people foundation (bios are fetched off these ids).
people: list[dict] = []
for c in (credits.get("cast") or [])[:30]:
if c.get("name") and c.get("id"):
@@ -266,7 +266,7 @@ def parse_tmdb(payload: dict, media_type: str) -> dict:
c.get("name") for c in (payload.get("production_companies") or []) if c.get("name")
][:8]
or None,
# The ISO code beside TMDB's English name (E30): the client names the country in the reader's
# The ISO code beside TMDB's English name: the client names the country in the reader's
# language from the code; the name is the fallback for a code it cannot name.
"production_countries": [
{"code": c.get("iso_3166_1") or None, "name": c["name"]}
@@ -373,14 +373,14 @@ def enrich_batch(db: Session, tmdb: TmdbClient, limit: int) -> int:
def _country(c: dict | str) -> dict:
"""One stored production country as served: `{code, name}`. A row enriched before E30 stored the
"""One stored production country as served: `{code, name}`. An older row stored the
bare English name; until `backfill_country_codes` reaches it, it serves with no code (the client
shows the name)."""
return c if isinstance(c, dict) else {"code": None, "name": c}
def backfill_country_codes(db: Session, tmdb: TmdbClient) -> int:
"""Give the countries stored before E30 (bare English names) their ISO codes, from TMDB's own
"""Give the older countries (bare English names) their ISO codes, from TMDB's own
country list — the names came from that list, so they match it exactly. One request, and only
while such a row is left: re-fetching each title instead would take the enrichment batch days.
A name the list does not hold keeps no code. Returns the number of rows rewritten."""
@@ -418,7 +418,7 @@ def run_enrich(db: Session) -> dict:
def run_people_enrich(db: Session) -> dict:
"""Scheduler entry point of people_enrich (E30 S6): a batch of person bios (the most-credited
"""Scheduler entry point of people_enrich: a batch of person bios (the most-credited
first) and a batch of portraits. Each half is a no-op — never an error — without its TMDB key /
without Plex."""
batch = settings.people_enrich_batch
@@ -475,7 +475,7 @@ def resolve_recs(db: Session, recs: list[dict] | None, *, lang: str) -> list[dic
"""Tag each recommendation with whether it's in OUR library (matched by guid `tmdb:<id>`, bucketed
by media_type — see resolve_library_guids) and, if so, its rating_key — so the client can deep-link
the ones we have and merely display the rest. One we have is named as the library names it, in
`lang` (E30); one we lack keeps the title TMDB gave."""
`lang`; one we lack keeps the title TMDB gave."""
if not recs:
return []
movies, shows = resolve_library_guids(db, recs)
@@ -515,7 +515,7 @@ def serve_block(db: Session, row, *, lang: str) -> dict | None:
}
# --- Person enrichment (E5 S6) -------------------------------------------------------------------
# --- Person enrichment -------------------------------------------------------------------
# A person's bio + birth facts + a "Known for" credit pool, fetched once — by the people_enrich job, or
# by the first person-page open if the job has not reached them yet — and served from the plex_people row. `known_for` stores the top combined-credits ids; the person
# endpoint resolves them to library rows by guid at serve time (like recommendations).
@@ -525,7 +525,7 @@ def serve_block(db: Session, row, *, lang: str) -> dict | None:
# credits are often not the 15 a given library holds.
_KNOWN_FOR_MAX = 40
# TMDB genre ids of the shows a person visits as a guest (News, Reality, Talk). Ranked by popularity,
# talk-show visits crowded out the films (Tom Cruise: 14 of 15 were talk shows; E30 S6).
# talk-show visits crowded out the films (Tom Cruise: 14 of 15 were talk shows).
_GUEST_GENRES = {10763, 10764, 10767}
# A credit in which the person plays themselves ("Self", "Himself - Guest", "Herself (archive footage)").
_SELF_CREDIT = re.compile(r"^\s*(self|himself|herself|themselves)\b", re.IGNORECASE)
@@ -646,7 +646,7 @@ _PEOPLE_FAILS_IN_A_ROW = 3
def enrich_people_batch(db: Session, tmdb: TmdbClient, limit: int) -> dict:
"""Fetch up to `limit` linked, not-yet-enriched people (E30 S6), the most-credited first, so a
"""Fetch up to `limit` linked, not-yet-enriched people, the most-credited first, so a
person page reads its bio from the row instead of waiting on TMDB. Each person is committed as it
lands: a page open during the run already sees it. A failed lookup is skipped (left unstamped for the
next run); _PEOPLE_FAILS_IN_A_ROW of them in a row end the batch."""
+6 -7
View File
@@ -11,7 +11,7 @@ already pages) and upsert them into the owner's `plex_states`. Plex wins on over
exist only in Siftlode are left untouched (union, Plex-authoritative on the intersection).
Phases B (Siftlode→Plex push via scrobble/timeline) and C (incremental Plex→Siftlode via the watch
history + onDeck) build on this and land in later ships. As of S2 Plex is the source of truth: the
history + onDeck) build on this and land in later ships. Plex is now the source of truth: the
player pushes the resume position back to Plex on the periodic checkpoint (not just the final flush),
and the incremental pull resolves a genuine divergence to Plex's state (see `_pull_apply`) rather than
running a last-write-wins comparison.
@@ -81,7 +81,7 @@ def _scan_plex_states(
The settled keys matter to the reconcile alone: it reads "absent from the scan" as "Plex
un-watched it", which a walk cut short by a Plex hiccup does not mean (see `_paginate`). Settled
is decided PER SECTION: one section that never settles (a library emptied on Plex while its
mirrored items are kept) must not stop un-watches syncing in all the others (E32 review)."""
mirrored items are kept) must not stop un-watches syncing in all the others."""
item_id_by_rk = _rk_to_id(db)
wanted = _enabled_section_keys(db)
# The sections we mirror something from: a total of 0 from one of those contradicts the mirror.
@@ -117,7 +117,7 @@ def _scan_plex_states(
out[item_id] = res
# A section the server SAYS is empty (total 0, nothing listed) settles too. Believed only for
# a section we mirror nothing from: one whose items we hold answering 0 is a Plex that lost
# the library for a moment, not one that un-watched all of it (E32 review) — so it is left
# the library for a moment, not one that un-watched all of it — so it is left
# unsettled, and its rows are not cleared.
listed = scanned > before
said_empty = progress.get("total") == 0 and not listed and key not in mirrored
@@ -402,7 +402,7 @@ def _pull_apply(
if st is not None and not st.synced_to_plex:
st.synced_to_plex = True # already matches Plex → just settle the flag, no churn
return None
# Plex-authoritative (S2): on a genuine divergence for an item Plex knows about, Plex wins — no
# Plex-authoritative: on a genuine divergence for an item Plex knows about, Plex wins — no
# last-write-wins tug-of-war. This is safe because prompt write-back (playback.py pushes the resume
# position to Plex on the ~10s checkpoint, not just on the final flush) keeps Plex current, so a
# local edit that's "newer" than Plex has almost always already been pushed and matched above by
@@ -620,9 +620,8 @@ def _clear_unwatched(db: Session, user_id: int, plex_states: dict, settled: set[
"""Clear the user's rows Plex no longer has a record for — the mirrored ones (Plex un-watched
them); hidden and dirty rows stay (a dirty one is re-pushed by `_repush_dirty`). Only rows of the
sections the scan SETTLED: a walk a Plex hiccup cut short leaves out rows it merely failed to
reach, and clearing those un-watched them here until the next complete run (E32 S2); an item of
a library the allowlist leaves out is absent from every scan, and was cleared on every run (E32
review)."""
reach, and clearing those un-watched them here until the next complete run; an item of
a library the allowlist leaves out is absent from every scan, and was cleared on every run."""
n = 0
rows = db.execute(
select(PlexState)
+1 -1
View File
@@ -25,7 +25,7 @@ def report(current: int, total: int | None = None, phase: str | None = None) ->
class JobFailed(RuntimeError):
"""A job's own failure, said so the reader's language can name it (E31): the client reads
"""A job's own failure, said so the reader's language can name it: the client reads
`scheduler.failure.<reason>` and interpolates `params` (keep them to values the reader may see —
numbers, ids, short codes). The message is the English line the log and `last_error` keep. An
upstream's own error (a timeout, a refused connection) is not one of these: it stays as raised,
+11 -3
View File
@@ -16,7 +16,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.db import SessionLocal
from app.models import ApiQuotaUsage, QuotaEvent
@@ -204,6 +204,14 @@ def record_usage(units: int) -> None:
)
with SessionLocal() as qs:
qs.execute(stmt)
# Audit detail: who/what spent it (per-user attribution; NULL actor = system).
qs.add(QuotaEvent(user_id=_actor_id.get(), action=_action.get(), units=units))
# Audit detail: who/what spent it (per-user attribution; NULL actor = system), and the
# background run it belongs to, so the run's row can say what it cost.
qs.add(
QuotaEvent(
user_id=_actor_id.get(),
action=_action.get(),
units=units,
job_run_id=logs.current_run_id(),
)
)
qs.commit()
-68
View File
@@ -1,68 +0,0 @@
"""In-process WebSocket connection registry for live server→client push.
Generic groundwork (notification module phase 2 is the first user): connections are kept in
this process's memory, keyed by user_id, so a request handler can push an event to all of a
user's open tabs/devices. Single-uvicorn-process assumption — if we ever run multiple workers,
replace the in-memory map with a Redis pub/sub fan-out behind the same `push()` interface (same
note as app.ratelimit).
`push()` is intentionally SYNC-callable: ordinary (sync) route handlers run in a threadpool, so
they schedule the actual send onto the WebSocket event loop via run_coroutine_threadsafe. The
connect/disconnect/_push coroutines all run on that one loop, so the connection map is only ever
mutated from a single thread.
"""
import asyncio
from collections import defaultdict
from starlette.websockets import WebSocket
from app import logs
log = logs.get_logger(__name__)
class ConnectionManager:
def __init__(self) -> None:
self._conns: dict[int, set[WebSocket]] = defaultdict(set)
self._loop: asyncio.AbstractEventLoop | None = None
async def connect(self, user_id: int, ws: WebSocket) -> None:
self._loop = asyncio.get_running_loop()
await ws.accept()
self._conns[user_id].add(ws)
async def disconnect(self, user_id: int, ws: WebSocket) -> None:
conns = self._conns.get(user_id)
if conns:
conns.discard(ws)
if not conns:
self._conns.pop(user_id, None)
async def _push(self, user_id: int, payload: dict) -> None:
for ws in list(self._conns.get(user_id, ())):
try:
await ws.send_json(payload)
except Exception as exc: # noqa: BLE001 — one dead socket must not stop the others
# Stale socket; it'll be cleaned up by its own disconnect handler. But this is
# also where a live update goes missing: the UI simply never changes, and there
# is no request behind it whose failure anyone could look up.
log.debug(
"a live update could not be pushed to a socket",
extra=logs.event(
"realtime.push_failed", user_id=user_id, reason=type(exc).__name__
),
)
def push(self, user_id: int, payload: dict) -> None:
"""Schedule a push onto the WebSocket event loop. Safe to call from a sync route
handler (threadpool). No-op if no WebSocket has ever connected (no loop captured)."""
loop = self._loop
if loop is None:
return
try:
asyncio.run_coroutine_threadsafe(self._push(user_id, payload), loop)
except Exception: # noqa: BLE001 — a closed loop or socket must not fail the caller
log.debug("ws push failed", exc_info=True)
manager = ConnectionManager()
+17
View File
@@ -0,0 +1,17 @@
"""The account filter the activity page's lists share (runs, admin actions, server log, quota): one
filter bar drives them all, so the value it sends means the same on every tab.
The value is an account id, or `none` — the rows no account made (the background) — a value to filter
on, not the absence of a filter.
"""
from sqlalchemy import ColumnElement
NONE = "none"
# The `user` query parameter's shape on every route that takes it.
PATTERN = r"^(none|[0-9]{1,10})$"
def condition(column, value: str) -> ColumnElement[bool]:
"""`column` (an account id column) narrowed to `value`."""
return column.is_(None) if value == NONE else column == int(value)
+47 -5
View File
@@ -1,4 +1,4 @@
"""Route-layer shared helpers for the API contract (R6 S1a).
"""Route-layer shared helpers for the API contract.
Small, dependency-light guards that several route modules had hand-rolled with subtle
drift. Keeping them here (imported by the route modules) means the "fetch a row or 404"
@@ -6,19 +6,55 @@ and "fetch a row this user owns or 404" idioms have one definition, one status c
one existence-leak policy.
"""
from typing import IO, Any, TypeVar
from typing import Any, Protocol, TypeVar
from fastapi import HTTPException, Request
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy.orm import Session
from app.db import Base
from app.models import User
from app.models import POSITION_MAX_S, User
from app.issues import gitea
from app.refusal import refused
from app.youtube.client import YouTubeError, refusal_reason
M = TypeVar("M", bound=Base)
# Resume-position thresholds of a player that only keeps a resume point (the YouTube player, the
# library's): positions below this are "didn't really start" and within this of the end are
# "basically finished" — neither is worth storing, so the position is cleared instead (no progress
# bar). The client's local resume (`lib/resume.ts`) keeps the same two numbers, held equal by
# `resume.contract.test.ts` beside it. NB: the Plex player intentionally uses a LARGER finish margin
# (30s) because it marks long-form items watched, not just clears a resume bar — see
# routes/plex/playback.py.
PROGRESS_MIN_SECONDS = 5
FINISH_MARGIN_SECONDS = 10
class ProgressIn(BaseModel):
"""A player's resume checkpoint (sent every few seconds while it plays, and as a beacon when the
page goes away)."""
# Finite only: JSON parsing here accepts `Infinity`/`NaN`, and the handler's `int()` turned
# one into a 500.
model_config = ConfigDict(allow_inf_nan=False)
# Accept a float and truncate (`kept_position`) — the player may post a raw currentTime (a float),
# which the old `int(payload.get(...))` silently truncated. A plain `int` field would 422 that on
# the progress hot path, losing the resume position; `float` keeps the old tolerance.
# Bounded above by the int4 column (see POSITION_MAX_S): past it the flush was a 500.
position_seconds: float = Field(0, le=POSITION_MAX_S)
duration_seconds: float = Field(0, le=POSITION_MAX_S)
def kept_position(body: ProgressIn) -> int:
"""The whole-second resume point worth storing for `body`, or 0 for "clear it" (too early to
count as started, or close enough to the end to count as finished)."""
position = max(0, int(body.position_seconds))
duration = int(body.duration_seconds)
near_end = duration > 0 and position > duration - FINISH_MARGIN_SECONDS
return position if position >= PROGRESS_MIN_SECONDS and not near_end else 0
def not_found(what: str = "generic") -> HTTPException:
"""The 404 for a row that is gone — structured, so the bilingual UI says it in the reader's
@@ -86,8 +122,14 @@ async def capped_body(request: Request, limit: int) -> bytes:
return bytes(body)
class ByteSink(Protocol):
"""Where `capped_stream_to` writes: a file, or anything that takes the bytes on their way to one."""
def write(self, data: bytes, /) -> object: ...
async def capped_stream_to(
request: Request, out: IO[bytes], limit: int, refusal: HTTPException
request: Request, out: ByteSink, limit: int, refusal: HTTPException
) -> int:
"""`capped_body`'s sibling for a body too large to hold: the same counting rule, written
straight to `out` instead of accumulated, returning the size.
@@ -128,7 +170,7 @@ async def capped_stream_to(
def issue_refused(exc: gitea.GiteaError) -> HTTPException:
"""The refusal an issue-tracker failure (E16) answers as — the send, the preview and the
"""The refusal an issue-tracker failure answers as — the send, the preview and the
admin's connection test alike. Literal reasons, one per case: the frontend's contract test reads
them to require a sentence for each in both languages."""
match exc.reason:
+66
View File
@@ -0,0 +1,66 @@
"""Keyset paging over `(timestamp desc, id desc)`, shared by the admin history lists (server log,
audit log, background runs).
A timestamp alone is not unique (a browser log row's `ts` is clamped into its consent window; a run
and its siblings can start in the same microsecond), and an offset would skip or repeat rows while
writers add new ones at the top — so a page continues from the last row's `(timestamp, id)`.
"""
import base64
import binascii
import json
from datetime import datetime, timezone
from fastapi import HTTPException
from sqlalchemy import and_, or_, select
from sqlalchemy.orm import Session
from app.models import User
PAGE_MAX = 200
def utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the columns are `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def encode_cursor(ts: datetime, row_id: int) -> str:
return base64.urlsafe_b64encode(json.dumps([ts.isoformat(), row_id]).encode()).decode()
def decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
at = datetime.fromisoformat(ts)
return (at if at.tzinfo is not None else at.replace(tzinfo=timezone.utc)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def older_than(ts_col, id_col, cursor: str):
"""The rows after `cursor` in `(ts desc, id desc)` order. The `ts <=` conjunct is implied by the
OR; it is there so the planner can bound the timestamp index scan instead of filtering every
newer row first."""
ts, row_id = decode_cursor(cursor)
return and_(ts_col <= ts, or_(ts_col < ts, and_(ts_col == ts, id_col < row_id)))
def page(rows: list, limit: int, ts_of) -> tuple[list, str | None]:
"""`rows` was read with `limit + 1` (to know whether there is a next page without a count query):
the page itself, and the cursor that continues after it (None on the last page)."""
more = len(rows) > limit
rows = rows[:limit]
return rows, encode_cursor(ts_of(rows[-1]), rows[-1].id) if more else None
def accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the row
was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
+26 -134
View File
@@ -1,5 +1,4 @@
"""Admin-only onboarding: review access requests (Invites), approve/deny, manual add.
Also the demo-account admin surface: the demo email whitelist + a manual state reset."""
"""Admin-only onboarding: review access requests (Invites), approve/deny, manual add."""
from collections.abc import Iterable
from datetime import datetime, timezone
@@ -19,15 +18,12 @@ from app.auth import (
count_admins,
env_allowed_canonicals,
find_user_by_email,
get_or_create_demo_user,
is_allowed,
operator_contact,
purge_user,
)
from app.db import get_db
from app.demo import reset_demo_state
from app.models import (
DemoWhitelist,
Invite,
User,
)
@@ -131,7 +127,7 @@ def _welcome_mail_parts(db: Session, email: str) -> tuple[str | None, bool, int]
"""
hours = int(APPROVAL_VERIFY_TTL.total_seconds() // 3600)
user = find_user_by_email(db, email)
if user is None or user.is_demo:
if user is None:
return None, False, hours
if user.email_verified or not email_mod.email_enabled(db):
return None, True, hours
@@ -142,20 +138,21 @@ def _activate_user_by_email(db: Session, email: str) -> None:
"""Approving access activates a matching (e.g. pending password-registration) account so
it can sign in. No-op if no such user exists yet (a Google user is created at first login).
E9.5 S1: this path deliberately does NOT revoke a preset password. Revoking here destroyed the
This path deliberately does NOT revoke a preset password. Revoking here destroyed the
password a legitimate registrant had just chosen and made "your password was cleared for
security" the DEFAULT experience, because approve-before-verify is the normal ordering.
S1 justified that by saying `password_login` refuses an unverified account BEFORE it looks at
`is_active`, so a pre-emptor could not sign in "approved or not". A review of the finished epic
showed that argument only holds while the address stays unverified — and S6's approval mail
hands the real owner a confirmation link precisely so they will end that state. Nothing replaced
the revoke, and that is deliberate: round 2 of the same review tried "the newest registration
owns an unverified account" and reverted it, because the rule is symmetric — an attacker can
always register last — and it handed anonymous callers a write onto a real account's credential.
The owner's recourse is password-reset, which proves the mailbox, rewrites the hash and bumps
the session epoch. The residual is logged as SEC-16. The revoke also stays on the Google-adoption path
(auth.revoke_preset_password_if_unverified), where ownership is actually attested."""
That was once justified by saying `password_login` refuses an unverified account BEFORE it
looks at `is_active`, so a pre-emptor could not sign in "approved or not". A review of the
finished epic showed that argument only holds while the address stays unverified — and the
approval mail hands the real owner a confirmation link precisely so they will end that state.
Nothing replaced the revoke, and that is deliberate: the same review later tried "the newest
registration owns an unverified account" and reverted it, because the rule is symmetric — an
attacker can always register last — and it handed anonymous callers a write onto a real
account's credential. The owner's recourse is password-reset, which proves the mailbox, rewrites
the hash and bumps the session epoch. The residual is logged as SEC-16. The revoke also stays on
the Google-adoption path (auth.revoke_untrusted_preset_password), where ownership is
actually attested."""
# Canonical, with the literal spelling preferred when both exist: approving
# `npeter83.temu@gmail.com` must activate the account that registered as `npeter83temu@gmail.com`,
# since Gmail delivers both to one mailbox and the admin decided about the mailbox.
@@ -400,19 +397,18 @@ def _serialize_user(u: User) -> dict:
"is_active": u.is_active,
"is_suspended": u.is_suspended,
"email_verified": u.email_verified,
"is_demo": u.is_demo,
"has_password": u.password_hash is not None,
# Distinguishes "this account never had a password" (Google-only) from "SEC-10 cleared it" —
# the same distinction _send_reset_if_eligible needs, and the one an admin has to see before
# deciding what kind of help someone needs.
"password_revoked_at": iso(u.password_revoked_at),
"has_google": u.google_sub is not None,
# Only when it DIFFERS from the account address (E9.5 S3): "connected" is not the same
# Only when it DIFFERS from the account address: "connected" is not the same
# statement as "connected as someone else", and the second one is the one that explains a
# confusing sign-in.
"google_email": u.google_email if u.google_email != u.email else None,
"created_at": iso(u.created_at),
# NULL = no sign-in since the column existed (0085), or the shared demo account.
# NULL = no sign-in since the column existed (0085).
"last_login_at": iso(u.last_login_at),
}
@@ -435,14 +431,12 @@ def set_user_role(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Promote/demote a user. Guards: can't change your own role, can't touch the demo
account, and can't remove the last remaining admin. Emails the user when it actually changes."""
"""Promote/demote a user. Guards: can't change your own role and can't remove the
last remaining admin. Emails the user when it actually changes."""
role = body.role
if role not in ("user", "admin"):
raise refused(400, "invalid_role")
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise refused(400, "demo_role")
if target.id == admin.id:
raise refused(400, "own_role")
if (
@@ -490,14 +484,12 @@ def set_user_suspended(
db: Session = Depends(get_db),
) -> dict:
"""Suspend or un-suspend an account. A suspended user can't sign in by any method and any
live session is dropped on its next request. Guards: can't suspend the demo account, can't
suspend yourself, and can't suspend the last active admin (that would lock admin out). On
live session is dropped on its next request; their queued downloads wait (the worker skips them)
until they are un-suspended. Guards: can't suspend yourself, and can't suspend the last active admin (that would lock admin out). On
un-suspend the user is emailed they can return (suspend is announced reactively, on a blocked
sign-in)."""
suspended = body.suspended
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise refused(400, "demo_suspend")
if target.id == admin.id:
raise refused(400, "own_suspend")
if (
@@ -538,10 +530,8 @@ def admin_delete_user(
) -> dict:
"""Admin-triggered account deletion — the same full erasure a user can perform on themselves
(cascade delete of all personal data + access-request cleanup + Google-grant revoke). Guards:
not the demo account, not yourself (use Settings → Account), and not the last admin."""
not yourself (use Settings → Account), and not the last admin."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise refused(400, "demo_delete")
if target.id == admin.id:
raise refused(400, "own_delete")
if (
@@ -563,10 +553,7 @@ def admin_delete_user(
return {"deleted": user_id}
# --- Demo account: email whitelist + state reset -------------------------------------
# --- Stuck sign-ups: the admin lifeline (E9.5 S5, roadmap UX-25) ---------------------------------
# --- Stuck sign-ups: the admin lifeline -----------------------------------------------------------
def _pending_access_rows(db: Session) -> list[User]:
@@ -587,7 +574,6 @@ def _pending_access_rows(db: Session) -> list[User]:
db.execute(
select(User)
.where(
User.is_demo.is_(False),
User.is_suspended.is_(False),
or_(User.is_active.is_(False), User.email_verified.is_(False)),
)
@@ -625,7 +611,7 @@ def user_access(
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Everything the account panel shows about why this person can or cannot get in (E12 S5).
"""Everything the account panel shows about why this person can or cannot get in.
Its own endpoint rather than extra columns on /users: `is_allowed` and `can_activate` each run a
query, and the panel opens for ONE user at a time — folding them into the list would pay that cost
@@ -644,7 +630,7 @@ def create_set_password_link(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Mint a one-time link that lets this account set a password, with no email involved (E12 S5).
"""Mint a one-time link that lets this account set a password, with no email involved.
The answer to SEC-15, SEC-16 and "no SMTP and someone forgot their password" at once. The admin
hands the link over by whatever channel they trust; opening it sets a password and signs out
@@ -653,13 +639,8 @@ def create_set_password_link(
Deliberately NOT gated on the account's state. It is useful for a pre-empted address, for a
revoked password, for a Google-only account that wants one, and for a plain forgotten password on
an instance with no mail — narrowing it to one of those would only make the admin's real case the
one it refuses. The demo account is the single exception: it is shared, so handing anyone a way to
change its password is handing them everyone else's session."""
one it refuses."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(
status_code=400, detail={"code": "setpw", "reason": "demo", "email": target.email}
)
url, expires_at = issue_set_password_link(db, target)
# The URL itself is NOT recorded: the audit log is read by humans and kept long, and a live
# credential-setting link is the last thing that belongs in it. That the link was issued, by whom,
@@ -692,8 +673,6 @@ def activate_user(
already do exactly this by re-entering the address in the whitelist box (`add_invite` →
`_activate_user_by_email`). What was missing was any way to know they had to."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise refused(400, "demo_activate")
if target.is_active:
return _serialize_user(target) # idempotent: a double click is not an error
if not is_allowed(db, target.email):
@@ -731,7 +710,7 @@ def resend_user_verification(
Not rate-limited: an admin is not a threat model to enumerate against, and the deliberate silence
of the public endpoint is exactly what this exists to escape."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo or target.email_verified:
if target.email_verified:
raise HTTPException(
status_code=409,
detail={"code": "resend", "reason": "not_applicable", "email": target.email},
@@ -757,93 +736,6 @@ def resend_user_verification(
return {"ok": True, "email": target.email}
def _serialize_demo(row: DemoWhitelist) -> dict:
return {
"id": row.id,
"email": row.email,
"note": row.note,
"added_by": row.added_by,
"created_at": iso(row.created_at),
}
@router.get("/demo/whitelist")
def list_demo_whitelist(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> list[dict]:
rows = (
db.execute(select(DemoWhitelist).order_by(DemoWhitelist.created_at.desc())).scalars().all()
)
return [_serialize_demo(r) for r in rows]
class AddDemoWhitelistIn(BaseModel):
email: str = ""
note: str | None = None
@router.post("/demo/whitelist")
def add_demo_whitelist(
body: AddDemoWhitelistIn,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Add an email that may enter the shared demo account from the login page (no Google
sign-in). Idempotent — re-adding an existing email just returns it."""
email = body.email.strip().lower()
if not valid_email(email):
raise refused(400, "invalid_email")
row = db.execute(select(DemoWhitelist).where(DemoWhitelist.email == email)).scalar_one_or_none()
if row is None:
row = DemoWhitelist(
email=email,
note=(body.note or "").strip() or None,
added_by=admin.email,
)
db.add(row)
audit.record(
db, admin.id, AuditAction.DEMO_ADD, target_type="demo", target_id=email, summary=email
)
db.commit()
return _serialize_demo(row)
@router.delete("/demo/whitelist/{entry_id}")
def remove_demo_whitelist(
entry_id: int,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
row = db.get(DemoWhitelist, entry_id)
if row is not None:
audit.record(
db,
admin.id,
AuditAction.DEMO_REMOVE,
target_type="demo",
target_id=row.email,
summary=row.email,
)
db.delete(row)
db.commit()
return {"deleted": entry_id}
@router.post("/demo/reset")
def reset_demo(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
"""Manually clean up the shared demo sandbox back to a baseline. A scheduled job does the
same automatically (admin-tunable interval); this is the admin's on-demand button."""
demo = get_or_create_demo_user(db)
seeded = reset_demo_state(db, demo)
audit.record(
db,
admin.id,
AuditAction.DEMO_RESET,
summary=f"{seeded} sample playlists re-seeded",
after={"playlists_seeded": seeded}, # the count as data: named in the reader's language
)
db.commit()
return {"ok": True, "reset": True, "playlists_seeded": seeded}
@router.post("/purge-discovery")
def purge_discovery(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
"""Reclaim all un-kept discovery content NOW (ignoring the grace period): live-search result
+33 -72
View File
@@ -1,4 +1,4 @@
"""The admin diagnostics console's read side (E22.6): query `log_events`, the server's stored log.
"""The admin diagnostics console's read side: query `log_events`, the server's stored log.
Admin-only and read-only — rows leave the table through the retention job (`log_retention_days`),
never through here. Reading is not audited: it changes nothing, and the console's live tail polls,
@@ -8,33 +8,29 @@ What a row holds was redacted when it was written (`logs.record_fields`, and the
browser's rows), so this serves the columns as they are; a value the redactor took out reads as
`[redacted]` / `[email]`, which is the point — the console shows that something was there.
Paging is a keyset over `(ts desc, id desc)`: a browser row's `ts` is clamped into its consent
window, so `ts` alone is not unique, and an offset would skip or repeat rows while the writer adds
new ones at the top. The live tail re-reads the first page rather than asking for "ids after N":
Paging is a keyset over `(ts desc, id desc)` (`_keyset`). The live tail re-reads the first page rather than asking for "ids after N":
several processes write this table, and a batch's ids are taken when it inserts, not when it
commits, so a slower batch can land below an id the tail already passed — and would never show.
"""
import base64
import binascii
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import Text, and_, cast, func, or_, select
from fastapi import APIRouter, Depends, Query
from sqlalchemy import Text, cast, func, or_, select
from sqlalchemy.orm import Session
from app import logsink
from app.auth import admin_user
from app.db import get_db
from app.models import LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.utils import like_escape
router = APIRouter(prefix="/api/admin/logs", tags=["admin"])
PAGE_MAX = 200
# A facet list is a chip row, not a report: past this many distinct events the rest are reachable
# through the free-text search, and the response stays small whatever a self-host logs.
FACET_EVENTS_MAX = 100
@@ -42,38 +38,9 @@ FACET_EVENTS_MAX = 100
Level = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
# The filters a facet can leave out of its own count (`LogFilters.conditions`).
Facet = Literal["levels", "sources", "user", "event"]
# `none` selects the rows that have no account / no event code (a scheduler line, an anonymous
# request) — a real value to filter on, not the absence of a filter.
NONE = "none"
def _utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the column is `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def _encode_cursor(row: LogEvent) -> str:
return base64.urlsafe_b64encode(json.dumps([row.ts.isoformat(), row.id]).encode()).decode()
def _decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
return _utc(datetime.fromisoformat(ts)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def _accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the
line was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
# `none` selects the rows that have no event code, as the account filter's `none` (app/routes/_account)
# selects the rows of no account — a real value to filter on, not the absence of a filter.
NONE = account.NONE
@dataclass(frozen=True)
@@ -86,6 +53,7 @@ class LogFilters:
event: str | None
request_id: str | None
job_id: int | None
run_id: int | None
q: str | None
since: datetime | None
until: datetime | None
@@ -100,11 +68,7 @@ class LogFilters:
if self.sources and skip != "sources":
out.append(LogEvent.source.in_(self.sources))
if self.user is not None and skip != "user":
out.append(
LogEvent.user_id.is_(None)
if self.user == NONE
else LogEvent.user_id == int(self.user)
)
out.append(account.condition(LogEvent.user_id, self.user))
if self.event is not None and skip != "event":
out.append(
LogEvent.event.is_(None) if self.event == NONE else LogEvent.event == self.event
@@ -113,10 +77,12 @@ class LogFilters:
out.append(LogEvent.request_id == self.request_id)
if self.job_id is not None:
out.append(LogEvent.job_id == self.job_id)
if self.run_id is not None:
out.append(LogEvent.run_id == self.run_id)
if self.since is not None:
out.append(LogEvent.ts >= _utc(self.since))
out.append(LogEvent.ts >= utc(self.since))
if self.until is not None:
out.append(LogEvent.ts < _utc(self.until))
out.append(LogEvent.ts < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
@@ -134,17 +100,20 @@ class LogFilters:
def log_filters(
levels: list[Level] = Query(default=[]),
sources: list[str] = Query(default=[], max_length=8),
user: str | None = Query(default=None, pattern=r"^(none|[0-9]{1,10})$"),
user: str | None = Query(default=None, pattern=account.PATTERN),
event: str | None = Query(
default=None, max_length=64, pattern=r"^(none|[a-z][a-z0-9_]*(\.[a-z0-9_]+)*)$"
),
request_id: str | None = Query(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$"),
job_id: int | None = Query(default=None, ge=1),
run_id: int | None = Query(default=None, ge=1),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> LogFilters:
return LogFilters(list(levels), list(sources), user, event, request_id, job_id, q, since, until)
return LogFilters(
list(levels), list(sources), user, event, request_id, job_id, run_id, q, since, until
)
@router.get("")
@@ -159,28 +128,20 @@ def list_logs(
continues from a previous page's `next_cursor`."""
query = select(LogEvent).where(*filters.conditions())
if before is not None:
ts, row_id = _decode_cursor(before)
# The `ts <=` conjunct is implied by the OR; it is there so the planner can bound the
# `ts` index scan instead of filtering every newer row first.
query = query.where(
and_(
LogEvent.ts <= ts,
or_(LogEvent.ts < ts, and_(LogEvent.ts == ts, LogEvent.id < row_id)),
)
)
# One more than asked, to know whether there is a next page without a second count query.
rows = (
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
query = query.where(older_than(LogEvent.ts, LogEvent.id, before))
rows, cursor = page(
list(
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
),
limit,
lambda r: r.ts,
)
more = len(rows) > limit
rows = rows[:limit]
return {
"items": [logsink.serialize(r) for r in rows],
"accounts": _accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": _encode_cursor(rows[-1]) if more else None,
"accounts": accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": cursor,
}
@@ -211,6 +172,6 @@ def log_facets(
"sources": counts(LogEvent.source, "sources"),
"events": counts(LogEvent.event, "event", FACET_EVENTS_MAX),
"users": [{**u, "value": str(u["value"])} for u in users],
"accounts": _accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"accounts": accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"total": db.scalar(total) or 0,
}
+106 -29
View File
@@ -1,67 +1,144 @@
"""Admin audit log API: read the append-only who/what/when trail, and clear it.
Admin-only. The client renders these rows in a DataTable (client-side sort/filter/paginate), so
this returns the most recent window of rows (capped) plus the true total, rather than paging
server-side — the log stays small in practice (state-changing runs + admin actions only)."""
Admin-only. The activity page reads it a page at a time, newest first, filtered on the server like
the server log (`_keyset` paging), so the whole trail is reachable however long it grows."""
from fastapi import APIRouter, Depends
from sqlalchemy import delete, func, select
from dataclasses import dataclass
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy import Text, cast, delete, func, or_, select
from sqlalchemy.orm import Session
from app import audit
from app.audit import AuditAction
from app.auth import admin_user
from app.db import get_db
from app.models import AuditLog, User
from app.utils import iso
from app.models import AuditLog, LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.utils import iso, like_escape
router = APIRouter(prefix="/api/admin/audit", tags=["admin"])
# Cap the client-side window. The audit log is small by design; if it ever outgrows this we'd
# switch the DataTable to server-side paging.
MAX_ROWS = 2000
def _serialize(row: AuditLog, actors: dict[int, str]) -> dict:
def _serialize(row: AuditLog, lines: dict[str, int]) -> dict:
return {
"id": row.id,
"created_at": iso(row.created_at),
# actor_id NULL = the scheduler/background did it (client shows "System"); a set id that
# can't be resolved would mean a deleted user, but SET NULL makes that NULL — so present.
"actor": actors.get(row.actor_id) if row.actor_id is not None else None,
# NULL = the background did it (the client shows "System"); the address is in `accounts`.
"actor_id": row.actor_id,
"action": row.action,
"target_type": row.target_type,
"target_id": row.target_id,
"summary": row.summary,
"before": row.before,
"after": row.after,
"request_id": row.request_id,
# How many of that request's log lines are still kept: they go after days, the row after
# months, and a link to none would open an empty list.
"log_lines": lines.get(row.request_id, 0) if row.request_id else 0,
}
@dataclass(frozen=True)
class AuditFilters:
action: str | None
actor: str | None
q: str | None
since: datetime | None
until: datetime | None
def conditions(self, skip: Literal["action"] | None = None) -> list:
"""The WHERE conditions, ANDed. `skip="action"` leaves the action filter out, so the action
menu counts every action under the other filters (the way back from a pick)."""
out = []
if self.action is not None and skip != "action":
out.append(AuditLog.action == self.action)
if self.actor is not None:
out.append(account.condition(AuditLog.actor_id, self.actor))
if self.since is not None:
out.append(AuditLog.created_at >= utc(self.since))
if self.until is not None:
out.append(AuditLog.created_at < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
or_(
AuditLog.summary.ilike(like),
AuditLog.target_id.ilike(like),
AuditLog.action.ilike(like),
AuditLog.request_id.ilike(like),
cast(AuditLog.before, Text).ilike(like),
cast(AuditLog.after, Text).ilike(like),
)
)
return out
def audit_filters(
action: str | None = Query(default=None, max_length=48, pattern=r"^[a-z][a-z0-9_]*$"),
user: str | None = Query(default=None, pattern=account.PATTERN),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> AuditFilters:
return AuditFilters(action, user, q, since, until)
@router.get("")
def list_audit(
limit: int = 500,
filters: AuditFilters = Depends(audit_filters),
before: str | None = Query(default=None, max_length=200),
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
limit = max(1, min(limit, MAX_ROWS))
total = db.scalar(select(func.count()).select_from(AuditLog)) or 0
rows = (
db.execute(select(AuditLog).order_by(AuditLog.created_at.desc()).limit(limit))
.scalars()
.all()
"""One page of rows, newest first, matching every filter given; `before` continues from a
previous page's `next_cursor`."""
query = select(AuditLog).where(*filters.conditions())
if before is not None:
query = query.where(older_than(AuditLog.created_at, AuditLog.id, before))
order = (AuditLog.created_at.desc(), AuditLog.id.desc())
rows, cursor = page(
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
limit,
lambda r: r.created_at,
)
actor_ids = {r.actor_id for r in rows if r.actor_id is not None}
actors: dict[int, str] = {}
if actor_ids:
actors = dict(db.execute(select(User.id, User.email).where(User.id.in_(actor_ids))).all())
requests = {r.request_id for r in rows if r.request_id}
lines: dict[str, int] = {}
if requests:
counted = db.execute(
select(LogEvent.request_id, func.count())
.where(LogEvent.request_id.in_(requests))
.group_by(LogEvent.request_id)
)
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
return {
"items": [_serialize(r, actors) for r in rows],
"total": total,
"returned": len(rows),
"items": [_serialize(r, lines) for r in rows],
"accounts": accounts(db, {r.actor_id for r in rows if r.actor_id is not None}),
"next_cursor": cursor,
}
@router.get("/facets")
def audit_facets(
filters: AuditFilters = Depends(audit_filters),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""The actions present under the other filters, with counts — what the action menu offers."""
n = func.count().label("n")
stmt = (
select(AuditLog.action, n)
.where(*filters.conditions("action"))
.group_by(AuditLog.action)
.order_by(n.desc(), AuditLog.action)
)
return {"actions": [{"value": a, "count": c} for a, c in db.execute(stmt).all()]}
@router.delete("")
def clear_audit(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
"""Clear the audit log. Leaves ONE row recording who cleared it + how many (tamper-evidence)."""
+112 -147
View File
@@ -1,6 +1,6 @@
"""Auth HTTP endpoints (OAuth login/callback, register/verify, password, session mgmt).
Split out of app/auth.py (R12 S2); the core auth surface (current_user, scopes, token
Split out of app/auth.py; the core auth surface (current_user, scopes, token
helpers) stays in app/auth.py, imported below."""
import secrets
@@ -16,6 +16,7 @@ from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import apitokens
from app import email as email_mod
from app import logs
from app import mailtext
@@ -29,8 +30,8 @@ from app.auth import (
WRITE_SCOPES,
current_user,
env_admin_canonicals,
env_allowed_canonicals,
find_user_by_email,
get_or_create_demo_user,
google_enabled,
is_allowed,
log,
@@ -39,7 +40,7 @@ from app.auth import (
)
from app.config import settings
from app.db import SessionLocal, get_db
from app.models import AuthToken, DemoWhitelist, Invite, OAuthToken, User, display_text
from app.models import AuthToken, Invite, OAuthToken, User, display_text
from app.ratelimit import RateLimiter, client_key
from app.security import encrypt, hash_password, hash_token, verify_password
from app.utils import canonical_email, valid_email
@@ -53,7 +54,7 @@ VERIFY_TTL = timedelta(hours=24)
# in and sets no password.
APPROVAL_VERIFY_TTL = timedelta(days=7)
RESET_TTL = timedelta(hours=1)
# An admin-issued set-password link (E12 S5) travels out of band — the admin has to actually reach the
# An admin-issued set-password link travels out of band — the admin has to actually reach the
# person and hand it over — so the one-hour reset window is far too short. A day matches VERIFY_TTL and
# the affordance the admin was shown.
ADMIN_SETPW_TTL = timedelta(hours=24)
@@ -72,7 +73,6 @@ _reset_limiter = RateLimiter(max_events=5, window_seconds=300)
# Separate from _reset_limiter on purpose: opening a link must not eat the budget for REQUESTING one.
_reset_check_limiter = RateLimiter(max_events=10, window_seconds=300)
_verify_resend_limiter = RateLimiter(max_events=5, window_seconds=300)
_request_access_limiter = RateLimiter(max_events=5, window_seconds=300)
_suspend_email_limiter = RateLimiter(max_events=1, window_seconds=3600)
@@ -92,8 +92,6 @@ def _notify_suspended(background: BackgroundTasks, user: User) -> None:
)
_demo_limiter = RateLimiter(max_events=5, window_seconds=60)
router = APIRouter(prefix="/auth", tags=["auth"])
_oauth = OAuth()
@@ -161,24 +159,12 @@ def notify_admins_of_request(
background.add_task(email_mod.send_admin_new_request, recipients, lang, requester)
def is_demo_allowed(db: Session, email: str) -> bool:
"""Whether this email may enter the shared demo account (no Google sign-in)."""
if not email:
return False
return (
db.execute(
select(DemoWhitelist).where(DemoWhitelist.email == email.lower())
).scalar_one_or_none()
is not None
)
class InviteRequest(NamedTuple):
"""What an access request settled on.
`invite` is the row the request now lives under — which is NOT always the address that was just
typed: one Gmail mailbox spelled two ways settles on whichever row already exists, and that row's
address is where mail will go. Callers that tell the user what happened need it (E9.5 S4).
address is where mail will go. Callers that tell the user what happened need it.
`is_new` is the old boolean return in a name: true only when a row was created or a denied one
reopened, i.e. exactly when admins should hear about it. A repeat request must not re-spam them.
"""
@@ -196,14 +182,14 @@ def upsert_pending_invite(db: Session, email: str, lang: str | None = None) -> I
not reopen a decision that has already been made for that mailbox. The row is still STORED under
the address as typed, because that is where mail has to go.
`lang` is the language of the surface the request came from — the sign-in page, or the locale
Google reported for a refused sign-in. The latest one wins: someone who asks again from a
Hungarian page is telling us something newer than last time. Anything unsupported is ignored
`lang` is the language the request came in — the locale Google reported for a refused sign-in.
The latest one wins: someone who asks again under another locale is telling us something newer
than last time. Anything unsupported is ignored
rather than stored, so a forged body cannot put junk in the column.
The commit contract is unchanged: this still commits ONLY when it writes something. Callers rely
on that — `_register_account` persists its own flushed user precisely because a no-op here
commits nothing (E9.5 S6).
commits nothing.
"""
email = (email or "").lower()
if not valid_email(email):
@@ -215,6 +201,12 @@ def upsert_pending_invite(db: Session, email: str, lang: str | None = None) -> I
.scalars()
.all()
)
# An address the env already lets in has nothing to ask for, and a pending row for it would be
# worse than nothing: `is_allowed` answers from the invite rows once any exist, so a stranger
# registering a sibling spelling (`b.ob@` beside an env-granted `bob@`) would lock the real
# owner out of a grant no admin could see.
if not rows and canon in env_allowed_canonicals():
return InviteRequest(None, False)
# An approved row wins, so a denied duplicate can't mask an approval. Then a row that is not
# DENIED, preferring the exact spelling — a denied sibling must never be picked while a pending
# row exists, or the reopen branch below would fire and put one person in the admin's queue
@@ -254,7 +246,7 @@ def _access_notice_url(kind: str, email: str = "", inv: Invite | None = None) ->
It rides the FRAGMENT, not the query string, for the same reason the verify/reset tokens do
(SB3): a fragment never reaches the server, a proxy access log or a Referer header — and this
notice now carries the user's address. Naming it is the point of E9.5 S4: on 2026-09-03 a
notice now carries the user's address. Naming it is the point: on 2026-09-03 a
registration as `npeter83.temu@` met an OAuth claim of `npeter83temu@`, and a screen that said
"request recorded" without saying FOR WHAT hid the mismatch for a day.
@@ -306,9 +298,10 @@ class EmailPasswordIn(BaseModel):
password: str | None = None
# The language the sign-up page was in. Captured at registration because it is the ONLY moment
# the choice exists anywhere the server can see: it lives in the browser until an account does,
# and a preference nobody recorded cannot be backfilled later (the lesson `google_email` taught
# in E9.5 S3 — those rows still read NULL until each user signs in again). Nothing reads it yet;
# the bilingual mail templates are E9.5 S7.
# and a preference nobody recorded cannot be backfilled later (as `google_email` showed: those
# rows still read NULL until each user signs in again). It is stored as the account's
# `preferences["language"]`, which the UI starts in and every mail to the account is written in
# (`mailtext.lang_for_user` / `lang_for_email`).
lang: str | None = None
@@ -408,7 +401,7 @@ async def callback(
return RedirectResponse(url=_access_notice_url("requested", email, req.invite))
return RedirectResponse(url=_access_notice_url("denied"))
user = db.query(User).filter(User.google_sub == sub).one_or_none()
user = db.execute(select(User).where(User.google_sub == sub)).scalar_one_or_none()
if user is None:
# No account carries this Google identity yet. If one already exists for this MAILBOX —
# e.g. an email+password registration — adopt the Google identity onto it instead of
@@ -421,7 +414,11 @@ async def callback(
# (ids 13 and 14). Exact spelling still wins when both exist, so an account that already
# matches literally is never passed over for a canonical sibling.
canon = canonical_email(email)
candidates = db.query(User).filter(User.email_canonical == canon).order_by(User.id).all()
candidates = (
db.execute(select(User).where(User.email_canonical == canon).order_by(User.id))
.scalars()
.all()
)
existing = next((u for u in candidates if u.email == email), None) or next(
iter(candidates), None
)
@@ -447,9 +444,10 @@ async def callback(
return RedirectResponse(url=_access_notice_url("denied"))
user = existing
user.google_sub = sub
# SEC-10: drop any password pre-set on this account while its email was still unverified,
# now that Google attests ownership (bumps the epoch to revoke any session it minted).
revoke_preset_password_if_unverified(user)
# SEC-10: drop any password nobody proved the mailbox for, now that Google attests
# ownership (bumps the epoch to revoke any session it minted). Runs before the
# activation below, which is what it reads.
revoke_untrusted_preset_password(db, user)
# Google verified the email and is_allowed granted access, so finish activating a
# previously pending password registration (otherwise current_user would bounce it).
user.email_verified = True
@@ -480,7 +478,16 @@ async def callback(
# default for a brand-new account (its role attribute is still unset before flush).
# Canonical, like every other read of the env lists: a dotted or `+tag`ged ADMIN_EMAILS entry
# would otherwise never bootstrap its admin, because Google answers with the plain primary.
#
# Becoming admin this way drops any password the account holds, on whichever sign-in it
# happens. Nothing here can tell who chose that password: without SMTP registration marks the
# address verified with no proof (and SMTP configured later never revisits it), and an admin's
# Activate vouches for access, not for the password. So a stranger who registered the address
# first, and was let in, would otherwise see their own live session become an admin one. The
# real owner, signed in through Google right now, sets a new password in Settings.
if canonical_email(email) in env_admin_canonicals():
if user.role != "admin" and user.password_hash is not None:
_revoke_password(db, user)
user.role = "admin"
elif not user.role:
user.role = "user"
@@ -515,7 +522,7 @@ def _sync_google_email(db: Session, user: User, email: str) -> None:
of the spelling they chose, because sign-in matches the stored address first. Google returns the
account's PRIMARY address, which need not be what they registered with — and can never contain a
`+` at all — so without this guard every plus-tagged registration lost its tag on the first Google
sign-in. (E9.5 S3 made that reachable by matching adoption across spellings; the rewrite itself is
sign-in. (Matching adoption across spellings made that reachable; the rewrite itself is
older.)
Never overwrite with an address another account already owns either: that would hit the unique
@@ -525,7 +532,9 @@ def _sync_google_email(db: Session, user: User, email: str) -> None:
return
if canonical_email(user.email) == canonical_email(email):
return # same mailbox, their spelling — leave it alone
clash = db.query(User).filter(User.email == email, User.id != user.id).one_or_none()
clash = db.execute(
select(User).where(User.email == email, User.id != user.id)
).scalar_one_or_none()
if clash is None:
user.email = email
@@ -582,9 +591,11 @@ def _complete_link(
if link_uid != request.session.get("user_id") and link_uid not in accounts:
return RedirectResponse(url="/?link=error")
user = db.get(User, link_uid)
if user is None or user.is_demo:
if user is None:
return RedirectResponse(url="/?link=error")
other = db.query(User).filter(User.google_sub == sub, User.id != user.id).one_or_none()
other = db.execute(
select(User).where(User.google_sub == sub, User.id != user.id)
).scalar_one_or_none()
if other is not None:
return RedirectResponse(url="/?link=conflict")
if user.google_sub is not None and user.google_sub != sub:
@@ -628,73 +639,12 @@ async def logout(request: Request):
return JSONResponse({"ok": True, "switched": False})
class AccessRequestIn(EmailIn):
# The language of the page this request was made from. Same reasoning as registration's `lang`
# (and the same shape): it is the ONLY moment this preference exists anywhere the server can
# see, since the requester has no account to carry it, and it decides what language the approval
# or denial mail is written in.
lang: str | None = None
@router.post("/request-access")
async def request_access(
body: AccessRequestIn,
request: Request,
background: BackgroundTasks,
db: Session = Depends(get_db),
) -> dict:
"""Public: ask for access. Idempotent — repeat requests for the same email don't
duplicate or re-spam admins (upsert answers `is_new=False` for an already-pending row).
Answers a uniform ``"pending"`` for EVERY outcome — already-allowed, newly-pending,
already-pending, and rate-limited alike. The old ``"approved"`` reply for an allow-listed
address made this a public enumeration oracle for the allow-list (which the rest of the module
is careful to avoid, cf. demo_login); an already-allowed user can still just sign in with Google.
Rate-limited per client IP like register/login/reset, and a block returns "pending" too so the
endpoint can't be probed for throttling either."""
email = (body.email or "").strip().lower()
if not valid_email(email):
raise refused(400, "invalid_email")
if not _request_access_limiter.allow(client_key(request)):
return {"ok": True, "status": "pending"}
if is_allowed(db, email):
return {"ok": True, "status": "pending"} # already allowed — don't reveal it
req = upsert_pending_invite(db, email, body.lang)
if req.is_new:
notify_admins_of_request(db, background, email)
return {"ok": True, "status": "pending"}
@router.post("/demo")
def demo_login(body: EmailIn, request: Request, db: Session = Depends(get_db)) -> dict:
"""Hidden demo entry: a whitelisted email logs straight into the shared demo account,
no Google OAuth. The login page fires this quietly (debounced) as the user types/pastes,
so there is no visible 'demo login' button. Rate-limited per IP; when blocked it answers
exactly like a non-match (authenticated=false) so the field can't be hammered as an
enumeration oracle. On a match it sets the session and the client reloads into the app."""
if not _demo_limiter.allow(client_key(request)):
return {"ok": True, "authenticated": False}
email = (body.email or "").strip().lower()
if valid_email(email) and is_demo_allowed(db, email):
demo = get_or_create_demo_user(db)
# Isolate the demo: drop any real-account wallet on this tab first, so the demo session
# can't switch to / act as a previously signed-in account via the multi-account header.
request.session.clear()
request.session["user_id"] = demo.id
_remember_epoch(request, demo)
# The demo key IS an address, so it is not in the line — see the denied branch above.
log.info("signed in", extra=logs.event("auth.login.ok", method="demo", uid=demo.id))
return {"ok": True, "authenticated": True}
return {"ok": True, "authenticated": False}
def _establish_session(request: Request, user: User) -> None:
"""Sign `user` in for this browser session — the ONE sign-in path, shared by the Google callback
and password login, so both join the same account switcher and both stamp `last_login_at` (the
caller commits). Multi-session: every account that authenticated in this browser is remembered so
the user can switch between them; only accounts that actually proved themselves HERE land in that
wallet (it is the proof they may be switched to). The demo entry deliberately does not come
through here: it clears the wallet instead, and its shared account has no meaningful last login."""
wallet (it is the proof they may be switched to)."""
accounts = [a for a in (request.session.get("account_ids") or []) if a != user.id]
accounts.append(user.id)
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
@@ -784,7 +734,7 @@ def _consume_password_token(db: Session, raw: str | None) -> tuple[User, str] |
def issue_set_password_link(db: Session, user: User) -> tuple[str, datetime]:
"""An admin-issued, single-use link that lets `user` set a password without any email (E12 S5).
"""An admin-issued, single-use link that lets `user` set a password without any email.
One mechanism answers three things: SEC-15 (on a no-SMTP instance every registration is
auto-verified, so a pre-empted address cannot be recovered by mail), SEC-16 (a pre-empted address
@@ -866,10 +816,10 @@ def _register_account(email: str, password: str, lang: str | None = None) -> Non
# for the same input — so it is not a new capability, just the end of a dead end. An
# account that is already confirmed still gets nothing: there is nothing to confirm.
#
# And a re-send is ALL it does: the submitted password is discarded. E9.5's closing
# And a re-send is ALL it does: the submitted password is discarded. A closing
# review tried "while the address is unverified, the newest registration owns it"
# (replace the hash, kill the outstanding verify tokens) as a fix for address
# pre-emption, and round 2 showed it was worse than the problem — DO NOT REPEAT IT.
# pre-emption, and it proved worse than the problem — DO NOT REPEAT IT.
# The rule is symmetric, and an attacker can always be last: it hands an unauthenticated
# caller the power to overwrite a real registrant's chosen password and delete the
# confirmation link already in their inbox, and on a SEC-10-stripped row (password_hash
@@ -878,7 +828,7 @@ def _register_account(email: str, password: str, lang: str | None = None) -> Non
# primitive: password-reset accepts this account, proves the mailbox, rewrites the hash
# and bumps the session epoch. The residual is logged as SEC-16.
user = existing
if not user.email_verified and not user.is_demo and email_mod.email_enabled():
if not user.email_verified and email_mod.email_enabled():
send_verification_link(db, user)
return
# Without working SMTP we can't deliver a verification link, so email ownership can't gate
@@ -918,7 +868,7 @@ def _register_account(email: str, password: str, lang: str | None = None) -> Non
db.commit()
if email_ok:
send_verification_link(db, user)
# Fired unconditionally, unlike the other two callers which gate on a fresh invite row: for
# Fired unconditionally, unlike the Google callback, which gates on a fresh invite row: for
# an address the admin already approved, nothing lands in the Access-requests queue, so this
# is the only place the account's existence is announced at all. Delivery is a separate
# matter — with no working SMTP `_send` drops it, which is why the pre-approved registrant
@@ -928,7 +878,7 @@ def _register_account(email: str, password: str, lang: str | None = None) -> Non
def issue_verification_link(db: Session, user: User, ttl: timedelta | None = None) -> str:
"""A fresh confirmation link for `user`, returned rather than sent — for a mail that carries the
link inside a larger message (the branching approval/invitation mails, E9.5 S6).
link inside a larger message (the branching approval/invitation mails).
Same token machinery and the same fragment (SB3) as the standalone confirmation mail, so the two
kinds of link cannot drift apart in shape or in how they are consumed."""
@@ -943,8 +893,8 @@ def send_verification_link(db: Session, user: User) -> bool:
address is the one its owner chose to be reachable at, and a `+tag` there is usually deliberate.
The token rides the URL FRAGMENT (#) so it can never reach a proxy/access log or a Referer (SB3).
One definition, three callers — registration, the self-service resend, and the admin-side resend
(E9.5 S5). The link's shape and TTL must not drift between them; a verification mail that differs
One definition, three callers — registration, the self-service resend, and the admin-side resend.
The link's shape and TTL must not drift between them; a verification mail that differs
by who triggered it is a bug waiting to be discovered by exactly one user.
Returns whether the mail actually went out, so a caller with a human waiting on the answer (the
@@ -967,8 +917,8 @@ def verify_email_confirm(body: TokenIn, db: Session = Depends(get_db)) -> dict:
dropped the password this account registered with, so sign-in would answer with the deliberately
uniform "Invalid email or password" — true, but unreadable to the person who just proved they own
the mailbox. Saying so leaks nothing: holding a valid single-use verify token for this account
already proves that much. Since E9.5 S1 the common source of that state is gone (admin approval no
longer revokes), leaving accounts stripped before that change and the Google-adoption path, so this
already proves that much. The common source of that state is gone (admin approval no longer
revokes), leaving accounts stripped before that change and the Google-adoption path, so this
flag is rare now — but it must stay, because those users have no other way back in."""
user = _consume_token(db, body.token, "verify")
if user is None:
@@ -1012,7 +962,7 @@ def _resend_verify_if_eligible(email: str) -> None:
return
with SessionLocal() as db:
user = find_user_by_email(db, email)
if user is not None and not user.email_verified and not user.is_demo:
if user is not None and not user.email_verified:
send_verification_link(db, user)
@@ -1028,7 +978,7 @@ def _login_refused(status: int, reason: str, user: User | None = None, **extra)
that proof (`invalid`, `rate_limited`) carry nothing at all, so they stay byte-identical for a
real account and an unknown one.
Since E9.5 S1 the two gates are order-independent, so `approved` really can be true while
The two gates are order-independent, so `approved` really can be true while
`email_verified` is false, and vice versa — the SPA renders whichever is actually outstanding
rather than assuming a sequence.
"""
@@ -1058,11 +1008,11 @@ def password_login(
email = (body.email or "").strip().lower()
password = body.password or ""
user = find_user_by_email(db, email)
# Always run the KDF (against a decoy hash for an unknown / passwordless / demo account) so the
# Always run the KDF (against a decoy hash for an unknown / passwordless account) so the
# response time is the same whether or not the email exists — no timing-based enumeration.
candidate_hash = user.password_hash if (user and not user.is_demo) else None
candidate_hash = user.password_hash if user else None
password_ok = verify_password(password, candidate_hash or _DECOY_PASSWORD_HASH)
if user is None or user.is_demo or not password_ok:
if user is None or not password_ok:
raise _login_refused(401, "invalid")
if user.is_suspended:
_notify_suspended(background, user)
@@ -1101,7 +1051,7 @@ def password_reset_request(
def _send_reset_if_eligible(email: str) -> None:
"""Background worker for password_reset_request: issue a reset token + email the link, but ONLY
for a real (non-demo) password account. Runs after the response with its own DB session, so the
for a password account. Runs after the response with its own DB session, so the
request's timing never reveals whether the account exists. Token rides the URL FRAGMENT (#) so it
can't leak into proxy/access logs or a Referer (SB3).
@@ -1112,7 +1062,7 @@ def _send_reset_if_eligible(email: str) -> None:
proves identity through the session rather than by mail."""
with SessionLocal() as db:
user = find_user_by_email(db, email)
if user is None or user.is_demo:
if user is None:
return
if user.password_hash or user.password_revoked_at:
raw = _issue_token(db, user, "reset", RESET_TTL)
@@ -1169,9 +1119,8 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
# about. The person still confirms their address the normal way.
if kind == "reset":
user.email_verified = True
bump_session_epoch(
user
) # SA4: a reset (often a compromise response) kills all existing sessions.
# SA4: a reset (often a compromise response) kills all existing sessions — and API tokens.
bump_session_epoch(db, user)
# Burn any other outstanding reset tokens for this user.
for row in db.execute(
select(AuthToken).where(
@@ -1194,43 +1143,69 @@ def _remember_epoch(request: Request, user: User) -> None:
request.session["epochs"] = epochs
def bump_session_epoch(user: User) -> None:
def bump_session_epoch(db: Session, user: User) -> None:
"""Invalidate every EXISTING signed cookie for this account by advancing its epoch (SA4). Cookies
minted before the bump carry a now-stale epoch and are rejected by `current_user`."""
user.session_epoch = (user.session_epoch or 0) + 1
minted before the bump carry a now-stale epoch and are rejected by `current_user`. The account's
API tokens go with them (`apitokens.revoke_all`): a token minted from a stolen session must not
outlive the answer to that theft. Not committed — the caller commits it with its change.
The user's row is locked before the tokens go, the lock `create_token` takes too: a token made at
the same moment either commits first and is deleted here, or waits and sees the new epoch. FOR NO
KEY UPDATE (`key_share`): the two still exclude each other, but a row that only refers to the user
(a notice, an audit line) is not held up — a token's expiry notice would deadlock with it."""
if user.id is None:
user.session_epoch = (user.session_epoch or 0) + 1
return
epoch = db.execute(
select(User.session_epoch).where(User.id == user.id).with_for_update(key_share=True)
).scalar_one()
user.session_epoch = (epoch or 0) + 1
apitokens.revoke_all(db, user.id)
def revoke_preset_password_if_unverified(user: User) -> None:
def revoke_untrusted_preset_password(db: Session, user: User) -> None:
"""SEC-10 (account pre-emption): a password set on an account whose email was NEVER verified is
untrusted — whoever registered the address (to pre-empt the real owner) could have chosen it. When
the real owner attests ownership through Google, drop that password and bump the epoch to revoke any
session it minted; the owner re-sets one via password-reset. No-op for an already-verified account,
so a legit no-SMTP registration is unaffected.
session it minted; the owner re-sets one via password-reset, or from Settings while the Google
session lasts.
The Google-adoption callback is the ONLY caller (E9.5 S1). Admin approval used to call this too, and
dropping it was right for the reason S1 gave — the revoke destroyed the password a legitimate
A NEVER-ACTIVATED account is untrusted too, verified or not: without SMTP, registration marks the
address verified with no proof at all. For an ADMIN_EMAILS / ALLOWED_EMAILS address no invite row
gates the registration, so a stranger who registered it first would otherwise be activated here
with their own password still working. (An account the callback makes ADMIN loses its password
whatever its state — see the ADMIN_EMAILS block there.)
The Google-adoption callback is the ONLY caller. Admin approval used to call this too, and
dropping it was right for the reason given — the revoke destroyed the password a legitimate
registrant had just chosen, and since the admin normally decides before the user reads their mail,
that made the "your password was cleared" detour the DEFAULT path rather than an edge case.
S1 also claimed the revoke was REDUNDANT there, because `password_login` refuses an unverified
The revoke was also called REDUNDANT there, because `password_login` refuses an unverified
account before it reads `is_active`. That part was wrong, and the epic's own review caught it: the
gate holds only while the address is unverified, and S6's approval mail exists to get the real
gate holds only while the address is unverified, and the approval mail exists to get the real
owner to end that state — at which point a pre-emptor's password works. Nothing replaced the
revoke, and that is deliberate. The review tried "the newest registration owns an unverified
account" and reverted it: the rule is symmetric, so an attacker can simply register last, and it
would have handed anonymous callers a write onto a real account's credential. What the owner
actually has is password-reset, which proves the mailbox, rewrites the hash and bumps the epoch.
The residual is logged as SEC-16; re-adding a caller here would put "your password was cleared"
back in front of every legitimate registrant, which is the whole reason S1 removed it.
back in front of every legitimate registrant, which is the whole reason it was removed.
`password_revoked_at` is what makes "the owner re-sets one via password-reset" actually true. The
revoke leaves `password_hash IS NULL`, which is also what a Google-only account looks like — and
_send_reset_if_eligible keyed off exactly that, so a revoked account silently got no reset link and
was locked out for good (it could not sign in either). The stamp separates the two cases."""
if not user.email_verified:
user.password_hash = None
user.password_revoked_at = datetime.now(timezone.utc)
bump_session_epoch(user)
if not user.email_verified or (not user.is_active and user.password_hash is not None):
_revoke_password(db, user)
def _revoke_password(db: Session, user: User) -> None:
"""Drop the account's password, stamp why (so password-reset still accepts it) and end every
session it signed in."""
user.password_hash = None
user.password_revoked_at = datetime.now(timezone.utc)
bump_session_epoch(db, user)
@router.post("/logout-others")
@@ -1239,12 +1214,10 @@ def logout_others(
) -> dict:
"""SA4: sign every OTHER session for this account out — bump the epoch so all existing cookies
are rejected, then re-stamp THIS cookie so the caller stays signed in. For "I left myself logged
in somewhere" / a suspected session compromise. The demo account (shared) is excluded."""
if user.is_demo:
raise HTTPException(status_code=403, detail={"code": "demo"})
in somewhere" / a suspected session compromise."""
# Commit the bump BEFORE re-stamping this cookie, so a failed commit can't strand the current
# session at a newer epoch than the DB (see set_password).
bump_session_epoch(user)
bump_session_epoch(db, user)
db.commit()
_remember_epoch(request, user)
log.info("Logout-others: uid=%s (epoch=%s)", user.id, user.session_epoch)
@@ -1260,8 +1233,6 @@ async def link_google(
"""Start linking a Google account to the signed-in (e.g. password) account. Requests only
the identity scopes — YouTube access is granted separately via /auth/upgrade. The callback
sees `oauth_link_uid` and attaches the identity instead of creating a new account."""
if user.is_demo:
return RedirectResponse(url="/")
client = google_oauth(db)
if client is None:
return RedirectResponse(url="/?link=error")
@@ -1287,8 +1258,6 @@ def set_password(
"""Set or change the signed-in account's password. Setting a first password (Google-only
account) needs no current password — the session already proves identity. Changing an
existing one requires the current password, so a hijacked session can't silently rotate it."""
if user.is_demo:
raise HTTPException(status_code=403, detail={"code": "demo"})
new_password = body.password or ""
if len(new_password) < PASSWORD_MIN_LEN:
raise refused(400, "weak_password", min=PASSWORD_MIN_LEN)
@@ -1300,7 +1269,7 @@ def set_password(
# SA4: a password change logs out every OTHER session; re-stamp this cookie at the new epoch so
# the person making the change stays signed in here. Commit BEFORE re-stamping so a failed commit
# can't leave the cookie at a newer epoch than the DB (which would wrongly 401 THIS session).
bump_session_epoch(user)
bump_session_epoch(db, user)
db.commit()
_remember_epoch(request, user)
log.info("Password set/changed: uid=%s", user.id)
@@ -1319,10 +1288,6 @@ async def upgrade(
(unsubscribe). The shared callback stores whatever Google returns, so `can_read` /
`can_write` flip on once granted. Anything other than an explicit `write` defaults to
the least-privileged read scope."""
# This is the browser-facing redirect entry point, so the shared demo account is bounced
# straight home (no YouTube link is ever attached to it) rather than shown a raw 403.
if user.is_demo:
return RedirectResponse(url="/")
client = google_oauth(db)
if client is None:
return RedirectResponse(url="/")
+30 -27
View File
@@ -3,16 +3,17 @@ hidden, tags) and per-channel sync state. Local reads/writes plus YouTube-side w
(subscribe / unsubscribe) that require the optional write scope."""
from datetime import datetime, timezone
from functools import partial
from fastapi import APIRouter, BackgroundTasks, Depends
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import Select, func, select, update
from sqlalchemy.orm import Session
from app.collation import collated, sorted_words
from app.mailtext import lang_for_user
from app import logs, quota
from app.auth import admin_user, current_user, require_human, require_write_scope
from app import logs, quota, scheduler
from app.auth import admin_user, current_user, require_write_scope
from app.db import get_db
from app.models import (
LIVE_OR_UPCOMING,
@@ -67,8 +68,8 @@ def list_channels(user: User = Depends(current_user), db: Session = Depends(get_
# Per-channel aggregates over the shared catalog (channel_aggregates), computed on read rather than
# denormalised: an Index Only Scan over a ~20 MB index, where it used to read every video's heap page
# (681 MB on prod, a 2–6 s first open once something else had pushed it out of memory, E31 S4d).
# (The Normal/Shorts/Live breakdown moved to the channel-detail page in R14, so it is no longer
# (681 MB on prod, a 2–6 s first open once something else had pushed it out of memory).
# (The Normal/Shorts/Live breakdown moved to the channel-detail page, so it is no longer
# computed here — the manager table dropped its N/S/L column.)
channel_ids = [c.id for _, c in rows]
agg: dict[str, dict] = {}
@@ -163,8 +164,8 @@ def discover_channels(
can judge a channel BEFORE subscribing — but we do NOT pull its videos. Most-present
channels come first; subscribing is a separate action."""
# Resolve & cache the user's own channel id so it can be excluded — you don't subscribe
# to yourself. Lazy + cached; needs OAuth, so demo / token-less users just skip it.
if user.yt_channel_id is None and user.token is not None and not user.is_demo:
# to yourself. Lazy + cached; needs OAuth, so token-less users just skip it.
if user.yt_channel_id is None and user.token is not None:
try:
with (
quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER),
@@ -192,8 +193,7 @@ def discover_channels(
# (fresh subscriber counts/thumbnails carry through without a re-query). Only the title can
# change, which affects the tie-break order → re-sort in Python instead of re-hitting the DB.
need = [ch.id for ch, _, _ in rows if ch.details_synced_at is None][:DISCOVERY_ENRICH_CAP]
# Skip the quota-spending enrich for the demo account (it just sees the stub metadata).
if need and not user.is_demo:
if need:
try:
with (
quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER),
@@ -277,7 +277,7 @@ def _channel_detail_dict(
"explored": explored,
"blocked": blocked,
"stored_videos": stored,
# Per-type breakdown (Normal / Shorts / Live) — R14 moved it off the manager table onto the
# Per-type breakdown (Normal / Shorts / Live) — moved off the manager table onto the
# channel page. Same aggregation the manager list uses (count_normal = stored − shorts − live).
"count_normal": count_normal,
"count_short": count_short,
@@ -311,12 +311,12 @@ def channel_detail(
Lazily fills the About metadata (view count, join date, banner, links) the first time it's
needed (a stub) and refreshes it once it is stale (`DETAILS_MAX_AGE`), which is how a channel nobody subscribes to (an explored one, outside the
subscription import) gets a banner that still resolves. The enrich uses the API key (1 unit,
cached after), so it's skipped for the quota-less demo account."""
cached after)."""
channel = get_or_404(db, Channel, channel_id, what="channel")
# No `published_at is None` sentinel: a terminated channel never gets one, so it re-fetched on
# every view; rows enriched before that field existed are long past DETAILS_MAX_AGE anyway.
stale = details_stale(channel, datetime.now(timezone.utc))
if stale and not user.is_demo:
if stale:
try:
with (
quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER),
@@ -350,7 +350,7 @@ def channel_detail(
)
blocked = _is_blocked(db, user, channel_id)
# Stored total + the Normal/Shorts/Live split in one grouped pass (same filters as the manager
# list), so the channel page can show the per-type breakdown R14 moved off the table.
# list), so the channel page can show the per-type breakdown that moved off the table.
n_all, n_short, n_live = db.execute(
select(
func.count(Video.id),
@@ -389,7 +389,7 @@ def channel_detail(
def explore_channel(
channel_id: str,
page_token: str | None = None,
user: User = Depends(require_human),
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Open an un-subscribed channel for browsing. Records the exploration (a per-user,
@@ -397,7 +397,7 @@ def explore_channel(
drives the later cleanup), flags the channel ephemeral unless someone organically follows
it, then ingests one page of its uploads (newest-first) marked via_explore + enriched.
Returns the next page token so the page can "load more". Guarded against the shared
backfill quota reserve; blocked for the demo account (it spends shared quota)."""
backfill quota reserve."""
channel = get_or_404(db, Channel, channel_id, what="channel")
if _is_blocked(db, user, channel_id):
raise refused(403, "channel_blocked")
@@ -482,7 +482,6 @@ class ChannelUpdateIn(BaseModel):
def update_channel(
channel_id: str,
body: ChannelUpdateIn,
background: BackgroundTasks,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
@@ -493,10 +492,7 @@ def update_channel(
sub.priority = body.priority
if body.hidden is not None:
sub.hidden = body.hidden
# The demo account is SHARED, so it must not persist deep_requested (a quota-relevant flag) onto
# the shared subscription — skip the write entirely for demo, which also keeps deep_turned_on
# False so the immediate backfill below can never fire for it.
if body.deep_requested is not None and not user.is_demo:
if body.deep_requested is not None:
# Opt this channel into (or out of) full-history backfill. The deep scheduler
# picks the flag up on its next run; turning it off won't undo already-fetched
# videos, it just stops further deep paging if nobody else still wants it.
@@ -507,17 +503,24 @@ def update_channel(
# Opting a channel into full history should give an immediate feed: if we haven't even
# fetched its recent uploads yet, do that one channel now (cheap) instead of waiting for
# the scheduler. (deep_turned_on is only ever True for a non-demo user — see the guard above.)
# the scheduler.
if deep_turned_on and channel is not None and channel.recent_synced_at is None:
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_BACKFILL_RECENT):
run_recent_backfill(db, [channel], max_channels=1)
# …and kick the FULL deep backfill for this one channel immediately in the background, so the
# whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks. The
# request returns right away; the channel page's live poll shows the count climb. The scheduler
# still finishes a very large channel's tail on its normal runs (deep_requested stays set).
if deep_turned_on:
background.add_task(backfill_channel_now, channel_id, user.id)
# …and kick the FULL deep backfill for this one channel immediately, as a recorded run of its own,
# so the whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks.
# The request returns right away; the channel page's live poll shows the count climb. The
# scheduler still finishes a very large channel's tail on its normal runs (deep_requested stays set).
# A channel whose history is all in gets no run: it would only say "nothing to do", one kept row
# per flip of the flag.
if deep_turned_on and not (channel is not None and channel.backfill_done):
scheduler.run_ad_hoc(
scheduler.CHANNEL_BACKFILL,
partial(backfill_channel_now, channel_id, user.id),
actor_id=user.id,
target=channel_id,
)
return {
"ok": True,
+6 -4
View File
@@ -61,7 +61,7 @@ def set_config(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
s = sysconfig.spec(key)
s = sysconfig.config_page_spec(key)
if s is None:
raise not_found("config_key")
if "value" not in body.model_fields_set:
@@ -89,7 +89,9 @@ def set_config(
before=None if s.secret else {"value": _redact(old)},
after=None if s.secret else {"value": _redact(new)},
)
db.commit()
# The override is committed even when the trail skips it: re-saving the value in force still
# pins it as an override, and an uncommitted write would be dropped with the session.
db.commit()
_apply_runtime(db, key)
return sysconfig.describe(db)
@@ -100,7 +102,7 @@ def reset_config(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
s = sysconfig.spec(key)
s = sysconfig.config_page_spec(key)
if s is None:
raise not_found("config_key")
old = None if s.secret else sysconfig.get(db, key)
@@ -139,7 +141,7 @@ def test_issue_tracker(
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Check the configured issue tracker (E16): the repository answers with this token, takes
"""Check the configured issue tracker: the repository answers with this token, takes
issues, and is PRIVATE — the one it must be, since an issue carries its reporter's log."""
try:
return {"ok": True, **gitea.check(db)}

Some files were not shown because too many files have changed in this diff Show More