Compare commits

...
53 Commits
Author SHA1 Message Date
peter c8247c0dde release: 0.107.2 (improvement/gate-runner) 2026-10-17 04:37:30 +02:00
peter 722c37a6d0 chore(release): 0.107.2 — the gate lists and runs single lanes, the backend tests use every core 2026-10-17 04:30:33 +02:00
peter 5ba5679239 test: tests that assumed an idle machine wait for what they assert, or carry a budget for their work
Under a busy machine (the gate's lanes side by side, 12 busy cores) these went red at random: a press landed while the button was still busy, an answer was settled before it was asked for, a page grew past one wait's budget, a lazy chunk was transformed inside a 1 s wait, focus was taken a frame after the press. They now wait for the condition they depend on, preload the chunks they open, and the tests that do heavy work (whole-repo scans, the type-aware linter, module resets) carry an explicit budget naming that work. The whole suite four times under 12 busy-loop processes: red 4/4 before, 0/4 after.
2026-10-17 01:08:15 +02:00
peter 926e5e9899 chore(gate): check.sh can list its lanes and run one of them
--list prints the lane names, --only "<lane>" runs that one lane, so a runner can schedule the lanes itself and a red lane can be re-run on its own. An --only run reports that lane, never the gate, and an unknown name exits 2 instead of passing with nothing run; the contract test holds both.
2026-10-16 02:53:30 +02:00
peter 9251918e08 chore(test): the DB lane runs on every core — each pytest-xdist worker gets its own database 2026-10-16 01:13:31 +02:00
peter 2b6d968833 release: 0.107.1 (improvement/gate-timing) 2026-10-16 00:45:45 +02:00
peter 95e6b2c3d9 chore(release): 0.107.1 — a failed download's explanation says press Retry 2026-10-16 00:33:59 +02:00
peter 80c55fbf1b merge bug/download-retry-copy into improvement/gate-timing 2026-10-16 00:33:38 +02:00
peter 011f269152 chore(gate): every lane reports how long it took, and a run can log the times for comparison 2026-10-16 00:13:47 +02:00
peter dcb244dd04 fix(downloads): a failed download's explanation says press Retry, not retry it from a list that is gone 2026-10-15 22:56:40 +02:00
peter 08a9f4e255 release: 0.107.0 (feature/e39b-machine-access) 2026-10-15 06:20:35 +02:00
peter 1673134c36 chore(release): 0.107.0 — API tokens for your machines, share a whole folder by link 2026-10-15 06:09:23 +02:00
peter 27427eef32 fix(files): ending every session and making a token lock the user without holding up rows that refer to it 2026-10-15 06:05:51 +02:00
peter a248ca6c13 fix(files): the token list is fresh after the sessions end, a copy again reads Copied again, the Copied button says so 2026-10-15 05:47:00 +02:00
peter 620f84dba4 fix(files): a token made as every session ends goes with them, and a file being shared holds its job 2026-10-15 05:47:00 +02:00
peter 2efa7d5a75 fix(files): an upload is said once in any order, copy buttons speak, a link list that failed says so
- The queue holds an upload it sees land done while this tab's upload still waits for its answer, so
  the dialog's notice is the only one whichever comes first; another device's upload follows after.
- One `useCopyFeedback` for every Copy beside a value (token, share link, set-password link): Copied
  for a moment, a notice a screen reader hears, a warning when the clipboard refuses; the token
  dialog's two Copy buttons are named by what they copy.
- The share dialog's link list shows loading, and a load error with Retry, never "no links" — and no
  New link beside a list it has not seen.
- The settings texts say that signing out elsewhere or a password change revokes the API tokens.
- DESIGN.md logs the folder-link decisions; the design lab keeps only the chosen renderings of the
  Links, Tokens and Files rounds.
2026-10-15 04:58:17 +02:00
peter 827df7494c fix(files): ending every session ends every API token, and the share links hold their folder
- A password reset or change, "log out everywhere" and a revoked untrusted password revoke all of the
  account's API tokens too (audited): one minted from a stolen session no longer outlives the answer.
- A bearer token or link grant with a non-ASCII character is refused (401/403), not a 500.
- Making a token's grants or a folder link holds the folder, so a racing delete cannot fail the
  foreign key; a link's view count is one atomic UPDATE.
- A folder link's card counts exactly the files its page lists (one `links.folder_files`).
- A file with no extension is served without a trailing dot.
- 0106's downgrade names a Hub folder so it never collides with one of the user's folders.
- The speed baseline reads share tokens through the app (the database keeps them hashed/encrypted).
- Tests: the token scope on the real session stack, the folder poster, the Hub refused for a link,
  0106's downgrade data step, 0107 without a key; docs/comments follow (three folder kinds, folder
  links, key change, the cleanup job's token warnings).
2026-10-15 04:58:16 +02:00
peter 86712b6b0e feat(files): share a folder by link — its page lists what is in it, plays a video above the list
A /watch link may now open one of its owner's own folders (migration 0108: folder_id beside job_id,
exactly one set; a folder's links go with it). The page shows what is in the folder when it is
opened, newest first: a video or a song plays in the same player a file link has, above the list,
each keeping its own resume point; a document only downloads, never opens as a page. A file moved
out stops opening through the link at its next request. Password, expiry and stream-only work as on
a file link; the unfurl names the folder and counts its files, never their names. Source folders
and the Hub cannot be shared.

The folder menu gets "Copy link" and "Share with a link…" (the Share dialog's link half); the rail
marks a folder a live link opens.

Two fixes in the Share dialog on the way: revoking the last link no longer mints a new one at once,
and the new-link form goes back to downloadable after making a link.
2026-10-15 02:23:24 +02:00
peter 90567718b2 feat(files): share-link tokens are kept only hashed and encrypted, and every link change is audited
A /watch link's token is the credential, and it sat in the database as written. Migration 0107
replaces the column with the token's SHA-256 (the lookup) and a Fernet-encrypted copy (so the owner
can copy the same link again); every existing address keeps working, and the downgrade decrypts
them back. Without a copy this key can read, a link still opens but its address cannot be shown:
"Copy link" then takes the newest link that has one or makes a new one, and the Share dialog says
so in that row instead of a broken URL. Making, changing and revoking a link are audited, never
with its token or password.
2026-10-15 01:02:29 +02:00
peter c31d7f9d65 fix(files): an upload is announced once, as an upload, and a finish shows in Files at once
An upload that came back done was announced twice: by the upload dialog ("File uploaded") and
again by the tray, which saw a job turn done ("Downloaded: …"). The dialog now marks its job as
announced, so the queue keeps it for the tray but does not say it again. An upload the worker
converted first is still the tray's to announce, now as "Uploaded: …".

A finished job's notice offered "Open" while Files still listed the folder without it (its own
poll runs every 15 s): the tray now refreshes the library the moment it sees a finish.
2026-10-14 23:54:19 +02:00
peter db56f25387 feat(files): API tokens in Settings, the Hub folder in Files, a notice before a token runs out
Settings gains an API tokens tab: a row per token (name and prefix; each
folder with its rights; last use and expiry, "expires soon" marked),
revoked from its menu behind a confirm. A new token is the Hub exchange in
one click, or folders and rights picked per folder (no Write where an
upload cannot land); expiry 30/90 days, 1 year or never (warned). The
token shows once in the same dialog, with Copy and a ready curl line.

The Hub folder closes the Sources in the Files rail, takes uploads and
moved-in files. A token about to run out warns its user once, a week
before (the Files GC's sixth pass); the notice opens the tokens tab.
2026-10-14 23:33:34 +02:00
peter 4f58d15d3f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 21:19:14 +02:00
peter 2c66ce1cf4 release: 0.106.2 (improvement/thumbs-and-apt-refresh) 2026-10-14 21:17:39 +02:00
peter ce45445d17 chore(release): 0.106.2 — pictures that fail to load retry instead of breaking 2026-10-14 21:06:18 +02:00
peter 383d394e32 fix(ui): a picture that fails to load shows the card's own frame, then retries
A bare <img> sent a Referer that Google's image CDNs refuse and had no retry and no fallback, so a
Home shelf card showed the browser's broken-image mark. ui/RemoteImage gathers what Avatar, the
channel banner and the download rows had each hand-rolled: no Referer, up to three retries past the
browser's negative cache, the caller's fallback (by default the card's empty frame) instead of the
broken mark, the failure remembered by URL. Every picture now goes through it; a lint rule refuses
a bare <img>.
2026-10-14 20:51:08 +02:00
peter 06e8291896 build(image): rebuild the apt-get upgrade layer when APT_REFRESH changes
A build cache kept the upgrade layer for as long as nothing above it changed, so a publish shipped
curl and librsvg a release behind fixes already in the Debian archive (the CVE gate refused it).
The release build passes the day; empty, the default for a self-hosted build, leaves the cache alone.
2026-10-14 20:51:07 +02:00
peter db6b60029f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 20:17:05 +02:00
peter 02134799cc release: 0.106.1 (bug/plex-settings-outside-click) 2026-10-14 20:03:38 +02:00
peter c8bc843517 chore(release): 0.106.1 — the Plex player survives a click outside its Settings panel 2026-10-14 19:44:25 +02:00
peter 759a7536b9 fix(history): an overlay opened while a back() is in flight waits to push its entry
The Plex player's ⋯ → Settings closed the menu (its entry went back) and opened the panel a task
later, before that back() had landed. The panel's push was the entry the landing back() took, so
the books counted one entry more than the history held, and closing the panel by a click outside
it stepped back over the player's own entry: the player closed.

A push now waits while a back() of ours is in flight; the popstate that lands the last one
reconciles again.
2026-10-14 19:43:51 +02:00
peter b2d34e6e6c feat(files): API tokens — a machine reaches the library folder by folder
A user can make a token for a machine of theirs (`/api/tokens`, signed-in only): `sfl_` + 32
random bytes + a checksum, only its SHA-256 kept, shown once. Sent as a Bearer header it is
honoured on six `/api/files` routes only — folders, list, file, hash, upload, delete — and there
only in the folders its grants name, with the rights they give (list / read / write / delete).
One check, `FilesCaller.may`, in every route; a folder or file out of reach answers like a
missing one; the files shared with the user are never in reach. A token beats a cookie on the
same request.

The Hub preset makes the user's one Hub folder (a new folder kind) and grants it every right.
Refused tokens are slowed down per client, a runaway machine per token; last use is stamped at
most once a minute; making and revoking a token is audited, and a token's uploads and deletes
name it in their audit rows.
2026-10-14 18:40:23 +02:00
peter c335d54929 release: 0.106.0 (feature/e39-files) 2026-10-13 03:34:56 +02:00
peter 7188ad7d71 chore(release): 0.106.0 — the Files library 2026-10-13 03:24:14 +02:00
peter ebbae6794c fix(files): review round 3, the page — a search waits for the server, the viewer keeps its file, a drop ends the drag
- past the list's cap a search waits in a skeleton, never showing the newest files' 'no match' for a
  file the server is about to find; a sort keeps its stand-in rows
- the viewer keeps the file it was opened on when a server answer replaces the rows under it
- a drop another zone took still clears the page's drop overlay
- 'The first N items' only for a server sort; the past-the-end notice no longer claims the file exists
- a failed server search is tested to say so
2026-10-13 03:15:59 +02:00
peter c9baf68b16 fix(files): review round 3, the server — a fetched locked re-read, a finish that follows its file, an eviction that counts again
- set_retention and delete_own re-read the folder under its lock and use that row (an unfetched ORM
  result never refreshes the object); a folder gone meanwhile is refused (409), and the route audits
  the retention actually replaced
- a finishing download is stamped only while it is still in the folder (and keep) it was read with:
  a move or a folder delete landing meanwhile sends the read round again, so no file is done with no
  expiry
- the eviction pass counts the disk before each candidate and skips one asked for since it began
- docs: source builds get their own volume-copy snippet
- race tests assert the lock wait; one built on a committing delete never raced
2026-10-13 03:15:59 +02:00
peter 5bb101e493 fix(files): review round 2, the page — the viewer's Escape, the dialog's drop, the capped list's honesty
- Escape closes the file viewer again: its layer sits above its dialog's, so it takes the Escape.
- The page's drop guard leaves a drag the open Upload dialog's zone accepted alone (it ran last and
  refused that drop).
- A capped list's server sort or search keeps the rows on screen until its answer is in (no skeleton
  under the pressed header), says "The first N items" when that answer is cut too, and an arrival is
  decided on the rows shown: one past the end of a cut list is said to be so, not gone.
- The search an arrival clears is taken at once.
- Tests: confirm names asserted per confirm, a fitting text pages under the arrows, a deleted file
  leaves the tray's finished list.
2026-10-13 02:32:50 +02:00
peter 54a6bbf07f fix(files): review round 2, the server — locks the GC and folders read under, the mount check, the old mount name
- GC: the expiry warning marks each file conditionally on the expiry it saw, one commit each (a file
  deleted meanwhile no longer fails the run; one re-timed keeps its fresh expiry unwarned); eviction
  counts the disk again when a candidate was freed meanwhile, so it never evicts for space already free.
- Folders: a retention change locks the folder and its files, and a finishing file reads the retention
  under a share lock, so neither misses the other; a folder delete locks the files it takes home (a
  file moved out meanwhile stays put, one deleted meanwhile no longer fails the delete).
- Upload: the folder is checked before the probe and held only once the job names it.
- Dedup enqueue locks the cached asset, so a release meanwhile yields a fresh one instead of a 500.
- Mount check reads the kernel's mount table instead of comparing devices (the vfs storage driver gave
  a mounted volume the device of /), and only inside a container; the API also logs a fresh, empty
  volume in the library's place.
- Compose files fall back to DOWNLOAD_HOST_PATH; the upgrade notes cover the from-source pull.
- Folder routes call owned_or_404 directly.
2026-10-13 02:32:49 +02:00
peter 8b2260c9a8 fix(files): review round 1, the page — honest states, the keyboard, the tray's words
- A library past the list's cap is searched and sorted on the server; the count says "the newest".
- Each folder and each search opens on its first page (DataTable pageScope, useCardPager scope).
- A file dropped where the page takes none never reaches the browser (which would unload the app).
- The viewer's arrows wait while a layer opened from it is on top, and scroll a wide text line;
  a text shows its first 5000 lines and says where the rest is.
- Confirms name the file (no more "your downloads"); the asking menu rows end in "…".
- The tray names a running step beside its bar, shows a pause mark for paused work, polls slowly
  while only paused work waits, and forgets a finished file that was deleted.
- An unchanged rename sends nothing; a paste opens the download dialog only with nothing on top;
  the link box never replaces what was typed.
- Honest states: folders that failed to load, a file no longer there, the storage figure; the rail
  says "nothing downloaded" only when there is nothing.
- Focus after deleting the open folder and when the format form opens; the format name's label.
- ui/Meter (one danger line for both quota bars) and ui/RenameField in the platform layer.
- Copy fixes, stale comments and the speculative hub key; a contract test for the list's
  parameters; e2e: the Files page and the paste intake.
2026-10-13 00:14:09 +02:00
peter 815083bcc4 fix(files): review round 1, the server — the tray's window, the GC's locks, a file's origin, the upgrade path
- The queue keeps a done job by when it became done (ready_at), not by its last write: renaming,
  keeping or moving an old file, or the GC's expiry warning, no longer brings it back as just finished.
- GC expiry and eviction re-read each file under the asset→job locks the user routes take, and commit
  one file at a time: a delete, a re-download or a keep made while the GC runs is honoured, never a
  second release that frees a file another user still holds.
- A file records the origin it was placed under (download_jobs.source_key, migration 0105): an edit
  whose parent expired still goes back to its folder, not to Uploads.
- A folder a file is being put in is held (FOR SHARE) until it lands; a delete of it waits and takes
  that file home too, instead of failing on the foreign key.
- The hash route reads at most two files at once (429 past that) and holds no connection meanwhile.
- 0104's backfill dates old files by their request date only: no notice prod wrote names its job.
- The old DOWNLOAD_ROOT still finds the files; a files root that is not a mounted volume stops the
  worker and is logged by the API; the volume-copy recipe looks the project-prefixed names up.
- Stale comments about the /downloads mount and the Downloads page.
2026-10-13 00:14:09 +02:00
peter 96509fe356 fix(files): a file is added when it is done, not when it was asked for
A download that failed for days and then went through sat days down the library, under its
request date, and read as missing. A job now records when it became done (`ready_at`), stamped
where the expiry already is (`folders.stamp_done`, `finalize_if_running`) and on an insert that is
already done; a check constraint holds every path to done to it. The library sorts and dates its
"Added" by it; the queue keeps the request order.

The migration dates a file from before by its "ready to watch" notice where the inbox still holds
one, else by its request date, where it already sat.
2026-10-12 22:19:37 +02:00
peter 132fb56428 fix(downloads): a site the downloader no longer understands says so, and offers a retry
yt-dlp asks for a bug report on an extraction failure it did not expect: the site answered in a way its extractor does not read (measured on a TikTok link: fails on every attempt under yt-dlp 2026.07.04, downloads under 2026.08.19). That failure read as "Something went wrong" with no retry; it is now its own category, site_changed, with a Retry in the failure window. A contract test holds every failure category to its copy in both languages.
2026-10-12 20:54:47 +02:00
peter f6bce23a94 fix(files): the Storage tab drops its user picker — every user already has a row and a menu
The list holds every user, files or not, so the picker above it only repeated a row's Edit quota. The empty text names what the list holds: users.
2026-10-12 20:06:30 +02:00
peter fb8fb980e5 feat(files): a file that is not media opens in a viewer — an image, a PDF, text as text
- GET /api/files/{id}/preview: an allow-list on the stored extension (downloads/preview.py) picks
  image / PDF / text; served inline under `CSP: sandbox; frame-ancestors 'self'`, the type from the
  table; HTML and SVG arrive as plain text; a text is its first 1 MiB (X-Preview-Truncated), sent
  as UTF-8 whatever it was written in (BOM, UTF-8, else cp1250); a NUL means binary: 415
- the list says which preview each file gets (asset.preview); FE/BE contract on kinds and header
- the viewer (shared dialog shell): ← / → through the list's such files in its order, Save, open in
  a tab, the file's menu; one stage height for every kind; a no-preview card offers the file
- the name, the card's picture and the menu's Preview open it; an image up to 8 MB is its own
  thumbnail
2026-10-12 15:28:31 +02:00
peter 6707550513 fix(files): a steady list header, every user on the Storage tab, the name gets the row's room
- The Files list starts at the scroller's top: the padding above it let the sticky header ride up
  12px on the first scroll, then stop.
- Users -> Storage lists every user with their quota, not only those already holding a file — a
  quota is set before anyone downloads.
- On a wide Files list only the "…" holds a place; Copy link, Share and Save come up over the date
  while the row is hovered or holds the keyboard (a card keeps them in its line). Keeps is narrower.
2026-10-11 20:33:48 +02:00
peter 078194731b feat(files): the Downloads page goes — the tray, a Storage tab and a Downloads settings tab take its place
- The Downloads module, its rail icon and badge are gone. A bookmark, the remembered page and an old
  history entry that name it land on Files (urlState.resolvePage); `g d` goes to Files too.
- Users page: a Storage tab — the disk in numbers, every user's use against their own limit with
  their file count, every user's downloads; aligned tables, one menu per row (edit / reset quota,
  open source, copy, the owner's quota). The admin storage rows carry quota + file count, the admin
  job list its owner.
- Settings: a Downloads tab — the download formats in one list (duplicate a built-in, edit or delete
  your own) and the site sign-ins, moved from Account.
- "View" on an added download, a clip or a converting upload opens the tray on that row (unfolded,
  back if closed; a finished one opens as a file; under a player it opens without taking focus).
- The download dialog opens on the format used last.
- DataTable: `pager={false}` for a short list, `cardRow` to give a lone "…" its own card line.
- Welcome and README speak of Files; a new Files screenshot.
2026-10-11 19:37:13 +02:00
peter 1c4f0042d2 feat(files): download from anywhere — a link box, a pasted link, the palette; the running downloads in a tray
- Files toolbar link box (paste button, Enter); a lone http(s) link pasted onto any page opens the one
  app-wide download dialog, which now names the link; the palette's "Download from a link…" focuses
  the box on its history entry.
- The download tray, bottom right on every page: what waits, runs, is paused or failed (reason +
  Retry, the queue's right-click / "…" menu), and what finished in this tab as "Done · Open" until
  closed; folds to a pill (per-account pref); closed, it stays shut — across a reload — until the
  work changes or something finishes. Back to top steps up over it, the page end keeps clear.
- A finish is announced once: "Downloaded: …", whether a phone plays it, Copy link + Open (toasts
  and notices may carry a second action).
- The inbox's ready notice opens the file in its folder: download_done carries job_id + folder_id.
- GET /api/downloads is the queue only (in flight, failed, finished in the last 10 minutes), with
  each row's source key.
2026-10-11 16:56:13 +02:00
peter d22c64f4dd feat(files): folders managed in place, a grid per folder, drop to upload, open on one file
- Own folders are made, renamed and deleted from the rail (right click, long press,
  Shift+F10) or the "…" beside an open folder's name; any folder's retention is a
  submenu (7/30/90 days, until deleted). Deleting confirms where its files go.
- List or grid per folder, sharing one sort; the grid is the one media card.
- A file dropped anywhere on the page opens Upload with it, bound for the folder
  on screen.
- A link (?item=) or the upload notice opens Files on the file: its page turned
  to, the keyboard on it, a ring round it.
- An in-place name field left by Enter or Escape hands the focus back to what
  replaced it (the row, the entry, the "…") instead of dropping it on <body>.
- The table's saved page size is per account.
2026-10-11 15:59:12 +02:00
peter 85aa917b38 feat(files): the Files page — the library in one aligned list, its folders in a rail
Files is the one library: every done file, the ones shared with me, or one
folder's, picked in a side rail (sources, own folders, the storage meter) and
mirrored to ?folder=. One table with every kind of value in its own column;
Copy link, Share and Save in sight, every action on a right click: play,
cut, rename in the row, move to an own folder or back home, keep, details,
re-download, delete.

The Downloads page keeps only the queue until the tray replaces it. The old
/api/downloads library routes are gone: list shared, upload, the details
PATCH, delete, the file and the usage now live under /api/files, and a
listed file says which source it came from. The job's "expired" flag is gone
with the state it described.
2026-10-11 05:34:25 +02:00
peter 1a1665b8ec feat(files): the Files library API — folders, moving and keeping files, upload into a folder, the file's hash
/api/files is the library's home: list a user's done files (by folder, searched, sorted, or the
ones shared with them), rename / move / keep / delete a file, make, rename and delete own folders
(a deleted folder's files go back to their source folders), and set how long any folder keeps a
file — done files shift by the difference, never closer than the GC's warning window.

An upload can name its folder (an own folder or Uploads), checked before the body is read and again
before the file lands; it is hashed while it streams, and any file can be hashed on first ask. The
download says the hash (Repr-Digest) once it is known; a new file at an asset's path forgets it.

Uploads, deletes, keeps, moves and folder changes are audited (ids only). The fileless "expired"
rows of the old retention are deleted by migration 0103: an expired file now leaves the library.

The shared route helpers moved to routes/files.py; the old /api/downloads library routes delegate
to the new ones until the client moves over.
2026-10-11 04:42:55 +02:00
peter bf99444ded feat(files): every file sits in a library folder, and keeps its own expiry
The Files library starts here. A user's every download, clip and upload now
lands in a source folder made per origin (YouTube, Plex clips, uploads, one per
other site), decided once in downloads/folders.py at every place a job is made.
How long a file stays moves from the shared asset to the user's job: stamped
from its folder when it is done, never for a file the user keeps. The GC warns
each user once, then takes the expired file out of their library; the cache cap
never evicts a kept file, and an evicted one leaves every holder's library.

The media mount is /files now (FILES_ROOT, FILES_HOST_PATH, volume
siftlode_files); docs/self-hosting.md says how to carry an old volume over.
Migration 0102 places every existing job and carries each done file's expiry
over from its asset.

safe_abs_path takes a missing path and answers None, which drops one baselined
type error.
2026-10-11 03:30:34 +02:00
peter 07e0990ecf design(files): the one-library round in the lab — aligned list columns, the tray, the done notice
The Files page replaces the Downloads page: list rows with one column per value kind and a
right-click menu, an app-wide download tray, a done notice that copies the link. Picked
variants are logged in DESIGN.md.
2026-10-11 02:59:51 +02:00
peter e2ed2bf874 merge: chore/e2e-retry-connect into E39 (rides the next release) 2026-10-11 02:30:16 +02:00
peter b9eb8458b3 test(e2e): a localhost connect the machine dropped is tried again on the test side
On a Windows host a burst of test traffic can drop a loopback connect now and then (every port,
any program — measured on the dev desktop). The browser resends it and only waits; Node (libuv)
sends a loopback SYN once and fails at ~300 ms with "connect ETIMEDOUT". `e2e/retryConnect.ts` wraps
Playwright's request context so that one error is tried again (0.5 / 1 / 2 / 4 / 8 s, the system's
own retry waits) while every other failure goes straight to the test; global.setup's health check,
login and /api/me and speed.measure's share probe go through it. Unit-tested; e2e 61/61.
2026-10-11 00:45:13 +02:00
304 changed files with 20705 additions and 4087 deletions
+5 -5
View File
@@ -73,13 +73,13 @@ SMTP_FROM=
# of them and false on the rest, to avoid double quota use and write races.
SCHEDULER_ENABLED=true
# --- Download center ---
# The Download Center adds a `worker` container (runs the yt-dlp/ffmpeg job loop) and a small
# --- Files library & downloads ---
# Downloading adds a `worker` container (runs the yt-dlp/ffmpeg job loop) and a small
# `bgutil-pot` sidecar (mints YouTube tokens) — both come up automatically with docker compose.
# Downloaded media defaults to a Docker-managed named volume. Set DOWNLOAD_HOST_PATH to a host
# The Files library defaults to a Docker-managed named volume. Set FILES_HOST_PATH to a host
# directory instead — e.g. one your Plex server can read — to keep the Plex-style tree there.
# The path must be writable by the container user (uid of `appuser`, 1000): chown 1000:1000 <dir>.
# DOWNLOAD_HOST_PATH=/mnt/media/youtube
# FILES_HOST_PATH=/mnt/media/youtube
# --- Plex integration (optional) ---
# The optional Plex module browses/plays your Plex library, playing the LOCAL physical file
@@ -89,6 +89,6 @@ SCHEDULER_ENABLED=true
# to /plex-media inside the container (the local side of the admin "path map", whose default is
# /data=/plex-media). Leave unset if you don't use Plex.
# PLEX_MEDIA_HOST_PATH=/mnt/media
# Optional: scratch dir for on-the-fly HLS remux segments. Defaults to DOWNLOAD_ROOT; point it at
# Optional: scratch dir for on-the-fly HLS remux segments. Defaults to FILES_ROOT; point it at
# fast local storage if your download volume is slow.
# PLEX_HLS_DIR=/var/tmp/plex-hls
+2 -2
View File
@@ -29,9 +29,9 @@ frontend/e2e/.auth/
*.sql.gz
backups/
# Download center: locally downloaded media (the DOWNLOAD_ROOT bind mount at the repo root).
# Download center: locally downloaded media (the FILES_ROOT bind mount at the repo root).
# Anchored with a leading slash so it does NOT also ignore the backend/app/downloads package.
/downloads/
/files/
# OS / editor
.DS_Store
+46
View File
@@ -7,6 +7,52 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.107.2 — 2026-10-17
### New
- Development: the gate script can list its checks (scripts/check.sh --list) and run just one (--only), and the backend tests run on every CPU core, each worker with its own database.
## 0.107.1 — 2026-10-16
### Fixed
- Downloads: a failed download's explanation now says to press Retry, instead of pointing to a download list that no longer exists.
## 0.107.0 — 2026-10-15
### New
- API tokens: Settings › API tokens gives one machine of yours a key to the Files folders you pick — or, in one click, to a Hub folder for swapping files with it. Signing out your other sessions or changing your password revokes every token.
- Share by link: a link can now open a whole folder of yours — its page lists what is in it and plays a video above the list.
### Fixed
- Uploads: a finished upload is announced once, and shows in Files at once.
## 0.106.2 — 2026-10-14
### Fixed
- Pictures: a thumbnail, poster or channel picture that fails to load is retried, and meanwhile the card shows its empty frame instead of a broken-image mark.
## 0.106.1 — 2026-10-14
### Fixed
- Plex player: clicking outside the Settings panel closes the panel again instead of closing the player.
## 0.106.0 — 2026-10-13
### New
- Files: one library for everything you keep — downloads, clips and your own uploads of any file — in a folder per site plus folders of your own, each keeping its files as long as you choose. Documents, pictures and text open in a viewer.
- Downloads: paste a link anywhere on a page to download it, and the tray shows what is running and what is done. The Downloads page is gone: your files are in Files, and your download settings are under Settings › Downloads.
### Fixed
- Self-hosting: the library is now mounted at /files (FILES_HOST_PATH). Old settings keep working; if you used the default volume, copy it over once — see the upgrade notes.
## 0.105.0 — 2026-10-10
### New
+3 -3
View File
@@ -5,7 +5,7 @@ be developed that way. This file is the rules of the road: read it before you to
loaded into every agent session, so keep it **short and true**. A rule here that has gone stale is
worse than no rule — if you change how the code works, change this file in the same commit.
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a download center). It is a
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a files library with downloads). It is a
**modular monolith**: one Postgres, one FastAPI process (uvicorn + APScheduler), one worker process
(yt-dlp/ffmpeg), one PO-token sidecar. React 19 (with the React Compiler) + react-query frontend, with a
router-free navigation provider.
@@ -62,7 +62,7 @@ how it gets built. Concretely:
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell (a shared `/watch` link boots from its own
@@ -114,7 +114,7 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
Not repeated here, because a lint rule refuses the wrong one and its message names the right one:
buttons (`ui/Button`, `IconButton`, `Pressable`, `Chip`, `DragHandle`), text fields and native
controls (`ui/Input`, `Checkbox`, `Slider`, …), glass surfaces (`SURFACE.*`), a native `<select>`
(`SegmentedToggle` / `FilterSelect`), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
(`SegmentedToggle` / `FilterSelect`), a picture (`ui/RemoteImage`, never a bare `<img>`), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
## House rules that bite (learned from real bugs)
+10 -4
View File
@@ -39,13 +39,13 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
WORKDIR /app
# Create the app user BEFORE the copies so files land already-owned via COPY --chown (a trailing
# `chown -R /app` would rewrite every file into a duplicate layer). /downloads must be appuser-owned
# `chown -R /app` would rewrite every file into a duplicate layer). /files must be appuser-owned
# so a fresh named volume mounted there inherits that ownership (Docker copies the image dir's
# ownership into a new empty volume) — the worker/API can then write without a manual chown. A bind
# mount to a host path instead needs that path writable by this uid (see docs/self-hosting.md).
RUN adduser --disabled-password --gecos "" appuser \
&& mkdir -p /downloads \
&& chown appuser /app /downloads
&& mkdir -p /files \
&& chown appuser /app /files
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode.
# intel-media-va-driver (iHD): the VA driver ffmpeg's h264_vaapi needs — libva2 ships in the base but
@@ -55,8 +55,14 @@ RUN adduser --disabled-password --gecos "" appuser \
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# `apt-get upgrade`: the base image is a snapshot, and the Debian security fixes released since it
# was cut are only in the archive — the first Trivy scan found one (pcre2) fixed upstream but not in it.
# APT_REFRESH: the upgrade only runs when this layer is rebuilt, and a build cache keeps it for as long
# as nothing above it changes — a publish shipped an image whose curl and librsvg were a release behind
# fixes already in the archive. The release build passes the day here, so the layer is rebuilt at
# least daily; empty (the default, a self-hosted build) leaves the cache alone.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
ARG APT_REFRESH=
RUN echo "apt refresh: ${APT_REFRESH:-cached}" \
&& apt-get update \
&& apt-get upgrade -y --no-install-recommends \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
+4 -3
View File
@@ -22,10 +22,11 @@ blocker and SponsorBlock keep working.
tags to slice the feed by.
- **Playlists with two-way YouTube sync** — build them locally, keep them in sync in both directions.
- **In-app player** with resume, plus keyboard/scroll controls.
- **Download Center** — save videos to the server with yt-dlp in a Plex-friendly layout (format
presets, per-user storage quota), trim / crop / split & join them in a built-in editor, play them
- **Files** — one library for everything you keep: paste a link anywhere in the app to download it
with yt-dlp (format presets, per-user storage quota, the running downloads in a tray), upload your
own files, sort them into folders, trim / crop / split & join videos in a built-in editor, play them
where you left off, then save to your device, share with another user, or hand out a public watch
link.
link that plays on any phone.
- **Plex library** *(optional)* — point Siftlode at your own Plex server to browse and play its
movies and shows alongside your feed, with watch state kept in sync.
- **Multi-user** with per-user private state, a shared catalog, and a fair daily API-quota guard.
+1 -1
View File
@@ -1 +1 @@
0.105.0
0.107.2
-50
View File
@@ -727,30 +727,6 @@
}
],
"./app/routes/downloads.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 29,
"endColumn": 33,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 29,
"endColumn": 33,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 40,
"endColumn": 50,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
@@ -790,22 +766,6 @@
"endColumn": 54,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 29,
"endColumn": 33,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 40,
"endColumn": 50,
"lineCount": 1
}
}
],
"./app/routes/playlists.py": [
@@ -1196,16 +1156,6 @@
}
}
],
"./app/routes/public.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 57,
"endColumn": 71,
"lineCount": 1
}
}
],
"./app/routes/scheduler.py": [
{
"code": "reportArgumentType",
@@ -6,7 +6,7 @@ Create Date: 2026-07-07
Some sources (a direct media URL like a reddit v.redd.it HLS manifest) carry no thumbnail, so the
download shows a blank image box. The worker now extracts a representative frame with ffmpeg and
saves it as a poster sidecar; `poster_path` (relative to download_root) records it so the card /
saves it as a poster sidecar; `poster_path` (relative to files_root) records it so the card /
watch page / link preview can fall back to it. Nullable — only set when there was no thumbnail.
"""
@@ -0,0 +1,193 @@
"""The Files library: folders, and a file's retention on the user's job
Revision ID: 0102_file_folders
Revises: 0101_plex_text_plain
Create Date: 2026-10-11
Every download, clip and upload of a user now sits in a folder of their Files library
(`app.downloads.folders`): a source folder made per origin (YouTube, Plex, uploads, one per other
site) or one of their own. How long a file stays moves from the shared asset to each user's job
(`download_jobs.expires_at`, stamped from the folder's `retention_days`), so one user keeping a file
no longer depends on another's copy; the asset's own `expires_at` / `gc_notified` go.
This places every existing job in its source folder and carries each done job's expiry over from
the asset it holds (an upload keeps none). The source-key rule is copied here, frozen, from
`folders.source_key_for` as it is today — a migration must not follow the code's later changes.
"""
from urllib.parse import urlsplit
import sqlalchemy as sa
from alembic import op
revision: str = "0102_file_folders"
down_revision: str | None = "0101_plex_text_plain"
branch_labels: str | None = None
depends_on: str | None = None
_FRONT_DOORS = ("www.", "m.", "mobile.", "web.")
_YOUTUBE_HOSTS = {"youtube.com", "youtu.be", "youtube-nocookie.com", "music.youtube.com"}
_DEFAULT_RETENTION_DAYS = 30 # config.download_retention_days on the day this was written
def _site_key(url: str) -> str:
host = (urlsplit(url).hostname or "").lower().rstrip(".")
for door in _FRONT_DOORS:
if host.startswith(door) and host.count(".") > 1:
host = host[len(door) :]
break
if host in _YOUTUBE_HOSTS:
return "youtube"
return f"site:{host or 'unknown'}"
def _asset_key(kind: str | None, ref: str | None) -> str:
if kind == "youtube":
return "youtube"
if kind == "url" and ref:
return _site_key(ref)
return "upload"
def _key(job, jobs_by_id, assets) -> str:
kind = job.source_kind
if kind == "youtube":
return "youtube"
if kind == "upload":
return "upload"
if kind == "clip":
spec = job.edit_spec or {}
return "plex" if spec.get("source") == "plex" else "youtube"
if kind == "url":
return _site_key(job.source_ref)
if kind == "edit":
parent = jobs_by_id.get(job.source_job_id)
if parent is not None and parent.id != job.id:
return _key(parent, jobs_by_id, assets)
asset = assets.get(job.source_asset_id)
if asset is not None:
return _asset_key(asset.source_kind, asset.source_ref)
return "upload"
def upgrade() -> None:
op.create_table(
"file_folders",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("kind", sa.String(16), nullable=False),
sa.Column("source_key", sa.String(260)), # "site:" + a host (<= 253)
sa.Column("name", sa.String(255)),
sa.Column("retention_days", sa.Integer()),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
sa.CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
op.create_index("ix_file_folders_user_id", "file_folders", ["user_id"])
op.create_index(
"uq_file_folder_own_name",
"file_folders",
["user_id", sa.text("lower(name)")],
unique=True,
postgresql_where=sa.text("name IS NOT NULL"),
)
op.add_column(
"download_jobs",
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="RESTRICT"),
),
)
op.add_column(
"download_jobs",
sa.Column("kept", sa.Boolean(), server_default="false", nullable=False),
)
op.add_column("download_jobs", sa.Column("expires_at", sa.DateTime(timezone=True)))
op.add_column(
"download_jobs",
sa.Column("expiry_warned", sa.Boolean(), server_default="false", nullable=False),
)
op.create_index("ix_download_jobs_folder_id", "download_jobs", ["folder_id"])
op.create_index("ix_download_jobs_expires_at", "download_jobs", ["expires_at"])
conn = op.get_bind()
raw = conn.execute(
sa.text("SELECT value FROM system_config WHERE key = 'download_retention_days'")
).scalar()
retention = int(raw) if raw not in (None, "") else _DEFAULT_RETENTION_DAYS
jobs = conn.execute(
sa.text(
"SELECT j.id, j.user_id, j.source_kind, j.source_ref, j.edit_spec, j.source_job_id,"
" j.source_asset_id, j.status, j.asset_id, a.expires_at AS asset_expires_at"
" FROM download_jobs j LEFT JOIN media_assets a ON a.id = j.asset_id"
)
).fetchall()
jobs_by_id = {j.id: j for j in jobs}
assets = {
a.id: a
for a in conn.execute(sa.text("SELECT id, source_kind, source_ref FROM media_assets"))
}
folder_ids: dict[tuple[int, str], int] = {}
for job in jobs:
key = _key(job, jobs_by_id, assets)
slot = (job.user_id, key)
if slot not in folder_ids:
folder_ids[slot] = conn.execute(
sa.text(
"INSERT INTO file_folders (user_id, kind, source_key, retention_days)"
" VALUES (:u, 'source', :k, :r) RETURNING id"
),
{"u": job.user_id, "k": key, "r": None if key == "upload" else retention},
).scalar_one()
# A done file keeps the expiry it had (its asset's); an upload had none and keeps none.
expires = job.asset_expires_at if job.status == "done" and key != "upload" else None
conn.execute(
sa.text("UPDATE download_jobs SET folder_id = :f, expires_at = :e WHERE id = :id"),
{"f": folder_ids[slot], "e": expires, "id": job.id},
)
op.drop_column("media_assets", "expires_at")
op.drop_column("media_assets", "gc_notified")
def downgrade() -> None:
op.add_column("media_assets", sa.Column("expires_at", sa.DateTime(timezone=True)))
op.add_column(
"media_assets",
sa.Column("gc_notified", sa.Boolean(), server_default="false", nullable=False),
)
# The asset goes when its last holder would: the latest of its holders' expiries (a holder that
# never expires keeps the asset unexpired).
op.execute(
"UPDATE media_assets a SET expires_at = sub.e FROM ("
" SELECT asset_id, CASE WHEN bool_or(expires_at IS NULL) THEN NULL ELSE max(expires_at) END"
" AS e FROM download_jobs WHERE asset_id IS NOT NULL AND status = 'done' GROUP BY asset_id"
") sub WHERE a.id = sub.asset_id AND a.source_kind <> 'upload'"
)
op.drop_index("ix_download_jobs_expires_at", table_name="download_jobs")
op.drop_index("ix_download_jobs_folder_id", table_name="download_jobs")
op.drop_column("download_jobs", "expiry_warned")
op.drop_column("download_jobs", "expires_at")
op.drop_column("download_jobs", "kept")
op.drop_column("download_jobs", "folder_id")
op.drop_index("uq_file_folder_own_name", table_name="file_folders")
op.drop_index("ix_file_folders_user_id", table_name="file_folders")
op.drop_table("file_folders")
@@ -0,0 +1,32 @@
"""The Files library: a file's SHA-256, and the expired rows retired
Revision ID: 0103_files_sha256
Revises: 0102_file_folders
Create Date: 2026-10-11
`media_assets.sha256` holds the file's hash once it is known (an upload is hashed while it streams
in, any other file on the first ask), so a machine fetching a file can check what it got.
An expired file now leaves its user's library (`downloads.gc`). Before, an expired download stayed
as a fileless "expired · download again" row (a done job with no asset). Those rows hold nothing on
disk and the library no longer shows them, so they are deleted here rather than left behind
unseen; their shares and public links go with them (ON DELETE CASCADE), and those already served
nothing. The downgrade cannot bring them back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0103_files_sha256"
down_revision: str | None = "0102_file_folders"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("media_assets", sa.Column("sha256", sa.String(64)))
op.execute("DELETE FROM download_jobs WHERE status = 'done' AND asset_id IS NULL")
def downgrade() -> None:
op.drop_column("media_assets", "sha256")
@@ -0,0 +1,38 @@
"""A file's "Added" is when it became done
Revision ID: 0104_job_ready_at
Revises: 0103_files_sha256
Create Date: 2026-10-12
`download_jobs.ready_at` is when a job became done, and the Files library sorts and dates by it: a
download that failed for days and then went through used to sit days down the list, under its
request date. Every path to done stamps it, held by `ck_download_jobs_ready_at`.
A done job from before has no record of that moment (its "ready to watch" notice never named the
job, so it cannot be matched back), so it takes its request date, which is where it already sat.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0104_job_ready_at"
down_revision: str | None = "0103_files_sha256"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("ready_at", sa.DateTime(timezone=True)))
op.execute(
"""
UPDATE download_jobs SET ready_at = created_at WHERE status = 'done'
"""
)
op.create_check_constraint(
"ck_download_jobs_ready_at", "download_jobs", "status <> 'done' OR ready_at IS NOT NULL"
)
def downgrade() -> None:
op.drop_constraint("ck_download_jobs_ready_at", "download_jobs", type_="check")
op.drop_column("download_jobs", "ready_at")
@@ -0,0 +1,37 @@
"""A file remembers where it came from
Revision ID: 0105_job_source_key
Revises: 0104_job_ready_at
Create Date: 2026-10-12
`download_jobs.source_key` is the origin a file came from — the source folder it may go back to —
recorded when it is placed. An edit used to work its origin out through its parent each time, and
once the parent was deleted or expired it fell back to "upload": moved into an own folder, it could
no longer go back to its folder, and deleting that folder sent it to Uploads to stay forever.
A file in a source folder takes that folder's key. One already in an own folder is left NULL and
keeps being worked out (`folders.source_key_for`), as before.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0105_job_source_key"
down_revision: str | None = "0104_job_ready_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("source_key", sa.String(260)))
op.execute(
"""
UPDATE download_jobs j SET source_key = f.source_key
FROM file_folders f
WHERE f.id = j.folder_id AND f.kind = 'source'
"""
)
def downgrade() -> None:
op.drop_column("download_jobs", "source_key")
+110
View File
@@ -0,0 +1,110 @@
"""Machine access to the Files library: API tokens bound to folders, and the Hub folder
Revision ID: 0106_api_tokens
Revises: 0105_job_source_key
Create Date: 2026-10-14
`api_tokens` is a user's token for a machine (a script, the hub): only its SHA-256 is kept, plus the
first characters to recognise it by, and whether its user was told it is about to run out. `api_token_grants` says what it may do in which folder — list,
read, write, delete — so a token reaches nothing it was not given, folder by folder.
A third folder kind, `hub`: the one folder a user exchanges files with their hub machine through.
Like a source folder its name is the client's to render; like an own folder a file may be moved in.
One per user.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0106_api_tokens"
down_revision: str | None = "0105_job_source_key"
branch_labels: str | None = None
depends_on: str | None = None
_KIND_OLD = (
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
)
_KIND_NEW = _KIND_OLD + " OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)"
def upgrade() -> None:
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_NEW)
op.create_index(
"uq_file_folder_hub",
"file_folders",
["user_id"],
unique=True,
postgresql_where=sa.text("kind = 'hub'"),
)
op.create_table(
"api_tokens",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("name", sa.String(80), nullable=False),
sa.Column("prefix", sa.String(16), nullable=False),
sa.Column("token_hash", sa.String(64), nullable=False, unique=True),
sa.Column("expires_at", sa.DateTime(timezone=True)),
sa.Column("last_used_at", sa.DateTime(timezone=True)),
sa.Column("expiry_warned", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_table(
"api_token_grants",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"token_id",
sa.Integer(),
sa.ForeignKey("api_tokens.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("can_list", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_read", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_write", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_delete", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
sa.CheckConstraint(
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
),
)
def downgrade() -> None:
op.drop_table("api_token_grants")
op.drop_table("api_tokens")
op.drop_index("uq_file_folder_hub", table_name="file_folders")
# A Hub folder may hold files (their foreign key RESTRICTs a delete): it becomes an own folder,
# named "Hub <id>" — or "Hub <id> (2)", … when its user already has a folder of that name,
# which `uq_file_folder_own_name` (per user, case-blind) would refuse.
conn = op.get_bind()
hubs = conn.execute(sa.text("SELECT id, user_id FROM file_folders WHERE kind = 'hub'")).all()
for folder_id, user_id in hubs:
name, n = f"Hub {folder_id}", 1
while conn.execute(
sa.text("SELECT 1 FROM file_folders WHERE user_id = :u AND lower(name) = lower(:n)"),
{"u": user_id, "n": name},
).first():
n += 1
name = f"Hub {folder_id} ({n})"
conn.execute(
sa.text("UPDATE file_folders SET kind = 'own', name = :n WHERE id = :i"),
{"n": name, "i": folder_id},
)
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_OLD)
@@ -0,0 +1,66 @@
"""Share-link tokens no longer stored in the clear
Revision ID: 0107_link_token_at_rest
Revises: 0106_api_tokens
Create Date: 2026-10-15
A `/watch` link's token is the credential itself, and it sat in `download_links.token` as written,
so a copy of the database was every shared file. Now a link is found by the token's SHA-256
(`token_hash`), and the token is kept only Fernet-encrypted (`token_enc`, TOKEN_ENCRYPTION_KEY) so its
owner can copy the same link again. Without a key the copy is not kept: the link still opens, its
address just cannot be shown again (`app/downloads/links.py`).
Every existing link is converted here, so the addresses already sent keep working. The downgrade
writes the tokens back from their encrypted copies; a link with none left (no key, or a key since
changed) gets a fresh random token, which is the same as revoking it.
"""
import hashlib
import secrets
import sqlalchemy as sa
from alembic import op
from app import security
revision: str = "0107_link_token_at_rest"
down_revision: str | None = "0106_api_tokens"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_links", sa.Column("token_hash", sa.String(64)))
op.add_column("download_links", sa.Column("token_enc", sa.Text()))
conn = op.get_bind()
sealing = security.can_encrypt()
for link_id, token in conn.execute(sa.text("SELECT id, token FROM download_links")).fetchall():
conn.execute(
sa.text("UPDATE download_links SET token_hash = :h, token_enc = :e WHERE id = :id"),
{
"id": link_id,
"h": hashlib.sha256(token.encode()).hexdigest(),
"e": security.encrypt(token) if sealing else None,
},
)
op.alter_column("download_links", "token_hash", nullable=False)
op.create_unique_constraint("uq_download_link_token_hash", "download_links", ["token_hash"])
op.drop_constraint("uq_download_link_token", "download_links", type_="unique")
op.drop_column("download_links", "token")
def downgrade() -> None:
op.add_column("download_links", sa.Column("token", sa.String(64)))
conn = op.get_bind()
rows = conn.execute(sa.text("SELECT id, token_enc FROM download_links")).fetchall()
for link_id, enc in rows:
token = security.decrypt_optional(enc) or secrets.token_urlsafe(24)
conn.execute(
sa.text("UPDATE download_links SET token = :t WHERE id = :id"),
{"id": link_id, "t": token},
)
op.alter_column("download_links", "token", nullable=False)
op.create_unique_constraint("uq_download_link_token", "download_links", ["token"])
op.drop_constraint("uq_download_link_token_hash", "download_links", type_="unique")
op.drop_column("download_links", "token_enc")
op.drop_column("download_links", "token_hash")
@@ -0,0 +1,40 @@
"""A share link may open a folder
Revision ID: 0108_folder_links
Revises: 0107_link_token_at_rest
Create Date: 2026-10-15
A `/watch` link opened one file. Now it may open one of its owner's own folders instead: the link's
page lists what is in the folder when it is opened. `folder_id` sits beside `job_id`, exactly one of
the two set; a folder's links go with the folder.
The downgrade has nowhere to keep a folder link, so it deletes them: they stop opening.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0108_folder_links"
down_revision: str | None = "0107_link_token_at_rest"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"download_links",
sa.Column("folder_id", sa.Integer(), sa.ForeignKey("file_folders.id", ondelete="CASCADE")),
)
op.create_index("ix_download_links_folder_id", "download_links", ["folder_id"])
op.alter_column("download_links", "job_id", nullable=True)
op.create_check_constraint(
"ck_download_link_target", "download_links", "(job_id IS NULL) <> (folder_id IS NULL)"
)
def downgrade() -> None:
op.execute("DELETE FROM download_links WHERE folder_id IS NOT NULL")
op.drop_constraint("ck_download_link_target", "download_links", type_="check")
op.alter_column("download_links", "job_id", nullable=False)
op.drop_index("ix_download_links_folder_id", table_name="download_links")
op.drop_column("download_links", "folder_id")
+2 -1
View File
@@ -20,7 +20,8 @@ Three layers, by *who* runs the code — not by feature:
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline;
`issues/bundle.py` is the one definition of a bug report's files, for the zip export and the tracker.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth` (with `apitokens`, a machine's
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`,
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
+273
View File
@@ -0,0 +1,273 @@
"""Machine access to the Files library: API tokens, and what each one may do where.
A user makes a token for a machine of theirs — their hub, a script — and it reaches their library as
`Authorization: Bearer sfl_…`. Only on `/api/files`: the routes there take `files_caller` instead of
`current_user`, and nothing else does (a test holds the route table to that), so a token can never
manage the account, the tokens, or anything outside the library. And only in the folders its grants
name, with the rights they give: list, read, write, delete. Every route asks the one question,
`FilesCaller.may(folder_id, right)`; a signed-in user may everything in their own folders.
The token is `sfl_` + 32 random bytes in base62 + a CRC32 of them: a mistyped or cut-off token is
refused without a database read, and a secret scanner can tell one by its shape. Only its SHA-256 is
kept (`security.hash_token`) — the user sees it once, when it is made — with its first characters to
tell tokens apart by. It is looked up by that hash on every request: no signed claims, so a deleted
token stops working at once. A token about to run out warns its user once, a week before
(`warn_expiring`), so a machine does not lose its key unannounced.
"""
import secrets
import zlib
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from fastapi import Depends, Request
from sqlalchemy import delete, or_, select, update
from sqlalchemy.orm import Session
from app import audit, debugsession, logs
from app.audit import AuditAction
from app.auth import current_user
from app.db import get_db
from app.models import ApiToken, ApiTokenGrant, User
from app.notifications import create_notification
from app.ratelimit import RateLimiter, client_key
from app.refusal import refused
from app.security import hash_token
from app.utils import iso
log = logs.get_logger(__name__)
PREFIX = "sfl_"
_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
_BODY_BYTES = 32
_BODY_LEN = 43 # 62**43 > 2**256: every 32-byte value fits
_CHECK_LEN = 6 # 62**6 > 2**32: every CRC32 fits
TOKEN_LEN = len(PREFIX) + _BODY_LEN + _CHECK_LEN
# What a token is shown by once it is made: enough to tell a user's tokens apart, far too little to
# narrow a guess (8 of its 43 random characters).
SHOWN_LEN = len(PREFIX) + 8
# A machine that runs away (a loop in a script) is held back per token; a caller trying tokens is
# held back per client, counting only the refusals.
_calls = RateLimiter(600, 60)
_bad = RateLimiter(20, 300)
# How stale `last_used_at` may get before a request writes it again: a busy script costs one write a
# minute, not one per request.
_TOUCH_EVERY = timedelta(minutes=1)
# How long before a token runs out its user is told (`warn_expiring`); the token list marks it
# "expires soon" for the same span.
EXPIRY_WARNING = timedelta(days=7)
class Right(StrEnum):
LIST = "list"
READ = "read"
WRITE = "write"
DELETE = "delete"
# Each right's column on a grant. A dict over the enum, so a new right without a column fails here.
GRANT_COLUMNS = {
Right.LIST: "can_list",
Right.READ: "can_read",
Right.WRITE: "can_write",
Right.DELETE: "can_delete",
}
def _b62(value: int, width: int) -> str:
out = []
for _ in range(width):
value, digit = divmod(value, 62)
out.append(_ALPHABET[digit])
return "".join(reversed(out))
def _check(body: str) -> str:
return _b62(zlib.crc32(body.encode()), _CHECK_LEN)
def mint() -> str:
"""A new token, in full. The caller stores its hash and shows it to the user, once."""
body = _b62(int.from_bytes(secrets.token_bytes(_BODY_BYTES)), _BODY_LEN)
return PREFIX + body + _check(body)
def well_formed(raw: str) -> bool:
"""Whether `raw` has a token's shape and its checksum adds up."""
if len(raw) != TOKEN_LEN or not raw.startswith(PREFIX):
return False
body, check = raw[len(PREFIX) : -_CHECK_LEN], raw[-_CHECK_LEN:]
# The check too: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it.
if any(c not in _ALPHABET for c in body + check):
return False
return secrets.compare_digest(_check(body), check)
def grant_rights(grant: ApiTokenGrant) -> frozenset[Right]:
return frozenset(r for r, col in GRANT_COLUMNS.items() if getattr(grant, col))
@dataclass(frozen=True)
class FilesCaller:
"""Who is asking the Files library, and where they may do what. `grants` is None for a signed-in
user (everything in their own folders); for a token, folder id → its rights there."""
user: User
token_id: int | None = None
token_prefix: str | None = None
grants: Mapping[int, frozenset[Right]] | None = None
def may(self, folder_id: int | None, right: Right) -> bool:
if self.grants is None:
return True
return folder_id is not None and right in self.grants.get(folder_id, frozenset())
def folders_with(self, right: Right) -> set[int] | None:
"""The folders `right` is granted in; None = any of the user's own."""
if self.grants is None:
return None
return {f for f, rights in self.grants.items() if right in rights}
@property
def via(self) -> str | None:
"""How an audit row says a token did it, None when the user did."""
return None if self.token_prefix is None else f"api token {self.token_prefix}…"
def bearer_token(request: Request) -> str | None:
"""The token of an `Authorization: Bearer` header; None when the request carries none (another
scheme counts as none — it is not ours to judge)."""
scheme, _, value = request.headers.get("authorization", "").partition(" ")
if scheme.lower() != "bearer":
return None
return value.strip()
def session_caller(request: Request, db: Session = Depends(get_db)) -> User | None:
"""The signed-in user, or None when the request brings a token instead: a token wins over a
cookie, so a machine's request is judged by its grants alone, whatever else it carries."""
if bearer_token(request) is not None:
return None
return current_user(request, db)
def files_caller(
request: Request,
db: Session = Depends(get_db),
user: User | None = Depends(session_caller),
) -> FilesCaller:
"""The `/api/files` caller: a signed-in user, or a valid token's user with its grants."""
if user is not None:
return FilesCaller(user=user)
return _token_caller(request, db, bearer_token(request) or "")
def _refuse_token(key: str, reason: str) -> Exception:
_bad.allow(key)
log.info(
"a request brought an API token that does not hold",
extra=logs.event("apitoken.refused", reason=reason),
)
return refused(401, "not_authenticated")
def _token_caller(request: Request, db: Session, raw: str) -> FilesCaller:
key = client_key(request)
if _bad.exhausted(key):
raise refused(429, "api_token_rate_limited")
if not well_formed(raw):
raise _refuse_token(key, "malformed")
token = db.execute(
select(ApiToken).where(ApiToken.token_hash == hash_token(raw))
).scalar_one_or_none()
if token is None:
raise _refuse_token(key, "unknown")
now = datetime.now(timezone.utc)
if token.expires_at is not None and token.expires_at <= now:
raise _refuse_token(key, "expired")
user = db.get(User, token.user_id)
if user is None or not user.is_active or user.is_suspended:
raise _refuse_token(key, "account")
if not _calls.allow(str(token.id)):
raise refused(429, "api_token_rate_limited")
grants = {
g.folder_id: grant_rights(g)
for g in db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == token.id))
.scalars()
.all()
}
caller = FilesCaller(user=user, token_id=token.id, token_prefix=token.prefix, grants=grants)
db.execute(
update(ApiToken)
.where(
ApiToken.id == token.id,
or_(ApiToken.last_used_at.is_(None), ApiToken.last_used_at < now - _TOUCH_EVERY),
)
.values(last_used_at=now)
.execution_options(synchronize_session=False)
)
# Committed here, ahead of the route: a read-only route never commits, and a write route must
# not carry this along into a rollback of its own.
db.commit()
logs.set_user(user.id, *debugsession.level_for(user, None))
request.state.user_id = user.id
return caller
def revoke_all(db: Session, user_id: int) -> int:
"""Revoke every token of `user_id`, each with its audit row (NOT committed). A token is a
credential like a session, so whatever ends every session of an account — a password reset or
change, "log out everywhere", a revoked untrusted password — ends its tokens too: one minted
from a stolen session must not outlive the owner's answer to that theft. Returns how many."""
gone = (
db.execute(delete(ApiToken).where(ApiToken.user_id == user_id).returning(ApiToken.id))
.scalars()
.all()
)
for token_id in gone:
audit.record(
db,
user_id,
AuditAction.API_TOKEN_REVOKE,
target_type="api_token",
target_id=token_id,
summary="revoked with every session of the account",
)
return len(gone)
def expiring(token: ApiToken, now: datetime) -> bool:
"""Whether `token` runs out within the warning span (and has not yet)."""
return token.expires_at is not None and now < token.expires_at <= now + EXPIRY_WARNING
def warn_expiring(db: Session) -> int:
"""Tell each user whose token runs out within `EXPIRY_WARNING` — once per token: the mark and the
notice commit together, and a token made again (a new row) warns afresh. Returns how many."""
now = datetime.now(timezone.utc)
hits = db.execute(
update(ApiToken)
.where(
ApiToken.expiry_warned.is_(False),
ApiToken.expires_at.is_not(None),
ApiToken.expires_at > now,
ApiToken.expires_at <= now + EXPIRY_WARNING,
)
.values(expiry_warned=True)
.returning(ApiToken.id, ApiToken.user_id, ApiToken.name, ApiToken.expires_at)
.execution_options(synchronize_session=False)
).all()
for token_id, user_id, name, expires_at in hits:
create_notification(
db,
user_id,
"api_token_expiring",
title=f"API token {name} expires soon", # English fallback; the client names it from data
data={"token_id": token_id, "name": name, "expires_at": iso(expires_at)},
commit=False,
)
db.commit()
return len(hits)
+21
View File
@@ -58,6 +58,27 @@ class AuditAction:
# A user sent a debug session's issue to the tracker — the one act that takes their log
# off this instance, so it is recorded (target = the session, summary = the issue's URL).
ISSUE_CREATE = "issue_create"
# The Files library (`routes.files`): what a user put in, took out, kept or reorganised. Ids
# only — a file's or folder's name is the user's own text, not the admin's to read here.
FILE_UPLOAD = "file_upload"
FILE_DELETE = "file_delete"
FILE_KEEP = "file_keep"
FILE_UNKEEP = "file_unkeep"
FILE_MOVE = "file_move"
FILE_FOLDER_CREATE = "file_folder_create"
FILE_FOLDER_RENAME = "file_folder_rename"
FILE_FOLDER_RETENTION = "file_folder_retention"
FILE_FOLDER_DELETE = "file_folder_delete"
# A machine's token to the library (`routes.tokens`): made (its folders and rights) or revoked.
# What a token then did is the file rows above, their summary naming it.
API_TOKEN_CREATE = "api_token_create"
API_TOKEN_REVOKE = "api_token_revoke"
# A public `/watch` link to a file or a folder (`routes.downloads`): made, its settings changed,
# revoked.
# Never its token or password — only whether it has one.
DOWNLOAD_LINK_CREATE = "download_link_create"
DOWNLOAD_LINK_UPDATE = "download_link_update"
DOWNLOAD_LINK_REVOKE = "download_link_revoke"
def record(
+2 -2
View File
@@ -92,8 +92,8 @@ ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# So this 413 is a refusal a real user meets on a default install, not a theoretical edge, and
# it has to be as readable as the route's own: the client localizes `too_large` (`errors.tooLarge`)
# exactly as it does `upload_too_large`, and the upload dialog refuses an over-cap file before
# sending it at all, off the `upload_max_bytes` that `GET /api/downloads/usage` reports.
("POST", "/api/downloads/upload"): settings.download_upload_max_bytes,
# sending it at all, off the `upload_max_bytes` that `GET /api/files/usage` reports.
("POST", "/api/files/upload"): settings.download_upload_max_bytes,
("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
}
+9 -5
View File
@@ -1,7 +1,7 @@
from pathlib import Path
from typing import Literal, get_args, get_origin
from pydantic import Field, ValidationInfo, field_validator, model_validator
from pydantic import AliasChoices, Field, ValidationInfo, field_validator, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
# Shipped placeholder; production must override it. Used by the startup guard below.
@@ -171,9 +171,13 @@ class Settings(BaseSettings):
feed_default_hidden_live: str = "live,upcoming"
# --- Download center (yt-dlp) ---
# Root directory (inside the container) for downloaded media — a Docker volume/bind mount.
# Root directory (inside the container) for the files library — a Docker volume/bind mount.
# Env-only (a mount path can't live in the DB). The worker lays a Plex-style tree under it.
download_root: str = "/downloads"
# DOWNLOAD_ROOT is its old name: an install still on the old compose file (mounted at
# /downloads, DOWNLOAD_ROOT=/downloads) keeps finding its files after it pulls a new image.
files_root: str = Field(
default="/files", validation_alias=AliasChoices("FILES_ROOT", "DOWNLOAD_ROOT", "files_root")
)
# Whether THIS process runs the download worker loop. On for the dedicated worker
# container, off for the API (which keeps the scheduler). Mirrors scheduler_enabled.
worker_enabled: bool = False
@@ -345,8 +349,8 @@ class Settings(BaseSettings):
plex_transcode_crf: int = 20
plex_transcode_max_height: int = 1080
# Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
# dir under download_root. On localdev the download_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's download_root is fast
# dir under files_root. On localdev the files_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's files_root is fast
# local disk, so the default is fine there.
plex_hls_dir: str = ""
+1 -1
View File
@@ -89,7 +89,7 @@ def run() -> dict:
if a.poster_path is None:
# Record a self-hosted poster for every download (the reliable og:image source):
# ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
rel = editmod.ensure_poster(a, settings.download_root)
rel = editmod.ensure_poster(a, settings.files_root)
if rel:
a.poster_path = rel
filled_poster += 1
+4 -4
View File
@@ -732,7 +732,7 @@ def thumbnail_to_jpg(src: Path | None, dest: Path, timeout_s: int = 30) -> Path
return dest if dest.exists() else None
def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
def ensure_poster(asset, files_root: str, timeout_s: int = 30) -> str | None:
"""Generate a poster frame for a ready asset that has no thumbnail (e.g. a direct media URL
like a reddit HLS manifest, which yt-dlp can't get a thumbnail for). Written as the media's
`<base>.jpg` sidecar — so Plex uses it too and `delete_asset_files` cleans it up — and returned
@@ -740,7 +740,7 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
short clip) dodges black intro frames."""
if not asset.rel_path:
return None
root = Path(download_root)
root = Path(files_root)
src = root / asset.rel_path
if not src.exists():
return None
@@ -756,7 +756,7 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
return rel if dest.exists() else None
def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict | None:
def ensure_storyboard(asset, files_root: str, timeout_s: int = 90) -> dict | None:
"""Return the filmstrip meta for a ready source asset, generating the sprite on first use.
Best-effort + time-bounded: for a very long/high-res source the one-pass decode can be slow,
@@ -768,7 +768,7 @@ def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict |
if not asset.rel_path:
return None
geom = storyboard_geometry(asset.duration_s)
root = Path(download_root)
root = Path(files_root)
rel = f".storyboards/asset-{asset.id}.jpg"
dest = root / rel
+7
View File
@@ -144,6 +144,13 @@ _RULES: tuple[tuple[str, str], ...] = (
("postprocessing", "postprocess"),
("produced no output file", "postprocess"),
("no output file", "postprocess"),
# yt-dlp's own verdict that its extractor did not understand the site's answer: it appends this
# request for a bug report to an UNEXPECTED extraction failure, never to an expected one (a private,
# removed or geo-blocked video says so in words matched above). Measured 2026-10-12: a TikTok link
# failed with it on every attempt under yt-dlp 2026.07.04 and downloads under 2026.08.19 — the
# site changed and a newer yt-dlp is the cure. After the media-format and post-processing rules,
# whose texts can carry the same request; before the network ones, which are about the transport.
("please report this issue", "site_changed"),
# Network / transport.
("unable to download", "network"),
("http error", "network"),
+312
View File
@@ -0,0 +1,312 @@
"""The folders of the Files library — where each of a user's files sits.
A folder is a row, never a directory: on disk a file stays where the download layout put it (one
shared asset can be held by several users), so a rename or a move is a column update. Three kinds:
- a SOURCE folder is made automatically, one per origin a user has files from (YouTube, Plex clips,
uploads, one per other site). Its `source_key` names the origin; the client renders its name
from the key, in the reader's language.
- an OWN folder is the user's, with the name they gave it.
- the HUB folder (one per user, made when they first give a machine a token for it, `hub_folder`) is
where they swap files with that machine. Its name, like a source folder's, is the client's; a file
may be moved in like into an own folder, and it keeps a file until it is deleted.
Every file of a user is in exactly one folder. The folder a new file lands in is decided here
(`source_key_for` + `source_folder`), at every place a job is created, so the rule lives once. A file
moves only into one of the user's own folders, their Hub folder, or back to its own source folder
(`move`): a YouTube file in the TikTok folder would be a lie the folder's name tells.
Retention is the folder's: a source folder keeps a file `download_retention_days` (the value when the
folder was made; the user can change it), Uploads, own folders and the Hub keep it until it is deleted. A
done file carries its own `expires_at`, stamped by `done_expiry` when it finishes — unless the user
kept it — and its `ready_at`, the moment it finished (`stamp_done`).
"""
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from urllib.parse import urlsplit
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobStatus
from app.models import DownloadJob, FileFolder, MediaAsset
class FolderKind(StrEnum):
SOURCE = "source"
OWN = "own"
HUB = "hub"
class SourceKey(StrEnum):
"""The fixed origins. Any other site is `site:<host>` (see `site_key`)."""
YOUTUBE = "youtube"
PLEX = "plex"
UPLOAD = "upload"
SITE_PREFIX = "site:"
# A host's leading label that only says "the same site, another front door".
_FRONT_DOORS = ("www.", "m.", "mobile.", "web.")
_YOUTUBE_HOSTS = {"youtube.com", "youtu.be", "youtube-nocookie.com", "music.youtube.com"}
def site_key(url: str) -> str:
"""The source key of a page on some site: `site:<host>` without its front door (www., m.),
or the YouTube key for a YouTube address that arrived as a plain URL."""
host = (urlsplit(url).hostname or "").lower().rstrip(".")
for door in _FRONT_DOORS:
if host.startswith(door) and host.count(".") > 1:
host = host[len(door) :]
break
if host in _YOUTUBE_HOSTS:
return SourceKey.YOUTUBE
return f"{SITE_PREFIX}{host or 'unknown'}"
def source_key_for(db: Session, job: DownloadJob) -> str:
"""The origin a job's file came from — the source folder it belongs in.
The one recorded when it was placed, else: youtube → YouTube; a clip → the clip's source
(YouTube or Plex); an upload → Uploads; a URL → its site; an edit → wherever its parent's file
came from (its parent job, or, once that is deleted, the parent's asset)."""
if job.source_key:
return job.source_key
kind = job.source_kind
if kind == "youtube":
return SourceKey.YOUTUBE
if kind == "upload":
return SourceKey.UPLOAD
if kind == "clip":
return (
SourceKey.PLEX if (job.edit_spec or {}).get("source") == "plex" else SourceKey.YOUTUBE
)
if kind == "url":
return site_key(job.source_ref)
if kind == "edit":
parent = db.get(DownloadJob, job.source_job_id) if job.source_job_id else None
if parent is not None and parent.id != job.id:
return source_key_for(db, parent)
asset = db.get(MediaAsset, job.source_asset_id) if job.source_asset_id else None
if asset is not None:
if asset.source_kind == "youtube":
return SourceKey.YOUTUBE
if asset.source_kind == "upload":
return SourceKey.UPLOAD
if asset.source_kind == "url":
return site_key(asset.source_ref)
return SourceKey.UPLOAD
def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
"""The user's source folder for `key`, made on first use. A race on the unique (user, key) is
settled by re-selecting, inside a SAVEPOINT so the caller's own pending writes survive."""
stmt = select(FileFolder).where(FileFolder.user_id == user_id, FileFolder.source_key == key)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
retention = (
None if key == SourceKey.UPLOAD else sysconfig.get_int(db, "download_retention_days")
)
folder = FileFolder(
user_id=user_id, kind=FolderKind.SOURCE, source_key=key, retention_days=retention
)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: the expected outcome of the race the savepoint exists for — another request made
# the same folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def hub_folder(db: Session, user_id: int) -> FileFolder:
"""The user's Hub folder, made on first use; a race on its one-per-user index is settled like
`source_folder`'s."""
stmt = select(FileFolder).where(
FileFolder.user_id == user_id, FileFolder.kind == FolderKind.HUB
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
folder = FileFolder(user_id=user_id, kind=FolderKind.HUB)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: another request made the Hub folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def place(db: Session, job: DownloadJob) -> None:
"""Put a new job's file in its source folder, and record that origin on it. Called wherever a
job is created; the job must carry its `user_id`, `source_kind` and source refs already."""
key = source_key_for(db, job)
job.source_key = key
job.folder_id = source_folder(db, job.user_id, key).id
def done_expiry(db: Session, folder_id: int | None, kept: bool) -> datetime | None:
"""When a file that finishes now goes: its folder's retention from now, or never (kept, or a
folder that keeps files)."""
if kept or folder_id is None:
return None
# Read under a share lock, held to the caller's commit: a retention change (`set_retention` locks
# the folder) waits for this file and restamps it too, or this read waits for the change and
# takes the new value — never the old value stamped on a file the change did not see.
days = db.execute(
select(FileFolder.retention_days)
.where(FileFolder.id == folder_id)
.with_for_update(read=True)
).scalar()
return None if days is None else datetime.now(timezone.utc) + timedelta(days=days)
def stamp_done(db: Session, job: DownloadJob) -> None:
"""Stamp a job that just became done (in this session): when it did, and its expiry."""
job.ready_at = datetime.now(timezone.utc)
job.expires_at = done_expiry(db, job.folder_id, job.kept)
job.expiry_warned = False
class FolderError(Exception):
"""A folder change the rules refuse; `reason` is the refusal the route answers with."""
def __init__(self, reason: str) -> None:
super().__init__(reason)
self.reason = reason
def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The user's folder `folder_id` that a file is about to be put in — held (FOR SHARE) until the
caller's transaction ends, so a delete of the folder waits for the file and then takes it home
with the rest (`delete_own` locks it FOR UPDATE), never failing on its foreign key. `hold=False`
only looks (a check ahead of a long wait, which must not hold a lock through it). Another user's
folder answers like a missing one."""
stmt = select(FileFolder).where(FileFolder.id == folder_id)
if hold:
stmt = stmt.with_for_update(read=True)
folder = db.execute(stmt.execution_options(populate_existing=True)).scalar_one_or_none()
if folder is None or folder.user_id != user_id:
raise FolderError("file_folder_gone")
return folder
def upload_target(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The folder an upload asked to land in, if it may: one of the user's own folders, their Hub
folder, or their Uploads folder. A file put in a site's folder would claim an origin it does not
have."""
folder = own_folder(db, user_id, folder_id, hold=hold)
if folder.kind == FolderKind.SOURCE and folder.source_key != SourceKey.UPLOAD:
raise FolderError("file_folder_not_its_source")
return folder
def restamp(db: Session, job: DownloadJob) -> None:
"""Start a file's clock again from now: it moved to another folder, or its user stopped keeping
it. A file not done yet gets its expiry when it finishes (`stamp_done`), so it has none now."""
done = job.status == JobStatus.DONE
job.expires_at = done_expiry(db, job.folder_id, job.kept) if done else None
job.expiry_warned = False
def set_kept(db: Session, job: DownloadJob, kept: bool) -> None:
"""Keep a file until its user deletes it, or hand it back to its folder's retention."""
job.kept = kept
restamp(db, job)
def move(db: Session, job: DownloadJob, folder: FileFolder) -> None:
"""Put a file in `folder` — one of its user's own folders, their Hub folder, or the source folder
it came from.
The caller has checked that `folder` is the job's user's."""
if folder.kind == FolderKind.SOURCE and folder.source_key != source_key_for(db, job):
raise FolderError("file_folder_not_its_source")
if folder.id == job.folder_id:
return
job.folder_id = folder.id
restamp(db, job)
def _hold_for_change(db: Session, folder: FileFolder) -> None:
"""Lock `folder` FOR UPDATE and read it afresh into the same object: a change waits out a file
being put in it or stamped from it right now (`own_folder`, `done_expiry`), and works from what
another change that landed first left, not from what the caller read before the wait. The row is
consumed on purpose: an ORM result refreshes the object only as its rows are fetched."""
fresh = db.execute(
select(FileFolder)
.where(FileFolder.id == folder.id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if fresh is None:
raise FolderError("file_folder_gone")
def set_retention(db: Session, folder: FileFolder, days: int | None) -> int | None:
"""Change how long `folder` keeps a file (None = until deleted). The done files already in it
move by the difference — each keeps the age it has — but none ends up closer than the GC's
warning window, so a shortened retention still warns before it takes anything. Returns the
retention it replaced (equal to `days` when nothing changed)."""
_hold_for_change(db, folder) # so the file arriving now is among the ones restamped below
old = folder.retention_days
if old == days:
return old
folder.retention_days = days
now = datetime.now(timezone.utc)
floor = now + timedelta(days=sysconfig.get_int(db, "download_gc_grace_days"))
# Locked as they are read: one moved out or deleted meanwhile is no longer this folder's to time.
jobs = db.execute(
select(DownloadJob)
.where(
DownloadJob.folder_id == folder.id,
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
)
.with_for_update()
.execution_options(populate_existing=True)
).scalars()
for job in jobs:
if days is None:
job.expires_at = None
elif old is None or job.expires_at is None:
job.expires_at = max(now + timedelta(days=days), floor)
else:
job.expires_at = max(job.expires_at + timedelta(days=days - old), floor)
job.expiry_warned = False
return old
def delete_own(db: Session, folder: FileFolder) -> int:
"""Delete one of a user's own folders. Its files are not deleted: each goes back to the source
folder it came from, its clock starting there now. Returns how many moved."""
if folder.kind != FolderKind.OWN:
raise FolderError("file_folder_not_own")
_hold_for_change(db, folder) # so the file arriving now is taken home too
# Locked as they are read: a file moved out meanwhile stays where it went, and one deleted
# meanwhile is not written back (its gone row would fail the delete).
jobs = (
db.execute(
select(DownloadJob)
.where(DownloadJob.folder_id == folder.id)
.with_for_update()
.execution_options(populate_existing=True)
)
.scalars()
.all()
)
for job in jobs:
job.folder_id = source_folder(db, job.user_id, source_key_for(db, job)).id
restamp(db, job)
db.flush() # the files leave before the folder goes: its foreign key RESTRICTs the delete
db.delete(folder)
return len(jobs)
+192 -87
View File
@@ -1,32 +1,36 @@
"""Download retention garbage collector (a scheduler job).
Five passes each run:
1. Pre-expiry warning — for ready assets expiring within the grace window, notify each holder
once (gc_notified flag) so a download isn't silently deleted.
2. Expiry deletion — delete assets past their TTL (files + row). The FK SET NULL nulls the
holders' jobs' asset_id, so those jobs surface as "expired" (re-downloadable) in the UI.
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed
ready assets (never an upload) until back under it.
Six passes each run:
1. Pre-expiry warning — a done file whose `expires_at` (its folder's retention, `folders`) falls
within the grace window, and that its user did not keep, warns its user once (`expiry_warned`).
2. Expiry — such a file past its `expires_at` leaves the library: the job is deleted and its hold
released, so the shared file goes with its last holder.
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed ready
assets (never an upload, never one a user kept) until back under it; each holder's file leaves
their library, with a notice.
4. Orphan reap — delete errored, fileless, unreferenced asset rows (they carry no TTL, so
passes 1-3 skip them; benign row-count hygiene).
5. Orphan FILE sweep — delete on-disk files/dirs no surviving asset owns (cruft left when a
re-download renames an asset, e.g. old accented channel dirs, or an upload's stranded file).
It skips the dot-prefixed system trees (all but `.uploads`) and anything newer than an hour,
so an in-flight download is never touched.
6. Token expiry warning — a machine's token for the library about to run out warns its user once
(`apitokens.warn_expiring`): the library's other clock.
Runs in the API process (which mounts DOWNLOAD_ROOT). Pure disk/DB work — no YouTube quota.
Runs in the API process (which mounts FILES_ROOT). Pure disk/DB work — no YouTube quota.
"""
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
from sqlalchemy import func, or_, select
from sqlalchemy import func, or_, select, update
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app import apitokens, logs, sysconfig
from app.config import settings
from app.downloads import quota, storage
from app.downloads import quota, service, storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, MediaAsset
from app.notifications import create_notification
@@ -37,42 +41,118 @@ _UPLOAD_TREE = ".uploads"
_ORPHAN_MIN_AGE_S = 3600 # only sweep files older than this, so an in-flight download is never hit
def _holders(db: Session, asset_id: int) -> list[int]:
return list(
db.execute(
select(DownloadJob.user_id).where(DownloadJob.asset_id == asset_id).distinct()
).scalars()
def _notify(db: Session, job: DownloadJob, asset: MediaAsset | None, ntype: str) -> None:
title = job.display_name or (asset.title if asset else None) or "download"
create_notification(
db,
user_id=job.user_id,
type=ntype,
title=title, # English fallback; the client renders a translated message from data
data={
"kind": ntype,
"title": title,
"video_id": asset.source_ref if asset else None,
"job_id": job.id,
},
commit=False,
)
def _notify_holders(db: Session, asset: MediaAsset, ntype: str) -> None:
title = asset.title or "download"
for uid in _holders(db, asset.id):
create_notification(
db,
user_id=uid,
type=ntype,
title=title, # English fallback; the client renders a translated message from data
data={"kind": ntype, "title": asset.title, "video_id": asset.source_ref},
commit=False,
def _relock_expired(
db: Session, job_id: int, asset_id: int | None, now: datetime
) -> DownloadJob | None:
"""Re-read an expired job under the lock order the user routes take (`own_job_for_update`: the
asset, then the job) and say whether it is still to go. Between the pass's SELECT and here a
user may have deleted or re-downloaded it (dropping it again would release its asset a second
time, freeing a file another job still holds) or kept or moved it (it is no longer expired)."""
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
still = (
job is not None
and job.asset_id == asset_id
and job.status == JobStatus.DONE
and not job.kept
and job.expires_at is not None
and job.expires_at <= now
)
return job if still else None
def _mark_warned(db: Session, job_id: int, expires_at: datetime) -> DownloadJob | None:
"""Mark one file warned if it still goes when the pass saw it go — not deleted, kept, moved or
re-timed since (each of those sets a fresh expiry, and its own warning to come) — and return it.
The UPDATE's row lock holds it so until the commit."""
hit = db.execute(
update(DownloadJob)
.where(
DownloadJob.id == job_id,
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expiry_warned.is_(False),
DownloadJob.expires_at == expires_at,
)
.values(expiry_warned=True)
.returning(DownloadJob.id)
.execution_options(synchronize_session=False)
).scalar_one_or_none()
return db.get(DownloadJob, job_id) if hit is not None else None
def _delete_asset(db: Session, asset: MediaAsset, ntype: str) -> None:
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
_notify_holders(db, asset, ntype)
db.delete(asset) # jobs.asset_id -> NULL via FK ON DELETE SET NULL
def _on_disk(db: Session) -> int:
"""The bytes the cache holds now (every asset whose file is on disk)."""
return (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).scalar()
or 0
)
def _sweep_orphan_files(db: Session, download_root: str) -> int:
"""Delete media/sidecar files (and now-empty dirs) under DOWNLOAD_ROOT that no `ready` asset
def _relock_evictable(db: Session, asset_id: int) -> tuple[MediaAsset | None, list[DownloadJob]]:
"""Re-read an eviction candidate and its holders under their locks (the asset, then its jobs —
the order the user routes take): between the pass's SELECT and here a holder may have kept it,
or every holder deleted it."""
asset = db.execute(
select(MediaAsset)
.where(MediaAsset.id == asset_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
holders = (
db.execute(
select(DownloadJob)
.where(DownloadJob.asset_id == asset_id)
.with_for_update()
.execution_options(populate_existing=True)
)
.scalars()
.all()
)
return asset, list(holders)
def _drop_job(db: Session, job: DownloadJob, ntype: str) -> None:
"""Take a file out of its user's library: tell them, release the job's hold on the asset (the
file goes with the last holder), delete the row (its shares and links cascade)."""
_notify(db, job, db.get(MediaAsset, job.asset_id) if job.asset_id else None, ntype)
service.release_asset(db, job)
db.delete(job)
def _sweep_orphan_files(db: Session, files_root: str) -> int:
"""Delete media/sidecar files (and now-empty dirs) under FILES_ROOT that no `ready` asset
owns — cruft left when a re-download changes an asset's on-disk name (e.g. an old accented
channel dir, or a poster from a superseded naming scheme) — or, under `.uploads`, that no upload
row names at all. Skips the other dot-prefixed system trees and only removes files older than
`_ORPHAN_MIN_AGE_S`, so a just-completed or in-flight download is never deleted, and only files
NOT owned by a surviving asset."""
root = Path(download_root)
root = Path(files_root)
if not root.is_dir():
return 0
keep: set[Path] = set()
@@ -151,74 +231,95 @@ def run_download_gc(db: Session) -> dict:
grace_days = sysconfig.get_int(db, "download_gc_grace_days")
warned = expired = evicted = reaped = 0
# 1. Pre-expiry warnings.
# 1. Pre-expiry warnings — one per file, the first run inside the grace window. Each mark is
# conditional on the expiry the SELECT saw and committed on its own: a file deleted since is
# passed over (a write to its gone row would fail the whole run), and one kept, moved or re-timed
# since keeps its fresh expiry unwarned.
warn_before = now + timedelta(days=grace_days)
soon = (
db.execute(
select(MediaAsset).where(
MediaAsset.status == "ready",
MediaAsset.gc_notified.is_(False),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at > now,
MediaAsset.expires_at <= warn_before,
)
soon = db.execute(
select(DownloadJob.id, DownloadJob.expires_at).where(
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expiry_warned.is_(False),
DownloadJob.expires_at.is_not(None),
DownloadJob.expires_at > now,
DownloadJob.expires_at <= warn_before,
)
.scalars()
.all()
)
for asset in soon:
_notify_holders(db, asset, "download_expiring")
asset.gc_notified = True
warned += 1
if soon:
).all()
db.rollback() # the pass-1 snapshot holds no locks; each mark takes its own below
for job_id, expires_at in soon:
job = _mark_warned(db, job_id, expires_at)
if job is not None:
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
_notify(db, job, asset, "download_expiring")
warned += 1
db.commit()
# 2. Expiry deletion (ready assets past TTL, and any errored leftovers with a TTL).
dead = (
db.execute(
select(MediaAsset).where(
MediaAsset.status.in_(("ready", "error")),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at <= now,
)
# 2. Expiry — a file past its folder's retention leaves the library (warned in pass 1).
# Each file is re-checked under its locks and committed on its own: the release unlinks the
# file at once, so it must never act on what the SELECT saw a while ago.
dead = db.execute(
select(DownloadJob.id, DownloadJob.asset_id).where(
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expires_at.is_not(None),
DownloadJob.expires_at <= now,
)
.scalars()
.all()
)
for asset in dead:
_delete_asset(db, asset, "download_expired")
expired += 1
if dead:
).all()
db.rollback() # the pass-2 snapshot holds no locks; each file takes its own below
for job_id, asset_id in dead:
job = _relock_expired(db, job_id, asset_id, now)
if job is not None:
_drop_job(db, job, "download_expired")
expired += 1
db.commit()
# 3. LRU eviction over the total cache cap.
# 3. LRU eviction over the total cache cap: the disk's safety net. Never an upload (no source to
# fetch again) and never a file anyone kept; an evicted file leaves every holder's library.
cap = sysconfig.get_int(db, "download_total_max_bytes")
if cap > 0:
total = (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).scalar()
or 0
)
if total > cap:
# An upload counts toward the total (it is disk like any other) but is never evicted:
# it has no source to fetch again, and it lives until its owner deletes it.
candidates = (
if _on_disk(db) > cap:
kept_assets = select(DownloadJob.asset_id).where(
DownloadJob.kept.is_(True), DownloadJob.asset_id.is_not(None)
)
candidate_ids = (
db.execute(
select(MediaAsset)
.where(MediaAsset.status == "ready", MediaAsset.source_kind != "upload")
select(MediaAsset.id)
.where(
MediaAsset.status == "ready",
MediaAsset.source_kind != "upload",
MediaAsset.id.not_in(kept_assets),
)
.order_by(MediaAsset.last_access_at.asc().nulls_first())
)
.scalars()
.all()
)
for asset in candidates:
if total <= cap:
db.rollback()
for asset_id in candidate_ids:
# Counted afresh before each: a file deleted since (this one, an upload, a candidate
# further down) has freed its space, and no other user's file goes for that space.
if _on_disk(db) <= cap:
break
total -= asset.size_bytes or 0
_delete_asset(db, asset, "download_evicted")
# Locked and re-read like an expiry: since the SELECT a holder may have kept it,
# every holder deleted it, or someone asked for it again (a fresh cache hit).
asset, holders = _relock_evictable(db, asset_id)
if (
asset is None
or asset.status != "ready"
or any(j.kept for j in holders)
or (asset.last_access_at is not None and asset.last_access_at > now)
):
db.commit()
continue
for job in holders:
_notify(db, job, asset, "download_evicted")
db.delete(job)
if asset.rel_path:
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
evicted += 1
db.commit()
db.commit()
# 4. Reap orphaned errored rows (no file, no holder). get_or_create_asset resets a still-
# referenced errored row to pending on re-enqueue, so these only accumulate once every
@@ -257,7 +358,7 @@ def run_download_gc(db: Session) -> dict:
)
for asset in unheld:
if asset.rel_path: # no holder is left to tell, so no notice — just the file and the row
storage.delete_asset_files(settings.download_root, asset.rel_path)
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
reaped += 1
if unheld:
@@ -265,7 +366,10 @@ def run_download_gc(db: Session) -> dict:
# 5. Orphan FILE sweep — remove on-disk cruft no surviving asset owns (old accented dirs, stale
# posters). Runs last so the keep-set reflects the assets that survived passes 2-4.
swept = _sweep_orphan_files(db, settings.download_root)
swept = _sweep_orphan_files(db, settings.files_root)
# 6. A machine's token about to run out.
tokens_warned = apitokens.warn_expiring(db)
result = {
"warned": warned,
@@ -273,6 +377,7 @@ def run_download_gc(db: Session) -> dict:
"evicted": evicted,
"reaped": reaped,
"swept": swept,
"tokens_warned": tokens_warned,
}
log.info("download_gc %s", result)
return result
+2 -2
View File
@@ -68,8 +68,8 @@ def _reset_asset_pending(asset_id: int) -> None:
def _staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"asset-{asset_id}"
return Path(settings.files_root) / ".staging" / f"asset-{asset_id}"
def _edit_staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"edit-{asset_id}"
return Path(settings.files_root) / ".staging" / f"edit-{asset_id}"
+75 -11
View File
@@ -1,10 +1,15 @@
"""Public "share by link" helpers for the download center.
A DownloadLink is a capability URL: the unguessable `token` grants anyone read access to one
download's file on the login-free `/watch/{token}` page. Password-protected links additionally
require a short-lived signed **grant** on the file request — because a `<video src>` can't send a
header, the password is exchanged once at `/unlock` for an HMAC grant appended to the file URL,
so the raw password never rides in the media URL (or server logs).
A DownloadLink is a capability URL: the unguessable token grants anyone read access to one
download's file, or to the files in one of its owner's own folders (`folder_files`), on the
login-free `/watch/{token}` page. So the token is never stored as written:
the link is found by its SHA-256, and the owner's copy is Fernet-encrypted — a database copy alone
opens nothing. Without that copy (no TOKEN_ENCRYPTION_KEY, or a key since changed) the link still
opens; only its address cannot be shown again, and the owner makes a new link to share.
Password-protected links additionally require a short-lived signed **grant** on the file request —
because a `<video src>` can't send a header, the password is exchanged once at `/unlock` for an HMAC
grant appended to the file URL, so the raw password never rides in the media URL (or server logs).
"""
import hashlib
@@ -12,16 +17,71 @@ import hmac
import secrets
from datetime import datetime, timezone
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import security
from app.config import settings
from app.models import DownloadLink
from app.downloads import storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_GRANT_TTL = 6 * 3600 # a watch session's grant is good for 6h, then the viewer re-unlocks
# The most files a folder link lists: as many as the owner's own Files list shows (`routes.files.
# MAX_LIST`; a test holds the two equal — this layer may not import that one).
FOLDER_MAX = 500
def new_token() -> str:
return secrets.token_urlsafe(24) # ~32 chars, 192 bits
def seal(link: DownloadLink, token: str) -> None:
"""Store `token` on `link` the only ways it is kept: its hash, and its encrypted copy."""
link.token_hash = security.hash_token(token)
link.token_enc = security.encrypt(token) if security.can_encrypt() else None
def find(db: Session, token: str) -> DownloadLink | None:
"""The link a `/watch` token names, by its hash."""
return db.execute(
select(DownloadLink).where(DownloadLink.token_hash == security.hash_token(token))
).scalar_one_or_none()
def recover(link: DownloadLink) -> str | None:
"""The link's token from its encrypted copy, or None when there is none this key can read."""
return security.decrypt_optional(link.token_enc)
def in_folder(folder: FileFolder) -> tuple:
"""Which jobs a link to `folder` reaches: the done files in it NOW, the folder owner's. A file
moved out stops opening through the link at its next request; one moved in starts."""
return (
DownloadJob.folder_id == folder.id,
DownloadJob.user_id == folder.user_id,
DownloadJob.status == JobStatus.DONE,
)
def folder_files(db: Session, folder: FileFolder) -> list[tuple[DownloadJob, MediaAsset]]:
"""What a link to `folder` shows — its page lists exactly these, and its link card counts them:
each file `in_folder` whose asset is ready and whose bytes are on disk (so the list never offers
a dead row), newest first, at most `FOLDER_MAX`."""
rows = db.execute(
select(DownloadJob, MediaAsset)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(*in_folder(folder), MediaAsset.status == "ready")
.order_by(DownloadJob.ready_at.desc().nulls_last(), DownloadJob.id.desc())
.limit(FOLDER_MAX)
).all()
return [
(job, asset)
for job, asset in rows
if storage.safe_abs_path(settings.files_root, asset.rel_path) is not None
]
def is_expired(link: DownloadLink) -> bool:
return link.expires_at is not None and link.expires_at <= datetime.now(timezone.utc)
@@ -55,18 +115,22 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
return False
if exp < datetime.now(timezone.utc).timestamp():
return False
return hmac.compare_digest(sig, _sign(f"{token}.{exp}.{password_version}"))
# As bytes: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it, and
# `sig` is whatever the URL carried.
return hmac.compare_digest(sig.encode(), _sign(f"{token}.{exp}.{password_version}").encode())
# --- serialization -------------------------------------------------------------------------
def owner_view(link: DownloadLink, base_url: str = "") -> dict:
"""What the file owner sees when managing their links."""
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
"""What the file owner sees when managing their links. `token` is the one just made (it is in
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
token = token or recover(link)
return {
"id": link.id,
"token": link.token,
"url": f"{base_url}/watch/{link.token}",
"token": token,
"url": f"{base_url}/watch/{token}" if token else None,
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
+32 -6
View File
@@ -11,13 +11,12 @@ import html
import re
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import links as linksmod
from app.downloads import storage
from app.models import DownloadJob, DownloadLink, MediaAsset
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
@@ -74,22 +73,25 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
def _watch_tags(token: str, db: Session) -> str | None:
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
return None
url = f"{settings.app_base}/watch/{token}"
folder = link.folder_id is not None
if link.password_hash:
# Password-gated: don't leak the title/thumbnail in the unfurl.
return "\n ".join(
[
"<title>Siftlode</title>",
_tag("og:title", "Password-protected video"),
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
_tag("og:description", "Shared via Siftlode"),
_tag("og:type", "video.other"),
_tag("og:type", "website" if folder else "video.other"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
]
)
if folder:
return _folder_tags(db, link, url)
job = db.get(DownloadJob, link.job_id)
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if job is None or asset is None or asset.status != "ready":
@@ -135,7 +137,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
if image_dims is None and image.endswith("poster.jpg") and asset.poster_path:
# safe_abs_path returns None if the poster file is missing (gc'd) or escapes the root —
# guard it, else _jpeg_size(None) would raise an uncaught AttributeError on this public page.
poster_abs = storage.safe_abs_path(settings.download_root, asset.poster_path)
poster_abs = storage.safe_abs_path(settings.files_root, asset.poster_path)
image_dims = _jpeg_size(poster_abs) if poster_abs else None
if image_dims:
tags.append(_tag("og:image:width", str(image_dims[0])))
@@ -147,6 +149,30 @@ def _watch_tags(token: str, db: Session) -> str | None:
return "\n ".join(tags)
def _folder_tags(db: Session, link: DownloadLink, url: str) -> str | None:
"""A folder link's card: the folder's name and how many files it holds — never their names,
which the page shows only to whoever opens it."""
folder = db.get(FileFolder, link.folder_id)
if folder is None or not folder.name:
return None
count = len(linksmod.folder_files(db, folder)) # the page's own list, so the two agree
title = folder.name.strip()
about = f"{count} file{'' if count == 1 else 's'} · Shared via Siftlode"
return "\n ".join(
[
f"<title>{html.escape(title)}</title>",
_tag("og:title", title),
_tag("og:description", about),
_tag("og:type", "website"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
_tag("twitter:title", title, attr="name"),
_tag("twitter:description", about, attr="name"),
_tag("twitter:card", "summary", attr="name"),
]
)
def render_watch_html(token: str, page_html: Path, db: Session) -> str | None:
"""Return the watch page (watch.html) with the OG block replaced for this token, or None to
serve the generic page unchanged (invalid/expired/not-ready link, or a template without the
+32 -33
View File
@@ -3,7 +3,7 @@
import re
import shutil
import time
from datetime import date, datetime, timedelta, timezone
from datetime import date, datetime, timezone
from pathlib import Path
import yt_dlp
@@ -95,16 +95,24 @@ def _notify_download_done(db, user_id: int | None, title: str | None, job_id, *,
notification failure must never mask the completed job, so it is isolated + rolled back (mirrors
scheduler._notify_done). `kind` ("download"/"edit"/"upload") lets the inbox pick accurate copy; the
English title here is only a fallback for an unknown type. `user_id`/`title` are captured by the
caller BEFORE its commit, so reading them here needs no post-commit refresh of an expired row."""
caller BEFORE its commit; only the folder is read here, once, for the notice's "Open"."""
if user_id is None:
return
try:
# The notice's "Open" lands on the file in its folder: both ride along (a folder the file
# has left since is the page's to notice — it then looks among every file).
job = db.get(DownloadJob, job_id)
create_notification(
db,
user_id=user_id,
type="download_done",
title=f"Ready to watch: {title}" if title else "Ready to watch",
data={"title": title, "kind": kind},
data={
"title": title,
"kind": kind,
"job_id": job_id,
"folder_id": job.folder_id if job else None,
},
)
except Exception:
db.rollback()
@@ -434,14 +442,12 @@ def _extract_with_fallback(
raise last_exc # unreachable (loop either returns or raises)
def _download_settings() -> tuple[str, int]:
"""Layout + retention days from the DB config (admin-overridable on the Configuration page),
falling back to the env defaults. Read here, not from `settings`, so an admin edit takes effect."""
def _download_layout() -> str:
"""The on-disk layout from the DB config (admin-overridable on the Configuration page), falling
back to the env default. Read here, not from `settings`, so an admin edit takes effect. (How long
a file stays is its folder's, stamped on the job when it is done — `folders.done_expiry`.)"""
with SessionLocal() as db:
return (
sysconfig.get_str(db, "download_layout"),
sysconfig.get_int(db, "download_retention_days"),
)
return sysconfig.get_str(db, "download_layout")
def _ensure_phone_safe(
@@ -588,19 +594,18 @@ def _download(
thumbnail_url=info.get("thumbnail"),
)
layout, retention_days = _download_settings()
layout = _download_layout()
rel = storage.rel_path(meta, ext, layout, asset_id=asset_id)
thumb = _find_thumbnail(staging, produced)
# Convert the raw thumbnail (webp/png/jpeg) to a real JPEG ourselves — robustly — for the
# poster + og:image sidecar, now that yt-dlp's fatal thumbnail convertor is gone. On failure
# this is None and write_sidecars writes no poster, so ensure_poster cuts a frame instead.
poster_jpg = editmod.thumbnail_to_jpg(thumb, staging / "poster.jpg")
storage.place_file(produced, settings.download_root, rel)
nfo_ok = storage.write_sidecars(settings.download_root, rel, meta, poster_jpg)
storage.place_file(produced, settings.files_root, rel)
nfo_ok = storage.write_sidecars(settings.files_root, rel, meta, poster_jpg)
final = storage.abs_path(settings.download_root, rel)
final = storage.abs_path(settings.files_root, rel)
size = final.stat().st_size if final.exists() else None
expires = datetime.now(timezone.utc) + timedelta(days=retention_days)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
@@ -626,12 +631,11 @@ def _download(
# (og:image) source — a remote thumbnail, esp. Facebook's signed CDN URL, expires
# and can't be relied on cross-origin. write_sidecars already wrote <base>.jpg from
# the thumbnail; ensure_poster returns that, or cuts a frame if there was none.
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
asset.source_webpage_url = info.get("webpage_url")
asset.nfo_written = nfo_ok
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = expires
job = db.get(DownloadJob, job_id)
# Capture the owner before commit expires the row (avoids a refresh SELECT just to notify).
owner_id = job.user_id if job else None
@@ -840,7 +844,7 @@ def _process_edit(job_id: int, asset_id: int) -> None:
if not src_ready or not src_rel:
_fail_edit(job_id, asset_id, "The source download is no longer available.")
return
src_path = storage.abs_path(settings.download_root, src_rel)
src_path = storage.abs_path(settings.files_root, src_rel)
if not src_path.exists():
_fail_edit(job_id, asset_id, "The source file is missing.")
return
@@ -887,13 +891,11 @@ def _process_edit(job_id: int, asset_id: int) -> None:
raise StatedFailure(dlerrors.CLIP_TOO_BIG_MESSAGE)
rel = storage.edit_rel_path(user_id, asset_id, out_ext)
storage.place_file(dest_staging, settings.download_root, rel)
final = storage.abs_path(settings.download_root, rel)
storage.place_file(dest_staging, settings.files_root, rel)
final = storage.abs_path(settings.files_root, rel)
size = final.stat().st_size if final.exists() else None
# The clip's real frame: a re-encode evens an odd crop or source (`editmod.EVEN_FRAME`).
_, _, out_w, out_h, _ = probe.probe_media(final)
_, retention_days = _download_settings()
expires = datetime.now(timezone.utc) + timedelta(days=retention_days)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
@@ -912,7 +914,6 @@ def _process_edit(job_id: int, asset_id: int) -> None:
asset.width, asset.height = src_w, src_h
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = expires
job = db.get(DownloadJob, job_id)
# Capture owner + title before commit expires the rows (avoids a refresh SELECT).
owner_id = job.user_id if job else None
@@ -1086,10 +1087,9 @@ def _process_clip(job_id: int, asset_id: int) -> None:
raise _OwnerRefused(dlerrors.CLIP_TOO_BIG_MESSAGE)
ext = out.suffix.lstrip(".").lower() or "mp4"
rel = storage.edit_rel_path(user_id, asset_id, ext)
storage.place_file(out, settings.download_root, rel)
final = storage.abs_path(settings.download_root, rel)
storage.place_file(out, settings.files_root, rel)
final = storage.abs_path(settings.files_root, rel)
vcodec, acodec, width, height, _ = probe.probe_media(final)
_, retention_days = _download_settings()
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
@@ -1108,10 +1108,9 @@ def _process_clip(job_id: int, asset_id: int) -> None:
asset.media_kind = mediakind.derive(vcodec, acodec)
# The clip's own frame as its poster: the link preview shows the moment, and a Plex
# item has no public thumbnail at all.
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = datetime.now(timezone.utc) + timedelta(days=retention_days)
fields: dict = {"status": JobStatus.DONE, "progress": 100, "phase": None}
# A video outside the catalog was queued as "Clip (clip)": the fetch has its title now.
placeholder = service.clip_title(None, None)
@@ -1230,7 +1229,7 @@ def _free_unheld_upload(asset_id: int) -> bool:
if asset is None or asset.ref_count != 0:
return False
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
db.commit()
return True
@@ -1253,7 +1252,7 @@ def _convert_upload(job_id: int, asset_id: int) -> None:
duration = asset.duration_s if asset else None
job = db.get(DownloadJob, job_id)
owner_id = job.user_id if job else None
src = storage.safe_abs_path(settings.download_root, old_rel) if old_rel else None
src = storage.safe_abs_path(settings.files_root, old_rel) if old_rel else None
if old_rel is None or src is None or not src.exists():
_fail_edit(job_id, asset_id, dlerrors.UPLOAD_MISSING_MESSAGE, what="upload")
return
@@ -1267,10 +1266,10 @@ def _convert_upload(job_id: int, asset_id: int) -> None:
new_rel = storage.with_ext(old_rel, old_ext, ext)
if safe != src:
_check_converted_fits(owner_id, old_size, safe.stat().st_size)
storage.place_file(safe, settings.download_root, new_rel)
storage.place_file(safe, settings.files_root, new_rel)
if new_rel != old_rel:
placed = storage.abs_path(settings.download_root, new_rel)
final = storage.abs_path(settings.download_root, new_rel)
placed = storage.abs_path(settings.files_root, new_rel)
final = storage.abs_path(settings.files_root, new_rel)
vcodec, acodec, width, height, dur = probe.probe_media(final)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
+107
View File
@@ -0,0 +1,107 @@
"""What a library file previews as — the single home of the preview allow-list.
A generic file (`media_kind == "file"`; video and audio have their players) previews by its stored
`container`, the upload's cleaned extension, and ONLY through this table: an image is shown as an
image, a PDF in the browser's own viewer, and a text format as plain text. Anything else has no
preview. The served Content-Type comes from here too, never from a guess on the name, and HTML / SVG
sit in the TEXT list on purpose: rendered on this origin, an uploaded page would run with the
reader's session. The route adds `CSP: sandbox` on top, so even a mislabelled file (a "png" that is
really a page) gets no script and no origin.
The serializer hands the kind to the client (`asset.preview`), so the library and the server can
never disagree on which files open a preview.
"""
import codecs
from typing import Literal
from app.downloads import mediakind
PreviewKind = Literal["image", "pdf", "text"]
IMAGE_TYPES: dict[str, str] = {
"png": "image/png",
"jpg": "image/jpeg",
"jpeg": "image/jpeg",
"gif": "image/gif",
"webp": "image/webp",
"avif": "image/avif",
"bmp": "image/bmp",
}
PDF_TYPE = "application/pdf"
# Read as text: notes, data, subtitles, config and source. HTML and SVG belong here (see above).
TEXT_EXTS: frozenset[str] = frozenset(
{
"txt", "text", "md", "markdown", "log", "csv", "tsv", "json", "jsonl", "ndjson", "xml",
"yaml", "yml", "toml", "ini", "conf", "cfg", "properties", "env", "srt", "vtt", "ass", "ssa",
"sub", "nfo", "cue", "m3u", "m3u8", "html", "htm", "xhtml", "svg", "css", "scss", "less",
"js", "mjs", "cjs", "ts", "tsx", "jsx", "py", "rb", "php", "pl", "lua", "go", "rs", "java",
"kt", "kts", "swift", "c", "h", "cc", "cpp", "hpp", "cs", "sh", "bash", "zsh", "fish", "ps1",
"bat", "cmd", "sql", "r", "dart", "diff", "patch", "tex", "rst", "adoc", "gradle", "dockerfile",
}
) # fmt: skip
TEXT_TYPE = "text/plain; charset=utf-8"
# A text preview is the file's head: enough to read, never a whole log loaded into a tab.
TEXT_PREVIEW_BYTES = 1 << 20
# A NUL byte in the first this-many bytes means the "text" is not text.
BINARY_SNIFF_BYTES = 8192
def kind(media_kind: str | None, container: str | None) -> PreviewKind | None:
"""The preview a library file gets, or None (a media file plays instead; others download)."""
if media_kind != mediakind.FILE:
return None
ext = (container or "").lower()
if ext in IMAGE_TYPES:
return "image"
if ext == "pdf":
return "pdf"
if ext in TEXT_EXTS:
return "text"
return None
def media_type(preview: PreviewKind, container: str | None) -> str:
"""The Content-Type a preview is served with: from this table, never guessed from the name."""
if preview == "image":
return IMAGE_TYPES[(container or "").lower()]
if preview == "pdf":
return PDF_TYPE
return TEXT_TYPE
# A text that is not UTF-8 is read as Central European Windows: what an old Hungarian subtitle or
# note is written in (its ő and ű are not in Latin-1).
FALLBACK_ENCODING = "cp1250"
_BOMS = (
(codecs.BOM_UTF8, "utf-8-sig"),
(codecs.BOM_UTF16_LE, "utf-16"),
(codecs.BOM_UTF16_BE, "utf-16"),
)
def looks_binary(head: bytes) -> bool:
"""Whether the start of a "text" file is binary after all (a NUL in it)."""
return b"\x00" in head[:BINARY_SNIFF_BYTES]
def text_utf8(head: bytes) -> bytes | None:
"""A text file's head as UTF-8, the one charset the preview is served in; None when it is not text.
A byte-order mark names its encoding (UTF-16 is full of NULs, so it is read before the binary
check); else UTF-8 is kept as it is — a character cut at the size cap included, the reader shows
that one as a replacement mark — and anything else is read as `FALLBACK_ENCODING`."""
for bom, encoding in _BOMS:
if head.startswith(bom):
return head.decode(encoding, errors="replace").encode()
if looks_binary(head):
return None
try:
head.decode("utf-8")
except UnicodeDecodeError as e: # silent-ok: not UTF-8 is an answer — read it as the fallback
if not (e.reason == "unexpected end of data" and e.end == len(head)):
return head.decode(FALLBACK_ENCODING, errors="replace").encode()
return head
+71 -4
View File
@@ -19,7 +19,7 @@ from app.downloads import adapters
from app.downloads import clip as clipmod
from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import formats, mediakind, quota, storage
from app.downloads import folders, formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobKind, JobStatus
from app.models import (
DISPLAY_TEXT_MAX,
@@ -49,7 +49,7 @@ def source_url(source_kind: str, source_ref: str) -> str:
def reference_url(job, asset) -> str | None:
"""The clean "downloaded from" URL for a job — shared by the private Downloads page and the public
"""The clean "downloaded from" URL for a job — shared by the Files library and the public
watch page so both surfaces always agree. None for edit derivatives (a clip's source is the user's
own earlier download, not a web page) and for a Plex clip (the library is not a public page); a
YouTube clip links its video at the clip's start; else the canonical page URL yt-dlp recorded; else
@@ -65,6 +65,23 @@ def reference_url(job, asset) -> str | None:
return None
def library_left_behind(db: Session, files_root: str) -> bool:
"""True when the library names files and none of the newest of them is under `files_root` — a
fresh, empty volume mounted in the library's place (a compose file changed under an install, its
old volume left behind). The mount check cannot see that: the new volume is a real mount."""
rels = (
db.execute(
select(MediaAsset.rel_path)
.where(MediaAsset.status == "ready", MediaAsset.rel_path.is_not(None))
.order_by(MediaAsset.id.desc())
.limit(20)
)
.scalars()
.all()
)
return bool(rels) and not any(storage.abs_path(files_root, r).exists() for r in rels if r)
def get_or_create_asset(
db: Session, source_kind: str, source_ref: str, format_sig: str
) -> MediaAsset:
@@ -77,7 +94,12 @@ def get_or_create_asset(
MediaAsset.source_ref == source_ref,
MediaAsset.format_sig == format_sig,
)
asset = db.execute(stmt).scalar_one_or_none()
# Locked to the caller's commit: a release or the GC freeing this asset meanwhile (they lock it
# too) either waits for the new hold, or has deleted the row by the time this read returns — and
# the caller then gets a fresh one, never a job pointing at a row that is gone.
asset = db.execute(
stmt.with_for_update().execution_options(populate_existing=True)
).scalar_one_or_none()
if asset is not None:
# A previously-FAILED shared asset must be re-attempted for the new job about to hold it —
# reset it to pending (+clear the error) so the worker actually re-downloads instead of
@@ -198,12 +220,14 @@ def enqueue(
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
# Cache hit: the file already exists — no worker round-trip needed.
if asset.status == "ready":
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
@@ -277,11 +301,13 @@ def enqueue_edit(
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
if asset.status == "ready": # identical clip already produced for this user → instant
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
@@ -364,11 +390,13 @@ def enqueue_clip(
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
if asset.status == "ready": # the same range was already cut for this user → instant
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
@@ -392,6 +420,8 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
if not fields:
return False
db.flush() # asset-then-job lock order: push pending asset writes before we lock the job row
if fields.get("status") == JobStatus.DONE:
return _finalize_done(db, job_id, {**fields, "ready_at": datetime.now(timezone.utc)})
# A Core UPDATE with the guard as a WHERE — same atomic "only while still running" write as the
# rest of the codebase's 2.0 statements, no hand-built SQL string. synchronize_session=False: this
# is a terminal write, so we don't pay to sync the (about-to-be-discarded) ORM identity map. Bound
@@ -405,6 +435,41 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
return result.rowcount > 0
def _finalize_done(db: Session, job_id: int, fields: dict) -> bool:
"""`finalize_if_running` for a job that finished: done is when a file's retention starts, its
folder's unless the user kept it. The expiry is written only while the job is still where it was
read from — a move, a keep, or its folder's delete (which sends its files home) landing while the
folder's retention was read sends the read round again, never stamping a folder's retention on a
file now in another one."""
while True:
held = db.execute(
select(DownloadJob.folder_id, DownloadJob.kept, DownloadJob.status).where(
DownloadJob.id == job_id
)
).one_or_none()
if held is None or held.status != JobStatus.RUNNING:
return False
stamped = {
**fields,
"expires_at": folders.done_expiry(db, held.folder_id, held.kept),
"expiry_warned": False,
}
hit = db.execute(
update(DownloadJob)
.where(
DownloadJob.id == job_id,
DownloadJob.status == JobStatus.RUNNING.value,
DownloadJob.folder_id.is_not_distinct_from(held.folder_id),
DownloadJob.kept == held.kept,
)
.values(**stamped)
.returning(DownloadJob.id)
.execution_options(synchronize_session=False)
).scalar_one_or_none()
if hit is not None:
return True
def release_asset(db: Session, job: DownloadJob) -> None:
"""Drop this job's hold on its asset when it leaves a holding state. Once NO job holds the
asset anymore, delete the file + row immediately — a deleted download should free its disk,
@@ -436,7 +501,7 @@ def release_asset(db: Session, job: DownloadJob) -> None:
)
if new_count == 0 and freeable:
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
@@ -565,4 +630,6 @@ def redownload(db: Session, job: DownloadJob) -> None:
# name would survive the rebuild).
job.display_name = None
job.queue_pos = next_queue_pos(db) # tail of the queue, not its stale original slot
job.expires_at = None # stamped again when the rebuild is done
job.expiry_warned = False
db.commit()
+4
View File
@@ -53,3 +53,7 @@ NOT_REFETCHABLE = frozenset({JobKind.EDIT, JobKind.UPLOAD, JobKind.CLIP})
# States a route sets to STOP a running job. The worker's supervisor must step in (kill the child, free
# the asset, leave the route's status intact) when it observes one of these on a job it is running.
STOPPED_BY_ROUTE = frozenset({JobStatus.PAUSED, JobStatus.CANCELED})
# A job that will not move again on its own: the queue lists it only briefly after it got here.
FINISHED = frozenset({JobStatus.DONE, JobStatus.CANCELED})
+79 -20
View File
@@ -11,6 +11,7 @@ a Plex library using a metadata agent picks up rich info. Channel = "show", vide
date-based — the community-standard mapping for YouTube-in-Plex.
"""
import os
import re
import shutil
import unicodedata
@@ -67,6 +68,57 @@ _FOLD_MAP = str.maketrans(
)
def _mount_points(mountinfo: str) -> list[str]:
"""The mount points a /proc/<pid>/mountinfo table lists (its fifth field, octal escapes undone)."""
points = []
for line in mountinfo.splitlines():
fields = line.split(" ")
if len(fields) > 4:
points.append(re.sub(r"\\([0-7]{3})", lambda m: chr(int(m.group(1), 8)), fields[4]))
return points
def _in_container() -> bool:
return Path("/.dockerenv").exists() or Path("/run/.containerenv").exists()
def _read_mountinfo() -> str | None:
try:
return Path("/proc/self/mountinfo").read_text()
# silent-ok: no mount table to read (not Linux) — the caller then cannot tell, and refuses nothing.
except OSError:
return None
def files_root_problem(
root: str, mountinfo: str | None = None, in_container: bool | None = None
) -> str | None:
"""Why `root` cannot be the files library's volume, or None when it can. In a container, a root
that is no mount of its own (nor inside one) is the image's empty directory: the library would
answer 404 for every file, and what a download writes would go when the container does — the
state an install is in when its compose file still mounts the library somewhere else.
Asked of the kernel's mount table, not by comparing devices: a storage driver that keeps the
container's root on the host's own disk (vfs, the fallback under Docker-in-LXC on older ZFS) gives
a mounted volume the same device as `/`. Outside a container a plain directory is a fine root."""
path = Path(root)
if not path.is_dir():
return f"{root} does not exist"
if in_container if in_container is not None else _in_container():
table = mountinfo if mountinfo is not None else _read_mountinfo()
if table is not None:
real = os.path.realpath(root)
mounted = any(
m != "/" and (real == m or real.startswith(m.rstrip("/") + "/"))
for m in _mount_points(table)
)
if not mounted:
return f"{root} is not a mounted volume (it is the container's own empty directory)"
if not os.access(path, os.W_OK):
return f"{root} is not writable"
return None
def _ascii_fold(text: str) -> str:
"""Transliterate to pure ASCII: apply the explicit map, NFKD-split accented letters and drop
the combining marks (á→a, ő→o), then strip any leftover non-ASCII (non-Latin scripts that have
@@ -122,7 +174,7 @@ def display_filename(name: str) -> str:
def rel_path(meta: MediaMeta, ext: str, layout: str = "plex", asset_id: int | None = None) -> str:
"""Path of the media file relative to DOWNLOAD_ROOT (forward slashes).
"""Path of the media file relative to FILES_ROOT (forward slashes).
`asset_id` disambiguates the file. A MediaAsset is unique on (source_kind, source_ref,
format_sig), so the SAME video downloaded in two formats is two distinct assets — but without
@@ -144,7 +196,7 @@ def rel_path(meta: MediaMeta, ext: str, layout: str = "plex", asset_id: int | No
def edit_rel_path(user_id: int, asset_id: int, ext: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a phase-2 editor derivative.
"""On-disk path (relative to FILES_ROOT) for a phase-2 editor derivative.
Kept in a per-user `.edits/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (these are user clips, not canonical episodes; there are no .nfo/poster
@@ -161,7 +213,7 @@ def _clean_upload_ext(filename: str) -> str:
def upload_rel_path(user_id: int, asset_id: int, filename: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a locally-uploaded file.
"""On-disk path (relative to FILES_ROOT) for a locally-uploaded file.
Kept in a per-user `.uploads/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (an upload is arbitrary user content, not a canonical episode; no .nfo/poster
@@ -242,32 +294,39 @@ def served_media_type(media_kind: str | None, container: str | None) -> str | No
def download_filename(display_name: str, container: str | None, path: Path) -> str:
"""The Content-Disposition filename for a served download: the cleaned display name carrying a single
correct extension — the asset container, else the file's own suffix — without doubling it when the
name already ends in that extension. Shared by the private download and the public watch endpoints."""
name already ends in that extension. A file with no extension at all is served without one —
never as `name.` (Windows drops a trailing dot, so the saved file would not be the one named).
Shared by the private download and the public watch endpoints."""
ext = container or path.suffix.lstrip(".")
base = display_filename(display_name)
if not ext:
return base
if base.lower().endswith(f".{ext.lower()}"):
base = base[: -(len(ext) + 1)]
return f"{base}.{ext}"
def abs_path(download_root: str, rel: str) -> Path:
return Path(download_root) / rel
def abs_path(files_root: str, rel: str) -> Path:
return Path(files_root) / rel
def safe_abs_path(download_root: str, rel: str) -> Path | None:
"""The resolved absolute path for `rel`, but ONLY if it stays inside DOWNLOAD_ROOT and exists on
disk — otherwise None. Centralizes the path-traversal + existence guard every file-serving endpoint
shares, so the containment check lives in exactly one place."""
root = Path(download_root).resolve()
path = abs_path(download_root, rel).resolve()
def safe_abs_path(files_root: str, rel: str | None) -> Path | None:
"""The resolved absolute path for `rel`, but ONLY if it stays inside FILES_ROOT and exists on
disk — otherwise None (no `rel` at all included: an asset without a file has nothing to serve).
Centralizes the path-traversal + existence guard every file-serving endpoint shares, so the
containment check lives in exactly one place."""
if not rel:
return None
root = Path(files_root).resolve()
path = abs_path(files_root, rel).resolve()
if root not in path.parents or not path.exists():
return None
return path
def place_file(staging_file: Path, download_root: str, rel: str) -> None:
"""Move a completed staging file into its final location under DOWNLOAD_ROOT."""
dest = abs_path(download_root, rel)
def place_file(staging_file: Path, files_root: str, rel: str) -> None:
"""Move a completed staging file into its final location under FILES_ROOT."""
dest = abs_path(files_root, rel)
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(staging_file), str(dest))
@@ -301,9 +360,9 @@ def _sidecar(media: Path, ext: str) -> Path:
return media.with_suffix(ext)
def write_sidecars(download_root: str, rel: str, meta: MediaMeta, thumb_src: Path | None) -> bool:
def write_sidecars(files_root: str, rel: str, meta: MediaMeta, thumb_src: Path | None) -> bool:
"""Write `<basename>.nfo` + `<basename>.jpg` next to the media file. Returns True if written."""
media = abs_path(download_root, rel)
media = abs_path(files_root, rel)
try:
_sidecar(media, ".nfo").write_text(_nfo_xml(meta), encoding="utf-8")
if thumb_src and thumb_src.exists():
@@ -319,9 +378,9 @@ def write_sidecars(download_root: str, rel: str, meta: MediaMeta, thumb_src: Pat
return False
def delete_asset_files(download_root: str, rel: str) -> None:
def delete_asset_files(files_root: str, rel: str) -> None:
"""Remove the media file and its sidecars; prune now-empty parent dirs (best effort)."""
media = abs_path(download_root, rel)
media = abs_path(files_root, rel)
for p in (media, _sidecar(media, ".nfo"), _sidecar(media, ".jpg")):
try:
p.unlink()
@@ -335,7 +394,7 @@ def delete_asset_files(download_root: str, rel: str) -> None:
extra=logs.event("download.asset.file_unlink_failed", reason=type(exc).__name__),
)
# Prune empty Season/Channel dirs up to (but not including) the root.
root = Path(download_root).resolve()
root = Path(files_root).resolve()
parent = media.parent
while parent != root and root in parent.parents:
try:
+37 -6
View File
@@ -31,9 +31,11 @@ from pathlib import Path
from sqlalchemy.orm import Session
from app import audit
from app.audit import AuditAction
from app.config import settings
from app.downloads import edit as editmod
from app.downloads import mediakind, playable, probe, quota, service, storage
from app.downloads import folders, mediakind, playable, probe, quota, service, storage
from app.downloads.status import JobKind, JobStatus
from app.models import DownloadJob, MediaAsset, display_text
@@ -53,14 +55,26 @@ def ingest_upload(
src_path: Path,
size: int,
display_name: str | None = None,
sha256: str | None = None,
folder_id: int | None = None,
via: str | None = None,
) -> DownloadJob:
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return its job —
`done`, or `queued` for the worker's phone-safe conversion (see the module docstring). `src_path`
is MOVED into place on success (the caller stages it to a temp file first).
is MOVED into place on success (the caller stages it to a temp file first). `sha256` is the
hash of those bytes, when the caller took it on the way in; `folder_id` the folder it asked for
(`folders.upload_target`), else its Uploads folder; `via` what the audit row names as the doer
when it was not the user themselves (an API token).
Raises folders.FolderError when that folder is not one an upload may land in (or is gone).
Raises quota.QuotaExceeded (holdings/footprint cap) — checked BEFORE any disk write, so a
rejected upload leaves nothing behind and the caller just cleans up its temp file."""
quota.check_upload(db, user_id, size) # the file's size is known → charge it up front
if folder_id is not None:
# Refused before any work; held only once the job is about to name it (below), not through
# the probe and the poster, which a rename or delete of the folder would wait out.
folders.upload_target(db, user_id, folder_id, hold=False)
data = probe.probe_json(src_path)
vcodec, acodec, width, height, duration = probe.facts(data)
@@ -81,18 +95,21 @@ def ingest_upload(
db.flush() # need asset.id for the on-disk path
rel = storage.upload_rel_path(user_id, asset.id, filename)
storage.place_file(src_path, settings.download_root, rel)
storage.place_file(src_path, settings.files_root, rel)
# From here the file is on disk under `.uploads/`. If the transaction fails before it commits, the
# asset/job rows roll back and nothing names the placed file: the per-user delete path keys off
# asset.rel_path (which is gone), and the GC's `.uploads` sweep reaches it only an hour later. So
# roll back the on-disk side too — unless the commit itself raised after it landed (see below).
asset_id = asset.id
try:
final = storage.abs_path(settings.download_root, rel)
final = storage.abs_path(settings.files_root, rel)
actual_size = final.stat().st_size if final.exists() else size
asset.status = "pending" if convert else "ready"
asset.rel_path = rel
# The hash names the bytes that arrived: a conversion replaces them, so it is kept only
# when the file stays as it came.
asset.sha256 = None if convert else sha256
asset.size_bytes = actual_size
asset.container = ext or None
asset.vcodec = vcodec
@@ -108,7 +125,7 @@ def ingest_upload(
# A frame poster gives the library row a thumbnail + the share link an og:image. Audio has
# no frame and a generic file no player, so neither gets one. Best-effort (ffmpeg decodes
# server side even for a codec the browser can't play).
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
job = DownloadJob(
user_id=user_id,
@@ -123,7 +140,21 @@ def ingest_upload(
queue_pos=service.next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
if folder_id is not None:
job.folder_id = folders.upload_target(db, user_id, folder_id).id
if not convert:
folders.stamp_done(db, job)
service.incr_ref_count(db, asset.id)
db.flush()
audit.record(
db,
user_id,
AuditAction.FILE_UPLOAD,
target_type="file",
target_id=job.id,
summary=via,
)
db.commit()
except Exception:
db.rollback()
@@ -139,7 +170,7 @@ def ingest_upload(
# return would skip that orphan. unlink is missing_ok, so the poster path is a harmless no-op
# when none was generated (audio/file, or a video whose poster never started).
for leaked in (rel, str(Path(rel).with_suffix(".jpg"))):
storage.abs_path(settings.download_root, leaked).unlink(missing_ok=True)
storage.abs_path(settings.files_root, leaked).unlink(missing_ok=True)
raise
db.refresh(job)
return job
+3 -1
View File
@@ -690,7 +690,9 @@ _POLLED_ROUTES = frozenset(
"/api/me/notifications/unread_count",
"/api/downloads",
"/api/downloads/index",
"/api/downloads/usage",
"/api/files",
"/api/files/folders",
"/api/files/usage",
"/api/sync/my-status",
"/api/admin/scheduler",
"/api/admin/downloads",
+24 -1
View File
@@ -28,6 +28,8 @@ log = logs.get_logger(__name__)
from app import bodylimit, collation, logsink, state, sysconfig # noqa: E402 — logging is configured first
from app.config import settings # noqa: E402 — logging is configured first
from app.db import SessionLocal, engine # noqa: E402 — logging is configured first
from app.downloads import service as download_service # noqa: E402 — logging is configured first
from app.downloads import storage # noqa: E402 — logging is configured first
from app.plex import stream as plex_stream # noqa: E402 — logging is configured first
from app.plex import subtitles as plex_subtitles # noqa: E402 — logging is configured first
from app.routes import ( # noqa: E402 — logging is configured first
@@ -42,6 +44,7 @@ from app.routes import ( # noqa: E402 — logging is configured first
discover as discover_routes,
downloads,
feed,
files as files_routes,
health,
me,
notifications,
@@ -55,6 +58,7 @@ from app.routes import ( # noqa: E402 — logging is configured first
setup as setup_routes,
sync,
tags,
tokens as tokens_routes,
version,
)
from app.scheduler import shutdown_scheduler, start_scheduler # noqa: E402 — logging is configured first
@@ -63,6 +67,10 @@ from app.scheduler import shutdown_scheduler, start_scheduler # noqa: E402 —
@asynccontextmanager
async def lifespan(app: FastAPI):
log.info("Siftlode starting up")
problem = storage.files_root_problem(settings.files_root)
if problem:
# The rest of the app still serves; the files library cannot (the worker refuses to start).
log.critical("The files library is not reachable: %s. See docs/self-hosting.md.", problem)
# The stored copy of the log (log_events). The entrypoint has already migrated the schema.
logsink.start()
# First-run: if the instance isn't configured yet, mint a fresh one-time setup token and print
@@ -78,6 +86,18 @@ async def lifespan(app: FastAPI):
# With the traceback: this catch spans a connection, a query and a coercion at boot,
# and `%s` of the exception does not say which of them raised.
log.exception("could not read the stored log level; staying at %s", current_level())
if not problem:
try:
left = download_service.library_left_behind(db, settings.files_root)
except Exception: # a database not reachable yet: the check is a hint, never a gate
log.exception("could not check the files library against its database")
else:
if left:
log.critical(
"The files library is empty but the database names files: the volume "
"mounted at %s is not the one that holds them. See docs/self-hosting.md.",
settings.files_root,
)
if not state.is_configured(db):
token = state.rotate_setup_token(db)
url = f"{settings.app_base}/setup?token={token}"
@@ -267,7 +287,8 @@ async def static_cache_headers(request, call_next):
# Baseline security response headers (SEC-1). A minimal, safe set that needs no per-route tuning:
# nosniff stops MIME-confusion, the Referrer-Policy keeps the (private) app URL from leaking to
# external sites the user follows, `frame-ancestors 'none'` blocks click-jacking by refusing to be
# iframed anywhere (we never embed our own pages; we embed OTHERS, which this does not affect), and
# iframed anywhere (we embed OTHERS, which this does not affect; the one own answer the app frames, a
# library file's preview, sets its own `frame-ancestors 'self'` — these are defaults, `setdefault`), and
# Permissions-Policy turns off powerful features the app never uses. `X-Frame-Options: DENY` duplicates
# the frame-ancestors intent for legacy engines that never implemented CSP frame-ancestors (belt and
# suspenders — modern browsers use the CSP directive and ignore the header). A full Content-Security-Policy
@@ -316,6 +337,8 @@ app.include_router(saved_views.router)
app.include_router(plex_routes.router)
app.include_router(downloads.router)
app.include_router(downloads.admin_router)
app.include_router(files_routes.router)
app.include_router(tokens_routes.router)
app.include_router(public_routes.router)
app.include_router(admin.router)
app.include_router(config_routes.router)
+153 -20
View File
@@ -1,4 +1,4 @@
from datetime import date, datetime
from datetime import date, datetime, timezone
from sqlalchemy import (
BigInteger,
@@ -18,6 +18,7 @@ from sqlalchemy import (
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
@@ -38,7 +39,7 @@ def display_text(raw: str | None) -> str | None:
blank -> None.
One definition because there are SIX writers and they had drifted: `POST /api/downloads`,
`POST /api/downloads/edit`, `POST /api/downloads/upload`, the metadata PATCH's two fields, and
`POST /api/downloads/edit`, `POST /api/files/upload`, the library PATCH's two fields, and
the Google sign-in callback. Only the PATCH truncated. Truncating is the right answer rather
than refusing: the name is a label on work the user has already done — a file already
transferred, an account already authenticated — and losing its tail is not worth failing the
@@ -826,12 +827,13 @@ class MediaAsset(Base, TimestampMixin):
links to this ready asset instantly — no re-download, no cost. Only the SOURCE download is
shared; per-user edited derivatives (phase 2) are never cached here.
`rel_path` is relative to `settings.download_root` (the mount), so the physical filename is
`rel_path` is relative to `settings.files_root` (the mount), so the physical filename is
system-decided and bound to the source id — never renamed (the user's custom name is display
metadata on `download_job`, applied only when downloading to their own machine). The naming
fields (`title`/`uploader`/`upload_date`) are denormalized so the Plex-tree path can be built
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here)
plus `last_access_at`/`expires_at` drive retention + LRU eviction in the GC job."""
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here) frees
the file with its last holder, and `last_access_at` orders the GC's LRU eviction. How long a file
stays is not the asset's: each holder's job carries its own `expires_at` (its library folder's)."""
__tablename__ = "media_assets"
__table_args__ = (
@@ -847,10 +849,10 @@ class MediaAsset(Base, TimestampMixin):
status: Mapped[str] = mapped_column(
String(16), default="pending", server_default="pending", index=True
) # "pending" | "ready" | "error"
rel_path: Mapped[str | None] = mapped_column(Text) # relative to download_root
rel_path: Mapped[str | None] = mapped_column(Text) # relative to files_root
storyboard_path: Mapped[str | None] = mapped_column(Text) # phase-2 filmstrip mosaic
# A locally-generated poster frame (ffmpeg), set only when the source had NO thumbnail so the
# card / watch page / link preview can still show an image. Relative to download_root.
# card / watch page / link preview can still show an image. Relative to files_root.
poster_path: Mapped[str | None] = mapped_column(Text)
size_bytes: Mapped[int | None] = mapped_column(BigInteger)
duration_s: Mapped[int | None] = mapped_column(Integer)
@@ -866,8 +868,6 @@ class MediaAsset(Base, TimestampMixin):
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Set once the GC has warned holders of the upcoming expiry, so it doesn't warn every run.
gc_notified: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Denormalized naming/metadata for the Plex tree + ad-hoc display.
title: Mapped[str | None] = mapped_column(Text)
uploader: Mapped[str | None] = mapped_column(String(255))
@@ -877,13 +877,21 @@ class MediaAsset(Base, TimestampMixin):
upload_date: Mapped[date | None] = mapped_column(Date)
thumbnail_url: Mapped[str | None] = mapped_column(Text)
# yt-dlp's canonical page URL for the source (webpage_url) — the clean "downloaded from"
# reference shown on the Downloads page. Null until the worker extracts it (older/queued rows
# reference shown in the Files library. Null until the worker extracts it (older/queued rows
# fall back to a URL derived from source_kind+source_ref).
source_webpage_url: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
ref_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
last_access_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The file's SHA-256 (hex), so a machine fetching it can check what it got. Hashed while an upload
# streams in, otherwise on the first ask (`routes.files`); NULL = not hashed yet.
sha256: Mapped[str | None] = mapped_column(String(64))
@validates("rel_path")
def _forget_hash(self, _key: str, rel_path: str | None) -> str | None:
# A hash names the bytes at `rel_path`: a new file there (a conversion, a retry) is not them.
self.sha256 = None
return rel_path
class DownloadProfile(Base, TimestampMixin):
@@ -907,6 +915,92 @@ class DownloadProfile(Base, TimestampMixin):
sort_order: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class FileFolder(Base, TimestampMixin):
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
__tablename__ = "file_folders"
__table_args__ = (
UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
Index(
"uq_file_folder_own_name",
"user_id",
func.lower(text("name")),
unique=True,
postgresql_where=text("name IS NOT NULL"),
),
# One Hub folder per user.
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
kind: Mapped[str] = mapped_column(String(16)) # folders.FolderKind
# "site:" + a host name (DNS caps one at 253 characters).
source_key: Mapped[str | None] = mapped_column(String(260))
name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
retention_days: Mapped[int | None] = mapped_column(Integer)
class ApiToken(Base, TimestampMixin):
"""A user's token for a machine to reach their Files library with (`app.apitokens`): sent as
`Authorization: Bearer sfl_…`, honoured on `/api/files` only, and only in the folders its
grants name. Only the SHA-256 of the token is kept — it is shown once, when it is made — plus
its first characters (`prefix`), so the user can tell their tokens apart."""
__tablename__ = "api_tokens"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
name: Mapped[str] = mapped_column(String(80))
prefix: Mapped[str] = mapped_column(String(16))
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) # NULL = never
last_used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The one notice its user gets before it runs out (`apitokens.warn_expiring`).
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class ApiTokenGrant(Base):
"""What one token may do in one folder. A folder with no grant is out of the token's reach."""
__tablename__ = "api_token_grants"
__table_args__ = (
UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
CheckConstraint(
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
token_id: Mapped[int] = mapped_column(ForeignKey("api_tokens.id", ondelete="CASCADE"))
folder_id: Mapped[int] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
can_list: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_read: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_write: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_delete: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
def _ready_on_insert(context) -> datetime | None:
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
done = context.get_current_parameters().get("status") == "done"
return datetime.now(timezone.utc) if done else None
class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
"""A user's request to download one source — the per-user layer over `MediaAsset`.
@@ -925,7 +1019,13 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
point at the parent download the clip was cut from; `edit_spec` is the (trim/crop) recipe."""
__tablename__ = "download_jobs"
__table_args__ = (Index("ix_download_jobs_claim", "status", "queue_pos"),)
__table_args__ = (
Index("ix_download_jobs_claim", "status", "queue_pos"),
# Every way a job becomes done stamps when (`folders.stamp_done`, `finalize_if_running`).
CheckConstraint(
"status <> 'done' OR ready_at IS NOT NULL", name="ck_download_jobs_ready_at"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
@@ -974,6 +1074,27 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
# The host-specific failure-catalog entry on top of the category (`facebook.login_wall`), or NULL.
error_detail: Mapped[str | None] = mapped_column(String(48)) # see downloads.errors.diagnose
queue_pos: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# The Files library: the folder this file sits in (`downloads.folders.place` at creation; an own
# folder is emptied back into the source folders before it is deleted, hence RESTRICT), whether
# the user keeps it past its folder's retention, and when it goes (stamped once done; NULL =
# never). `expiry_warned` marks the one warning the GC sends before it deletes the file.
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="RESTRICT"), index=True
)
# The origin the file came from (the source folder's key), recorded where it is placed: an edit
# traces its origin through its parent, and that parent may be deleted or expire later — the
# file must still know which source folder it may go back to. NULL only for a file placed before
# the column existed outside a source folder; `folders.source_key_for` works those out.
source_key: Mapped[str | None] = mapped_column(String(260))
kept: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# When the file became done — what the library's "Added" means: a download that failed for days
# and then went through is new to its user, so it sorts as new. `created_at` stays when it was
# asked for (the queue's order). Set again when a file is made anew (a retry, a re-download).
ready_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), default=_ready_on_insert
)
class DownloadShare(Base, TimestampMixin):
@@ -1008,21 +1129,33 @@ class DownloadQuota(Base, UpdatedAtMixin):
class DownloadLink(Base, TimestampMixin):
"""A public capability link to one download — the "share by link" surface.
"""A public capability link to one download, or to one of its owner's own folders — the "share
by link" surface.
Anyone holding the unguessable `token` can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style). Optional per-link controls: `expires_at`, `allow_download`
(stream-only vs downloadable), and a `password_hash` (argon2, like a user password). Revoke =
delete the row. Distinct from `DownloadShare`, which grants a *registered* user access via the
in-app "Shared with me" list."""
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
the folder at the time it is opened. The token itself is never stored as written: a link is
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
grants a *registered* user access via the in-app "Shared with me" list."""
__tablename__ = "download_links"
__table_args__ = (
# A link opens exactly one thing: a file or a folder.
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int] = mapped_column(
job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
token: Mapped[str] = mapped_column(String(64), unique=True)
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
password_hash: Mapped[str | None] = mapped_column(Text)
+1 -1
View File
@@ -10,4 +10,4 @@ from pathlib import Path
from app.config import settings
CACHE_DIR = Path(settings.download_root) / ".plex-img-cache"
CACHE_DIR = Path(settings.files_root) / ".plex-img-cache"
+1 -1
View File
@@ -41,7 +41,7 @@ log = logs.get_logger(__name__)
_HLS_ROOT = (
Path(settings.plex_hls_dir)
if settings.plex_hls_dir
else Path(settings.download_root) / ".plex-hls"
else Path(settings.files_root) / ".plex-hls"
)
_SEG_SECONDS = 6
# Cold-start warm-up. A freshly-spawned ffmpeg has produced only the first segment (or two) by the
+8 -2
View File
@@ -6,7 +6,7 @@ and "fetch a row this user owns or 404" idioms have one definition, one status c
one existence-leak policy.
"""
from typing import IO, Any, TypeVar
from typing import Any, Protocol, TypeVar
from fastapi import HTTPException, Request
from pydantic import BaseModel, ConfigDict, Field
@@ -122,8 +122,14 @@ async def capped_body(request: Request, limit: int) -> bytes:
return bytes(body)
class ByteSink(Protocol):
"""Where `capped_stream_to` writes: a file, or anything that takes the bytes on their way to one."""
def write(self, data: bytes, /) -> object: ...
async def capped_stream_to(
request: Request, out: IO[bytes], limit: int, refusal: HTTPException
request: Request, out: ByteSink, limit: int, refusal: HTTPException
) -> int:
"""`capped_body`'s sibling for a body too large to hold: the same counting rule, written
straight to `out` instead of accumulated, returning the size.
+28 -14
View File
@@ -16,6 +16,7 @@ from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import apitokens
from app import email as email_mod
from app import logs
from app import mailtext
@@ -446,7 +447,7 @@ async def callback(
# SEC-10: drop any password nobody proved the mailbox for, now that Google attests
# ownership (bumps the epoch to revoke any session it minted). Runs before the
# activation below, which is what it reads.
revoke_untrusted_preset_password(user)
revoke_untrusted_preset_password(db, user)
# Google verified the email and is_allowed granted access, so finish activating a
# previously pending password registration (otherwise current_user would bounce it).
user.email_verified = True
@@ -486,7 +487,7 @@ async def callback(
# real owner, signed in through Google right now, sets a new password in Settings.
if canonical_email(email) in env_admin_canonicals():
if user.role != "admin" and user.password_hash is not None:
_revoke_password(user)
_revoke_password(db, user)
user.role = "admin"
elif not user.role:
user.role = "user"
@@ -1118,9 +1119,8 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
# about. The person still confirms their address the normal way.
if kind == "reset":
user.email_verified = True
bump_session_epoch(
user
) # SA4: a reset (often a compromise response) kills all existing sessions.
# SA4: a reset (often a compromise response) kills all existing sessions — and API tokens.
bump_session_epoch(db, user)
# Burn any other outstanding reset tokens for this user.
for row in db.execute(
select(AuthToken).where(
@@ -1143,13 +1143,27 @@ def _remember_epoch(request: Request, user: User) -> None:
request.session["epochs"] = epochs
def bump_session_epoch(user: User) -> None:
def bump_session_epoch(db: Session, user: User) -> None:
"""Invalidate every EXISTING signed cookie for this account by advancing its epoch (SA4). Cookies
minted before the bump carry a now-stale epoch and are rejected by `current_user`."""
user.session_epoch = (user.session_epoch or 0) + 1
minted before the bump carry a now-stale epoch and are rejected by `current_user`. The account's
API tokens go with them (`apitokens.revoke_all`): a token minted from a stolen session must not
outlive the answer to that theft. Not committed — the caller commits it with its change.
The user's row is locked before the tokens go, the lock `create_token` takes too: a token made at
the same moment either commits first and is deleted here, or waits and sees the new epoch. FOR NO
KEY UPDATE (`key_share`): the two still exclude each other, but a row that only refers to the user
(a notice, an audit line) is not held up — a token's expiry notice would deadlock with it."""
if user.id is None:
user.session_epoch = (user.session_epoch or 0) + 1
return
epoch = db.execute(
select(User.session_epoch).where(User.id == user.id).with_for_update(key_share=True)
).scalar_one()
user.session_epoch = (epoch or 0) + 1
apitokens.revoke_all(db, user.id)
def revoke_untrusted_preset_password(user: User) -> None:
def revoke_untrusted_preset_password(db: Session, user: User) -> None:
"""SEC-10 (account pre-emption): a password set on an account whose email was NEVER verified is
untrusted — whoever registered the address (to pre-empt the real owner) could have chosen it. When
the real owner attests ownership through Google, drop that password and bump the epoch to revoke any
@@ -1183,15 +1197,15 @@ def revoke_untrusted_preset_password(user: User) -> None:
_send_reset_if_eligible keyed off exactly that, so a revoked account silently got no reset link and
was locked out for good (it could not sign in either). The stamp separates the two cases."""
if not user.email_verified or (not user.is_active and user.password_hash is not None):
_revoke_password(user)
_revoke_password(db, user)
def _revoke_password(user: User) -> None:
def _revoke_password(db: Session, user: User) -> None:
"""Drop the account's password, stamp why (so password-reset still accepts it) and end every
session it signed in."""
user.password_hash = None
user.password_revoked_at = datetime.now(timezone.utc)
bump_session_epoch(user)
bump_session_epoch(db, user)
@router.post("/logout-others")
@@ -1203,7 +1217,7 @@ def logout_others(
in somewhere" / a suspected session compromise."""
# Commit the bump BEFORE re-stamping this cookie, so a failed commit can't strand the current
# session at a newer epoch than the DB (see set_password).
bump_session_epoch(user)
bump_session_epoch(db, user)
db.commit()
_remember_epoch(request, user)
log.info("Logout-others: uid=%s (epoch=%s)", user.id, user.session_epoch)
@@ -1255,7 +1269,7 @@ def set_password(
# SA4: a password change logs out every OTHER session; re-stamp this cookie at the new epoch so
# the person making the change stays signed in here. Commit BEFORE re-stamping so a failed commit
# can't leave the cookie at a newer epoch than the DB (which would wrongly 401 THIS session).
bump_session_epoch(user)
bump_session_epoch(db, user)
db.commit()
_remember_epoch(request, user)
log.info("Password set/changed: uid=%s", user.id)
+217 -498
View File
@@ -2,20 +2,16 @@
All routes are behind `current_user`; admin routes add `admin_user`. Live job state is polled by the
frontend (the worker is a separate process); this module never downloads — it enqueues, manages job
state, serves finished files (range-aware, with the user's custom display name), and shares them.
state, and shares finished files. The library itself — listing, renaming, moving, deleting and
serving a done file, and uploading one — is `routes.files`.
"""
import os
import re
import tempfile
from typing import Literal
from collections.abc import AsyncIterator, Sequence
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from pathlib import Path
from urllib.parse import parse_qs, unquote, urlparse
from urllib.parse import parse_qs, urlparse
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import FileResponse
from pydantic import BaseModel, Field
from sqlalchemy import delete, func, select
@@ -27,18 +23,20 @@ from app.collation import collated
from app import audit, sysconfig
from app.auth import admin_user, current_user
from app.config import settings
from app.db import get_db, release_read_connection
from app.db import get_db
from app.downloads import clip as clipmod
from app.downloads import credentials as dlcreds
from app.downloads import edit as editmod
from app.downloads import folders
from app.downloads import links as linksmod
from app.downloads import quota, service, storage
from app.downloads import upload as uploadmod
from app.downloads import urlguard
from app.downloads.status import NOT_REFETCHABLE, JobStatus
from app.downloads.folders import FolderKind
from app.downloads.status import FINISHED, NOT_REFETCHABLE, JobStatus
from app.models import (
DownloadJob,
DownloadLink,
FileFolder,
DownloadProfile,
DownloadQuota,
DownloadShare,
@@ -49,12 +47,22 @@ from app.models import (
)
from app.routes._common import (
ProgressIn,
capped_stream_to,
get_or_404,
kept_position,
not_found,
owned_or_404,
)
from app.routes.files import (
MAX_LIST,
accessible_job,
assets_for,
own_job_for_update,
positions_for,
quota_detail,
serialize,
serialize_job,
source_keys,
)
from app.security import hash_password
from app.userscope import is_messageable_user, messageable_clauses
from app.utils import iso
@@ -65,11 +73,6 @@ admin_router = APIRouter(prefix="/api/admin/downloads", tags=["admin-downloads"]
_VIDEO_ID = re.compile(r"^[A-Za-z0-9_-]{11}$")
# Cap the per-user job listings so a heavy library can't return an unbounded result set. The
# Download Center renders newest-first; 500 is far beyond any real manual-download history (and
# matches the admin listing's own cap).
_MAX_LIST = 500
# --- source + serialization ----------------------------------------------------------------
@@ -113,115 +116,6 @@ def resolve_source(raw: str) -> tuple[str, str]:
raise refused(400, "invalid_link")
def _serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
ready = asset is not None and asset.status == "ready" and bool(asset.rel_path)
return {
"id": job.id,
"status": job.status,
"progress": job.progress,
"speed_bps": job.speed_bps,
"eta_s": job.eta_s,
"phase": job.phase,
"error": job.error,
"error_code": job.error_code,
"error_detail": job.error_detail,
"queue_pos": job.queue_pos,
"created_at": iso(job.created_at),
"source_kind": job.source_kind,
"source_ref": job.source_ref,
"source_url": service.reference_url(job, asset),
# A locally-generated poster to fall back to when the source had no thumbnail.
"poster_url": f"/api/downloads/{job.id}/poster.jpg"
if (asset is not None and asset.poster_path)
else None,
"profile_id": job.profile_id,
"spec": job.profile_snapshot,
"display_name": job.display_name,
"display_uploader": job.display_uploader,
"display_uploader_url": job.display_uploader_url,
"extra_links": job.extra_links or [],
"job_kind": job.job_kind,
"source_job_id": job.source_job_id,
"edit_spec": job.edit_spec,
"can_download": ready and job.status == JobStatus.DONE,
"expired": job.asset_id is None and job.status == JobStatus.DONE,
"asset": None
if asset is None
else {
"id": asset.id,
"status": asset.status,
"title": asset.title,
"uploader": asset.uploader,
"upload_date": iso(asset.upload_date),
"duration_s": asset.duration_s,
"size_bytes": asset.size_bytes,
"width": asset.width,
"height": asset.height,
"container": asset.container,
# "video" | "audio" | "file" — the library gates play/edit on this (media only); a
# generic uploaded file is download-only. Null while still in flight (set at finalize).
"media_kind": asset.media_kind,
"uploader_url": asset.uploader_url,
"thumbnail_url": asset.thumbnail_url,
"expires_at": iso(asset.expires_at),
},
}
def _assets_for(db: Session, jobs: list[DownloadJob]) -> dict[int, MediaAsset]:
ids = {j.asset_id for j in jobs if j.asset_id}
if not ids:
return {}
rows = db.execute(select(MediaAsset).where(MediaAsset.id.in_(ids))).scalars()
return {a.id: a for a in rows}
def _positions_for(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, int]:
"""`user`'s resume point in each of `jobs` that has one (the library player's `library_states`)."""
ids = [j.id for j in jobs]
if not ids:
return {}
rows = db.execute(
select(LibraryState.job_id, LibraryState.position_seconds).where(
LibraryState.user_id == user.id, LibraryState.job_id.in_(ids)
)
).all()
return {job_id: pos for job_id, pos in rows}
def _serialize_job(db: Session, job: DownloadJob) -> dict:
"""Resolve a job's cache asset (if any) and serialize the pair — the resolve-then-serialize step
every single-job handler shares."""
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
return _serialize(job, asset)
def _own_job_for_update(db: Session, user: User, job_id: int) -> DownloadJob:
"""The ownership check (`owned_or_404`) for a transition that moves its asset's ref count (resume, cancel, delete, redownload)
or writes the status those decide on (pause). The job is re-read under a row lock, so two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`service.finalize_if_running`), so the two can never deadlock."""
while True:
asset_id = owned_or_404(db, DownloadJob, job_id, user, what="download").asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
raise not_found("download")
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
# --- profiles ------------------------------------------------------------------------------
@@ -431,122 +325,8 @@ def enqueue_download(
try:
job = service.enqueue(db, user.id, source_kind, source_ref, spec, profile_id, display_name)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
return _serialize_job(db, job)
# The upload's filename and display name travel as HEADERS, not as multipart parts. That is not a
# style choice: a DECLARED body parameter (`file: UploadFile = File(...)`) is read and spooled by
# FastAPI *before the first dependency runs*, so the auth dependency could not gate it and an anonymous
# caller could make the process write the whole per-file allowance to temp, per request (SEC-20
# bounded the number but could not stop the writing). Reading the body ourselves, from a dependency
# declared AFTER the auth one, is the only shape that closes it — the same one
# `POST /api/debug-sessions/{sid}/screenshots` uses, and `test_body_limit.py` now enforces it for
# every route with a raised ceiling rather than leaving it to a comment like this one.
#
# Headers are latin-1 on the wire, so both are percent-encoded by the client and unquoted here; a
# Hungarian filename would otherwise be unsendable.
UPLOAD_FILENAME_HEADER = "x-upload-filename"
UPLOAD_DISPLAY_NAME_HEADER = "x-upload-display-name"
@dataclass(frozen=True)
class SpooledUpload:
"""What the body dependency hands the handler: the bytes already on disk, plus the two names
that came alongside them."""
path: Path
size: int
user_id: int
filename: str
display_name: str | None
def _header_text(request: Request, name: str) -> str:
"""One percent-encoded header, decoded. A header that is not valid UTF-8 once unquoted is the
caller's doing, not a server fault — it degrades to empty rather than 500ing, and the route's
own fallback names the file."""
raw = request.headers.get(name, "")
try:
return unquote(raw, errors="strict").strip()
except (UnicodeDecodeError, ValueError):
# silent-ok: a mis-encoded header is the caller's own malformed request, and this one is
# cosmetic — it names the file. Refusing the upload over it would fail a 500 MiB transfer on
# a label; logging it would hand any client a free log line per request.
return ""
async def spool_upload(
request: Request,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> AsyncIterator[SpooledUpload]:
"""Stream the request body to a staging temp file, enforcing the per-file cap as it arrives.
`user` is taken but unused: it is the whole point. Dependencies resolve in declaration order, so
naming `current_user` here makes the auth check a precondition of reading the body at all — a
stranger is refused before a byte moves. The `yield` shape owns the temp file's life: ingest
MOVES it into place on success (the unlink is then a no-op), and on any other exit — cap hit,
empty, quota, probe or move error, a client that vanished mid-upload — it is removed."""
cap = sysconfig.get_int(db, "download_upload_max_bytes")
user_id = user.id
# Hand the pooled connection back BEFORE the transfer starts — see `db.release_read_connection`
# for why, and for the precondition it enforces. Everything this dependency still needs is
# already an int or a header in hand: touching an ORM attribute below would reload `user` and
# check a connection straight back out.
release_read_connection(db)
staging = Path(settings.download_root) / ".staging" / "uploads"
staging.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(staging))
tmp_path = Path(tmp)
try:
with os.fdopen(fd, "wb") as out:
size = await capped_stream_to(
request,
out,
cap,
HTTPException(status_code=422, detail={"code": "upload_too_large", "limit": cap}),
)
yield SpooledUpload(
path=tmp_path,
size=size,
user_id=user_id,
filename=_header_text(request, UPLOAD_FILENAME_HEADER) or "upload",
display_name=display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER)),
)
finally:
tmp_path.unlink(missing_ok=True)
@router.post("/upload")
def upload_download(
upload: SpooledUpload = Depends(spool_upload),
db: Session = Depends(get_db),
) -> dict:
"""Ingest a local file into the caller's library. Any file type is accepted; media
(video/audio) gets the full function set, a generic file is download-only. The body IS the file
(see the header note above); `spool_upload` has already authenticated the caller and streamed it
to a temp path under the download volume, so what is left here is the non-yt-dlp ingest which
probes + places it and returns its job — `done`, or `queued` for the worker's phone-safe
conversion (`downloads.upload`).
A sync (threadpool) handler on purpose: the ingest does blocking work — an ffprobe of up to 30s
plus sync DB writes — so it must NOT run on the event loop. The streaming half is async because
reading the body is; keeping them in separate callables is what lets each run where it belongs."""
if upload.size == 0:
raise HTTPException(status_code=400, detail={"code": "upload_empty"})
try:
job = uploadmod.ingest_upload(
db,
upload.user_id,
filename=upload.filename,
src_path=upload.path,
size=upload.size,
display_name=upload.display_name,
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
return _serialize_job(db, job)
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
return serialize_job(db, job)
class EnqueueEditIn(BaseModel):
@@ -564,7 +344,7 @@ def enqueue_edit(
The source may be the user's own download OR one shared with them — either way the resulting
clip is a NEW per-user derivative in the editor's own library (counts against their quota); the
source file is only read, never modified, so editing a shared video is safe."""
source_job = _accessible_job(db, user, body.source_job_id)
source_job = accessible_job(db, user, body.source_job_id)
if source_job.status != JobStatus.DONE or source_job.asset_id is None:
raise refused(409, "edit_unfinished")
spec = body.edit_spec
@@ -572,10 +352,10 @@ def enqueue_edit(
try:
job = service.enqueue_edit(db, user.id, source_job, spec, display_name)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
except service.EditError as e:
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason}) from e
return _serialize_job(db, job)
return serialize_job(db, job)
class ClipCropIn(BaseModel):
@@ -617,57 +397,49 @@ def enqueue_clip(
detail = {"code": "clip", "reason": e.reason, **e.params}
raise HTTPException(status_code=status, detail=detail) from e
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
return _serialize_job(db, job)
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
return serialize_job(db, job)
def _quota_detail(e: quota.QuotaExceeded) -> dict:
"""Structured error the trilingual client localizes (reason key + numbers), instead of a
pre-baked English string with a dot-decimal GB value. See lib/api.ts localizeDetail()."""
return {"code": "quota", "reason": e.reason, "limit": e.limit, "current": e.current}
# How long a finished job stays in the queue: the tray polls it from every page and must see a job
# turn done (its "Done · Open" row and the done notice) — the finished file itself is the library's.
# A done job is timed from when it became done, never from its last write: a rename, a move or the
# GC's expiry warning touches an old file's row, and the tray would announce it as just finished.
QUEUE_RECENT = timedelta(minutes=10)
@router.get("")
def list_downloads(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
"""The queue: what is waiting, running, paused or failed, and what finished a moment ago."""
recent = datetime.now(timezone.utc) - QUEUE_RECENT
jobs = (
db.execute(
select(DownloadJob)
.where(DownloadJob.user_id == user.id)
.where(
DownloadJob.user_id == user.id,
DownloadJob.status.not_in(FINISHED)
| ((DownloadJob.status == JobStatus.DONE) & (DownloadJob.ready_at >= recent))
| ((DownloadJob.status == JobStatus.CANCELED) & (DownloadJob.updated_at >= recent)),
)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(_MAX_LIST)
.limit(MAX_LIST)
)
.scalars()
.all()
)
assets = _assets_for(db, jobs)
positions = _positions_for(db, user, jobs)
assets = assets_for(db, jobs)
positions = positions_for(db, user, jobs)
sources = source_keys(db, jobs)
return [
{
**_serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None),
**serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None),
"position_seconds": positions.get(j.id, 0),
"source_key": sources[j.id],
}
for j in jobs
]
@router.get("/usage")
def my_usage(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
"""The caller's download quota, plus the per-file upload cap.
The cap rides here rather than in `quota.usage` because it is not a quota value — it is a
`sysconfig` knob about one route — and this is the route layer, where composing the two is the
job. The upload dialog needs it to refuse an over-cap file BEFORE sending it: both server-side
caps answer too late to be pleasant (the edge ceiling refuses the request outright, see
`bodylimit.ROUTE_MAX_BODY_BYTES`), and making the user push 600 MB up the wire to be told the
limit is 500 is a bad answer even when it is a correct one. It is a convenience, not the
boundary: the caps stay where they are and still refuse a client that ignores this.
"""
return {
**quota.usage(db, user.id),
"upload_max_bytes": sysconfig.get_int(db, "download_upload_max_bytes"),
}
# Priority when a source has several jobs (e.g. a done one plus a fresh queued one).
_INDEX_RANK: dict[str, int] = {
JobStatus.DONE: 4,
@@ -694,106 +466,22 @@ def my_download_index(user: User = Depends(current_user), db: Session = Depends(
return out
@router.get("/shared")
def shared_with_me(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
jobs = (
db.execute(
select(DownloadJob)
.join(DownloadShare, DownloadShare.job_id == DownloadJob.id)
.where(DownloadShare.to_user_id == user.id)
.order_by(DownloadShare.created_at.desc(), DownloadJob.id.desc())
.limit(_MAX_LIST)
)
.scalars()
.all()
)
assets = _assets_for(db, jobs)
positions = _positions_for(db, user, jobs)
out = []
for j in jobs:
data = _serialize(j, assets.get(j.asset_id))
data["shared"] = True
data["position_seconds"] = positions.get(j.id, 0)
out.append(data)
return out
_MAX_EXTRA_LINKS = 8
def _clean_url(raw) -> str | None:
"""Accept only a plausible http(s) URL (these get rendered as clickable links + embedded in a
public watch page); anything else is dropped rather than trusted."""
if not isinstance(raw, str):
return None
url = raw.strip()
if not url:
return None
if not url.startswith(("http://", "https://")):
return None
# The third site in this module handing user input to `urlparse`, which raises on a malformed
# authority. This one already has a "not a plausible URL" answer, so the typo belongs in it.
try:
parsed = urlparse(url)
# silent-ok: a typo in a link someone pasted. None is this function's "not a plausible URL"
# answer, the caller already has one, and any line here would carry the URL itself.
except ValueError:
return None
if not parsed.netloc:
return None
return url[:2048]
class UpdateDownloadMetaIn(BaseModel):
# PATCH display-only overrides; presence (model_fields_set) decides what's touched. A field
# set to null/"" clears the override (falls back to the on-disk system value).
display_name: str | None = None
display_uploader: str | None = None
display_uploader_url: str | None = None
extra_links: list | None = None
@router.patch("/{job_id}")
def update_download_meta(
job_id: int,
body: UpdateDownloadMetaIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Update a download's display metadata: title, channel name + link, and extra reference URLs.
All are display-only overrides — the on-disk file keeps its system-decided, id-bound name."""
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
sent = body.model_fields_set
if "display_name" in sent:
job.display_name = display_text(body.display_name)
if "display_uploader" in sent:
job.display_uploader = display_text(body.display_uploader)
if "display_uploader_url" in sent:
job.display_uploader_url = _clean_url(body.display_uploader_url)
if "extra_links" in sent:
raw = body.extra_links or []
cleaned = [u for u in (_clean_url(x) for x in raw) if u][:_MAX_EXTRA_LINKS]
job.extra_links = cleaned or None
db.commit()
return _serialize_job(db, job)
@router.post("/{job_id}/pause")
def pause_download(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
job = _own_job_for_update(db, user, job_id)
job = own_job_for_update(db, user, job_id)
service.pause(db, job)
return _serialize_job(db, job)
return serialize_job(db, job)
@router.post("/{job_id}/resume")
def resume_download(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
job = _own_job_for_update(db, user, job_id)
job = own_job_for_update(db, user, job_id)
service.resume(db, job)
return _serialize_job(db, job)
return serialize_job(db, job)
@router.post("/{job_id}/redownload")
@@ -812,7 +500,7 @@ def redownload_download(
under the current format/naming rules — never silently reuses a shared cached file — and (b)
never deletes a file another job/user still shares: the old asset is only removed if THIS job
was its last holder."""
job = _own_job_for_update(db, user, job_id)
job = own_job_for_update(db, user, job_id)
if job.job_kind in NOT_REFETCHABLE:
# None has a re-fetchable source: an editor clip is derived locally, an upload IS the
# source, an ad-hoc clip is a range of a source the clip itself does not re-fetch.
@@ -820,82 +508,16 @@ def redownload_download(
if job.status in (JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED):
raise refused(409, "still_downloading")
service.redownload(db, job)
return _serialize_job(db, job)
return serialize_job(db, job)
@router.post("/{job_id}/cancel")
def cancel_download(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
job = _own_job_for_update(db, user, job_id)
job = own_job_for_update(db, user, job_id)
service.cancel(db, job)
return _serialize_job(db, job)
@router.delete("/{job_id}")
def delete_download(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
job = _own_job_for_update(db, user, job_id)
service.delete(db, job)
return {"deleted": job_id}
# --- file download (range-aware, custom display name) --------------------------------------
def _accessible_job(db: Session, user: User, job_id: int) -> DownloadJob:
job = db.get(DownloadJob, job_id)
if job is None:
raise not_found("download")
if job.user_id == user.id:
return job
shared = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == user.id
)
).first()
if shared is None:
raise not_found("download")
return job
@router.get("/{job_id}/file")
def download_file(
job_id: int,
request: Request,
user: User = Depends(current_user),
db: Session = Depends(get_db),
):
job = _accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or asset.status != "ready" or not asset.rel_path:
raise refused(409, "download_not_ready")
path = storage.safe_abs_path(settings.download_root, asset.rel_path)
if path is None:
raise not_found("file")
filename = storage.download_filename(
job.display_name or asset.title or asset.source_ref, asset.container, path
)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
# The served Content-Type comes from the ffprobe-verified media_kind + a container allow-list
# (storage.served_media_type), NEVER guessed from the file's own extension. A generic 'file' (and
# any media whose container we don't recognise — e.g. a media/HTML polyglot named "evil.html")
# returns None → a neutral force-download attachment, so uploaded HTML/SVG can't execute inline on
# this origin. Recognised media is served with its real type so the in-app player streams it. Both
# are range-aware (FileResponse → 206).
media_type = storage.served_media_type(asset.media_kind, asset.container)
if media_type is None:
return FileResponse(
path,
filename=filename,
media_type="application/octet-stream",
content_disposition_type="attachment",
)
return FileResponse(path, filename=filename, media_type=media_type)
return serialize_job(db, job)
# --- editor filmstrip (scrub track) --------------------------------------------------------
@@ -910,7 +532,7 @@ def set_library_progress(
) -> dict:
"""Checkpoint the library player's resume point in a download the caller can play (their own, or
one shared with them). A position too early or too close to the end clears it instead."""
_accessible_job(db, user, job_id)
accessible_job(db, user, job_id)
position = kept_position(body)
mine = (LibraryState.user_id == user.id) & (LibraryState.job_id == job_id)
if not position:
@@ -942,11 +564,11 @@ def storyboard_meta(
Best-effort: for a very long/high-res source the sprite may not generate in time — then
`available` is False and the editor shows a plain timeline."""
job = _accessible_job(db, user, job_id)
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or asset.status != "ready" or not asset.rel_path:
raise refused(409, "download_not_ready")
meta = editmod.ensure_storyboard(asset, settings.download_root)
meta = editmod.ensure_storyboard(asset, settings.files_root)
if meta is None:
return {"available": False}
db.commit() # persist storyboard_path if it was just generated
@@ -963,11 +585,11 @@ def storyboard_meta(
@router.get("/{job_id}/poster.jpg")
def poster_image(job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)):
"""The locally-generated poster frame for a download whose source had no thumbnail."""
job = _accessible_job(db, user, job_id)
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or not asset.poster_path:
raise not_found("poster")
path = storage.safe_abs_path(settings.download_root, asset.poster_path)
path = storage.safe_abs_path(settings.files_root, asset.poster_path)
if path is None:
raise not_found("poster")
return FileResponse(path, media_type="image/jpeg")
@@ -977,11 +599,11 @@ def poster_image(job_id: int, user: User = Depends(current_user), db: Session =
def storyboard_image(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
):
job = _accessible_job(db, user, job_id)
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or not asset.storyboard_path:
raise not_found("filmstrip")
path = storage.safe_abs_path(settings.download_root, asset.storyboard_path)
path = storage.safe_abs_path(settings.files_root, asset.storyboard_path)
if path is None:
raise not_found("filmstrip")
return FileResponse(path, media_type="image/jpeg")
@@ -1070,24 +692,6 @@ def unshare_download(
return {"ok": True}
@router.delete("/shared/{job_id}")
def remove_shared_with_me(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Remove a download that was shared WITH this user from their 'Shared with me' list. Deletes
only the recipient's share grant — the owner's job and the physical file are untouched (this is
a per-user dismissal, not a delete)."""
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == user.id
)
).scalar_one_or_none()
if row is not None:
db.delete(row)
db.commit()
return {"ok": True, "removed": job_id}
# --- recipients (for the internal "share with a user" picker) ------------------------------
@@ -1116,8 +720,12 @@ def _own_link(db: Session, user: User, link_id: int) -> DownloadLink:
link = db.get(DownloadLink, link_id)
if link is None:
raise not_found("link")
job = db.get(DownloadJob, link.job_id)
if job is None or job.user_id != user.id:
target = (
db.get(DownloadJob, link.job_id)
if link.job_id is not None
else db.get(FileFolder, link.folder_id)
)
if target is None or target.user_id != user.id:
raise not_found("link")
return link
@@ -1130,21 +738,17 @@ def _link_expiry(days: int | None) -> datetime | None:
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
def _links_of(db: Session, *where) -> list[dict]:
rows = db.execute(select(DownloadLink).where(*where).order_by(DownloadLink.id.desc()))
return [linksmod.owner_view(row) for row in rows.scalars()]
@router.get("/{job_id}/links")
def list_links(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
owned_or_404(db, DownloadJob, job_id, user, what="download")
rows = (
db.execute(
select(DownloadLink)
.where(DownloadLink.job_id == job_id)
.order_by(DownloadLink.id.desc())
)
.scalars()
.all()
)
return [linksmod.owner_view(row) for row in rows]
return _links_of(db, DownloadLink.job_id == job_id)
class CreateLinkIn(BaseModel):
@@ -1158,6 +762,34 @@ class CreateLinkIn(BaseModel):
expires_days: int | None = None
def _new_link(
db: Session, user: User, body: CreateLinkIn, **target: int
) -> tuple[DownloadLink, str]:
"""Make a link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and
the token it is opened by — which is in hand only now."""
pw = body.password.strip()
token = linksmod.new_token()
link = DownloadLink(
**target,
created_by=user.id,
allow_download=body.allow_download,
password_hash=hash_password(pw) if pw else None,
expires_at=_link_expiry(body.expires_days),
)
linksmod.seal(link, token)
db.add(link)
db.flush()
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_CREATE,
target_type="download_link",
target_id=link.id,
after={**target, **_link_audit(link)},
)
return link, token
@router.post("/{job_id}/links")
def create_link(
job_id: int,
@@ -1165,22 +797,73 @@ def create_link(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
job = _shareable_job(db, user, job_id)
if job.status != JobStatus.DONE or job.asset_id is None:
raise refused(409, "share_unfinished")
pw = body.password.strip()
link = DownloadLink(
job_id=job_id,
token=linksmod.new_token(),
created_by=user.id,
allow_download=body.allow_download,
password_hash=hash_password(pw) if pw else None,
expires_at=_link_expiry(body.expires_days),
)
db.add(link)
link, token = _new_link(db, user, body, job_id=job_id)
db.commit()
db.refresh(link)
return linksmod.owner_view(link)
return linksmod.owner_view(link, token=token)
def _shareable_job(db: Session, user: User, job_id: int) -> DownloadJob:
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
rather than failing the link's foreign key — or, done first, leaves a 404."""
owned_or_404(db, DownloadJob, job_id, user, what="download")
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update(read=True)
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
raise not_found("download")
return job
def _shareable_folder(db: Session, user: User, folder_id: int) -> FileFolder:
"""One of the user's own folders. A source folder lets its files go on its own clock, and the Hub
is the user's channel to a machine of theirs: neither is a shelf to hand out. Held (FOR SHARE)
for the rest of the transaction, so a delete of it at the same moment waits and takes the new
link with it, rather than failing the link's foreign key."""
owned_or_404(db, FileFolder, folder_id, user, what="folder")
try:
folder = folders.own_folder(db, user.id, folder_id)
except folders.FolderError as e:
raise refused(409, e.reason) from e
if folder.kind != FolderKind.OWN:
raise refused(409, "share_folder_kind")
return folder
@router.get("/folders/{folder_id}/links")
def list_folder_links(
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
owned_or_404(db, FileFolder, folder_id, user, what="folder")
return _links_of(db, DownloadLink.folder_id == folder_id)
@router.post("/folders/{folder_id}/links")
def create_folder_link(
folder_id: int,
body: CreateLinkIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
_shareable_folder(db, user, folder_id)
link, token = _new_link(db, user, body, folder_id=folder_id)
db.commit()
return linksmod.owner_view(link, token=token)
def _link_audit(link: DownloadLink) -> dict:
"""A link's settings as the audit records them: whether it has a password, never which."""
return {
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": iso(link.expires_at),
}
class UpdateLinkIn(BaseModel):
@@ -1197,6 +880,7 @@ def update_link(
db: Session = Depends(get_db),
) -> dict:
link = _own_link(db, user, link_id)
before = _link_audit(link)
sent = body.model_fields_set
if "allow_download" in sent:
link.allow_download = bool(body.allow_download)
@@ -1208,6 +892,17 @@ def update_link(
# Bump the version so any grant issued under the previous password stops working now, rather
# than living out its 6h TTL (setting/clearing/changing the password all count as a rotation).
link.password_version += 1
after = _link_audit(link)
if after != before or "password" in sent:
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
target_type="download_link",
target_id=link_id,
before=before,
after=after,
)
db.commit()
return linksmod.owner_view(link)
@@ -1218,8 +913,16 @@ def revoke_link(
) -> dict:
link = _own_link(db, user, link_id)
db.delete(link)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
target_type="download_link",
target_id=link_id,
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
)
db.commit()
return {"deleted": link_id}
return {"ok": True, "deleted": link_id}
# --- admin ---------------------------------------------------------------------------------
@@ -1233,19 +936,20 @@ def admin_list_downloads(
db.execute(
select(DownloadJob)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(_MAX_LIST)
.limit(MAX_LIST)
)
.scalars()
.all()
)
assets = _assets_for(db, jobs)
assets = assets_for(db, jobs)
emails = {
u.id: u.email
for u in db.execute(select(User).where(User.id.in_({j.user_id for j in jobs}))).scalars()
}
out = []
for j in jobs:
data = _serialize(j, assets.get(j.asset_id))
data = serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None)
data["user_id"] = j.user_id
data["user_email"] = emails.get(j.user_id)
out.append(data)
return out
@@ -1259,24 +963,39 @@ def admin_storage(admin: User = Depends(admin_user), db: Session = Depends(get_d
select(func.count(), func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).one()
total_assets = db.execute(select(func.count()).select_from(MediaAsset)).scalar()
per_user = db.execute(
select(DownloadJob.user_id, func.count(func.distinct(DownloadJob.asset_id)))
.where(DownloadJob.asset_id.is_not(None))
.group_by(DownloadJob.user_id)
).all()
emails = {
u.id: u.email
for u in db.execute(select(User).where(User.id.in_([uid for uid, _ in per_user]))).scalars()
# EVERY user, not only those holding a file: a quota is set before anyone downloads.
emails: dict[int, str] = {
uid: email
for uid, email in db.execute(select(User.id, User.email).order_by(User.email)).tuples()
}
uids = list(emails)
# What each one's Files lists: their finished files.
files: dict[int, int] = {
uid: n
for uid, n in db.execute(
select(DownloadJob.user_id, func.count())
.where(DownloadJob.status == JobStatus.DONE)
.group_by(DownloadJob.user_id)
).tuples()
}
def row(uid: int) -> dict:
lim = quota.resolve(db, uid)
return {
"user_id": uid,
"email": emails[uid],
"footprint_bytes": quota.footprint(db, uid),
"files": files.get(uid, 0),
"max_bytes": lim.max_bytes,
"unlimited": lim.unlimited,
}
return {
"files_on_disk": on_disk[0],
"total_bytes": int(on_disk[1]),
"total_assets": total_assets,
"total_cap_bytes": sysconfig.get_int(db, "download_total_max_bytes"),
"per_user": [
{"user_id": uid, "email": emails.get(uid), "footprint_bytes": quota.footprint(db, uid)}
for uid, _ in per_user
],
"per_user": [row(uid) for uid in uids],
}
+945
View File
@@ -0,0 +1,945 @@
"""The Files library HTTP API — a user's files, the folders they sit in, and the file itself.
A file in the library is a user's DONE job (`DownloadJob`) over its shared `MediaAsset`: the job is
the per-user layer (name, folder, kept, expiry, shares and links all hang off it), the asset the
bytes. Folders are rows (`downloads.folders`), so a rename or a move never touches the disk.
In-flight jobs are not here: they are the download tray's (`routes.downloads`). The helpers above
the routes are shared with that module — serialization, the ownership locks, the upload spool —
because a download and a library file are one row at two moments.
A machine with an API token (`app.apitokens`) may use the routes that take `files_caller`: list the
folders and files, fetch a file and its hash, upload, delete — each only in a folder its token was
granted that right in, and never the files shared with the user. A folder or file out of its reach
answers like a missing one. Everything else here is the signed-in user's alone (`current_user`).
"""
import base64
import hashlib
import os
import tempfile
import threading
from collections.abc import AsyncIterator, Sequence
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import BinaryIO, Literal
from urllib.parse import unquote, urlparse
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import FileResponse, Response
from pydantic import BaseModel, Field
from sqlalchemy import func, or_, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import audit, sysconfig
from app.apitokens import FilesCaller, Right, files_caller
from app.audit import AuditAction
from app.auth import current_user
from app.collation import collated
from app.config import settings
from app.db import get_db, release_read_connection
from app.downloads import folders, preview, quota, service, storage
from app.downloads import upload as uploadmod
from app.downloads.status import JobStatus
from app.mailtext import lang_for_user
from app.models import (
DISPLAY_TEXT_MAX,
DownloadJob,
DownloadLink,
DownloadShare,
FileFolder,
LibraryState,
MediaAsset,
User,
display_text,
)
from app.refusal import refused
from app.routes._common import capped_stream_to, not_found, owned_or_404
from app.utils import iso, like_escape
router = APIRouter(prefix="/api/files", tags=["files"])
# Cap the listing so a heavy library can't return an unbounded result set (the old Download Center's
# own cap; far beyond any real library).
MAX_LIST = 500
# The longest a folder may keep a file, in days — ten years is "forever" in all but name.
_MAX_RETENTION_DAYS = 3650
_HASH_CHUNK = 1 << 20
# How many files the hash route reads at once: each holds a threadpool thread for the whole read.
_HASHERS = 2
_HASHING = threading.BoundedSemaphore(_HASHERS)
# --- shared with routes.downloads -------------------------------------------------------
def serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
ready = asset is not None and asset.status == "ready" and bool(asset.rel_path)
return {
"id": job.id,
"status": job.status,
"progress": job.progress,
"speed_bps": job.speed_bps,
"eta_s": job.eta_s,
"phase": job.phase,
"error": job.error,
"error_code": job.error_code,
"error_detail": job.error_detail,
"queue_pos": job.queue_pos,
"created_at": iso(job.created_at),
"ready_at": iso(job.ready_at),
"source_kind": job.source_kind,
"source_ref": job.source_ref,
"source_url": service.reference_url(job, asset),
# A locally-generated poster to fall back to when the source had no thumbnail.
"poster_url": f"/api/downloads/{job.id}/poster.jpg"
if (asset is not None and asset.poster_path)
else None,
"profile_id": job.profile_id,
"spec": job.profile_snapshot,
"display_name": job.display_name,
"display_uploader": job.display_uploader,
"display_uploader_url": job.display_uploader_url,
"extra_links": job.extra_links or [],
"job_kind": job.job_kind,
"source_job_id": job.source_job_id,
"edit_spec": job.edit_spec,
"can_download": ready and job.status == JobStatus.DONE,
# The Files library: its folder, whether the user keeps it, and when it goes (NULL = never).
"folder_id": job.folder_id,
"kept": job.kept,
"expires_at": iso(job.expires_at),
"asset": None
if asset is None
else {
"id": asset.id,
"status": asset.status,
"title": asset.title,
"uploader": asset.uploader,
"upload_date": iso(asset.upload_date),
"duration_s": asset.duration_s,
"size_bytes": asset.size_bytes,
"width": asset.width,
"height": asset.height,
"container": asset.container,
# "video" | "audio" | "file" — the library gates play/edit on this (media only); a
# generic uploaded file is download-only. Null while still in flight (set at finalize).
"media_kind": asset.media_kind,
"uploader_url": asset.uploader_url,
"thumbnail_url": asset.thumbnail_url,
# "image" | "pdf" | "text" | null — what `/preview` shows (`downloads/preview.py`).
"preview": preview.kind(asset.media_kind, asset.container),
},
}
def assets_for(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, MediaAsset]:
ids = {j.asset_id for j in jobs if j.asset_id}
if not ids:
return {}
rows = db.execute(select(MediaAsset).where(MediaAsset.id.in_(ids))).scalars()
return {a.id: a for a in rows}
def positions_for(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, int]:
"""`user`'s resume point in each of `jobs` that has one (the library player's `library_states`)."""
ids = [j.id for j in jobs]
if not ids:
return {}
rows = db.execute(
select(LibraryState.job_id, LibraryState.position_seconds).where(
LibraryState.user_id == user.id, LibraryState.job_id.in_(ids)
)
).all()
return {job_id: pos for job_id, pos in rows}
def serialize_job(db: Session, job: DownloadJob) -> dict:
"""Resolve a job's cache asset (if any) and serialize the pair — the resolve-then-serialize step
every single-job handler shares."""
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
return serialize(job, asset)
def own_job_for_update(db: Session, user: User, job_id: int) -> DownloadJob:
"""The ownership check (`owned_or_404`) for a transition that moves its asset's ref count (resume, cancel, delete, redownload)
or writes the status those decide on (pause). The job is re-read under a row lock, so two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`service.finalize_if_running`), so the two can never deadlock."""
while True:
asset_id = owned_or_404(db, DownloadJob, job_id, user, what="download").asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
raise not_found("download")
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
def quota_detail(e: quota.QuotaExceeded) -> dict:
"""Structured error the trilingual client localizes (reason key + numbers), instead of a
pre-baked English string with a dot-decimal GB value. See lib/api.ts localizeDetail()."""
return {"code": "quota", "reason": e.reason, "limit": e.limit, "current": e.current}
# The upload's filename and display name travel as HEADERS, not as multipart parts. That is not a
# style choice: a DECLARED body parameter (`file: UploadFile = File(...)`) is read and spooled by
# FastAPI *before the first dependency runs*, so the auth dependency could not gate it and an anonymous
# caller could make the process write the whole per-file allowance to temp, per request (SEC-20
# bounded the number but could not stop the writing). Reading the body ourselves, from a dependency
# declared AFTER the auth one, is the only shape that closes it — the same one
# `POST /api/debug-sessions/{sid}/screenshots` uses, and `test_body_limit.py` now enforces it for
# every route with a raised ceiling rather than leaving it to a comment like this one.
#
# Headers are latin-1 on the wire, so both are percent-encoded by the client and unquoted here; a
# Hungarian filename would otherwise be unsendable.
UPLOAD_FILENAME_HEADER = "x-upload-filename"
UPLOAD_DISPLAY_NAME_HEADER = "x-upload-display-name"
# The folder the file goes in (its id); without it, the user's Uploads folder.
UPLOAD_FOLDER_HEADER = "x-upload-folder"
@dataclass(frozen=True)
class SpooledUpload:
"""What the body dependency hands the handler: the bytes already on disk and their hash, plus
the names and the folder that came alongside them, and who sent them."""
path: Path
size: int
sha256: str
user_id: int
filename: str
display_name: str | None
folder_id: int | None
via: str | None = None
def _header_text(request: Request, name: str) -> str:
"""One percent-encoded header, decoded. A header that is not valid UTF-8 once unquoted is the
caller's doing, not a server fault — it degrades to empty rather than 500ing, and the route's
own fallback names the file."""
raw = request.headers.get(name, "")
try:
return unquote(raw, errors="strict").strip()
except (UnicodeDecodeError, ValueError):
# silent-ok: a mis-encoded header is the caller's own malformed request, and this one is
# cosmetic — it names the file. Refusing the upload over it would fail a 500 MiB transfer on
# a label; logging it would hand any client a free log line per request.
return ""
def _upload_folder(db: Session, caller: FilesCaller, raw: str) -> int | None:
"""The folder id an upload asked for (`UPLOAD_FOLDER_HEADER`), checked; None when it asked for
none — which a token may not: it writes only where it was granted to."""
if not raw.strip():
if caller.grants is not None:
raise refused(400, "api_token_folder_required")
return None
try:
folder_id = int(raw)
except ValueError as e:
raise refused(409, "file_folder_gone") from e
if not caller.may(folder_id, Right.WRITE):
raise not_found("folder")
try:
return folders.upload_target(db, caller.user.id, folder_id, hold=False).id
except folders.FolderError as e:
raise refused(409, e.reason) from e
class _HashingSink:
"""The upload's file, hashing the bytes on their way in: the hash costs no second read."""
def __init__(self, out: BinaryIO) -> None:
self.out = out
self.hash = hashlib.sha256()
def write(self, data: bytes, /) -> int:
self.hash.update(data)
return self.out.write(data)
async def spool_upload(
request: Request,
caller: FilesCaller = Depends(files_caller),
db: Session = Depends(get_db),
) -> AsyncIterator[SpooledUpload]:
"""Stream the request body to a staging temp file, enforcing the per-file cap as it arrives.
`caller` comes first on purpose. Dependencies resolve in declaration order, so naming
`files_caller` here makes the auth check a precondition of reading the body at all — a stranger,
or a token without the right to write in that folder, is refused before a byte moves. The
`yield` shape owns the temp file's life: ingest MOVES it into place on success (the unlink is
then a no-op), and on any other exit — cap hit, empty, quota, probe or move error, a client that
vanished mid-upload — it is removed."""
cap = sysconfig.get_int(db, "download_upload_max_bytes")
user_id = caller.user.id
via = caller.via
# A folder the file may not go in is refused before the transfer, not after it.
folder_id = _upload_folder(db, caller, request.headers.get(UPLOAD_FOLDER_HEADER, ""))
# Hand the pooled connection back BEFORE the transfer starts — see `db.release_read_connection`
# for why, and for the precondition it enforces. Everything this dependency still needs is
# already an int, a string or a header in hand: touching an ORM attribute below (`caller.user`)
# would reload it and check a connection straight back out.
release_read_connection(db)
staging = Path(settings.files_root) / ".staging" / "uploads"
staging.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(staging))
tmp_path = Path(tmp)
try:
with os.fdopen(fd, "wb") as out:
sink = _HashingSink(out)
size = await capped_stream_to(
request,
sink,
cap,
HTTPException(status_code=422, detail={"code": "upload_too_large", "limit": cap}),
)
yield SpooledUpload(
path=tmp_path,
size=size,
sha256=sink.hash.hexdigest(),
user_id=user_id,
filename=_header_text(request, UPLOAD_FILENAME_HEADER) or "upload",
display_name=display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER)),
folder_id=folder_id,
via=via,
)
finally:
tmp_path.unlink(missing_ok=True)
def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob:
"""A file the caller may read: their own, or one shared with them."""
job = db.get(DownloadJob, job_id)
if job is None:
raise not_found("download")
if job.user_id == user.id:
return job
shared = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == user.id
)
).first()
if shared is None:
raise not_found("download")
return job
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob:
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
own in a folder it may read."""
if caller.grants is None:
return accessible_job(db, caller.user, job_id)
job = db.get(DownloadJob, job_id)
if job is None or job.user_id != caller.user.id or not caller.may(job.folder_id, Right.READ):
raise not_found("download")
return job
# --- folders -------------------------------------------------------------------------------
def _serialize_folder(folder: FileFolder, count: int, size: int, linked: bool = False) -> dict:
return {
"id": folder.id,
"kind": folder.kind,
# A source folder's name is the client's to render from its key, in the reader's language.
"source_key": folder.source_key,
"name": folder.name,
"retention_days": folder.retention_days,
"count": count,
"size_bytes": size,
"created_at": iso(folder.created_at),
# A live public link opens it: whatever is moved in is shown to whoever holds the link.
"has_link": linked,
}
def _linked(db: Session, folder_ids: list[int]) -> set[int]:
"""Which of these folders a live public link opens (one not past its expiry)."""
now = datetime.now(timezone.utc)
rows = db.execute(
select(DownloadLink.folder_id).where(
DownloadLink.folder_id.in_(folder_ids),
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
)
).scalars()
return {folder_id for folder_id in rows if folder_id is not None}
@router.get("/folders")
def list_folders(
caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
) -> list[dict]:
"""The caller's folders with how many files each holds and their size (a token: the ones it may
list). A source folder its files have all left stays (count 0): the client decides whether to
show it."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
stmt = select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
).where(FileFolder.user_id == caller.user.id)
listable = caller.folders_with(Right.LIST)
if listable is not None:
stmt = stmt.where(FileFolder.id.in_(listable))
rows = db.execute(
stmt.outerjoin(DownloadJob, done)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.group_by(FileFolder.id)
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
).all()
linked = _linked(db, [f.id for f, _, _ in rows])
return [_serialize_folder(f, count, int(size), f.id in linked) for f, count, size in rows]
class FolderIn(BaseModel):
name: str = Field(min_length=1, max_length=DISPLAY_TEXT_MAX)
def _folder_name(raw: str) -> str:
name = display_text(raw)
if not name:
raise refused(400, "file_folder_name_required")
return name
@router.post("/folders")
def create_folder(
body: FolderIn, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
folder = FileFolder(user_id=user.id, kind=folders.FolderKind.OWN, name=_folder_name(body.name))
try:
with db.begin_nested():
db.add(folder)
db.flush()
except IntegrityError as e: # the unique (user, lower(name)): one folder per name
raise refused(409, "file_folder_name_taken") from e
audit.record(
db, user.id, AuditAction.FILE_FOLDER_CREATE, target_type="file_folder", target_id=folder.id
)
db.commit()
return _serialize_folder(folder, 0, 0)
class UpdateFolderIn(BaseModel):
# Presence (model_fields_set) decides what is touched; retention_days null = keep forever.
name: str | None = Field(default=None, min_length=1, max_length=DISPLAY_TEXT_MAX)
retention_days: int | None = Field(default=None, ge=1, le=_MAX_RETENTION_DAYS)
@router.patch("/folders/{folder_id}")
def update_folder(
folder_id: int,
body: UpdateFolderIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Rename one of the caller's own folders (a source folder's name is its origin's), and/or set
how long any of their folders keeps a file (`folders.set_retention`)."""
folder = owned_or_404(db, FileFolder, folder_id, user, what="folder")
sent = body.model_fields_set
if "name" in sent:
if folder.kind != folders.FolderKind.OWN:
raise refused(409, "file_folder_not_own")
folder.name = _folder_name(body.name or "")
try:
with db.begin_nested():
db.flush()
except IntegrityError as e:
raise refused(409, "file_folder_name_taken") from e
audit.record(
db,
user.id,
AuditAction.FILE_FOLDER_RENAME,
target_type="file_folder",
target_id=folder.id,
)
if "retention_days" in sent:
# The value replaced is the one read under the folder's lock, not the one read above it.
try:
before = folders.set_retention(db, folder, body.retention_days)
except folders.FolderError as e:
raise refused(409, e.reason) from e
if before != body.retention_days:
audit.record(
db,
user.id,
AuditAction.FILE_FOLDER_RETENTION,
target_type="file_folder",
target_id=folder.id,
before={"retention_days": before},
after={"retention_days": body.retention_days},
)
db.commit()
count, size = _folder_totals(db, folder.id)
return _serialize_folder(folder, count, size, folder.id in _linked(db, [folder.id]))
def _folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
count, size = db.execute(
select(func.count(DownloadJob.id), func.coalesce(func.sum(MediaAsset.size_bytes), 0))
.select_from(DownloadJob)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.folder_id == folder_id, DownloadJob.status == JobStatus.DONE)
).one()
return count, int(size)
@router.delete("/folders/{folder_id}")
def delete_folder(
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Delete one of the caller's own folders; its files go back to their source folders."""
folder = owned_or_404(db, FileFolder, folder_id, user, what="folder")
try:
moved = folders.delete_own(db, folder)
except folders.FolderError as e:
raise refused(409, e.reason) from e
audit.record(
db,
user.id,
AuditAction.FILE_FOLDER_DELETE,
target_type="file_folder",
target_id=folder_id,
after={"files_moved": moved},
)
db.commit()
return {"ok": True, "deleted": folder_id, "files_moved": moved}
@router.get("/usage")
def my_usage(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
"""The caller's storage quota, plus the per-file upload cap.
The cap rides here rather than in `quota.usage` because it is not a quota value — it is a
`sysconfig` knob about one route — and this is the route layer, where composing the two is the
job. The upload dialog needs it to refuse an over-cap file BEFORE sending it: both server-side
caps answer too late to be pleasant (the edge ceiling refuses the request outright, see
`bodylimit.ROUTE_MAX_BODY_BYTES`), and making the user push 600 MB up the wire to be told the
limit is 500 is a bad answer even when it is a correct one. It is a convenience, not the
boundary: the caps stay where they are and still refuse a client that ignores this.
"""
return {
**quota.usage(db, user.id),
"upload_max_bytes": sysconfig.get_int(db, "download_upload_max_bytes"),
}
@router.delete("/shared/{job_id}")
def remove_shared_with_me(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Remove a file that was shared WITH this user from their 'Shared with me' list. Deletes
only the recipient's share grant — the owner's job and the physical file are untouched (this is
a per-user dismissal, not a delete)."""
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == user.id
)
).scalar_one_or_none()
if row is not None:
db.delete(row)
db.commit()
return {"ok": True, "removed": job_id}
# --- files ---------------------------------------------------------------------------------
@router.get("")
def list_files(
folder_id: int | None = None,
q: str = Query("", max_length=200),
sort: Literal["added", "name", "size"] = "added",
order: Literal["asc", "desc"] = "desc",
scope: Literal["mine", "shared"] = "mine",
caller: FilesCaller = Depends(files_caller),
db: Session = Depends(get_db),
) -> list[dict]:
"""The caller's done files (`scope=mine`, optionally one folder's), or the ones shared with
them (`scope=shared`); searched by name and channel, sorted by when they were added, by name
(in the reader's alphabet) or by size. A token lists only in the folders it may list, and never
the shared ones."""
user = caller.user
listable = caller.folders_with(Right.LIST)
if listable is not None:
if scope == "shared":
raise refused(403, "api_token_scope")
if folder_id is not None and folder_id not in listable:
raise not_found("folder")
stmt = (
select(DownloadJob)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.status == JobStatus.DONE)
)
shared = scope == "shared"
if shared:
stmt = stmt.join(DownloadShare, DownloadShare.job_id == DownloadJob.id).where(
DownloadShare.to_user_id == user.id
)
else:
stmt = stmt.where(DownloadJob.user_id == user.id)
if folder_id is not None:
stmt = stmt.where(DownloadJob.folder_id == folder_id)
elif listable is not None:
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
term = q.strip()
if term:
like = f"%{like_escape(term)}%"
stmt = stmt.where(
or_(
name.ilike(like),
func.coalesce(DownloadJob.display_uploader, MediaAsset.uploader, "").ilike(like),
)
)
# A file is added to its owner's list when it is done, not when it was asked for (a download that
# failed for days and then went through is new); one shared with the caller, when it was shared.
key = {
"added": DownloadShare.created_at if shared else DownloadJob.ready_at,
"name": collated(name, lang_for_user(user)),
"size": MediaAsset.size_bytes,
}[sort]
first = key.asc() if order == "asc" else key.desc()
tie = DownloadJob.id.asc() if order == "asc" else DownloadJob.id.desc()
jobs = db.execute(stmt.order_by(first.nulls_last(), tie).limit(MAX_LIST)).scalars().all()
assets = assets_for(db, jobs)
positions = positions_for(db, user, jobs)
shared_at = _shared_at(db, user, jobs) if shared else {}
sources = source_keys(db, jobs)
return [
{
**serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None),
"position_seconds": positions.get(j.id, 0),
"shared": shared,
"shared_at": iso(shared_at.get(j.id)),
"source_key": sources[j.id],
}
for j in jobs
]
def source_keys(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, str]:
"""Where each file came from (its source folder's key), for the list's Source column, the
"back to its source folder" move and the download tray's site: a file in a source folder has its folder's key; one moved into
an own folder (or someone else's, shared) is traced back by `folders.source_key_for`."""
ids = {j.folder_id for j in jobs if j.folder_id is not None}
by_id = (
{f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
if ids
else {}
)
out: dict[int, str] = {}
for j in jobs:
folder = by_id.get(j.folder_id) if j.folder_id is not None else None
if folder is not None and folder.kind == folders.FolderKind.SOURCE and folder.source_key:
out[j.id] = folder.source_key
else:
out[j.id] = folders.source_key_for(db, j)
return out
def _shared_at(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, datetime]:
rows = db.execute(
select(DownloadShare.job_id, DownloadShare.created_at).where(
DownloadShare.to_user_id == user.id,
DownloadShare.job_id.in_([j.id for j in jobs]),
)
).all()
return {job_id: at for job_id, at in rows}
_MAX_EXTRA_LINKS = 8
def _clean_url(raw) -> str | None:
"""Accept only a plausible http(s) URL (these get rendered as clickable links + embedded in a
public watch page); anything else is dropped rather than trusted."""
if not isinstance(raw, str):
return None
url = raw.strip()
if not url:
return None
if not url.startswith(("http://", "https://")):
return None
# `urlparse` raises on a malformed authority; this function already has a "not a plausible URL"
# answer, so the typo belongs in it.
try:
parsed = urlparse(url)
# silent-ok: a typo in a link someone pasted. None is this function's "not a plausible URL"
# answer, the caller already has one, and any line here would carry the URL itself.
except ValueError:
return None
if not parsed.netloc:
return None
return url[:2048]
class UpdateFileIn(BaseModel):
# Presence (model_fields_set) decides what is touched; a null/"" text falls back to the system
# value (the title, the channel). All display-only: the file on disk keeps its own name.
name: str | None = None
uploader: str | None = None
uploader_url: str | None = None
extra_links: list | None = None
folder_id: int | None = None
kept: bool | None = None
@router.patch("/{job_id}")
def update_file(
job_id: int,
body: UpdateFileIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Rename a file and its channel, set its reference links (all display-only: the file on disk
keeps its own name), move it to another folder (`folders.move`), or keep it until deleted /
hand it back to its folder's retention."""
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
sent = body.model_fields_set
if "name" in sent:
job.display_name = display_text(body.name)
if "uploader" in sent:
job.display_uploader = display_text(body.uploader)
if "uploader_url" in sent:
job.display_uploader_url = _clean_url(body.uploader_url)
if "extra_links" in sent:
raw = body.extra_links or []
cleaned = [u for u in (_clean_url(x) for x in raw) if u][:_MAX_EXTRA_LINKS]
job.extra_links = cleaned or None
if "folder_id" in sent:
if body.folder_id is None:
raise refused(400, "file_folder_gone")
before = job.folder_id
try:
folders.move(db, job, folders.own_folder(db, user.id, body.folder_id))
except folders.FolderError as e:
raise refused(409, e.reason) from e
if job.folder_id != before:
audit.record(
db,
user.id,
AuditAction.FILE_MOVE,
target_type="file",
target_id=job.id,
before={"folder_id": before},
after={"folder_id": job.folder_id},
)
if "kept" in sent and body.kept is not None and body.kept != job.kept:
folders.set_kept(db, job, body.kept)
action = AuditAction.FILE_KEEP if body.kept else AuditAction.FILE_UNKEEP
audit.record(db, user.id, action, target_type="file", target_id=job.id)
db.commit()
return serialize_job(db, job)
@router.delete("/{job_id}")
def delete_file(
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
) -> dict:
"""Delete a file from the caller's library; the bytes go with their last holder. A token
deletes only in a folder it may delete in — judged on the row as locked, so a file moved out of
that folder meanwhile is out of its reach."""
job = own_job_for_update(db, caller.user, job_id)
if not caller.may(job.folder_id, Right.DELETE):
db.rollback() # lets go of the locks; nothing was written
raise not_found("download")
audit.record(
db,
caller.user.id,
AuditAction.FILE_DELETE,
target_type="file",
target_id=job_id,
summary=caller.via,
)
service.delete(db, job) # commits, the audit row with it
return {"ok": True, "deleted": job_id}
@router.post("/upload")
def upload_file(
upload: SpooledUpload = Depends(spool_upload),
db: Session = Depends(get_db),
) -> dict:
"""Put a local file in the caller's library — in the folder `UPLOAD_FOLDER_HEADER` names, else
their Uploads folder. Any file type; media gets the player, a generic file is download-only.
The body IS the file (see the header note above `UPLOAD_FILENAME_HEADER`); `spool_upload` has
authenticated the caller, checked the folder and streamed it to disk already.
A sync (threadpool) handler on purpose: the ingest does blocking work — an ffprobe of up to 30s
plus sync DB writes — so it must NOT run on the event loop."""
return serialize_job(db, ingest_spooled(db, upload))
def ingest_spooled(db: Session, upload: SpooledUpload) -> DownloadJob:
"""The spooled upload into the library (`upload.ingest_upload`, which commits its own rows),
its refusals answered as HTTP."""
if upload.size == 0:
raise HTTPException(status_code=400, detail={"code": "upload_empty"})
try:
job = uploadmod.ingest_upload(
db,
upload.user_id,
filename=upload.filename,
src_path=upload.path,
size=upload.size,
display_name=upload.display_name,
sha256=upload.sha256,
folder_id=upload.folder_id,
via=upload.via,
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
except folders.FolderError as e: # the folder went while the file was on its way
raise refused(409, e.reason) from e
return job
def _ready_file(db: Session, job: DownloadJob) -> tuple[MediaAsset, Path]:
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or asset.status != "ready" or not asset.rel_path:
raise refused(409, "download_not_ready")
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
if path is None:
raise not_found("file")
return asset, path
def serve_file(db: Session, job: DownloadJob) -> FileResponse:
"""The file of `job` (the caller may read it), range-aware, named by the user's display name.
The served Content-Type comes from the ffprobe-verified media_kind + a container allow-list
(`storage.served_media_type`), NEVER guessed from the file's own extension. A generic 'file' (and
any media whose container we don't recognise — e.g. a media/HTML polyglot named "evil.html")
returns None → a neutral force-download attachment, so uploaded HTML/SVG can't execute inline on
this origin. Recognised media is served with its real type so the in-app player streams it. Both
are range-aware (FileResponse → 206). A file whose hash is known says so (`Repr-Digest`, RFC
9530: the digest of the whole file, whichever range is asked for)."""
asset, path = _ready_file(db, job)
filename = storage.download_filename(
job.display_name or asset.title or asset.source_ref, asset.container, path
)
headers = {}
if asset.sha256:
digest = base64.b64encode(bytes.fromhex(asset.sha256)).decode()
headers["Repr-Digest"] = f"sha-256=:{digest}:"
asset.last_access_at = datetime.now(timezone.utc) # the handler commits it
media_type = storage.served_media_type(asset.media_kind, asset.container)
if media_type is None:
return FileResponse(
path,
filename=filename,
media_type="application/octet-stream",
content_disposition_type="attachment",
headers=headers,
)
return FileResponse(path, filename=filename, media_type=media_type, headers=headers)
@router.get("/{job_id}/file")
def get_file(
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
):
response = serve_file(db, readable_job(db, caller, job_id))
db.commit()
return response
# A preview's own headers. `sandbox` leaves the answer no script and an opaque origin, whatever the
# file turns out to hold; `frame-ancestors 'self'` lets the app's viewer frame a PDF (every other
# answer keeps the app-wide 'none', main.py), with X-Frame-Options saying the same for old engines.
PREVIEW_HEADERS = {
"Content-Security-Policy": "sandbox; frame-ancestors 'self'",
"X-Frame-Options": "SAMEORIGIN",
}
# Set on a text preview cut at `preview.TEXT_PREVIEW_BYTES`, so the viewer can say it shows the start.
TRUNCATED_HEADER = "X-Preview-Truncated"
@router.get("/{job_id}/preview")
def preview_file(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> Response:
"""The file shown in the library's viewer: an image or a PDF inline, a text format as plain text
(its first `TEXT_PREVIEW_BYTES`). The type comes from `downloads/preview.py`'s allow-list, so
an HTML or SVG upload arrives as text, never as a page. Anything else: 415 `file_no_preview`."""
job = accessible_job(db, user, job_id)
asset, path = _ready_file(db, job)
kind = preview.kind(asset.media_kind, asset.container)
if kind is None:
raise refused(415, "file_no_preview")
headers = dict(PREVIEW_HEADERS)
if kind == "text":
with path.open("rb") as f:
head = f.read(preview.TEXT_PREVIEW_BYTES + 1)
if len(head) > preview.TEXT_PREVIEW_BYTES:
head = head[: preview.TEXT_PREVIEW_BYTES]
headers[TRUNCATED_HEADER] = "1"
text = preview.text_utf8(head)
if text is None:
raise refused(415, "file_no_preview")
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
return Response(text, media_type=preview.media_type(kind, asset.container), headers=headers)
filename = storage.download_filename(
job.display_name or asset.title or asset.source_ref, asset.container, path
)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
return FileResponse(
path,
filename=filename,
media_type=preview.media_type(kind, asset.container),
content_disposition_type="inline",
headers=headers,
)
@router.get("/{job_id}/sha256")
def get_sha256(
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
) -> dict:
"""The file's SHA-256 (hex), hashed now if it never was and kept on the asset for every holder.
A sync handler: hashing a large file is blocking disk work, so it runs in the threadpool — and
every other sync route shares that pool, so only `_HASHERS` files are hashed at a time (a caller
past that is told to come back), and none of them holds a database connection while it reads."""
asset, path = _ready_file(db, readable_job(db, caller, job_id))
if asset.sha256:
return {"sha256": asset.sha256}
asset_id = asset.id
if not _HASHING.acquire(blocking=False):
raise refused(429, "file_hash_busy")
try:
release_read_connection(db)
h = hashlib.sha256()
with path.open("rb") as f:
while chunk := f.read(_HASH_CHUNK):
h.update(chunk)
except OSError as e:
raise not_found("file") from e
finally:
_HASHING.release()
digest = h.hexdigest()
db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id, MediaAsset.sha256.is_(None))
.values(sha256=digest)
)
db.commit()
return {"sha256": digest}
+152 -56
View File
@@ -9,7 +9,7 @@ a short-lived signed grant that the media URL carries (a `<video src>` can't sen
from fastapi import APIRouter, Depends, Request
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy import select, update
from sqlalchemy.orm import Session
from app.config import settings
@@ -17,7 +17,7 @@ from app.db import get_db
from app.downloads import links as linksmod
from app.downloads import storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, MediaAsset, User
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset, User
from app.ratelimit import RateLimiter, client_key
from app.security import verify_password
from app.refusal import refused
@@ -38,7 +38,7 @@ _unlock_failures = RateLimiter(max_events=50, window_seconds=300)
def _link_or_404(db: Session, token: str) -> DownloadLink:
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
raise refused(404, "link_gone")
return link
@@ -55,8 +55,7 @@ def _being_made(db: Session, job: DownloadJob) -> bool:
return owner is not None and not owner.is_suspended
def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAsset]:
job = db.get(DownloadJob, link.job_id)
def _ready_job(db: Session, job: DownloadJob | None) -> tuple[DownloadJob, MediaAsset]:
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if (
job is not None
@@ -73,42 +72,120 @@ def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAs
raise refused(404, "video_gone")
# Verify the file is actually on disk here (not just flagged ready), so the watch page never
# loads a player for a missing file — meta and file agree.
path = storage.safe_abs_path(settings.download_root, asset.rel_path)
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
if path is None:
raise refused(404, "video_gone")
return job, asset
def _ready_asset(db: Session, link: DownloadLink) -> MediaAsset:
return _ready_target(db, link)[1]
def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAsset]:
"""A file link's file. A folder link names no one file: its files are asked for by id."""
if link.job_id is None:
raise refused(404, "video_gone")
return _ready_job(db, db.get(DownloadJob, link.job_id))
def _file_url(token: str, grant: str | None = None) -> str:
base = f"/api/public/watch/{token}/file"
return f"{base}?g={grant}" if grant else base
def _link_folder(db: Session, link: DownloadLink) -> FileFolder:
folder = db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
if folder is None:
raise refused(404, "link_gone")
return folder
def _poster_url(token: str, asset, grant: str | None = None) -> str | None:
def _folder_job(db: Session, link: DownloadLink, job_id: int) -> tuple[DownloadJob, MediaAsset]:
folder = _link_folder(db, link)
job = db.execute(
select(DownloadJob).where(DownloadJob.id == job_id, *linksmod.in_folder(folder))
).scalar_one_or_none()
if job is None:
raise refused(404, "video_gone")
return _ready_job(db, job)
def _base(token: str, job_id: int | None) -> str:
"""A file link's file lives under its token; a folder link's files, each under its id."""
return f"/api/public/watch/{token}" + (f"/files/{job_id}" if job_id is not None else "")
def _file_url(token: str, grant: str | None = None, job_id: int | None = None) -> str:
url = f"{_base(token, job_id)}/file"
return f"{url}?g={grant}" if grant else url
def _poster_url(
token: str, asset: MediaAsset, grant: str | None = None, job_id: int | None = None
) -> str | None:
if not asset.poster_path:
return None
base = f"/api/public/watch/{token}/poster.jpg"
return f"{base}?g={grant}" if grant else base
url = f"{_base(token, job_id)}/poster.jpg"
return f"{url}?g={grant}" if grant else url
def _folder_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
"""A folder link's page: the folder's name and each file's own watch metadata
(`linksmod.folder_files`: ready, on disk, newest first)."""
folder = _link_folder(db, link)
files = [
{
"id": job.id,
**linksmod.public_meta(
link,
job,
asset,
_file_url(token, grant, job.id),
_poster_url(token, asset, grant, job.id),
),
}
for job, asset in linksmod.folder_files(db, folder)
]
return {
"needs_password": False,
"kind": "folder",
"name": folder.name,
"allow_download": link.allow_download,
"files": files,
}
def _file_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
job, asset = _ready_target(db, link)
return {
"needs_password": False,
"kind": "file",
**linksmod.public_meta(
link, job, asset, _file_url(token, grant), _poster_url(token, asset, grant)
),
}
def _opened(db: Session, link: DownloadLink, token: str, grant: str | None = None) -> dict:
"""The page an open link shows, counted as one view (the handler commits the count)."""
meta = (
_folder_meta(db, link, token, grant)
if link.folder_id is not None
else _file_meta(db, link, token, grant)
)
# In SQL, not `+= 1` on the loaded row: two opens at once would both write the same count.
db.execute(
update(DownloadLink)
.where(DownloadLink.id == link.id)
.values(view_count=DownloadLink.view_count + 1)
.execution_options(synchronize_session=False)
)
return meta
@router.get("/watch/{token}")
def watch_meta(token: str, db: Session = Depends(get_db)) -> dict:
"""Public metadata for the watch page. For a password link, returns only `needs_password`
until the viewer unlocks it (no title/thumbnail leak)."""
"""Public metadata for the watch page: one file's, or a folder's list. For a password link,
returns only `needs_password` until the viewer unlocks it (no title/thumbnail leak)."""
link = _link_or_404(db, token)
if link.password_hash:
return {"needs_password": True}
job, asset = _ready_target(db, link)
link.view_count += 1
# Which kind it opens, for the password prompt's title — never what is in it.
return {"needs_password": True, "kind": "folder" if link.folder_id is not None else "file"}
meta = _opened(db, link, token)
db.commit()
return {
"needs_password": False,
**linksmod.public_meta(link, job, asset, _file_url(token), _poster_url(token, asset)),
}
return meta
class WatchUnlockIn(BaseModel):
@@ -119,7 +196,7 @@ class WatchUnlockIn(BaseModel):
def watch_unlock(
token: str, body: WatchUnlockIn, request: Request, db: Session = Depends(get_db)
) -> dict:
"""Exchange the link password for a signed grant carried by the media URL."""
"""Exchange the link password for a signed grant carried by the media URLs."""
# The link first: a token naming none must leave no limiter entry behind, or posting made-up tokens
# would grow the limiter without bound.
link = _link_or_404(db, token)
@@ -128,37 +205,28 @@ def watch_unlock(
raise refused(429, "too_many_attempts")
if not link.password_hash:
# Not a password link — nothing to unlock; just hand back the plain meta.
job, asset = _ready_target(db, link)
return {
"ok": True,
"needs_password": False,
**linksmod.public_meta(link, job, asset, _file_url(token), _poster_url(token, asset)),
}
meta = _opened(db, link, token)
db.commit()
return {"ok": True, **meta}
if not verify_password(body.password, link.password_hash):
_unlock_failures.allow(token)
raise refused(403, "wrong_share_password")
job, asset = _ready_target(db, link)
link.view_count += 1
db.commit()
grant = linksmod.make_grant(token, link.password_version)
return {
"ok": True,
"needs_password": False,
**linksmod.public_meta(
link, job, asset, _file_url(token, grant), _poster_url(token, asset, grant)
),
}
meta = _opened(db, link, token, grant)
db.commit()
return {"ok": True, **meta}
@router.get("/watch/{token}/file")
def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
"""Range-aware stream of the shared file. Inline by default (plays in the page); an
`allow_download` link serves it as an attachment. Password links require a valid grant."""
link = _link_or_404(db, token)
def _granted(link: DownloadLink, token: str, g: str | None) -> None:
"""A password link's files want the grant its unlock handed out."""
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
raise refused(403, "share_password_required")
asset = _ready_asset(db, link)
path = storage.safe_abs_path(settings.download_root, asset.rel_path)
def _serve(link: DownloadLink, asset: MediaAsset) -> FileResponse:
"""Range-aware stream of a shared file. Inline by default (plays in the page); an
`allow_download` link serves it as an attachment."""
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
if path is None:
raise refused(404, "video_gone")
@@ -184,17 +252,45 @@ def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
)
def _poster(asset: MediaAsset) -> FileResponse:
if not asset.poster_path:
raise not_found("poster")
path = storage.safe_abs_path(settings.files_root, asset.poster_path)
if path is None:
raise not_found("poster")
return FileResponse(path, media_type="image/jpeg")
@router.get("/watch/{token}/file")
def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
"""A file link's file. Password links require a valid grant."""
link = _link_or_404(db, token)
_granted(link, token, g)
return _serve(link, _ready_target(db, link)[1])
@router.get("/watch/{token}/poster.jpg")
def watch_poster(token: str, g: str | None = None, db: Session = Depends(get_db)):
"""The generated poster frame for a shared thumbnail-less video (used as the page/video poster
and the link-preview image). Password links require a valid grant, like the file."""
link = _link_or_404(db, token)
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
raise refused(403, "share_password_required")
job, asset = _ready_target(db, link)
if not asset.poster_path:
raise not_found("poster")
path = storage.safe_abs_path(settings.download_root, asset.poster_path)
if path is None:
raise not_found("poster")
return FileResponse(path, media_type="image/jpeg")
_granted(link, token, g)
return _poster(_ready_target(db, link)[1])
@router.get("/watch/{token}/files/{job_id}/file")
def watch_folder_file(token: str, job_id: int, g: str | None = None, db: Session = Depends(get_db)):
"""One file of a folder link's folder, while it is in it. Password links require a grant."""
link = _link_or_404(db, token)
_granted(link, token, g)
return _serve(link, _folder_job(db, link, job_id)[1])
@router.get("/watch/{token}/files/{job_id}/poster.jpg")
def watch_folder_poster(
token: str, job_id: int, g: str | None = None, db: Session = Depends(get_db)
):
"""The poster frame of one file of a folder link's folder, under the same terms as the file."""
link = _link_or_404(db, token)
_granted(link, token, g)
return _poster(_folder_job(db, link, job_id)[1])
+185
View File
@@ -0,0 +1,185 @@
"""A user's API tokens for their Files library (`app.apitokens`): list, make, revoke.
Signed-in only — `current_user`, never a token: a token cannot make itself a wider one. A new
token's value is in the answer that made it and nowhere after; only its hash is kept.
The Hub preset is the one machine a user swaps files with: it makes (or reuses) their Hub folder
and gives the token every right in it, so the hub both drops files there and picks up the ones put
there for it.
"""
from datetime import datetime, timedelta, timezone
from typing import Literal
from fastapi import APIRouter, Depends
from pydantic import BaseModel, Field
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import apitokens, audit
from app.apitokens import GRANT_COLUMNS, Right
from app.audit import AuditAction
from app.auth import current_user
from app.db import get_db
from app.downloads import folders
from app.models import ApiToken, ApiTokenGrant, FileFolder, User
from app.refusal import refused
from app.routes._common import owned_or_404
from app.security import hash_token
from app.utils import iso
router = APIRouter(prefix="/api/tokens", tags=["tokens"])
# Far more than one person's machines; a ceiling, so a script gone wrong cannot mint without end.
MAX_TOKENS = 20
_MAX_GRANTS = 50
_MAX_EXPIRY_DAYS = 3650
class GrantIn(BaseModel):
folder_id: int
rights: list[Right] = Field(min_length=1, max_length=len(Right))
class TokenIn(BaseModel):
name: str = Field(min_length=1, max_length=80)
# None = never expires.
expires_days: int | None = Field(default=None, ge=1, le=_MAX_EXPIRY_DAYS)
preset: Literal["hub"] | None = None
grants: list[GrantIn] = Field(default_factory=list, max_length=_MAX_GRANTS)
def _serialize(db: Session, token: ApiToken) -> dict:
rows = db.execute(
select(ApiTokenGrant, FileFolder)
.join(FileFolder, FileFolder.id == ApiTokenGrant.folder_id)
.where(ApiTokenGrant.token_id == token.id)
.order_by(FileFolder.id)
).all()
now = datetime.now(timezone.utc)
return {
"id": token.id,
"name": token.name,
"prefix": token.prefix,
"created_at": iso(token.created_at),
"expires_at": iso(token.expires_at),
"expired": token.expires_at is not None and token.expires_at <= now,
"expiring": apitokens.expiring(token, now),
"last_used_at": iso(token.last_used_at),
"grants": [
{
"folder_id": folder.id,
"kind": folder.kind,
"source_key": folder.source_key,
"name": folder.name,
"rights": sorted(apitokens.grant_rights(grant)),
}
for grant, folder in rows
],
}
@router.get("")
def list_tokens(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
tokens = (
db.execute(select(ApiToken).where(ApiToken.user_id == user.id).order_by(ApiToken.id))
.scalars()
.all()
)
return [_serialize(db, t) for t in tokens]
def _wanted_grants(db: Session, user: User, body: TokenIn) -> dict[int, set[Right]]:
"""Folder id → the rights the new token gets there; every folder checked to be the user's, and
writing only where an upload may land (`folders.upload_target`)."""
if body.preset == "hub":
if body.grants:
raise refused(400, "api_token_preset_grants")
return {folders.hub_folder(db, user.id).id: set(Right)}
if not body.grants:
raise refused(400, "api_token_no_grants")
wanted: dict[int, set[Right]] = {}
for grant in body.grants:
wanted.setdefault(grant.folder_id, set()).update(grant.rights)
for folder_id, rights in wanted.items():
try:
if Right.WRITE in rights:
folders.upload_target(db, user.id, folder_id)
else:
folders.own_folder(db, user.id, folder_id)
except folders.FolderError as e:
raise refused(409, e.reason) from e
return wanted
@router.post("")
def create_token(
body: TokenIn, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Make a token; its value is in this answer only."""
name = body.name.strip()
if not name:
raise refused(400, "api_token_name_required")
# The user's row, locked: two requests at once cannot both slip under the ceiling, and a
# session ended since `current_user` read the epoch (unlocked) cannot mint a token that outlives
# it — `bump_session_epoch` takes this lock before it revokes them all. No key update: a row that
# only refers to the user (a folder delete's audit line) must not wait on it, or the two deadlock.
epoch = db.execute(
select(User.session_epoch).where(User.id == user.id).with_for_update(key_share=True)
).scalar_one()
if (epoch or 0) != (user.session_epoch or 0):
raise refused(401, "not_authenticated")
count = db.execute(
select(func.count(ApiToken.id)).where(ApiToken.user_id == user.id)
).scalar_one()
if count >= MAX_TOKENS:
raise refused(409, "api_token_limit", limit=MAX_TOKENS)
wanted = _wanted_grants(db, user, body)
raw = apitokens.mint()
token = ApiToken(
user_id=user.id,
name=name,
prefix=raw[: apitokens.SHOWN_LEN],
token_hash=hash_token(raw),
expires_at=None
if body.expires_days is None
else datetime.now(timezone.utc) + timedelta(days=body.expires_days),
)
db.add(token)
db.flush()
for folder_id, rights in wanted.items():
db.add(
ApiTokenGrant(
token_id=token.id,
folder_id=folder_id,
**{col: right in rights for right, col in GRANT_COLUMNS.items()},
)
)
db.flush()
audit.record(
db,
user.id,
AuditAction.API_TOKEN_CREATE,
target_type="api_token",
target_id=token.id,
after={
"grants": {str(f): sorted(r) for f, r in wanted.items()},
"expires_at": iso(token.expires_at),
},
)
db.commit()
return {"ok": True, **_serialize(db, token), "token": raw}
@router.delete("/{token_id}")
def revoke_token(
token_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Revoke a token: it stops working at its next request (its grants go with it)."""
token = owned_or_404(db, ApiToken, token_id, user, what="api_token")
db.delete(token)
audit.record(
db, user.id, AuditAction.API_TOKEN_REVOKE, target_type="api_token", target_id=token_id
)
db.commit()
return {"ok": True, "deleted": token_id}
+2 -1
View File
@@ -370,7 +370,8 @@ def _explore_cleanup_job() -> None:
def _download_gc_job() -> None:
# Retention GC for the download center: pre-expiry warnings, TTL deletion, LRU eviction.
# Retention GC for the download center: pre-expiry warnings, TTL deletion, LRU eviction — and
# the week-ahead warning before an API token runs out (`apitokens.warn_expiring`).
# Pure disk/DB work (no YouTube quota).
_job("download_gc", run_download_gc)
+6
View File
@@ -46,6 +46,12 @@ _fernet: Fernet | None = (
)
def can_encrypt() -> bool:
"""Whether a value can be stored encrypted here: TOKEN_ENCRYPTION_KEY is set. Production
refuses to start without it (`config`); a local box may run without one."""
return _fernet is not None
def _require_fernet() -> Fernet:
if _fernet is None:
raise RuntimeError("TOKEN_ENCRYPTION_KEY is not configured")
+8 -2
View File
@@ -37,7 +37,7 @@ from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import credentials, formats, quota, service
from app.downloads import credentials, formats, quota, service, storage
from app.downloads import errors as dlerrors
from app.downloads.jobdb import (
_edit_staging_dir,
@@ -710,12 +710,18 @@ def main() -> None:
# `log_events`, so on the deploy that first adds the table the worker may start before the API
# has migrated — those few seconds of batches fail and say so on stderr; nothing else breaks.
logsink.start()
problem = storage.files_root_problem(settings.files_root)
if problem:
# Refuse, loudly: downloading into a root that is not the volume loses every file with the
# container. The queue waits until the mount is fixed (see docs/self-hosting.md).
log.critical("The files library cannot be written: %s. The worker will not start.", problem)
raise SystemExit(1)
_recover_orphans()
global _target
_target = max(1, settings.download_worker_concurrency) # used only if the DB can't answer
threads: dict[int, threading.Thread] = {}
level = _sync_knobs(threads, None)
log.info("Download worker started (concurrency=%d, root=%s).", _target, settings.download_root)
log.info("Download worker started (concurrency=%d, root=%s).", _target, settings.files_root)
while not _stop.wait(_KNOB_SECONDS):
level = _sync_knobs(threads, level)
log.info("Download worker stopping…")
+4 -1
View File
@@ -16,6 +16,9 @@ services:
POSTGRES_DB: siftlode_test
tmpfs:
- /var/lib/postgresql/data # ephemeral + fast; the schema is rebuilt per run
# One database per pytest-xdist worker, each through the app's own pool (5 + 10 overflow): the default 100
# connections would run out at 8 workers.
command: ["postgres", "-c", "max_connections=300"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U siftlode -d siftlode_test"]
interval: 1s
@@ -37,4 +40,4 @@ services:
- ./:/app
working_dir: /app
# Coverage is on demand, not in the gate: see docs/development.md.
command: ["python", "-m", "pytest"]
command: ["python", "-m", "pytest", "-n", "auto"]
+6 -1
View File
@@ -7,4 +7,9 @@
# not an unknown marker that quietly does nothing.
testpaths = tests
pythonpath = .
addopts = -q -rs --strict-markers
# `-p tests.db_per_worker`: under pytest-xdist (`-n`) every worker gets its own database — DATABASE_URL + `_gw<N>`,
# created on first use — because the `db` fixture's TRUNCATE-per-test isolation assumes one process per database.
# It is a plugin so it runs before conftest imports the app (whose settings read DATABASE_URL once). Serial: no-op.
# The file is a verbatim copy of a shared tool (see its docstring); change the source, not the copy.
addopts = -q -rs --strict-markers -p tests.db_per_worker
db_per_worker_env = DATABASE_URL
+3
View File
@@ -16,3 +16,6 @@ pytest-cov==7.1.0
ruff==0.15.21
httpx2==2.13.0
pip-audit==2.10.1
# pytest-xdist: the DB lane runs on every core (`-n auto` in docker-compose.test.yml); tests/db_per_worker.py gives
# each worker its own database. Same tests, same strictness — only side by side.
pytest-xdist==3.8.0
+15 -3
View File
@@ -27,7 +27,7 @@ from starlette.requests import HTTPConnection
import app.models # register every model on Base.metadata
import app.state as app_state
from app import security
from app import apitokens, security
from app.auth import current_user
from app.config import settings
from app.db import Base, SessionLocal, engine, get_db
@@ -242,8 +242,16 @@ def as_user(db, override_dep, _configured):
raise HTTPException(status_code=401, detail="Not authenticated")
return registry[uid]
def _session_or_token(request: Request) -> User | None:
# The Files routes' session half (`apitokens.session_caller`): a request carrying a token is
# left to the real token check, exactly as in production.
if apitokens.bearer_token(request) is not None:
return None
return _resolver(request)
override_dep(get_db, db.info["request"])
override_dep(current_user, _resolver)
override_dep(apitokens.session_caller, _session_or_token)
def _as(user: User) -> TestClient:
registry[user.id] = user
@@ -254,20 +262,24 @@ def as_user(db, override_dep, _configured):
@pytest.fixture
def upload_headers():
"""Build the headers `POST /api/downloads/upload` takes (the file is the raw body).
"""Build the headers the upload routes take (the file is the raw body).
Shared, because two test modules drive that route and a second copy of this is exactly the drift
that lets the wire shape change in one place only. The body IS the file now; the names ride as
percent-encoded headers (`quote` mirrors the client's `encodeURIComponent`, since headers are
latin-1 and a Hungarian filename would not survive raw)."""
def _headers(filename: str, display_name: str | None = None) -> dict[str, str]:
def _headers(
filename: str, display_name: str | None = None, folder: int | str | None = None
) -> dict[str, str]:
headers = {
"content-type": "application/octet-stream",
"x-upload-filename": quote(filename),
}
if display_name is not None:
headers["x-upload-display-name"] = quote(display_name)
if folder is not None:
headers["x-upload-folder"] = str(folder)
return headers
return _headers
+94
View File
@@ -0,0 +1,94 @@
"""Give every pytest-xdist worker its own database, so a suite that shares one test database can run in parallel.
Generic (source of truth: the memory repo's tools/pytest-parallel/db_per_worker.py); a project carries a verbatim copy
next to its conftest — tests often run in a container that cannot see this repo — and
`tools/pytest-parallel/sync-check.sh <copy>` reports drift. Keep the copy byte-identical; change it here.
Use, as a pytest plugin (it then runs before any conftest, i.e. before the app reads its settings):
pytest.ini: addopts = -p tests.db_per_worker (the copy's import path; `python -m pytest` puts cwd on sys.path)
db_per_worker_env = DATABASE_URL (the environment variable that holds the test database)
or call `apply("<ENV VAR>")` yourself before the app is imported. The variable may hold:
- a Postgres URL (`postgresql://…/name`, `postgresql+psycopg://…`): the worker gets `name_gw<N>`, created from the
base database's server if missing (empty — the project's own schema fixture migrates it, as for one process);
- a SQLite URL (`sqlite:///path.db`) or a plain file path: the worker gets `path-gw<N>.db`.
Without xdist (no PYTEST_XDIST_WORKER) or with the variable unset it changes nothing, so a serial run is unaffected.
"""
from __future__ import annotations
import os
import re
from urllib.parse import urlsplit, urlunsplit
import pytest
_PG = re.compile(r"^postgres(?:ql)?(?:\+\w+)?://", re.I)
def pytest_addoption(parser: pytest.Parser) -> None:
parser.addini(
"db_per_worker_env",
"env var holding the test database (a Postgres URL, SQLite URL or file path)",
)
@pytest.hookimpl(tryfirst=True)
def pytest_load_initial_conftests(early_config: pytest.Config) -> None:
var = early_config.getini("db_per_worker_env")
if var:
apply(str(var))
def apply(var: str) -> str | None:
worker = os.environ.get("PYTEST_XDIST_WORKER")
value = os.environ.get(var)
if not worker or not value:
return value
if _PG.match(value):
new = _postgres(value, worker)
else:
new = _file(value, worker)
os.environ[var] = new
return new
def _postgres(url: str, worker: str) -> str:
parts = urlsplit(url)
base = parts.path.lstrip("/")
name = f"{base}_{worker}"
if not re.fullmatch(r"\w+", name):
raise ValueError(f"db_per_worker: refusing an unusual database name {name!r}")
# The server, through the BASE database (it exists; `postgres` may not be reachable for this role).
plain = urlunsplit(
(re.sub(r"\+\w+$", "", parts.scheme), parts.netloc, parts.path, parts.query, "")
)
_create_if_missing(plain, name)
return urlunsplit((parts.scheme, parts.netloc, f"/{name}", parts.query, parts.fragment))
def _create_if_missing(dsn: str, name: str) -> None:
try:
import psycopg # psycopg 3
conn = psycopg.connect(dsn, autocommit=True)
except ImportError:
import psycopg2 # fallback for projects on psycopg2
conn = psycopg2.connect(dsn)
conn.autocommit = True
try:
cur = conn.cursor()
cur.execute("SELECT 1 FROM pg_database WHERE datname = %s", (name,))
if cur.fetchone() is None:
cur.execute(f'CREATE DATABASE "{name}"') # name checked to be \w+ above
finally:
conn.close()
def _file(value: str, worker: str) -> str:
prefix = ""
m = re.match(r"^(sqlite(?:\+\w+)?:///)(.*)$", value, re.I)
if m:
prefix, value = m.group(1), m.group(2)
root, ext = os.path.splitext(value)
return f"{prefix}{root}-{worker}{ext}"
+648
View File
@@ -0,0 +1,648 @@
"""API tokens for the Files library (`app.apitokens`, `routes.tokens`), DB-lane.
A token is made, listed and revoked by its signed-in user only; its value is in the answer that made
it and nowhere after. With it a machine reaches `/api/files` and nothing else, and there only the
folders its grants name, with the rights they give — every right checked in a folder it has and in
one it has not. A token that is malformed, unknown, expired, revoked, or whose account is suspended
holds nothing, and a client trying tokens is slowed down.
"""
from datetime import datetime, timedelta, timezone
import pytest
from fastapi import Depends, Request
from fastapi.testclient import TestClient
from sqlalchemy import select, update
from app import apitokens, audit
from app.apitokens import files_caller
from app.auth import current_user
from app.downloads import folders, service
from app.downloads import gc as gcmod
from app.downloads import upload as uploadmod
from app.downloads.status import JobStatus
from app.config import settings
from app.db import SessionLocal, get_db
from app.main import app
from app.notifications import create_notification
from app.models import (
ApiToken,
ApiTokenGrant,
AuditLog,
DownloadJob,
FileFolder,
MediaAsset,
Notification,
User,
)
from app.ratelimit import RateLimiter
from app.routes import tokens as tokens_routes
from app.routes.auth import bump_session_epoch
from app.security import hash_token
@pytest.fixture
def _root(tmp_path, monkeypatch):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
return tmp_path
@pytest.fixture
def _no_probe(monkeypatch):
# A file ffprobe cannot read: a generic file, done at once (no conversion).
monkeypatch.setattr(uploadmod.probe, "probe_json", lambda _p: None)
def _own(db, user_id, name):
folder = FileFolder(user_id=user_id, kind=folders.FolderKind.OWN, name=name)
db.add(folder)
db.commit()
return folder
def _file_in(db, root, user_id, folder, name):
"""A done file of `user_id`'s with bytes on disk, sitting in `folder`."""
asset = MediaAsset(
source_kind="upload",
source_ref=name,
format_sig=f"sig-{name}",
status="ready",
rel_path=f"{name}.bin",
size_bytes=4,
ref_count=0,
title=name,
media_kind="file",
container="bin",
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
source_kind="upload",
source_ref=name,
profile_snapshot={},
status=JobStatus.DONE,
)
db.add(job)
db.flush()
folders.place(db, job)
job.folder_id = folder.id
folders.stamp_done(db, job)
service.incr_ref_count(db, asset.id)
db.commit()
(root / f"{name}.bin").write_bytes(b"data")
return job
def _bearer(raw: str) -> TestClient:
return TestClient(app, headers={"authorization": f"Bearer {raw}"})
def _make(c, **body) -> dict:
r = c.post("/api/tokens", json={"name": "a machine", **body})
assert r.status_code == 200, r.text
return r.json()
def _audits(db, action):
return db.execute(select(AuditLog).where(AuditLog.action == action)).scalars().all()
# --- the token itself ----------------------------------------------------------------------------
def test_a_token_has_its_shape_and_a_checksum_that_refuses_a_typo():
raw = apitokens.mint()
assert raw.startswith("sfl_") and len(raw) == apitokens.TOKEN_LEN
assert apitokens.well_formed(raw)
assert apitokens.mint() != raw
flipped = raw[:10] + ("A" if raw[10] != "A" else "B") + raw[11:]
assert not apitokens.well_formed(flipped)
assert not apitokens.well_formed(raw[:-1])
assert not apitokens.well_formed("xxx_" + raw[4:])
# A header byte past ASCII arrives as one latin-1 character: refused, never a 500.
assert not apitokens.well_formed(raw[:-1] + "é")
def test_a_non_ascii_token_is_refused_as_any_bad_one(db, client):
# The test client re-sends a high byte as two UTF-8 bytes, which arrive as two latin-1
# characters: one "A" fewer keeps the header a token's length, so it is the checksum that meets
# them, not the length check.
raw = ("Bearer sfl_" + "A" * 47 + "é").encode("latin-1")
r = TestClient(app, headers={"authorization": raw}).get("/api/files/folders")
assert r.status_code == 401 and r.json()["detail"]["reason"] == "not_authenticated"
def test_the_hub_preset_makes_one_hub_folder_with_every_right_and_shows_the_token_once(
db, make_user, as_user
):
alice = make_user()
c = as_user(alice)
made = _make(c, preset="hub")
raw = made["token"]
assert apitokens.well_formed(raw) and made["prefix"] == raw[: apitokens.SHOWN_LEN]
[grant] = made["grants"]
assert grant["kind"] == "hub" and grant["rights"] == ["delete", "list", "read", "write"]
# Only the hash is kept, and the value is not in the list.
stored = db.execute(select(ApiToken)).scalar_one()
assert stored.token_hash != raw and raw not in str(c.get("/api/tokens").json())
# A second hub token reuses the one Hub folder.
again = _make(c, preset="hub")
assert again["grants"][0]["folder_id"] == grant["folder_id"]
hubs = db.execute(select(FileFolder).where(FileFolder.kind == "hub")).scalars().all()
assert len(hubs) == 1
assert len(_audits(db, "api_token_create")) == 2
def test_a_token_is_granted_only_the_users_own_folders_and_writes_only_where_uploads_land(
db, make_user, as_user
):
alice, bob = make_user(), make_user()
theirs = _own(db, bob.id, "Bob's")
c = as_user(alice)
r = c.post(
"/api/tokens",
json={"name": "x", "grants": [{"folder_id": theirs.id, "rights": ["read"]}]},
)
assert r.status_code == 409 and r.json()["detail"]["reason"] == "file_folder_gone"
youtube = folders.source_folder(db, alice.id, folders.SourceKey.YOUTUBE)
db.commit()
r = c.post(
"/api/tokens",
json={"name": "x", "grants": [{"folder_id": youtube.id, "rights": ["write"]}]},
)
assert r.json()["detail"]["reason"] == "file_folder_not_its_source"
assert c.post("/api/tokens", json={"name": "x"}).json()["detail"]["reason"] == (
"api_token_no_grants"
)
assert db.execute(select(ApiToken)).all() == []
@pytest.mark.parametrize("right", ["read", "write"])
def test_a_folder_granted_is_held_until_the_token_is_in(db, make_user, right):
"""A delete of the folder at the same moment waits and takes the new grant with it, rather than
failing the grant's foreign key."""
import threading
alice = make_user()
fid = _own(db, alice.id, "Mine").id
body = tokens_routes.TokenIn(
name="x", grants=[tokens_routes.GrantIn(folder_id=fid, rights=[right])]
)
tokens_routes._wanted_grants(db, alice, body)
outcome: list[object] = []
def delete_it():
other = SessionLocal()
try:
outcome.append(folders.delete_own(other, other.get(FileFolder, fid)))
other.commit()
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=delete_it)
t.start()
t.join(timeout=1)
assert t.is_alive() # the delete waits for the token on its way in
db.commit()
t.join(timeout=10)
assert not isinstance(outcome[0], Exception), outcome
def test_a_user_has_at_most_the_ceiling_of_tokens(db, make_user, as_user, monkeypatch):
monkeypatch.setattr(tokens_routes, "MAX_TOKENS", 2)
c = as_user(make_user())
_make(c, preset="hub")
_make(c, preset="hub")
r = c.post("/api/tokens", json={"name": "third", "preset": "hub"})
assert r.status_code == 409 and r.json()["detail"]["reason"] == "api_token_limit"
# --- what a token reaches ------------------------------------------------------------------------
@pytest.mark.parametrize("right", ["list", "read", "write", "delete"])
def test_each_right_holds_in_its_folder_only(
db, make_user, as_user, _root, _no_probe, upload_headers, right
):
"""One right in folder A; folder B has none. Every Files route a token may use, in both."""
alice = make_user()
a, b = _own(db, alice.id, "A"), _own(db, alice.id, "B")
in_a, in_b = _file_in(db, _root, alice.id, a, "fa"), _file_in(db, _root, alice.id, b, "fb")
raw = _make(as_user(alice), grants=[{"folder_id": a.id, "rights": [right]}])["token"]
m = _bearer(raw)
folders_seen = [f["id"] for f in m.get("/api/files/folders").json()]
assert folders_seen == ([a.id] if right == "list" else [])
listed = m.get("/api/files", params={"folder_id": a.id})
assert listed.status_code == (200 if right == "list" else 404)
if right == "list":
assert [f["id"] for f in listed.json()] == [in_a.id]
assert [f["id"] for f in m.get("/api/files").json()] == [in_a.id]
else:
assert m.get("/api/files").json() == []
assert m.get("/api/files", params={"folder_id": b.id}).status_code == 404
read_a = m.get(f"/api/files/{in_a.id}/file")
assert read_a.status_code == (200 if right == "read" else 404)
assert m.get(f"/api/files/{in_a.id}/sha256").status_code == (200 if right == "read" else 404)
assert m.get(f"/api/files/{in_b.id}/file").status_code == 404
up_a = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt", folder=a.id))
assert up_a.status_code == (200 if right == "write" else 404)
if right == "write":
assert up_a.json()["folder_id"] == a.id
up_b = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt", folder=b.id))
assert up_b.status_code == 404
no_folder = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt"))
assert no_folder.status_code == 400
assert no_folder.json()["detail"]["reason"] == "api_token_folder_required"
assert m.delete(f"/api/files/{in_b.id}").status_code == 404
assert db.get(DownloadJob, in_b.id) is not None
del_a = m.delete(f"/api/files/{in_a.id}")
assert del_a.status_code == (200 if right == "delete" else 404)
db.expire_all()
assert (db.get(DownloadJob, in_a.id) is None) == (right == "delete")
def test_a_token_never_reaches_another_users_files_or_the_shared_ones(
db, make_user, as_user, _root
):
alice, bob = make_user(), make_user()
hub_raw = _make(as_user(alice), preset="hub")["token"]
bobs = _file_in(db, _root, bob.id, _own(db, bob.id, "B"), "bf")
m = _bearer(hub_raw)
assert m.get(f"/api/files/{bobs.id}/file").status_code == 404
assert m.delete(f"/api/files/{bobs.id}").status_code == 404
shared = m.get("/api/files", params={"scope": "shared"})
assert shared.status_code == 403 and shared.json()["detail"]["reason"] == "api_token_scope"
def test_a_token_is_judged_by_its_grants_even_beside_a_signed_in_cookie(db, make_user, client):
"""Through the real session stack (`client`, a real sign-in): the `as_user` fixture stands in
for the very dependency this is about."""
alice = make_user(email="alice@example.com")
a = _own(db, alice.id, "A")
c = client
signed_in = c.post(
"/auth/password-login",
json={"email": "alice@example.com", "password": "correct horse battery"},
)
assert signed_in.status_code == 200
raw = _make(c, grants=[{"folder_id": a.id, "rights": ["read"]}])["token"]
assert len(c.get("/api/files/folders").json()) == 1 # the user sees every folder
both = c.get("/api/files/folders", headers={"authorization": f"Bearer {raw}"})
assert both.json() == [] # the token may not list: its grants decide
def _signed_in(client, email="alice@example.com") -> TestClient:
r = client.post(
"/auth/password-login", json={"email": email, "password": "correct horse battery"}
)
assert r.status_code == 200, r.text
return client
def test_a_token_works_on_the_files_routes_only(db, make_user, client):
"""Through the real session stack: under `as_user` every route without its header answers 401
whatever `current_user` does with a token, so these refusals would prove nothing there."""
make_user(email="alice@example.com")
raw = _make(_signed_in(client), preset="hub")["token"]
m = _bearer(raw)
assert m.get("/api/files/folders").status_code == 200
# Making a wider token, or reading the account, takes the user's session.
assert m.get("/api/me").json() == {"authenticated": False} # the token is no one here
assert m.get("/api/tokens").status_code == 401
assert m.post("/api/tokens", json={"name": "wider", "preset": "hub"}).status_code == 401
assert m.get("/api/files/usage").status_code == 401
assert m.patch("/api/files/1", json={"name": "x"}).status_code == 401
def test_only_the_files_routes_take_a_token():
"""The route table, not a list kept by hand: every route that resolves `files_caller` is under
/api/files, and these are all of them. The walk recurses into included routers (see
`test_body_limit._declared_routes` for why `app.routes` alone sees none of ours)."""
takers, nodes, routes = set(), [app], []
while nodes:
for route in getattr(nodes.pop(), "routes", ()) or ():
inner = getattr(route, "original_router", None)
if inner is not None:
nodes.append(inner)
else:
routes.append(route)
assert len(routes) > 100 # the walk found the app's routes, not just the shell's
for route in routes:
dependant = getattr(route, "dependant", None)
if dependant is None:
continue
stack, calls = [dependant], set()
while stack:
d = stack.pop()
calls.add(d.call)
stack.extend(d.dependencies)
if files_caller in calls:
for method in route.methods:
takers.add((method, route.path))
assert all(path.startswith("/api/files") for _, path in takers)
assert takers == {
("GET", "/api/files/folders"),
("GET", "/api/files"),
("GET", "/api/files/{job_id}/file"),
("GET", "/api/files/{job_id}/sha256"),
("POST", "/api/files/upload"),
("DELETE", "/api/files/{job_id}"),
}
def _reset_password(db, client, alice):
from app.routes.auth import _issue_token
link = _issue_token(db, alice, "reset", timedelta(hours=1))
r = client.post(
"/auth/password-reset/confirm", json={"token": link, "password": "a brand new password"}
)
assert r.status_code == 200, r.text
def _change_password(db, client, alice):
r = client.post(
"/auth/set-password",
json={"password": "a brand new password", "current_password": "correct horse battery"},
)
assert r.status_code == 200, r.text
def _log_out_others(db, client, alice):
assert client.post("/auth/logout-others").status_code == 200
def _revoke_untrusted_password(db, client, alice):
from app.routes.auth import _revoke_password
_revoke_password(db, alice)
db.commit()
@pytest.mark.parametrize(
"end_sessions",
[_reset_password, _change_password, _log_out_others, _revoke_untrusted_password],
)
def test_whatever_ends_every_session_ends_every_token_too(db, make_user, client, end_sessions):
"""A token minted from a stolen session must not outlive the owner's answer to the theft."""
alice = make_user(email="alice@example.com")
bob = make_user(email="bob@example.com")
raw = _make(_signed_in(client), preset="hub")["token"]
bobs = apitokens.mint()
db.add(ApiToken(user_id=bob.id, name="b", prefix=bobs[:12], token_hash=hash_token(bobs)))
db.commit()
assert _bearer(raw).get("/api/files/folders").status_code == 200
end_sessions(db, client, alice)
assert _bearer(raw).get("/api/files/folders").status_code == 401
assert db.execute(select(ApiToken.user_id)).scalars().all() == [bob.id] # only hers went
[revoked] = _audits(db, "api_token_revoke")
assert revoked.actor_id == alice.id and "every session" in (revoked.summary or "")
def test_a_token_asked_for_as_every_session_ends_is_refused(db, make_user, client, override_dep):
"""The session was still good when its request came in, and ended before the token was made."""
alice = make_user(email="alice@example.com")
c = _signed_in(client)
def ended_meanwhile(request: Request, session=Depends(get_db)):
user = current_user(request, session)
other = SessionLocal()
try:
bump_session_epoch(other, other.get(User, alice.id))
other.commit()
finally:
other.close()
return user
override_dep(current_user, ended_meanwhile)
r = c.post("/api/tokens", json={"name": "late", "preset": "hub"})
assert r.status_code == 401
assert db.execute(select(ApiToken.id)).scalars().all() == []
def _meanwhile(work):
"""Run `work(other_session)` in another thread and session, committed; (thread, outcome)."""
import threading
outcome: list[object] = []
def run():
other = SessionLocal()
try:
outcome.append(work(other))
other.commit()
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=run)
t.start()
return t, outcome
def test_ending_every_session_never_deadlocks_with_a_tokens_expiry_notice(db, make_user):
"""The notice's sweep holds the token's row and then writes a row naming the user; the sessions'
end holds the user and then deletes the token. A lock on the user that a mere reference to it
waits on would make that a deadlock — one of the two failing."""
alice = make_user()
raw = apitokens.mint()
soon = datetime.now(timezone.utc) + timedelta(days=1)
tok = ApiToken(
user_id=alice.id, name="t", prefix=raw[:12], token_hash=hash_token(raw), expires_at=soon
)
db.add(tok)
db.commit()
# `warn_expiring`, its two halves held apart: the token row taken…
db.execute(update(ApiToken).where(ApiToken.id == tok.id).values(expiry_warned=True))
t, outcome = _meanwhile(lambda other: bump_session_epoch(other, other.get(User, alice.id)))
t.join(timeout=1)
assert t.is_alive() # the sessions' end waits for the token row
# …and then its notice, naming the user.
create_notification(db, alice.id, "api_token_expiring", title="t", commit=False)
db.commit()
t.join(timeout=10)
assert outcome == [None]
db.expire_all()
assert db.execute(select(ApiToken.id)).scalars().all() == []
def test_a_token_granted_a_folder_being_deleted_never_deadlocks(db, make_user):
"""The delete holds the folder and then writes its audit line naming the user; the token holds
the user and then waits for the folder."""
alice = make_user()
fid = _own(db, alice.id, "Mine").id
folders.delete_own(db, db.get(FileFolder, fid)) # the folder held, as the delete route does
body = tokens_routes.TokenIn(
name="x", grants=[tokens_routes.GrantIn(folder_id=fid, rights=["read"])]
)
t, outcome = _meanwhile(
lambda other: tokens_routes.create_token(body, other.get(User, alice.id), other)
)
t.join(timeout=1)
assert t.is_alive() # the token waits for the folder
audit.record(db, alice.id, "file_folder_delete", target_type="file_folder", target_id=fid)
db.commit()
t.join(timeout=10)
[refusal] = outcome
assert getattr(refusal, "status_code", None) == 409, refusal # the folder went first
def test_a_token_made_as_every_session_ends_goes_with_them(db, make_user):
"""The other order: the token is on its way in when the sessions end."""
import threading
alice = make_user(email="alice@example.com")
db.execute(select(User.id).where(User.id == alice.id).with_for_update()) # as create_token
raw = apitokens.mint()
db.add(ApiToken(user_id=alice.id, name="t", prefix=raw[:12], token_hash=hash_token(raw)))
db.flush()
outcome: list[object] = []
def end_sessions():
other = SessionLocal()
try:
bump_session_epoch(other, other.get(User, alice.id))
other.commit()
outcome.append("done")
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=end_sessions)
t.start()
t.join(timeout=1)
assert t.is_alive() # it waits for the token being made
db.commit()
t.join(timeout=10)
assert outcome == ["done"]
db.expire_all()
assert db.execute(select(ApiToken.id)).scalars().all() == []
# --- a token that does not hold ------------------------------------------------------------------
def _refused(m: TestClient) -> int:
return m.get("/api/files/folders").status_code
def test_a_malformed_unknown_expired_revoked_or_suspended_token_holds_nothing(
db, make_user, as_user
):
alice = make_user()
c = as_user(alice)
raw = _make(c, preset="hub")["token"]
assert _refused(_bearer(raw)) == 200
assert _refused(_bearer("sfl_short")) == 401
assert _refused(_bearer(apitokens.mint())) == 401 # well formed, never made
expiring = _make(c, preset="hub", expires_days=1)
token = db.get(ApiToken, expiring["id"])
token.expires_at = datetime.now(timezone.utc) - timedelta(seconds=1)
db.commit()
assert _refused(_bearer(expiring["token"])) == 401
assert c.get("/api/tokens").json()[1]["expired"] is True
assert c.delete(f"/api/tokens/{expiring['id']}").json() == {
"ok": True,
"deleted": expiring["id"],
}
assert (
db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == expiring["id"])).all()
== []
)
assert len(_audits(db, "api_token_revoke")) == 1
revoked = _make(c, preset="hub")
assert c.delete(f"/api/tokens/{revoked['id']}").status_code == 200
assert _refused(_bearer(revoked["token"])) == 401
alice.is_suspended = True
db.commit()
assert _refused(_bearer(raw)) == 401
def test_a_token_about_to_run_out_warns_its_user_once_and_is_marked_in_the_list(
db, make_user, as_user
):
alice, bob = make_user(), make_user()
c = as_user(alice)
soon = _make(c, preset="hub", expires_days=3)
_make(c, preset="hub", expires_days=30) # outside the span
_make(c, preset="hub") # never runs out
gone = _make(c, preset="hub", expires_days=1)
db.get(ApiToken, gone["id"]).expires_at = datetime.now(timezone.utc) - timedelta(seconds=1)
_make(as_user(bob), preset="hub", expires_days=2)
db.commit()
assert [t["expiring"] for t in c.get("/api/tokens").json()] == [True, False, False, False]
assert apitokens.warn_expiring(db) == 2 # alice's and bob's; the run-out one is not "soon"
notes = db.execute(select(Notification).where(Notification.user_id == alice.id)).scalars()
[note] = notes.all()
assert note.type == "api_token_expiring"
assert note.data["token_id"] == soon["id"] and note.data["name"] == "a machine"
assert note.data["expires_at"] == soon["expires_at"]
assert apitokens.warn_expiring(db) == 0 # once per token
def test_the_files_gc_runs_the_token_warning(db, make_user, as_user, _root):
_make(as_user(make_user()), preset="hub", expires_days=3)
assert gcmod.run_download_gc(db)["tokens_warned"] == 1
def test_another_users_token_cannot_be_revoked(db, make_user, as_user):
alice, bob = make_user(), make_user()
made = _make(as_user(alice), preset="hub")
assert as_user(bob).delete(f"/api/tokens/{made['id']}").status_code == 404
assert db.get(ApiToken, made["id"]) is not None
def test_a_client_trying_tokens_is_slowed_down(db, make_user, as_user, monkeypatch):
alice = make_user()
raw = _make(as_user(alice), preset="hub")["token"]
monkeypatch.setattr(apitokens, "_bad", RateLimiter(3, 300))
for _ in range(3):
assert _refused(_bearer(apitokens.mint())) == 401
slowed = _bearer(apitokens.mint()).get("/api/files/folders")
assert slowed.status_code == 429
assert slowed.json()["detail"]["reason"] == "api_token_rate_limited"
# Even a good token from that client waits out the window: a guesser cannot tell a hit.
assert _refused(_bearer(raw)) == 429
def test_a_runaway_machine_is_held_back_per_token(db, make_user, as_user, monkeypatch):
c = as_user(make_user())
first, second = _make(c, preset="hub")["token"], _make(c, preset="hub")["token"]
monkeypatch.setattr(apitokens, "_calls", RateLimiter(2, 60))
assert [_refused(_bearer(first)) for _ in range(3)] == [200, 200, 429]
assert _refused(_bearer(second)) == 200
# --- what a token leaves behind ------------------------------------------------------------------
def test_a_tokens_use_is_stamped_and_its_uploads_and_deletes_name_it_in_the_audit(
db, make_user, as_user, _root, _no_probe, upload_headers
):
alice = make_user()
made = _make(as_user(alice), preset="hub")
hub = made["grants"][0]["folder_id"]
assert db.get(ApiToken, made["id"]).last_used_at is None
m = _bearer(made["token"])
up = m.post("/api/files/upload", content=b"hi", headers=upload_headers("h.txt", folder=hub))
assert up.status_code == 200
db.expire_all()
assert db.get(ApiToken, made["id"]).last_used_at is not None
assert m.delete(f"/api/files/{up.json()['id']}").status_code == 200
via = f"api token {made['prefix']}…"
assert [a.summary for a in _audits(db, "file_upload")] == [via]
assert [a.summary for a in _audits(db, "file_delete")] == [via]
+1 -1
View File
@@ -493,7 +493,7 @@ def _revoke_as_google_adoption(db, user) -> None:
Google callback adopting an account whose email was never verified."""
from app.routes.auth import revoke_untrusted_preset_password
revoke_untrusted_preset_password(user)
revoke_untrusted_preset_password(db, user)
db.commit()
+5 -5
View File
@@ -221,7 +221,7 @@ def test_the_default_ceiling_clears_the_largest_body_a_default_route_takes():
CAPPED_BODY_SITES = {
("diagnostics.py", "_parsed_events", "capped_body", "BODY_MAX_BYTES"),
("debug_sessions.py", "_image_body", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
("downloads.py", "spool_upload", "capped_stream_to", "cap"),
("files.py", "spool_upload", "capped_stream_to", "cap"),
}
@@ -367,7 +367,7 @@ def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
assert ("POST", "/api/debug-sessions/{sid}/screenshots") in bodylimit.ROUTE_MAX_BODY_BYTES
# The upload site's limit is the sysconfig knob, which is bounded BY the edge ceiling rather
# than being a constant to compare against — that pairing is its own test below.
assert ("POST", "/api/downloads/upload") in bodylimit.ROUTE_MAX_BODY_BYTES
assert ("POST", "/api/files/upload") in bodylimit.ROUTE_MAX_BODY_BYTES
def test_release_read_connection_ends_the_transaction_it_was_called_for(db):
@@ -467,7 +467,7 @@ def test_every_body_reading_site_hands_its_pooled_connection_back_first():
assert not missing, (
"a route reads its own request body while still inside its read transaction, pinning a "
f"pooled connection for the whole transfer: {missing}. Call "
"db.release_read_connection(db) before the await — see routes/downloads.py:spool_upload."
"db.release_read_connection(db) before the await — see routes/files.py:spool_upload."
)
@@ -487,7 +487,7 @@ def test_the_upload_ceiling_and_the_admin_knob_cannot_drift_apart():
`errors.tooLarge` exists and why the dialog checks the file against `upload_max_bytes` first.
"""
spec = next(s for s in sysconfig.SPECS if s.key == "download_upload_max_bytes")
edge = bodylimit.ROUTE_MAX_BODY_BYTES[("POST", "/api/downloads/upload")]
edge = bodylimit.ROUTE_MAX_BODY_BYTES[("POST", "/api/files/upload")]
assert spec.max == edge
@@ -540,7 +540,7 @@ def test_no_route_with_a_raised_ceiling_declares_a_body_parameter(method, templa
"its dependencies, so FastAPI reads it before the first dependency runs and the raised "
"ceiling is all that stands between an anonymous caller and the whole allowance. Read the "
"body yourself, from a dependency declared AFTER the auth one — see "
"routes/downloads.py:spool_upload."
"routes/files.py:spool_upload."
)
+2 -2
View File
@@ -374,7 +374,7 @@ def test_a_clip_is_not_redownloadable(db, make_user, as_user, yt_video):
c = as_user(make_user())
job = _clip(c, source="youtube", ref=yt_video).json()
row = db.get(DownloadJob, job["id"])
row.status = "done"
row.status, row.ready_at = "done", row.created_at
db.commit()
r = c.post(f"/api/downloads/{job['id']}/redownload")
assert (r.status_code, r.json()["detail"]["reason"]) == (409, "not_redownloadable")
@@ -395,7 +395,7 @@ def test_a_clip_job_reaches_the_clip_entry(monkeypatch):
def worker(db, tmp_path, monkeypatch):
"""The worker path with its external programs replaced: ffmpeg writes the file it was given, the
probe answers what the test says, the phone-safe step and the poster pass through."""
monkeypatch.setattr(settings, "download_root", str(tmp_path / "root"))
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
seen: dict = {}
def ffmpeg(cmd, *_a, **_k):
+25 -4
View File
@@ -5,6 +5,14 @@ from app.downloads import errors as dlerrors
from app.downloads.errors import classify
# Verbatim from dev job 128 (2026-10-09, yt-dlp 2026.07.04); the same link downloads under 2026.08.19.
_TIKTOK_UNEXPECTED = (
"ERROR: [TikTok] 7418778475890232609: Unexpected response from webpage request; please report "
"this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue "
"template. Confirm you are on the latest version using yt-dlp -U"
)
def test_real_ytdlp_messages_map_to_categories():
# Verbatim-shaped yt-dlp / ffmpeg output → the category the UI explains.
cases = {
@@ -32,6 +40,7 @@ def test_real_ytdlp_messages_map_to_categories():
"The source file is missing.": "source_missing",
"ERROR: unable to download video data: HTTP Error 403: Forbidden": "network",
"ERROR: Unable to download webpage: <urlopen error timed out>": "network",
_TIKTOK_UNEXPECTED: "site_changed",
}
for msg, expected in cases.items():
assert classify(msg) == expected, f"{msg!r} → {classify(msg)!r}, expected {expected!r}"
@@ -211,13 +220,25 @@ def test_our_post_without_a_video_verdict_names_the_group_link():
def test_a_catalog_entry_never_leaves_its_host():
# "Cannot parse data" is yt-dlp's wording for a broken extractor on every site.
# "Cannot parse data" is yt-dlp's wording for a broken extractor on every site: elsewhere it is
# only the category — the site changed under the extractor.
other = _FB_PARSE.replace("[facebook]", "[vimeo]")
assert dlerrors.diagnose(other, url="https://vimeo.com/1") == ("unknown", None)
assert dlerrors.diagnose(other, url="https://vimeo.com/1") == ("site_changed", None)
# The Facebook text for a link that is not Facebook's (a look-alike host) is not the Facebook wall.
assert dlerrors.diagnose(_FB_PARSE, url="https://facebook.com.evil/reel/1") == ("unknown", None)
assert dlerrors.diagnose(_FB_PARSE, url="https://facebook.com.evil/reel/1") == (
"site_changed",
None,
)
# No link known: the category only.
assert dlerrors.diagnose(_FB_PARSE) == ("unknown", None)
assert dlerrors.diagnose(_FB_PARSE) == ("site_changed", None)
def test_a_bug_report_request_keeps_the_more_specific_category():
# yt-dlp asks for the bug report on these too; what they say about the media wins.
assert classify(_FB_NO_FORMATS) == "format_unavailable"
assert classify("ERROR: [x] 1: Unable to extract title; please report this issue") == (
"format_unavailable"
)
def test_diagnose_falls_back_to_the_category():
@@ -80,7 +80,7 @@ def _asset_and_job(db, user, *, sig, asset_status, job_status, error=None):
def _fail_the_child(monkeypatch, tmp_path, *, cookie: bool):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
cookiefile = tmp_path / "cookies.txt"
cookiefile.write_text("# Netscape HTTP Cookie File\n")
@@ -230,7 +230,7 @@ def test_a_download_yt_dlp_gave_no_duration_stores_the_files_duration(db, monkey
# row lost its trim action. Through the real finalize, with a real 3-second file.
u = _user(db, "fb-duration@example.com")
a, j = _asset_and_job(db, u, sig="shared", asset_status="downloading", job_status="running")
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
def extract(job_id, asset_id, url, spec, staging, adapter, **kwargs):
@@ -350,7 +350,7 @@ def test_a_signed_in_facebook_download_is_titled_with_its_posts_text(db, monkeyp
# file name and the share preview all said "Video". The post's text is in the description.
u = _user(db, "fb-title@example.com")
a, j = _asset_and_job(db, u, sig="shared", asset_status="downloading", job_status="running")
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
def extract(job_id, asset_id, url, spec, staging, adapter, **kwargs):
@@ -401,7 +401,7 @@ def test_an_mp3_download_is_stored_as_an_mp3(db, monkeypatch, tmp_path):
# video/mp4, which iPad's WebKit will not play. It stays an .mp3 (served as audio/mpeg).
u = _user(db, "fb-mp3@example.com")
a, j = _asset_and_job(db, u, sig="shared", asset_status="downloading", job_status="running")
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
def extract(job_id, asset_id, url, spec, staging, adapter, **kwargs):
@@ -430,7 +430,7 @@ def test_a_video_download_is_still_made_browser_playable(db, monkeypatch, tmp_pa
a, j = _asset_and_job(
db, u, sig=f"mkv-{mode}", asset_status="downloading", job_status="running"
)
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
def extract(job_id, asset_id, url, spec, staging, adapter, **kwargs):
+11 -9
View File
@@ -5,7 +5,7 @@ unlocked read: two overlapping requests for one job (a double-click, a second ta
gateway timeout) both saw it holding its asset and both released the hold. The count then reaches 0
while another job still holds the asset, and GC or the ready-file delete removes it under that job.
A pause racing a cancel did the same one step later: its unlocked write turned the released job back
into a holder, which a later delete releases again. `_own_job_for_update` makes the second request
into a holder, which a later delete releases again. `own_job_for_update` makes the second request
wait and decide on the first one's committed result.
"""
@@ -15,15 +15,12 @@ import pytest
from fastapi import HTTPException
from sqlalchemy import select
from app.apitokens import FilesCaller
from app.db import SessionLocal
from app.downloads import service
from app.models import DownloadJob, MediaAsset, User
from app.routes.downloads import (
_own_job_for_update,
cancel_download,
delete_download,
pause_download,
)
from app.routes.downloads import cancel_download, pause_download
from app.routes.files import delete_file, own_job_for_update
_REF = "https://example.com/video"
@@ -54,11 +51,16 @@ def _asset_held_by_two_jobs(db, status="error"):
return mine, asset, jobs[0]
def delete_file_as(job_id, *, user, db):
"""`delete_file` as the user's own request (it takes the Files caller, not the user)."""
return delete_file(job_id, caller=FilesCaller(user=user), db=db)
@pytest.mark.parametrize(
("status", "transition", "route", "left"),
[
("error", service.cancel, cancel_download, "canceled"),
("error", service.delete, delete_download, None),
("error", service.delete, delete_file_as, None),
("queued", service.cancel, pause_download, "canceled"),
],
)
@@ -80,7 +82,7 @@ def test_overlapping_requests_for_one_job_release_its_hold_once(
finished.set()
try:
locked = _own_job_for_update(first, user, job_id)
locked = own_job_for_update(first, user, job_id)
other = threading.Thread(target=the_other_request)
other.start()
# The second request waits for the first one's locks instead of deciding on the same row.
+4 -4
View File
@@ -188,7 +188,7 @@ def test_list_downloads_is_capped(db, monkeypatch):
_job(db, u.id, queue_pos=i, status="done")
db.commit()
monkeypatch.setattr(dl, "_MAX_LIST", 2)
monkeypatch.setattr(dl, "MAX_LIST", 2)
rows = dl.list_downloads(user=u, db=db)
assert len(rows) == 2 # bounded by the cap, not the 3 rows on disk
@@ -259,7 +259,7 @@ def test_a_refusal_for_one_user_does_not_poison_the_shared_asset(db, monkeypatch
that shared row — correct for an upstream failure, wrong for "YOUR storage is full", which
`worker._process_job` would then hand verbatim to someone with an empty library.
"""
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
over, other = User(email="over@example.com"), User(email="other@example.com")
db.add_all([over, other])
db.flush()
@@ -306,7 +306,7 @@ def test_a_download_over_the_space_left_says_so_and_releases_the_shared_asset(
# yt-dlp refuses a file over `max_filesize` by saying so and returning with no file (pinned in
# test_formats). That is the owner's storage, not a processing fault: their refusal, in words
# true of a cap with space still left, and the shared asset goes back for the next job.
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
u = User(email="nearcap@example.com")
db.add(u)
db.flush()
@@ -352,7 +352,7 @@ def test_a_download_that_outgrows_the_space_left_unannounced_is_refused_and_not_
# yt-dlp's `max_filesize` only stops a plain HTTP file that announces its size: YouTube's
# fragmented formats never meet it (measured on localdev: 718 MB fetched with 1 MB left). The
# file it produced is held to what is left by its real size, as a clip is.
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
u = User(email="unannounced@example.com")
db.add(u)
db.flush()
+1 -1
View File
@@ -77,7 +77,7 @@ def test_resume_resets_a_failed_shared_asset_to_pending(db, as_user):
def test_delete_releases_and_removes_the_job(db, as_user):
u, asset, job = _seed(db, job_status="done", asset_status="ready", ref_count=1, rel_path=None)
r = as_user(u).request("DELETE", f"/api/downloads/{job.id}")
r = as_user(u).request("DELETE", f"/api/files/{job.id}")
assert r.status_code == 200, r.text
db.expire_all()
assert db.get(DownloadJob, job.id) is None
+1 -1
View File
@@ -21,7 +21,7 @@ pytestmark = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg n
def test_an_editor_clip_larger_than_what_is_left_is_refused_and_not_placed(
db, make_user, tmp_path, monkeypatch
):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
src = tmp_path / "in.mp4"
subprocess.run(
[
+6 -10
View File
@@ -258,7 +258,7 @@ def test_an_edit_that_crashes_is_said_once(sink, finalized, monkeypatch, tmp_pat
"""The line `_process_edit`'s handler used to log before `_fail_edit` logged it again."""
(tmp_path / "src.mp4").write_bytes(b"x")
monkeypatch.setattr(pipeline, "SessionLocal", _EditDb)
monkeypatch.setattr(pipeline.settings, "download_root", str(tmp_path))
monkeypatch.setattr(pipeline.settings, "files_root", str(tmp_path))
monkeypatch.setattr(pipeline, "_edit_staging_dir", lambda asset_id: tmp_path / "staging")
def odd_spec(*a, **k):
@@ -300,7 +300,7 @@ def test_a_failed_download_line_does_not_put_back_what_the_adapter_took_out(sink
RuntimeError("dictionary changed size during iteration"),
RecursionError("maximum recursion depth exceeded"),
NotImplementedError(),
FileNotFoundError(2, "No such file", "/downloads/Some_Channel/Season_2024/A_Private_Title"),
FileNotFoundError(2, "No such file", "/files/Some_Channel/Season_2024/A_Private_Title"),
],
ids=lambda e: type(e).__name__,
)
@@ -499,11 +499,9 @@ def test_a_failed_poster_or_storyboard_names_no_file(sink, tmp_path):
@pytest.mark.parametrize(
"err",
[
subprocess.TimeoutExpired(
["ffmpeg", "-i", "/downloads/Some_Channel/A_Private_Title.mp4"], 30
),
FileNotFoundError(2, "No such file", "/downloads/Some_Channel/A_Private_Title.jpg"),
subprocess.CalledProcessError(1, ["ffmpeg", "-i", "/downloads/Some_Channel/x.mp4"]),
subprocess.TimeoutExpired(["ffmpeg", "-i", "/files/Some_Channel/A_Private_Title.mp4"], 30),
FileNotFoundError(2, "No such file", "/files/Some_Channel/A_Private_Title.jpg"),
subprocess.CalledProcessError(1, ["ffmpeg", "-i", "/files/Some_Channel/x.mp4"]),
],
ids=lambda e: type(e).__name__,
)
@@ -523,9 +521,7 @@ def test_a_thumbnail_that_cannot_be_copied_names_no_file(sink, monkeypatch, tmp_
src.write_bytes(b"\xff\xd8\xff rest")
def refuse(a, b):
raise PermissionError(
13, "Permission denied", "/downloads/Some_Channel/A_Private_Title.jpg"
)
raise PermissionError(13, "Permission denied", "/files/Some_Channel/A_Private_Title.jpg")
monkeypatch.setattr(editmod.shutil, "copyfile", refuse)
assert editmod.thumbnail_to_jpg(src, tmp_path / "out.jpg") is None
+807
View File
@@ -0,0 +1,807 @@
"""The Files library's folders and a file's retention: which folder a new file lands in, when a done
file goes, and the GC that takes it — warning once, never a file its user kept, never a file another
user still holds.
Pure-logic for the source-key rule; DB-backed for the rest.
"""
import hashlib
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import func, select
from app import sysconfig
from app.config import settings
from app.downloads import folders, formats, service
from app.downloads import gc as gcmod
from app.db import SessionLocal
from app.downloads.status import JobStatus
from app.models import DownloadJob, FileFolder, MediaAsset, Notification
NOW = datetime.now(timezone.utc)
@pytest.mark.parametrize(
("url", "key"),
[
("https://www.tiktok.com/@a/video/1", "site:tiktok.com"),
("https://m.facebook.com/groups/x/posts/2", "site:facebook.com"),
("https://vimeo.com/123", "site:vimeo.com"),
("https://youtu.be/abcdefghijk", "youtube"),
("https://music.youtube.com/watch?v=abcdefghijk", "youtube"),
("https://www.youtube.com/watch?v=abcdefghijk", "youtube"),
# A front door is only stripped when a real host remains behind it.
("https://m.co/x", "site:m.co"),
("not a url", "site:unknown"),
],
)
def test_a_url_lands_in_its_sites_folder(url, key):
assert folders.site_key(url) == key
@pytest.fixture
def _root(tmp_path, monkeypatch):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
return tmp_path
def _asset(db, kind="youtube", ref="abcdefghijk", status="ready", rel="a.mp4", size=10):
asset = MediaAsset(
source_kind=kind,
source_ref=ref,
format_sig=hashlib.sha256(f"{kind}:{ref}".encode()).hexdigest()[:32],
status=status,
rel_path=rel,
size_bytes=size,
ref_count=0,
title="A title",
)
db.add(asset)
db.flush()
return asset
def _job(db, user_id, asset, *, kind="youtube", ref=None, status=JobStatus.DONE, **kw):
job = DownloadJob(
user_id=user_id,
asset_id=asset.id if asset else None,
source_kind=kind,
source_ref=ref or (asset.source_ref if asset else "x"),
profile_snapshot={},
status=status,
**kw,
)
db.add(job)
db.flush()
folders.place(db, job)
if asset is not None:
service.incr_ref_count(db, asset.id)
db.commit()
return job
def _folder(db, job) -> FileFolder:
return db.get(FileFolder, job.folder_id)
def test_every_kind_of_job_lands_in_its_source_folder(db, make_user):
uid = make_user().id
yt = _job(db, uid, _asset(db))
site = _job(db, uid, _asset(db, "url", "https://www.tiktok.com/@a/video/1"), kind="url")
up = _job(db, uid, _asset(db, "upload", "u1"), kind="upload")
plex = _job(
db, uid, _asset(db, "clip", "c1"), kind="clip", edit_spec={"source": "plex", "ref": "9"}
)
ytclip = _job(
db, uid, _asset(db, "clip", "c2"), kind="clip", edit_spec={"source": "youtube", "ref": "v"}
)
# An edit of a Plex clip: only its parent knows it came from Plex (the clip's asset says "clip").
edit = _job(
db, uid, _asset(db, "edit", "e1"), kind="edit", source_job_id=plex.id,
source_asset_id=plex.asset_id,
) # fmt: skip
keys = {j.id: _folder(db, j).source_key for j in (yt, site, up, plex, ytclip, edit)}
assert keys == {
yt.id: "youtube",
site.id: "site:tiktok.com",
up.id: "upload",
plex.id: "plex",
ytclip.id: "youtube",
edit.id: "plex", # an edit sits with the file it was cut from
}
# One folder per origin per user: the two YouTube files share theirs.
assert _folder(db, yt).id == _folder(db, ytclip).id
def test_an_edit_whose_parent_is_gone_follows_the_parents_asset(db, make_user):
uid = make_user().id
src = _asset(db, "url", "https://vimeo.com/1")
edit = DownloadJob(
user_id=uid, source_kind="edit", source_ref="e", profile_snapshot={},
source_job_id=None, source_asset_id=src.id, status=JobStatus.DONE,
) # fmt: skip
assert folders.source_key_for(db, edit) == "site:vimeo.com"
def test_an_edit_keeps_its_origin_after_its_parent_and_its_file_are_gone(db, make_user):
# The parent download expires, its file goes with its last holder, and both of the edit's
# pointers to it are cleared: the edit must still go back to the YouTube folder it came from.
uid = make_user().id
parent = _job(db, uid, _asset(db))
edit = _job(
db, uid, _asset(db, "edit", "e1"), kind="edit", source_job_id=parent.id,
source_asset_id=parent.asset_id,
) # fmt: skip
home = _folder(db, edit)
own = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="Mine")
db.add(own)
db.flush()
folders.move(db, edit, own)
edit.source_job_id = edit.source_asset_id = None # what ON DELETE SET NULL leaves
db.delete(parent)
db.commit()
folders.move(db, edit, home)
assert edit.folder_id == home.id
folders.move(db, edit, own)
folders.delete_own(db, own)
assert edit.folder_id == home.id
def test_a_folder_deleted_while_a_file_goes_in_takes_that_file_home_too(db, make_user):
# Before, the delete missed the file still on its way in and failed on the foreign key (a 500),
# or the file's insert did after the folder went.
import threading
uid = make_user().id
own = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="Mine")
db.add(own)
db.commit()
own_id = own.id
job = _job(db, uid, _asset(db))
home = job.folder_id
target = folders.own_folder(db, uid, own_id) # held from the check until this commits
outcome: list[object] = []
def delete_it():
other = SessionLocal()
try:
outcome.append(folders.delete_own(other, other.get(FileFolder, own_id)))
other.commit()
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=delete_it)
t.start()
t.join(timeout=1)
assert t.is_alive() # the delete waits for the file, even before it is written
folders.move(db, job, target)
db.commit()
t.join(timeout=10)
assert outcome == [1]
db.expire_all()
assert db.get(DownloadJob, job.id).folder_id == home
assert db.get(FileFolder, own_id) is None
def test_folders_are_per_user(db, make_user):
a, b = make_user().id, make_user().id
asset = _asset(db)
ja, jb = _job(db, a, asset), _job(db, b, asset)
assert ja.folder_id != jb.folder_id
assert (_folder(db, ja).user_id, _folder(db, jb).user_id) == (a, b)
def test_a_source_folder_keeps_files_the_retention_days_and_uploads_forever(db, make_user):
sysconfig.set_value(db, "download_retention_days", 12)
db.commit()
uid = make_user().id
yt = _job(db, uid, _asset(db))
up = _job(db, uid, _asset(db, "upload", "u1"), kind="upload")
assert _folder(db, yt).retention_days == 12
assert _folder(db, up).retention_days is None
def test_enqueue_places_the_job_and_a_cache_hit_is_stamped_at_once(db, make_user):
uid = make_user().id
spec = formats.normalize({})
first = service.enqueue(db, uid, "youtube", "abcdefghijk", spec)
assert _folder(db, first).source_key == "youtube"
assert first.expires_at is None # queued: its retention starts when it is done
asset = db.get(MediaAsset, first.asset_id)
asset.status, asset.rel_path = "ready", "a.mp4"
db.commit()
other = make_user().id
hit = service.enqueue(db, other, "youtube", "abcdefghijk", spec)
assert hit.status == JobStatus.DONE
days = _folder(db, hit).retention_days
assert hit.expires_at is not None
assert abs(hit.expires_at - (NOW + timedelta(days=days))) < timedelta(minutes=5)
def test_a_finishing_worker_stamps_the_expiry_unless_kept(db, make_user):
uid = make_user().id
asset = _asset(db)
plain = _job(db, uid, asset, status=JobStatus.RUNNING)
kept = _job(db, uid, asset, status=JobStatus.RUNNING, kept=True)
for job in (plain, kept):
assert service.finalize_if_running(db, job.id, status=JobStatus.DONE, progress=100)
db.commit()
db.refresh(plain)
db.refresh(kept)
assert plain.expires_at is not None
assert kept.expires_at is None
def _gc(db):
return gcmod.run_download_gc(db)
def _notices(db, uid, kind):
return db.execute(
select(func.count())
.select_from(Notification)
.where(Notification.user_id == uid, Notification.type == kind)
).scalar()
def test_the_gc_warns_once_then_takes_the_file_out_of_the_library(db, make_user, _root):
(_root / "a.mp4").write_bytes(b"x")
uid = make_user().id
job = _job(db, uid, _asset(db), expires_at=NOW + timedelta(hours=12))
_gc(db)
_gc(db)
assert _notices(db, uid, "download_expiring") == 1
job.expires_at = NOW - timedelta(minutes=1)
db.commit()
assert _gc(db)["expired"] == 1
assert db.get(DownloadJob, job.id) is None
assert _notices(db, uid, "download_expired") == 1
# Its last holder gone, the shared file and its row go too.
assert not (_root / "a.mp4").exists()
assert db.execute(select(func.count()).select_from(MediaAsset)).scalar() == 0
@pytest.mark.parametrize("restart", ["unkeep", "move", "retention"])
def test_a_file_whose_clock_starts_again_is_warned_again(db, make_user, _root, restart):
# The GC warns once per expiry; a file whose expiry was moved must be warned before the new one.
(_root / "a.mp4").write_bytes(b"x")
uid = make_user().id
job = _job(db, uid, _asset(db), expires_at=NOW + timedelta(hours=12))
_gc(db)
assert job.expiry_warned
if restart == "unkeep":
folders.set_kept(db, job, True)
folders.set_kept(db, job, False)
elif restart == "move":
home = _folder(db, job)
own = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="Mine")
db.add(own)
db.flush()
folders.move(db, job, own)
folders.move(db, job, home)
else:
folders.set_retention(db, _folder(db, job), 90)
db.commit()
assert not job.expiry_warned
def test_the_gc_never_takes_a_kept_file(db, make_user, _root):
(_root / "a.mp4").write_bytes(b"x")
uid = make_user().id
job = _job(db, uid, _asset(db), kept=True, expires_at=NOW - timedelta(days=1))
assert _gc(db)["expired"] == 0
assert db.get(DownloadJob, job.id) is not None
def test_one_users_expiry_leaves_another_users_copy(db, make_user, _root):
(_root / "a.mp4").write_bytes(b"x")
a, b = make_user().id, make_user().id
asset = _asset(db)
gone = _job(db, a, asset, expires_at=NOW - timedelta(minutes=1))
stays = _job(db, b, asset, expires_at=NOW + timedelta(days=20))
_gc(db)
assert db.get(DownloadJob, gone.id) is None
assert db.get(DownloadJob, stays.id) is not None
assert (_root / "a.mp4").exists()
def _meanwhile(monkeypatch, act, step="_relock_expired"):
"""Run `act` in another session (another request) between a GC pass's SELECT and its handling of
each row it found — the window a user's click can land in. `step` is the pass's per-row re-read
(expiry, warning, eviction); `act` gets that row's id."""
real = getattr(gcmod, step)
def wrapped(db, key, *rest):
other = SessionLocal()
try:
act(other, key)
other.commit()
finally:
other.close()
return real(db, key, *rest)
monkeypatch.setattr(gcmod, step, wrapped)
def test_a_file_kept_while_the_gc_runs_stays(db, make_user, _root, monkeypatch):
(_root / "a.mp4").write_bytes(b"x")
job = _job(db, make_user().id, _asset(db), expires_at=NOW - timedelta(minutes=1))
_meanwhile(monkeypatch, lambda other, jid: setattr(other.get(DownloadJob, jid), "kept", True))
assert _gc(db)["expired"] == 0
db.expire_all()
assert db.get(DownloadJob, job.id) is not None
assert (_root / "a.mp4").exists()
def test_a_file_deleted_while_the_gc_runs_is_released_once(db, make_user, _root, monkeypatch):
# Released twice, the shared file would go from under the other user's copy.
(_root / "a.mp4").write_bytes(b"x")
a, b = make_user().id, make_user().id
asset = _asset(db)
_job(db, a, asset, expires_at=NOW - timedelta(minutes=1))
stays = _job(db, b, asset, expires_at=NOW + timedelta(days=20))
def user_deletes(other, jid):
service.delete(other, other.get(DownloadJob, jid))
_meanwhile(monkeypatch, user_deletes)
_gc(db)
db.expire_all()
assert db.get(DownloadJob, stays.id).asset_id == asset.id
assert db.get(MediaAsset, asset.id).ref_count == 1
assert (_root / "a.mp4").exists()
def test_the_cache_cap_never_evicts_a_kept_file_and_an_evicted_one_leaves_every_library(
db, make_user, _root
):
sysconfig.set_value(db, "download_total_max_bytes", 15)
db.commit()
a, b = make_user().id, make_user().id
for name in ("old.mp4", "new.mp4"):
(_root / name).write_bytes(b"x")
old = _asset(db, ref="oldoldoldol", rel="old.mp4", size=10)
new = _asset(db, ref="newnewnewne", rel="new.mp4", size=10)
old.last_access_at = NOW - timedelta(days=9)
new.last_access_at = NOW - timedelta(days=1)
kept = _job(db, a, old, kept=True) # the older file, but kept: the newer one goes instead
ja, jb = _job(db, a, new), _job(db, b, new)
assert _gc(db)["evicted"] == 1
assert db.get(DownloadJob, kept.id) is not None
assert db.get(DownloadJob, ja.id) is None and db.get(DownloadJob, jb.id) is None
assert _notices(db, a, "download_evicted") == 1
assert _notices(db, b, "download_evicted") == 1
assert (_root / "old.mp4").exists() and not (_root / "new.mp4").exists()
def test_a_file_deleted_while_the_gc_warns_does_not_stop_the_run(db, make_user, _root, monkeypatch):
# Written back from a stale read, the warning of a deleted file failed the whole run: nothing
# expired or was evicted that cycle, nor the next while the race lasted.
(_root / "a.mp4").write_bytes(b"x")
(_root / "b.mp4").write_bytes(b"x")
uid = make_user().id
_job(db, uid, _asset(db), expires_at=NOW + timedelta(hours=1))
gone = _job(
db, uid, _asset(db, ref="bbbbbbbbbbb", rel="b.mp4"), expires_at=NOW - timedelta(minutes=1)
)
def user_deletes(other, jid):
service.delete(other, other.get(DownloadJob, jid))
_meanwhile(monkeypatch, user_deletes, step="_mark_warned")
result = _gc(db)
assert (result["warned"], result["expired"]) == (0, 1)
db.expire_all()
assert db.get(DownloadJob, gone.id) is None
assert _notices(db, uid, "download_expiring") == 0
def test_a_file_re_timed_while_the_gc_warns_keeps_its_new_expiry_unwarned(
db, make_user, _root, monkeypatch
):
# Moved, un-kept or its folder's retention changed: a fresh expiry, warned when it comes — not
# marked warned now for a date it no longer has.
uid = make_user().id
job = _job(db, uid, _asset(db), expires_at=NOW + timedelta(hours=1))
later = NOW + timedelta(days=30)
def re_timed(other, jid):
moved = other.get(DownloadJob, jid)
moved.expires_at = later
moved.expiry_warned = False
_meanwhile(monkeypatch, re_timed, step="_mark_warned")
assert _gc(db)["warned"] == 0
db.expire_all()
fresh = db.get(DownloadJob, job.id)
assert (fresh.expires_at, fresh.expiry_warned) == (later, False)
assert _notices(db, uid, "download_expiring") == 0
def _two_over_the_cap(db, make_user, _root):
sysconfig.set_value(db, "download_total_max_bytes", 15)
db.commit()
a, b = make_user().id, make_user().id
for name in ("old.mp4", "new.mp4"):
(_root / name).write_bytes(b"x")
old = _asset(db, ref="oldoldoldol", rel="old.mp4", size=10)
new = _asset(db, ref="newnewnewne", rel="new.mp4", size=10)
old.last_access_at = NOW - timedelta(days=9)
new.last_access_at = NOW - timedelta(days=1)
return _job(db, a, old), _job(db, b, new)
def test_a_file_deleted_while_the_cache_is_over_its_cap_frees_that_space_and_no_more(
db, make_user, _root, monkeypatch
):
# Its bytes still counted, the pass went on to evict another user's file for space already free.
jold, jnew = _two_over_the_cap(db, make_user, _root)
def holder_deletes(other, asset_id):
if asset_id == jold.asset_id:
service.delete(other, other.get(DownloadJob, jold.id))
_meanwhile(monkeypatch, holder_deletes, step="_relock_evictable")
assert _gc(db)["evicted"] == 0
db.expire_all()
assert db.get(DownloadJob, jnew.id) is not None
assert (_root / "new.mp4").exists()
def test_a_file_kept_while_the_cache_is_over_its_cap_stays(db, make_user, _root, monkeypatch):
jold, jnew = _two_over_the_cap(db, make_user, _root)
def holder_keeps(other, asset_id):
if asset_id == jold.asset_id:
other.get(DownloadJob, jold.id).kept = True
_meanwhile(monkeypatch, holder_keeps, step="_relock_evictable")
assert _gc(db)["evicted"] == 1 # the newer one goes in its place
db.expire_all()
assert db.get(DownloadJob, jold.id) is not None
assert (_root / "old.mp4").exists()
assert db.get(DownloadJob, jnew.id) is None
def _in_thread(work):
"""Run `work(other_session)` in another thread and session; returns (thread, outcome list)."""
import threading
outcome: list[object] = []
def run():
other = SessionLocal()
try:
outcome.append(work(other))
other.commit()
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=run)
t.start()
return t, outcome
def test_a_retention_changed_while_a_file_goes_in_times_that_file_too(db, make_user):
# The change used to list the folder's files before the arriving one was in, so that one kept the
# old retention — deleted later from a folder now set to keep files forever.
uid = make_user().id
own = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="Mine", retention_days=7)
db.add(own)
db.commit()
own_id = own.id
job = _job(db, uid, _asset(db))
target = folders.own_folder(db, uid, own_id) # held from the check until this commits
t, outcome = _in_thread(
lambda other: folders.set_retention(other, other.get(FileFolder, own_id), None)
)
t.join(timeout=1)
assert t.is_alive() # the change waits for the file being put in
folders.move(db, job, target) # stamped with the 7 days it sees
db.commit()
t.join(timeout=10)
assert outcome == [7] # the retention it replaced
db.expire_all()
assert db.get(DownloadJob, job.id).expires_at is None
def test_a_retention_changed_while_a_file_finishes_there_times_that_file_too(db, make_user):
# The worker's finish read the folder's old retention unlocked, while the change listed the done
# files before this one was among them.
uid = make_user().id
own = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="Mine", retention_days=7)
db.add(own)
db.commit()
own_id = own.id
job = _job(db, uid, _asset(db), status=JobStatus.RUNNING)
folders.move(db, job, own)
db.commit()
job.status = JobStatus.DONE
folders.stamp_done(db, job) # the finish: stamped with the 7 days it reads
db.flush()
t, outcome = _in_thread(
lambda other: folders.set_retention(other, other.get(FileFolder, own_id), None)
)
t.join(timeout=1)
assert t.is_alive() # the change waits for the file being stamped
db.commit()
t.join(timeout=10)
assert outcome == [7] # the retention it replaced
db.expire_all()
assert db.get(DownloadJob, job.id).expires_at is None
def _deleted_uncommitted(db, job):
"""`service.delete` up to its commit: the file is gone in this session, its row still locked."""
service.release_asset(db, job)
db.delete(job)
db.flush()
def test_a_file_moved_out_while_its_folder_is_deleted_stays_where_it_went(db, make_user):
uid = make_user().id
f = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="F")
g = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="G")
db.add_all([f, g])
db.commit()
f_id, g_id = f.id, g.id
job = _job(db, uid, _asset(db))
folders.move(db, job, f)
db.commit()
folders.move(db, job, folders.own_folder(db, uid, g_id))
db.flush() # the move is written, not yet committed
t, outcome = _in_thread(lambda other: folders.delete_own(other, other.get(FileFolder, f_id)))
t.join(timeout=1)
assert t.is_alive() # the delete waits for the file it would take home
db.commit()
t.join(timeout=10)
assert outcome == [0]
db.expire_all()
assert db.get(DownloadJob, job.id).folder_id == g_id
def test_a_file_deleted_while_its_folder_is_deleted_does_not_fail_the_delete(db, make_user, _root):
(_root / "a.mp4").write_bytes(b"x")
uid = make_user().id
f = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="F")
db.add(f)
db.commit()
f_id = f.id
job = _job(db, uid, _asset(db))
folders.move(db, job, f)
db.commit()
_deleted_uncommitted(db, db.get(DownloadJob, job.id))
t, outcome = _in_thread(lambda other: folders.delete_own(other, other.get(FileFolder, f_id)))
t.join(timeout=1)
assert t.is_alive() # the delete waits for the file being deleted
db.commit()
t.join(timeout=10)
assert outcome == [0]
db.expire_all()
assert db.get(FileFolder, f_id) is None
def _own_with_a_file(db, uid, retention=30):
"""A user's own folder keeping files `retention` days, holding one done file due in as many."""
own = FileFolder(
user_id=uid, kind=folders.FolderKind.OWN, name="Mine", retention_days=retention
)
db.add(own)
db.flush()
job = _job(db, uid, _asset(db))
job.folder_id = own.id
job.expires_at = NOW + timedelta(days=retention)
db.commit()
return own.id, job.id
def _elsewhere(act):
other = SessionLocal()
try:
act(other)
other.commit()
finally:
other.close()
def test_a_retention_change_works_from_what_a_change_before_it_left(db, make_user):
# The locked re-read was never fetched, so the change shifted the files from the value read before
# its wait: 30 -> 10 -> 60 left them due in 40 days under a folder saying 60.
own_id, job_id = _own_with_a_file(db, make_user().id)
stale = db.get(FileFolder, own_id) # read before the other change lands
_elsewhere(lambda other: folders.set_retention(other, other.get(FileFolder, own_id), 10))
assert folders.set_retention(db, stale, 60) == 10
db.commit()
db.expire_all()
due = db.get(DownloadJob, job_id).expires_at - NOW
assert timedelta(days=59) < due < timedelta(days=61)
def test_a_retention_change_to_a_folder_deleted_meanwhile_is_refused(db, make_user):
own_id, _ = _own_with_a_file(db, make_user().id)
stale = db.get(FileFolder, own_id)
_elsewhere(lambda other: folders.delete_own(other, other.get(FileFolder, own_id)))
with pytest.raises(folders.FolderError, match="file_folder_gone"):
folders.set_retention(db, stale, 60)
def test_a_file_moved_out_while_its_folder_retention_changes_keeps_its_new_folders(db, make_user):
uid = make_user().id
own_id, job_id = _own_with_a_file(db, uid)
g = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="G", retention_days=7)
db.add(g)
db.commit()
g_id = g.id
folders.move(db, db.get(DownloadJob, job_id), folders.own_folder(db, uid, g_id))
db.flush() # the move is written, not yet committed
t, outcome = _in_thread(
lambda other: folders.set_retention(other, other.get(FileFolder, own_id), None)
)
t.join(timeout=1)
assert t.is_alive() # the change waits for the file it would time
db.commit()
t.join(timeout=10)
assert outcome == [30]
db.expire_all()
job = db.get(DownloadJob, job_id)
assert job.folder_id == g_id
assert job.expires_at is not None # G's 7 days, not the forever of the folder it left
def test_a_file_deleted_while_its_folder_retention_changes_does_not_fail_the_change(
db, make_user, _root
):
(_root / "a.mp4").write_bytes(b"x")
own_id, job_id = _own_with_a_file(db, make_user().id)
_deleted_uncommitted(db, db.get(DownloadJob, job_id))
t, outcome = _in_thread(
lambda other: folders.set_retention(other, other.get(FileFolder, own_id), None)
)
t.join(timeout=1)
assert t.is_alive() # the change waits for the file it would time
db.commit()
t.join(timeout=10)
assert outcome == [30]
def test_a_file_finishing_while_its_folder_is_deleted_gets_its_new_folders_retention(db, make_user):
# The finish read the folder unlocked, then waited out the delete and found no folder: the file
# was done with no expiry at all, in a source folder that keeps files 30 days.
uid = make_user().id
own = FileFolder(user_id=uid, kind=folders.FolderKind.OWN, name="Mine", retention_days=7)
db.add(own)
db.commit()
own_id = own.id
job = _job(db, uid, _asset(db), status=JobStatus.RUNNING)
job_id = job.id
folders.move(db, job, own)
db.commit()
folders.delete_own(db, db.get(FileFolder, own_id))
db.flush() # the delete, not yet committed
t, outcome = _in_thread(
lambda other: service.finalize_if_running(other, job_id, status=JobStatus.DONE)
)
t.join(timeout=1)
assert t.is_alive() # the finish waits for the delete
db.commit()
t.join(timeout=10)
assert outcome == [True]
db.expire_all()
done = db.get(DownloadJob, job_id)
assert done.folder_id != own_id
assert done.expires_at is not None
def _an_upload_over_the_cap_too(db, uid, _root):
(_root / "u.bin").write_bytes(b"x")
return _job(db, uid, _asset(db, "upload", "u1", rel="u.bin", size=10), kind="upload")
def test_space_freed_elsewhere_while_the_cache_is_over_its_cap_evicts_no_more(
db, make_user, _root, monkeypatch
):
# The pass counted the disk once: an upload deleted meanwhile still counted, so a second user's
# file went for space already free.
jold, jnew = _two_over_the_cap(db, make_user, _root)
up = _an_upload_over_the_cap_too(db, make_user().id, _root)
def upload_deleted(other, asset_id):
if asset_id == jold.asset_id:
service.delete(other, other.get(DownloadJob, up.id))
_meanwhile(monkeypatch, upload_deleted, step="_relock_evictable")
assert _gc(db)["evicted"] == 1
db.expire_all()
assert db.get(DownloadJob, jnew.id) is not None
assert (_root / "new.mp4").exists()
def test_a_file_asked_for_again_while_the_cache_is_over_its_cap_stays(
db, make_user, _root, monkeypatch
):
# A cache hit between the pass's SELECT and its lock: the file the user just asked for went.
jold, jnew = _two_over_the_cap(db, make_user, _root)
def asked_again(other, asset_id):
if asset_id == jold.asset_id:
other.get(MediaAsset, asset_id).last_access_at = datetime.now(timezone.utc)
_meanwhile(monkeypatch, asked_again, step="_relock_evictable")
_gc(db)
db.expire_all()
assert db.get(DownloadJob, jold.id) is not None
assert (_root / "old.mp4").exists()
def test_a_file_asked_for_while_its_cached_copy_is_freed_gets_a_fresh_one(db, make_user):
# The enqueue read the cached copy unlocked: freed meanwhile, its job pointed at a row that was gone
# (a foreign-key failure, a 500).
asset = _asset(db)
asset_id, sig = asset.id, asset.format_sig
db.commit()
db.execute(select(MediaAsset).where(MediaAsset.id == asset_id).with_for_update())
db.delete(db.get(MediaAsset, asset_id))
db.flush() # the release, not yet committed
t, outcome = _in_thread(
lambda other: service.get_or_create_asset(other, "youtube", "abcdefghijk", sig).id
)
t.join(timeout=1)
assert t.is_alive() # it waits for the release
db.commit()
t.join(timeout=10)
[fresh] = outcome
assert isinstance(fresh, int) and fresh != asset_id
def test_an_empty_volume_in_the_librarys_place_is_noticed(db, make_user, _root):
# A fresh volume is a real, writable mount: only the files the library names tell it is the wrong one.
assert service.library_left_behind(db, str(_root)) is False # nothing named yet
_asset(db, rel="a.mp4")
db.commit()
assert service.library_left_behind(db, str(_root)) is True
(_root / "a.mp4").write_bytes(b"x")
assert service.library_left_behind(db, str(_root)) is False
def test_deleting_a_user_takes_their_folders_and_files_with_them(db, make_user):
# Both tables cascade from users; the files' folder FK refuses deleting a folder that still holds
# files, which must not stop the account's own deletion.
user = make_user()
_job(db, user.id, _asset(db))
db.delete(user)
db.commit()
assert db.execute(select(func.count()).select_from(FileFolder)).scalar() == 0
assert db.execute(select(func.count()).select_from(DownloadJob)).scalar() == 0
def test_the_longest_host_has_room_for_its_folder(db, make_user):
label = "a" * 63
host = ".".join([label, label, label, "b" * 58]) + ".hu" # 253 characters, the DNS maximum
assert len(host) == 253
uid = make_user().id
job = _job(db, uid, _asset(db, "url", f"https://{host}/v"), kind="url")
assert _folder(db, job).source_key == f"site:{host}"
def test_the_ready_notice_names_the_file_and_its_folder(db, make_user):
# The notice's "Open" goes to the file in its folder, so both ride along as data.
from app.downloads import pipeline
user = make_user()
job = _job(db, user.id, _asset(db))
pipeline._notify_download_done(db, user.id, "A title", job.id, kind="download")
db.commit()
note = db.scalars(select(Notification).where(Notification.user_id == user.id)).one()
assert note.type == "download_done"
assert note.data == {
"title": "A title",
"kind": "download",
"job_id": job.id,
"folder_id": job.folder_id,
}
assert job.folder_id is not None
+670
View File
@@ -0,0 +1,670 @@
"""The Files library API (`routes.files`), DB-lane: folders (list, own create/rename/delete, a
folder's retention), the file list (done only, by folder, searched, sorted, shared with me), a
file's rename / move / keep / delete, the upload into a folder, and the file's hash.
Every route is also held to the caller's own rows: another user's folder or file answers like a
missing one.
"""
import base64
import hashlib
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import select
from app import sysconfig
from app.config import settings
from app.downloads import folders, service
from app.downloads import upload as uploadmod
from app.downloads.status import JobStatus
from app.models import AuditLog, DownloadJob, DownloadShare, FileFolder, MediaAsset
NOW = datetime.now(timezone.utc)
@pytest.fixture
def _root(tmp_path, monkeypatch):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
return tmp_path
def _file(db, user_id, *, kind="youtube", ref=None, title="A title", size=10, rel=None, **kw):
"""A done library file: an asset with bytes on disk-path terms, and the user's job over it."""
n = db.execute(select(MediaAsset.id)).all()
ref = ref or f"vid{len(n):08d}"
asset = MediaAsset(
source_kind=kind,
source_ref=ref,
format_sig=f"sig-{ref}",
status="ready",
rel_path=rel or f"{ref}.mp4",
size_bytes=size,
ref_count=0,
title=title,
media_kind="video",
container="mp4",
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
source_kind=kind,
source_ref=ref,
profile_snapshot={},
status=kw.pop("status", JobStatus.DONE),
**kw,
)
db.add(job)
db.flush()
folders.place(db, job)
folders.stamp_done(db, job)
service.incr_ref_count(db, asset.id)
db.commit()
return job
def _own(db, user_id, name="Saját"):
folder = FileFolder(user_id=user_id, kind=folders.FolderKind.OWN, name=name)
db.add(folder)
db.commit()
return folder
def _audits(db, action):
return db.execute(select(AuditLog).where(AuditLog.action == action)).scalars().all()
# --- folders -------------------------------------------------------------------------------
def test_the_folder_list_counts_each_folders_done_files_and_is_the_callers_own(
db, make_user, as_user
):
alice, bob = make_user(), make_user()
_file(db, alice.id, size=10)
_file(db, alice.id, size=5)
_file(db, alice.id, kind="url", ref="https://vimeo.com/1", size=7)
_file(db, alice.id, status=JobStatus.RUNNING) # not in the library yet
_file(db, bob.id)
got = {
(f["kind"], f["source_key"]): (f["count"], f["size_bytes"])
for f in as_user(alice).get("/api/files/folders").json()
}
assert got == {("source", "youtube"): (2, 15), ("source", "site:vimeo.com"): (1, 7)}
def test_an_own_folder_is_made_once_per_name_whatever_its_case(db, make_user, as_user):
alice = make_user()
c = as_user(alice)
made = c.post("/api/files/folders", json={"name": "Nyaralás"})
assert made.status_code == 200, made.text
assert (made.json()["kind"], made.json()["name"]) == ("own", "Nyaralás")
again = c.post("/api/files/folders", json={"name": "NYARALÁS"})
assert again.status_code == 409
assert again.json()["detail"]["reason"] == "file_folder_name_taken"
blank = c.post("/api/files/folders", json={"name": " "})
assert blank.json()["detail"]["reason"] == "file_folder_name_required"
# Another user may have a folder of the same name.
assert as_user(make_user()).post("/api/files/folders", json={"name": "Nyaralás"}).is_success
assert len(_audits(db, "file_folder_create")) == 2
def test_only_an_own_folder_is_renamed_and_only_by_its_user(db, make_user, as_user):
alice = make_user()
c = as_user(alice)
src = db.get(FileFolder, _file(db, alice.id).folder_id)
own = _own(db, alice.id, "Régi")
taken = _own(db, alice.id, "Foglalt")
ok = c.patch(f"/api/files/folders/{own.id}", json={"name": "Új"})
assert ok.json()["name"] == "Új"
clash = c.patch(f"/api/files/folders/{own.id}", json={"name": taken.name.lower()})
assert clash.json()["detail"]["reason"] == "file_folder_name_taken"
no = c.patch(f"/api/files/folders/{src.id}", json={"name": "YouTube 2"})
assert no.status_code == 409 and no.json()["detail"]["reason"] == "file_folder_not_own"
stranger = as_user(make_user()).patch(f"/api/files/folders/{own.id}", json={"name": "X"})
assert stranger.status_code == 404
db.refresh(own)
assert own.name == "Új"
def test_a_shorter_retention_moves_each_files_expiry_but_never_inside_the_warning_window(
db, make_user, as_user
):
sysconfig.set_value(db, "download_gc_grace_days", 3)
db.commit()
alice = make_user()
fresh = _file(db, alice.id) # expires in 30 days
old = _file(db, alice.id)
old.expires_at = NOW + timedelta(days=10) # 20 days old already
kept = _file(db, alice.id, kept=True)
db.commit()
folder_id = fresh.folder_id
r = as_user(alice).patch(f"/api/files/folders/{folder_id}", json={"retention_days": 20})
assert r.json()["retention_days"] == 20
for j in (fresh, old, kept):
db.refresh(j)
# 30 → 20 days: each loses 10 — the one with 10 left would be due now, so it gets the window.
assert abs(fresh.expires_at - (NOW + timedelta(days=20))) < timedelta(minutes=5)
assert abs(old.expires_at - (NOW + timedelta(days=3))) < timedelta(minutes=5)
assert kept.expires_at is None
forever = as_user(alice).patch(f"/api/files/folders/{folder_id}", json={"retention_days": None})
assert forever.json()["retention_days"] is None
db.refresh(fresh)
assert fresh.expires_at is None
rows = _audits(db, "file_folder_retention")
assert [(a.before, a.after) for a in rows] == [
({"retention_days": 30}, {"retention_days": 20}),
({"retention_days": 20}, {"retention_days": None}),
]
def test_a_retention_set_on_a_folder_that_kept_forever_starts_its_files_clock_now(
db, make_user, as_user
):
alice = make_user()
up = _file(db, alice.id, kind="upload", ref="u1")
assert up.expires_at is None
r = as_user(alice).patch(f"/api/files/folders/{up.folder_id}", json={"retention_days": 7})
assert r.status_code == 200, r.text
db.refresh(up)
assert abs(up.expires_at - (NOW + timedelta(days=7))) < timedelta(minutes=5)
def test_deleting_an_own_folder_sends_its_files_home_and_a_source_folder_cannot_be_deleted(
db, make_user, as_user
):
alice = make_user()
c = as_user(alice)
yt = _file(db, alice.id)
site = _file(db, alice.id, kind="url", ref="https://vimeo.com/2")
home = {yt.id: yt.folder_id, site.id: site.folder_id}
own = _own(db, alice.id)
for j in (yt, site):
assert c.patch(f"/api/files/{j.id}", json={"folder_id": own.id}).is_success
r = c.delete(f"/api/files/folders/{own.id}")
assert r.json() == {"ok": True, "deleted": own.id, "files_moved": 2}
assert db.get(FileFolder, own.id) is None
for j in (yt, site):
db.refresh(j)
assert j.folder_id == home[j.id]
# Back in a source folder, its clock runs again from now.
assert abs(j.expires_at - (NOW + timedelta(days=30))) < timedelta(minutes=5)
src = c.delete(f"/api/files/folders/{yt.folder_id}")
assert src.status_code == 409 and src.json()["detail"]["reason"] == "file_folder_not_own"
other = _own(db, alice.id, "Más")
assert as_user(make_user()).delete(f"/api/files/folders/{other.id}").status_code == 404
assert len(_audits(db, "file_folder_delete")) == 1
# --- the file list -------------------------------------------------------------------------
def test_the_list_holds_the_callers_done_files_by_folder(db, make_user, as_user):
alice, bob = make_user(), make_user()
yt = _file(db, alice.id)
site = _file(db, alice.id, kind="url", ref="https://vimeo.com/3")
_file(db, alice.id, status=JobStatus.QUEUED)
_file(db, bob.id)
c = as_user(alice)
assert {f["id"] for f in c.get("/api/files").json()} == {yt.id, site.id}
only = c.get("/api/files", params={"folder_id": site.folder_id}).json()
assert [f["id"] for f in only] == [site.id]
assert only[0]["folder_id"] == site.folder_id and only[0]["shared"] is False
# Bob's folder id names nothing of Alice's.
bobs = db.execute(select(FileFolder.id).where(FileFolder.user_id == bob.id)).scalar()
assert c.get("/api/files", params={"folder_id": bobs}).json() == []
def test_the_list_is_searched_by_name_and_channel_with_wildcards_taken_literally(
db, make_user, as_user
):
alice = make_user()
hit = _file(db, alice.id, title="Balaton 100% nyár")
by_channel = _file(db, alice.id, title="Más", display_uploader="Balatoni csatorna")
_file(db, alice.id, title="Balaton 1000 nyár")
c = as_user(alice)
assert [f["id"] for f in c.get("/api/files", params={"q": "100%"}).json()] == [hit.id]
found = {f["id"] for f in c.get("/api/files", params={"q": "balatoni"}).json()}
assert found == {by_channel.id}
def test_the_list_sorts_by_name_size_or_when_added(db, make_user, as_user):
alice = make_user()
b = _file(db, alice.id, title="Béka", size=30)
a = _file(db, alice.id, title="alma", size=10)
z = _file(db, alice.id, title="Zebra", size=20)
c = as_user(alice)
def ids(**params):
return [f["id"] for f in c.get("/api/files", params=params).json()]
assert ids() == [z.id, a.id, b.id] # newest first
assert ids(sort="name", order="asc") == [a.id, b.id, z.id] # the reader's alphabet, any case
assert ids(sort="size", order="desc") == [b.id, z.id, a.id]
def test_a_file_is_added_when_it_is_done_not_when_it_was_asked_for(db, make_user, as_user):
"""A download that failed for days and then went through is new: it lists first, dated today."""
alice = make_user()
early = _file(db, alice.id, created_at=NOW - timedelta(days=1))
early.ready_at = NOW - timedelta(days=1)
late = _file(db, alice.id, status=JobStatus.RUNNING, created_at=NOW - timedelta(days=5))
late.ready_at = None
db.commit()
assert service.finalize_if_running(db, late.id, status=JobStatus.DONE)
db.commit()
db.expire_all() # the finalize is a Core UPDATE; the route reads this same session
got = as_user(alice).get("/api/files").json()
assert [f["id"] for f in got] == [late.id, early.id]
ready = datetime.fromisoformat(got[0]["ready_at"])
assert ready > NOW - timedelta(minutes=5)
assert datetime.fromisoformat(got[0]["created_at"]) < NOW - timedelta(days=4)
def test_a_job_made_done_is_ready_as_it_is_made(db, make_user):
# An upload kept as it came is inserted done; the database refuses a done job without the moment.
job = DownloadJob(
user_id=make_user().id,
source_kind="upload",
source_ref="r",
profile_snapshot={},
status=JobStatus.DONE,
)
db.add(job)
db.commit()
assert job.ready_at is not None
def test_shared_with_me_lists_what_others_shared_and_nothing_else(db, make_user, as_user):
alice, bob, carol = make_user(), make_user(), make_user()
theirs = _file(db, bob.id)
_file(db, bob.id) # not shared
for_carol = _file(db, bob.id)
db.add(DownloadShare(job_id=theirs.id, from_user_id=bob.id, to_user_id=alice.id))
db.add(DownloadShare(job_id=for_carol.id, from_user_id=bob.id, to_user_id=carol.id))
db.commit()
got = as_user(alice).get("/api/files", params={"scope": "shared"}).json()
assert [(f["id"], f["shared"]) for f in got] == [(theirs.id, True)]
def test_shared_with_me_is_ordered_by_when_it_was_shared_and_can_be_dismissed(
db, make_user, as_user
):
"""A friend's old file shared today is new to me: "added" is the share's date there. Dismissing
one drops only my grant; the owner's file stays."""
alice, bob = make_user(), make_user()
old = _file(db, bob.id, created_at=NOW - timedelta(days=30))
new = _file(db, bob.id, created_at=NOW - timedelta(days=1))
db.add(
DownloadShare(
job_id=new.id,
from_user_id=bob.id,
to_user_id=alice.id,
created_at=NOW - timedelta(hours=2),
)
)
db.add(
DownloadShare(
job_id=old.id,
from_user_id=bob.id,
to_user_id=alice.id,
created_at=NOW - timedelta(hours=1),
)
)
db.commit()
c = as_user(alice)
got = c.get("/api/files", params={"scope": "shared"}).json()
assert [f["id"] for f in got] == [old.id, new.id]
assert got[0]["shared_at"] is not None and got[0]["shared_at"] > got[1]["shared_at"]
r = c.delete(f"/api/files/shared/{old.id}")
assert r.json() == {"ok": True, "removed": old.id}
assert [f["id"] for f in c.get("/api/files", params={"scope": "shared"}).json()] == [new.id]
assert db.get(DownloadJob, old.id) is not None
def test_each_listed_file_says_where_it_came_from_wherever_it_sits(db, make_user, as_user):
"""The Source column: a file's own source folder, also after it moved into an own folder."""
alice = make_user()
yt = _file(db, alice.id)
site = _file(db, alice.id, kind="url", ref="https://www.tiktok.com/@x/video/1")
moved = _file(db, alice.id)
own = _own(db, alice.id)
c = as_user(alice)
assert c.patch(f"/api/files/{moved.id}", json={"folder_id": own.id}).status_code == 200
got = {f["id"]: f["source_key"] for f in c.get("/api/files").json()}
assert got == {yt.id: "youtube", site.id: "site:tiktok.com", moved.id: "youtube"}
# --- one file ------------------------------------------------------------------------------
def test_a_file_is_renamed_kept_and_given_back_to_its_folders_retention(db, make_user, as_user):
alice = make_user()
job = _file(db, alice.id)
c = as_user(alice)
assert c.patch(f"/api/files/{job.id}", json={"name": "Új név"}).json()["display_name"] == (
"Új név"
)
kept = c.patch(f"/api/files/{job.id}", json={"kept": True}).json()
assert (kept["kept"], kept["expires_at"]) == (True, None)
back = c.patch(f"/api/files/{job.id}", json={"kept": False}).json()
assert back["kept"] is False and back["expires_at"] is not None
# Setting what it already is records nothing.
assert c.patch(f"/api/files/{job.id}", json={"kept": False}).status_code == 200
assert len(_audits(db, "file_keep")) == 1 and len(_audits(db, "file_unkeep")) == 1
assert as_user(make_user()).patch(f"/api/files/{job.id}", json={"kept": True}).status_code == (
404
)
def test_a_files_channel_and_links_are_set_and_only_plausible_links_are_kept(
db, make_user, as_user
):
alice = make_user()
job = _file(db, alice.id)
c = as_user(alice)
got = c.patch(
f"/api/files/{job.id}",
json={
"uploader": " Saját csatorna ",
"uploader_url": "mailto:someone@example.com",
"extra_links": ["https://a.example/x", "ftp://b.example", "", 7, "http://[::1/x"]
+ [f"https://c.example/{i}" for i in range(10)],
},
).json()
assert got["display_uploader"] == "Saját csatorna"
assert got["display_uploader_url"] is None
assert got["extra_links"][0] == "https://a.example/x" and len(got["extra_links"]) == 8
cleared = c.patch(f"/api/files/{job.id}", json={"uploader": "", "extra_links": []}).json()
assert (cleared["display_uploader"], cleared["extra_links"]) == (None, [])
def test_the_usage_carries_the_quota_and_the_upload_cap(db, make_user, as_user):
got = as_user(make_user()).get("/api/files/usage").json()
assert got["upload_max_bytes"] == sysconfig.get_int(db, "download_upload_max_bytes")
assert "footprint_bytes" in got and "max_bytes" in got
def test_a_file_moves_to_an_own_folder_or_home_and_nowhere_else(db, make_user, as_user):
alice, bob = make_user(), make_user()
job = _file(db, alice.id)
home = job.folder_id
vimeo = db.get(FileFolder, _file(db, alice.id, kind="url", ref="https://vimeo.com/4").folder_id)
own = _own(db, alice.id)
bobs = _own(db, bob.id, "Bobé")
c = as_user(alice)
moved = c.patch(f"/api/files/{job.id}", json={"folder_id": own.id}).json()
assert (moved["folder_id"], moved["expires_at"]) == (own.id, None) # an own folder keeps it
wrong = c.patch(f"/api/files/{job.id}", json={"folder_id": vimeo.id})
assert wrong.json()["detail"]["reason"] == "file_folder_not_its_source"
foreign = c.patch(f"/api/files/{job.id}", json={"folder_id": bobs.id})
assert foreign.json()["detail"]["reason"] == "file_folder_gone"
home_again = c.patch(f"/api/files/{job.id}", json={"folder_id": home}).json()
assert home_again["folder_id"] == home and home_again["expires_at"] is not None
assert [(a.before, a.after) for a in _audits(db, "file_move")] == [
({"folder_id": home}, {"folder_id": own.id}),
({"folder_id": own.id}, {"folder_id": home}),
]
def test_a_deleted_file_leaves_the_library_and_the_log_says_so(db, make_user, as_user, _root):
alice = make_user()
job = _file(db, alice.id)
assert as_user(make_user()).delete(f"/api/files/{job.id}").status_code == 404
r = as_user(alice).delete(f"/api/files/{job.id}")
assert r.json() == {"ok": True, "deleted": job.id}
db.expire_all()
assert db.get(DownloadJob, job.id) is None
[row] = _audits(db, "file_delete")
assert (row.actor_id, row.target_id) == (alice.id, str(job.id))
def test_a_file_shared_with_someone_is_theirs_to_watch_not_to_change(db, make_user, as_user, _root):
# The read routes let a share's recipient in; the write routes must not.
alice, friend = make_user(), make_user()
job = _file(db, alice.id, title="Mine")
db.add(DownloadShare(job_id=job.id, from_user_id=alice.id, to_user_id=friend.id))
db.commit()
c = as_user(friend)
assert c.patch(f"/api/files/{job.id}", json={"name": "Theirs"}).status_code == 404
assert c.patch(f"/api/files/{job.id}", json={"kept": True}).status_code == 404
assert c.delete(f"/api/files/{job.id}").status_code == 404
db.expire_all()
still = db.get(DownloadJob, job.id)
assert still is not None and still.display_name is None and not still.kept
# --- upload, the file, its hash ------------------------------------------------------------
@pytest.fixture
def _no_probe(monkeypatch):
# A file ffprobe cannot read: a generic file, done at once (no conversion).
monkeypatch.setattr(uploadmod.probe, "probe_json", lambda _p: None)
def test_an_upload_lands_in_the_folder_it_asked_for_with_its_hash(
db, make_user, as_user, _root, _no_probe, upload_headers
):
alice = make_user()
own = _own(db, alice.id)
body = b"hello files"
c = as_user(alice)
r = c.post("/api/files/upload", content=body, headers=upload_headers("a.txt", folder=own.id))
assert r.status_code == 200, r.text
job = db.get(DownloadJob, r.json()["id"])
assert job.folder_id == own.id and job.expires_at is None
asset = db.get(MediaAsset, job.asset_id)
assert asset.sha256 == hashlib.sha256(body).hexdigest()
plain = c.post("/api/files/upload", content=b"x", headers=upload_headers("b.txt")).json()
assert db.get(FileFolder, plain["folder_id"]).source_key == "upload"
assert len(_audits(db, "file_upload")) == 2
@pytest.mark.parametrize(
("which", "reason"),
[
("site", "file_folder_not_its_source"), # YouTube's folder: no upload came from there
("foreign", "file_folder_gone"),
("garbage", "file_folder_gone"),
],
)
def test_an_upload_to_a_folder_it_may_not_use_is_refused_before_its_body_is_read(
db, make_user, as_user, _root, _no_probe, upload_headers, monkeypatch, which, reason
):
from app.routes import files as files_routes
alice, bob = make_user(), make_user()
folder = {
"site": lambda: _file(db, alice.id).folder_id,
"foreign": lambda: _own(db, bob.id).id,
"garbage": lambda: "12abc",
}[which]()
streamed = []
async def spy(*args, **kwargs):
streamed.append(True)
raise AssertionError("the body was read")
monkeypatch.setattr(files_routes, "capped_stream_to", spy)
before = db.execute(select(DownloadJob.id)).all()
r = as_user(alice).post(
"/api/files/upload", content=b"data", headers=upload_headers("a.txt", folder=folder)
)
assert (r.status_code, r.json()["detail"]["reason"]) == (409, reason)
assert streamed == []
assert db.execute(select(DownloadJob.id)).all() == before
def test_an_upload_holds_its_folder_only_once_its_file_is_about_to_be_in_it(
db, make_user, as_user, _root, upload_headers, monkeypatch
):
# Held from the start, the folder waited out the probe and the poster (up to a minute) before a
# rename or a delete of it could go on; not held at all, a delete could take it from under the file.
alice = make_user()
own = _own(db, alice.id)
steps: list[object] = []
real = folders.own_folder
def own_folder(db_, user_id, folder_id, *, hold=True):
steps.append(("folder", hold))
return real(db_, user_id, folder_id, hold=hold)
def probe(_p):
steps.append("probe")
monkeypatch.setattr(folders, "own_folder", own_folder)
monkeypatch.setattr(uploadmod.probe, "probe_json", probe)
r = as_user(alice).post(
"/api/files/upload", content=b"x", headers=upload_headers("a.txt", folder=own.id)
)
assert r.status_code == 200, r.text
# The route's own look before the body, the ingest's look, the probe, then the hold.
assert steps[-3:] == [("folder", False), "probe", ("folder", True)]
def test_a_file_moved_holds_the_folder_it_goes_to(db, make_user, as_user, monkeypatch):
alice = make_user()
job = _file(db, alice.id)
own = _own(db, alice.id)
holds: list[bool] = []
real = folders.own_folder
def own_folder(db_, user_id, folder_id, *, hold=True):
holds.append(hold)
return real(db_, user_id, folder_id, hold=hold)
monkeypatch.setattr(folders, "own_folder", own_folder)
r = as_user(alice).patch(f"/api/files/{job.id}", json={"folder_id": own.id})
assert r.status_code == 200, r.text
assert holds == [True]
def test_a_folder_deleted_while_its_upload_streams_refuses_the_upload(
db, make_user, as_user, _root, _no_probe, upload_headers, monkeypatch
):
from app.routes import files as files_routes
alice = make_user()
own = _own(db, alice.id)
real = files_routes.capped_stream_to
async def deleting_meanwhile(*args, **kwargs):
size = await real(*args, **kwargs)
db.delete(db.get(FileFolder, own.id))
db.commit()
return size
monkeypatch.setattr(files_routes, "capped_stream_to", deleting_meanwhile)
r = as_user(alice).post(
"/api/files/upload", content=b"data", headers=upload_headers("a.txt", folder=own.id)
)
assert (r.status_code, r.json()["detail"]["reason"]) == (409, "file_folder_gone")
assert db.execute(select(DownloadJob.id).where(DownloadJob.user_id == alice.id)).all() == []
assert not (_root / ".uploads").exists() # checked before the file was placed
def test_the_file_carries_its_hash_once_known_and_the_hash_is_taken_on_first_ask(
db, make_user, as_user, _root
):
alice, friend = make_user(), make_user()
body = b"\x00\x01video bytes"
job = _file(db, alice.id, rel="v.mp4")
(_root / "v.mp4").write_bytes(body)
c = as_user(alice)
first = c.get(f"/api/files/{job.id}/file")
assert first.status_code == 200 and first.content == body
assert "repr-digest" not in first.headers # not hashed yet: nothing claimed
want = hashlib.sha256(body).hexdigest()
assert c.get(f"/api/files/{job.id}/sha256").json() == {"sha256": want}
assert db.get(MediaAsset, job.asset_id).sha256 == want
ranged = c.get(f"/api/files/{job.id}/file", headers={"Range": "bytes=0-3"})
assert ranged.status_code == 206
digest = base64.b64encode(bytes.fromhex(want)).decode()
assert ranged.headers["repr-digest"] == f"sha-256=:{digest}:"
# A friend it was shared with may read it too; a stranger is told it does not exist.
db.add(DownloadShare(job_id=job.id, from_user_id=alice.id, to_user_id=friend.id))
db.commit()
assert as_user(friend).get(f"/api/files/{job.id}/sha256").json() == {"sha256": want}
assert as_user(make_user()).get(f"/api/files/{job.id}/sha256").status_code == 404
def test_only_a_few_files_are_hashed_at_once(db, make_user, as_user, _root):
# Each hash holds a thread of the pool every sync route shares for the whole read; a caller
# past the few that may run is told to come back, not queued behind them.
from app.routes import files as filesroute
alice = make_user()
job = _file(db, alice.id, rel="v.mp4")
(_root / "v.mp4").write_bytes(b"bytes")
held = [filesroute._HASHING.acquire(blocking=False) for _ in range(filesroute._HASHERS)]
try:
assert all(held)
busy = as_user(alice).get(f"/api/files/{job.id}/sha256")
finally:
for _ in held:
filesroute._HASHING.release()
assert busy.status_code == 429
assert busy.json()["detail"]["reason"] == "file_hash_busy"
assert as_user(alice).get(f"/api/files/{job.id}/sha256").status_code == 200
def test_a_new_file_at_an_assets_path_forgets_the_old_hash(db, make_user):
job = _file(db, make_user().id)
asset = db.get(MediaAsset, job.asset_id)
asset.sha256 = "ab" * 32
asset.rel_path = "converted.mp4"
assert asset.sha256 is None
# --- the queue (the tray's poll) ------------------------------------------------------------------
def test_the_queue_is_what_is_in_flight_and_what_just_finished_not_the_library(
db, make_user, as_user
):
# The tray polls this on every page: a finished file is the library's, so it is listed only
# while the tray may still be showing it done.
from sqlalchemy import update
user = make_user()
old = _file(db, user.id, title="Old")
fresh = _file(db, user.id, title="Fresh")
waiting = _file(db, user.id, title="Waiting", status=JobStatus.QUEUED)
stuck = _file(db, user.id, title="Stuck", status=JobStatus.ERROR)
paused = _file(db, user.id, title="Paused", status=JobStatus.PAUSED)
stopped = _file(db, user.id, title="Stopped", status=JobStatus.ERROR)
dropped = _file(db, user.id, title="Dropped", status=JobStatus.CANCELED)
for job in (old, stopped, dropped):
db.execute(
update(DownloadJob)
.where(DownloadJob.id == job.id)
.values(updated_at=NOW - timedelta(hours=1), ready_at=NOW - timedelta(hours=1))
)
db.commit()
rows = as_user(user).get("/api/downloads").json()
# An old failure still waits for the user; an old finished or cancelled job does not.
assert {row["id"] for row in rows} == {fresh.id, waiting.id, stuck.id, paused.id, stopped.id}
# The tray names where each came from, as the library does.
assert {row["source_key"] for row in rows} == {"youtube"}
def test_an_old_file_written_to_does_not_come_back_as_just_finished(db, make_user, as_user):
# Renaming, keeping or moving touches the row; the tray would announce the file as ready again.
from sqlalchemy import update
user = make_user()
old = _file(db, user.id, title="Old")
db.execute(
update(DownloadJob)
.where(DownloadJob.id == old.id)
.values(updated_at=NOW - timedelta(days=30), ready_at=NOW - timedelta(days=30))
)
db.commit()
c = as_user(user)
assert (
c.patch(f"/api/files/{old.id}", json={"name": "Renamed", "kept": True}).status_code == 200
)
assert [row["id"] for row in c.get("/api/downloads").json()] == []
+174
View File
@@ -0,0 +1,174 @@
"""A library file's preview (`GET /api/files/{id}/preview`, `downloads/preview.py`), DB-lane.
What it may show is an allow-list on the stored extension: an image or a PDF inline, a text format
(HTML and SVG among them) as plain text, its head only. Every answer is sandboxed and may be framed
by the app alone; anything else, a binary "text" included, has no preview.
"""
import pytest
from sqlalchemy import select
from app.config import settings
from app.downloads import folders, preview, service
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadShare, MediaAsset
from app.routes.files import TRUNCATED_HEADER
@pytest.fixture
def _root(tmp_path, monkeypatch):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
return tmp_path
def _upload(db, root, user_id, ext, body, *, media_kind="file", title="notes"):
"""A done uploaded file named `title.ext` holding `body`."""
n = len(db.execute(select(MediaAsset.id)).all())
rel = f"up{n}.{ext}" if ext else f"up{n}"
(root / rel).write_bytes(body)
asset = MediaAsset(
source_kind="upload",
source_ref=f"up{n}",
format_sig=f"sig-up{n}",
status="ready",
rel_path=rel,
size_bytes=len(body),
ref_count=0,
title=title,
media_kind=media_kind,
container=ext,
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
source_kind="upload",
source_ref=f"up{n}",
profile_snapshot={},
status=JobStatus.DONE,
)
db.add(job)
db.flush()
folders.place(db, job)
service.incr_ref_count(db, asset.id)
db.commit()
return job
def _assert_sandboxed(r):
assert r.headers["content-security-policy"] == "sandbox; frame-ancestors 'self'"
assert r.headers["x-frame-options"] == "SAMEORIGIN"
assert r.headers["x-content-type-options"] == "nosniff"
@pytest.mark.parametrize(
"ext,ctype",
[
("png", "image/png"),
("JPG", "image/jpeg"),
("webp", "image/webp"),
("pdf", "application/pdf"),
],
)
def test_an_image_or_a_pdf_is_shown_inline_with_its_listed_type_in_a_sandbox(
db, make_user, as_user, _root, ext, ctype
):
alice = make_user()
job = _upload(db, _root, alice.id, ext, b"bytes of it")
r = as_user(alice).get(f"/api/files/{job.id}/preview")
assert r.status_code == 200 and r.content == b"bytes of it"
assert r.headers["content-type"] == ctype
assert r.headers["content-disposition"].startswith("inline;")
assert f'filename="notes.{ext}"' in r.headers["content-disposition"]
_assert_sandboxed(r)
@pytest.mark.parametrize("ext", ["html", "svg", "txt", "py"])
def test_html_svg_and_code_arrive_as_plain_text_never_as_a_page(db, make_user, as_user, _root, ext):
alice = make_user()
body = b"<svg onload=x><script>y</script></svg>"
job = _upload(db, _root, alice.id, ext, body)
r = as_user(alice).get(f"/api/files/{job.id}/preview")
assert r.status_code == 200 and r.content == body
assert r.headers["content-type"] == "text/plain; charset=utf-8"
assert TRUNCATED_HEADER.lower() not in r.headers
_assert_sandboxed(r)
def test_a_long_text_is_cut_to_its_head_and_says_so(db, make_user, as_user, _root):
alice = make_user()
body = b"a" * preview.TEXT_PREVIEW_BYTES + b"tail"
job = _upload(db, _root, alice.id, "log", body)
r = as_user(alice).get(f"/api/files/{job.id}/preview")
assert r.status_code == 200 and len(r.content) == preview.TEXT_PREVIEW_BYTES
assert r.headers[TRUNCATED_HEADER] == "1"
exact = _upload(db, _root, alice.id, "log", b"a" * preview.TEXT_PREVIEW_BYTES)
whole = as_user(alice).get(f"/api/files/{exact.id}/preview")
assert len(whole.content) == preview.TEXT_PREVIEW_BYTES
assert TRUNCATED_HEADER.lower() not in whole.headers
@pytest.mark.parametrize(
"ext,body,media_kind",
[
("zip", b"PK\x03\x04", "file"), # not on the list
("docx", b"PK\x03\x04", "file"),
(None, b"plain", "file"), # no extension
("txt", b"text\x00with a nul", "file"), # a "text" that is binary
("mp4", b"video", "video"), # media plays in the player instead
("png", b"png", "video"), # an extension alone never makes media a preview
],
)
def test_anything_else_has_no_preview(db, make_user, as_user, _root, ext, body, media_kind):
alice = make_user()
job = _upload(db, _root, alice.id, ext, body, media_kind=media_kind)
r = as_user(alice).get(f"/api/files/{job.id}/preview")
assert r.status_code == 415
assert r.json()["detail"] == {"code": "refused", "reason": "file_no_preview"}
@pytest.mark.parametrize(
"body",
[
"Árvíztűrő tükörfúrógép".encode("cp1250"), # an old Hungarian subtitle
"Árvíztűrő tükörfúrógép".encode("utf-8-sig"),
"Árvíztűrő tükörfúrógép".encode("utf-16"), # a BOM, then NULs: text all the same
"Árvíztűrő tükörfúrógép".encode("utf-8"),
],
)
def test_a_text_arrives_as_utf8_whatever_it_was_written_in(db, make_user, as_user, _root, body):
alice = make_user()
job = _upload(db, _root, alice.id, "srt", body)
r = as_user(alice).get(f"/api/files/{job.id}/preview")
assert r.status_code == 200
assert r.content.decode("utf-8") == "Árvíztűrő tükörfúrógép"
def test_a_character_cut_at_the_cap_keeps_the_text_utf8(db, make_user, as_user, _root):
alice = make_user()
body = b"a" * (preview.TEXT_PREVIEW_BYTES - 1) + "ő".encode() # its 2nd byte past the cap
job = _upload(db, _root, alice.id, "txt", body)
r = as_user(alice).get(f"/api/files/{job.id}/preview")
assert r.content == body[: preview.TEXT_PREVIEW_BYTES] # not read as cp1250
def test_the_list_says_which_preview_each_file_gets(db, make_user, as_user, _root):
alice = make_user()
for ext in ["png", "pdf", "html", "zip"]:
_upload(db, _root, alice.id, ext, b"x", title=ext)
_upload(db, _root, alice.id, "mp4", b"x", media_kind="video", title="mp4")
got = {
f["asset"]["container"]: f["asset"]["preview"]
for f in as_user(alice).get("/api/files").json()
}
assert got == {"png": "image", "pdf": "pdf", "html": "text", "zip": None, "mp4": None}
def test_a_preview_is_the_holders_or_a_friends_it_was_shared_with(db, make_user, as_user, _root):
alice, friend = make_user(), make_user()
job = _upload(db, _root, alice.id, "png", b"img")
assert as_user(make_user()).get(f"/api/files/{job.id}/preview").status_code == 404
db.add(DownloadShare(job_id=job.id, from_user_id=alice.id, to_user_id=friend.id))
db.commit()
assert as_user(friend).get(f"/api/files/{job.id}/preview").status_code == 200
+397
View File
@@ -0,0 +1,397 @@
"""A `/watch` link to one of its owner's own folders (routes/downloads.py, routes/public.py), DB-lane.
The link shows what is in the folder when it is opened: a file moved in appears, one moved out stops
opening at its next request. Only an own folder can be shared — a source folder lets its files go on
its own clock, the Hub is the channel to a machine. A password, an expiry and stream-only work as on
a file link; a document is never served to open as a page.
"""
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
from alembic import command
from alembic.config import Config
from sqlalchemy import select, text
from app.config import settings
from app.db import engine
from app.downloads import folders, og
from app.downloads import probe as probemod
from app.models import AuditLog, DownloadJob, DownloadLink, FileFolder, MediaAsset
_VIDEO = {
"streams": [
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
{"codec_type": "audio", "codec_name": "aac"},
],
"format": {"format_name": "mov,mp4", "duration": "5"},
}
@pytest.fixture
def probe(monkeypatch):
"""What ffprobe says the next upload is: a phone-safe video, or (None) a file it cannot read."""
said = {"data": _VIDEO}
monkeypatch.setattr(probemod, "probe_json", lambda _p: said["data"])
return said
@pytest.fixture
def shelf(as_user, make_user, tmp_path, monkeypatch, upload_headers, probe):
"""An owner with an own folder "Friends" holding two videos, a third video in their Uploads, and
a function to upload more."""
monkeypatch.setattr(settings, "files_root", str(tmp_path))
user = make_user()
owner = as_user(user)
folder = owner.post("/api/files/folders", json={"name": "Friends"}).json()
def upload(name: str, folder_id: int | None = None, body: bytes = b"fake-video") -> int:
r = owner.post(
"/api/files/upload", content=body, headers=upload_headers(name, folder=folder_id)
)
assert r.status_code == 200, r.text
return r.json()["id"]
inside = [upload("a.mp4", folder["id"]), upload("b.mp4", folder["id"])]
outside = upload("c.mp4")
return {
"user": user,
"owner": owner,
"folder": folder["id"],
"inside": inside,
"outside": outside,
"upload": upload,
}
def _link(shelf, **body) -> dict:
r = shelf["owner"].post(f"/api/downloads/folders/{shelf['folder']}/links", json=body)
assert r.status_code == 200, r.text
return r.json()
def _ids(client, token: str) -> list[int]:
meta = client.get(f"/api/public/watch/{token}").json()
return [f["id"] for f in meta["files"]]
def test_a_folder_link_lists_what_is_in_the_folder(shelf, client):
token = _link(shelf)["token"]
meta = client.get(f"/api/public/watch/{token}").json()
assert meta["kind"] == "folder"
assert meta["name"] == "Friends"
assert sorted(f["id"] for f in meta["files"]) == sorted(shelf["inside"])
# Each one plays and downloads through the link, under its own id.
for f in meta["files"]:
assert f["file_url"] == f"/api/public/watch/{token}/files/{f['id']}/file"
served = client.get(f["file_url"])
assert served.status_code == 200
assert served.content == b"fake-video"
def test_a_file_outside_the_folder_does_not_open_through_it(shelf, client):
token = _link(shelf)["token"]
r = client.get(f"/api/public/watch/{token}/files/{shelf['outside']}/file")
assert r.status_code == 404
# Nor does the folder link open as a file link.
assert client.get(f"/api/public/watch/{token}/file").status_code == 404
def test_the_link_follows_the_folder_as_files_move(shelf, client):
token = _link(shelf)["token"]
owner, moved_out = shelf["owner"], shelf["inside"][0]
other = owner.post("/api/files/folders", json={"name": "Other"}).json()["id"]
def move(job_id: int, folder_id: int) -> None:
r = owner.patch(f"/api/files/{job_id}", json={"folder_id": folder_id})
assert r.status_code == 200, r.text
move(shelf["outside"], shelf["folder"])
move(moved_out, other)
assert sorted(_ids(client, token)) == sorted([shelf["inside"][1], shelf["outside"]])
assert client.get(f"/api/public/watch/{token}/files/{moved_out}/file").status_code == 404
url = f"/api/public/watch/{token}/files/{shelf['outside']}/file"
assert client.get(url).status_code == 200
def test_only_a_folder_of_ones_own_can_be_shared(shelf, db, as_user, make_user):
owner = shelf["owner"]
source = db.execute(
select(FileFolder.id).where(
FileFolder.user_id == shelf["user"].id, FileFolder.kind == "source"
)
).scalar_one()
r = owner.post(f"/api/downloads/folders/{source}/links", json={})
assert r.status_code == 409
assert r.json()["detail"]["reason"] == "share_folder_kind"
# Nor the Hub: whatever a machine drops there would be public the moment it lands.
hub = folders.hub_folder(db, shelf["user"].id).id
db.commit()
r = owner.post(f"/api/downloads/folders/{hub}/links", json={})
assert r.status_code == 409
assert r.json()["detail"]["reason"] == "share_folder_kind"
stranger = as_user(make_user())
assert (
stranger.post(f"/api/downloads/folders/{shelf['folder']}/links", json={}).status_code == 404
)
assert stranger.get(f"/api/downloads/folders/{shelf['folder']}/links").status_code == 404
link_id = _link(shelf)["id"]
patch = stranger.patch(f"/api/downloads/links/{link_id}", json={"allow_download": False})
assert patch.status_code == 404
assert stranger.delete(f"/api/downloads/links/{link_id}").status_code == 404
def test_a_password_folder_link_shows_nothing_until_unlocked(shelf, client):
token = _link(shelf, password="open sesame")["token"]
assert client.get(f"/api/public/watch/{token}").json() == {
"needs_password": True,
"kind": "folder",
}
bare = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/file"
assert client.get(bare).status_code == 403
unlocked = client.post(f"/api/public/watch/{token}/unlock", json={"password": "open sesame"})
assert unlocked.status_code == 200
files = unlocked.json()["files"]
assert len(files) == 2
assert all("?g=" in f["file_url"] for f in files)
assert client.get(files[0]["file_url"]).status_code == 200
def test_a_files_poster_opens_by_the_links_own_rules(shelf, db, client):
"""The poster route on its own: a password link's poster needs the unlock grant, and a file
outside the folder has none through it, grant or not."""
for job_id in (shelf["inside"][0], shelf["outside"]):
asset = db.get(MediaAsset, db.get(DownloadJob, job_id).asset_id)
asset.poster_path = f"poster-{job_id}.jpg"
(Path(settings.files_root) / asset.poster_path).write_bytes(b"a poster frame")
db.commit()
token = _link(shelf, password="open sesame")["token"]
inside = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/poster.jpg"
assert client.get(inside).status_code == 403
unlocked = client.post(f"/api/public/watch/{token}/unlock", json={"password": "open sesame"})
[entry] = [f for f in unlocked.json()["files"] if f["id"] == shelf["inside"][0]]
assert entry["poster_url"].startswith(inside + "?g=")
got = client.get(entry["poster_url"])
assert got.status_code == 200 and got.content == b"a poster frame"
grant = entry["poster_url"].split("?g=", 1)[1]
outside = f"/api/public/watch/{token}/files/{shelf['outside']}/poster.jpg?g={grant}"
assert client.get(outside).status_code == 404
def test_every_open_counts_as_a_view(shelf, client):
made = _link(shelf)
for _ in range(3):
assert client.get(f"/api/public/watch/{made['token']}").status_code == 200
[listed] = shelf["owner"].get(f"/api/downloads/folders/{shelf['folder']}/links").json()
assert listed["view_count"] == 3
def test_the_folder_list_is_capped_like_the_owners_files_list():
from app.downloads import links as linksmod
from app.routes.files import MAX_LIST
assert linksmod.FOLDER_MAX == MAX_LIST
def test_a_document_in_a_shared_folder_only_downloads(shelf, client, probe):
probe["data"] = None
doc = shelf["upload"]("notes.html", shelf["folder"], b"<script>1</script>")
token = _link(shelf, allow_download=False)["token"]
assert doc in _ids(client, token)
served = client.get(f"/api/public/watch/{token}/files/{doc}/file")
assert served.status_code == 200
assert served.headers["content-type"].startswith("application/octet-stream")
assert "attachment" in served.headers["content-disposition"].lower()
def test_a_stream_only_folder_link_plays_inline(shelf, client):
token = _link(shelf, allow_download=False)["token"]
meta = client.get(f"/api/public/watch/{token}").json()
assert meta["allow_download"] is False
served = client.get(meta["files"][0]["file_url"])
assert "inline" in served.headers["content-disposition"].lower()
def test_an_expired_or_deleted_folder_link_is_gone(shelf, client, db):
made = _link(shelf)
db.get(DownloadLink, made["id"]).expires_at = datetime.now(timezone.utc) - timedelta(minutes=1)
db.commit()
assert client.get(f"/api/public/watch/{made['token']}").status_code == 404
token = _link(shelf)["token"]
assert shelf["owner"].delete(f"/api/files/folders/{shelf['folder']}").status_code == 200
assert client.get(f"/api/public/watch/{token}").status_code == 404
db.expire_all()
assert db.execute(select(DownloadLink)).scalars().all() == []
def test_the_folder_list_marks_a_folder_a_live_link_opens(shelf, db):
owner = shelf["owner"]
def marked() -> dict[int, bool]:
return {f["id"]: f["has_link"] for f in owner.get("/api/files/folders").json()}
assert marked()[shelf["folder"]] is False
made = _link(shelf)
assert marked()[shelf["folder"]] is True
# A rename answers with the mark too, so the rail does not lose it.
renamed = owner.patch(f"/api/files/folders/{shelf['folder']}", json={"name": "Pals"}).json()
assert renamed["has_link"] is True
db.get(DownloadLink, made["id"]).expires_at = datetime.now(timezone.utc) - timedelta(minutes=1)
db.commit()
assert marked()[shelf["folder"]] is False
live = _link(shelf)
assert owner.delete(f"/api/downloads/links/{live['id']}").status_code == 200
assert marked()[shelf["folder"]] is False
assert not any(marked()[f] for f in marked() if f != shelf["folder"])
def test_making_and_revoking_a_folder_link_is_audited(shelf, db):
made = _link(shelf)
assert shelf["owner"].delete(f"/api/downloads/links/{made['id']}").status_code == 200
db.expire_all()
rows = {
a.action: a.after
for a in db.execute(
select(AuditLog).where(AuditLog.action.like("download_link_%"))
).scalars()
}
assert rows["download_link_create"]["folder_id"] == shelf["folder"]
assert "job_id" not in rows["download_link_create"]
assert rows["download_link_revoke"] == {"folder_id": shelf["folder"]}
def test_the_unfurl_names_the_folder_and_counts_its_files_only(shelf, db):
token = _link(shelf)["token"]
tags = og._watch_tags(token, db)
assert "Friends" in tags
assert "2 files" in tags
names = db.execute(
select(DownloadJob.display_name).where(DownloadJob.id.in_(shelf["inside"]))
).scalars()
for name in names:
assert not name or name not in tags
# A file whose bytes are gone is off the page, and so off the card's count.
gone = db.get(MediaAsset, db.get(DownloadJob, shelf["inside"][0]).asset_id)
(Path(settings.files_root) / gone.rel_path).unlink()
assert "1 file " in og._watch_tags(token, db)
locked = _link(shelf, password="pw")["token"]
hidden = og._watch_tags(locked, db)
assert "Friends" not in hidden and "Password-protected folder" in hidden
assert 'content="website"' in hidden and "video.other" not in hidden
def test_the_downgrade_drops_folder_links_and_keeps_file_links(shelf, db, client):
_link(shelf)
file_link = shelf["owner"].post(f"/api/downloads/{shelf['outside']}/links", json={}).json()
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
cfg = Config("alembic.ini")
command.downgrade(cfg, "0107_link_token_at_rest")
try:
with engine.connect() as conn:
left = conn.execute(text("SELECT job_id FROM download_links")).scalars().all()
finally:
command.upgrade(cfg, "head")
assert left == [shelf["outside"]]
assert client.get(f"/api/public/watch/{file_link['token']}").status_code == 200
def test_a_password_file_link_says_it_opens_a_file(shelf, client):
made = (
shelf["owner"]
.post(f"/api/downloads/{shelf['outside']}/links", json={"password": "pw"})
.json()
)
meta = client.get(f"/api/public/watch/{made['token']}").json()
assert meta == {"needs_password": True, "kind": "file"}
def test_a_file_whose_bytes_are_gone_is_not_offered(shelf, client, db):
from pathlib import Path
from app.models import MediaAsset
token = _link(shelf)["token"]
gone = shelf["inside"][0]
asset = db.get(MediaAsset, db.get(DownloadJob, gone).asset_id)
(Path(settings.files_root) / asset.rel_path).unlink()
assert _ids(client, token) == [shelf["inside"][1]]
assert client.get(f"/api/public/watch/{token}/files/{gone}/file").status_code == 404
def _meanwhile(work):
"""Run `work(other_session)` in another thread and session, committed; (thread, outcome)."""
import threading
from app.db import SessionLocal
outcome: list[object] = []
def run():
other = SessionLocal()
try:
outcome.append(work(other))
other.commit()
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=run)
t.start()
return t, outcome
def test_two_opens_at_once_count_as_two_views(shelf, db):
from app.routes import public
made = _link(shelf)
link = db.get(DownloadLink, made["id"])
public._opened(db, link, made["token"]) # this open's count is not committed yet
t, outcome = _meanwhile(
lambda other: public._opened(other, other.get(DownloadLink, made["id"]), made["token"])
)
t.join(timeout=1)
db.commit()
t.join(timeout=10)
assert not isinstance(outcome[0], Exception), outcome
db.expire_all()
assert db.get(DownloadLink, made["id"]).view_count == 2
def test_a_file_shared_as_it_is_deleted_takes_its_new_link_with_it(shelf, db):
from app.routes import downloads as routes
job_id = shelf["outside"]
routes._shareable_job(db, shelf["user"], job_id) # held from the check until this commits
t, outcome = _meanwhile(lambda other: other.delete(other.get(DownloadJob, job_id)))
t.join(timeout=1)
assert t.is_alive() # the delete waits for the link on its way in
db.add(DownloadLink(job_id=job_id, token_hash="h" * 64, created_by=shelf["user"].id))
db.commit()
t.join(timeout=10)
assert outcome == [None]
assert db.execute(select(DownloadLink.id)).scalars().all() == []
def test_a_folder_shared_as_it_is_deleted_takes_its_new_link_with_it(shelf, db):
from app.routes import downloads as routes
fid = shelf["folder"]
routes._shareable_folder(db, shelf["user"], fid) # held from the check until this commits
t, outcome = _meanwhile(lambda other: folders.delete_own(other, other.get(FileFolder, fid)))
t.join(timeout=1)
assert t.is_alive() # the delete waits for the link on its way in
db.add(DownloadLink(folder_id=fid, token_hash="h" * 64, created_by=shelf["user"].id))
db.commit()
t.join(timeout=10)
assert not isinstance(outcome[0], Exception), outcome
assert db.execute(select(DownloadLink.id)).scalars().all() == []
+21 -5
View File
@@ -9,13 +9,26 @@ cannot reach, is ever read or written. DB-backed (real Postgres).
import pytest
from sqlalchemy import select
from app.models import DownloadJob, DownloadShare, LibraryState
from app.models import DownloadJob, DownloadShare, LibraryState, MediaAsset
from app.routes._common import ProgressIn, kept_position
def _done_job(db, user) -> DownloadJob:
asset = MediaAsset(
source_kind="url",
source_ref="https://example.com/a.mp4",
format_sig=f"sig-{user.id}",
status="ready",
rel_path=f"a-{user.id}.mp4",
ref_count=1,
media_kind="video",
container="mp4",
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=user.id,
asset_id=asset.id,
source_kind="url",
source_ref="https://example.com/a.mp4",
status="done",
@@ -59,7 +72,7 @@ def test_a_checkpoint_is_listed_back_on_the_item(db, make_user, as_user):
"position_seconds": 754,
}
[item] = c.get("/api/downloads").json()
[item] = c.get("/api/files").json()
assert item["position_seconds"] == 754
# A second checkpoint moves it rather than adding a row.
assert _checkpoint(c, job.id, 900) == 900
@@ -69,7 +82,7 @@ def test_a_checkpoint_is_listed_back_on_the_item(db, make_user, as_user):
def test_an_item_never_played_lists_zero(db, make_user, as_user):
me = make_user()
_done_job(db, me)
[item] = as_user(me).get("/api/downloads").json()
[item] = as_user(me).get("/api/files").json()
assert item["position_seconds"] == 0
@@ -110,8 +123,11 @@ def test_a_recipient_keeps_their_own_place_in_a_shared_item(db, make_user, as_us
assert _checkpoint(as_user(owner), job.id, 100) == 100
assert _checkpoint(as_user(friend), job.id, 2000) == 2000
assert as_user(owner).get("/api/downloads").json()[0]["position_seconds"] == 100
assert as_user(friend).get("/api/downloads/shared").json()[0]["position_seconds"] == 2000
assert as_user(owner).get("/api/files").json()[0]["position_seconds"] == 100
assert (
as_user(friend).get("/api/files", params={"scope": "shared"}).json()[0]["position_seconds"]
== 2000
)
def test_an_item_the_caller_cannot_play_is_a_404_and_writes_nothing(db, make_user, as_user):
+227
View File
@@ -0,0 +1,227 @@
"""A `/watch` link's token at rest (downloads/links.py, migration 0107), DB-lane.
The token is the credential, so the database keeps only its SHA-256 (to find the link by) and a
Fernet-encrypted copy (for its owner to copy the same link again). Without a copy this key can read,
the link still opens and only its address cannot be shown again. Making, changing and revoking a
link is audited, never with its token or password.
"""
import json
import pytest
from alembic import command
from alembic.config import Config
from cryptography.fernet import Fernet
from sqlalchemy import select, text
from app import security
from app.config import settings
from app.db import engine
from app.downloads import probe as probemod
from app.models import AuditLog, DownloadLink
_VIDEO = {
"streams": [
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
{"codec_type": "audio", "codec_name": "aac"},
],
"format": {"format_name": "mov,mp4", "duration": "5"},
}
@pytest.fixture
def key(monkeypatch):
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
@pytest.fixture
def owner_job(as_user, make_user, tmp_path, monkeypatch, upload_headers):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
monkeypatch.setattr(probemod, "probe_json", lambda _p: _VIDEO)
owner = as_user(make_user())
job = owner.post("/api/files/upload", content=b"fake-video", headers=upload_headers("clip.mp4"))
assert job.status_code == 200, job.text
return owner, job.json()["id"]
def _row(db, link_id: int) -> dict:
db.expire_all()
row = db.execute(
text("SELECT * FROM download_links WHERE id = :id"), {"id": link_id}
).mappings()
return dict(row.one())
def _audits(db, action: str) -> list[AuditLog]:
db.expire_all()
return list(db.execute(select(AuditLog).where(AuditLog.action == action)).scalars())
def test_the_token_is_kept_only_hashed_and_encrypted(db, key, owner_job, client):
owner, job_id = owner_job
made = owner.post(f"/api/downloads/{job_id}/links", json={}).json()
token = made["token"]
assert made["url"] == f"/watch/{token}"
row = _row(db, made["id"])
assert "token" not in row
assert token not in json.dumps(row, default=str)
assert row["token_hash"] == security.hash_token(token)
assert security.decrypt(row["token_enc"]) == token
# The address it was given opens it, and its owner can copy the same one again.
assert client.get(f"/api/public/watch/{token}").status_code == 200
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [entry["url"] for entry in listed] == [f"/watch/{token}"]
def test_a_token_naming_no_link_finds_none(db, key, owner_job, client):
owner, job_id = owner_job
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
other = token[:-1] + ("A" if token[-1] != "A" else "B")
assert client.get(f"/api/public/watch/{other}").status_code == 404
# Its hash is no key either: the lookup hashes what it is given.
hashed = security.hash_token(token)
assert client.get(f"/api/public/watch/{hashed}").status_code == 404
def test_without_a_key_the_link_opens_but_its_address_is_not_kept(db, owner_job, client):
assert not security.can_encrypt()
owner, job_id = owner_job
made = owner.post(f"/api/downloads/{job_id}/links", json={}).json()
# The one who made it has it in hand…
assert made["url"] == f"/watch/{made['token']}"
assert _row(db, made["id"])["token_enc"] is None
assert client.get(f"/api/public/watch/{made['token']}").status_code == 200
# …but it cannot be shown again.
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [(entry["token"], entry["url"]) for entry in listed] == [(None, None)]
def test_a_changed_key_hides_the_address_and_the_link_still_opens(
db, key, owner_job, client, monkeypatch
):
owner, job_id = owner_job
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [entry["url"] for entry in listed] == [None]
assert client.get(f"/api/public/watch/{token}").status_code == 200
def test_making_changing_and_revoking_a_link_is_audited_without_its_secrets(db, key, owner_job):
owner, job_id = owner_job
pw = "a secret word"
made = owner.post(
f"/api/downloads/{job_id}/links", json={"password": pw, "allow_download": False}
).json()
link_id = made["id"]
patch = f"/api/downloads/links/{link_id}"
assert owner.patch(patch, json={"allow_download": True}).status_code == 200
assert owner.patch(patch, json={"password": "another one"}).status_code == 200
# Nothing changed: nothing to record.
assert owner.patch(patch, json={"allow_download": True}).status_code == 200
gone = owner.delete(f"/api/downloads/links/{link_id}")
assert gone.json() == {"ok": True, "deleted": link_id}
(create,) = _audits(db, "download_link_create")
assert create.target_id == str(link_id)
assert create.after == {
"job_id": job_id,
"allow_download": False,
"has_password": True,
"expires_at": None,
}
updates = _audits(db, "download_link_update")
assert [(u.before["allow_download"], u.after["allow_download"]) for u in updates] == [
(False, True),
(True, True), # the password changed: a rotation is recorded though the flags are equal
]
(revoke,) = _audits(db, "download_link_revoke")
assert revoke.target_id == str(link_id)
assert revoke.after == {"job_id": job_id}
trail = json.dumps(
[[a.summary, a.before, a.after] for a in [create, *updates, revoke]], default=str
)
for secret in (made["token"], pw, "another one"):
assert secret not in trail
def test_the_migration_keeps_every_address_already_sent(db, key, owner_job, client):
owner, job_id = owner_job
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
cfg = Config("alembic.ini")
command.downgrade(cfg, "0106_api_tokens")
try:
with engine.begin() as conn:
conn.execute(
text("INSERT INTO download_links (job_id, token) VALUES (:j, 'an-old-token-sent')"),
{"j": job_id},
)
finally:
command.upgrade(cfg, "head")
(row,) = db.execute(text("SELECT token_hash, token_enc FROM download_links")).mappings().all()
assert row["token_hash"] == security.hash_token("an-old-token-sent")
assert security.decrypt(row["token_enc"]) == "an-old-token-sent"
assert client.get("/api/public/watch/an-old-token-sent").status_code == 200
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [entry["url"] for entry in listed] == ["/watch/an-old-token-sent"]
def test_the_downgrade_writes_the_tokens_back(db, key, owner_job):
owner, job_id = owner_job
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
cfg = Config("alembic.ini")
command.downgrade(cfg, "0106_api_tokens")
try:
with engine.connect() as conn:
back = conn.execute(text("SELECT token FROM download_links")).scalars().all()
finally:
command.upgrade(cfg, "head")
assert back == [token]
assert db.execute(select(DownloadLink.token_hash)).scalars().all() == [
security.hash_token(token)
]
def test_without_a_key_the_migration_keeps_the_address_opening_but_not_shown(db, owner_job, client):
"""No key to seal with: the hash alone is kept, the link still opens, its address is gone."""
assert not security.can_encrypt()
owner, job_id = owner_job
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
cfg = Config("alembic.ini")
command.downgrade(cfg, "0106_api_tokens")
try:
with engine.begin() as conn:
conn.execute(
text("INSERT INTO download_links (job_id, token) VALUES (:j, 'an-old-token-sent')"),
{"j": job_id},
)
finally:
command.upgrade(cfg, "head")
(row,) = db.execute(text("SELECT token_hash, token_enc FROM download_links")).mappings().all()
assert row["token_hash"] == security.hash_token("an-old-token-sent")
assert row["token_enc"] is None
assert client.get("/api/public/watch/an-old-token-sent").status_code == 200
assert [e["url"] for e in owner.get(f"/api/downloads/{job_id}/links").json()] == [None]
def test_a_downgrade_with_no_readable_copy_writes_a_fresh_token(db, key, owner_job, monkeypatch):
"""A key since changed reads no copy: the old column gets a random token, so the link is
revoked rather than left with no token (the column is NOT NULL there)."""
owner, job_id = owner_job
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
cfg = Config("alembic.ini")
command.downgrade(cfg, "0106_api_tokens")
try:
with engine.connect() as conn:
back = conn.execute(text("SELECT token FROM download_links")).scalars().all()
finally:
command.upgrade(cfg, "head")
assert len(back) == 1 and back[0] and back[0] != token
+3
View File
@@ -53,6 +53,9 @@ class TestGrant:
def test_malformed_grants_are_rejected_not_crashed(self):
for bad in [None, "", "no-dot", "notanumber.sig"]:
assert check_grant("tok", bad, 0) is False
# The query string decodes as UTF-8: a signature past ASCII is just a wrong one.
future = int(datetime.now(timezone.utc).timestamp()) + _GRANT_TTL
assert check_grant("tok", f"{future}.é", 0) is False
def _asset(**over):
+141
View File
@@ -0,0 +1,141 @@
"""Migration 0102's data step, DB-lane: every existing job lands in its source folder, and a done
file's expiry moves from the shared asset to the user's job.
The schema half is checked by test_migrations.py; a row placed in the wrong folder is invisible there.
"""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def test_existing_jobs_are_placed_and_keep_their_expiry(db, make_user):
alice, bob = make_user().id, make_user().id
cfg = Config("alembic.ini")
command.downgrade(cfg, "0101_plex_text_plain")
try:
with engine.begin() as conn:
def asset(kind, ref, expires=None):
return conn.execute(
text(
"INSERT INTO media_assets (source_kind, source_ref, format_sig, status,"
" ref_count, expires_at) VALUES (:k, :r, :sig, 'ready', 1, :e)"
" RETURNING id"
),
{"k": kind, "r": ref, "sig": kind + ref, "e": expires},
).scalar_one()
def job(user, kind, ref, asset_id, status="done", spec=None, parent=None):
return conn.execute(
text(
"INSERT INTO download_jobs (user_id, asset_id, source_kind, source_ref,"
" profile_snapshot, status, edit_spec, source_job_id)"
" VALUES (:u, :a, :k, :r, '{}', :s, CAST(:spec AS json), :p) RETURNING id"
),
{"u": user, "a": asset_id, "k": kind, "r": ref, "s": status,
"spec": spec, "p": parent},
).scalar_one() # fmt: skip
yt = asset("youtube", "abcdefghijk", "2030-01-02T00:00:00+00:00")
tt = asset("url", "https://www.tiktok.com/@a/video/1", "2030-01-03T00:00:00+00:00")
up = asset("upload", "u1")
cl = asset("clip", "c1", "2030-01-04T00:00:00+00:00")
j_yt = job(alice, "youtube", "abcdefghijk", yt)
j_yt_bob = job(bob, "youtube", "abcdefghijk", yt, status="queued")
j_tt = job(alice, "url", "https://www.tiktok.com/@a/video/1", tt)
j_up = job(alice, "upload", "u1", up)
j_cl = job(alice, "clip", "c1", cl, spec='{"source": "plex", "ref": "9"}')
j_ed = job(alice, "edit", "e1", cl, parent=j_tt)
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
rows = conn.execute(
text(
"SELECT j.id, f.user_id, f.source_key, f.retention_days, j.expires_at::date"
" FROM download_jobs j JOIN file_folders f ON f.id = j.folder_id"
)
).all()
folders = conn.execute(text("SELECT count(*) FROM file_folders")).scalar()
got = {r[0]: (r[1], r[2], r[3], str(r[4]) if r[4] else None) for r in rows}
assert got == {
j_yt: (alice, "youtube", 30, "2030-01-02"),
j_yt_bob: (bob, "youtube", 30, None), # not done yet: no expiry until it is
j_tt: (alice, "site:tiktok.com", 30, "2030-01-03"),
j_up: (alice, "upload", None, None), # an upload never expires
j_cl: (alice, "plex", 30, "2030-01-04"),
j_ed: (alice, "site:tiktok.com", 30, "2030-01-04"),
}
# alice: youtube, tiktok, upload, plex; bob: youtube.
assert folders == 5
def test_the_admins_retention_is_kept_and_the_downgrade_gives_the_asset_back_its_expiry(
db, make_user
):
alice, bob = make_user().id, make_user().id
cfg = Config("alembic.ini")
command.downgrade(cfg, "0101_plex_text_plain")
try:
with engine.begin() as conn:
conn.execute(
text(
"INSERT INTO system_config (key, value) VALUES ('download_retention_days', '7')"
" ON CONFLICT (key) DO UPDATE SET value = '7'"
)
)
yt = conn.execute(
text(
"INSERT INTO media_assets (source_kind, source_ref, format_sig, status,"
" ref_count, expires_at) VALUES ('youtube', 'abcdefghijk', 'sig', 'ready', 2,"
" '2030-01-02T00:00:00+00:00') RETURNING id"
)
).scalar_one()
for user in (alice, bob):
conn.execute(
text(
"INSERT INTO download_jobs (user_id, asset_id, source_kind, source_ref,"
" profile_snapshot, status) VALUES (:u, :a, 'youtube', 'abcdefghijk',"
" '{}', 'done')"
),
{"u": user, "a": yt},
)
command.upgrade(cfg, "head")
with engine.connect() as conn:
retention = conn.execute(
text(
"SELECT DISTINCT retention_days FROM file_folders WHERE source_key = 'youtube'"
)
).all()
def asset_expiry_after_downgrade(alice_at, bob_at):
with engine.begin() as conn:
for user, at in ((alice, alice_at), (bob, bob_at)):
conn.execute(
text(
"UPDATE download_jobs SET expires_at = CAST(:at AS timestamptz)"
" WHERE asset_id = :a AND user_id = :u"
),
{"at": at, "a": yt, "u": user},
)
command.downgrade(cfg, "0101_plex_text_plain")
with engine.connect() as conn:
got = conn.execute(
text("SELECT expires_at::date FROM media_assets WHERE id = :a"), {"a": yt}
).scalar()
command.upgrade(cfg, "head")
return str(got) if got else None
# The shared file goes when its last holder's copy would; one holder keeping it keeps it.
latest = asset_expiry_after_downgrade(
"2030-02-01T00:00:00+00:00", "2030-03-01T00:00:00+00:00"
)
kept_by_one = asset_expiry_after_downgrade("2030-02-01T00:00:00+00:00", None)
finally:
command.upgrade(cfg, "head")
assert retention == [(7,)]
assert latest == "2030-03-01"
assert kept_by_one is None
+62
View File
@@ -0,0 +1,62 @@
"""Migration 0103's data step, DB-lane: the fileless "expired" rows of the old retention go, every
other job stays.
An expired file now leaves the library (`downloads.gc`); before, it stayed as a done job with no
asset. Left in place, those rows would sit unseen in no listing forever.
"""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def test_the_fileless_expired_rows_go_and_every_other_job_stays(db, make_user):
uid = make_user().id
cfg = Config("alembic.ini")
command.downgrade(cfg, "0102_file_folders")
try:
with engine.begin() as conn:
folder = conn.execute(
text(
"INSERT INTO file_folders (user_id, kind, source_key, retention_days)"
" VALUES (:u, 'source', 'youtube', 30) RETURNING id"
),
{"u": uid},
).scalar_one()
asset = conn.execute(
text(
"INSERT INTO media_assets (source_kind, source_ref, format_sig, status, ref_count)"
" VALUES ('youtube', 'abcdefghijk', 'sig', 'ready', 1) RETURNING id"
)
).scalar_one()
def job(status, asset_id):
return conn.execute(
text(
"INSERT INTO download_jobs (user_id, asset_id, source_kind, source_ref,"
" profile_snapshot, status, folder_id) VALUES (:u, :a, 'youtube',"
" 'abcdefghijk', '{}', :s, :f) RETURNING id"
),
{"u": uid, "a": asset_id, "s": status, "f": folder},
).scalar_one()
expired = job("done", None)
alive = job("done", asset)
failed = job("error", None) # a failed job has no file either, and is not expired
queued = job("queued", None)
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
left = set(conn.execute(text("SELECT id FROM download_jobs")).scalars())
has_hash = conn.execute(
text(
"SELECT count(*) FROM information_schema.columns"
" WHERE table_name = 'media_assets' AND column_name = 'sha256'"
)
).scalar()
assert left == {alive, failed, queued}
assert expired not in left
assert has_hash == 1
+42
View File
@@ -0,0 +1,42 @@
"""Migration 0104's data step, DB-lane: a done file from before is dated by when it was asked for
(where it already sat); a job not done has no date."""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def test_a_done_file_takes_its_request_time(db, make_user):
uid = make_user().id
cfg = Config("alembic.ini")
command.downgrade(cfg, "0103_files_sha256")
try:
with engine.begin() as conn:
def job(status, created):
return conn.execute(
text(
"INSERT INTO download_jobs (user_id, source_kind, source_ref,"
" profile_snapshot, status, created_at) VALUES (:u, 'url', 'x', '{}', :s,"
" CAST(:c AS timestamptz)) RETURNING id"
),
{"u": uid, "s": status, "c": created},
).scalar_one()
done = job("done", "2026-10-02T10:00:00+00:00")
failed = job("error", "2026-10-03T10:00:00+00:00")
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
rows = dict(
conn.execute(
text(
"SELECT id, to_char(ready_at AT TIME ZONE 'UTC', 'MM-DD HH24') FROM download_jobs"
)
).all()
)
assert rows[done] == "10-02 10"
assert rows[failed] is None
+45
View File
@@ -0,0 +1,45 @@
"""Migration 0105's data step, DB-lane: a file in a source folder records that folder's key as its
origin; a file in an own folder is left to be worked out, as before."""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def test_a_file_in_a_source_folder_records_its_origin(db, make_user):
uid = make_user().id
cfg = Config("alembic.ini")
command.downgrade(cfg, "0104_job_ready_at")
try:
with engine.begin() as conn:
def folder(kind, key, name):
return conn.execute(
text(
"INSERT INTO file_folders (user_id, kind, source_key, name) VALUES"
" (:u, :k, :s, :n) RETURNING id"
),
{"u": uid, "k": kind, "s": key, "n": name},
).scalar_one()
def job(folder_id):
return conn.execute(
text(
"INSERT INTO download_jobs (user_id, source_kind, source_ref,"
" profile_snapshot, status, folder_id) VALUES (:u, 'url', 'x', '{}',"
" 'queued', :f) RETURNING id"
),
{"u": uid, "f": folder_id},
).scalar_one()
in_source = job(folder("source", "site:vimeo.com", None))
in_own = job(folder("own", None, "Mine"))
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
rows = dict(conn.execute(text("SELECT id, source_key FROM download_jobs")).all())
assert rows[in_source] == "site:vimeo.com"
assert rows[in_own] is None
+50
View File
@@ -0,0 +1,50 @@
"""Migration 0106's downgrade data step, DB-lane: a Hub folder holding a file becomes an own folder
that keeps the file — under a name its user's own folders do not already have (their names are
unique per user, case-blind), and with its token gone with the tokens table."""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def test_a_hub_folder_with_a_file_comes_back_as_an_own_folder_beside_a_namesake(db, make_user):
uid = make_user().id
with engine.begin() as conn:
hub = conn.execute(
text("INSERT INTO file_folders (user_id, kind) VALUES (:u, 'hub') RETURNING id"),
{"u": uid},
).scalar_one()
conn.execute(
text("INSERT INTO file_folders (user_id, kind, name) VALUES (:u, 'own', :n)"),
{"u": uid, "n": f"hub {hub}"}, # the name the downgrade would pick, in another case
)
held = conn.execute(
text(
"INSERT INTO download_jobs (user_id, source_kind, source_ref, profile_snapshot,"
" status, folder_id) VALUES (:u, 'upload', 'x', '{}', 'queued', :f) RETURNING id"
),
{"u": uid, "f": hub},
).scalar_one()
conn.execute(
text(
"INSERT INTO api_tokens (user_id, name, prefix, token_hash) VALUES"
" (:u, 'hub', 'sfl_abcd1234', 'h') "
),
{"u": uid},
)
cfg = Config("alembic.ini")
command.downgrade(cfg, "0105_job_source_key")
try:
with engine.connect() as conn:
kind, name = conn.execute(
text("SELECT kind, name FROM file_folders WHERE id = :i"), {"i": hub}
).one()
folder_of_job = conn.execute(
text("SELECT folder_id FROM download_jobs WHERE id = :j"), {"j": held}
).scalar_one()
finally:
command.upgrade(cfg, "head")
assert (kind, name) == ("own", f"Hub {hub} (2)")
assert folder_of_job == hub
+1 -1
View File
@@ -46,7 +46,7 @@ def test_an_odd_sized_4_4_4_video_converts_to_the_even_frame_below_it(tmp_path):
def test_an_odd_crop_makes_a_clip_and_stores_the_frame_it_has(db, make_user, tmp_path, monkeypatch):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
src = tmp_path / "in.mp4"
_ffmpeg(
"-f", "lavfi", "-i", "testsrc2=size=160x120:rate=10:duration=2",
+2 -4
View File
@@ -127,7 +127,7 @@ _FILES = {
def media_root(tmp_path, monkeypatch):
root = tmp_path / "media"
root.mkdir()
monkeypatch.setattr(settings, "download_root", str(root))
monkeypatch.setattr(settings, "files_root", str(root))
return root
@@ -141,9 +141,7 @@ def test_what_a_link_serves_passes_the_rule(
subprocess.run(["ffmpeg", "-y", "-loglevel", "error", *args, str(src)], check=True)
owner = as_user(make_user())
r = owner.post(
"/api/downloads/upload", content=src.read_bytes(), headers=upload_headers(filename)
)
r = owner.post("/api/files/upload", content=src.read_bytes(), headers=upload_headers(filename))
assert r.status_code == 200, r.text
job_id = r.json()["id"]
assert r.json()["status"] == "queued" # every one of these fails the rule as it came
+2 -4
View File
@@ -19,15 +19,13 @@ _PW = "open sesame"
@pytest.fixture
def unlock_url(as_user, make_user, tmp_path, monkeypatch, upload_headers):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
video = {"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"}
audio = {"codec_type": "audio", "codec_name": "aac"}
data = {"streams": [video, audio], "format": {"format_name": "mov,mp4", "duration": "5"}}
monkeypatch.setattr(probemod, "probe_json", lambda _p: data)
owner = as_user(make_user())
job = owner.post(
"/api/downloads/upload", content=b"fake-video", headers=upload_headers("clip.mp4")
)
job = owner.post("/api/files/upload", content=b"fake-video", headers=upload_headers("clip.mp4"))
assert job.status_code == 200, job.text
link = owner.post(f"/api/downloads/{job.json()['id']}/links", json={"password": _PW})
assert link.status_code == 200, link.text
+3 -1
View File
@@ -46,7 +46,9 @@ SERIALIZERS = {
"_collection_card",
"_playlist_card",
"_serialize",
"_serialize_job",
"_serialize_folder",
"serialize",
"serialize_job",
"_serialize_profile",
"_serialize_user",
"_summary",
+2 -2
View File
@@ -27,14 +27,14 @@ def pages(tmp_path, monkeypatch):
@pytest.fixture
def link_token(as_user, make_user, tmp_path, monkeypatch, upload_headers):
monkeypatch.setattr(settings, "download_root", str(tmp_path / "media"))
monkeypatch.setattr(settings, "files_root", str(tmp_path / "media"))
video = {"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"}
audio = {"codec_type": "audio", "codec_name": "aac"}
data = {"streams": [video, audio], "format": {"format_name": "mov,mp4", "duration": "5"}}
monkeypatch.setattr(probemod, "probe_json", lambda _p: data)
owner = as_user(make_user())
job = owner.post(
"/api/downloads/upload",
"/api/files/upload",
content=b"fake-video",
headers=upload_headers("clip.mp4", display_name="Holiday clip"),
)
+40 -1
View File
@@ -6,6 +6,7 @@ from datetime import date
from app.downloads.storage import (
MediaMeta,
download_filename,
files_root_problem,
rel_path,
safe_abs_path,
sanitize,
@@ -95,12 +96,15 @@ class TestDownloadFilename:
def test_does_not_double_an_extension_already_present(self):
assert download_filename("clip.mp4", "mp4", Path("/x/f.mp4")) == "clip.mp4"
def test_no_extension_is_served_without_a_trailing_dot(self):
assert download_filename("Makefile", None, Path("/x/upload-7")) == "Makefile"
def test_falls_back_to_the_file_suffix_when_no_container(self):
assert download_filename("song", None, Path("/x/f.m4a")) == "song.m4a"
class TestSafeAbsPath:
"""The path-traversal guard: only paths that resolve INSIDE download_root AND exist are returned."""
"""The path-traversal guard: only paths that resolve INSIDE files_root AND exist are returned."""
def test_returns_the_path_for_a_file_inside_the_root(self, tmp_path: Path):
(tmp_path / "sub").mkdir()
@@ -174,3 +178,38 @@ def test_sidecars_keep_the_full_basename_when_the_title_has_a_dot(tmp_path: Path
assert f"{stem}.nfo" in names, names
assert f"{stem}.jpg" in names, names
assert "Chan_-_2026-08-12_-_Sonka_Hazilag.nfo" not in names # the old truncated orphan
def _mounts(*points):
"""A /proc/self/mountinfo table with these mount points (the fields around them are filler)."""
return "\n".join(f"{i} 1 0:{i} / {p} rw - ext4 /dev/x rw" for i, p in enumerate(points, 22))
def test_a_files_root_on_the_containers_own_disk_is_refused(tmp_path):
# The image's empty /files, left unmounted by an old compose file: every file would 404 and
# every download would go with the container.
root = str(tmp_path)
plain = _mounts("/", "/proc", "/etc/hosts")
assert "not a mounted volume" in files_root_problem(root, plain, in_container=True)
assert "does not exist" in files_root_problem(str(tmp_path / "gone"))
# Its own mount, or inside one: the volume — whatever device it reports. On a storage driver that
# keeps the container's root on the host's disk (vfs) a volume has the same device as /.
assert files_root_problem(root, _mounts("/", root), in_container=True) is None
assert files_root_problem(root, _mounts("/", str(tmp_path.parent)), in_container=True) is None
# A mount point with a space in it, as the kernel escapes it.
spaced = tmp_path / "my files"
spaced.mkdir()
escaped = str(spaced).replace(" ", "\\040")
assert files_root_problem(str(spaced), _mounts("/", escaped), in_container=True) is None
# Outside a container a plain directory is a fine root.
assert files_root_problem(root, plain, in_container=False) is None
def test_the_old_name_of_the_files_root_still_finds_the_files(monkeypatch):
from app.config import Settings
monkeypatch.delenv("FILES_ROOT", raising=False)
monkeypatch.setenv("DOWNLOAD_ROOT", "/downloads")
assert Settings().files_root == "/downloads"
monkeypatch.setenv("FILES_ROOT", "/files")
assert Settings().files_root == "/files"
+40 -48
View File
@@ -5,7 +5,7 @@ generic file (force-download so uploaded HTML/SVG can't execute inline), quota a
size known up front, the per-file cap enforced while streaming, filename sanitization, and the two
guards that keep an upload out of paths that assume a re-fetchable source (edit / redownload).
DB-backed (real Postgres) — skipped when no database is reachable. `settings.download_root` is
DB-backed (real Postgres) — skipped when no database is reachable. `settings.files_root` is
pointed at a per-test tmp dir so ingest's on-disk moves are isolated. The ingest/route tests
monkeypatch the shared `probe.probe_json` so they don't shell out to ffprobe; the classifier's OWN
parsing (image → file, cover-art skip, dims, duration) is covered directly by `TestProbeClassifier`
@@ -51,7 +51,7 @@ def test_derive_media_kind():
@pytest.fixture
def _root(tmp_path, monkeypatch):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
return tmp_path
@@ -212,7 +212,7 @@ def test_an_anonymous_upload_is_refused_before_a_byte_of_it_is_read(client):
multipart the old route would have spooled, not the raw body the new one expects."""
body, pulled = _counting_multipart()
r = client.post(
"/api/downloads/upload",
"/api/files/upload",
content=body,
headers={"content-type": f"multipart/form-data; boundary={_BOUNDARY}"},
)
@@ -227,7 +227,7 @@ def test_an_authenticated_upload_still_has_its_body_read(as_user, make_user, _ro
body, pulled = _counting_body()
c = as_user(make_user())
r = c.post(
"/api/downloads/upload",
"/api/files/upload",
content=body,
headers={"content-type": "application/octet-stream", "x-upload-filename": "clip.bin"},
)
@@ -281,7 +281,7 @@ def test_upload_route_generic_forces_download(
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
r = c.post(
"/api/downloads/upload",
"/api/files/upload",
content=b"<script>alert(1)</script>",
headers=upload_headers("report.html"),
)
@@ -290,7 +290,7 @@ def test_upload_route_generic_forces_download(
assert job["asset"]["media_kind"] == "file"
assert job["can_download"] is True
# The served file must be a neutral force-download attachment — never inline text/html.
fr = c.get(f"/api/downloads/{job['id']}/file")
fr = c.get(f"/api/files/{job['id']}/file")
assert fr.status_code == 200
assert fr.headers["content-type"].startswith("application/octet-stream")
assert "attachment" in fr.headers["content-disposition"].lower()
@@ -307,7 +307,7 @@ def test_upload_route_media_html_extension_not_served_as_html(
_probe(monkeypatch, ("h264", "aac", 320, 240, 5)) # ffprobe says it's video…
c = as_user(make_user())
r = c.post(
"/api/downloads/upload",
"/api/files/upload",
content=b"<script>alert(1)</script>",
headers=upload_headers("evil.html"),
)
@@ -320,11 +320,11 @@ def test_upload_route_media_html_extension_not_served_as_html(
assert job["status"] == "queued"
row = db.get(DownloadJob, job["id"])
db.get(MediaAsset, row.asset_id).status = "ready"
row.status = "done"
row.status, row.ready_at = "done", row.created_at
db.commit()
# Authenticated endpoint: never text/html.
fr = c.get(f"/api/downloads/{job['id']}/file")
fr = c.get(f"/api/files/{job['id']}/file")
assert fr.status_code == 200
assert fr.headers["content-type"].startswith("application/octet-stream")
assert "attachment" in fr.headers["content-disposition"].lower()
@@ -348,7 +348,7 @@ def test_public_watch_file_forces_download_for_generic_file(
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
job = c.post(
"/api/downloads/upload",
"/api/files/upload",
content=b"<script>alert(1)</script>",
headers=upload_headers("notes.html"),
).json()
@@ -369,7 +369,7 @@ def test_public_watch_file_serves_media_inline_with_real_type(
_probe(monkeypatch, ("h264", "aac", 640, 360, 12))
c = as_user(make_user())
job = c.post(
"/api/downloads/upload",
"/api/files/upload",
content=b"fake-mp4-bytes",
headers=upload_headers("clip.mp4"),
).json()
@@ -389,7 +389,7 @@ def test_upload_route_enforces_per_file_cap(
db.commit()
c = as_user(make_user())
r = c.post(
"/api/downloads/upload",
"/api/files/upload",
content=b"way over the cap",
headers=upload_headers("big.bin"),
)
@@ -412,14 +412,10 @@ def test_the_cap_admits_a_body_of_EXACTLY_the_cap_and_refuses_one_byte_more(
db.commit()
c = as_user(make_user())
at_the_cap = c.post(
"/api/downloads/upload", content=payload, headers=upload_headers("exact.bin")
)
at_the_cap = c.post("/api/files/upload", content=payload, headers=upload_headers("exact.bin"))
assert at_the_cap.status_code == 200, at_the_cap.text
over = c.post(
"/api/downloads/upload", content=payload + b"x", headers=upload_headers("over.bin")
)
over = c.post("/api/files/upload", content=payload + b"x", headers=upload_headers("over.bin"))
assert over.status_code == 422
assert over.json()["detail"] == {"code": "upload_too_large", "limit": len(payload)}
@@ -437,9 +433,7 @@ def test_a_non_ascii_filename_survives_the_header_round_trip(
"""
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
r = c.post(
"/api/downloads/upload", content=b"data", headers=upload_headers("nyaralás ősszel.bin")
)
r = c.post("/api/files/upload", content=b"data", headers=upload_headers("nyaralás ősszel.bin"))
assert r.status_code == 200, r.text
asset = db.execute(select(MediaAsset).order_by(MediaAsset.id.desc())).scalars().first()
assert asset is not None
@@ -467,9 +461,7 @@ def test_a_long_display_name_is_truncated_on_EVERY_route_that_takes_one(
c = as_user(make_user())
# 1. the upload route — the name arrives in a request header.
up = c.post(
"/api/downloads/upload", content=b"video", headers=upload_headers("a.mp4", long_name)
)
up = c.post("/api/files/upload", content=b"video", headers=upload_headers("a.mp4", long_name))
assert up.status_code == 200, up.text
assert up.json()["display_name"] == bound
@@ -494,10 +486,10 @@ def test_a_long_display_name_is_truncated_on_EVERY_route_that_takes_one(
assert edit.status_code == 200, edit.text
assert edit.json()["display_name"] == bound
# 4 + 5. the metadata PATCH's two fields.
# 4 + 5. the library PATCH's two fields.
patched = c.patch(
f"/api/downloads/{up.json()['id']}",
json={"display_name": long_name, "display_uploader": long_name},
f"/api/files/{up.json()['id']}",
json={"name": long_name, "uploader": long_name},
)
assert patched.status_code == 200, patched.text
assert patched.json()["display_name"] == bound
@@ -584,7 +576,7 @@ def _display_write_sites() -> dict[tuple[str, str], set[str]]:
# one value that needs no bound, and a clear is still a write.
# service.enqueue / enqueue_edit / `display_name` is already bounded by its caller; the fallback
# enqueue_clip slices the source's title to the same width.
# downloads.upload_download passes through `spool_upload`'s value, bounded there.
# files.ingest_spooled passes through `spool_upload`'s value, bounded there.
# routes enqueue_clip the `display_name=` keyword passes on the name bound above it.
# pipeline._download / _process_clip / worker._finish_from_cache
# slice a fetched or stored title to the column's width.
@@ -610,12 +602,12 @@ DISPLAY_WRITE_SITES = {
("routes/downloads.py", "enqueue_download"): {"display_text(body.display_name)"},
("routes/downloads.py", "enqueue_edit"): {"display_text(body.display_name)"},
("routes/downloads.py", "enqueue_clip"): {"display_text(body.display_name)", "display_name"},
("routes/downloads.py", "update_download_meta"): {
"display_text(body.display_name)",
"display_text(body.display_uploader)",
("routes/files.py", "ingest_spooled"): {"upload.display_name"},
("routes/files.py", "update_file"): {
"display_text(body.name)",
"display_text(body.uploader)",
},
("routes/downloads.py", "upload_download"): {"upload.display_name"},
("routes/downloads.py", "spool_upload"): {
("routes/files.py", "spool_upload"): {
"display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER))"
},
}
@@ -665,7 +657,7 @@ def test_the_display_name_header_reaches_the_job_and_is_bounded(
c = as_user(make_user())
ok = c.post(
"/api/downloads/upload",
"/api/files/upload",
content=b"data",
headers=upload_headers("clip.bin", "Nyári vakáció"),
)
@@ -674,7 +666,7 @@ def test_the_display_name_header_reaches_the_job_and_is_bounded(
long_name = "n" * 300
bounded = c.post(
"/api/downloads/upload", content=b"data2", headers=upload_headers("clip2.bin", long_name)
"/api/files/upload", content=b"data2", headers=upload_headers("clip2.bin", long_name)
)
assert bounded.status_code == 200, bounded.text
assert bounded.json()["display_name"] == "n" * DISPLAY_TEXT_MAX
@@ -694,12 +686,12 @@ def test_the_upload_holds_no_db_transaction_while_the_body_streams(
the checkout is its consequence (measured), and the pool counter is shared with whatever else
the test process is holding.
"""
from app.routes import downloads as downloads_routes
from app.routes import files as files_routes
_probe(monkeypatch, (None, None, None, None, None))
seen: dict[str, bool] = {}
real_stream = downloads_routes.capped_stream_to
real_release = downloads_routes.release_read_connection
real_stream = files_routes.capped_stream_to
real_release = files_routes.release_read_connection
def spy_release(session):
# The PRECONDITION, so this test cannot pass vacuously. Asserting only "no transaction
@@ -714,10 +706,10 @@ def test_the_upload_holds_no_db_transaction_while_the_body_streams(
seen["while_streaming"] = bool(db.in_transaction())
return await real_stream(request, out, limit, refusal)
monkeypatch.setattr(downloads_routes, "release_read_connection", spy_release)
monkeypatch.setattr(downloads_routes, "capped_stream_to", spy_stream)
monkeypatch.setattr(files_routes, "release_read_connection", spy_release)
monkeypatch.setattr(files_routes, "capped_stream_to", spy_stream)
c = as_user(make_user())
r = c.post("/api/downloads/upload", content=b"data", headers=upload_headers("held.bin"))
r = c.post("/api/files/upload", content=b"data", headers=upload_headers("held.bin"))
assert r.status_code == 200, r.text
assert seen["before"] is True, (
"nothing had opened a transaction before the release, so this test proves nothing — the "
@@ -739,14 +731,14 @@ def test_the_usage_route_reports_the_per_file_upload_cap(as_user, make_user, db)
from app import sysconfig
c = as_user(make_user())
r = c.get("/api/downloads/usage")
r = c.get("/api/files/usage")
assert r.status_code == 200, r.text
assert r.json()["upload_max_bytes"] == sysconfig.get_int(db, "download_upload_max_bytes")
# And it is the KNOB, not a constant: a lowered override must show through.
sysconfig.set_value(db, "download_upload_max_bytes", 4096)
db.commit()
assert c.get("/api/downloads/usage").json()["upload_max_bytes"] == 4096
assert c.get("/api/files/usage").json()["upload_max_bytes"] == 4096
def test_a_refused_upload_leaves_nothing_in_the_staging_directory(
@@ -761,10 +753,10 @@ def test_a_refused_upload_leaves_nothing_in_the_staging_directory(
sysconfig.set_value(db, "download_upload_max_bytes", 8)
db.commit()
staging = Path(settings.download_root) / ".staging" / "uploads"
staging = Path(settings.files_root) / ".staging" / "uploads"
c = as_user(make_user())
r = c.post("/api/downloads/upload", content=b"x" * 4096, headers=upload_headers("leaky.bin"))
r = c.post("/api/files/upload", content=b"x" * 4096, headers=upload_headers("leaky.bin"))
assert r.status_code == 422
assert staging.exists(), "the dependency did not get as far as creating the staging dir"
assert list(staging.iterdir()) == [], "a refused upload left its partial file behind"
@@ -786,7 +778,7 @@ def test_upload_cap_cannot_be_disabled_with_zero(db):
def test_upload_route_rejects_empty_file(as_user, make_user, _root, monkeypatch, upload_headers):
c = as_user(make_user())
r = c.post("/api/downloads/upload", content=b"", headers=upload_headers("empty.bin"))
r = c.post("/api/files/upload", content=b"", headers=upload_headers("empty.bin"))
assert r.status_code == 400
assert r.json()["detail"]["code"] == "upload_empty"
@@ -794,7 +786,7 @@ def test_upload_route_rejects_empty_file(as_user, make_user, _root, monkeypatch,
def test_redownload_rejects_an_upload(as_user, make_user, _root, monkeypatch, upload_headers):
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
job = c.post("/api/downloads/upload", content=b"data", headers=upload_headers("a.bin")).json()
job = c.post("/api/files/upload", content=b"data", headers=upload_headers("a.bin")).json()
r = c.post(f"/api/downloads/{job['id']}/redownload")
assert r.status_code == 409
@@ -804,7 +796,7 @@ def test_edit_rejects_a_generic_upload_source(
):
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
job = c.post("/api/downloads/upload", content=b"data", headers=upload_headers("a.bin")).json()
job = c.post("/api/files/upload", content=b"data", headers=upload_headers("a.bin")).json()
r = c.post(
"/api/downloads/edit",
json={"source_job_id": job["id"], "edit_spec": {"trim": {"start_s": 0, "end_s": 1}}},
+47 -7
View File
@@ -29,7 +29,7 @@ from sqlalchemy.orm import Session
@pytest.fixture
def _root(tmp_path, monkeypatch):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
return tmp_path
@@ -407,7 +407,7 @@ def test_the_sweep_keeps_an_aged_upload_file_its_row_names_in_any_state(
_probe(monkeypatch)
job, asset = _ingest(db, _root, make_user().id)
path = storage.abs_path(str(_root), asset.rel_path)
job.status, asset.status = job_status, asset_status
job.status, asset.status, job.ready_at = job_status, asset_status, job.created_at
db.commit()
_aged(path)
@@ -427,7 +427,11 @@ def _route_moved(db, job, asset, job_status: str, ref_count: int) -> None:
"""What the cancel/pause/delete route leaves behind, written as SQL: the session's copy of
`ref_count` is stale (the routes count it in SQL), so an ORM assignment of the same value would
write nothing."""
db.execute(update(DownloadJob).where(DownloadJob.id == job.id).values(status=job_status))
db.execute(
update(DownloadJob)
.where(DownloadJob.id == job.id)
.values(status=job_status, ready_at=DownloadJob.created_at)
)
db.execute(update(MediaAsset).where(MediaAsset.id == asset.id).values(ref_count=ref_count))
db.commit()
@@ -549,7 +553,7 @@ def test_a_link_says_preparing_only_while_its_video_is_being_made(
_probe(monkeypatch, vcodec="h264")
c = as_user(make_user())
job = c.post(
"/api/downloads/upload", content=b"h264 bytes", headers=upload_headers("clip.mp4")
"/api/files/upload", content=b"h264 bytes", headers=upload_headers("clip.mp4")
).json()
link = c.post(f"/api/downloads/{job['id']}/links", json={"allow_download": False}).json()
assert c.get(f"/api/public/watch/{link['token']}").status_code == 200
@@ -570,7 +574,7 @@ def test_a_link_to_a_suspended_owners_waiting_rebuild_says_it_is_gone(
owner = make_user()
c = as_user(owner)
job = c.post(
"/api/downloads/upload", content=b"h264 bytes", headers=upload_headers("clip.mp4")
"/api/files/upload", content=b"h264 bytes", headers=upload_headers("clip.mp4")
).json()
link = c.post(f"/api/downloads/{job['id']}/links", json={"allow_download": False}).json()
db.get(MediaAsset, job["asset"]["id"]).status = "pending"
@@ -593,9 +597,45 @@ def test_the_admin_storage_view_counts_a_waiting_upload(db, as_user, make_user,
_probe(monkeypatch)
u = make_user()
_ingest(db, _root, u.id, size=10)
r = as_user(make_user(role="admin")).get("/api/admin/downloads/storage").json()
admin = make_user(role="admin")
r = as_user(admin).get("/api/admin/downloads/storage").json()
assert (r["files_on_disk"], r["total_bytes"]) == (1, 10)
assert {p["user_id"]: p["footprint_bytes"] for p in r["per_user"]} == {u.id: 10}
assert {p["user_id"]: p["footprint_bytes"] for p in r["per_user"]} == {u.id: 10, admin.id: 0}
def test_the_admin_storage_view_names_each_users_quota_and_files(
db, as_user, make_user, _root, monkeypatch
):
# A row says how full the user is against THEIR limit (a custom one, here), and how many files
# their Files lists — the finished ones, not one still waiting for its conversion.
_probe(monkeypatch)
u = make_user()
done, _ = _ingest(db, _root, u.id, size=10)
waiting, _ = _ingest(db, _root, u.id, name="b.mov", size=5)
done.status, waiting.status = "done", "queued"
done.ready_at = done.created_at
db.commit()
admin = as_user(make_user(role="admin"))
put = admin.put(f"/api/admin/downloads/quota/{u.id}", json={"max_bytes": 1000})
assert put.status_code == 200
rows = admin.get("/api/admin/downloads/storage").json()["per_user"]
mine = [(p["files"], p["max_bytes"], p["unlimited"]) for p in rows if p["user_id"] == u.id]
assert mine == [(1, 1000, False)]
# Every download names its owner, so its row can open THEIR quota.
jobs = admin.get("/api/admin/downloads").json()
assert {(j["id"], j["user_id"]) for j in jobs} == {(done.id, u.id), (waiting.id, u.id)}
def test_the_admin_storage_view_lists_every_user_even_one_without_a_file(db, as_user, make_user):
# The admin sets a quota BEFORE anyone downloads: a user holding nothing is still a row.
empty = make_user()
admin_user = make_user(role="admin")
admin = as_user(admin_user)
rows = admin.get("/api/admin/downloads/storage").json()["per_user"]
by_id = {p["user_id"]: p for p in rows}
assert {empty.id, admin_user.id} <= by_id.keys()
assert (by_id[empty.id]["email"], by_id[empty.id]["files"]) == (empty.email, 0)
assert by_id[empty.id]["footprint_bytes"] == 0
@pytest.mark.parametrize(
+1 -1
View File
@@ -32,7 +32,7 @@ def test_a_clip_the_phone_safe_step_changed_is_stored_under_its_new_extension(
):
# A fast trim stream-copies its source's container; when the phone-safe step remuxes it (an
# audio clip out of an .mp4 becomes an .m4a), the stored path and container follow the new file.
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(settings, "files_root", str(tmp_path))
data = {
"streams": [{"codec_type": "audio", "codec_name": "aac"}],
"format": {"format_name": "mov,mp4", "duration": "10"},
+15
View File
@@ -67,6 +67,7 @@ def test_worker_boots_with_the_stored_concurrency_and_log_level(monkeypatch, sto
monkeypatch.setattr(worker, "_wait_for_schema", lambda: True)
monkeypatch.setattr(worker.logsink, "start", lambda: None)
monkeypatch.setattr(worker, "_recover_orphans", lambda: None)
monkeypatch.setattr(worker.storage, "files_root_problem", lambda _root: None)
threading.Timer(0.5, worker._stop.set).start()
worker.main()
@@ -107,3 +108,17 @@ def test_an_unreadable_database_keeps_what_runs(monkeypatch, levels, loops):
assert worker._target == 2
assert sorted(threads) == [0, 1] # the loops still run on the boot value
assert levels == []
def test_the_worker_will_not_start_on_a_files_root_that_is_not_the_volume(monkeypatch, loops):
# Files downloaded into the container's own empty /files go with the container: refuse, loudly.
monkeypatch.setattr(worker.settings, "worker_enabled", True)
monkeypatch.setattr(worker.logs, "configure_logging", lambda _source: None)
monkeypatch.setattr(worker.signal, "signal", lambda *_a: None)
monkeypatch.setattr(worker, "_wait_for_schema", lambda: True)
monkeypatch.setattr(worker.logsink, "start", lambda: None)
monkeypatch.setattr(worker.storage, "files_root_problem", lambda _root: "not a mounted volume")
with pytest.raises(SystemExit) as exit_:
worker.main()
assert exit_.value.code == 1
assert loops == []
+9 -9
View File
@@ -17,7 +17,7 @@
# TOKEN_ENCRYPTION_KEY (Fernet), GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET,
# OAUTH_REDIRECT_URL=https://siftlode.example.com/auth/callback, ALLOWED_EMAILS, ADMIN_EMAILS,
# YOUTUBE_API_KEY. APP_VERSION is exported by the deploy step (selects the image tag).
# Optional: DOWNLOAD_HOST_PATH (host dir for the download tree) and PLEX_MEDIA_HOST_PATH (host dir
# Optional: FILES_HOST_PATH (host dir for the download tree) and PLEX_MEDIA_HOST_PATH (host dir
# holding the Plex library, mounted read-only for the optional Plex module).
name: siftlode
@@ -72,8 +72,8 @@ services:
# The public instance owns the background scheduler (single writer).
SCHEDULER_ENABLED: "true"
# Download center: the API serves finished files + builds filmstrips (read-only root is fine —
# it only writes to the /downloads mount + tmpfs). The yt-dlp job loop runs in `worker` below.
DOWNLOAD_ROOT: /downloads
# it only writes to the /files mount + tmpfs). The yt-dlp job loop runs in `worker` below.
FILES_ROOT: /files
WORKER_ENABLED: "false"
depends_on:
db:
@@ -84,9 +84,9 @@ services:
# (10.x.x.x:8080) and on the trusted home LAN. Never forwarded to the public internet.
- "8080:8000"
volumes:
# Downloaded media. Defaults to a Docker-managed named volume; set DOWNLOAD_HOST_PATH in .env
# Downloaded media. Defaults to a Docker-managed named volume; set FILES_HOST_PATH in .env
# to a host directory (e.g. one your Plex server reads) to keep the Plex-style tree there.
- ${DOWNLOAD_HOST_PATH:-siftlode_downloads}:/downloads
- ${FILES_HOST_PATH:-${DOWNLOAD_HOST_PATH:-siftlode_files}}:/files
# Plex media, read-only. Only used when the optional Plex module is enabled: set
# PLEX_MEDIA_HOST_PATH in .env to the host directory holding the Plex library (the local side
# of plex_path_map, default /plex-media). Left unset → an empty named volume (harmless; the
@@ -113,7 +113,7 @@ services:
restart: unless-stopped
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
# DB (job queue) and the downloads mount with the API. Not read-only (yt-dlp/deno/ffmpeg write to
# DB (job queue) and the /files mount with the API. Not read-only (yt-dlp/deno/ffmpeg write to
# $HOME caches + the staging dir); heavier limits since it does the video encoding.
worker:
image: registry.example.com/you/siftlode:${APP_VERSION:-latest}
@@ -124,7 +124,7 @@ services:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB:-siftlode}
# The API owns the scheduler; the worker must not also run it.
SCHEDULER_ENABLED: "false"
DOWNLOAD_ROOT: /downloads
FILES_ROOT: /files
WORKER_ENABLED: "true"
depends_on:
db:
@@ -137,7 +137,7 @@ services:
condition: service_started
networks: [internal]
volumes:
- ${DOWNLOAD_HOST_PATH:-siftlode_downloads}:/downloads
- ${FILES_HOST_PATH:-${DOWNLOAD_HOST_PATH:-siftlode_files}}:/files
# Plex media (read-only) — same mount as the api; used by Plex clip jobs (a later phase).
- ${PLEX_MEDIA_HOST_PATH:-siftlode_plex_media}:/plex-media:ro
mem_limit: 2g
@@ -167,7 +167,7 @@ services:
volumes:
siftlode_pgdata:
siftlode_downloads:
siftlode_files:
# Empty fallback for the Plex media mount when PLEX_MEDIA_HOST_PATH is unset (Plex module off).
siftlode_plex_media:
+3 -3
View File
@@ -21,21 +21,21 @@ services:
environment:
# This instance owns its scheduler (its own DB, so no double-write concern).
SCHEDULER_ENABLED: "true"
# Plex HLS remux scratch on fast container-local storage (the /downloads bind-mount is a slow
# Plex HLS remux scratch on fast container-local storage (the /files bind-mount is a slow
# Windows Docker Desktop mount in dev; prod uses fast local disk so it keeps the default).
PLEX_HLS_DIR: /var/tmp/plex-hls
security_opt:
- apparmor:unconfined
volumes:
# Downloaded media lives on the host so you can inspect the generated Plex tree. Gitignored.
- ./downloads:/downloads
- ./files:/files
- plex_media:/plex-media:ro
worker:
security_opt:
- apparmor:unconfined
volumes:
- ./downloads:/downloads
- ./files:/files
- plex_media:/plex-media:ro
bgutil-pot:
+8 -8
View File
@@ -48,8 +48,8 @@ services:
# This instance owns the background scheduler (single writer).
SCHEDULER_ENABLED: "true"
# Download center: the API serves finished files + builds filmstrips; the worker (below) runs
# the yt-dlp job loop. Read-only root is fine — it only writes to the /downloads mount + tmpfs.
DOWNLOAD_ROOT: /downloads
# the yt-dlp job loop. Read-only root is fine — it only writes to the /files mount + tmpfs.
FILES_ROOT: /files
WORKER_ENABLED: "false"
depends_on:
db:
@@ -60,9 +60,9 @@ services:
# front for HTTPS / public exposure — and set OAUTH_REDIRECT_URL to the https URL.
- "${HTTP_PORT:-8080}:8000"
volumes:
# Downloaded media. Defaults to a Docker-managed named volume; set DOWNLOAD_HOST_PATH in .env
# Downloaded media. Defaults to a Docker-managed named volume; set FILES_HOST_PATH in .env
# to a host directory (e.g. one your Plex server reads) to keep the Plex-style tree there.
- ${DOWNLOAD_HOST_PATH:-siftlode_downloads}:/downloads
- ${FILES_HOST_PATH:-${DOWNLOAD_HOST_PATH:-siftlode_files}}:/files
# Plex library, read-only. The Plex module streams files itself, so it needs to SEE them: set
# PLEX_MEDIA_HOST_PATH in .env to the host directory holding the library (the local side of the
# admin "path map", whose default is /data=/plex-media). Left unset it falls back to an empty
@@ -85,7 +85,7 @@ services:
restart: unless-stopped
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
# DB (job queue) and the downloads mount with the API. Not read-only (yt-dlp/deno/ffmpeg write to
# DB (job queue) and the /files mount with the API. Not read-only (yt-dlp/deno/ffmpeg write to
# $HOME caches + the staging dir).
worker:
image: git.b1fr0st.eu/peter/siftlode:${IMAGE_TAG:-latest}
@@ -98,7 +98,7 @@ services:
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB:-siftlode}
SCHEDULER_ENABLED: "false"
DOWNLOAD_ROOT: /downloads
FILES_ROOT: /files
WORKER_ENABLED: "true"
depends_on:
db:
@@ -111,7 +111,7 @@ services:
condition: service_started
networks: [internal]
volumes:
- ${DOWNLOAD_HOST_PATH:-siftlode_downloads}:/downloads
- ${FILES_HOST_PATH:-${DOWNLOAD_HOST_PATH:-siftlode_files}}:/files
# Same read-only Plex library as the API — the worker resolves library paths too.
- ${PLEX_MEDIA_HOST_PATH:-siftlode_plex_media}:/plex-media:ro
security_opt:
@@ -136,7 +136,7 @@ services:
volumes:
siftlode_pgdata:
siftlode_downloads:
siftlode_files:
# Empty fallback for the Plex media mount when PLEX_MEDIA_HOST_PATH is unset (Plex module off).
siftlode_plex_media:
+7 -7
View File
@@ -37,7 +37,7 @@ services:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD:-siftlode}@db:5432/${POSTGRES_DB:-siftlode}
# Download center: the API serves finished files + builds filmstrips; the worker (below) runs
# the yt-dlp job loop.
DOWNLOAD_ROOT: /downloads
FILES_ROOT: /files
WORKER_ENABLED: "false"
depends_on:
db:
@@ -45,9 +45,9 @@ services:
ports:
- "${APP_PORT:-8080}:8000"
volumes:
# Downloaded media. Defaults to a Docker-managed named volume; set DOWNLOAD_HOST_PATH in .env
# Downloaded media. Defaults to a Docker-managed named volume; set FILES_HOST_PATH in .env
# to a host directory (e.g. one your Plex server reads) to keep the Plex-style tree there.
- ${DOWNLOAD_HOST_PATH:-downloads}:/downloads
- ${FILES_HOST_PATH:-${DOWNLOAD_HOST_PATH:-files}}:/files
# Plex library, read-only. The Plex module streams files itself, so it needs to SEE them: set
# PLEX_MEDIA_HOST_PATH in .env to the host directory holding the library (the local side of the
# admin "path map", whose default is /data=/plex-media). Left unset it falls back to an empty
@@ -63,7 +63,7 @@ services:
restart: unless-stopped
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
# DB (job queue) + the downloads mount with the API.
# DB (job queue) + the /files mount with the API.
worker:
build:
context: .
@@ -80,7 +80,7 @@ services:
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD:-siftlode}@db:5432/${POSTGRES_DB:-siftlode}
SCHEDULER_ENABLED: "false"
DOWNLOAD_ROOT: /downloads
FILES_ROOT: /files
WORKER_ENABLED: "true"
depends_on:
db:
@@ -91,7 +91,7 @@ services:
bgutil-pot:
condition: service_started
volumes:
- ${DOWNLOAD_HOST_PATH:-downloads}:/downloads
- ${FILES_HOST_PATH:-${DOWNLOAD_HOST_PATH:-files}}:/files
# Same read-only Plex library as the API — the worker resolves library paths too.
- ${PLEX_MEDIA_HOST_PATH:-plex_media}:/plex-media:ro
logging: *logging
@@ -107,6 +107,6 @@ services:
volumes:
pgdata:
downloads:
files:
# Empty fallback for the Plex media mount when PLEX_MEDIA_HOST_PATH is unset (Plex module off).
plex_media:
+122 -113
View File
@@ -661,116 +661,125 @@ of a full-width panel and nothing on the right.
## Decision log
| Date | Decision | Chosen | Over | Why |
| ---------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
| Date | Decision | Chosen | Over | Why |
| ---------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
+5
View File
@@ -44,6 +44,8 @@ A change is done when this is green, from the repo root:
bash scripts/check.sh # every lane
bash scripts/check.sh lint # the static lanes only (no tests, build or audits) — the quick loop
bash scripts/check.sh --brief # either mode: one line per lane, output only for a red one
bash scripts/check.sh --list # the lane names, one per line (runs nothing)
bash scripts/check.sh --only "vitest" # one lane, by its exact name — never "the gate"
bash scripts/lint-changed.sh # ruff + eslint + prettier on the uncommitted files only
```
@@ -107,6 +109,9 @@ Coverage is on demand, never enforced and not in the gate (E35): `npm run test -
`frontend/`, and for the backend
`docker compose -f backend/docker-compose.test.yml run --rm test python -m pytest --cov=app` (then
`… down -v`).
The DB lane runs on every core (`pytest -n auto`, pytest-xdist): `tests/db_per_worker.py` gives each worker its own
database (`siftlode_test_gw<N>`, created and migrated on first use), so the same tests run side by side. A serial
run (no `-n`) still works and uses `siftlode_test` itself.
The Playwright suite (`npm run e2e`) runs against the `:8080` stack built from your tree; the release
requires it. Before an image is pushed, it is scanned with Trivy (HIGH/CRITICAL with a fix fails;
exceptions in `.trivyignore.yaml`, each with a reason and an expiry).
+58 -10
View File
@@ -8,7 +8,7 @@ configured in a web wizard on first start. There's no editing of config files by
- A machine with **Docker** and the **Docker Compose plugin** (Docker Desktop on Windows/macOS,
or Docker Engine on Linux).
- A few hundred MB of disk and ~1 GB RAM free for the app itself. The optional Download Center
- A few hundred MB of disk and ~1 GB RAM free for the app itself. The Files library
stores media too — budget disk for whatever you download (it's bounded by per-user quotas you
set as admin).
- Optional: a domain name + reverse proxy if you want HTTPS / public access (see below).
@@ -44,7 +44,9 @@ The installer asks for the **public URL** where the instance will be reached (ju
local `.env` file — keep that file private. **Do not change `TOKEN_ENCRYPTION_KEY` once the app has
stored any secret** (Google credentials, API keys, SMTP password, Plex token): those values are
encrypted with it, and a changed key makes them unreadable. If it is ever lost or rotated, clear the
affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page,
affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page.
Share links are kept with it too: after a key change every existing link still opens, but its
address can no longer be shown — its owner makes a new link to share it again,
- pulls the image and starts the app + database,
- prints the **setup wizard URL**, which looks like `…/setup?token=…`.
@@ -97,20 +99,20 @@ Only that peer is trusted, so a request sent straight to port `8080` (bypassing
app is exposed directly with no proxy. Make sure your proxy actually sets `X-Forwarded-For` (Caddy's
`reverse_proxy` and Nginx's `proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for` both do).
## Download Center (media storage)
## Files (media storage)
The stack includes a **Download Center**: an admin-enabled feature that saves videos to the server
The stack includes **Files**: one library that saves videos to the server
with yt-dlp (Plex-friendly folders + `.nfo`/poster), lets users trim/crop/join clips, and shares
them. It runs two extra containers that come up automatically — a `worker` (the download/edit job
loop) and a small `bgutil-pot` sidecar (mints YouTube tokens so downloads aren't bot-blocked). No
configuration is required; per-user storage quotas are set on the admin **Downloads → System** page.
configuration is required; per-user storage quotas are set on the admin **Users → Storage** tab.
By default the media lives in a Docker-managed volume (`siftlode_downloads`). To keep it somewhere
By default the media lives in a Docker-managed volume (`siftlode_files`). To keep it somewhere
you can reach from other apps — e.g. a folder your **Plex** server indexes — point it at a host
directory by adding this to `.env` and re-running `up -d`:
```bash
DOWNLOAD_HOST_PATH=/mnt/media/youtube
FILES_HOST_PATH=/mnt/media/youtube
```
The directory must be writable by the container user (uid `1000`): `sudo chown -R 1000:1000 <dir>`.
@@ -163,9 +165,11 @@ then cannot read its own secrets.
2. **`.env`** — in particular `TOKEN_ENCRYPTION_KEY`. Your Google client secret, SMTP password, Plex
token and every stored refresh token are encrypted with it *inside* the database. Restore a dump
against a different key and those values are lost — you would have to clear the affected rows from
`system_config` and re-enter every secret by hand. Keep `.env` beside the dump, and just as private.
3. **Downloaded media** *(only if you use the Download Center)* — the `siftlode_downloads` volume, or
whatever `DOWNLOAD_HOST_PATH` points at. Not needed to bring the instance back; the jobs and their
`system_config` and re-enter every secret by hand — and no existing share link's address can be
shown again (the links still open; their owners make new ones to share). Keep `.env` beside the
dump, and just as private.
3. **Downloaded media** *(only if you use Files)* — the `siftlode_files` volume, or
whatever `FILES_HOST_PATH` points at. Not needed to bring the instance back; the jobs and their
metadata are in the database, the files themselves are not.
Dump and restore the database with these two commands. They also exist as scripts
@@ -254,6 +258,50 @@ Then `pull` + `up -d` stays on that version until you change the line. Released
changed in them are listed in the app itself, under **Release Notes** in the account menu, and in
[`CHANGELOG.md`](../CHANGELOG.md).
**The media mount was renamed** (the Files library): it is mounted at `/files` (`FILES_ROOT`), set
from `FILES_HOST_PATH`, and the default volume is `siftlode_files`. Until you move, nothing breaks: a
compose file from before still mounts `/downloads` and sets `DOWNLOAD_ROOT`, and the app still reads
that name. To move to the new layout, download the new
[`docker-compose.selfhost.yml`](../docker-compose.selfhost.yml) over your old one, then:
- **If you set a host directory**, nothing to do: the new compose file still reads
`DOWNLOAD_HOST_PATH` when `FILES_HOST_PATH` is not set. Rename it in `.env` when it suits you.
- **If you used the default volume**, copy it over once before you start the new version (Docker
cannot rename a volume). Compose names a volume after the project, which is your folder's name
unless you set one, so look the old one up rather than guessing it:
```bash
docker compose -f docker-compose.selfhost.yml down
docker volume ls --format '{{.Name}}' | grep 'siftlode_downloads$' # e.g. mysiftlode_siftlode_downloads
OLD=mysiftlode_siftlode_downloads # the name it printed
NEW="${OLD%siftlode_downloads}siftlode_files"
docker volume create "$NEW"
docker run --rm -v "$OLD":/from -v "$NEW":/to alpine cp -a /from/. /to/
docker compose -f docker-compose.selfhost.yml up -d
```
Open the Files library and check your files play; only then remove the old volume
(`docker volume rm "$OLD"`).
- **If you build from source**, a `git pull` brings the new `docker-compose.yml`, whose default
volume is `files` (it was `downloads`). Copy it over before you start the new version — with your
own compose file, not the self-host one above, whose volumes are other ones:
```bash
docker compose down
docker volume ls --format '{{.Name}}' | grep '_downloads$' # e.g. siftlode_downloads
OLD=siftlode_downloads # the name it printed
NEW="${OLD%downloads}files"
docker volume create "$NEW"
docker run --rm -v "$OLD":/from -v "$NEW":/to alpine cp -a /from/. /to/
docker compose up -d --build
```
If the files library is not mounted where the app looks, the API says so in its log at start
(`The files library is not reachable`), and the download worker refuses to start rather than write
files into a container that throws them away. A fresh, empty volume in the library's place (the old
one not copied over) is a real mount, so the worker runs; the API's log says that too, at start
(`The files library is empty but the database names files`). Copy the old volume over, as above.
**The registry keeps only the newest few releases, not every version the changelog lists.** A pin is
"decide when you take the next hop", not "stay here forever": once you are several releases behind,
your pinned tag can no longer be pulled — which matters the day you rebuild the host, prune your
+7
View File
@@ -17,6 +17,13 @@ siftlode e2e-seed # (re)seed the test account only, no tests
The localdev stack must be up (`siftlode dev`). `global.setup.ts` reseeds the account and logs in
once, sharing the session via `e2e/.auth/state.json` (git-ignored).
**A dropped localhost connect** — on a Windows host, a burst of test traffic can drop a loopback
connect now and then (every port, any program). The browser resends it and only waits (a loader
outlives an `expect` timeout); Node does not, so a request from the test side fails with
`connect ETIMEDOUT`. Those requests go through `retryConnect` (`e2e/retryConnect.ts`), which tries
that one error again and passes every other failure straight on. Don't run two e2e suites on the
same machine at once.
**Browsers** — every flow runs in three projects: `chromium` (Desktop Chrome), `webkit` (Desktop
Safari) and `ipad` (iPad gen 7 landscape: 1080×810, touch, mobile). Install the engines once per
machine with `npx playwright install chromium webkit`; run one project with
+42
View File
@@ -0,0 +1,42 @@
import { test, expect } from "@playwright/test";
import { gotoPage, openApp } from "./helpers";
// The Files library end to end, against the real app: the page opens on the whole library, and a
// link — typed into its box, or pasted onto any page — opens the one download dialog on it. Nothing
// is downloaded: the dialog is closed, so the run leaves the E2E account's library as it found it.
test.describe("Files", () => {
const LINK = "https://www.youtube.com/watch?v=jNQXAC9IVRw";
const dialog = (page: import("@playwright/test").Page) =>
page.getByRole("dialog", { name: "Download video" });
test("opens on the whole library, and its link box hands a link to the download dialog", async ({
page,
}) => {
await openApp(page);
await gotoPage(page, "files");
await expect(page.getByRole("heading", { name: "All files" })).toBeVisible();
const box = page.getByRole("textbox", { name: "A link to download" });
await box.fill(LINK);
await box.press("Enter");
await expect(dialog(page)).toBeVisible();
await expect(dialog(page)).toContainText("jNQXAC9IVRw");
await page.keyboard.press("Escape");
await expect(dialog(page)).toBeHidden();
});
test("a link pasted onto a page (not into a field) opens the download dialog", async ({
page,
}) => {
await openApp(page);
await page.evaluate((text) => {
const e = new Event("paste", { bubbles: true, cancelable: true });
Object.defineProperty(e, "clipboardData", {
value: { getData: (type: string) => (type === "text/plain" ? text : "") },
});
document.body.dispatchEvent(e);
}, LINK);
await expect(dialog(page)).toBeVisible();
await page.keyboard.press("Escape");
await expect(dialog(page)).toBeHidden();
});
});
+8 -3
View File
@@ -3,6 +3,7 @@ import { execFileSync } from "node:child_process";
import { mkdirSync, readFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { retryConnect } from "./retryConnect";
/**
* One-time setup for the characterization net (the `setup` project the tests depend on):
@@ -32,8 +33,12 @@ setup("seed + authenticate", async ({ page }) => {
"E2E_PASSWORD is not set — run the suite through `siftlode e2e` (it reads the secret store)",
).not.toBe("");
// These requests run in Node (sharing the page's cookie jar), so a localhost connect the machine
// dropped is tried again (retryConnect).
const api = retryConnect(page.request);
// 1. Preflight: fail fast (with a useful message) if the stack isn't running.
const health = await page.request.get("/healthz");
const health = await api.get("/healthz");
expect(
health.ok(),
`localdev stack not reachable at ${BASE_URL} — start it with \`siftlode dev\``,
@@ -66,12 +71,12 @@ setup("seed + authenticate", async ({ page }) => {
// 3. Authenticate. page.request shares the browser context's cookie jar, so the `session`
// cookie it receives is captured by storageState below.
const login = await page.request.post("/auth/password-login", {
const login = await api.post("/auth/password-login", {
data: { email: EMAIL, password: PASSWORD },
});
expect(login.ok(), "password-login failed — is the E2E account seeded?").toBeTruthy();
const me = await (await page.request.get("/api/me")).json();
const me = await (await api.get("/api/me")).json();
expect(me.authenticated, "session not established after login").toBeTruthy();
// 4. Pin the UI locale, suppress the first-login onboarding wizard (a full-screen overlay
+1 -1
View File
@@ -10,7 +10,7 @@ test.describe("page switching", () => {
// `nav-admin`) that opens a panel of them — the spec still listed them as rail rows of their own,
// and it failed on every run for a while without anyone seeing it, because e2e was in no
// required gate (it is now part of `release`).
const RAIL = ["channels", "playlists", "stats", "settings", "feed"];
const RAIL = ["channels", "playlists", "files", "stats", "settings", "feed"];
const ADMIN = ["scheduler", "config", "users", "audit"];
test("each rail item activates its page", async ({ page }) => {
+34
View File
@@ -0,0 +1,34 @@
// Playwright's request context (it runs in Node), trying a connection again when it never came about —
// as every TCP stack does. Measured on the Windows dev desktop (2026-10-10, the work-group-timesheet
// suite): now and then, in bursts of 1–15 s, a connect to localhost is dropped — every port, IPv4 and
// IPv6, any program, no Docker needed; set off by any big test run on the machine. Every other client
// sends its SYN again and gets in a moment later, but Node (libuv) sends a loopback SYN only once on
// Windows and fails at ~300 ms with "connect ETIMEDOUT". The request never left this side, so trying it
// again is safe: the waits the system's own retries take (0.5, 1, 2, 4, 8 s). Anything else — a refused
// port, a reset, an answer — goes straight to the test. (The browser resends its own SYN: a page only
// waits, and nothing here can or should hide that.)
const METHODS = new Set(["get", "post", "put", "patch", "delete", "head", "fetch"]);
const NO_CONNECTION = /\bconnect ETIMEDOUT\b/;
const WAITS_MS = [500, 1000, 2000, 4000, 8000];
/** `ctx` with its request methods trying a connect that timed out again (see above). */
export function retryConnect<T extends object>(ctx: T): T {
return new Proxy(ctx, {
get(target, key) {
const v: unknown = Reflect.get(target, key, target);
if (typeof v !== "function") return v;
if (typeof key !== "string" || !METHODS.has(key)) return v.bind(target);
return async (...args: unknown[]) => {
for (let i = 0; ; i++) {
try {
return await v.apply(target, args);
} catch (e) {
const wait = WAITS_MS[i];
if (wait === undefined || !NO_CONNECTION.test(String((e as Error)?.message))) throw e;
await new Promise((r) => setTimeout(r, wait));
}
}
};
},
});
}
+28 -14
View File
@@ -1,6 +1,7 @@
import { test, type Page } from "@playwright/test";
import { execFileSync } from "node:child_process";
import { mkdirSync, writeFileSync } from "node:fs";
import { retryConnect } from "./retryConnect";
/**
* The speed baseline: numbers, not assertions — read by `scripts/audit_metrics.py --speed`, which
@@ -16,7 +17,7 @@ import { mkdirSync, writeFileSync } from "node:fs";
const RUNS = 5;
const OUT = "e2e/.speed/latest.json";
const CONTAINER = process.env.E2E_DB_CONTAINER || "siftlode-db-1";
const CONTAINER = process.env.E2E_API_CONTAINER || "siftlode-api-1";
const median = (xs: number[]) => [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)] ?? NaN;
@@ -31,20 +32,33 @@ const scriptBytes = (page: Page) =>
/** Public share links on the local stack that could play: no password, not expired, asset ready.
* The page itself is the judge of the last step (its file must be on THIS disk — a localdev DB
* seeded from prod lists assets whose files never came along), so the caller probes them. */
* seeded from prod lists assets whose files never came along), so the caller probes them. Read
* through the API container: the database keeps a link's token only hashed and encrypted, and only
* the app holds the key to read it back (`links.recover`). */
const SHARE_TOKENS_PY = `
from sqlalchemy import select
from app.db import SessionLocal
from app.downloads import links
from app.models import DownloadJob, DownloadLink, MediaAsset
with SessionLocal() as db:
for link in db.execute(
select(DownloadLink)
.join(DownloadJob, DownloadJob.id == DownloadLink.job_id)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadLink.password_hash.is_(None), MediaAsset.status == "ready")
.order_by(DownloadLink.id)
).scalars():
token = None if links.is_expired(link) else links.recover(link)
if token:
print(token)
`;
function shareTokens(): string[] {
const sql =
"select l.token from download_links l join download_jobs j on j.id = l.job_id" +
" join media_assets a on a.id = j.asset_id" +
" where l.password_hash is null and (l.expires_at is null or l.expires_at > now())" +
" and a.status = 'ready' order by l.id";
// Not caught: a database this cannot reach (no docker, a renamed container) is a failed
// Not caught: an API container this cannot reach (no docker, a renamed container) is a failed
// measurement, not "no playable share link on the local stack".
const out = execFileSync(
"docker",
["exec", CONTAINER, "psql", "-U", "siftlode", "-d", "siftlode", "-tAc", sql],
{ encoding: "utf8" },
);
const out = execFileSync("docker", ["exec", CONTAINER, "python", "-c", SHARE_TOKENS_PY], {
encoding: "utf8",
});
return out
.split("\n")
.map((t) => t.trim())
@@ -83,7 +97,7 @@ test("speed baseline", async ({ browser, page: page0 }) => {
// the visitor's tap — the page has no autoplay).
let token: string | null = null;
for (const t of shareTokens()) {
if ((await page0.request.get(`/api/public/watch/${t}`)).ok()) {
if ((await retryConnect(page0.request).get(`/api/public/watch/${t}`)).ok()) {
token = t;
break;
}

Some files were not shown because too many files have changed in this diff Show More