Compare commits

..
57 Commits
Author SHA1 Message Date
peter c85ae42b31 release: 0.108.0 (feature/e40-job-history) 2026-10-18 21:48:23 +02:00
peter cb9a89682a chore(release): 0.108.0 — one Activity page for runs, admin actions, the server log and quota; every background run has a history 2026-10-18 21:40:52 +02:00
peter b842e4e871 fix(ui): a looping pulse stops for reduced motion; the e2e waits for skeleton bones, not for any pulse
- Seven looping pulses (a running job's dot, the scheduler's progress bar, a queued download, the
  download tray, a playlist upload, a channel's full-history action, the setup wizard's test mail)
  ran regardless of prefers-reduced-motion, which docs/DESIGN.md promises to respect: each is
  motion-safe now, held by a contract test.
- The a11y suite's "settled" wait took every aria-hidden pulse for a loading skeleton, so the
  Scheduler legend's running dot, which pulses for good, kept it waiting until it timed out. A
  skeleton bone now says it is one (data-skeleton) and the wait looks for that.
2026-10-18 21:40:52 +02:00
peter 5082636f15 fix(activity): a run nothing here has open reads as interrupted, a rollback sums a stopped run's spend
- The scheduler keeps one registry of the runs this process has open, for the interval jobs and the
  ad-hoc runs alike. A row the table still says is `running` but no run here has open (its outcome
  failed to write) reads as interrupted, as the next start's sweep will close it: no pulsing dot
  beside "No run recorded yet".
- Migration 0109's downgrade sums a still-running run's quota events into its spend before the link
  to them is dropped, as the startup sweep would.
- A test holds that a dialog closing onto the same address leaves an open log line open.
2026-10-18 21:21:19 +02:00
peter 13b7577d9f fix(activity): an ad-hoc run's row is one run's, a rollback keeps a run whole, Back reads one entry's filters
- The channel-backfill row says "running" from the runs this process has open, not from the table:
  a row whose outcome failed to write no longer reads as running until a restart. The run still
  going leads the row, so a newer run's outcome never sits under the older one's channel.
- Migration 0109's downgrade carries every run's trigger, spend and start in the audit row, and the
  upgrade restores them: a manual run whose admin was deleted stays manual.
- Back/Forward is read by the page's one popstate listener, both halves in one render; the server
  log adopts its half during that render, so no read goes out with one entry's filters and the
  other's range.
- The server log's failed facets use the other tabs' words (the menus may be short); the accounts
  message shows only when the menu really lacks them.
2026-10-18 17:32:22 +02:00
peter fad656988a fix(activity): runs keep their true outcome and spend, the page reaches every kept row, a closing dialog leaves the server log alone
Backend
- A run's outcome write sits outside the job's own try: a database blip after a run that
  succeeded no longer records it as failed (the row waits for the next start's sweep).
- An interrupted run keeps what it spent (the sweep sums its quota rows, as finish does).
- The Scheduler's ad-hoc row says running while any backfill runs, not only the newest.
- recent_runs reads each job's last runs off its index instead of ranking the whole table.
- An interval change is stored with its audit row in one transaction, then rescheduled.
- One API instance per database is the stated contract (.env.example, jobruns).
- Turning full history on for a channel already complete records no run.
- Migration 0109 round-trips interrupted and ad-hoc runs; the run words' DB CHECKs are
  test-held to the enums; the account filter is one helper for the four activity reads.
- An audit row says how many of its request's log lines are still kept.

Frontend
- An "All" period reaches every row still kept (runs and admin actions outlive 90 days).
- Back/Forward is followed in place by the server log: a closing dialog no longer turns
  Live off; a pivot still opens a fresh log.
- Runs moved over from the audit log read their one-liner in the reader's language.
- Filter choices or accounts that failed to load say so beside the filters, with a retry.
- An old server-log link keeps its account, text and range; a remembered retired page
  opens its tab.
- A request whose lines are gone says so instead of linking to an empty list.
- The log lines and the history rows share one row frame, and the server log one list
  frame with the other tabs; an empty list is announced; each day's quota is readable.
2026-10-18 15:38:32 +02:00
peter 3021be5166 fix(activity): a run's quota reads with its unit, its outcome keeps room at a large text size, a token's audit rows read in Hungarian
The run list showed the quota spend as a bare figure and, at the largest text size on a 1024-wide
screen, its fixed columns left the outcome column no width at all. The outcome now keeps 10rem and
the trigger column gives way; the audit list's summary keeps the same floor. A run that stored no
log line says so instead of linking to an empty list. An API token revoke and a token's file work
rendered the server's English line under the Hungarian UI; both are rebuilt in the reader's
language, and the test holds the server's wording.
2026-10-18 04:22:52 +02:00
peter c288ba147b fix(activity): an old audit/log link opens its tab, times read relative, one name for admin actions
Seen in the browser, not in the page's own tests: the navigation provider rewrites a retired page id
to `activity` before the page's chunk reads the URL, so `?page=audit` landed on the Runs tab. The
retired-page map now carries the params that open the successor's part (`at`), and the provider
writes them as it rewrites the page — on load and on Back alike.

A list time is relative with the exact second on hover (the DESIGN copy rule); a run moved over from
the audit log, whose start and end are one instant, shows no length instead of an invented "1 sec".
The audit log is called "admin actions" everywhere it is named (the clear button and its confirm, the
Configuration group, the cleanup job, which also prunes the run history its description left out).
2026-10-18 04:00:38 +02:00
peter fe6308d844 refactor(activity): the window's hours stay private to its URL state 2026-10-18 03:48:30 +02:00
peter 24448e7c6f feat(activity): one Activity page — runs, admin actions, server log and quota under one filter bar
The audit log, the server log and the admin quota view were three pages (the quota one a Stats tab),
each with its own filters, and the background runs had no list at all. Now one page ("Napló" /
"Activity") reads all four, each still its own table on the server: a shared bar (text, a period of
1 h to 90 days defaulting to 7, an account) and each tab's own filters, the whole view in the URL.
The old page ids open their tab.

Server side: the audit log and the new run list (`/api/admin/runs`) filter and page like the server
log, through one keyset helper (`routes/_keyset`) the log now shares; an audit row keeps the request
it was made in (migration 0111), so its log lines are a pivot away, as a run's are (`lrun`). The
quota breakdown takes since/until and one account. The Scheduler rows get a history button to their
job's runs.

Also fixed on the way: re-saving the value in force on the Configuration page staged its override
without committing it (the commit sat inside the audit branch), so the save was lost.
2026-10-18 03:45:18 +02:00
peter d13483a593 feat(config): job intervals live in sysconfig, every setting says when it applies, the unread SponsorBlock default goes
An interval override was a row of its own table beside the one store every other admin knob uses; it
is now the `job_interval_<job>` sysconfig key (migration 0110 copies them over and drops the table).
The Scheduler page stays its one editor: the Configuration page and the setup wizard neither list nor
write those keys, since a write there would skip the reschedule.

Each ConfigSpec now states `applies` (required, no default). Every key is `live`, held by a scan that
finds no read at import time; a `restart` key is refused until the page can say so. A second scan
requires a reader for every key on the Configuration page — `sponsorblock_default` had none since it
was added, so it leaves the registry, the translations and the store.
2026-10-18 02:23:15 +02:00
peter 3bab2e7dc7 docs(scheduler): the snapshot docstring names the ad-hoc runs it now carries 2026-10-18 01:29:46 +02:00
peter 1b1e3cdcf2 feat(scheduler): each row shows its last three runs, an interrupted run as such, and the ad-hoc runs beside the jobs
A row's one dot said how the LAST run went, and an interrupted run fell through to "idle" — a grey dot
that read as "no run recorded". The row now leads with its last three runs, newest first, each named
for a screen reader and in its tooltip (how it went, when, who started it); an interrupted run is a
ring, set apart from a skipped run and a failure, and its line says the server restarted under it.

A channel's full-history backfill gets its own "Ad-hoc runs" block: its runs, its channel and what
it fetched, no interval and no run button — it needs its target. AD_HOC_JOBS names every such job;
the backend-id contract holds their names in both languages, and run_ad_hoc refuses any other id.
2026-10-18 01:20:53 +02:00
peter 27c2fcbd2e feat(scheduler): a channel's "get full history" backfill is a recorded run, and a row knows its last three runs
The one-channel deep backfill ran as a FastAPI BackgroundTask: what it fetched, what it spent and
how it failed reached only the log. It is now a one-shot run on the scheduler (`channel_backfill`,
trigger ad_hoc, actor = the user, target = the channel) with its quota tagged to the run. Its
per-channel exclusion with the scheduled backfill and the two-run cap are unchanged, and a run
they turn away says why in its result instead of returning nothing. A failure is no longer
swallowed: the run records it.

The scheduler snapshot carries each job's last three runs (one windowed query), so the page can
show an interrupted or skipped run instead of the latest outcome alone.
2026-10-18 01:05:24 +02:00
peter 911527a548 fix(scheduler): the dashboard no longer says a job has not run "this session" — runs are kept across restarts 2026-10-17 23:22:13 +02:00
peter 8fd91b7b15 fix(scheduler): keep run history 180 days, as long as the audit log kept the runs it moved over 2026-10-17 23:15:17 +02:00
peter 45c0ee7d58 feat(scheduler): every background run is a durable row
What a job last did lived in the API process's memory and was gone at
every restart; the audit log kept a run only when it was manual, failed
or changed something. Now each run is a job_runs row written at its
start and its end in its own session: who started it, its outcome as
data, what it spent of the YouTube quota (quota_events.job_run_id) and
the lines it logged (log_events.run_id, filterable in the log console).

- a run a stopped process left open is marked interrupted at startup
- "run now" is a one-shot job on the scheduler, which now always starts
  (interval jobs only with SCHEDULER_ENABLED); the manual actor is reset
  after the run, since pool threads are reused
- the dashboard's last run comes from job_runs, so it survives a deploy
- runs leave the audit log: migration 0109 moves the old job_run rows
  over (and back on downgrade); the audit page's run folding goes
- job_run_retention_days (default 30) prunes old runs in audit_gc
2026-10-17 23:08:55 +02:00
peter 4b1f6d7ec2 release: 0.107.3 (improvement/dev-tooling) 2026-10-17 21:34:53 +02:00
peter 51bf780100 chore(release): 0.107.3 — text on glass holds 4.5:1; the e2e suite runs axe, the gate lints every shell script 2026-10-17 21:21:37 +02:00
peter 3ada34073b fix(theme): text on glass holds 4.5:1 — light --muted kept at 6:1 on the page, tonal fills opaque
Pixel-measured over every scheme x mode x backdrop: the item count fell to 3.8:1 (dark too: its 8%
fill let the toolbar glass and the thumbnails under it through), Channels' tabs to 4.1:1 and the
current Files row to 4.4:1. Light --muted moves 15% toward --fg in every scheme; the badges, the item
count and the current rail row paint opaque bg-tint* fills mixed into the surface. The token contract
now asks 6:1 of --muted on the bare page and checks those three sites use the tints.
2026-10-17 20:06:54 +02:00
peter d74f2a9ca1 fix(deps): source-map-js 1.2.2 — the indexed-source-map denial-of-service advisory (dev tooling only) 2026-10-17 17:51:10 +02:00
peter a8dddc8c24 test(e2e): axe-core's WCAG A + AA rules over every page and the player (chromium only); contrast stays with the visual check 2026-10-17 17:48:53 +02:00
peter 460bdec406 chore(tooling): drop scripts/lint-changed.sh — the shared gate runner's Stop hook lints the turn's files, shellcheck included 2026-10-17 17:22:35 +02:00
peter 66181d1394 chore(gate): shellcheck joins the gate — every tracked shell script, the host binary held to a pin 2026-10-17 15:31:34 +02:00
peter 0a1fef28ac docs(claude): the YouTube write-error helper, the overlay and z-scale rule, and px-frozen sizes join the canonical patterns 2026-10-17 14:47:57 +02:00
peter fe572d9207 docs: a copy-only change runs the Conformance check's step 4 alone, in both languages 2026-10-17 14:27:45 +02:00
peter c8247c0dde release: 0.107.2 (improvement/gate-runner) 2026-10-17 04:37:30 +02:00
peter 722c37a6d0 chore(release): 0.107.2 — the gate lists and runs single lanes, the backend tests use every core 2026-10-17 04:30:33 +02:00
peter 5ba5679239 test: tests that assumed an idle machine wait for what they assert, or carry a budget for their work
Under a busy machine (the gate's lanes side by side, 12 busy cores) these went red at random: a press landed while the button was still busy, an answer was settled before it was asked for, a page grew past one wait's budget, a lazy chunk was transformed inside a 1 s wait, focus was taken a frame after the press. They now wait for the condition they depend on, preload the chunks they open, and the tests that do heavy work (whole-repo scans, the type-aware linter, module resets) carry an explicit budget naming that work. The whole suite four times under 12 busy-loop processes: red 4/4 before, 0/4 after.
2026-10-17 01:08:15 +02:00
peter 926e5e9899 chore(gate): check.sh can list its lanes and run one of them
--list prints the lane names, --only "<lane>" runs that one lane, so a runner can schedule the lanes itself and a red lane can be re-run on its own. An --only run reports that lane, never the gate, and an unknown name exits 2 instead of passing with nothing run; the contract test holds both.
2026-10-16 02:53:30 +02:00
peter 9251918e08 chore(test): the DB lane runs on every core — each pytest-xdist worker gets its own database 2026-10-16 01:13:31 +02:00
peter 2b6d968833 release: 0.107.1 (improvement/gate-timing) 2026-10-16 00:45:45 +02:00
peter 95e6b2c3d9 chore(release): 0.107.1 — a failed download's explanation says press Retry 2026-10-16 00:33:59 +02:00
peter 80c55fbf1b merge bug/download-retry-copy into improvement/gate-timing 2026-10-16 00:33:38 +02:00
peter 011f269152 chore(gate): every lane reports how long it took, and a run can log the times for comparison 2026-10-16 00:13:47 +02:00
peter dcb244dd04 fix(downloads): a failed download's explanation says press Retry, not retry it from a list that is gone 2026-10-15 22:56:40 +02:00
peter 08a9f4e255 release: 0.107.0 (feature/e39b-machine-access) 2026-10-15 06:20:35 +02:00
peter 1673134c36 chore(release): 0.107.0 — API tokens for your machines, share a whole folder by link 2026-10-15 06:09:23 +02:00
peter 27427eef32 fix(files): ending every session and making a token lock the user without holding up rows that refer to it 2026-10-15 06:05:51 +02:00
peter a248ca6c13 fix(files): the token list is fresh after the sessions end, a copy again reads Copied again, the Copied button says so 2026-10-15 05:47:00 +02:00
peter 620f84dba4 fix(files): a token made as every session ends goes with them, and a file being shared holds its job 2026-10-15 05:47:00 +02:00
peter 2efa7d5a75 fix(files): an upload is said once in any order, copy buttons speak, a link list that failed says so
- The queue holds an upload it sees land done while this tab's upload still waits for its answer, so
  the dialog's notice is the only one whichever comes first; another device's upload follows after.
- One `useCopyFeedback` for every Copy beside a value (token, share link, set-password link): Copied
  for a moment, a notice a screen reader hears, a warning when the clipboard refuses; the token
  dialog's two Copy buttons are named by what they copy.
- The share dialog's link list shows loading, and a load error with Retry, never "no links" — and no
  New link beside a list it has not seen.
- The settings texts say that signing out elsewhere or a password change revokes the API tokens.
- DESIGN.md logs the folder-link decisions; the design lab keeps only the chosen renderings of the
  Links, Tokens and Files rounds.
2026-10-15 04:58:17 +02:00
peter 827df7494c fix(files): ending every session ends every API token, and the share links hold their folder
- A password reset or change, "log out everywhere" and a revoked untrusted password revoke all of the
  account's API tokens too (audited): one minted from a stolen session no longer outlives the answer.
- A bearer token or link grant with a non-ASCII character is refused (401/403), not a 500.
- Making a token's grants or a folder link holds the folder, so a racing delete cannot fail the
  foreign key; a link's view count is one atomic UPDATE.
- A folder link's card counts exactly the files its page lists (one `links.folder_files`).
- A file with no extension is served without a trailing dot.
- 0106's downgrade names a Hub folder so it never collides with one of the user's folders.
- The speed baseline reads share tokens through the app (the database keeps them hashed/encrypted).
- Tests: the token scope on the real session stack, the folder poster, the Hub refused for a link,
  0106's downgrade data step, 0107 without a key; docs/comments follow (three folder kinds, folder
  links, key change, the cleanup job's token warnings).
2026-10-15 04:58:16 +02:00
peter 86712b6b0e feat(files): share a folder by link — its page lists what is in it, plays a video above the list
A /watch link may now open one of its owner's own folders (migration 0108: folder_id beside job_id,
exactly one set; a folder's links go with it). The page shows what is in the folder when it is
opened, newest first: a video or a song plays in the same player a file link has, above the list,
each keeping its own resume point; a document only downloads, never opens as a page. A file moved
out stops opening through the link at its next request. Password, expiry and stream-only work as on
a file link; the unfurl names the folder and counts its files, never their names. Source folders
and the Hub cannot be shared.

The folder menu gets "Copy link" and "Share with a link…" (the Share dialog's link half); the rail
marks a folder a live link opens.

Two fixes in the Share dialog on the way: revoking the last link no longer mints a new one at once,
and the new-link form goes back to downloadable after making a link.
2026-10-15 02:23:24 +02:00
peter 90567718b2 feat(files): share-link tokens are kept only hashed and encrypted, and every link change is audited
A /watch link's token is the credential, and it sat in the database as written. Migration 0107
replaces the column with the token's SHA-256 (the lookup) and a Fernet-encrypted copy (so the owner
can copy the same link again); every existing address keeps working, and the downgrade decrypts
them back. Without a copy this key can read, a link still opens but its address cannot be shown:
"Copy link" then takes the newest link that has one or makes a new one, and the Share dialog says
so in that row instead of a broken URL. Making, changing and revoking a link are audited, never
with its token or password.
2026-10-15 01:02:29 +02:00
peter c31d7f9d65 fix(files): an upload is announced once, as an upload, and a finish shows in Files at once
An upload that came back done was announced twice: by the upload dialog ("File uploaded") and
again by the tray, which saw a job turn done ("Downloaded: …"). The dialog now marks its job as
announced, so the queue keeps it for the tray but does not say it again. An upload the worker
converted first is still the tray's to announce, now as "Uploaded: …".

A finished job's notice offered "Open" while Files still listed the folder without it (its own
poll runs every 15 s): the tray now refreshes the library the moment it sees a finish.
2026-10-14 23:54:19 +02:00
peter db56f25387 feat(files): API tokens in Settings, the Hub folder in Files, a notice before a token runs out
Settings gains an API tokens tab: a row per token (name and prefix; each
folder with its rights; last use and expiry, "expires soon" marked),
revoked from its menu behind a confirm. A new token is the Hub exchange in
one click, or folders and rights picked per folder (no Write where an
upload cannot land); expiry 30/90 days, 1 year or never (warned). The
token shows once in the same dialog, with Copy and a ready curl line.

The Hub folder closes the Sources in the Files rail, takes uploads and
moved-in files. A token about to run out warns its user once, a week
before (the Files GC's sixth pass); the notice opens the tokens tab.
2026-10-14 23:33:34 +02:00
peter 4f58d15d3f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 21:19:14 +02:00
peter 2c66ce1cf4 release: 0.106.2 (improvement/thumbs-and-apt-refresh) 2026-10-14 21:17:39 +02:00
peter ce45445d17 chore(release): 0.106.2 — pictures that fail to load retry instead of breaking 2026-10-14 21:06:18 +02:00
peter 383d394e32 fix(ui): a picture that fails to load shows the card's own frame, then retries
A bare <img> sent a Referer that Google's image CDNs refuse and had no retry and no fallback, so a
Home shelf card showed the browser's broken-image mark. ui/RemoteImage gathers what Avatar, the
channel banner and the download rows had each hand-rolled: no Referer, up to three retries past the
browser's negative cache, the caller's fallback (by default the card's empty frame) instead of the
broken mark, the failure remembered by URL. Every picture now goes through it; a lint rule refuses
a bare <img>.
2026-10-14 20:51:08 +02:00
peter 06e8291896 build(image): rebuild the apt-get upgrade layer when APT_REFRESH changes
A build cache kept the upgrade layer for as long as nothing above it changed, so a publish shipped
curl and librsvg a release behind fixes already in the Debian archive (the CVE gate refused it).
The release build passes the day; empty, the default for a self-hosted build, leaves the cache alone.
2026-10-14 20:51:07 +02:00
peter db6b60029f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 20:17:05 +02:00
peter 02134799cc release: 0.106.1 (bug/plex-settings-outside-click) 2026-10-14 20:03:38 +02:00
peter c8bc843517 chore(release): 0.106.1 — the Plex player survives a click outside its Settings panel 2026-10-14 19:44:25 +02:00
peter 759a7536b9 fix(history): an overlay opened while a back() is in flight waits to push its entry
The Plex player's ⋯ → Settings closed the menu (its entry went back) and opened the panel a task
later, before that back() had landed. The panel's push was the entry the landing back() took, so
the books counted one entry more than the history held, and closing the panel by a click outside
it stepped back over the player's own entry: the player closed.

A push now waits while a back() of ours is in flight; the popstate that lands the last one
reconciles again.
2026-10-14 19:43:51 +02:00
peter b2d34e6e6c feat(files): API tokens — a machine reaches the library folder by folder
A user can make a token for a machine of theirs (`/api/tokens`, signed-in only): `sfl_` + 32
random bytes + a checksum, only its SHA-256 kept, shown once. Sent as a Bearer header it is
honoured on six `/api/files` routes only — folders, list, file, hash, upload, delete — and there
only in the folders its grants name, with the rights they give (list / read / write / delete).
One check, `FilesCaller.may`, in every route; a folder or file out of reach answers like a
missing one; the files shared with the user are never in reach. A token beats a cookie on the
same request.

The Hub preset makes the user's one Hub folder (a new folder kind) and grants it every right.
Refused tokens are slowed down per client, a runaway machine per token; last use is stamped at
most once a minute; making and revoking a token is audited, and a token's uploads and deletes
name it in their audit rows.
2026-10-14 18:40:23 +02:00
263 changed files with 13440 additions and 3385 deletions
+5 -3
View File
@@ -68,9 +68,11 @@ SMTP_FROM=
# --- Scheduler ---
# The background scheduler (subscription sync, backfill, enrichment) runs inside the app.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. If you ever run
# more than one instance against the same database, keep SCHEDULER_ENABLED=true on exactly one
# of them and false on the rest, to avoid double quota use and write races.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. Run ONE app instance
# per database: a starting instance closes every run still open as interrupted (it assumes the
# previous process is gone), and "is this job already running" is held in the process. A second
# instance would cut the first one's runs short and could run the same job twice.
# SCHEDULER_ENABLED=false keeps the recurring jobs off; "run now" still works.
SCHEDULER_ENABLED=true
# --- Files library & downloads ---
+58
View File
@@ -7,6 +7,64 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.108.0 — 2026-10-18
### New
- Activity: one admin page for background runs, admin actions, the server log and quota, with one time range, account and search. A run opens to its outcome and its log lines, an admin action to its request's lines; old audit and log links open their tab.
- Scheduler: each job shows its last three runs (a run cut short by a restart is marked) and a button opens all of them; turning on a channel's full history is now a run of its own, listed under Ad-hoc runs.
- Configuration: how long the run history is kept (Activity › Run history, 180 days by default). Stats keeps your own numbers; the system quota view moved to Activity › Quota.
### Fixed
- Configuration: saving a setting with the value already in force now keeps it, instead of dropping the change.
- Configuration: the unused SponsorBlock default setting is gone.
## 0.107.3 — 2026-10-17
### New
- Development: the end-to-end suite runs the WCAG A and AA accessibility checks over every page and the player, and every shell script is linted in the gate.
### Fixed
- Readability: muted text in the light themes is a shade darker, and the item count, the labels and the selected Files folder sit on a solid tint, so text on the frosted-glass bars stays readable (4.5:1) in every scheme.
## 0.107.2 — 2026-10-17
### New
- Development: the gate script can list its checks (scripts/check.sh --list) and run just one (--only), and the backend tests run on every CPU core, each worker with its own database.
## 0.107.1 — 2026-10-16
### Fixed
- Downloads: a failed download's explanation now says to press Retry, instead of pointing to a download list that no longer exists.
## 0.107.0 — 2026-10-15
### New
- API tokens: Settings › API tokens gives one machine of yours a key to the Files folders you pick — or, in one click, to a Hub folder for swapping files with it. Signing out your other sessions or changing your password revokes every token.
- Share by link: a link can now open a whole folder of yours — its page lists what is in it and plays a video above the list.
### Fixed
- Uploads: a finished upload is announced once, and shows in Files at once.
## 0.106.2 — 2026-10-14
### Fixed
- Pictures: a thumbnail, poster or channel picture that fails to load is retried, and meanwhile the card shows its empty frame instead of a broken-image mark.
## 0.106.1 — 2026-10-14
### Fixed
- Plex player: clicking outside the Settings panel closes the panel again instead of closing the player.
## 0.106.0 — 2026-10-13
### New
+10 -7
View File
@@ -12,7 +12,7 @@ router-free navigation provider.
## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
The local gate is **ruff (`check` + `format --check`) + shellcheck + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip + vitest + vite build + pytest + pip-audit + npm audit**. It runs
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
@@ -55,14 +55,15 @@ how it gets built. Concretely:
- **Challenge unsound requests.** If a request is technically ill-advised and you can back it with real
arguments, push back or propose an alternative — don't comply blindly.
- **Verify UI in the browser before calling it done.** tsc/unit/DOM tests miss visual and interaction
bugs; do a real visual check of any UI change, and run DESIGN.md's four-step Conformance check.
bugs; do a real visual check of any UI change, and run DESIGN.md's Conformance check (a copy-only
change runs its step 4 alone: every state the new text renders in, in both languages).
- **Be terse everywhere else.** Outside those decision points, lead with the answer or decision. During
implementation and debugging especially: no restating the same point reworded, no filler.
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell (a shared `/watch` link boots from its own
@@ -91,8 +92,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` and `jobruns` own their session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
@@ -101,7 +102,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect | a bare `catch {}` or `notify()` with no error attached |
| FE floating or full-screen layer | `ui/Overlay` (portals to `<body>`) or the popover engine (`AnchoredPopover` portals itself); stacking by the z-index scale (`z-menu` … `z-popover`, docs/DESIGN.md "Layers") | one rendered in place inside a virtualized row or a masked/transformed scroller — clipped or painted over; a magic `z-[n]` |
| FE player on a native media element | `lib/mediaSession` (`useMediaSession`) + `lib/mediaKeys` + `lib/resume` (`useResume`), the element held as STATE | a private `navigator.mediaSession` / `keydown` / position-saving effect per player |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
@@ -114,7 +116,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
Not repeated here, because a lint rule refuses the wrong one and its message names the right one:
buttons (`ui/Button`, `IconButton`, `Pressable`, `Chip`, `DragHandle`), text fields and native
controls (`ui/Input`, `Checkbox`, `Slider`, …), glass surfaces (`SURFACE.*`), a native `<select>`
(`SegmentedToggle` / `FilterSelect`), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
(`SegmentedToggle` / `FilterSelect`), a picture (`ui/RemoteImage`, never a bare `<img>`), a box or font size (the spacing scale or rem,
never a px-frozen `w-[Npx]` / `text-[Npx]`: the text-size slider scales the root font), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
## House rules that bite (learned from real bugs)
+7 -1
View File
@@ -55,8 +55,14 @@ RUN adduser --disabled-password --gecos "" appuser \
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# `apt-get upgrade`: the base image is a snapshot, and the Debian security fixes released since it
# was cut are only in the archive — the first Trivy scan found one (pcre2) fixed upstream but not in it.
# APT_REFRESH: the upgrade only runs when this layer is rebuilt, and a build cache keeps it for as long
# as nothing above it changes — a publish shipped an image whose curl and librsvg were a release behind
# fixes already in the archive. The release build passes the day here, so the layer is rebuilt at
# least daily; empty (the default, a self-hosted build) leaves the cache alone.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
ARG APT_REFRESH=
RUN echo "apt refresh: ${APT_REFRESH:-cached}" \
&& apt-get update \
&& apt-get upgrade -y --no-install-recommends \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
+1 -1
View File
@@ -1 +1 @@
0.106.0
0.108.0
-34
View File
@@ -630,41 +630,7 @@
}
}
],
"./app/routes/admin_logs.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 15,
"endColumn": 47,
"lineCount": 1
}
}
],
"./app/routes/audit.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 17,
"endColumn": 98,
"lineCount": 1
}
},
{
"code": "reportAssignmentType",
"range": {
"startColumn": 17,
"endColumn": 98,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 22,
"endColumn": 97,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
+110
View File
@@ -0,0 +1,110 @@
"""Machine access to the Files library: API tokens bound to folders, and the Hub folder
Revision ID: 0106_api_tokens
Revises: 0105_job_source_key
Create Date: 2026-10-14
`api_tokens` is a user's token for a machine (a script, the hub): only its SHA-256 is kept, plus the
first characters to recognise it by, and whether its user was told it is about to run out. `api_token_grants` says what it may do in which folder — list,
read, write, delete — so a token reaches nothing it was not given, folder by folder.
A third folder kind, `hub`: the one folder a user exchanges files with their hub machine through.
Like a source folder its name is the client's to render; like an own folder a file may be moved in.
One per user.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0106_api_tokens"
down_revision: str | None = "0105_job_source_key"
branch_labels: str | None = None
depends_on: str | None = None
_KIND_OLD = (
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
)
_KIND_NEW = _KIND_OLD + " OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)"
def upgrade() -> None:
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_NEW)
op.create_index(
"uq_file_folder_hub",
"file_folders",
["user_id"],
unique=True,
postgresql_where=sa.text("kind = 'hub'"),
)
op.create_table(
"api_tokens",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("name", sa.String(80), nullable=False),
sa.Column("prefix", sa.String(16), nullable=False),
sa.Column("token_hash", sa.String(64), nullable=False, unique=True),
sa.Column("expires_at", sa.DateTime(timezone=True)),
sa.Column("last_used_at", sa.DateTime(timezone=True)),
sa.Column("expiry_warned", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_table(
"api_token_grants",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"token_id",
sa.Integer(),
sa.ForeignKey("api_tokens.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("can_list", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_read", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_write", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_delete", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
sa.CheckConstraint(
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
),
)
def downgrade() -> None:
op.drop_table("api_token_grants")
op.drop_table("api_tokens")
op.drop_index("uq_file_folder_hub", table_name="file_folders")
# A Hub folder may hold files (their foreign key RESTRICTs a delete): it becomes an own folder,
# named "Hub <id>" — or "Hub <id> (2)", … when its user already has a folder of that name,
# which `uq_file_folder_own_name` (per user, case-blind) would refuse.
conn = op.get_bind()
hubs = conn.execute(sa.text("SELECT id, user_id FROM file_folders WHERE kind = 'hub'")).all()
for folder_id, user_id in hubs:
name, n = f"Hub {folder_id}", 1
while conn.execute(
sa.text("SELECT 1 FROM file_folders WHERE user_id = :u AND lower(name) = lower(:n)"),
{"u": user_id, "n": name},
).first():
n += 1
name = f"Hub {folder_id} ({n})"
conn.execute(
sa.text("UPDATE file_folders SET kind = 'own', name = :n WHERE id = :i"),
{"n": name, "i": folder_id},
)
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_OLD)
@@ -0,0 +1,66 @@
"""Share-link tokens no longer stored in the clear
Revision ID: 0107_link_token_at_rest
Revises: 0106_api_tokens
Create Date: 2026-10-15
A `/watch` link's token is the credential itself, and it sat in `download_links.token` as written,
so a copy of the database was every shared file. Now a link is found by the token's SHA-256
(`token_hash`), and the token is kept only Fernet-encrypted (`token_enc`, TOKEN_ENCRYPTION_KEY) so its
owner can copy the same link again. Without a key the copy is not kept: the link still opens, its
address just cannot be shown again (`app/downloads/links.py`).
Every existing link is converted here, so the addresses already sent keep working. The downgrade
writes the tokens back from their encrypted copies; a link with none left (no key, or a key since
changed) gets a fresh random token, which is the same as revoking it.
"""
import hashlib
import secrets
import sqlalchemy as sa
from alembic import op
from app import security
revision: str = "0107_link_token_at_rest"
down_revision: str | None = "0106_api_tokens"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_links", sa.Column("token_hash", sa.String(64)))
op.add_column("download_links", sa.Column("token_enc", sa.Text()))
conn = op.get_bind()
sealing = security.can_encrypt()
for link_id, token in conn.execute(sa.text("SELECT id, token FROM download_links")).fetchall():
conn.execute(
sa.text("UPDATE download_links SET token_hash = :h, token_enc = :e WHERE id = :id"),
{
"id": link_id,
"h": hashlib.sha256(token.encode()).hexdigest(),
"e": security.encrypt(token) if sealing else None,
},
)
op.alter_column("download_links", "token_hash", nullable=False)
op.create_unique_constraint("uq_download_link_token_hash", "download_links", ["token_hash"])
op.drop_constraint("uq_download_link_token", "download_links", type_="unique")
op.drop_column("download_links", "token")
def downgrade() -> None:
op.add_column("download_links", sa.Column("token", sa.String(64)))
conn = op.get_bind()
rows = conn.execute(sa.text("SELECT id, token_enc FROM download_links")).fetchall()
for link_id, enc in rows:
token = security.decrypt_optional(enc) or secrets.token_urlsafe(24)
conn.execute(
sa.text("UPDATE download_links SET token = :t WHERE id = :id"),
{"id": link_id, "t": token},
)
op.alter_column("download_links", "token", nullable=False)
op.create_unique_constraint("uq_download_link_token", "download_links", ["token"])
op.drop_constraint("uq_download_link_token_hash", "download_links", type_="unique")
op.drop_column("download_links", "token_enc")
op.drop_column("download_links", "token_hash")
@@ -0,0 +1,40 @@
"""A share link may open a folder
Revision ID: 0108_folder_links
Revises: 0107_link_token_at_rest
Create Date: 2026-10-15
A `/watch` link opened one file. Now it may open one of its owner's own folders instead: the link's
page lists what is in the folder when it is opened. `folder_id` sits beside `job_id`, exactly one of
the two set; a folder's links go with the folder.
The downgrade has nowhere to keep a folder link, so it deletes them: they stop opening.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0108_folder_links"
down_revision: str | None = "0107_link_token_at_rest"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"download_links",
sa.Column("folder_id", sa.Integer(), sa.ForeignKey("file_folders.id", ondelete="CASCADE")),
)
op.create_index("ix_download_links_folder_id", "download_links", ["folder_id"])
op.alter_column("download_links", "job_id", nullable=True)
op.create_check_constraint(
"ck_download_link_target", "download_links", "(job_id IS NULL) <> (folder_id IS NULL)"
)
def downgrade() -> None:
op.execute("DELETE FROM download_links WHERE folder_id IS NOT NULL")
op.drop_constraint("ck_download_link_target", "download_links", type_="check")
op.alter_column("download_links", "job_id", nullable=False)
op.drop_index("ix_download_links_folder_id", table_name="download_links")
op.drop_column("download_links", "folder_id")
+160
View File
@@ -0,0 +1,160 @@
"""Background runs get a durable history
Revision ID: 0109_job_runs
Revises: 0108_folder_links
Create Date: 2026-10-17
What a scheduler job last did lived in the API process's memory, gone at every restart; the audit
log kept a row for a run only when it was manual, failed, or changed something. Now every run is a
`job_runs` row, written when it starts and when it ends. A run's quota spend points at it
(`quota_events.job_run_id`), and so do the lines it logged (`log_events.run_id`, no foreign key: the
log writer runs behind the event).
Runs are not admin actions, so the audit log stops carrying them: the existing `job_run` audit rows
MOVE into `job_runs` (start = end = when the row was written, the only time it kept), and leave the
audit log. The downgrade moves every run back into the audit log as such a row (one still running as
interrupted, what the audit row has no column for in its `after`), then drops the table.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0109_job_runs"
down_revision: str | None = "0108_folder_links"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"job_runs",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("job", sa.String(40), nullable=False),
sa.Column("trigger", sa.String(16), nullable=False),
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="SET NULL")),
sa.Column("target", sa.String(128)),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True)),
sa.Column("summary", sa.Text()),
sa.Column("result", postgresql.JSONB()),
sa.Column("error", sa.Text()),
sa.Column("failure", postgresql.JSONB()),
sa.Column("quota_units", sa.Integer(), nullable=False, server_default="0"),
sa.CheckConstraint(
"status IN ('running', 'ok', 'error', 'skipped', 'interrupted')",
name="ck_job_runs_status",
),
sa.CheckConstraint(
"trigger IN ('scheduled', 'manual', 'ad_hoc')", name="ck_job_runs_trigger"
),
sa.CheckConstraint(
"(status = 'running') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
op.create_index("ix_job_runs_job_started", "job_runs", ["job", "started_at"])
op.create_index("ix_job_runs_actor_id", "job_runs", ["actor_id"])
op.create_index("ix_job_runs_started_at", "job_runs", ["started_at"])
op.add_column(
"quota_events",
sa.Column("job_run_id", sa.BigInteger(), sa.ForeignKey("job_runs.id", ondelete="SET NULL")),
)
op.create_index("ix_quota_events_job_run_id", "quota_events", ["job_run_id"])
op.add_column("log_events", sa.Column("run_id", sa.BigInteger()))
op.create_index(
"ix_log_events_run",
"log_events",
["run_id"],
postgresql_where=sa.text("run_id IS NOT NULL"),
)
# The audit trail's runs move over: the three statuses the scheduler wrote, and `interrupted`,
# which a downgrade writes back (as it does a run's trigger, target, spend and start, which the
# audit row has no column for: its actor alone cannot say "manual" once that admin is deleted),
# so a rollback and a redeploy keep every run as it was. A status the run table does
# not know would fail the CHECK.
op.execute(
"""
INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at, finished_at,
summary, result, error, failure, quota_units)
SELECT target_id,
coalesce(after::jsonb ->> 'trigger',
CASE WHEN actor_id IS NULL THEN 'scheduled' ELSE 'manual' END),
actor_id,
after::jsonb ->> 'target',
after::jsonb ->> 'status',
coalesce((after::jsonb ->> 'started_at')::timestamptz, created_at),
created_at,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb -> 'result' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb -> 'result' END,
coalesce((after::jsonb ->> 'quota_units')::int, 0)
FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
ORDER BY id
"""
)
op.execute(
"""
DELETE FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
"""
)
def downgrade() -> None:
# A run still `running` is one the rollback stopped: it goes back as interrupted. What the audit
# row's own columns cannot say rides in `after` (its trigger, target, spend and start), so the
# upgrade restores them.
op.execute(
"""
INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after, created_at)
SELECT actor_id,
'job_run',
'job',
job,
left(job || ': ' || st
|| coalesce(' — ' || CASE WHEN st = 'error' THEN error ELSE summary END, ''),
255),
(jsonb_build_object(
'status', st,
'summary', CASE WHEN st = 'error' THEN error ELSE summary END)
|| CASE
WHEN st = 'error' AND failure IS NOT NULL
THEN jsonb_build_object('result', failure)
WHEN st <> 'error' AND result IS NOT NULL
THEN jsonb_build_object('result', result)
ELSE '{}'::jsonb
END
|| jsonb_build_object('trigger', trigger, 'quota_units', units,
'started_at', started_at)
|| CASE WHEN target IS NOT NULL
THEN jsonb_build_object('target', target) ELSE '{}'::jsonb END)::json,
coalesce(finished_at, started_at)
FROM (SELECT *, CASE WHEN status = 'running' THEN 'interrupted' ELSE status END AS st,
-- A run stopped mid-way has not summed its spend yet: sum it as the startup
-- sweep would, before the link to it is dropped.
CASE WHEN status = 'running'
THEN (SELECT coalesce(sum(q.units), 0) FROM quota_events q
WHERE q.job_run_id = job_runs.id)
ELSE quota_units END AS units
FROM job_runs) AS r
ORDER BY id
"""
)
op.drop_index("ix_log_events_run", table_name="log_events")
op.drop_column("log_events", "run_id")
op.drop_index("ix_quota_events_job_run_id", table_name="quota_events")
op.drop_column("quota_events", "job_run_id")
op.drop_index("ix_job_runs_started_at", table_name="job_runs")
op.drop_index("ix_job_runs_actor_id", table_name="job_runs")
op.drop_index("ix_job_runs_job_started", table_name="job_runs")
op.drop_table("job_runs")
@@ -0,0 +1,79 @@
"""Job intervals move into sysconfig; the unread SponsorBlock default goes
Revision ID: 0110_intervals_in_sysconfig
Revises: 0109_job_runs
Create Date: 2026-10-18
An admin's interval override lived in its own table, `scheduler_settings`, beside the one store every
other admin knob uses. Each override becomes the `job_interval_<job>` sysconfig key and the table
goes. Only the jobs that still exist are copied: a row for any other id never did anything.
The `sponsorblock_default` setting was offered on the Configuration page but nothing ever read it, so
its key is gone and its stored value with it — the downgrade cannot bring that value back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0110_intervals_in_sysconfig"
down_revision: str | None = "0109_job_runs"
branch_labels: str | None = None
depends_on: str | None = None
# The interval jobs as of this revision (a migration does not read the live code's list).
_JOBS = (
"rss_poll",
"enrich",
"backfill",
"autotag",
"shorts",
"subscriptions",
"playlist_sync",
"maintenance",
"explore_cleanup",
"download_gc",
"plex_sync",
"plex_watch_sync",
"plex_watch_reconcile",
"tmdb_enrich",
"people_enrich",
"audit_gc",
"log_gc",
)
_PREFIX = "job_interval_"
def upgrade() -> None:
op.execute(
sa.text(
"INSERT INTO system_config (key, value, encrypted)"
" SELECT :prefix || job_id, interval_minutes::text, false FROM scheduler_settings"
" WHERE interval_minutes IS NOT NULL AND job_id IN :jobs"
" ON CONFLICT (key) DO NOTHING"
).bindparams(sa.bindparam("jobs", expanding=True), prefix=_PREFIX, jobs=list(_JOBS))
)
op.drop_table("scheduler_settings")
op.execute("DELETE FROM system_config WHERE key = 'sponsorblock_default'")
def downgrade() -> None:
op.create_table(
"scheduler_settings",
sa.Column("job_id", sa.String(40), primary_key=True),
sa.Column("interval_minutes", sa.Integer(), nullable=False),
)
op.execute(
sa.text(
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
" SELECT substr(key, :start), value::int FROM system_config WHERE key IN :keys"
).bindparams(
sa.bindparam("keys", expanding=True),
start=len(_PREFIX) + 1,
keys=[_PREFIX + j for j in _JOBS],
)
)
op.execute(
sa.text("DELETE FROM system_config WHERE key IN :keys").bindparams(
sa.bindparam("keys", expanding=True), keys=[_PREFIX + j for j in _JOBS]
)
)
@@ -0,0 +1,28 @@
"""An audit row keeps the request that made it
Revision ID: 0111_audit_request_id
Revises: 0110_intervals_in_sysconfig
Create Date: 2026-10-18
The activity page opens an admin action's log lines from its row: `audit_log.request_id` is the
request the change was made in, the same id `log_events.request_id` carries. Older rows have none —
the request that made them was never recorded.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0111_audit_request_id"
down_revision: str | None = "0110_intervals_in_sysconfig"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("audit_log", sa.Column("request_id", sa.String(64)))
op.create_index("ix_audit_log_request_id", "audit_log", ["request_id"])
def downgrade() -> None:
op.drop_index("ix_audit_log_request_id", table_name="audit_log")
op.drop_column("audit_log", "request_id")
+8 -5
View File
@@ -20,7 +20,9 @@ Three layers, by *who* runs the code — not by feature:
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline;
`issues/bundle.py` is the one definition of a bug report's files, for the zip export and the tracker.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth` (with `apitokens`, a machine's
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`, `jobruns` (every
background run's durable row, its words in `runstatus`),
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
@@ -35,7 +37,7 @@ pre-abstract for a single caller; don't leave shared logic duplicated once there
## Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
(only `quota` and `jobruns` own their session)"** — so read it that way, not as an exception to it. The commit belongs
to the **operation owner**; anything called *inside* another transaction only stages its change
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
@@ -44,7 +46,7 @@ mutation it records, so an audit entry can never exist without its change (or vi
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
`jobruns`' run rows — the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
attribute — see `service.decr_ref_count`.
@@ -66,8 +68,9 @@ mutation it records, so an audit entry can never exist without its change (or vi
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
startup, not user-tunable at runtime.
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
slated to fold into `sysconfig` — do not add more like it.)
goes in `SPECS`; don't invent a fourth store. Each spec states when a change takes effect
(`applies`, required): `live` keys are read where they are used, never at import (test-held). The
scheduler's job intervals are `SPECS` keys too, edited on the Scheduler page, not the Configuration page.
## The gate
+273
View File
@@ -0,0 +1,273 @@
"""Machine access to the Files library: API tokens, and what each one may do where.
A user makes a token for a machine of theirs — their hub, a script — and it reaches their library as
`Authorization: Bearer sfl_…`. Only on `/api/files`: the routes there take `files_caller` instead of
`current_user`, and nothing else does (a test holds the route table to that), so a token can never
manage the account, the tokens, or anything outside the library. And only in the folders its grants
name, with the rights they give: list, read, write, delete. Every route asks the one question,
`FilesCaller.may(folder_id, right)`; a signed-in user may everything in their own folders.
The token is `sfl_` + 32 random bytes in base62 + a CRC32 of them: a mistyped or cut-off token is
refused without a database read, and a secret scanner can tell one by its shape. Only its SHA-256 is
kept (`security.hash_token`) — the user sees it once, when it is made — with its first characters to
tell tokens apart by. It is looked up by that hash on every request: no signed claims, so a deleted
token stops working at once. A token about to run out warns its user once, a week before
(`warn_expiring`), so a machine does not lose its key unannounced.
"""
import secrets
import zlib
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from fastapi import Depends, Request
from sqlalchemy import delete, or_, select, update
from sqlalchemy.orm import Session
from app import audit, debugsession, logs
from app.audit import AuditAction
from app.auth import current_user
from app.db import get_db
from app.models import ApiToken, ApiTokenGrant, User
from app.notifications import create_notification
from app.ratelimit import RateLimiter, client_key
from app.refusal import refused
from app.security import hash_token
from app.utils import iso
log = logs.get_logger(__name__)
PREFIX = "sfl_"
_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
_BODY_BYTES = 32
_BODY_LEN = 43 # 62**43 > 2**256: every 32-byte value fits
_CHECK_LEN = 6 # 62**6 > 2**32: every CRC32 fits
TOKEN_LEN = len(PREFIX) + _BODY_LEN + _CHECK_LEN
# What a token is shown by once it is made: enough to tell a user's tokens apart, far too little to
# narrow a guess (8 of its 43 random characters).
SHOWN_LEN = len(PREFIX) + 8
# A machine that runs away (a loop in a script) is held back per token; a caller trying tokens is
# held back per client, counting only the refusals.
_calls = RateLimiter(600, 60)
_bad = RateLimiter(20, 300)
# How stale `last_used_at` may get before a request writes it again: a busy script costs one write a
# minute, not one per request.
_TOUCH_EVERY = timedelta(minutes=1)
# How long before a token runs out its user is told (`warn_expiring`); the token list marks it
# "expires soon" for the same span.
EXPIRY_WARNING = timedelta(days=7)
class Right(StrEnum):
LIST = "list"
READ = "read"
WRITE = "write"
DELETE = "delete"
# Each right's column on a grant. A dict over the enum, so a new right without a column fails here.
GRANT_COLUMNS = {
Right.LIST: "can_list",
Right.READ: "can_read",
Right.WRITE: "can_write",
Right.DELETE: "can_delete",
}
def _b62(value: int, width: int) -> str:
out = []
for _ in range(width):
value, digit = divmod(value, 62)
out.append(_ALPHABET[digit])
return "".join(reversed(out))
def _check(body: str) -> str:
return _b62(zlib.crc32(body.encode()), _CHECK_LEN)
def mint() -> str:
"""A new token, in full. The caller stores its hash and shows it to the user, once."""
body = _b62(int.from_bytes(secrets.token_bytes(_BODY_BYTES)), _BODY_LEN)
return PREFIX + body + _check(body)
def well_formed(raw: str) -> bool:
"""Whether `raw` has a token's shape and its checksum adds up."""
if len(raw) != TOKEN_LEN or not raw.startswith(PREFIX):
return False
body, check = raw[len(PREFIX) : -_CHECK_LEN], raw[-_CHECK_LEN:]
# The check too: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it.
if any(c not in _ALPHABET for c in body + check):
return False
return secrets.compare_digest(_check(body), check)
def grant_rights(grant: ApiTokenGrant) -> frozenset[Right]:
return frozenset(r for r, col in GRANT_COLUMNS.items() if getattr(grant, col))
@dataclass(frozen=True)
class FilesCaller:
"""Who is asking the Files library, and where they may do what. `grants` is None for a signed-in
user (everything in their own folders); for a token, folder id → its rights there."""
user: User
token_id: int | None = None
token_prefix: str | None = None
grants: Mapping[int, frozenset[Right]] | None = None
def may(self, folder_id: int | None, right: Right) -> bool:
if self.grants is None:
return True
return folder_id is not None and right in self.grants.get(folder_id, frozenset())
def folders_with(self, right: Right) -> set[int] | None:
"""The folders `right` is granted in; None = any of the user's own."""
if self.grants is None:
return None
return {f for f, rights in self.grants.items() if right in rights}
@property
def via(self) -> str | None:
"""How an audit row says a token did it, None when the user did."""
return None if self.token_prefix is None else f"api token {self.token_prefix}…"
def bearer_token(request: Request) -> str | None:
"""The token of an `Authorization: Bearer` header; None when the request carries none (another
scheme counts as none — it is not ours to judge)."""
scheme, _, value = request.headers.get("authorization", "").partition(" ")
if scheme.lower() != "bearer":
return None
return value.strip()
def session_caller(request: Request, db: Session = Depends(get_db)) -> User | None:
"""The signed-in user, or None when the request brings a token instead: a token wins over a
cookie, so a machine's request is judged by its grants alone, whatever else it carries."""
if bearer_token(request) is not None:
return None
return current_user(request, db)
def files_caller(
request: Request,
db: Session = Depends(get_db),
user: User | None = Depends(session_caller),
) -> FilesCaller:
"""The `/api/files` caller: a signed-in user, or a valid token's user with its grants."""
if user is not None:
return FilesCaller(user=user)
return _token_caller(request, db, bearer_token(request) or "")
def _refuse_token(key: str, reason: str) -> Exception:
_bad.allow(key)
log.info(
"a request brought an API token that does not hold",
extra=logs.event("apitoken.refused", reason=reason),
)
return refused(401, "not_authenticated")
def _token_caller(request: Request, db: Session, raw: str) -> FilesCaller:
key = client_key(request)
if _bad.exhausted(key):
raise refused(429, "api_token_rate_limited")
if not well_formed(raw):
raise _refuse_token(key, "malformed")
token = db.execute(
select(ApiToken).where(ApiToken.token_hash == hash_token(raw))
).scalar_one_or_none()
if token is None:
raise _refuse_token(key, "unknown")
now = datetime.now(timezone.utc)
if token.expires_at is not None and token.expires_at <= now:
raise _refuse_token(key, "expired")
user = db.get(User, token.user_id)
if user is None or not user.is_active or user.is_suspended:
raise _refuse_token(key, "account")
if not _calls.allow(str(token.id)):
raise refused(429, "api_token_rate_limited")
grants = {
g.folder_id: grant_rights(g)
for g in db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == token.id))
.scalars()
.all()
}
caller = FilesCaller(user=user, token_id=token.id, token_prefix=token.prefix, grants=grants)
db.execute(
update(ApiToken)
.where(
ApiToken.id == token.id,
or_(ApiToken.last_used_at.is_(None), ApiToken.last_used_at < now - _TOUCH_EVERY),
)
.values(last_used_at=now)
.execution_options(synchronize_session=False)
)
# Committed here, ahead of the route: a read-only route never commits, and a write route must
# not carry this along into a rollback of its own.
db.commit()
logs.set_user(user.id, *debugsession.level_for(user, None))
request.state.user_id = user.id
return caller
def revoke_all(db: Session, user_id: int) -> int:
"""Revoke every token of `user_id`, each with its audit row (NOT committed). A token is a
credential like a session, so whatever ends every session of an account — a password reset or
change, "log out everywhere", a revoked untrusted password — ends its tokens too: one minted
from a stolen session must not outlive the owner's answer to that theft. Returns how many."""
gone = (
db.execute(delete(ApiToken).where(ApiToken.user_id == user_id).returning(ApiToken.id))
.scalars()
.all()
)
for token_id in gone:
audit.record(
db,
user_id,
AuditAction.API_TOKEN_REVOKE,
target_type="api_token",
target_id=token_id,
summary="revoked with every session of the account",
)
return len(gone)
def expiring(token: ApiToken, now: datetime) -> bool:
"""Whether `token` runs out within the warning span (and has not yet)."""
return token.expires_at is not None and now < token.expires_at <= now + EXPIRY_WARNING
def warn_expiring(db: Session) -> int:
"""Tell each user whose token runs out within `EXPIRY_WARNING` — once per token: the mark and the
notice commit together, and a token made again (a new row) warns afresh. Returns how many."""
now = datetime.now(timezone.utc)
hits = db.execute(
update(ApiToken)
.where(
ApiToken.expiry_warned.is_(False),
ApiToken.expires_at.is_not(None),
ApiToken.expires_at > now,
ApiToken.expires_at <= now + EXPIRY_WARNING,
)
.values(expiry_warned=True)
.returning(ApiToken.id, ApiToken.user_id, ApiToken.name, ApiToken.expires_at)
.execution_options(synchronize_session=False)
).all()
for token_id, user_id, name, expires_at in hits:
create_notification(
db,
user_id,
"api_token_expiring",
title=f"API token {name} expires soon", # English fallback; the client names it from data
data={"token_id": token_id, "name": name, "expires_at": iso(expires_at)},
commit=False,
)
db.commit()
return len(hits)
+17 -7
View File
@@ -1,9 +1,9 @@
"""Admin/scheduler audit trail: an append-only record of who changed what and when.
"""Admin audit trail: an append-only record of who changed what and when.
Design (locked): log at ACTION / job-run granularity, NEVER per-row — a scheduler run that
touches thousands of videos is ONE audit row summarising the outcome. Automatic job runs are
only logged when they actually changed something (or errored); all admin actions and all manual
runs are always logged. Secrets are never stored (config changes log the key name only).
Design (locked): log at ACTION granularity, NEVER per-row — an action that touches thousands of
rows is ONE audit row summarising the outcome. Background runs are not actions: they have their
own history (app/jobruns.py). Secrets are never stored (config changes log the key name only).
Each row keeps the request that made it, so the request's log lines are one click away.
`record()` only db.add()s the row — the caller commits it in the SAME transaction as the
mutation it describes, so the trail can't drift from reality (and a rolled-back mutation drops
@@ -15,6 +15,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from sqlalchemy.orm import Session
from app import logs
from app.models import AuditLog
@@ -48,8 +49,6 @@ class AuditAction:
MAINTENANCE_TUNE = "maintenance_tune"
SYNC_PAUSE = "sync_pause"
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
@@ -69,6 +68,16 @@ class AuditAction:
FILE_FOLDER_RENAME = "file_folder_rename"
FILE_FOLDER_RETENTION = "file_folder_retention"
FILE_FOLDER_DELETE = "file_folder_delete"
# A machine's token to the library (`routes.tokens`): made (its folders and rights) or revoked.
# What a token then did is the file rows above, their summary naming it.
API_TOKEN_CREATE = "api_token_create"
API_TOKEN_REVOKE = "api_token_revoke"
# A public `/watch` link to a file or a folder (`routes.downloads`): made, its settings changed,
# revoked.
# Never its token or password — only whether it has one.
DOWNLOAD_LINK_CREATE = "download_link_create"
DOWNLOAD_LINK_UPDATE = "download_link_update"
DOWNLOAD_LINK_REVOKE = "download_link_revoke"
def record(
@@ -94,6 +103,7 @@ def record(
summary=(summary or None) and summary[:255],
before=before,
after=after,
request_id=logs.current_request_id(),
)
)
+4 -2
View File
@@ -268,10 +268,12 @@ class Settings(BaseSettings):
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
quota_retention_days: int = 365
# Background-run history (job_runs): days kept (0 = forever). The audit log's 180, which is how
# long a run lived there before it moved here: ~1.8k rows a day at the default intervals
# (measured 2026-10-17), ~320k at the limit. The same daily cleanup job prunes it.
job_run_retention_days: int = 180
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: DownloadLayout = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
sponsorblock_default: bool = False
# --- Diagnostics / logging ---
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
+33 -7
View File
@@ -1,20 +1,23 @@
"""The folders of the Files library — where each of a user's files sits.
A folder is a row, never a directory: on disk a file stays where the download layout put it (one
shared asset can be held by several users), so a rename or a move is a column update. Two kinds:
shared asset can be held by several users), so a rename or a move is a column update. Three kinds:
- a SOURCE folder is made automatically, one per origin a user has files from (YouTube, Plex clips,
uploads, one per other site). Its `source_key` names the origin; the client renders its name
from the key, in the reader's language.
- an OWN folder is the user's, with the name they gave it.
- the HUB folder (one per user, made when they first give a machine a token for it, `hub_folder`) is
where they swap files with that machine. Its name, like a source folder's, is the client's; a file
may be moved in like into an own folder, and it keeps a file until it is deleted.
Every file of a user is in exactly one folder. The folder a new file lands in is decided here
(`source_key_for` + `source_folder`), at every place a job is created, so the rule lives once. A file
moves only into one of the user's own folders or back to its own source folder (`move`): a YouTube
file in the TikTok folder would be a lie the folder's name tells.
moves only into one of the user's own folders, their Hub folder, or back to its own source folder
(`move`): a YouTube file in the TikTok folder would be a lie the folder's name tells.
Retention is the folder's: a source folder keeps a file `download_retention_days` (the value when the
folder was made; the user can change it), Uploads and own folders keep it until it is deleted. A
folder was made; the user can change it), Uploads, own folders and the Hub keep it until it is deleted. A
done file carries its own `expires_at`, stamped by `done_expiry` when it finishes — unless the user
kept it — and its `ready_at`, the moment it finished (`stamp_done`).
"""
@@ -35,6 +38,7 @@ from app.models import DownloadJob, FileFolder, MediaAsset
class FolderKind(StrEnum):
SOURCE = "source"
OWN = "own"
HUB = "hub"
class SourceKey(StrEnum):
@@ -123,6 +127,26 @@ def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
return folder
def hub_folder(db: Session, user_id: int) -> FileFolder:
"""The user's Hub folder, made on first use; a race on its one-per-user index is settled like
`source_folder`'s."""
stmt = select(FileFolder).where(
FileFolder.user_id == user_id, FileFolder.kind == FolderKind.HUB
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
folder = FileFolder(user_id=user_id, kind=FolderKind.HUB)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: another request made the Hub folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def place(db: Session, job: DownloadJob) -> None:
"""Put a new job's file in its source folder, and record that origin on it. Called wherever a
job is created; the job must carry its `user_id`, `source_kind` and source refs already."""
@@ -178,8 +202,9 @@ def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True)
def upload_target(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The folder an upload asked to land in, if it may: one of the user's own folders, or their
Uploads folder. A file put in a site's folder would claim an origin it does not have."""
"""The folder an upload asked to land in, if it may: one of the user's own folders, their Hub
folder, or their Uploads folder. A file put in a site's folder would claim an origin it does not
have."""
folder = own_folder(db, user_id, folder_id, hold=hold)
if folder.kind == FolderKind.SOURCE and folder.source_key != SourceKey.UPLOAD:
raise FolderError("file_folder_not_its_source")
@@ -201,7 +226,8 @@ def set_kept(db: Session, job: DownloadJob, kept: bool) -> None:
def move(db: Session, job: DownloadJob, folder: FileFolder) -> None:
"""Put a file in `folder` — one of its user's own folders, or the source folder it came from.
"""Put a file in `folder` — one of its user's own folders, their Hub folder, or the source folder
it came from.
The caller has checked that `folder` is the job's user's."""
if folder.kind == FolderKind.SOURCE and folder.source_key != source_key_for(db, job):
raise FolderError("file_folder_not_its_source")
+8 -2
View File
@@ -1,6 +1,6 @@
"""Download retention garbage collector (a scheduler job).
Five passes each run:
Six passes each run:
1. Pre-expiry warning — a done file whose `expires_at` (its folder's retention, `folders`) falls
within the grace window, and that its user did not keep, warns its user once (`expiry_warned`).
2. Expiry — such a file past its `expires_at` leaves the library: the job is deleted and its hold
@@ -14,6 +14,8 @@ Five passes each run:
re-download renames an asset, e.g. old accented channel dirs, or an upload's stranded file).
It skips the dot-prefixed system trees (all but `.uploads`) and anything newer than an hour,
so an in-flight download is never touched.
6. Token expiry warning — a machine's token for the library about to run out warns its user once
(`apitokens.warn_expiring`): the library's other clock.
Runs in the API process (which mounts FILES_ROOT). Pure disk/DB work — no YouTube quota.
"""
@@ -25,7 +27,7 @@ from pathlib import Path
from sqlalchemy import func, or_, select, update
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app import apitokens, logs, sysconfig
from app.config import settings
from app.downloads import quota, service, storage
from app.downloads.status import JobStatus
@@ -366,12 +368,16 @@ def run_download_gc(db: Session) -> dict:
# posters). Runs last so the keep-set reflects the assets that survived passes 2-4.
swept = _sweep_orphan_files(db, settings.files_root)
# 6. A machine's token about to run out.
tokens_warned = apitokens.warn_expiring(db)
result = {
"warned": warned,
"expired": expired,
"evicted": evicted,
"reaped": reaped,
"swept": swept,
"tokens_warned": tokens_warned,
}
log.info("download_gc %s", result)
return result
+75 -11
View File
@@ -1,10 +1,15 @@
"""Public "share by link" helpers for the download center.
A DownloadLink is a capability URL: the unguessable `token` grants anyone read access to one
download's file on the login-free `/watch/{token}` page. Password-protected links additionally
require a short-lived signed **grant** on the file request — because a `<video src>` can't send a
header, the password is exchanged once at `/unlock` for an HMAC grant appended to the file URL,
so the raw password never rides in the media URL (or server logs).
A DownloadLink is a capability URL: the unguessable token grants anyone read access to one
download's file, or to the files in one of its owner's own folders (`folder_files`), on the
login-free `/watch/{token}` page. So the token is never stored as written:
the link is found by its SHA-256, and the owner's copy is Fernet-encrypted — a database copy alone
opens nothing. Without that copy (no TOKEN_ENCRYPTION_KEY, or a key since changed) the link still
opens; only its address cannot be shown again, and the owner makes a new link to share.
Password-protected links additionally require a short-lived signed **grant** on the file request —
because a `<video src>` can't send a header, the password is exchanged once at `/unlock` for an HMAC
grant appended to the file URL, so the raw password never rides in the media URL (or server logs).
"""
import hashlib
@@ -12,16 +17,71 @@ import hmac
import secrets
from datetime import datetime, timezone
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import security
from app.config import settings
from app.models import DownloadLink
from app.downloads import storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_GRANT_TTL = 6 * 3600 # a watch session's grant is good for 6h, then the viewer re-unlocks
# The most files a folder link lists: as many as the owner's own Files list shows (`routes.files.
# MAX_LIST`; a test holds the two equal — this layer may not import that one).
FOLDER_MAX = 500
def new_token() -> str:
return secrets.token_urlsafe(24) # ~32 chars, 192 bits
def seal(link: DownloadLink, token: str) -> None:
"""Store `token` on `link` the only ways it is kept: its hash, and its encrypted copy."""
link.token_hash = security.hash_token(token)
link.token_enc = security.encrypt(token) if security.can_encrypt() else None
def find(db: Session, token: str) -> DownloadLink | None:
"""The link a `/watch` token names, by its hash."""
return db.execute(
select(DownloadLink).where(DownloadLink.token_hash == security.hash_token(token))
).scalar_one_or_none()
def recover(link: DownloadLink) -> str | None:
"""The link's token from its encrypted copy, or None when there is none this key can read."""
return security.decrypt_optional(link.token_enc)
def in_folder(folder: FileFolder) -> tuple:
"""Which jobs a link to `folder` reaches: the done files in it NOW, the folder owner's. A file
moved out stops opening through the link at its next request; one moved in starts."""
return (
DownloadJob.folder_id == folder.id,
DownloadJob.user_id == folder.user_id,
DownloadJob.status == JobStatus.DONE,
)
def folder_files(db: Session, folder: FileFolder) -> list[tuple[DownloadJob, MediaAsset]]:
"""What a link to `folder` shows — its page lists exactly these, and its link card counts them:
each file `in_folder` whose asset is ready and whose bytes are on disk (so the list never offers
a dead row), newest first, at most `FOLDER_MAX`."""
rows = db.execute(
select(DownloadJob, MediaAsset)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(*in_folder(folder), MediaAsset.status == "ready")
.order_by(DownloadJob.ready_at.desc().nulls_last(), DownloadJob.id.desc())
.limit(FOLDER_MAX)
).all()
return [
(job, asset)
for job, asset in rows
if storage.safe_abs_path(settings.files_root, asset.rel_path) is not None
]
def is_expired(link: DownloadLink) -> bool:
return link.expires_at is not None and link.expires_at <= datetime.now(timezone.utc)
@@ -55,18 +115,22 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
return False
if exp < datetime.now(timezone.utc).timestamp():
return False
return hmac.compare_digest(sig, _sign(f"{token}.{exp}.{password_version}"))
# As bytes: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it, and
# `sig` is whatever the URL carried.
return hmac.compare_digest(sig.encode(), _sign(f"{token}.{exp}.{password_version}").encode())
# --- serialization -------------------------------------------------------------------------
def owner_view(link: DownloadLink, base_url: str = "") -> dict:
"""What the file owner sees when managing their links."""
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
"""What the file owner sees when managing their links. `token` is the one just made (it is in
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
token = token or recover(link)
return {
"id": link.id,
"token": link.token,
"url": f"{base_url}/watch/{link.token}",
"token": token,
"url": f"{base_url}/watch/{token}" if token else None,
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
+31 -5
View File
@@ -11,13 +11,12 @@ import html
import re
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import links as linksmod
from app.downloads import storage
from app.models import DownloadJob, DownloadLink, MediaAsset
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
@@ -74,22 +73,25 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
def _watch_tags(token: str, db: Session) -> str | None:
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
return None
url = f"{settings.app_base}/watch/{token}"
folder = link.folder_id is not None
if link.password_hash:
# Password-gated: don't leak the title/thumbnail in the unfurl.
return "\n ".join(
[
"<title>Siftlode</title>",
_tag("og:title", "Password-protected video"),
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
_tag("og:description", "Shared via Siftlode"),
_tag("og:type", "video.other"),
_tag("og:type", "website" if folder else "video.other"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
]
)
if folder:
return _folder_tags(db, link, url)
job = db.get(DownloadJob, link.job_id)
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if job is None or asset is None or asset.status != "ready":
@@ -147,6 +149,30 @@ def _watch_tags(token: str, db: Session) -> str | None:
return "\n ".join(tags)
def _folder_tags(db: Session, link: DownloadLink, url: str) -> str | None:
"""A folder link's card: the folder's name and how many files it holds — never their names,
which the page shows only to whoever opens it."""
folder = db.get(FileFolder, link.folder_id)
if folder is None or not folder.name:
return None
count = len(linksmod.folder_files(db, folder)) # the page's own list, so the two agree
title = folder.name.strip()
about = f"{count} file{'' if count == 1 else 's'} · Shared via Siftlode"
return "\n ".join(
[
f"<title>{html.escape(title)}</title>",
_tag("og:title", title),
_tag("og:description", about),
_tag("og:type", "website"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
_tag("twitter:title", title, attr="name"),
_tag("twitter:description", about, attr="name"),
_tag("twitter:card", "summary", attr="name"),
]
)
def render_watch_html(token: str, page_html: Path, db: Session) -> str | None:
"""Return the watch page (watch.html) with the OG block replaced for this token, or None to
serve the generic page unchanged (invalid/expired/not-ready link, or a template without the
+5 -1
View File
@@ -294,9 +294,13 @@ def served_media_type(media_kind: str | None, container: str | None) -> str | No
def download_filename(display_name: str, container: str | None, path: Path) -> str:
"""The Content-Disposition filename for a served download: the cleaned display name carrying a single
correct extension — the asset container, else the file's own suffix — without doubling it when the
name already ends in that extension. Shared by the private download and the public watch endpoints."""
name already ends in that extension. A file with no extension at all is served without one —
never as `name.` (Windows drops a trailing dot, so the saved file would not be the one named).
Shared by the private download and the public watch endpoints."""
ext = container or path.suffix.lstrip(".")
base = display_filename(display_name)
if not ext:
return base
if base.lower().endswith(f".{ext.lower()}"):
base = base[: -(len(ext) + 1)]
return f"{base}.{ext}"
+11 -2
View File
@@ -57,12 +57,14 @@ def ingest_upload(
display_name: str | None = None,
sha256: str | None = None,
folder_id: int | None = None,
via: str | None = None,
) -> DownloadJob:
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return its job —
`done`, or `queued` for the worker's phone-safe conversion (see the module docstring). `src_path`
is MOVED into place on success (the caller stages it to a temp file first). `sha256` is the
hash of those bytes, when the caller took it on the way in; `folder_id` the folder it asked for
(`folders.upload_target`), else its Uploads folder.
(`folders.upload_target`), else its Uploads folder; `via` what the audit row names as the doer
when it was not the user themselves (an API token).
Raises folders.FolderError when that folder is not one an upload may land in (or is gone).
@@ -145,7 +147,14 @@ def ingest_upload(
folders.stamp_done(db, job)
service.incr_ref_count(db, asset.id)
db.flush()
audit.record(db, user_id, AuditAction.FILE_UPLOAD, target_type="file", target_id=job.id)
audit.record(
db,
user_id,
AuditAction.FILE_UPLOAD,
target_type="file",
target_id=job.id,
summary=via,
)
db.commit()
except Exception:
db.rollback()
+131
View File
@@ -0,0 +1,131 @@
"""The durable history of background runs — the `job_runs` table.
A run is written twice: a `running` row when it starts, its outcome when it ends. Each write is its
OWN short session, committed at once, for the reason `quota.record_usage` gives: the run's own work
may roll back (its `except: db.rollback()`), and the history of what happened must not go with it —
nor may this bookkeeping commit the job's half-done session.
Only the API process runs jobs (the worker never starts the scheduler), and one API process serves a
database (`.env.example`, the Scheduler section: the single-run guard is in its memory too), so at its
startup nothing can still be running: `interrupt_leftovers` closes what a stopped process left open.
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete, func, select, union_all, update
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import JobRun, QuotaEvent
from app.runstatus import RunStatus, RunTrigger
def start(job: str, trigger: RunTrigger, actor_id: int | None, target: str | None = None) -> int:
"""Open a run's row; returns its id."""
with SessionLocal() as s:
row = JobRun(
job=job,
trigger=trigger,
actor_id=actor_id,
target=target,
status=RunStatus.RUNNING,
started_at=datetime.now(timezone.utc),
)
s.add(row)
s.commit()
return row.id
def finish(
run_id: int,
status: RunStatus,
*,
summary: str | None = None,
result: dict | None = None,
error: str | None = None,
failure: dict | None = None,
) -> None:
"""Close a run with its outcome and what it spent. Conditional on the row still running: a row
the startup sweep already called interrupted is not rewritten by a thread that outlived it."""
with SessionLocal() as s:
units = s.scalar(
select(func.coalesce(func.sum(QuotaEvent.units), 0)).where(
QuotaEvent.job_run_id == run_id
)
)
s.execute(
update(JobRun)
.where(JobRun.id == run_id, JobRun.status == RunStatus.RUNNING)
.values(
status=status,
finished_at=datetime.now(timezone.utc),
summary=summary,
result=result,
error=error,
failure=failure,
quota_units=units or 0,
)
)
s.commit()
def interrupt_leftovers() -> int:
"""Every row a previous process left `running` → `interrupted`, with what it spent until then
(its quota rows outlive it, as `finish` sums them). Called once at startup, before the scheduler
can open a new run. Returns how many."""
spent = (
select(func.coalesce(func.sum(QuotaEvent.units), 0))
.where(QuotaEvent.job_run_id == JobRun.id)
.scalar_subquery()
)
with SessionLocal() as s:
res = s.execute(
update(JobRun)
.where(JobRun.status == RunStatus.RUNNING)
.values(
status=RunStatus.INTERRUPTED,
finished_at=datetime.now(timezone.utc),
quota_units=spent,
)
)
s.commit()
return res.rowcount if isinstance(res, CursorResult) else 0
def recent_runs(db: Session, jobs: list[str], per_job: int) -> dict[str, list[JobRun]]:
"""Each of `jobs`' last `per_job` runs, newest first, by job id — one query, each job's part a
short walk of `ix_job_runs_job_started`, so a poll costs the same at any history length (a window
over the whole table grew with every run kept)."""
if not jobs:
return {}
newest = union_all(
*(
select(JobRun.id)
.where(JobRun.job == job)
.order_by(JobRun.started_at.desc(), JobRun.id.desc())
.limit(per_job)
for job in jobs
)
).subquery()
rows = db.execute(
select(JobRun)
.where(JobRun.id.in_(select(newest.c.id)))
.order_by(JobRun.job, JobRun.started_at.desc(), JobRun.id.desc())
).scalars()
out: dict[str, list[JobRun]] = {}
for r in rows:
out.setdefault(r.job, []).append(r)
return out
def gc(db: Session, retention_days: int) -> int:
"""Delete finished runs older than `retention_days` (0 = keep forever). A run still going is
never reaped. The caller commits."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
res = db.execute(
delete(JobRun).where(JobRun.started_at < cutoff, JobRun.status != RunStatus.RUNNING)
)
return res.rowcount if isinstance(res, CursorResult) else 0
+15 -2
View File
@@ -10,8 +10,8 @@ the logs were both unjoinable and leaky. `configure_logging()` replaces all four
Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
ts · level · sev · source · logger · event · msg · attrs
plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
`job_id`) and the app version. That shape is what the admin log console filters on and what a Gitea
issue attachment will carry, so it is deliberately flat and stable.
`job_id`, `run_id`) and the app version. That shape is what the admin log console filters on and
what a Gitea issue attachment will carry, so it is deliberately flat and stable.
Call sites stay ordinary stdlib logging:
@@ -81,15 +81,28 @@ _debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
"log_debug_session", default=None
)
_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
# The background run (app/jobruns.py) this context is doing; quota's spend is tagged with it too.
_run_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_run_id", default=None)
_CONTEXT_VARS = {
"request_id": _request_id,
"user_id": _user_id,
"debug_session_id": _debug_session,
"job_id": _job_id,
"run_id": _run_id,
}
def current_request_id() -> str | None:
"""The HTTP request this context is serving, if any — the request middleware sets it."""
return _request_id.get()
def current_run_id() -> int | None:
"""The background run this context is doing, if any — `bind(run_id=…)` sets it."""
return _run_id.get()
class _RequestScope:
"""What the auth dependency learns about a request, for every line logged after it.
+1
View File
@@ -362,6 +362,7 @@ def serialize(row: LogEvent) -> dict:
"request_id": row.request_id,
"user_id": row.user_id,
"job_id": row.job_id,
"run_id": row.run_id,
"debug_session_id": row.debug_session_id,
"version": row.version,
}
+4
View File
@@ -52,12 +52,14 @@ from app.routes import ( # noqa: E402 — logging is configured first
plex as plex_routes,
public as public_routes,
quota,
runs as runs_routes,
saved_views,
scheduler as scheduler_routes,
search as search_routes,
setup as setup_routes,
sync,
tags,
tokens as tokens_routes,
version,
)
from app.scheduler import shutdown_scheduler, start_scheduler # noqa: E402 — logging is configured first
@@ -337,12 +339,14 @@ app.include_router(plex_routes.router)
app.include_router(downloads.router)
app.include_router(downloads.admin_router)
app.include_router(files_routes.router)
app.include_router(tokens_routes.router)
app.include_router(public_routes.router)
app.include_router(admin.router)
app.include_router(config_routes.router)
app.include_router(scheduler_routes.router)
app.include_router(audit_routes.router)
app.include_router(admin_logs_routes.router)
app.include_router(runs_routes.router)
app.include_router(quota.router)
app.include_router(version.router)
app.include_router(setup_routes.router)
+122 -22
View File
@@ -24,6 +24,7 @@ from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from app.db import Base
from app.runstatus import RunStatus, RunTrigger, sql_in
from app.utils import canonical_email
# Every user-supplied display string in this schema shares one width, and it lives HERE so the
@@ -563,14 +564,20 @@ class QuotaEvent(Base):
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), index=True
)
# The background run that spent it (app/jobruns.py); NULL = a request's own spend. SET NULL: a
# reaped run leaves the spend in the budget's history.
job_run_id: Mapped[int | None] = mapped_column(
ForeignKey("job_runs.id", ondelete="SET NULL"), index=True
)
class AuditLog(Base):
"""Append-only admin/scheduler audit trail: who did what, when, and (where it applies)
the before->after values. `actor_id` is the acting admin; NULL means the scheduler /
background did it (SET NULL on user delete keeps the trail). Logged at ACTION / job-run
granularity — never per-row — so a scheduler run that touches thousands of videos is ONE
row summarising the outcome. `before`/`after` are small JSON snapshots (secrets excluded)."""
"""Append-only admin audit trail: who did what, when, and (where it applies) the before->after
values. `actor_id` is the acting admin; NULL means the background did it (SET NULL on user
delete keeps the trail). Logged per ACTION, never per row; a background run is not an action
and has its own history (`JobRun`). `before`/`after` are small JSON snapshots (secrets
excluded). `request_id` is the request that made the change, so its log lines are one click
away (NULL for rows written outside a request, and for rows older than the column)."""
__tablename__ = "audit_log"
@@ -587,6 +594,7 @@ class AuditLog(Base):
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), index=True
)
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
class LogEvent(Base):
@@ -609,6 +617,8 @@ class LogEvent(Base):
"debug_session_id",
postgresql_where="debug_session_id IS NOT NULL",
),
# A run's lines are `WHERE run_id = …`; request lines carry none.
Index("ix_log_events_run", "run_id", postgresql_where="run_id IS NOT NULL"),
)
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
@@ -624,7 +634,11 @@ class LogEvent(Base):
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
user_id: Mapped[int | None] = mapped_column(Integer)
debug_session_id: Mapped[str | None] = mapped_column(String(64))
# A DOWNLOAD job's id (download_jobs) — not a background run, which is `run_id`.
job_id: Mapped[int | None] = mapped_column(Integer)
# The background run (job_runs) that logged it. No foreign key, as `user_id`: the writer runs
# behind the event, and a run reaped meanwhile must not fail the batch its lines travel in.
run_id: Mapped[int | None] = mapped_column(BigInteger)
version: Mapped[str | None] = mapped_column(String(64))
@@ -689,15 +703,44 @@ class AppState(Base):
setup_token_hash: Mapped[str | None] = mapped_column(String(64))
class SchedulerSetting(Base):
"""Admin override for a scheduler job's run interval (minutes). One row per job id;
absent = use the env/config default. DB-backed so it's editable from the admin UI at
runtime without a redeploy (env stays a fallback)."""
class JobRun(Base):
"""One run of a background job — scheduled, an admin's "run now", or a user's ad-hoc work on one
target — written when it starts and again when it ends (app/jobruns.py), each in its own
committed session, so the history survives the job's own rollback and a restart. A row left
`running` by a process that died is set `interrupted` at the next startup.
__tablename__ = "scheduler_settings"
`quota_units` is what the run spent of the YouTube budget: the `quota_events` rows carrying its
id, summed when it ends. `summary` is the English one-line fallback; `result` / `failure` are the
same outcome as data, which the client names in the reader's language."""
job_id: Mapped[str] = mapped_column(String(40), primary_key=True)
interval_minutes: Mapped[int] = mapped_column(Integer)
__tablename__ = "job_runs"
__table_args__ = (
Index("ix_job_runs_job_started", "job", "started_at"),
CheckConstraint(f"status IN ({sql_in(RunStatus)})", name="ck_job_runs_status"),
CheckConstraint(f"trigger IN ({sql_in(RunTrigger)})", name="ck_job_runs_trigger"),
# Only a run still going has no end.
CheckConstraint(
f"(status = '{RunStatus.RUNNING}') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
job: Mapped[str] = mapped_column(String(40))
trigger: Mapped[str] = mapped_column(String(16))
# Who started it: the admin of a manual run, the user of an ad-hoc one; NULL = the schedule.
actor_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="SET NULL"), index=True
)
# What an ad-hoc run worked on (a channel id); NULL for the recurring jobs.
target: Mapped[str | None] = mapped_column(String(128))
status: Mapped[str] = mapped_column(String(16))
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
summary: Mapped[str | None] = mapped_column(Text)
result: Mapped[dict | None] = mapped_column(JSONB)
error: Mapped[str | None] = mapped_column(Text)
failure: Mapped[dict | None] = mapped_column(JSONB)
quota_units: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class SystemConfig(Base):
@@ -919,7 +962,8 @@ class FileFolder(Base, TimestampMixin):
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
user gets a file from that origin; an OWN folder (`name` set) is one the user made. Every file —
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
@@ -933,9 +977,12 @@ class FileFolder(Base, TimestampMixin):
unique=True,
postgresql_where=text("name IS NOT NULL"),
),
# One Hub folder per user.
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)",
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
@@ -949,6 +996,47 @@ class FileFolder(Base, TimestampMixin):
retention_days: Mapped[int | None] = mapped_column(Integer)
class ApiToken(Base, TimestampMixin):
"""A user's token for a machine to reach their Files library with (`app.apitokens`): sent as
`Authorization: Bearer sfl_…`, honoured on `/api/files` only, and only in the folders its
grants name. Only the SHA-256 of the token is kept — it is shown once, when it is made — plus
its first characters (`prefix`), so the user can tell their tokens apart."""
__tablename__ = "api_tokens"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
name: Mapped[str] = mapped_column(String(80))
prefix: Mapped[str] = mapped_column(String(16))
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) # NULL = never
last_used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The one notice its user gets before it runs out (`apitokens.warn_expiring`).
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class ApiTokenGrant(Base):
"""What one token may do in one folder. A folder with no grant is out of the token's reach."""
__tablename__ = "api_token_grants"
__table_args__ = (
UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
CheckConstraint(
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
token_id: Mapped[int] = mapped_column(ForeignKey("api_tokens.id", ondelete="CASCADE"))
folder_id: Mapped[int] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
can_list: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_read: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_write: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
can_delete: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
def _ready_on_insert(context) -> datetime | None:
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
@@ -1084,21 +1172,33 @@ class DownloadQuota(Base, UpdatedAtMixin):
class DownloadLink(Base, TimestampMixin):
"""A public capability link to one download — the "share by link" surface.
"""A public capability link to one download, or to one of its owner's own folders — the "share
by link" surface.
Anyone holding the unguessable `token` can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style). Optional per-link controls: `expires_at`, `allow_download`
(stream-only vs downloadable), and a `password_hash` (argon2, like a user password). Revoke =
delete the row. Distinct from `DownloadShare`, which grants a *registered* user access via the
in-app "Shared with me" list."""
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
the folder at the time it is opened. The token itself is never stored as written: a link is
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
grants a *registered* user access via the in-app "Shared with me" list."""
__tablename__ = "download_links"
__table_args__ = (
# A link opens exactly one thing: a file or a folder.
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int] = mapped_column(
job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
token: Mapped[str] = mapped_column(String(64), unique=True)
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
password_hash: Mapped[str | None] = mapped_column(Text)
+11 -3
View File
@@ -16,7 +16,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.db import SessionLocal
from app.models import ApiQuotaUsage, QuotaEvent
@@ -204,6 +204,14 @@ def record_usage(units: int) -> None:
)
with SessionLocal() as qs:
qs.execute(stmt)
# Audit detail: who/what spent it (per-user attribution; NULL actor = system).
qs.add(QuotaEvent(user_id=_actor_id.get(), action=_action.get(), units=units))
# Audit detail: who/what spent it (per-user attribution; NULL actor = system), and the
# background run it belongs to, so the run's row can say what it cost.
qs.add(
QuotaEvent(
user_id=_actor_id.get(),
action=_action.get(),
units=units,
job_run_id=logs.current_run_id(),
)
)
qs.commit()
+17
View File
@@ -0,0 +1,17 @@
"""The account filter the activity page's lists share (runs, admin actions, server log, quota): one
filter bar drives them all, so the value it sends means the same on every tab.
The value is an account id, or `none` — the rows no account made (the background) — a value to filter
on, not the absence of a filter.
"""
from sqlalchemy import ColumnElement
NONE = "none"
# The `user` query parameter's shape on every route that takes it.
PATTERN = r"^(none|[0-9]{1,10})$"
def condition(column, value: str) -> ColumnElement[bool]:
"""`column` (an account id column) narrowed to `value`."""
return column.is_(None) if value == NONE else column == int(value)
+66
View File
@@ -0,0 +1,66 @@
"""Keyset paging over `(timestamp desc, id desc)`, shared by the admin history lists (server log,
audit log, background runs).
A timestamp alone is not unique (a browser log row's `ts` is clamped into its consent window; a run
and its siblings can start in the same microsecond), and an offset would skip or repeat rows while
writers add new ones at the top — so a page continues from the last row's `(timestamp, id)`.
"""
import base64
import binascii
import json
from datetime import datetime, timezone
from fastapi import HTTPException
from sqlalchemy import and_, or_, select
from sqlalchemy.orm import Session
from app.models import User
PAGE_MAX = 200
def utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the columns are `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def encode_cursor(ts: datetime, row_id: int) -> str:
return base64.urlsafe_b64encode(json.dumps([ts.isoformat(), row_id]).encode()).decode()
def decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
at = datetime.fromisoformat(ts)
return (at if at.tzinfo is not None else at.replace(tzinfo=timezone.utc)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def older_than(ts_col, id_col, cursor: str):
"""The rows after `cursor` in `(ts desc, id desc)` order. The `ts <=` conjunct is implied by the
OR; it is there so the planner can bound the timestamp index scan instead of filtering every
newer row first."""
ts, row_id = decode_cursor(cursor)
return and_(ts_col <= ts, or_(ts_col < ts, and_(ts_col == ts, id_col < row_id)))
def page(rows: list, limit: int, ts_of) -> tuple[list, str | None]:
"""`rows` was read with `limit + 1` (to know whether there is a next page without a count query):
the page itself, and the cursor that continues after it (None on the last page)."""
more = len(rows) > limit
rows = rows[:limit]
return rows, encode_cursor(ts_of(rows[-1]), rows[-1].id) if more else None
def accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the row
was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
+32 -71
View File
@@ -8,33 +8,29 @@ What a row holds was redacted when it was written (`logs.record_fields`, and the
browser's rows), so this serves the columns as they are; a value the redactor took out reads as
`[redacted]` / `[email]`, which is the point — the console shows that something was there.
Paging is a keyset over `(ts desc, id desc)`: a browser row's `ts` is clamped into its consent
window, so `ts` alone is not unique, and an offset would skip or repeat rows while the writer adds
new ones at the top. The live tail re-reads the first page rather than asking for "ids after N":
Paging is a keyset over `(ts desc, id desc)` (`_keyset`). The live tail re-reads the first page rather than asking for "ids after N":
several processes write this table, and a batch's ids are taken when it inserts, not when it
commits, so a slower batch can land below an id the tail already passed — and would never show.
"""
import base64
import binascii
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import Text, and_, cast, func, or_, select
from fastapi import APIRouter, Depends, Query
from sqlalchemy import Text, cast, func, or_, select
from sqlalchemy.orm import Session
from app import logsink
from app.auth import admin_user
from app.db import get_db
from app.models import LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.utils import like_escape
router = APIRouter(prefix="/api/admin/logs", tags=["admin"])
PAGE_MAX = 200
# A facet list is a chip row, not a report: past this many distinct events the rest are reachable
# through the free-text search, and the response stays small whatever a self-host logs.
FACET_EVENTS_MAX = 100
@@ -42,38 +38,9 @@ FACET_EVENTS_MAX = 100
Level = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
# The filters a facet can leave out of its own count (`LogFilters.conditions`).
Facet = Literal["levels", "sources", "user", "event"]
# `none` selects the rows that have no account / no event code (a scheduler line, an anonymous
# request) — a real value to filter on, not the absence of a filter.
NONE = "none"
def _utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the column is `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def _encode_cursor(row: LogEvent) -> str:
return base64.urlsafe_b64encode(json.dumps([row.ts.isoformat(), row.id]).encode()).decode()
def _decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
return _utc(datetime.fromisoformat(ts)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def _accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the
line was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
# `none` selects the rows that have no event code, as the account filter's `none` (app/routes/_account)
# selects the rows of no account — a real value to filter on, not the absence of a filter.
NONE = account.NONE
@dataclass(frozen=True)
@@ -86,6 +53,7 @@ class LogFilters:
event: str | None
request_id: str | None
job_id: int | None
run_id: int | None
q: str | None
since: datetime | None
until: datetime | None
@@ -100,11 +68,7 @@ class LogFilters:
if self.sources and skip != "sources":
out.append(LogEvent.source.in_(self.sources))
if self.user is not None and skip != "user":
out.append(
LogEvent.user_id.is_(None)
if self.user == NONE
else LogEvent.user_id == int(self.user)
)
out.append(account.condition(LogEvent.user_id, self.user))
if self.event is not None and skip != "event":
out.append(
LogEvent.event.is_(None) if self.event == NONE else LogEvent.event == self.event
@@ -113,10 +77,12 @@ class LogFilters:
out.append(LogEvent.request_id == self.request_id)
if self.job_id is not None:
out.append(LogEvent.job_id == self.job_id)
if self.run_id is not None:
out.append(LogEvent.run_id == self.run_id)
if self.since is not None:
out.append(LogEvent.ts >= _utc(self.since))
out.append(LogEvent.ts >= utc(self.since))
if self.until is not None:
out.append(LogEvent.ts < _utc(self.until))
out.append(LogEvent.ts < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
@@ -134,17 +100,20 @@ class LogFilters:
def log_filters(
levels: list[Level] = Query(default=[]),
sources: list[str] = Query(default=[], max_length=8),
user: str | None = Query(default=None, pattern=r"^(none|[0-9]{1,10})$"),
user: str | None = Query(default=None, pattern=account.PATTERN),
event: str | None = Query(
default=None, max_length=64, pattern=r"^(none|[a-z][a-z0-9_]*(\.[a-z0-9_]+)*)$"
),
request_id: str | None = Query(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$"),
job_id: int | None = Query(default=None, ge=1),
run_id: int | None = Query(default=None, ge=1),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> LogFilters:
return LogFilters(list(levels), list(sources), user, event, request_id, job_id, q, since, until)
return LogFilters(
list(levels), list(sources), user, event, request_id, job_id, run_id, q, since, until
)
@router.get("")
@@ -159,28 +128,20 @@ def list_logs(
continues from a previous page's `next_cursor`."""
query = select(LogEvent).where(*filters.conditions())
if before is not None:
ts, row_id = _decode_cursor(before)
# The `ts <=` conjunct is implied by the OR; it is there so the planner can bound the
# `ts` index scan instead of filtering every newer row first.
query = query.where(
and_(
LogEvent.ts <= ts,
or_(LogEvent.ts < ts, and_(LogEvent.ts == ts, LogEvent.id < row_id)),
)
)
# One more than asked, to know whether there is a next page without a second count query.
rows = (
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
query = query.where(older_than(LogEvent.ts, LogEvent.id, before))
rows, cursor = page(
list(
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
),
limit,
lambda r: r.ts,
)
more = len(rows) > limit
rows = rows[:limit]
return {
"items": [logsink.serialize(r) for r in rows],
"accounts": _accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": _encode_cursor(rows[-1]) if more else None,
"accounts": accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": cursor,
}
@@ -211,6 +172,6 @@ def log_facets(
"sources": counts(LogEvent.source, "sources"),
"events": counts(LogEvent.event, "event", FACET_EVENTS_MAX),
"users": [{**u, "value": str(u["value"])} for u in users],
"accounts": _accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"accounts": accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"total": db.scalar(total) or 0,
}
+106 -29
View File
@@ -1,67 +1,144 @@
"""Admin audit log API: read the append-only who/what/when trail, and clear it.
Admin-only. The client renders these rows in a DataTable (client-side sort/filter/paginate), so
this returns the most recent window of rows (capped) plus the true total, rather than paging
server-side — the log stays small in practice (state-changing runs + admin actions only)."""
Admin-only. The activity page reads it a page at a time, newest first, filtered on the server like
the server log (`_keyset` paging), so the whole trail is reachable however long it grows."""
from fastapi import APIRouter, Depends
from sqlalchemy import delete, func, select
from dataclasses import dataclass
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy import Text, cast, delete, func, or_, select
from sqlalchemy.orm import Session
from app import audit
from app.audit import AuditAction
from app.auth import admin_user
from app.db import get_db
from app.models import AuditLog, User
from app.utils import iso
from app.models import AuditLog, LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.utils import iso, like_escape
router = APIRouter(prefix="/api/admin/audit", tags=["admin"])
# Cap the client-side window. The audit log is small by design; if it ever outgrows this we'd
# switch the DataTable to server-side paging.
MAX_ROWS = 2000
def _serialize(row: AuditLog, actors: dict[int, str]) -> dict:
def _serialize(row: AuditLog, lines: dict[str, int]) -> dict:
return {
"id": row.id,
"created_at": iso(row.created_at),
# actor_id NULL = the scheduler/background did it (client shows "System"); a set id that
# can't be resolved would mean a deleted user, but SET NULL makes that NULL — so present.
"actor": actors.get(row.actor_id) if row.actor_id is not None else None,
# NULL = the background did it (the client shows "System"); the address is in `accounts`.
"actor_id": row.actor_id,
"action": row.action,
"target_type": row.target_type,
"target_id": row.target_id,
"summary": row.summary,
"before": row.before,
"after": row.after,
"request_id": row.request_id,
# How many of that request's log lines are still kept: they go after days, the row after
# months, and a link to none would open an empty list.
"log_lines": lines.get(row.request_id, 0) if row.request_id else 0,
}
@dataclass(frozen=True)
class AuditFilters:
action: str | None
actor: str | None
q: str | None
since: datetime | None
until: datetime | None
def conditions(self, skip: Literal["action"] | None = None) -> list:
"""The WHERE conditions, ANDed. `skip="action"` leaves the action filter out, so the action
menu counts every action under the other filters (the way back from a pick)."""
out = []
if self.action is not None and skip != "action":
out.append(AuditLog.action == self.action)
if self.actor is not None:
out.append(account.condition(AuditLog.actor_id, self.actor))
if self.since is not None:
out.append(AuditLog.created_at >= utc(self.since))
if self.until is not None:
out.append(AuditLog.created_at < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
or_(
AuditLog.summary.ilike(like),
AuditLog.target_id.ilike(like),
AuditLog.action.ilike(like),
AuditLog.request_id.ilike(like),
cast(AuditLog.before, Text).ilike(like),
cast(AuditLog.after, Text).ilike(like),
)
)
return out
def audit_filters(
action: str | None = Query(default=None, max_length=48, pattern=r"^[a-z][a-z0-9_]*$"),
user: str | None = Query(default=None, pattern=account.PATTERN),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> AuditFilters:
return AuditFilters(action, user, q, since, until)
@router.get("")
def list_audit(
limit: int = 500,
filters: AuditFilters = Depends(audit_filters),
before: str | None = Query(default=None, max_length=200),
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
limit = max(1, min(limit, MAX_ROWS))
total = db.scalar(select(func.count()).select_from(AuditLog)) or 0
rows = (
db.execute(select(AuditLog).order_by(AuditLog.created_at.desc()).limit(limit))
.scalars()
.all()
"""One page of rows, newest first, matching every filter given; `before` continues from a
previous page's `next_cursor`."""
query = select(AuditLog).where(*filters.conditions())
if before is not None:
query = query.where(older_than(AuditLog.created_at, AuditLog.id, before))
order = (AuditLog.created_at.desc(), AuditLog.id.desc())
rows, cursor = page(
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
limit,
lambda r: r.created_at,
)
actor_ids = {r.actor_id for r in rows if r.actor_id is not None}
actors: dict[int, str] = {}
if actor_ids:
actors = dict(db.execute(select(User.id, User.email).where(User.id.in_(actor_ids))).all())
requests = {r.request_id for r in rows if r.request_id}
lines: dict[str, int] = {}
if requests:
counted = db.execute(
select(LogEvent.request_id, func.count())
.where(LogEvent.request_id.in_(requests))
.group_by(LogEvent.request_id)
)
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
return {
"items": [_serialize(r, actors) for r in rows],
"total": total,
"returned": len(rows),
"items": [_serialize(r, lines) for r in rows],
"accounts": accounts(db, {r.actor_id for r in rows if r.actor_id is not None}),
"next_cursor": cursor,
}
@router.get("/facets")
def audit_facets(
filters: AuditFilters = Depends(audit_filters),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""The actions present under the other filters, with counts — what the action menu offers."""
n = func.count().label("n")
stmt = (
select(AuditLog.action, n)
.where(*filters.conditions("action"))
.group_by(AuditLog.action)
.order_by(n.desc(), AuditLog.action)
)
return {"actions": [{"value": a, "count": c} for a, c in db.execute(stmt).all()]}
@router.delete("")
def clear_audit(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
"""Clear the audit log. Leaves ONE row recording who cleared it + how many (tamper-evidence)."""
+28 -14
View File
@@ -16,6 +16,7 @@ from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import apitokens
from app import email as email_mod
from app import logs
from app import mailtext
@@ -446,7 +447,7 @@ async def callback(
# SEC-10: drop any password nobody proved the mailbox for, now that Google attests
# ownership (bumps the epoch to revoke any session it minted). Runs before the
# activation below, which is what it reads.
revoke_untrusted_preset_password(user)
revoke_untrusted_preset_password(db, user)
# Google verified the email and is_allowed granted access, so finish activating a
# previously pending password registration (otherwise current_user would bounce it).
user.email_verified = True
@@ -486,7 +487,7 @@ async def callback(
# real owner, signed in through Google right now, sets a new password in Settings.
if canonical_email(email) in env_admin_canonicals():
if user.role != "admin" and user.password_hash is not None:
_revoke_password(user)
_revoke_password(db, user)
user.role = "admin"
elif not user.role:
user.role = "user"
@@ -1118,9 +1119,8 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
# about. The person still confirms their address the normal way.
if kind == "reset":
user.email_verified = True
bump_session_epoch(
user
) # SA4: a reset (often a compromise response) kills all existing sessions.
# SA4: a reset (often a compromise response) kills all existing sessions — and API tokens.
bump_session_epoch(db, user)
# Burn any other outstanding reset tokens for this user.
for row in db.execute(
select(AuthToken).where(
@@ -1143,13 +1143,27 @@ def _remember_epoch(request: Request, user: User) -> None:
request.session["epochs"] = epochs
def bump_session_epoch(user: User) -> None:
def bump_session_epoch(db: Session, user: User) -> None:
"""Invalidate every EXISTING signed cookie for this account by advancing its epoch (SA4). Cookies
minted before the bump carry a now-stale epoch and are rejected by `current_user`."""
user.session_epoch = (user.session_epoch or 0) + 1
minted before the bump carry a now-stale epoch and are rejected by `current_user`. The account's
API tokens go with them (`apitokens.revoke_all`): a token minted from a stolen session must not
outlive the answer to that theft. Not committed — the caller commits it with its change.
The user's row is locked before the tokens go, the lock `create_token` takes too: a token made at
the same moment either commits first and is deleted here, or waits and sees the new epoch. FOR NO
KEY UPDATE (`key_share`): the two still exclude each other, but a row that only refers to the user
(a notice, an audit line) is not held up — a token's expiry notice would deadlock with it."""
if user.id is None:
user.session_epoch = (user.session_epoch or 0) + 1
return
epoch = db.execute(
select(User.session_epoch).where(User.id == user.id).with_for_update(key_share=True)
).scalar_one()
user.session_epoch = (epoch or 0) + 1
apitokens.revoke_all(db, user.id)
def revoke_untrusted_preset_password(user: User) -> None:
def revoke_untrusted_preset_password(db: Session, user: User) -> None:
"""SEC-10 (account pre-emption): a password set on an account whose email was NEVER verified is
untrusted — whoever registered the address (to pre-empt the real owner) could have chosen it. When
the real owner attests ownership through Google, drop that password and bump the epoch to revoke any
@@ -1183,15 +1197,15 @@ def revoke_untrusted_preset_password(user: User) -> None:
_send_reset_if_eligible keyed off exactly that, so a revoked account silently got no reset link and
was locked out for good (it could not sign in either). The stamp separates the two cases."""
if not user.email_verified or (not user.is_active and user.password_hash is not None):
_revoke_password(user)
_revoke_password(db, user)
def _revoke_password(user: User) -> None:
def _revoke_password(db: Session, user: User) -> None:
"""Drop the account's password, stamp why (so password-reset still accepts it) and end every
session it signed in."""
user.password_hash = None
user.password_revoked_at = datetime.now(timezone.utc)
bump_session_epoch(user)
bump_session_epoch(db, user)
@router.post("/logout-others")
@@ -1203,7 +1217,7 @@ def logout_others(
in somewhere" / a suspected session compromise."""
# Commit the bump BEFORE re-stamping this cookie, so a failed commit can't strand the current
# session at a newer epoch than the DB (see set_password).
bump_session_epoch(user)
bump_session_epoch(db, user)
db.commit()
_remember_epoch(request, user)
log.info("Logout-others: uid=%s (epoch=%s)", user.id, user.session_epoch)
@@ -1255,7 +1269,7 @@ def set_password(
# SA4: a password change logs out every OTHER session; re-stamp this cookie at the new epoch so
# the person making the change stays signed in here. Commit BEFORE re-stamping so a failed commit
# can't leave the cookie at a newer epoch than the DB (which would wrongly 401 THIS session).
bump_session_epoch(user)
bump_session_epoch(db, user)
db.commit()
_remember_epoch(request, user)
log.info("Password set/changed: uid=%s", user.id)
+16 -9
View File
@@ -3,15 +3,16 @@ hidden, tags) and per-channel sync state. Local reads/writes plus YouTube-side w
(subscribe / unsubscribe) that require the optional write scope."""
from datetime import datetime, timezone
from functools import partial
from fastapi import APIRouter, BackgroundTasks, Depends
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import Select, func, select, update
from sqlalchemy.orm import Session
from app.collation import collated, sorted_words
from app.mailtext import lang_for_user
from app import logs, quota
from app import logs, quota, scheduler
from app.auth import admin_user, current_user, require_write_scope
from app.db import get_db
from app.models import (
@@ -481,7 +482,6 @@ class ChannelUpdateIn(BaseModel):
def update_channel(
channel_id: str,
body: ChannelUpdateIn,
background: BackgroundTasks,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
@@ -508,12 +508,19 @@ def update_channel(
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_BACKFILL_RECENT):
run_recent_backfill(db, [channel], max_channels=1)
# …and kick the FULL deep backfill for this one channel immediately in the background, so the
# whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks. The
# request returns right away; the channel page's live poll shows the count climb. The scheduler
# still finishes a very large channel's tail on its normal runs (deep_requested stays set).
if deep_turned_on:
background.add_task(backfill_channel_now, channel_id, user.id)
# …and kick the FULL deep backfill for this one channel immediately, as a recorded run of its own,
# so the whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks.
# The request returns right away; the channel page's live poll shows the count climb. The
# scheduler still finishes a very large channel's tail on its normal runs (deep_requested stays set).
# A channel whose history is all in gets no run: it would only say "nothing to do", one kept row
# per flip of the flag.
if deep_turned_on and not (channel is not None and channel.backfill_done):
scheduler.run_ad_hoc(
scheduler.CHANNEL_BACKFILL,
partial(backfill_channel_now, channel_id, user.id),
actor_id=user.id,
target=channel_id,
)
return {
"ok": True,
+5 -3
View File
@@ -61,7 +61,7 @@ def set_config(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
s = sysconfig.spec(key)
s = sysconfig.config_page_spec(key)
if s is None:
raise not_found("config_key")
if "value" not in body.model_fields_set:
@@ -89,7 +89,9 @@ def set_config(
before=None if s.secret else {"value": _redact(old)},
after=None if s.secret else {"value": _redact(new)},
)
db.commit()
# The override is committed even when the trail skips it: re-saving the value in force still
# pins it as an override, and an uncommitted write would be dropped with the session.
db.commit()
_apply_runtime(db, key)
return sysconfig.describe(db)
@@ -100,7 +102,7 @@ def reset_config(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
s = sysconfig.spec(key)
s = sysconfig.config_page_spec(key)
if s is None:
raise not_found("config_key")
old = None if s.secret else sysconfig.get(db, key)
+129 -26
View File
@@ -27,13 +27,16 @@ from app.db import get_db
from app.downloads import clip as clipmod
from app.downloads import credentials as dlcreds
from app.downloads import edit as editmod
from app.downloads import folders
from app.downloads import links as linksmod
from app.downloads import quota, service, storage
from app.downloads import urlguard
from app.downloads.folders import FolderKind
from app.downloads.status import FINISHED, NOT_REFETCHABLE, JobStatus
from app.models import (
DownloadJob,
DownloadLink,
FileFolder,
DownloadProfile,
DownloadQuota,
DownloadShare,
@@ -62,6 +65,7 @@ from app.routes.files import (
)
from app.security import hash_password
from app.userscope import is_messageable_user, messageable_clauses
from app.utils import iso
from app.refusal import refused
router = APIRouter(prefix="/api/downloads", tags=["downloads"])
@@ -716,8 +720,12 @@ def _own_link(db: Session, user: User, link_id: int) -> DownloadLink:
link = db.get(DownloadLink, link_id)
if link is None:
raise not_found("link")
job = db.get(DownloadJob, link.job_id)
if job is None or job.user_id != user.id:
target = (
db.get(DownloadJob, link.job_id)
if link.job_id is not None
else db.get(FileFolder, link.folder_id)
)
if target is None or target.user_id != user.id:
raise not_found("link")
return link
@@ -730,21 +738,17 @@ def _link_expiry(days: int | None) -> datetime | None:
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
def _links_of(db: Session, *where) -> list[dict]:
rows = db.execute(select(DownloadLink).where(*where).order_by(DownloadLink.id.desc()))
return [linksmod.owner_view(row) for row in rows.scalars()]
@router.get("/{job_id}/links")
def list_links(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
owned_or_404(db, DownloadJob, job_id, user, what="download")
rows = (
db.execute(
select(DownloadLink)
.where(DownloadLink.job_id == job_id)
.order_by(DownloadLink.id.desc())
)
.scalars()
.all()
)
return [linksmod.owner_view(row) for row in rows]
return _links_of(db, DownloadLink.job_id == job_id)
class CreateLinkIn(BaseModel):
@@ -758,6 +762,34 @@ class CreateLinkIn(BaseModel):
expires_days: int | None = None
def _new_link(
db: Session, user: User, body: CreateLinkIn, **target: int
) -> tuple[DownloadLink, str]:
"""Make a link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and
the token it is opened by — which is in hand only now."""
pw = body.password.strip()
token = linksmod.new_token()
link = DownloadLink(
**target,
created_by=user.id,
allow_download=body.allow_download,
password_hash=hash_password(pw) if pw else None,
expires_at=_link_expiry(body.expires_days),
)
linksmod.seal(link, token)
db.add(link)
db.flush()
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_CREATE,
target_type="download_link",
target_id=link.id,
after={**target, **_link_audit(link)},
)
return link, token
@router.post("/{job_id}/links")
def create_link(
job_id: int,
@@ -765,22 +797,73 @@ def create_link(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
job = _shareable_job(db, user, job_id)
if job.status != JobStatus.DONE or job.asset_id is None:
raise refused(409, "share_unfinished")
pw = body.password.strip()
link = DownloadLink(
job_id=job_id,
token=linksmod.new_token(),
created_by=user.id,
allow_download=body.allow_download,
password_hash=hash_password(pw) if pw else None,
expires_at=_link_expiry(body.expires_days),
)
db.add(link)
link, token = _new_link(db, user, body, job_id=job_id)
db.commit()
db.refresh(link)
return linksmod.owner_view(link)
return linksmod.owner_view(link, token=token)
def _shareable_job(db: Session, user: User, job_id: int) -> DownloadJob:
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
rather than failing the link's foreign key — or, done first, leaves a 404."""
owned_or_404(db, DownloadJob, job_id, user, what="download")
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update(read=True)
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
raise not_found("download")
return job
def _shareable_folder(db: Session, user: User, folder_id: int) -> FileFolder:
"""One of the user's own folders. A source folder lets its files go on its own clock, and the Hub
is the user's channel to a machine of theirs: neither is a shelf to hand out. Held (FOR SHARE)
for the rest of the transaction, so a delete of it at the same moment waits and takes the new
link with it, rather than failing the link's foreign key."""
owned_or_404(db, FileFolder, folder_id, user, what="folder")
try:
folder = folders.own_folder(db, user.id, folder_id)
except folders.FolderError as e:
raise refused(409, e.reason) from e
if folder.kind != FolderKind.OWN:
raise refused(409, "share_folder_kind")
return folder
@router.get("/folders/{folder_id}/links")
def list_folder_links(
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
owned_or_404(db, FileFolder, folder_id, user, what="folder")
return _links_of(db, DownloadLink.folder_id == folder_id)
@router.post("/folders/{folder_id}/links")
def create_folder_link(
folder_id: int,
body: CreateLinkIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
_shareable_folder(db, user, folder_id)
link, token = _new_link(db, user, body, folder_id=folder_id)
db.commit()
return linksmod.owner_view(link, token=token)
def _link_audit(link: DownloadLink) -> dict:
"""A link's settings as the audit records them: whether it has a password, never which."""
return {
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": iso(link.expires_at),
}
class UpdateLinkIn(BaseModel):
@@ -797,6 +880,7 @@ def update_link(
db: Session = Depends(get_db),
) -> dict:
link = _own_link(db, user, link_id)
before = _link_audit(link)
sent = body.model_fields_set
if "allow_download" in sent:
link.allow_download = bool(body.allow_download)
@@ -808,6 +892,17 @@ def update_link(
# Bump the version so any grant issued under the previous password stops working now, rather
# than living out its 6h TTL (setting/clearing/changing the password all count as a rotation).
link.password_version += 1
after = _link_audit(link)
if after != before or "password" in sent:
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
target_type="download_link",
target_id=link_id,
before=before,
after=after,
)
db.commit()
return linksmod.owner_view(link)
@@ -818,8 +913,16 @@ def revoke_link(
) -> dict:
link = _own_link(db, user, link_id)
db.delete(link)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
target_type="download_link",
target_id=link_id,
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
)
db.commit()
return {"deleted": link_id}
return {"ok": True, "deleted": link_id}
# --- admin ---------------------------------------------------------------------------------
+111 -38
View File
@@ -7,6 +7,11 @@ bytes. Folders are rows (`downloads.folders`), so a rename or a move never touch
In-flight jobs are not here: they are the download tray's (`routes.downloads`). The helpers above
the routes are shared with that module — serialization, the ownership locks, the upload spool —
because a download and a library file are one row at two moments.
A machine with an API token (`app.apitokens`) may use the routes that take `files_caller`: list the
folders and files, fetch a file and its hash, upload, delete — each only in a folder its token was
granted that right in, and never the files shared with the user. A folder or file out of its reach
answers like a missing one. Everything else here is the signed-in user's alone (`current_user`).
"""
import base64
@@ -29,6 +34,7 @@ from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import audit, sysconfig
from app.apitokens import FilesCaller, Right, files_caller
from app.audit import AuditAction
from app.auth import current_user
from app.collation import collated
@@ -41,6 +47,7 @@ from app.mailtext import lang_for_user
from app.models import (
DISPLAY_TEXT_MAX,
DownloadJob,
DownloadLink,
DownloadShare,
FileFolder,
LibraryState,
@@ -208,7 +215,7 @@ UPLOAD_FOLDER_HEADER = "x-upload-folder"
@dataclass(frozen=True)
class SpooledUpload:
"""What the body dependency hands the handler: the bytes already on disk and their hash, plus
the names and the folder that came alongside them."""
the names and the folder that came alongside them, and who sent them."""
path: Path
size: int
@@ -217,6 +224,7 @@ class SpooledUpload:
filename: str
display_name: str | None
folder_id: int | None
via: str | None = None
def _header_text(request: Request, name: str) -> str:
@@ -233,15 +241,21 @@ def _header_text(request: Request, name: str) -> str:
return ""
def _upload_folder(db: Session, user_id: int, raw: str) -> int | None:
def _upload_folder(db: Session, caller: FilesCaller, raw: str) -> int | None:
"""The folder id an upload asked for (`UPLOAD_FOLDER_HEADER`), checked; None when it asked for
none."""
none — which a token may not: it writes only where it was granted to."""
if not raw.strip():
if caller.grants is not None:
raise refused(400, "api_token_folder_required")
return None
try:
return folders.upload_target(db, user_id, int(raw), hold=False).id
folder_id = int(raw)
except ValueError as e:
raise refused(409, "file_folder_gone") from e
if not caller.may(folder_id, Right.WRITE):
raise not_found("folder")
try:
return folders.upload_target(db, caller.user.id, folder_id, hold=False).id
except folders.FolderError as e:
raise refused(409, e.reason) from e
@@ -260,24 +274,26 @@ class _HashingSink:
async def spool_upload(
request: Request,
user: User = Depends(current_user),
caller: FilesCaller = Depends(files_caller),
db: Session = Depends(get_db),
) -> AsyncIterator[SpooledUpload]:
"""Stream the request body to a staging temp file, enforcing the per-file cap as it arrives.
`user` is taken but unused: it is the whole point. Dependencies resolve in declaration order, so
naming `current_user` here makes the auth check a precondition of reading the body at all — a
stranger is refused before a byte moves. The `yield` shape owns the temp file's life: ingest
MOVES it into place on success (the unlink is then a no-op), and on any other exit — cap hit,
empty, quota, probe or move error, a client that vanished mid-upload — it is removed."""
`caller` comes first on purpose. Dependencies resolve in declaration order, so naming
`files_caller` here makes the auth check a precondition of reading the body at all — a stranger,
or a token without the right to write in that folder, is refused before a byte moves. The
`yield` shape owns the temp file's life: ingest MOVES it into place on success (the unlink is
then a no-op), and on any other exit — cap hit, empty, quota, probe or move error, a client that
vanished mid-upload — it is removed."""
cap = sysconfig.get_int(db, "download_upload_max_bytes")
user_id = user.id
user_id = caller.user.id
via = caller.via
# A folder the file may not go in is refused before the transfer, not after it.
folder_id = _upload_folder(db, user_id, request.headers.get(UPLOAD_FOLDER_HEADER, ""))
folder_id = _upload_folder(db, caller, request.headers.get(UPLOAD_FOLDER_HEADER, ""))
# Hand the pooled connection back BEFORE the transfer starts — see `db.release_read_connection`
# for why, and for the precondition it enforces. Everything this dependency still needs is
# already an int or a header in hand: touching an ORM attribute below would reload `user` and
# check a connection straight back out.
# already an int, a string or a header in hand: touching an ORM attribute below (`caller.user`)
# would reload it and check a connection straight back out.
release_read_connection(db)
staging = Path(settings.files_root) / ".staging" / "uploads"
staging.mkdir(parents=True, exist_ok=True)
@@ -300,6 +316,7 @@ async def spool_upload(
filename=_header_text(request, UPLOAD_FILENAME_HEADER) or "upload",
display_name=display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER)),
folder_id=folder_id,
via=via,
)
finally:
tmp_path.unlink(missing_ok=True)
@@ -322,10 +339,21 @@ def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob:
return job
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob:
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
own in a folder it may read."""
if caller.grants is None:
return accessible_job(db, caller.user, job_id)
job = db.get(DownloadJob, job_id)
if job is None or job.user_id != caller.user.id or not caller.may(job.folder_id, Right.READ):
raise not_found("download")
return job
# --- folders -------------------------------------------------------------------------------
def _serialize_folder(folder: FileFolder, count: int, size: int) -> dict:
def _serialize_folder(folder: FileFolder, count: int, size: int, linked: bool = False) -> dict:
return {
"id": folder.id,
"kind": folder.kind,
@@ -336,27 +364,47 @@ def _serialize_folder(folder: FileFolder, count: int, size: int) -> dict:
"count": count,
"size_bytes": size,
"created_at": iso(folder.created_at),
# A live public link opens it: whatever is moved in is shown to whoever holds the link.
"has_link": linked,
}
@router.get("/folders")
def list_folders(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
"""The caller's folders with how many files each holds and their size. A source folder its
files have all left stays (count 0): the client decides whether to show it."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
def _linked(db: Session, folder_ids: list[int]) -> set[int]:
"""Which of these folders a live public link opens (one not past its expiry)."""
now = datetime.now(timezone.utc)
rows = db.execute(
select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
select(DownloadLink.folder_id).where(
DownloadLink.folder_id.in_(folder_ids),
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
)
.outerjoin(DownloadJob, done)
).scalars()
return {folder_id for folder_id in rows if folder_id is not None}
@router.get("/folders")
def list_folders(
caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
) -> list[dict]:
"""The caller's folders with how many files each holds and their size (a token: the ones it may
list). A source folder its files have all left stays (count 0): the client decides whether to
show it."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
stmt = select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
).where(FileFolder.user_id == caller.user.id)
listable = caller.folders_with(Right.LIST)
if listable is not None:
stmt = stmt.where(FileFolder.id.in_(listable))
rows = db.execute(
stmt.outerjoin(DownloadJob, done)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(FileFolder.user_id == user.id)
.group_by(FileFolder.id)
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
).all()
return [_serialize_folder(f, count, int(size)) for f, count, size in rows]
linked = _linked(db, [f.id for f, _, _ in rows])
return [_serialize_folder(f, count, int(size), f.id in linked) for f, count, size in rows]
class FolderIn(BaseModel):
@@ -439,7 +487,7 @@ def update_folder(
)
db.commit()
count, size = _folder_totals(db, folder.id)
return _serialize_folder(folder, count, size)
return _serialize_folder(folder, count, size, folder.id in _linked(db, [folder.id]))
def _folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
@@ -520,12 +568,20 @@ def list_files(
sort: Literal["added", "name", "size"] = "added",
order: Literal["asc", "desc"] = "desc",
scope: Literal["mine", "shared"] = "mine",
user: User = Depends(current_user),
caller: FilesCaller = Depends(files_caller),
db: Session = Depends(get_db),
) -> list[dict]:
"""The caller's done files (`scope=mine`, optionally one folder's), or the ones shared with
them (`scope=shared`); searched by name and channel, sorted by when they were added, by name
(in the reader's alphabet) or by size."""
(in the reader's alphabet) or by size. A token lists only in the folders it may list, and never
the shared ones."""
user = caller.user
listable = caller.folders_with(Right.LIST)
if listable is not None:
if scope == "shared":
raise refused(403, "api_token_scope")
if folder_id is not None and folder_id not in listable:
raise not_found("folder")
stmt = (
select(DownloadJob)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
@@ -540,6 +596,8 @@ def list_files(
stmt = stmt.where(DownloadJob.user_id == user.id)
if folder_id is not None:
stmt = stmt.where(DownloadJob.folder_id == folder_id)
elif listable is not None:
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
term = q.strip()
if term:
@@ -693,11 +751,23 @@ def update_file(
@router.delete("/{job_id}")
def delete_file(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
) -> dict:
"""Delete a file from the caller's library; the bytes go with their last holder."""
job = own_job_for_update(db, user, job_id)
audit.record(db, user.id, AuditAction.FILE_DELETE, target_type="file", target_id=job_id)
"""Delete a file from the caller's library; the bytes go with their last holder. A token
deletes only in a folder it may delete in — judged on the row as locked, so a file moved out of
that folder meanwhile is out of its reach."""
job = own_job_for_update(db, caller.user, job_id)
if not caller.may(job.folder_id, Right.DELETE):
db.rollback() # lets go of the locks; nothing was written
raise not_found("download")
audit.record(
db,
caller.user.id,
AuditAction.FILE_DELETE,
target_type="file",
target_id=job_id,
summary=caller.via,
)
service.delete(db, job) # commits, the audit row with it
return {"ok": True, "deleted": job_id}
@@ -732,6 +802,7 @@ def ingest_spooled(db: Session, upload: SpooledUpload) -> DownloadJob:
display_name=upload.display_name,
sha256=upload.sha256,
folder_id=upload.folder_id,
via=upload.via,
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
@@ -782,8 +853,10 @@ def serve_file(db: Session, job: DownloadJob) -> FileResponse:
@router.get("/{job_id}/file")
def get_file(job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)):
response = serve_file(db, accessible_job(db, user, job_id))
def get_file(
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
):
response = serve_file(db, readable_job(db, caller, job_id))
db.commit()
return response
@@ -840,13 +913,13 @@ def preview_file(
@router.get("/{job_id}/sha256")
def get_sha256(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
) -> dict:
"""The file's SHA-256 (hex), hashed now if it never was and kept on the asset for every holder.
A sync handler: hashing a large file is blocking disk work, so it runs in the threadpool — and
every other sync route shares that pool, so only `_HASHERS` files are hashed at a time (a caller
past that is told to come back), and none of them holds a database connection while it reads."""
asset, path = _ready_file(db, accessible_job(db, user, job_id))
asset, path = _ready_file(db, readable_job(db, caller, job_id))
if asset.sha256:
return {"sha256": asset.sha256}
asset_id = asset.id
+149 -53
View File
@@ -9,7 +9,7 @@ a short-lived signed grant that the media URL carries (a `<video src>` can't sen
from fastapi import APIRouter, Depends, Request
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy import select, update
from sqlalchemy.orm import Session
from app.config import settings
@@ -17,7 +17,7 @@ from app.db import get_db
from app.downloads import links as linksmod
from app.downloads import storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, MediaAsset, User
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset, User
from app.ratelimit import RateLimiter, client_key
from app.security import verify_password
from app.refusal import refused
@@ -38,7 +38,7 @@ _unlock_failures = RateLimiter(max_events=50, window_seconds=300)
def _link_or_404(db: Session, token: str) -> DownloadLink:
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
raise refused(404, "link_gone")
return link
@@ -55,8 +55,7 @@ def _being_made(db: Session, job: DownloadJob) -> bool:
return owner is not None and not owner.is_suspended
def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAsset]:
job = db.get(DownloadJob, link.job_id)
def _ready_job(db: Session, job: DownloadJob | None) -> tuple[DownloadJob, MediaAsset]:
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if (
job is not None
@@ -79,36 +78,114 @@ def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAs
return job, asset
def _ready_asset(db: Session, link: DownloadLink) -> MediaAsset:
return _ready_target(db, link)[1]
def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAsset]:
"""A file link's file. A folder link names no one file: its files are asked for by id."""
if link.job_id is None:
raise refused(404, "video_gone")
return _ready_job(db, db.get(DownloadJob, link.job_id))
def _file_url(token: str, grant: str | None = None) -> str:
base = f"/api/public/watch/{token}/file"
return f"{base}?g={grant}" if grant else base
def _link_folder(db: Session, link: DownloadLink) -> FileFolder:
folder = db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
if folder is None:
raise refused(404, "link_gone")
return folder
def _poster_url(token: str, asset, grant: str | None = None) -> str | None:
def _folder_job(db: Session, link: DownloadLink, job_id: int) -> tuple[DownloadJob, MediaAsset]:
folder = _link_folder(db, link)
job = db.execute(
select(DownloadJob).where(DownloadJob.id == job_id, *linksmod.in_folder(folder))
).scalar_one_or_none()
if job is None:
raise refused(404, "video_gone")
return _ready_job(db, job)
def _base(token: str, job_id: int | None) -> str:
"""A file link's file lives under its token; a folder link's files, each under its id."""
return f"/api/public/watch/{token}" + (f"/files/{job_id}" if job_id is not None else "")
def _file_url(token: str, grant: str | None = None, job_id: int | None = None) -> str:
url = f"{_base(token, job_id)}/file"
return f"{url}?g={grant}" if grant else url
def _poster_url(
token: str, asset: MediaAsset, grant: str | None = None, job_id: int | None = None
) -> str | None:
if not asset.poster_path:
return None
base = f"/api/public/watch/{token}/poster.jpg"
return f"{base}?g={grant}" if grant else base
url = f"{_base(token, job_id)}/poster.jpg"
return f"{url}?g={grant}" if grant else url
def _folder_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
"""A folder link's page: the folder's name and each file's own watch metadata
(`linksmod.folder_files`: ready, on disk, newest first)."""
folder = _link_folder(db, link)
files = [
{
"id": job.id,
**linksmod.public_meta(
link,
job,
asset,
_file_url(token, grant, job.id),
_poster_url(token, asset, grant, job.id),
),
}
for job, asset in linksmod.folder_files(db, folder)
]
return {
"needs_password": False,
"kind": "folder",
"name": folder.name,
"allow_download": link.allow_download,
"files": files,
}
def _file_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
job, asset = _ready_target(db, link)
return {
"needs_password": False,
"kind": "file",
**linksmod.public_meta(
link, job, asset, _file_url(token, grant), _poster_url(token, asset, grant)
),
}
def _opened(db: Session, link: DownloadLink, token: str, grant: str | None = None) -> dict:
"""The page an open link shows, counted as one view (the handler commits the count)."""
meta = (
_folder_meta(db, link, token, grant)
if link.folder_id is not None
else _file_meta(db, link, token, grant)
)
# In SQL, not `+= 1` on the loaded row: two opens at once would both write the same count.
db.execute(
update(DownloadLink)
.where(DownloadLink.id == link.id)
.values(view_count=DownloadLink.view_count + 1)
.execution_options(synchronize_session=False)
)
return meta
@router.get("/watch/{token}")
def watch_meta(token: str, db: Session = Depends(get_db)) -> dict:
"""Public metadata for the watch page. For a password link, returns only `needs_password`
until the viewer unlocks it (no title/thumbnail leak)."""
"""Public metadata for the watch page: one file's, or a folder's list. For a password link,
returns only `needs_password` until the viewer unlocks it (no title/thumbnail leak)."""
link = _link_or_404(db, token)
if link.password_hash:
return {"needs_password": True}
job, asset = _ready_target(db, link)
link.view_count += 1
# Which kind it opens, for the password prompt's title — never what is in it.
return {"needs_password": True, "kind": "folder" if link.folder_id is not None else "file"}
meta = _opened(db, link, token)
db.commit()
return {
"needs_password": False,
**linksmod.public_meta(link, job, asset, _file_url(token), _poster_url(token, asset)),
}
return meta
class WatchUnlockIn(BaseModel):
@@ -119,7 +196,7 @@ class WatchUnlockIn(BaseModel):
def watch_unlock(
token: str, body: WatchUnlockIn, request: Request, db: Session = Depends(get_db)
) -> dict:
"""Exchange the link password for a signed grant carried by the media URL."""
"""Exchange the link password for a signed grant carried by the media URLs."""
# The link first: a token naming none must leave no limiter entry behind, or posting made-up tokens
# would grow the limiter without bound.
link = _link_or_404(db, token)
@@ -128,36 +205,27 @@ def watch_unlock(
raise refused(429, "too_many_attempts")
if not link.password_hash:
# Not a password link — nothing to unlock; just hand back the plain meta.
job, asset = _ready_target(db, link)
return {
"ok": True,
"needs_password": False,
**linksmod.public_meta(link, job, asset, _file_url(token), _poster_url(token, asset)),
}
meta = _opened(db, link, token)
db.commit()
return {"ok": True, **meta}
if not verify_password(body.password, link.password_hash):
_unlock_failures.allow(token)
raise refused(403, "wrong_share_password")
job, asset = _ready_target(db, link)
link.view_count += 1
db.commit()
grant = linksmod.make_grant(token, link.password_version)
return {
"ok": True,
"needs_password": False,
**linksmod.public_meta(
link, job, asset, _file_url(token, grant), _poster_url(token, asset, grant)
),
}
meta = _opened(db, link, token, grant)
db.commit()
return {"ok": True, **meta}
@router.get("/watch/{token}/file")
def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
"""Range-aware stream of the shared file. Inline by default (plays in the page); an
`allow_download` link serves it as an attachment. Password links require a valid grant."""
link = _link_or_404(db, token)
def _granted(link: DownloadLink, token: str, g: str | None) -> None:
"""A password link's files want the grant its unlock handed out."""
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
raise refused(403, "share_password_required")
asset = _ready_asset(db, link)
def _serve(link: DownloadLink, asset: MediaAsset) -> FileResponse:
"""Range-aware stream of a shared file. Inline by default (plays in the page); an
`allow_download` link serves it as an attachment."""
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
if path is None:
raise refused(404, "video_gone")
@@ -184,17 +252,45 @@ def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
)
@router.get("/watch/{token}/poster.jpg")
def watch_poster(token: str, g: str | None = None, db: Session = Depends(get_db)):
"""The generated poster frame for a shared thumbnail-less video (used as the page/video poster
and the link-preview image). Password links require a valid grant, like the file."""
link = _link_or_404(db, token)
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
raise refused(403, "share_password_required")
job, asset = _ready_target(db, link)
def _poster(asset: MediaAsset) -> FileResponse:
if not asset.poster_path:
raise not_found("poster")
path = storage.safe_abs_path(settings.files_root, asset.poster_path)
if path is None:
raise not_found("poster")
return FileResponse(path, media_type="image/jpeg")
@router.get("/watch/{token}/file")
def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
"""A file link's file. Password links require a valid grant."""
link = _link_or_404(db, token)
_granted(link, token, g)
return _serve(link, _ready_target(db, link)[1])
@router.get("/watch/{token}/poster.jpg")
def watch_poster(token: str, g: str | None = None, db: Session = Depends(get_db)):
"""The generated poster frame for a shared thumbnail-less video (used as the page/video poster
and the link-preview image). Password links require a valid grant, like the file."""
link = _link_or_404(db, token)
_granted(link, token, g)
return _poster(_ready_target(db, link)[1])
@router.get("/watch/{token}/files/{job_id}/file")
def watch_folder_file(token: str, job_id: int, g: str | None = None, db: Session = Depends(get_db)):
"""One file of a folder link's folder, while it is in it. Password links require a grant."""
link = _link_or_404(db, token)
_granted(link, token, g)
return _serve(link, _folder_job(db, link, job_id)[1])
@router.get("/watch/{token}/files/{job_id}/poster.jpg")
def watch_folder_poster(
token: str, job_id: int, g: str | None = None, db: Session = Depends(get_db)
):
"""The poster frame of one file of a folder link's folder, under the same terms as the file."""
link = _link_or_404(db, token)
_granted(link, token, g)
return _poster(_folder_job(db, link, job_id)[1])
+20 -9
View File
@@ -6,7 +6,7 @@ The aggregate daily budget lives in ApiQuotaUsage; here we read the per-spend au
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, Query
from sqlalchemy import func, select
from sqlalchemy.orm import Session
@@ -14,6 +14,8 @@ from app import quota
from app.auth import admin_user, current_user
from app.db import get_db
from app.models import QuotaEvent, User
from app.routes import _account as account
from app.routes._keyset import utc
router = APIRouter(prefix="/api/quota", tags=["quota"])
@@ -45,23 +47,32 @@ def my_usage(user: User = Depends(current_user), db: Session = Depends(get_db))
@router.get("/admin")
def admin_usage(
days: int = 30,
since: datetime | None = None,
until: datetime | None = None,
user: str | None = Query(default=None, pattern=account.PATTERN),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
days = max(1, min(days, 365))
since = datetime.now(timezone.utc) - timedelta(days=days)
"""Spend per account (and per action) and per day, between `since` (default: 30 days back) and
`until` (default: now), optionally for one account (`none` = the background)."""
now = datetime.now(timezone.utc)
lo = utc(since) or now - timedelta(days=30)
window = [QuotaEvent.created_at >= lo]
if until is not None:
window.append(QuotaEvent.created_at < utc(until))
if user is not None:
window.append(account.condition(QuotaEvent.user_id, user))
totals = db.execute(
select(QuotaEvent.user_id, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(QuotaEvent.user_id)
).all()
per_action: dict[int | None, dict[str, int]] = {}
for uid, action, units in db.execute(
select(QuotaEvent.user_id, QuotaEvent.action, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(QuotaEvent.user_id, QuotaEvent.action)
).all():
per_action.setdefault(uid, {})[action] = int(units)
@@ -84,16 +95,16 @@ def admin_usage(
]
rows.sort(key=lambda r: r["total"], reverse=True)
# Instance-wide daily series in Pacific days (matches the budget reset).
# The daily series in Pacific days (matches the budget reset), of the one account when filtered.
day_expr = func.date(func.timezone("America/Los_Angeles", QuotaEvent.created_at))
daily = [
{"day": str(day), "total": int(units)}
for day, units in db.execute(
select(day_expr, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(day_expr)
.order_by(day_expr)
).all()
]
return {"range_days": days, "rows": rows, "daily": daily}
return {"rows": rows, "daily": daily}
+156
View File
@@ -0,0 +1,156 @@
"""Admin background-run history: read `job_runs`, a page at a time.
Admin-only and read-only — rows leave the table through the retention job
(`job_run_retention_days`). Filtered on the server and paged by `_keyset`, like the server log and
the audit log beside it on the activity page. A row names what a reader needs without a second
request: the channel a user's ad-hoc run worked on, and how many log lines the run left (only
WARNING and up are stored, so most runs have none — the count says so before the click).
"""
from dataclasses import dataclass
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app import scheduler
from app.auth import admin_user
from app.db import get_db
from app.models import Channel, JobRun, LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.runstatus import RunStatus
from app.utils import iso, like_escape
router = APIRouter(prefix="/api/admin/runs", tags=["admin"])
Facet = Literal["job", "status"]
def _serialize(run: JobRun, titles: dict[str, str], lines: dict[int, int]) -> dict:
return {
"id": run.id,
"job": run.job,
"trigger": run.trigger,
"actor_id": run.actor_id,
"target": run.target,
"target_title": titles.get(run.target) if run.target is not None else None,
"status": run.status,
"started_at": iso(run.started_at),
"finished_at": iso(run.finished_at),
"summary": run.summary,
"result": run.result,
"error": run.error,
"failure": run.failure,
"quota_units": run.quota_units,
"log_lines": lines.get(run.id, 0),
}
@dataclass(frozen=True)
class RunFilters:
job: str | None
status: RunStatus | None
actor: str | None
q: str | None
since: datetime | None
until: datetime | None
def conditions(self, skip: Facet | None = None) -> list:
"""The WHERE conditions, ANDed. `skip` leaves one menu's own filter out, so that menu counts
its values under every other filter (the way back from a pick)."""
out = []
if self.job is not None and skip != "job":
out.append(JobRun.job == self.job)
if self.status is not None and skip != "status":
out.append(JobRun.status == self.status)
if self.actor is not None:
out.append(account.condition(JobRun.actor_id, self.actor))
if self.since is not None:
out.append(JobRun.started_at >= utc(self.since))
if self.until is not None:
out.append(JobRun.started_at < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
or_(
JobRun.job.ilike(like),
JobRun.target.ilike(like),
JobRun.summary.ilike(like),
JobRun.error.ilike(like),
)
)
return out
def run_filters(
job: str | None = Query(default=None, max_length=40, pattern=r"^[a-z][a-z0-9_]*$"),
status: RunStatus | None = None,
user: str | None = Query(default=None, pattern=account.PATTERN),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> RunFilters:
return RunFilters(job, status, user, q, since, until)
@router.get("")
def list_runs(
filters: RunFilters = Depends(run_filters),
before: str | None = Query(default=None, max_length=200),
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""One page of runs, newest first, matching every filter given; `before` continues from a
previous page's `next_cursor`."""
query = select(JobRun).where(*filters.conditions())
if before is not None:
query = query.where(older_than(JobRun.started_at, JobRun.id, before))
order = (JobRun.started_at.desc(), JobRun.id.desc())
runs, cursor = page(
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
limit,
lambda r: r.started_at,
)
channels = {r.target for r in runs if r.job in scheduler.AD_HOC_JOBS and r.target}
titles: dict[str, str] = {}
if channels:
rows = db.execute(select(Channel.id, Channel.title).where(Channel.id.in_(channels)))
titles = {cid: title for cid, title in rows.tuples() if title is not None}
lines: dict[int, int] = {}
if runs:
counted = db.execute(
select(LogEvent.run_id, func.count())
.where(LogEvent.run_id.in_([r.id for r in runs]))
.group_by(LogEvent.run_id)
)
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
return {
"items": [_serialize(r, titles, lines) for r in runs],
"accounts": accounts(db, {r.actor_id for r in runs if r.actor_id is not None}),
"next_cursor": cursor,
}
@router.get("/facets")
def run_facets(
filters: RunFilters = Depends(run_filters),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""What the job menu and the status chips offer, each counted under every filter but its own."""
def counts(col, skip: Facet) -> list[dict]:
n = func.count().label("n")
stmt = select(col, n).where(*filters.conditions(skip)).group_by(col).order_by(n.desc(), col)
return [{"value": v, "count": c} for v, c in db.execute(stmt).all()]
total = select(func.count()).select_from(JobRun).where(*filters.conditions("status"))
return {
"jobs": counts(JobRun.job, "job"),
"statuses": counts(JobRun.status, "status"),
"total": db.scalar(total) or 0,
}
+7 -5
View File
@@ -17,8 +17,9 @@ from app.scheduler import (
JOB_INTERVALS,
MAX_INTERVAL,
MIN_INTERVAL,
apply_interval,
reschedule,
scheduler_snapshot,
stage_interval,
trigger_all,
trigger_job,
)
@@ -42,14 +43,14 @@ def set_job_interval(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Change how often a scheduler job runs (minutes). Persisted as an override and applied
to the live scheduler immediately."""
"""Change how often a scheduler job runs (minutes). Stored as an override together with its
audit row, then applied to the live scheduler."""
if job_id not in JOB_INTERVALS:
raise not_found("job")
minutes = body.interval_minutes
if not (MIN_INTERVAL <= minutes <= MAX_INTERVAL):
raise refused(400, "invalid_interval", min=MIN_INTERVAL, max=MAX_INTERVAL)
applied = apply_interval(db, job_id, minutes)
applied = stage_interval(db, job_id, minutes)
audit.record(
db,
admin.id,
@@ -60,6 +61,7 @@ def set_job_interval(
after={"interval_minutes": applied},
)
db.commit()
reschedule(job_id, applied)
return {"ok": True, "id": job_id, "interval_minutes": applied}
@@ -117,7 +119,7 @@ def set_maintenance_settings(
@router.get("")
def get_scheduler(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
snap = scheduler_snapshot()
snap = scheduler_snapshot(db)
# Work still queued (shared catalog, so these are instance-wide, not per-user).
deep_requested = (
+3 -3
View File
@@ -91,12 +91,12 @@ def setup_admin(body: SetupAdminIn, db: Session = Depends(require_setup)) -> dic
@router.post("/config")
def setup_config(settings_map: dict[str, Any], db: Session = Depends(require_setup)) -> dict:
"""Persist provided DB-backed settings (Google creds, SMTP, quota…). Keys must be in the
sysconfig registry; empty values are skipped (left at the default). Genuinely dynamic
"""Persist provided DB-backed settings (Google creds, SMTP, quota…). Keys must be the
Configuration page's; empty values are skipped (left at the default). Genuinely dynamic
key→value map, so it stays a typed dict rather than a fixed model."""
saved: list[str] = []
for key, value in settings_map.items():
if sysconfig.spec(key) is None:
if sysconfig.config_page_spec(key) is None:
raise refused(400, "unknown_setting", key=key)
if value is None or value == "":
continue
+185
View File
@@ -0,0 +1,185 @@
"""A user's API tokens for their Files library (`app.apitokens`): list, make, revoke.
Signed-in only — `current_user`, never a token: a token cannot make itself a wider one. A new
token's value is in the answer that made it and nowhere after; only its hash is kept.
The Hub preset is the one machine a user swaps files with: it makes (or reuses) their Hub folder
and gives the token every right in it, so the hub both drops files there and picks up the ones put
there for it.
"""
from datetime import datetime, timedelta, timezone
from typing import Literal
from fastapi import APIRouter, Depends
from pydantic import BaseModel, Field
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import apitokens, audit
from app.apitokens import GRANT_COLUMNS, Right
from app.audit import AuditAction
from app.auth import current_user
from app.db import get_db
from app.downloads import folders
from app.models import ApiToken, ApiTokenGrant, FileFolder, User
from app.refusal import refused
from app.routes._common import owned_or_404
from app.security import hash_token
from app.utils import iso
router = APIRouter(prefix="/api/tokens", tags=["tokens"])
# Far more than one person's machines; a ceiling, so a script gone wrong cannot mint without end.
MAX_TOKENS = 20
_MAX_GRANTS = 50
_MAX_EXPIRY_DAYS = 3650
class GrantIn(BaseModel):
folder_id: int
rights: list[Right] = Field(min_length=1, max_length=len(Right))
class TokenIn(BaseModel):
name: str = Field(min_length=1, max_length=80)
# None = never expires.
expires_days: int | None = Field(default=None, ge=1, le=_MAX_EXPIRY_DAYS)
preset: Literal["hub"] | None = None
grants: list[GrantIn] = Field(default_factory=list, max_length=_MAX_GRANTS)
def _serialize(db: Session, token: ApiToken) -> dict:
rows = db.execute(
select(ApiTokenGrant, FileFolder)
.join(FileFolder, FileFolder.id == ApiTokenGrant.folder_id)
.where(ApiTokenGrant.token_id == token.id)
.order_by(FileFolder.id)
).all()
now = datetime.now(timezone.utc)
return {
"id": token.id,
"name": token.name,
"prefix": token.prefix,
"created_at": iso(token.created_at),
"expires_at": iso(token.expires_at),
"expired": token.expires_at is not None and token.expires_at <= now,
"expiring": apitokens.expiring(token, now),
"last_used_at": iso(token.last_used_at),
"grants": [
{
"folder_id": folder.id,
"kind": folder.kind,
"source_key": folder.source_key,
"name": folder.name,
"rights": sorted(apitokens.grant_rights(grant)),
}
for grant, folder in rows
],
}
@router.get("")
def list_tokens(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
tokens = (
db.execute(select(ApiToken).where(ApiToken.user_id == user.id).order_by(ApiToken.id))
.scalars()
.all()
)
return [_serialize(db, t) for t in tokens]
def _wanted_grants(db: Session, user: User, body: TokenIn) -> dict[int, set[Right]]:
"""Folder id → the rights the new token gets there; every folder checked to be the user's, and
writing only where an upload may land (`folders.upload_target`)."""
if body.preset == "hub":
if body.grants:
raise refused(400, "api_token_preset_grants")
return {folders.hub_folder(db, user.id).id: set(Right)}
if not body.grants:
raise refused(400, "api_token_no_grants")
wanted: dict[int, set[Right]] = {}
for grant in body.grants:
wanted.setdefault(grant.folder_id, set()).update(grant.rights)
for folder_id, rights in wanted.items():
try:
if Right.WRITE in rights:
folders.upload_target(db, user.id, folder_id)
else:
folders.own_folder(db, user.id, folder_id)
except folders.FolderError as e:
raise refused(409, e.reason) from e
return wanted
@router.post("")
def create_token(
body: TokenIn, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Make a token; its value is in this answer only."""
name = body.name.strip()
if not name:
raise refused(400, "api_token_name_required")
# The user's row, locked: two requests at once cannot both slip under the ceiling, and a
# session ended since `current_user` read the epoch (unlocked) cannot mint a token that outlives
# it — `bump_session_epoch` takes this lock before it revokes them all. No key update: a row that
# only refers to the user (a folder delete's audit line) must not wait on it, or the two deadlock.
epoch = db.execute(
select(User.session_epoch).where(User.id == user.id).with_for_update(key_share=True)
).scalar_one()
if (epoch or 0) != (user.session_epoch or 0):
raise refused(401, "not_authenticated")
count = db.execute(
select(func.count(ApiToken.id)).where(ApiToken.user_id == user.id)
).scalar_one()
if count >= MAX_TOKENS:
raise refused(409, "api_token_limit", limit=MAX_TOKENS)
wanted = _wanted_grants(db, user, body)
raw = apitokens.mint()
token = ApiToken(
user_id=user.id,
name=name,
prefix=raw[: apitokens.SHOWN_LEN],
token_hash=hash_token(raw),
expires_at=None
if body.expires_days is None
else datetime.now(timezone.utc) + timedelta(days=body.expires_days),
)
db.add(token)
db.flush()
for folder_id, rights in wanted.items():
db.add(
ApiTokenGrant(
token_id=token.id,
folder_id=folder_id,
**{col: right in rights for right, col in GRANT_COLUMNS.items()},
)
)
db.flush()
audit.record(
db,
user.id,
AuditAction.API_TOKEN_CREATE,
target_type="api_token",
target_id=token.id,
after={
"grants": {str(f): sorted(r) for f, r in wanted.items()},
"expires_at": iso(token.expires_at),
},
)
db.commit()
return {"ok": True, **_serialize(db, token), "token": raw}
@router.delete("/{token_id}")
def revoke_token(
token_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Revoke a token: it stops working at its next request (its grants go with it)."""
token = owned_or_404(db, ApiToken, token_id, user, what="api_token")
db.delete(token)
audit.record(
db, user.id, AuditAction.API_TOKEN_REVOKE, target_type="api_token", target_id=token_id
)
db.commit()
return {"ok": True, "deleted": token_id}
+37
View File
@@ -0,0 +1,37 @@
"""The words a background run's row is written with — the `job_runs` table's status and trigger.
One source for the literals: the scheduler writes them, the startup sweep rewrites `running`, the
model's CHECK constraints are built from these sets, and the dashboard reads them back (the frontend's
copies are held to them by `backendIds.contract.test.ts`). The migrated database's CHECKs are frozen
in its migration: a test holds them equal to these sets, so a new word fails until a migration adds
it (`test_runstatus.py`). The download pipeline's `cancelled`/`canceled` drift (`downloads/status.py`)
is why a status is never typed as a bare string.
"""
from enum import StrEnum
class RunStatus(StrEnum):
RUNNING = "running"
OK = "ok"
ERROR = "error"
# Sync was paused: the run was due and deliberately did nothing.
SKIPPED = "skipped"
# The process stopped while the run was going (a deploy, a crash): set at the next startup, so a
# row never claims to be running in a process that no longer exists.
INTERRUPTED = "interrupted"
class RunTrigger(StrEnum):
# The run's interval came round.
SCHEDULED = "scheduled"
# An admin pressed "run now" (or "run all").
MANUAL = "manual"
# A user's own action started work on one target (a channel's full history), outside the
# recurring jobs.
AD_HOC = "ad_hoc"
def sql_in(values: type[StrEnum]) -> str:
"""`'a', 'b'` — the IN-list a CHECK constraint names these values with."""
return ", ".join(f"'{v.value}'" for v in values)
+348 -191
View File
@@ -3,28 +3,31 @@ recent-first / deep backfill. Runs inside the API process (single worker)."""
import contextvars
import threading
from datetime import datetime, timezone
from datetime import timezone
from typing import Callable
from apscheduler.executors.pool import ThreadPoolExecutor
from apscheduler.jobstores.base import ConflictingIdError
from apscheduler.schedulers.background import BackgroundScheduler
from sqlalchemy import select
from app import audit, debugsession, logs, logsink, progress, quota
from app.audit import AuditAction
from app import audit, debugsession, jobruns, logs, logsink, progress, quota, sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads.gc import run_download_gc
from app.models import SchedulerSetting
from app.models import Channel, JobRun
from app.plex.sync import sync as run_plex_sync
from app.plex.tmdb import run_enrich as run_tmdb_enrich
from app.plex.tmdb import run_people_enrich
from app.plex.watch_sync import run_plex_watch_reconcile, run_plex_watch_sync
from app.notifications import create_notification
from app.runstatus import RunStatus, RunTrigger
from app.state import is_sync_paused
from app.sync.autotag import run_autotag_all
from app.sync.explore import purge_ephemeral
from app.sync.maintenance import run_maintenance
from app.sync.playlists import sync_all_playlists
from app.utils import iso
from app.sync.runner import (
run_deep_backfill,
run_enrich,
@@ -37,12 +40,31 @@ from app.sync.runner import (
log = logs.get_logger(__name__)
_scheduler: BackgroundScheduler | None = None
# Per-job run activity, kept in-memory for the admin Scheduler dashboard to read (the
# scheduler lives in the same single-worker process as the API, so a request can read it
# directly). Ephemeral by design — it reflects "what the scheduler is doing right now and
# how the last runs went" since this process started; durable history isn't the goal here.
# What is running RIGHT NOW, per job: the single-run claim and the live progress. In memory on
# purpose — the scheduler lives in the API's one process, and progress changes every second. What a
# run did is durable instead: its `job_runs` row (app/jobruns.py), which outlives a restart.
_activity: dict[str, dict] = {}
_activity_lock = threading.Lock()
# The runs this process has open (run id -> job), under the same lock. A row can stay `running` in
# the table after its outcome failed to write (_close), so the table cannot say what is going; this
# can, as the one process per database (app/jobruns.py) is the one running them.
_open_runs: dict[int, str] = {}
def _opened(run_id: int, job: str) -> None:
with _activity_lock:
_open_runs[run_id] = job
def _closed(run_id: int) -> None:
with _activity_lock:
_open_runs.pop(run_id, None)
def _open_run_ids() -> dict[int, str]:
with _activity_lock:
return dict(_open_runs)
# Set (per-thread) by a manual "run now" trigger to the id of the admin who clicked, so that
# run — and only that run, not the recurring scheduled ones — posts a completion notification
@@ -51,59 +73,36 @@ _manual_actor: contextvars.ContextVar[int | None] = contextvars.ContextVar(
"manual_actor", default=None
)
# Each interval job's env/config default period (minutes). The admin can override any of
# these at runtime (stored in scheduler_settings, applied live); see load_intervals.
# Each interval job's env/config default period (minutes). The admin can override any of these at
# runtime from the Scheduler page — a sysconfig key per job (`sysconfig.interval_key`), applied
# live; see load_intervals.
JOB_INTERVALS: dict[str, int] = {
"rss_poll": settings.rss_poll_minutes,
"enrich": settings.enrich_interval_minutes,
"backfill": settings.backfill_interval_minutes,
"autotag": settings.autotag_interval_minutes,
"shorts": settings.shorts_probe_interval_minutes,
"subscriptions": settings.subscriptions_resync_minutes,
"playlist_sync": settings.playlist_sync_minutes,
"maintenance": settings.maintenance_interval_minutes,
"explore_cleanup": settings.explore_cleanup_minutes,
"download_gc": settings.download_gc_minutes,
"plex_sync": settings.plex_sync_interval_min,
"plex_watch_sync": settings.plex_watch_sync_interval_min,
"plex_watch_reconcile": settings.plex_watch_reconcile_interval_min,
"tmdb_enrich": settings.tmdb_enrich_interval_min,
"people_enrich": settings.people_enrich_interval_min,
"audit_gc": settings.audit_gc_minutes,
"log_gc": settings.log_gc_minutes,
job: getattr(settings, attr) for job, attr in sysconfig.JOB_INTERVAL_DEFAULTS.items()
}
# Sane bounds for an admin-set interval (minutes).
MIN_INTERVAL = 1
MAX_INTERVAL = 1440 # one day
MIN_INTERVAL = sysconfig.MIN_INTERVAL
MAX_INTERVAL = sysconfig.MAX_INTERVAL
def load_intervals(db) -> dict[str, int]:
"""Effective per-job intervals: the env/config defaults overlaid with any admin overrides
saved in scheduler_settings."""
iv = dict(JOB_INTERVALS)
for row in db.execute(select(SchedulerSetting)).scalars():
if row.job_id in iv and row.interval_minutes:
iv[row.job_id] = row.interval_minutes
return iv
"""Effective per-job intervals: each job's sysconfig override, else its env/config default."""
return {job: sysconfig.get_int(db, sysconfig.interval_key(job)) for job in JOB_INTERVALS}
def apply_interval(db, job_id: str, minutes: int) -> int:
"""Persist a job's interval override and reschedule the live job immediately (if the
scheduler runs in this process). Returns the applied value."""
def stage_interval(db, job_id: str, minutes: int) -> int:
"""Stage a job's interval override (clamped) in `db`; returns the value. The caller commits it
with its audit row, then calls `reschedule` — the live job follows only what was stored."""
if job_id not in JOB_INTERVALS:
raise KeyError(job_id)
minutes = max(MIN_INTERVAL, min(MAX_INTERVAL, int(minutes)))
row = db.get(SchedulerSetting, job_id)
if row is None:
db.add(SchedulerSetting(job_id=job_id, interval_minutes=minutes))
else:
row.interval_minutes = minutes
db.commit()
sysconfig.set_value(db, sysconfig.interval_key(job_id), minutes)
return minutes
def reschedule(job_id: str, minutes: int) -> None:
"""Move the live interval job to `minutes` (if this process registered it)."""
if _scheduler is not None and _scheduler.get_job(job_id) is not None:
_scheduler.reschedule_job(job_id, trigger="interval", minutes=minutes)
log.info("rescheduled job %s to every %s min", job_id, minutes)
return minutes
def _record(name: str, **fields) -> None:
@@ -112,7 +111,12 @@ def _record(name: str, **fields) -> None:
def _notify_done(
db, actor_id: int, job_id: str, status: str, summary: str | None, result: dict | None = None
db,
actor_id: int,
job_id: str,
status: RunStatus,
summary: str | None,
result: dict | None = None,
) -> None:
"""Post a completion notice to the triggering admin's inbox (manual runs only). Best
effort — a notification failure must never mask the job's own outcome."""
@@ -130,49 +134,6 @@ def _notify_done(
log.exception("failed to write completion notification for job %s", job_id)
def _run_changed(result) -> bool:
"""Did an automatic job run actually do something worth an audit row? (No-op polls don't.)
A "change" = a truthy NUMERIC count; status-string dicts like {"skipped": "disabled"} (Plex
off — the default) are no-ops, not changes, so they don't flood the trail every interval."""
if result is None:
return False
if isinstance(result, dict):
# A phased job (backfill's recent/deep, people_enrich's bios/portraits) nests a result per
# phase: a count inside a phase is a change too.
return any(
_run_changed(v) if isinstance(v, dict) else isinstance(v, (int, float)) and bool(v)
for v in result.values()
)
if isinstance(result, (int, float)):
return result != 0
return bool(result)
def _audit_run(db, actor_id, job_id, status, summary, result) -> None:
"""Write ONE run-summary audit row per job run (never per-item). Manual runs (an admin
clicked "run now" → actor_id set) and errors are always logged; automatic runs only when
they changed something, so routine no-op polls don't spam the trail. Best-effort."""
if actor_id is None and status == "ok" and not _run_changed(result):
return
# The result as data too: the audit log names it in the reader's language, as the inbox
# does; `summary` stays the English line of the row's own summary column.
data = _result_data(result)
try:
audit.record(
db,
actor_id,
AuditAction.JOB_RUN,
target_type="job",
target_id=job_id,
summary=f"{job_id}: {status}" + (f" — {summary}" if summary else ""),
after={"status": status, "summary": summary, **({"result": data} if data else {})},
)
db.commit()
except Exception:
db.rollback()
log.exception("failed to write audit row for job %s", job_id)
def _job(name: str, fn) -> None:
# The one place both run paths converge (manual trigger AND scheduled fire), so claim the run
# HERE, atomically: whichever reaches this first wins, the other skips. This is the real
@@ -180,71 +141,88 @@ def _job(name: str, fn) -> None:
if not _claim_running(name):
log.info("job %s already running — skipping this trigger", name)
return
db = SessionLocal()
actor_id = _manual_actor.get() # set only for a manual "run now" trigger
_record(
name, last_started=datetime.now(timezone.utc).isoformat(), last_error=None, progress=None
) # running already claimed
trigger = RunTrigger.SCHEDULED if actor_id is None else RunTrigger.MANUAL
_record(name, progress=None) # running already claimed
db = SessionLocal()
run_id = None
try:
try:
run_id = jobruns.start(name, trigger, actor_id)
except Exception:
# No row to tell it in: the database is what failed. The log is the record.
log.exception("job %s not started: its run could not be recorded", name)
return
_opened(run_id, name)
with logs.bind(run_id=run_id):
_run(db, name, fn, run_id, actor_id)
finally:
# ALWAYS release the claim, even if the body raised before its own status update — so a
# failure can never wedge the job as permanently "running".
if run_id is not None:
_closed(run_id)
_record(name, running=False, progress=None)
db.close()
def _run(db, name: str, fn, run_id: int, actor_id: int | None) -> None:
"""The run itself, inside its `job_runs` row: the pause-skip, the work, and its outcome — on
the row, and in the triggering admin's inbox for a manual run."""
def sink(current, total, phase):
_record(name, progress={"current": current, "total": total, "phase": phase})
# Only the work is inside the try: writing its outcome is not the job, and a write that fails
# must not turn a run that succeeded into a failed one (see _close).
result = None
try:
if is_sync_paused(db):
log.info("job %s skipped (sync paused)", name)
_record(
name,
status="skipped",
last_finished=datetime.now(timezone.utc).isoformat(),
last_result="paused",
result=_PAUSED,
progress=None,
)
if actor_id is not None:
_notify_done(db, actor_id, name, "skipped", "sync paused", _PAUSED)
_audit_run(db, actor_id, name, "skipped", "sync paused", _PAUSED)
return
with progress.bind(sink):
result = fn(db)
summary = _summarize(result)
data = _result_data(result)
log.info("job %s -> %s", name, result)
_record(
name,
status="ok",
last_finished=datetime.now(timezone.utc).isoformat(),
last_result=summary,
result=data,
progress=None,
)
if actor_id is not None:
_notify_done(db, actor_id, name, "ok", summary, data)
_audit_run(db, actor_id, name, "ok", summary, result)
paused = is_sync_paused(db)
if not paused:
with progress.bind(sink):
result = fn(db)
except Exception as exc:
db.rollback()
log.exception("job %s failed", name, extra=logs.event("scheduler.job.failed", job=name))
err = str(exc) or exc.__class__.__name__
# A failure the job named (progress.JobFailed) rides along as data, as a result does, so the
# client says it in the reader's language; `err` stays the English line.
failure = (
{"failure": exc.reason, **exc.params} if isinstance(exc, progress.JobFailed) else None
)
_record(
name,
status="error",
last_finished=datetime.now(timezone.utc).isoformat(),
last_error=err,
result=failure,
progress=None,
)
err, failure = _failure(exc)
_close(run_id, RunStatus.ERROR, error=err, failure=failure)
if actor_id is not None:
_notify_done(db, actor_id, name, "error", err, failure)
_audit_run(db, actor_id, name, "error", err, failure)
finally:
# ALWAYS release the claim, even if the body raised before its own status update — so a
# failure can never wedge the job as permanently "running".
_record(name, running=False)
db.close()
_notify_done(db, actor_id, name, RunStatus.ERROR, err, failure)
return
if paused:
log.info("job %s skipped (sync paused)", name)
_close(run_id, RunStatus.SKIPPED, summary="sync paused", result=_PAUSED)
if actor_id is not None:
_notify_done(db, actor_id, name, RunStatus.SKIPPED, "sync paused", _PAUSED)
return
summary, data = _finish_ok(name, run_id, result)
if actor_id is not None:
_notify_done(db, actor_id, name, RunStatus.OK, summary, data)
def _close(run_id: int, status: RunStatus, **outcome) -> None:
"""Write a run's outcome. Its own failure (the database) is the log's to tell, never the job's:
the row then stays `running` until the next start's sweep closes it as interrupted."""
try:
jobruns.finish(run_id, status, **outcome)
except Exception:
log.exception("run %s: its outcome (%s) could not be recorded", run_id, status)
def _finish_ok(name: str, run_id: int, result) -> tuple[str | None, dict | None]:
"""Close a run that returned: its result on the row, as data and as the English one-liner."""
summary = _summarize(result)
data = _result_data(result)
log.info("job %s -> %s", name, result)
_close(run_id, RunStatus.OK, summary=summary, result=data)
return summary, data
def _failure(exc: Exception) -> tuple[str, dict | None]:
"""A raised run's English error line, and the failure the job named (progress.JobFailed) as
data, as a result is, so the client says it in the reader's language."""
err = str(exc) or exc.__class__.__name__
failure = {"failure": exc.reason, **exc.params} if isinstance(exc, progress.JobFailed) else None
return err, failure
# A skipped run's result, in the same shape as a job's own `{"skipped": …}`.
@@ -271,11 +249,13 @@ def _summarize(result) -> str | None:
return str(result)
def scheduler_snapshot() -> dict:
"""What the scheduler is doing, for the admin dashboard. `running_here` is False in
environments where the scheduler is disabled (e.g. local dev shares the prod-ish DB but
runs no scheduler), so the UI can say so while still showing DB-derived queue/quota."""
running_here = _scheduler is not None
def scheduler_snapshot(db) -> dict:
"""What the scheduler is doing, for the admin dashboard: each job's live state from this process
and its last runs from `job_runs`, so a restart no longer blanks "when did it last run, and how
did it go"; and the same for the ad-hoc runs users start on one target. `running_here` is False where the scheduler runs no interval jobs
(SCHEDULER_ENABLED=false), so the UI can say so while still showing the DB-derived history,
queue and quota."""
running_here = _scheduler is not None and settings.scheduler_enabled
next_runs: dict[str, str | None] = {}
live_intervals: dict[str, int] = {}
if _scheduler is not None:
@@ -287,6 +267,12 @@ def scheduler_snapshot() -> dict:
live_intervals[job.id] = round(iv.total_seconds() / 60)
with _activity_lock:
acts = {k: dict(v) for k, v in _activity.items()}
# Open before the read or after it: a run that started or ended while the rows were read is
# still counted as the open run it was, never as a stray one.
open_runs = _open_run_ids()
recent = jobruns.recent_runs(db, [*JOB_INTERVALS, *AD_HOC_JOBS], RECENT_RUNS)
open_runs |= _open_run_ids()
going_id = max((r for r, j in open_runs.items() if j == CHANNEL_BACKFILL), default=None)
jobs = []
for job_id, default_interval in JOB_INTERVALS.items():
a = acts.get(job_id, {})
@@ -296,16 +282,94 @@ def scheduler_snapshot() -> dict:
"interval_minutes": live_intervals.get(job_id, default_interval),
"next_run": next_runs.get(job_id),
"running": a.get("running", False),
"status": a.get("status"),
"last_started": a.get("last_started"),
"last_finished": a.get("last_finished"),
"last_result": a.get("last_result"),
"result": a.get("result"),
"last_error": a.get("last_error"),
"progress": a.get("progress"),
**_last_runs(recent.get(job_id, []), open_runs),
}
)
return {"running_here": running_here, "enabled": settings.scheduler_enabled, "jobs": jobs}
# Work a user starts on one target: no interval, no "run now" (it needs its target), but its
# runs are shown beside the jobs' — it spends the same quota. Its one in-memory guard is the
# target's own (sync/runner), so "running" is the runs this process has open: two channels can
# backfill at once, so a run still going counts even when a newer one already ended. That run then
# leads the row, so its channel, its "no outcome yet" and its dot are one run's, never the
# newer run's outcome under the older one's channel.
backfills = recent.get(CHANNEL_BACKFILL, [])
going = db.get(JobRun, going_id) if going_id is not None else None
if going is not None and going.status != RunStatus.RUNNING:
going = None # its outcome is written; only the bookkeeping has not caught up
if going is not None:
backfills = [going, *(r for r in backfills if r.id != going.id)][:RECENT_RUNS]
shown = backfills[0] if backfills else None
ad_hoc = [
{
"id": CHANNEL_BACKFILL,
"running": going is not None,
"target": shown.target if shown else None,
"target_title": (
db.scalar(select(Channel.title).where(Channel.id == shown.target))
if shown is not None and shown.target
else None
),
**_last_runs(backfills, open_runs),
}
]
return {
"running_here": running_here,
"enabled": settings.scheduler_enabled,
"jobs": jobs,
"ad_hoc": ad_hoc,
}
# How many past runs a Scheduler row shows as dots; the full history is the run list.
RECENT_RUNS = 3
def _seen(run, open_runs: dict[int, str]) -> RunStatus:
"""A run's status as the page tells it. A row `running` that no run here has open is one whose
outcome failed to write: the next start's sweep closes it as interrupted, so it reads so now —
not as a run still going beside a row that says nothing runs."""
if run.status == RunStatus.RUNNING and run.id not in open_runs:
return RunStatus.INTERRUPTED
return RunStatus(run.status)
def _last_runs(runs: list, open_runs: dict[int, str]) -> dict:
"""A row's run fields from its newest runs (newest first): how the last one went, and the few
before it for the history dots."""
run = runs[0] if runs else None
status = _seen(run, open_runs) if run is not None else None
# A run still going has no outcome yet: only an ended run says how it went.
ended = run if status is not None and status != RunStatus.RUNNING else None
return {
"status": status if ended else None,
"last_started": iso(run.started_at) if run else None,
# A stray row has no end of its own: it ended, as far as anyone can tell, where it began.
"last_finished": iso(ended.finished_at or ended.started_at) if ended else None,
"last_result": ended.summary if ended else None,
# The outcome as data: the result, or on a failed run the failure the job named.
"result": _outcome_data(ended),
"last_error": ended.error if ended else None,
"last_trigger": run.trigger if run else None,
"last_run_id": run.id if run else None,
"quota_units": ended.quota_units if ended else None,
"recent": [_run_brief(r, open_runs) for r in runs],
}
def _run_brief(run, open_runs: dict[int, str]) -> dict:
return {
"id": run.id,
"status": _seen(run, open_runs),
"trigger": run.trigger,
"started_at": iso(run.started_at),
"finished_at": iso(run.finished_at),
}
def _outcome_data(run) -> dict | None:
if run is None:
return None
return run.failure if run.status == RunStatus.ERROR else run.result
def _rss_job() -> None:
@@ -370,7 +434,8 @@ def _explore_cleanup_job() -> None:
def _download_gc_job() -> None:
# Retention GC for the download center: pre-expiry warnings, TTL deletion, LRU eviction.
# Retention GC for the download center: pre-expiry warnings, TTL deletion, LRU eviction — and
# the week-ahead warning before an API token runs out (`apitokens.warn_expiring`).
# Pure disk/DB work (no YouTube quota).
_job("download_gc", run_download_gc)
@@ -407,15 +472,16 @@ def _people_enrich_job() -> None:
def _audit_gc_job() -> None:
# The history cleanup: audit-log rows past audit_retention_days and YouTube quota history past
# quota_retention_days (0 = keep forever, each). Pure DB, no quota.
# The history cleanup: audit-log rows past audit_retention_days, YouTube quota history past
# quota_retention_days and finished runs past job_run_retention_days (0 = keep forever, each).
# Pure DB, no quota.
def work(db):
from app import sysconfig
reaped = audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))
quota_reaped = quota.gc(db, sysconfig.get_int(db, "quota_retention_days"))
runs_reaped = jobruns.gc(db, sysconfig.get_int(db, "job_run_retention_days"))
db.commit()
return {"reaped": reaped, "quota_reaped": quota_reaped}
return {"reaped": reaped, "quota_reaped": quota_reaped, "runs_reaped": runs_reaped}
_job("audit_gc", work)
@@ -424,7 +490,6 @@ def _log_gc_job() -> None:
# Prune log_events rows older than log_retention_days — the limit the privacy policy states, so
# this runs hourly rather than daily. Pure DB, no quota.
def work(db):
from app import sysconfig
days = sysconfig.get_int(db, "log_retention_days")
reaped = logsink.gc(db, days)
@@ -484,15 +549,29 @@ def running_job_ids() -> set[str]:
return {jid for jid, a in _activity.items() if a.get("running")}
def trigger_job(job_id: str, actor_id: int | None = None) -> str:
"""Run a job immediately in a background thread, independent of its interval schedule.
The wrapper handles its own DB session, activity tracking and pause-skip, so the live
dashboard reflects the run. Returns "started", "already_running", or raises KeyError for
an unknown job. Refusing a concurrent run keeps a manual trigger from overlapping a
scheduled run (APScheduler enforces max_instances=1 for the scheduled side).
def _as_actor(fn: Callable[[], None], actor_id: int | None) -> Callable[[], None]:
"""`fn` run as `actor_id`'s manual run. The actor is RESET afterwards: the executor's pool
threads are reused and keep their context from task to task, so an actor left set would make
the next scheduled run that thread picks up look like this admin's manual run."""
`actor_id` (the admin who clicked "run now") is stashed in a contextvar inside the new
thread so the run posts a completion notification to that admin's inbox."""
def run() -> None:
token = _manual_actor.set(actor_id)
try:
fn()
finally:
_manual_actor.reset(token)
return run
def trigger_job(job_id: str, actor_id: int | None = None) -> str:
"""Run a job now, independent of its interval schedule: a one-shot job on the same scheduler,
so a manual run shares the scheduled runs' executor and shutdown — no thread of its own. The
wrapper handles its own DB session, run row and pause-skip. Returns "started",
"already_running", or raises KeyError for an unknown job.
`actor_id` (the admin who clicked "run now") is set in a contextvar inside the run's thread, so
the run is recorded as theirs and posts a completion notification to their inbox."""
fn = JOB_FUNCS.get(job_id)
if fn is None:
raise KeyError(job_id)
@@ -501,13 +580,18 @@ def trigger_job(job_id: str, actor_id: int | None = None) -> str:
# _job claims, sees it's taken, and skips — no double run.
if _is_running(job_id):
return "already_running"
sched = _scheduler
if sched is None:
# Only between shutdown and exit: start_scheduler runs at every API startup.
raise RuntimeError("the scheduler is not running")
def run() -> None:
if actor_id is not None:
_manual_actor.set(actor_id)
fn()
threading.Thread(target=run, name=f"trigger-{job_id}", daemon=True).start()
try:
# One id per job: a second click before the first one-shot has even started is the same
# run, not a queued second one. misfire_grace_time=None: a busy pool delays it, never drops it.
sched.add_job(_as_actor(fn, actor_id), "date", id=f"{job_id}@now", misfire_grace_time=None)
# silent-ok: not a failure — the answer is the 409 the admin sees ("already running").
except ConflictingIdError:
return "already_running"
return "started"
@@ -520,22 +604,95 @@ def trigger_all(actor_id: int | None = None) -> list[str]:
return started
def start_scheduler() -> None:
global _scheduler
if not settings.scheduler_enabled or _scheduler is not None:
return
# Effective intervals = env defaults overlaid with any admin overrides from the DB.
db = SessionLocal()
# Work a user starts on one target outside the job list — today the "get full history" backfill of
# one channel. It runs on the same scheduler as a recorded run (trigger `ad_hoc`), so what it did and
# spent is in `job_runs` like any job's, not a background task only the log ever heard of.
CHANNEL_BACKFILL = "channel_backfill"
# Every ad-hoc job id: the page names each one (a frontend contract test reads this tuple).
AD_HOC_JOBS = (CHANNEL_BACKFILL,)
def run_ad_hoc(job: str, fn: Callable[[], dict], *, actor_id: int, target: str) -> None:
"""Run `fn` now as `actor_id`'s run of `job` on `target`, a one-shot job on the scheduler. Its
own exclusion (if any) lives in `fn`, which says so in its result; one id per job and target, so
a second request before the first has even started is that same run."""
if job not in AD_HOC_JOBS:
raise KeyError(job) # an unnamed job would show on the page as its bare id
sched = _scheduler
if sched is None:
# Only between shutdown and exit: start_scheduler runs at every API startup.
raise RuntimeError("the scheduler is not running")
try:
iv = load_intervals(db)
sched.add_job(
_ad_hoc,
"date",
args=(job, fn, actor_id, target),
id=f"{job}@{target}",
misfire_grace_time=None,
)
# silent-ok: not a failure — the run for this target is already queued, and it does this work.
except ConflictingIdError:
log.info("ad-hoc %s for %s already queued", job, target)
def _ad_hoc(job: str, fn: Callable[[], dict], actor_id: int, target: str) -> None:
try:
run_id = jobruns.start(job, RunTrigger.AD_HOC, actor_id, target)
except Exception:
# No row to tell it in: the database is what failed. The log is the record.
log.exception("ad-hoc %s for %s not started: its run could not be recorded", job, target)
return
_opened(run_id, job)
try:
_ad_hoc_work(job, fn, run_id)
finally:
db.close()
scheduler = BackgroundScheduler(timezone="UTC")
for job_id, fn in JOB_FUNCS.items():
scheduler.add_job(fn, "interval", minutes=iv[job_id], id=job_id)
_closed(run_id)
def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int) -> None:
with logs.bind(run_id=run_id):
# As in _run: only the work is inside the try.
try:
result = fn()
except Exception as exc:
log.exception("job %s failed", job, extra=logs.event("scheduler.job.failed", job=job))
err, failure = _failure(exc)
_close(run_id, RunStatus.ERROR, error=err, failure=failure)
return
_finish_ok(job, run_id, result)
# Room for every job at once (a "run all") beside a few ad-hoc runs: APScheduler's default pool of
# 10 would queue a scheduled fire behind long manual runs.
_POOL_SIZE = len(JOB_FUNCS) + 8
def start_scheduler() -> None:
"""Start this process's scheduler. It always starts, so "run now" works on every install; only
with SCHEDULER_ENABLED does it also register the recurring interval jobs. Before anything can
run, every run a previous process left open is closed as interrupted — one API process serves a
database (app/jobruns.py), so an open run here is a dead process's."""
global _scheduler
if _scheduler is not None:
return
left = jobruns.interrupt_leftovers()
if left:
log.warning("%s run(s) left running by the previous process marked interrupted", left)
scheduler = BackgroundScheduler(
timezone="UTC", executors={"default": ThreadPoolExecutor(_POOL_SIZE)}
)
if settings.scheduler_enabled:
# Effective intervals = env defaults overlaid with any admin overrides from the DB.
db = SessionLocal()
try:
iv = load_intervals(db)
finally:
db.close()
for job_id, fn in JOB_FUNCS.items():
scheduler.add_job(fn, "interval", minutes=iv[job_id], id=job_id)
scheduler.start()
_scheduler = scheduler
log.info("scheduler started")
log.info("scheduler started (interval jobs %s)", "on" if settings.scheduler_enabled else "off")
def shutdown_scheduler() -> None:
+6
View File
@@ -46,6 +46,12 @@ _fernet: Fernet | None = (
)
def can_encrypt() -> bool:
"""Whether a value can be stored encrypted here: TOKEN_ENCRYPTION_KEY is set. Production
refuses to start without it (`config`); a local box may run without one."""
return _fernet is not None
def _require_fernet() -> Fernet:
if _fernet is None:
raise RuntimeError("TOKEN_ENCRYPTION_KEY is not configured")
+22 -21
View File
@@ -437,7 +437,7 @@ def run_deep_backfill(db: Session, max_channels: int = 10, max_pages: int = 5) -
# Channels being deep-backfilled RIGHT NOW by EITHER path — the interactive backfill_channel_now (a
# FastAPI BackgroundTasks threadpool thread) OR the scheduler's run_deep_backfill (its own thread).
# one-shot run on the scheduler's pool, see scheduler.run_ad_hoc) OR the scheduler's run_deep_backfill.
# BOTH register the channel they're paging (under the lock) so the other skips it: no two runs page
# the same channel's `backfill_cursor` concurrently, whichever started first. `_immediate_running`
# counts only the interactive path, capped so one user's burst can't drain the shared budget (the
@@ -448,9 +448,9 @@ _backfill_lock = threading.Lock()
_MAX_CONCURRENT_IMMEDIATE = 2
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> None:
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, in a
background task with its OWN session, so the full catalogue lands in seconds instead of over the
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> dict:
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, as its
own recorded run with its OWN session, so the full catalogue lands in seconds instead of over the
paced scheduler's 10-minute ticks (max_pages=5/channel/tick). max_pages is high enough to finish
any realistic channel in a single run (400 pages = 20 000 videos; e.g. BBC Earth's ~3400 = 69
pages — the old 60-page cap stopped one page short of large channels, so their tail dribbled in
@@ -459,34 +459,35 @@ def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) ->
Quota-guarded, capped to _MAX_CONCURRENT_IMMEDIATE interactive runs so one user can't drain the
shared backfill budget by opting many channels in at once (the overflow, and any channel too large
to finish under the reserve, is finished by the scheduler since deep_requested stays set), and
mutually excluded per-channel with the scheduler via _backfill_in_progress. Best-effort — logs and
swallows failures."""
mutually excluded per-channel with the scheduler via _backfill_in_progress.
Returns what the run did, as a job result: the counts, or `{"skipped": <why>}` when it left the
channel to the scheduler. A failure raises — the run that wraps this records it."""
global _immediate_running
with _backfill_lock:
if channel_id in _backfill_in_progress or _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
return # already backfilling this channel, or too many interactive runs — leave it to the scheduler
if channel_id in _backfill_in_progress:
return {"skipped": "channel busy"}
if _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
return {"skipped": "at capacity"}
_backfill_in_progress.add(channel_id)
_immediate_running += 1
try:
with SessionLocal() as db:
user = get_service_user(db)
if user is None:
return
return {"skipped": "no service user"}
channel = db.get(Channel, channel_id)
if channel is None or channel.backfill_done or not channel.uploads_playlist_id:
return
return {"skipped": "nothing to do"}
if quota.backfill_reserve_reached():
return
try:
# Attribute the spend to the requesting user under the full-backfill action, so it
# shows up correctly in the usage breakdown (not the default OTHER/system bucket).
with quota.attribute(user_id, quota.QuotaAction.VIDEOS_BACKFILL_FULL):
with YouTubeClient(db, user) as yt:
backfill_channel_deep(db, yt, channel, max_pages)
reconcile_full_history(db)
except Exception:
db.rollback()
log.exception("Immediate backfill failed for channel %s", channel_id)
return {"skipped": "low quota"}
# Attribute the spend to the requesting user under the full-backfill action, so it
# shows up correctly in the usage breakdown (not the default OTHER/system bucket).
with quota.attribute(user_id, quota.QuotaAction.VIDEOS_BACKFILL_FULL):
with YouTubeClient(db, user) as yt:
added = backfill_channel_deep(db, yt, channel, max_pages)
reconciled = reconcile_full_history(db)
return {"videos_added": added, "reconciled_full": reconciled}
finally:
with _backfill_lock:
_backfill_in_progress.discard(channel_id)
+251 -61
View File
@@ -8,7 +8,8 @@ stored Fernet-encrypted at rest (requires TOKEN_ENCRYPTION_KEY; without it, secr
stored in the DB and stay env-only).
"""
from dataclasses import dataclass
from dataclasses import dataclass, field
from typing import Literal
from sqlalchemy import select
from sqlalchemy.orm import Session
@@ -27,6 +28,16 @@ _decrypt_warned: set[str] = set()
_coerce_warned: set[tuple[str, str]] = set()
# When a changed value takes effect. "live": the next read obeys it — every consumer reads the key
# where it uses it, never once at import (test-held: a read outside a function fails). A value read
# once at startup is live only when its writer also applies it (the job intervals reschedule, the log
# level is set as it is saved). "restart": the process has to start
# again. No key needs a restart today, and the Configuration page has no way to say so yet, so a
# "restart" key is refused until the page can (test-held).
Applies = Literal["live", "restart"]
LIVE: Applies = "live"
@dataclass(frozen=True)
class ConfigSpec:
key: str
@@ -37,6 +48,8 @@ class ConfigSpec:
min: int | None = None
max: int | None = None
allow_empty: bool = False # str keys: a stored "" is an explicit empty value, not "unset"
# Required, with no default: each key states it, so a new one cannot be live by omission.
applies: Applies = field(kw_only=True)
@property
def choices(self) -> tuple[str, ...]:
@@ -45,30 +58,78 @@ class ConfigSpec:
return choices_of(Settings.model_fields[self.default_attr].annotation)
# Each interval job of the scheduler and the `settings` attribute holding its default period
# (minutes). Its override is the `job_interval_<job>` key, set from the Scheduler page.
JOB_INTERVAL_DEFAULTS: dict[str, str] = {
"rss_poll": "rss_poll_minutes",
"enrich": "enrich_interval_minutes",
"backfill": "backfill_interval_minutes",
"autotag": "autotag_interval_minutes",
"shorts": "shorts_probe_interval_minutes",
"subscriptions": "subscriptions_resync_minutes",
"playlist_sync": "playlist_sync_minutes",
"maintenance": "maintenance_interval_minutes",
"explore_cleanup": "explore_cleanup_minutes",
"download_gc": "download_gc_minutes",
"plex_sync": "plex_sync_interval_min",
"plex_watch_sync": "plex_watch_sync_interval_min",
"plex_watch_reconcile": "plex_watch_reconcile_interval_min",
"tmdb_enrich": "tmdb_enrich_interval_min",
"people_enrich": "people_enrich_interval_min",
"audit_gc": "audit_gc_minutes",
"log_gc": "log_gc_minutes",
}
# Sane bounds for an admin-set interval (minutes).
MIN_INTERVAL = 1
MAX_INTERVAL = 1440 # one day
def interval_key(job: str) -> str:
return f"job_interval_{job}"
# Registry of DB-overridable keys. Add a key here + wire its read through get_*() to move it
# off env into the admin UI; the admin page renders it automatically from this list. A group's keys
# stay together, in the order its section lists them; a NEW group also needs its place in SECTIONS.
SPECS: tuple[ConfigSpec, ...] = (
# --- Access / registration ---
ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
ConfigSpec("allow_registration", "bool", "access", "allow_registration", applies=LIVE),
# --- Email / SMTP (one logical group; the password is the only secret) ---
ConfigSpec("smtp_host", "str", "email", "smtp_host"),
ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535),
ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True),
ConfigSpec("smtp_from", "str", "email", "smtp_from"),
ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True),
ConfigSpec("smtp_host", "str", "email", "smtp_host", applies=LIVE),
ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535, applies=LIVE),
ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True, applies=LIVE),
ConfigSpec("smtp_from", "str", "email", "smtp_from", applies=LIVE),
ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True, applies=LIVE),
# --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True, applies=LIVE),
# Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
ConfigSpec(
"youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True, applies=LIVE
),
# --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
# install wizard / Configuration page so no restart is needed when the creds change. ---
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
# --- Quota (shared YouTube Data API daily budget) ---
ConfigSpec("quota_daily_budget", "int", "quota", "quota_daily_budget", min=1, max=1_000_000),
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True, applies=LIVE),
ConfigSpec(
"backfill_quota_reserve", "int", "quota", "backfill_quota_reserve", min=0, max=1_000_000
"google_client_secret", "str", "google", "google_client_secret", secret=True, applies=LIVE
),
# --- Quota (shared YouTube Data API daily budget) ---
ConfigSpec(
"quota_daily_budget",
"int",
"quota",
"quota_daily_budget",
min=1,
max=1_000_000,
applies=LIVE,
),
ConfigSpec(
"backfill_quota_reserve",
"int",
"quota",
"backfill_quota_reserve",
min=0,
max=1_000_000,
applies=LIVE,
),
ConfigSpec(
"search_daily_limit_per_user",
@@ -77,10 +138,19 @@ SPECS: tuple[ConfigSpec, ...] = (
"search_daily_limit_per_user",
min=0,
max=100_000,
applies=LIVE,
),
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
ConfigSpec("search_source", "choice", "quota", "search_source"),
ConfigSpec("quota_retention_days", "int", "quota", "quota_retention_days", min=0, max=3650),
ConfigSpec("search_source", "choice", "quota", "search_source", applies=LIVE),
ConfigSpec(
"quota_retention_days",
"int",
"quota",
"quota_retention_days",
min=0,
max=3650,
applies=LIVE,
),
# --- Backfill (recent-first first pass per channel) ---
ConfigSpec(
"backfill_recent_max_videos",
@@ -89,19 +159,46 @@ SPECS: tuple[ConfigSpec, ...] = (
"backfill_recent_max_videos",
min=1,
max=100_000,
applies=LIVE,
),
ConfigSpec(
"backfill_recent_max_days", "int", "backfill", "backfill_recent_max_days", min=1, max=36_500
"backfill_recent_max_days",
"int",
"backfill",
"backfill_recent_max_days",
min=1,
max=36_500,
applies=LIVE,
),
ConfigSpec(
"rss_fallback_hours", "int", "backfill", "rss_fallback_hours", min=0, max=168, applies=LIVE
),
ConfigSpec("rss_fallback_hours", "int", "backfill", "rss_fallback_hours", min=0, max=168),
# --- Shorts probe ---
ConfigSpec(
"shorts_probe_max_seconds", "int", "shorts", "shorts_probe_max_seconds", min=1, max=3_600
"shorts_probe_max_seconds",
"int",
"shorts",
"shorts_probe_max_seconds",
min=1,
max=3_600,
applies=LIVE,
),
ConfigSpec(
"shorts_probe_batch", "int", "shorts", "shorts_probe_batch", min=1, max=10_000, applies=LIVE
),
ConfigSpec("shorts_probe_batch", "int", "shorts", "shorts_probe_batch", min=1, max=10_000),
# --- Batch sizes ---
ConfigSpec("enrich_batch_size", "int", "batch", "enrich_batch_size", min=1, max=50),
ConfigSpec("autotag_title_sample", "int", "batch", "autotag_title_sample", min=1, max=1_000),
ConfigSpec(
"enrich_batch_size", "int", "batch", "enrich_batch_size", min=1, max=50, applies=LIVE
),
ConfigSpec(
"autotag_title_sample",
"int",
"batch",
"autotag_title_sample",
min=1,
max=1_000,
applies=LIVE,
),
# --- Maintenance (rolling re-validation of already-known videos) ---
# Videos re-checked per maintenance run. Folded here from app_state (B-core-11) so every admin
# knob lives in one store; read by sync/maintenance.py, tuned from the Scheduler dashboard.
@@ -112,16 +209,33 @@ SPECS: tuple[ConfigSpec, ...] = (
"maintenance_revalidate_batch",
min=100,
max=100_000,
applies=LIVE,
),
# --- Discovery cleanup (search + explore ephemeral content) ---
# Days an explored-but-unsubscribed channel is kept before the cleanup job purges it.
ConfigSpec("explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650),
# Days an un-kept live-search result video is kept before the cleanup job removes it.
ConfigSpec("search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650),
# --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
ConfigSpec("download_total_max_bytes", "int", "downloads", "download_total_max_bytes", min=0),
ConfigSpec(
"download_default_max_bytes", "int", "downloads", "download_default_max_bytes", min=0
"explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650, applies=LIVE
),
# Days an un-kept live-search result video is kept before the cleanup job removes it.
ConfigSpec(
"search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650, applies=LIVE
),
# --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
ConfigSpec(
"download_total_max_bytes",
"int",
"downloads",
"download_total_max_bytes",
min=0,
applies=LIVE,
),
ConfigSpec(
"download_default_max_bytes",
"int",
"downloads",
"download_default_max_bytes",
min=0,
applies=LIVE,
),
ConfigSpec(
"download_default_max_concurrent",
@@ -130,6 +244,7 @@ SPECS: tuple[ConfigSpec, ...] = (
"download_default_max_concurrent",
min=1,
max=20,
applies=LIVE,
),
ConfigSpec(
"download_default_max_jobs",
@@ -138,6 +253,7 @@ SPECS: tuple[ConfigSpec, ...] = (
"download_default_max_jobs",
min=1,
max=100_000,
applies=LIVE,
),
# min=1, NOT 0: a stored 0 used to read as "no per-file cap" (the cap was enforced as
# `if cap and size > cap`, so a falsy value disabled it). Now the route passes this number
@@ -161,14 +277,27 @@ SPECS: tuple[ConfigSpec, ...] = (
"download_upload_max_bytes",
min=1,
max=settings.download_upload_max_bytes,
applies=LIVE,
),
ConfigSpec(
"download_retention_days", "int", "downloads", "download_retention_days", min=0, max=3_650
"download_retention_days",
"int",
"downloads",
"download_retention_days",
min=0,
max=3_650,
applies=LIVE,
),
ConfigSpec(
"download_gc_grace_days", "int", "downloads", "download_gc_grace_days", min=0, max=3_650
"download_gc_grace_days",
"int",
"downloads",
"download_gc_grace_days",
min=0,
max=3_650,
applies=LIVE,
),
ConfigSpec("download_layout", "choice", "downloads", "download_layout"),
ConfigSpec("download_layout", "choice", "downloads", "download_layout", applies=LIVE),
ConfigSpec(
"download_worker_concurrency",
"int",
@@ -176,60 +305,112 @@ SPECS: tuple[ConfigSpec, ...] = (
"download_worker_concurrency",
min=1,
max=16,
applies=LIVE,
),
ConfigSpec("sponsorblock_default", "bool", "downloads", "sponsorblock_default"),
# --- Audit log ---
# Days to keep admin audit-log rows before the audit_gc job prunes them (0 = keep forever).
ConfigSpec("audit_retention_days", "int", "audit", "audit_retention_days", min=0, max=3650),
ConfigSpec(
"audit_retention_days",
"int",
"audit",
"audit_retention_days",
min=0,
max=3650,
applies=LIVE,
),
# Days the job_runs table keeps a finished background run before the audit_gc job prunes it.
ConfigSpec(
"job_run_retention_days",
"int",
"audit",
"job_run_retention_days",
min=0,
max=3650,
applies=LIVE,
),
# --- Diagnostics ---
# Runtime log level for the API process. Applied without a restart — see routes/config.py and
# main.py's lifespan.
ConfigSpec("log_level", "choice", "diagnostics", "log_level"),
ConfigSpec("log_level", "choice", "diagnostics", "log_level", applies=LIVE),
# Days the log_events table keeps a row (the log_gc job prunes). No "0 = forever": stored logs
# carry account-linked request history, and the privacy policy names a limit.
ConfigSpec("log_retention_days", "int", "diagnostics", "log_retention_days", min=1, max=90),
ConfigSpec(
"log_retention_days",
"int",
"diagnostics",
"log_retention_days",
min=1,
max=90,
applies=LIVE,
),
# --- Plex integration (optional; local-file playback + Plex metadata) ---
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled"),
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url"),
ConfigSpec("plex_server_token", "str", "plex", "plex_server_token", secret=True),
ConfigSpec("plex_path_map", "str", "plex", "plex_path_map"),
ConfigSpec("plex_libraries", "str", "plex", "plex_libraries"),
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
# (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
ConfigSpec("plex_max_sessions", "int", "plex", "plex_max_sessions", min=1, max=64),
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled", applies=LIVE),
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url", applies=LIVE),
ConfigSpec("plex_server_token", "str", "plex", "plex_server_token", secret=True, applies=LIVE),
ConfigSpec("plex_path_map", "str", "plex", "plex_path_map", applies=LIVE),
ConfigSpec("plex_libraries", "str", "plex", "plex_libraries", applies=LIVE),
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map", applies=LIVE),
ConfigSpec(
"plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16, applies=LIVE
),
ConfigSpec(
"plex_max_sessions", "int", "plex", "plex_max_sessions", min=1, max=64, applies=LIVE
),
# Transcode re-encode tuning. The libx264 knobs apply only to the software fallback.
# crf 0–51 (x264 range); max_height 0 = source.
ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw"),
ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset"),
ConfigSpec("plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51),
ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw", applies=LIVE),
ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset", applies=LIVE),
ConfigSpec(
"plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
"plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51, applies=LIVE
),
ConfigSpec(
"plex_transcode_max_height",
"int",
"plex",
"plex_transcode_max_height",
min=0,
max=4320,
applies=LIVE,
),
# Audio loudness normalisation on the HLS path (remux + transcode alike, not direct play).
ConfigSpec("plex_audio_loudnorm", "bool", "plex", "plex_audio_loudnorm"),
# NOTE: plex_watch_sync_interval_min is a config.py default read by the watch-sync scheduler job
# (scheduler.py); it is intentionally NOT a runtime admin knob, so it stays out of SPECS.
ConfigSpec("plex_audio_loudnorm", "bool", "plex", "plex_audio_loudnorm", applies=LIVE),
# --- TMDB enrichment (optional) — the API key is the only secret ---
ConfigSpec("tmdb_api_key", "str", "tmdb", "tmdb_api_key", secret=True),
ConfigSpec("tmdb_language", "str", "tmdb", "tmdb_language"),
ConfigSpec("tmdb_api_key", "str", "tmdb", "tmdb_api_key", secret=True, applies=LIVE),
ConfigSpec("tmdb_language", "str", "tmdb", "tmdb_language", applies=LIVE),
# --- Bug reports: the Gitea issue tracker a debug session's issue goes to ---
ConfigSpec("issues_gitea_url", "str", "issues", "issues_gitea_url"),
ConfigSpec("issues_gitea_repo", "str", "issues", "issues_gitea_repo"),
ConfigSpec("issues_gitea_token", "str", "issues", "issues_gitea_token", secret=True),
# (tmdb_enrich_interval_min is a scheduler job, tuned from the Scheduler dashboard; tmdb_enrich_batch
# is a config.py default — both stay out of SPECS, like plex_sync_interval_min.)
ConfigSpec("issues_gitea_url", "str", "issues", "issues_gitea_url", applies=LIVE),
ConfigSpec("issues_gitea_repo", "str", "issues", "issues_gitea_repo", applies=LIVE),
ConfigSpec(
"issues_gitea_token", "str", "issues", "issues_gitea_token", secret=True, applies=LIVE
),
# --- Scheduler: how often each interval job runs (minutes), one key per job ---
*(
ConfigSpec(
interval_key(job),
"int",
"scheduler",
attr,
min=MIN_INTERVAL,
max=MAX_INTERVAL,
applies=LIVE,
)
for job, attr in JOB_INTERVAL_DEFAULTS.items()
),
)
# The admin page's sections (the owner's pick): sixteen groups as tabs wrapped to three rows at 133%
# zoom, so they are a vertical list under four headings, in this order. Every group of SPECS sits in
# exactly one section (test-held), so a new group cannot fall off the page.
# exactly one section, or is edited on its own page (test-held), so a new group cannot fall off.
SECTIONS: tuple[tuple[str, tuple[str, ...]], ...] = (
("access", ("access", "google", "email")),
("youtube", ("youtube", "quota", "backfill", "shorts", "batch", "explore")),
("media", ("downloads", "plex", "tmdb")),
("system", ("maintenance", "audit", "diagnostics", "issues")),
)
# Groups another admin page edits, so the Configuration page neither lists nor writes them: an
# interval is changed on the Scheduler page, which also reschedules the running job — a write that
# skipped it would leave the job on its old period while the page showed the new one.
OWN_PAGE_GROUPS: tuple[str, ...] = ("scheduler",)
_BY_KEY: dict[str, ConfigSpec] = {s.key: s for s in SPECS}
@@ -238,6 +419,12 @@ def spec(key: str) -> ConfigSpec | None:
return _BY_KEY.get(key)
def config_page_spec(key: str) -> ConfigSpec | None:
"""`spec`, for the Configuration page: None for a key edited on its own page as well."""
s = _BY_KEY.get(key)
return None if s is None or s.group in OWN_PAGE_GROUPS else s
def secrets_manageable() -> bool:
"""Secret values can only be stored when Fernet is configured (TOKEN_ENCRYPTION_KEY)."""
return bool(settings.token_encryption_key)
@@ -426,13 +613,16 @@ def reset(db: Session, key: str) -> None:
def describe(db: Session) -> dict:
"""Admin-UI metadata + current values, grouped. Secret values are never echoed —
only whether an override is set and whether an env default exists."""
"""Admin-UI metadata + current values, grouped — the keys of the Configuration page, not those
edited on their own page. Secret values are never echoed — only whether an override is set and
whether an env default exists."""
groups: dict[str, list[dict]] = {}
# Every override in one query: a db.get per key was one round trip each (two for a key with no
# row, which the session cannot cache), ~40 ms of the page's ~55 on prod.
rows = {r.key: r for r in db.scalars(select(SystemConfig))}
for s in SPECS:
if s.group in OWN_PAGE_GROUPS:
continue
row = rows.get(s.key)
is_set = row is not None
item: dict = {
+4 -1
View File
@@ -16,6 +16,9 @@ services:
POSTGRES_DB: siftlode_test
tmpfs:
- /var/lib/postgresql/data # ephemeral + fast; the schema is rebuilt per run
# One database per pytest-xdist worker, each through the app's own pool (5 + 10 overflow): the default 100
# connections would run out at 8 workers.
command: ["postgres", "-c", "max_connections=300"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U siftlode -d siftlode_test"]
interval: 1s
@@ -37,4 +40,4 @@ services:
- ./:/app
working_dir: /app
# Coverage is on demand, not in the gate: see docs/development.md.
command: ["python", "-m", "pytest"]
command: ["python", "-m", "pytest", "-n", "auto"]
+6 -1
View File
@@ -7,4 +7,9 @@
# not an unknown marker that quietly does nothing.
testpaths = tests
pythonpath = .
addopts = -q -rs --strict-markers
# `-p tests.db_per_worker`: under pytest-xdist (`-n`) every worker gets its own database — DATABASE_URL + `_gw<N>`,
# created on first use — because the `db` fixture's TRUNCATE-per-test isolation assumes one process per database.
# It is a plugin so it runs before conftest imports the app (whose settings read DATABASE_URL once). Serial: no-op.
# The file is a verbatim copy of a shared tool (see its docstring); change the source, not the copy.
addopts = -q -rs --strict-markers -p tests.db_per_worker
db_per_worker_env = DATABASE_URL
+3
View File
@@ -16,3 +16,6 @@ pytest-cov==7.1.0
ruff==0.15.21
httpx2==2.13.0
pip-audit==2.10.1
# pytest-xdist: the DB lane runs on every core (`-n auto` in docker-compose.test.yml); tests/db_per_worker.py gives
# each worker its own database. Same tests, same strictness — only side by side.
pytest-xdist==3.8.0
+9 -1
View File
@@ -27,7 +27,7 @@ from starlette.requests import HTTPConnection
import app.models # register every model on Base.metadata
import app.state as app_state
from app import security
from app import apitokens, security
from app.auth import current_user
from app.config import settings
from app.db import Base, SessionLocal, engine, get_db
@@ -242,8 +242,16 @@ def as_user(db, override_dep, _configured):
raise HTTPException(status_code=401, detail="Not authenticated")
return registry[uid]
def _session_or_token(request: Request) -> User | None:
# The Files routes' session half (`apitokens.session_caller`): a request carrying a token is
# left to the real token check, exactly as in production.
if apitokens.bearer_token(request) is not None:
return None
return _resolver(request)
override_dep(get_db, db.info["request"])
override_dep(current_user, _resolver)
override_dep(apitokens.session_caller, _session_or_token)
def _as(user: User) -> TestClient:
registry[user.id] = user
+94
View File
@@ -0,0 +1,94 @@
"""Give every pytest-xdist worker its own database, so a suite that shares one test database can run in parallel.
Generic (source of truth: the memory repo's tools/pytest-parallel/db_per_worker.py); a project carries a verbatim copy
next to its conftest — tests often run in a container that cannot see this repo — and
`tools/pytest-parallel/sync-check.sh <copy>` reports drift. Keep the copy byte-identical; change it here.
Use, as a pytest plugin (it then runs before any conftest, i.e. before the app reads its settings):
pytest.ini: addopts = -p tests.db_per_worker (the copy's import path; `python -m pytest` puts cwd on sys.path)
db_per_worker_env = DATABASE_URL (the environment variable that holds the test database)
or call `apply("<ENV VAR>")` yourself before the app is imported. The variable may hold:
- a Postgres URL (`postgresql://…/name`, `postgresql+psycopg://…`): the worker gets `name_gw<N>`, created from the
base database's server if missing (empty — the project's own schema fixture migrates it, as for one process);
- a SQLite URL (`sqlite:///path.db`) or a plain file path: the worker gets `path-gw<N>.db`.
Without xdist (no PYTEST_XDIST_WORKER) or with the variable unset it changes nothing, so a serial run is unaffected.
"""
from __future__ import annotations
import os
import re
from urllib.parse import urlsplit, urlunsplit
import pytest
_PG = re.compile(r"^postgres(?:ql)?(?:\+\w+)?://", re.I)
def pytest_addoption(parser: pytest.Parser) -> None:
parser.addini(
"db_per_worker_env",
"env var holding the test database (a Postgres URL, SQLite URL or file path)",
)
@pytest.hookimpl(tryfirst=True)
def pytest_load_initial_conftests(early_config: pytest.Config) -> None:
var = early_config.getini("db_per_worker_env")
if var:
apply(str(var))
def apply(var: str) -> str | None:
worker = os.environ.get("PYTEST_XDIST_WORKER")
value = os.environ.get(var)
if not worker or not value:
return value
if _PG.match(value):
new = _postgres(value, worker)
else:
new = _file(value, worker)
os.environ[var] = new
return new
def _postgres(url: str, worker: str) -> str:
parts = urlsplit(url)
base = parts.path.lstrip("/")
name = f"{base}_{worker}"
if not re.fullmatch(r"\w+", name):
raise ValueError(f"db_per_worker: refusing an unusual database name {name!r}")
# The server, through the BASE database (it exists; `postgres` may not be reachable for this role).
plain = urlunsplit(
(re.sub(r"\+\w+$", "", parts.scheme), parts.netloc, parts.path, parts.query, "")
)
_create_if_missing(plain, name)
return urlunsplit((parts.scheme, parts.netloc, f"/{name}", parts.query, parts.fragment))
def _create_if_missing(dsn: str, name: str) -> None:
try:
import psycopg # psycopg 3
conn = psycopg.connect(dsn, autocommit=True)
except ImportError:
import psycopg2 # fallback for projects on psycopg2
conn = psycopg2.connect(dsn)
conn.autocommit = True
try:
cur = conn.cursor()
cur.execute("SELECT 1 FROM pg_database WHERE datname = %s", (name,))
if cur.fetchone() is None:
cur.execute(f'CREATE DATABASE "{name}"') # name checked to be \w+ above
finally:
conn.close()
def _file(value: str, worker: str) -> str:
prefix = ""
m = re.match(r"^(sqlite(?:\+\w+)?:///)(.*)$", value, re.I)
if m:
prefix, value = m.group(1), m.group(2)
root, ext = os.path.splitext(value)
return f"{prefix}{root}-{worker}{ext}"
+205
View File
@@ -0,0 +1,205 @@
"""The activity page's reads: the audit log and the background runs, filtered on the server and
paged by a keyset, and the quota breakdown over a time range and one account.
Each list is asked for the rows a filter must keep AND must drop, and a page boundary is crossed
with rows that share a timestamp — the case an offset or a timestamp-only cursor gets wrong.
"""
from datetime import datetime, timedelta, timezone
from app.auth import admin_user
from app.main import app
from app.models import AuditLog, Channel, JobRun, LogEvent, QuotaEvent
from tests.test_body_limit import _route_objects
T0 = datetime(2026, 10, 1, 12, 0, tzinfo=timezone.utc)
def _audit(db, action, actor=None, at=T0, summary=None, request_id=None) -> int:
row = AuditLog(
actor_id=actor, action=action, summary=summary, created_at=at, request_id=request_id
)
db.add(row)
db.flush()
return row.id
def _run(db, job, status="ok", actor=None, at=T0, target=None, trigger="scheduled") -> int:
row = JobRun(
job=job,
trigger=trigger,
actor_id=actor,
target=target,
status=status,
started_at=at,
finished_at=None if status == "running" else at + timedelta(seconds=4),
quota_units=0,
)
db.add(row)
db.flush()
return row.id
def _ids(body) -> list[int]:
return [r["id"] for r in body["items"]]
def test_the_audit_list_filters_on_the_server_and_pages_across_a_shared_timestamp(
db, make_user, as_user
):
admin = make_user(role="admin")
a = _audit(db, "config_set", admin.id, T0, "quota_daily_budget = 9000")
b = _audit(db, "config_set", admin.id, T0, "log_level = DEBUG")
c = _audit(db, "config_set", admin.id, T0, "smtp_host = mail")
bg = _audit(db, "user_purged", None, T0 - timedelta(days=3))
db.commit()
client = as_user(admin)
first = client.get("/api/admin/audit", params={"limit": 2, "action": "config_set"}).json()
assert _ids(first) == [c, b]
second = client.get(
"/api/admin/audit",
params={"limit": 2, "action": "config_set", "before": first["next_cursor"]},
).json()
assert (_ids(second), second["next_cursor"]) == ([a], None)
# A page that ends exactly at the last row says there is nothing older.
whole = client.get("/api/admin/audit", params={"limit": 3, "action": "config_set"}).json()
assert (_ids(whole), whole["next_cursor"]) == ([c, b, a], None)
assert first["accounts"] == {str(admin.id): admin.email}
def ids(**params) -> list[int]:
return _ids(client.get("/api/admin/audit", params=params).json())
assert ids(user="none") == [bg]
assert ids(user=str(admin.id)) == [c, b, a]
assert ids(q="DEBUG") == [b]
assert ids(since=(T0 - timedelta(days=1)).isoformat()) == [c, b, a]
assert ids(until=(T0 - timedelta(days=1)).isoformat()) == [bg]
# The action menu counts every action under the other filters, its own pick included.
facets = client.get("/api/admin/audit/facets", params={"action": "config_set"}).json()
assert facets["actions"] == [
{"value": "config_set", "count": 3},
{"value": "user_purged", "count": 1},
]
def test_an_admin_action_keeps_the_request_it_was_made_in(db, make_user, as_user):
client = as_user(make_user(role="admin"))
r = client.patch("/api/admin/config/quota_daily_budget", json={"value": 9500})
assert r.status_code == 200, r.text
(row,) = client.get("/api/admin/audit", params={"action": "config_set"}).json()["items"]
assert row["request_id"] == r.headers["x-request-id"]
def test_an_admin_action_says_how_many_of_its_requests_lines_are_still_kept(db, make_user, as_user):
"""A request's lines are kept days, its audit row months: a link to lines gone would open an
empty list, so the row says how many are left, as a run does."""
admin = make_user(role="admin")
kept = _audit(db, "config_set", actor=admin.id, request_id="req-kept-0001")
gone = _audit(db, "config_set", actor=admin.id, request_id="req-gone-0001")
db.add_all(
[
LogEvent(
ts=T0,
level="INFO",
sev=20,
source="api",
logger="t",
msg="m",
request_id="req-kept-0001",
)
for _ in range(3)
]
)
db.commit()
items = as_user(admin).get("/api/admin/audit").json()["items"]
lines = {r["id"]: r["log_lines"] for r in items}
assert (lines[kept], lines[gone]) == (3, 0)
def test_the_run_list_filters_pages_and_names_the_channel_and_the_log_lines(db, make_user, as_user):
admin = make_user(role="admin")
db.add(Channel(id="UCchan", title="Aileen Wang"))
ok1 = _run(db, "rss_poll", at=T0)
ok2 = _run(db, "rss_poll", at=T0)
bad = _run(db, "backfill", "error", admin.id, T0, trigger="manual")
adhoc = _run(
db, "channel_backfill", "ok", admin.id, T0 - timedelta(hours=1), "UCchan", "ad_hoc"
)
for _ in range(2):
db.add(
LogEvent(ts=T0, level="WARNING", sev=30, source="api", logger="t", msg="m", run_id=bad)
)
db.commit()
client = as_user(admin)
first = client.get("/api/admin/runs", params={"limit": 2}).json()
rest = client.get("/api/admin/runs", params={"limit": 2, "before": first["next_cursor"]}).json()
assert _ids(first) + _ids(rest) == [bad, ok2, ok1, adhoc]
assert rest["next_cursor"] is None
rows = {r["id"]: r for r in first["items"] + rest["items"]}
assert (rows[bad]["log_lines"], rows[ok2]["log_lines"]) == (2, 0)
assert (rows[adhoc]["target_title"], rows[ok1]["target_title"]) == ("Aileen Wang", None)
def ids(**params) -> list[int]:
return _ids(client.get("/api/admin/runs", params=params).json())
assert ids(status="error") == [bad]
assert ids(job="rss_poll") == [ok2, ok1]
assert ids(user="none") == [ok2, ok1]
assert ids(user=str(admin.id)) == [bad, adhoc]
assert ids(q="UCchan") == [adhoc]
assert ids(since=(T0 - timedelta(minutes=30)).isoformat()) == [bad, ok2, ok1]
assert client.get("/api/admin/runs", params={"status": "finished"}).status_code == 422
facets = client.get("/api/admin/runs/facets", params={"status": "error"}).json()
assert facets["statuses"] == [{"value": "ok", "count": 3}, {"value": "error", "count": 1}]
assert facets["jobs"] == [{"value": "backfill", "count": 1}]
assert facets["total"] == 4
def test_the_quota_breakdown_covers_its_range_and_one_account(db, make_user, as_user):
admin = make_user(role="admin")
now = datetime.now(timezone.utc)
db.add_all(
[
QuotaEvent(user_id=admin.id, action="search", units=100, created_at=now),
QuotaEvent(user_id=None, action="backfill", units=7, created_at=now),
QuotaEvent(
user_id=admin.id, action="search", units=50, created_at=now - timedelta(days=10)
),
]
)
db.commit()
client = as_user(admin)
def totals(**params) -> dict:
body = client.get("/api/quota/admin", params=params).json()
# The chart is the same filter's: its days add up to the table's total.
assert sum(d["total"] for d in body["daily"]) == sum(r["total"] for r in body["rows"])
return {r["user_id"]: r["total"] for r in body["rows"]}
assert totals() == {admin.id: 150, None: 7} # the last 30 days by default
assert totals(since=(now - timedelta(days=1)).isoformat()) == {admin.id: 100, None: 7}
assert totals(until=(now - timedelta(days=1)).isoformat()) == {admin.id: 50}
assert totals(user="none") == {None: 7}
assert totals(user=str(admin.id)) == {admin.id: 150}
def test_every_admin_read_and_write_is_admin_only():
"""The route table, not a list kept by hand: every /api/admin route and the quota breakdown
resolve `admin_user`, so a dropped dependency cannot hand an account's runs, errors and e-mail
addresses to any signed-in user."""
routes = _route_objects(app)
admin = {k: r for k, r in routes.items() if k[1].startswith(("/api/admin", "/api/quota/admin"))}
assert ("GET", "/api/admin/runs") in admin and ("GET", "/api/quota/admin") in admin
open_ = []
for key, route in admin.items():
stack, calls = [route.dependant], set()
while stack:
d = stack.pop()
calls.add(d.call)
stack.extend(d.dependencies)
if admin_user not in calls:
open_.append(key)
assert open_ == []
+10 -1
View File
@@ -24,6 +24,7 @@ def _row(**kw) -> dict:
"request_id": None,
"user_id": None,
"job_id": None,
"run_id": None,
"version": "0.90.0",
}
row.update(kw)
@@ -80,6 +81,7 @@ def test_newest_first_and_every_row_once_across_pages_even_on_equal_timestamps(d
({"event": "none"}, {"warn"}),
({"request_id": "req-abcdef12"}, {"mine"}),
({"job_id": 42}, {"err"}),
({"run_id": 7}, {"warn"}), # a background run's lines, not a download job's
({"q": "FORBIDDEN"}, {"err"}), # in attrs, any case
({"q": "KeyError"}, {"warn"}), # in the stack
({"q": "100%"}, {"plain"}), # a literal, not a wildcard
@@ -102,7 +104,14 @@ def test_each_filter_selects_exactly_its_rows(db, admin, make_user, params, expe
job_id=42,
attrs={"code": "forbidden"},
),
_row(msg="warn", level="WARNING", sev=13, event=None, exc="Traceback …\nKeyError: 'x'"),
_row(
msg="warn",
level="WARNING",
sev=13,
event=None,
exc="Traceback …\nKeyError: 'x'",
run_id=7,
),
_row(msg="plain", ts=T0 - timedelta(hours=2), attrs={"note": "100% done"}),
# Matched by no filter above but `until`: another account, and `100%` read as a pattern.
_row(msg="decoy", ts=T0 - timedelta(hours=3), attrs={"note": "1000 done"}, user_id=99),
+648
View File
@@ -0,0 +1,648 @@
"""API tokens for the Files library (`app.apitokens`, `routes.tokens`), DB-lane.
A token is made, listed and revoked by its signed-in user only; its value is in the answer that made
it and nowhere after. With it a machine reaches `/api/files` and nothing else, and there only the
folders its grants name, with the rights they give — every right checked in a folder it has and in
one it has not. A token that is malformed, unknown, expired, revoked, or whose account is suspended
holds nothing, and a client trying tokens is slowed down.
"""
from datetime import datetime, timedelta, timezone
import pytest
from fastapi import Depends, Request
from fastapi.testclient import TestClient
from sqlalchemy import select, update
from app import apitokens, audit
from app.apitokens import files_caller
from app.auth import current_user
from app.downloads import folders, service
from app.downloads import gc as gcmod
from app.downloads import upload as uploadmod
from app.downloads.status import JobStatus
from app.config import settings
from app.db import SessionLocal, get_db
from app.main import app
from app.notifications import create_notification
from app.models import (
ApiToken,
ApiTokenGrant,
AuditLog,
DownloadJob,
FileFolder,
MediaAsset,
Notification,
User,
)
from app.ratelimit import RateLimiter
from app.routes import tokens as tokens_routes
from app.routes.auth import bump_session_epoch
from app.security import hash_token
@pytest.fixture
def _root(tmp_path, monkeypatch):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
return tmp_path
@pytest.fixture
def _no_probe(monkeypatch):
# A file ffprobe cannot read: a generic file, done at once (no conversion).
monkeypatch.setattr(uploadmod.probe, "probe_json", lambda _p: None)
def _own(db, user_id, name):
folder = FileFolder(user_id=user_id, kind=folders.FolderKind.OWN, name=name)
db.add(folder)
db.commit()
return folder
def _file_in(db, root, user_id, folder, name):
"""A done file of `user_id`'s with bytes on disk, sitting in `folder`."""
asset = MediaAsset(
source_kind="upload",
source_ref=name,
format_sig=f"sig-{name}",
status="ready",
rel_path=f"{name}.bin",
size_bytes=4,
ref_count=0,
title=name,
media_kind="file",
container="bin",
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
source_kind="upload",
source_ref=name,
profile_snapshot={},
status=JobStatus.DONE,
)
db.add(job)
db.flush()
folders.place(db, job)
job.folder_id = folder.id
folders.stamp_done(db, job)
service.incr_ref_count(db, asset.id)
db.commit()
(root / f"{name}.bin").write_bytes(b"data")
return job
def _bearer(raw: str) -> TestClient:
return TestClient(app, headers={"authorization": f"Bearer {raw}"})
def _make(c, **body) -> dict:
r = c.post("/api/tokens", json={"name": "a machine", **body})
assert r.status_code == 200, r.text
return r.json()
def _audits(db, action):
return db.execute(select(AuditLog).where(AuditLog.action == action)).scalars().all()
# --- the token itself ----------------------------------------------------------------------------
def test_a_token_has_its_shape_and_a_checksum_that_refuses_a_typo():
raw = apitokens.mint()
assert raw.startswith("sfl_") and len(raw) == apitokens.TOKEN_LEN
assert apitokens.well_formed(raw)
assert apitokens.mint() != raw
flipped = raw[:10] + ("A" if raw[10] != "A" else "B") + raw[11:]
assert not apitokens.well_formed(flipped)
assert not apitokens.well_formed(raw[:-1])
assert not apitokens.well_formed("xxx_" + raw[4:])
# A header byte past ASCII arrives as one latin-1 character: refused, never a 500.
assert not apitokens.well_formed(raw[:-1] + "é")
def test_a_non_ascii_token_is_refused_as_any_bad_one(db, client):
# The test client re-sends a high byte as two UTF-8 bytes, which arrive as two latin-1
# characters: one "A" fewer keeps the header a token's length, so it is the checksum that meets
# them, not the length check.
raw = ("Bearer sfl_" + "A" * 47 + "é").encode("latin-1")
r = TestClient(app, headers={"authorization": raw}).get("/api/files/folders")
assert r.status_code == 401 and r.json()["detail"]["reason"] == "not_authenticated"
def test_the_hub_preset_makes_one_hub_folder_with_every_right_and_shows_the_token_once(
db, make_user, as_user
):
alice = make_user()
c = as_user(alice)
made = _make(c, preset="hub")
raw = made["token"]
assert apitokens.well_formed(raw) and made["prefix"] == raw[: apitokens.SHOWN_LEN]
[grant] = made["grants"]
assert grant["kind"] == "hub" and grant["rights"] == ["delete", "list", "read", "write"]
# Only the hash is kept, and the value is not in the list.
stored = db.execute(select(ApiToken)).scalar_one()
assert stored.token_hash != raw and raw not in str(c.get("/api/tokens").json())
# A second hub token reuses the one Hub folder.
again = _make(c, preset="hub")
assert again["grants"][0]["folder_id"] == grant["folder_id"]
hubs = db.execute(select(FileFolder).where(FileFolder.kind == "hub")).scalars().all()
assert len(hubs) == 1
assert len(_audits(db, "api_token_create")) == 2
def test_a_token_is_granted_only_the_users_own_folders_and_writes_only_where_uploads_land(
db, make_user, as_user
):
alice, bob = make_user(), make_user()
theirs = _own(db, bob.id, "Bob's")
c = as_user(alice)
r = c.post(
"/api/tokens",
json={"name": "x", "grants": [{"folder_id": theirs.id, "rights": ["read"]}]},
)
assert r.status_code == 409 and r.json()["detail"]["reason"] == "file_folder_gone"
youtube = folders.source_folder(db, alice.id, folders.SourceKey.YOUTUBE)
db.commit()
r = c.post(
"/api/tokens",
json={"name": "x", "grants": [{"folder_id": youtube.id, "rights": ["write"]}]},
)
assert r.json()["detail"]["reason"] == "file_folder_not_its_source"
assert c.post("/api/tokens", json={"name": "x"}).json()["detail"]["reason"] == (
"api_token_no_grants"
)
assert db.execute(select(ApiToken)).all() == []
@pytest.mark.parametrize("right", ["read", "write"])
def test_a_folder_granted_is_held_until_the_token_is_in(db, make_user, right):
"""A delete of the folder at the same moment waits and takes the new grant with it, rather than
failing the grant's foreign key."""
import threading
alice = make_user()
fid = _own(db, alice.id, "Mine").id
body = tokens_routes.TokenIn(
name="x", grants=[tokens_routes.GrantIn(folder_id=fid, rights=[right])]
)
tokens_routes._wanted_grants(db, alice, body)
outcome: list[object] = []
def delete_it():
other = SessionLocal()
try:
outcome.append(folders.delete_own(other, other.get(FileFolder, fid)))
other.commit()
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=delete_it)
t.start()
t.join(timeout=1)
assert t.is_alive() # the delete waits for the token on its way in
db.commit()
t.join(timeout=10)
assert not isinstance(outcome[0], Exception), outcome
def test_a_user_has_at_most_the_ceiling_of_tokens(db, make_user, as_user, monkeypatch):
monkeypatch.setattr(tokens_routes, "MAX_TOKENS", 2)
c = as_user(make_user())
_make(c, preset="hub")
_make(c, preset="hub")
r = c.post("/api/tokens", json={"name": "third", "preset": "hub"})
assert r.status_code == 409 and r.json()["detail"]["reason"] == "api_token_limit"
# --- what a token reaches ------------------------------------------------------------------------
@pytest.mark.parametrize("right", ["list", "read", "write", "delete"])
def test_each_right_holds_in_its_folder_only(
db, make_user, as_user, _root, _no_probe, upload_headers, right
):
"""One right in folder A; folder B has none. Every Files route a token may use, in both."""
alice = make_user()
a, b = _own(db, alice.id, "A"), _own(db, alice.id, "B")
in_a, in_b = _file_in(db, _root, alice.id, a, "fa"), _file_in(db, _root, alice.id, b, "fb")
raw = _make(as_user(alice), grants=[{"folder_id": a.id, "rights": [right]}])["token"]
m = _bearer(raw)
folders_seen = [f["id"] for f in m.get("/api/files/folders").json()]
assert folders_seen == ([a.id] if right == "list" else [])
listed = m.get("/api/files", params={"folder_id": a.id})
assert listed.status_code == (200 if right == "list" else 404)
if right == "list":
assert [f["id"] for f in listed.json()] == [in_a.id]
assert [f["id"] for f in m.get("/api/files").json()] == [in_a.id]
else:
assert m.get("/api/files").json() == []
assert m.get("/api/files", params={"folder_id": b.id}).status_code == 404
read_a = m.get(f"/api/files/{in_a.id}/file")
assert read_a.status_code == (200 if right == "read" else 404)
assert m.get(f"/api/files/{in_a.id}/sha256").status_code == (200 if right == "read" else 404)
assert m.get(f"/api/files/{in_b.id}/file").status_code == 404
up_a = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt", folder=a.id))
assert up_a.status_code == (200 if right == "write" else 404)
if right == "write":
assert up_a.json()["folder_id"] == a.id
up_b = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt", folder=b.id))
assert up_b.status_code == 404
no_folder = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt"))
assert no_folder.status_code == 400
assert no_folder.json()["detail"]["reason"] == "api_token_folder_required"
assert m.delete(f"/api/files/{in_b.id}").status_code == 404
assert db.get(DownloadJob, in_b.id) is not None
del_a = m.delete(f"/api/files/{in_a.id}")
assert del_a.status_code == (200 if right == "delete" else 404)
db.expire_all()
assert (db.get(DownloadJob, in_a.id) is None) == (right == "delete")
def test_a_token_never_reaches_another_users_files_or_the_shared_ones(
db, make_user, as_user, _root
):
alice, bob = make_user(), make_user()
hub_raw = _make(as_user(alice), preset="hub")["token"]
bobs = _file_in(db, _root, bob.id, _own(db, bob.id, "B"), "bf")
m = _bearer(hub_raw)
assert m.get(f"/api/files/{bobs.id}/file").status_code == 404
assert m.delete(f"/api/files/{bobs.id}").status_code == 404
shared = m.get("/api/files", params={"scope": "shared"})
assert shared.status_code == 403 and shared.json()["detail"]["reason"] == "api_token_scope"
def test_a_token_is_judged_by_its_grants_even_beside_a_signed_in_cookie(db, make_user, client):
"""Through the real session stack (`client`, a real sign-in): the `as_user` fixture stands in
for the very dependency this is about."""
alice = make_user(email="alice@example.com")
a = _own(db, alice.id, "A")
c = client
signed_in = c.post(
"/auth/password-login",
json={"email": "alice@example.com", "password": "correct horse battery"},
)
assert signed_in.status_code == 200
raw = _make(c, grants=[{"folder_id": a.id, "rights": ["read"]}])["token"]
assert len(c.get("/api/files/folders").json()) == 1 # the user sees every folder
both = c.get("/api/files/folders", headers={"authorization": f"Bearer {raw}"})
assert both.json() == [] # the token may not list: its grants decide
def _signed_in(client, email="alice@example.com") -> TestClient:
r = client.post(
"/auth/password-login", json={"email": email, "password": "correct horse battery"}
)
assert r.status_code == 200, r.text
return client
def test_a_token_works_on_the_files_routes_only(db, make_user, client):
"""Through the real session stack: under `as_user` every route without its header answers 401
whatever `current_user` does with a token, so these refusals would prove nothing there."""
make_user(email="alice@example.com")
raw = _make(_signed_in(client), preset="hub")["token"]
m = _bearer(raw)
assert m.get("/api/files/folders").status_code == 200
# Making a wider token, or reading the account, takes the user's session.
assert m.get("/api/me").json() == {"authenticated": False} # the token is no one here
assert m.get("/api/tokens").status_code == 401
assert m.post("/api/tokens", json={"name": "wider", "preset": "hub"}).status_code == 401
assert m.get("/api/files/usage").status_code == 401
assert m.patch("/api/files/1", json={"name": "x"}).status_code == 401
def test_only_the_files_routes_take_a_token():
"""The route table, not a list kept by hand: every route that resolves `files_caller` is under
/api/files, and these are all of them. The walk recurses into included routers (see
`test_body_limit._declared_routes` for why `app.routes` alone sees none of ours)."""
takers, nodes, routes = set(), [app], []
while nodes:
for route in getattr(nodes.pop(), "routes", ()) or ():
inner = getattr(route, "original_router", None)
if inner is not None:
nodes.append(inner)
else:
routes.append(route)
assert len(routes) > 100 # the walk found the app's routes, not just the shell's
for route in routes:
dependant = getattr(route, "dependant", None)
if dependant is None:
continue
stack, calls = [dependant], set()
while stack:
d = stack.pop()
calls.add(d.call)
stack.extend(d.dependencies)
if files_caller in calls:
for method in route.methods:
takers.add((method, route.path))
assert all(path.startswith("/api/files") for _, path in takers)
assert takers == {
("GET", "/api/files/folders"),
("GET", "/api/files"),
("GET", "/api/files/{job_id}/file"),
("GET", "/api/files/{job_id}/sha256"),
("POST", "/api/files/upload"),
("DELETE", "/api/files/{job_id}"),
}
def _reset_password(db, client, alice):
from app.routes.auth import _issue_token
link = _issue_token(db, alice, "reset", timedelta(hours=1))
r = client.post(
"/auth/password-reset/confirm", json={"token": link, "password": "a brand new password"}
)
assert r.status_code == 200, r.text
def _change_password(db, client, alice):
r = client.post(
"/auth/set-password",
json={"password": "a brand new password", "current_password": "correct horse battery"},
)
assert r.status_code == 200, r.text
def _log_out_others(db, client, alice):
assert client.post("/auth/logout-others").status_code == 200
def _revoke_untrusted_password(db, client, alice):
from app.routes.auth import _revoke_password
_revoke_password(db, alice)
db.commit()
@pytest.mark.parametrize(
"end_sessions",
[_reset_password, _change_password, _log_out_others, _revoke_untrusted_password],
)
def test_whatever_ends_every_session_ends_every_token_too(db, make_user, client, end_sessions):
"""A token minted from a stolen session must not outlive the owner's answer to the theft."""
alice = make_user(email="alice@example.com")
bob = make_user(email="bob@example.com")
raw = _make(_signed_in(client), preset="hub")["token"]
bobs = apitokens.mint()
db.add(ApiToken(user_id=bob.id, name="b", prefix=bobs[:12], token_hash=hash_token(bobs)))
db.commit()
assert _bearer(raw).get("/api/files/folders").status_code == 200
end_sessions(db, client, alice)
assert _bearer(raw).get("/api/files/folders").status_code == 401
assert db.execute(select(ApiToken.user_id)).scalars().all() == [bob.id] # only hers went
[revoked] = _audits(db, "api_token_revoke")
assert revoked.actor_id == alice.id and "every session" in (revoked.summary or "")
def test_a_token_asked_for_as_every_session_ends_is_refused(db, make_user, client, override_dep):
"""The session was still good when its request came in, and ended before the token was made."""
alice = make_user(email="alice@example.com")
c = _signed_in(client)
def ended_meanwhile(request: Request, session=Depends(get_db)):
user = current_user(request, session)
other = SessionLocal()
try:
bump_session_epoch(other, other.get(User, alice.id))
other.commit()
finally:
other.close()
return user
override_dep(current_user, ended_meanwhile)
r = c.post("/api/tokens", json={"name": "late", "preset": "hub"})
assert r.status_code == 401
assert db.execute(select(ApiToken.id)).scalars().all() == []
def _meanwhile(work):
"""Run `work(other_session)` in another thread and session, committed; (thread, outcome)."""
import threading
outcome: list[object] = []
def run():
other = SessionLocal()
try:
outcome.append(work(other))
other.commit()
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=run)
t.start()
return t, outcome
def test_ending_every_session_never_deadlocks_with_a_tokens_expiry_notice(db, make_user):
"""The notice's sweep holds the token's row and then writes a row naming the user; the sessions'
end holds the user and then deletes the token. A lock on the user that a mere reference to it
waits on would make that a deadlock — one of the two failing."""
alice = make_user()
raw = apitokens.mint()
soon = datetime.now(timezone.utc) + timedelta(days=1)
tok = ApiToken(
user_id=alice.id, name="t", prefix=raw[:12], token_hash=hash_token(raw), expires_at=soon
)
db.add(tok)
db.commit()
# `warn_expiring`, its two halves held apart: the token row taken…
db.execute(update(ApiToken).where(ApiToken.id == tok.id).values(expiry_warned=True))
t, outcome = _meanwhile(lambda other: bump_session_epoch(other, other.get(User, alice.id)))
t.join(timeout=1)
assert t.is_alive() # the sessions' end waits for the token row
# …and then its notice, naming the user.
create_notification(db, alice.id, "api_token_expiring", title="t", commit=False)
db.commit()
t.join(timeout=10)
assert outcome == [None]
db.expire_all()
assert db.execute(select(ApiToken.id)).scalars().all() == []
def test_a_token_granted_a_folder_being_deleted_never_deadlocks(db, make_user):
"""The delete holds the folder and then writes its audit line naming the user; the token holds
the user and then waits for the folder."""
alice = make_user()
fid = _own(db, alice.id, "Mine").id
folders.delete_own(db, db.get(FileFolder, fid)) # the folder held, as the delete route does
body = tokens_routes.TokenIn(
name="x", grants=[tokens_routes.GrantIn(folder_id=fid, rights=["read"])]
)
t, outcome = _meanwhile(
lambda other: tokens_routes.create_token(body, other.get(User, alice.id), other)
)
t.join(timeout=1)
assert t.is_alive() # the token waits for the folder
audit.record(db, alice.id, "file_folder_delete", target_type="file_folder", target_id=fid)
db.commit()
t.join(timeout=10)
[refusal] = outcome
assert getattr(refusal, "status_code", None) == 409, refusal # the folder went first
def test_a_token_made_as_every_session_ends_goes_with_them(db, make_user):
"""The other order: the token is on its way in when the sessions end."""
import threading
alice = make_user(email="alice@example.com")
db.execute(select(User.id).where(User.id == alice.id).with_for_update()) # as create_token
raw = apitokens.mint()
db.add(ApiToken(user_id=alice.id, name="t", prefix=raw[:12], token_hash=hash_token(raw)))
db.flush()
outcome: list[object] = []
def end_sessions():
other = SessionLocal()
try:
bump_session_epoch(other, other.get(User, alice.id))
other.commit()
outcome.append("done")
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=end_sessions)
t.start()
t.join(timeout=1)
assert t.is_alive() # it waits for the token being made
db.commit()
t.join(timeout=10)
assert outcome == ["done"]
db.expire_all()
assert db.execute(select(ApiToken.id)).scalars().all() == []
# --- a token that does not hold ------------------------------------------------------------------
def _refused(m: TestClient) -> int:
return m.get("/api/files/folders").status_code
def test_a_malformed_unknown_expired_revoked_or_suspended_token_holds_nothing(
db, make_user, as_user
):
alice = make_user()
c = as_user(alice)
raw = _make(c, preset="hub")["token"]
assert _refused(_bearer(raw)) == 200
assert _refused(_bearer("sfl_short")) == 401
assert _refused(_bearer(apitokens.mint())) == 401 # well formed, never made
expiring = _make(c, preset="hub", expires_days=1)
token = db.get(ApiToken, expiring["id"])
token.expires_at = datetime.now(timezone.utc) - timedelta(seconds=1)
db.commit()
assert _refused(_bearer(expiring["token"])) == 401
assert c.get("/api/tokens").json()[1]["expired"] is True
assert c.delete(f"/api/tokens/{expiring['id']}").json() == {
"ok": True,
"deleted": expiring["id"],
}
assert (
db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == expiring["id"])).all()
== []
)
assert len(_audits(db, "api_token_revoke")) == 1
revoked = _make(c, preset="hub")
assert c.delete(f"/api/tokens/{revoked['id']}").status_code == 200
assert _refused(_bearer(revoked["token"])) == 401
alice.is_suspended = True
db.commit()
assert _refused(_bearer(raw)) == 401
def test_a_token_about_to_run_out_warns_its_user_once_and_is_marked_in_the_list(
db, make_user, as_user
):
alice, bob = make_user(), make_user()
c = as_user(alice)
soon = _make(c, preset="hub", expires_days=3)
_make(c, preset="hub", expires_days=30) # outside the span
_make(c, preset="hub") # never runs out
gone = _make(c, preset="hub", expires_days=1)
db.get(ApiToken, gone["id"]).expires_at = datetime.now(timezone.utc) - timedelta(seconds=1)
_make(as_user(bob), preset="hub", expires_days=2)
db.commit()
assert [t["expiring"] for t in c.get("/api/tokens").json()] == [True, False, False, False]
assert apitokens.warn_expiring(db) == 2 # alice's and bob's; the run-out one is not "soon"
notes = db.execute(select(Notification).where(Notification.user_id == alice.id)).scalars()
[note] = notes.all()
assert note.type == "api_token_expiring"
assert note.data["token_id"] == soon["id"] and note.data["name"] == "a machine"
assert note.data["expires_at"] == soon["expires_at"]
assert apitokens.warn_expiring(db) == 0 # once per token
def test_the_files_gc_runs_the_token_warning(db, make_user, as_user, _root):
_make(as_user(make_user()), preset="hub", expires_days=3)
assert gcmod.run_download_gc(db)["tokens_warned"] == 1
def test_another_users_token_cannot_be_revoked(db, make_user, as_user):
alice, bob = make_user(), make_user()
made = _make(as_user(alice), preset="hub")
assert as_user(bob).delete(f"/api/tokens/{made['id']}").status_code == 404
assert db.get(ApiToken, made["id"]) is not None
def test_a_client_trying_tokens_is_slowed_down(db, make_user, as_user, monkeypatch):
alice = make_user()
raw = _make(as_user(alice), preset="hub")["token"]
monkeypatch.setattr(apitokens, "_bad", RateLimiter(3, 300))
for _ in range(3):
assert _refused(_bearer(apitokens.mint())) == 401
slowed = _bearer(apitokens.mint()).get("/api/files/folders")
assert slowed.status_code == 429
assert slowed.json()["detail"]["reason"] == "api_token_rate_limited"
# Even a good token from that client waits out the window: a guesser cannot tell a hit.
assert _refused(_bearer(raw)) == 429
def test_a_runaway_machine_is_held_back_per_token(db, make_user, as_user, monkeypatch):
c = as_user(make_user())
first, second = _make(c, preset="hub")["token"], _make(c, preset="hub")["token"]
monkeypatch.setattr(apitokens, "_calls", RateLimiter(2, 60))
assert [_refused(_bearer(first)) for _ in range(3)] == [200, 200, 429]
assert _refused(_bearer(second)) == 200
# --- what a token leaves behind ------------------------------------------------------------------
def test_a_tokens_use_is_stamped_and_its_uploads_and_deletes_name_it_in_the_audit(
db, make_user, as_user, _root, _no_probe, upload_headers
):
alice = make_user()
made = _make(as_user(alice), preset="hub")
hub = made["grants"][0]["folder_id"]
assert db.get(ApiToken, made["id"]).last_used_at is None
m = _bearer(made["token"])
up = m.post("/api/files/upload", content=b"hi", headers=upload_headers("h.txt", folder=hub))
assert up.status_code == 200
db.expire_all()
assert db.get(ApiToken, made["id"]).last_used_at is not None
assert m.delete(f"/api/files/{up.json()['id']}").status_code == 200
via = f"api token {made['prefix']}…"
assert [a.summary for a in _audits(db, "file_upload")] == [via]
assert [a.summary for a in _audits(db, "file_delete")] == [via]
+1 -1
View File
@@ -1,6 +1,6 @@
"""An admin action that counts something keeps the count as data in its audit row's `after`, under
the key the audit log names in the reader's language (`scheduler.result.<key>`, frontend
auditColumns.tsx) — not only in the English `summary`. A renamed key would fall back to the raw word.
lib/jobResult.ts, called from lib/auditText.ts) — not only in the English `summary`. A renamed key would fall back to the raw word.
DB lane (Postgres).
"""
+1 -1
View File
@@ -493,7 +493,7 @@ def _revoke_as_google_adoption(db, user) -> None:
Google callback adopting an account whose email was never verified."""
from app.routes.auth import revoke_untrusted_preset_password
revoke_untrusted_preset_password(user)
revoke_untrusted_preset_password(db, user)
db.commit()
+64 -7
View File
@@ -107,25 +107,48 @@ def _setup_channel_and_sub(db, *, email):
def _patch_deep_requested(db, monkeypatch, override_dep, user):
submitted: list[tuple] = []
calls: list[tuple] = []
# background.add_task runs after the response in TestClient; record what would be scheduled.
# Record the run the route submits, and what its work would call, instead of running it.
monkeypatch.setattr(
channels_route.scheduler,
"run_ad_hoc",
lambda job, fn, **kw: submitted.append((job, kw)) or fn(),
)
monkeypatch.setattr(channels_route, "backfill_channel_now", lambda *a, **k: calls.append(a))
monkeypatch.setattr(state, "_configured_cache", True)
override_dep(get_db, db.info["request"])
override_dep(current_user, lambda: user)
resp = TestClient(app).patch("/api/channels/UC_wire", json={"deep_requested": True})
return resp, calls
return resp, submitted, calls
def test_deep_request_schedules_immediate_backfill(db, monkeypatch, override_dep):
def test_deep_request_starts_the_backfill_as_the_users_run_of_that_channel(
db, monkeypatch, override_dep
):
"""It was a BackgroundTask: what it did, spent or failed with reached only the log."""
ch, me = _setup_channel_and_sub(db, email="wire@example.com")
resp, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
resp, submitted, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
assert resp.status_code == 200, resp.text
assert resp.json()["deep_requested"] is True
# scheduled exactly once, with (channel_id, user_id) — the args backfill_channel_now needs.
assert submitted == [("channel_backfill", {"actor_id": me.id, "target": ch.id})]
# its work runs exactly once, with (channel_id, user_id) — the args backfill_channel_now needs.
assert calls == [(ch.id, me.id)]
def test_turning_full_history_on_for_a_finished_channel_records_no_run(
db, monkeypatch, override_dep
):
"""Its whole history is in already: a run would only say "nothing to do", and switching the flag
off and on again would otherwise write one such row each time, kept for months."""
ch, me = _setup_channel_and_sub(db, email="done@example.com")
ch.backfill_done = True
db.commit()
resp, submitted, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
assert resp.status_code == 200, resp.text
assert (submitted, calls) == ([], [])
class TestImmediateBackfillGuard:
"""backfill_channel_now coordinates with the paced scheduler through _backfill_in_progress (per-
channel mutual exclusion, both directions) + _immediate_running (a cap on interactive runs so a
@@ -148,15 +171,49 @@ class TestImmediateBackfillGuard:
def test_a_second_call_for_an_in_flight_channel_is_a_noop(self, monkeypatch):
runner._backfill_in_progress.add("UC_busy")
monkeypatch.setattr(runner, "SessionLocal", self._boom_session)
runner.backfill_channel_now("UC_busy", 1) # returns immediately; _boom_session never runs
# returns at once — _boom_session never runs — and the run says why it did nothing.
assert runner.backfill_channel_now("UC_busy", 1) == {"skipped": "channel busy"}
assert runner._backfill_in_progress == {"UC_busy"} # the in-flight entry is untouched
def test_over_the_concurrency_cap_falls_back_to_the_scheduler(self, monkeypatch):
runner._immediate_running = runner._MAX_CONCURRENT_IMMEDIATE
monkeypatch.setattr(runner, "SessionLocal", self._boom_session)
runner.backfill_channel_now("UC_new", 1) # over the cap → no-op, scheduler will handle it
# over the cap → no-op, the scheduler will handle it
assert runner.backfill_channel_now("UC_new", 1) == {"skipped": "at capacity"}
assert "UC_new" not in runner._backfill_in_progress
@staticmethod
def _one_channel(db, monkeypatch, cid):
ch = Channel(id=cid, title=cid, uploads_playlist_id="UU" + cid, backfill_done=False)
me = User(email=f"{cid}@example.com")
db.add_all([ch, me])
db.commit()
monkeypatch.setattr(runner, "get_service_user", lambda _db: me)
monkeypatch.setattr(runner.quota, "backfill_reserve_reached", lambda: False)
monkeypatch.setattr(runner, "YouTubeClient", lambda *a, **k: _NullYt())
return ch, me
def test_a_failure_reaches_its_run_and_still_frees_the_channel(self, db, monkeypatch):
"""The failure was logged and swallowed, so the run that now wraps it would read "ok"."""
ch, me = self._one_channel(db, monkeypatch, "UC_fail")
def boom(*_a):
raise RuntimeError("playlistItems 500")
monkeypatch.setattr(runner, "backfill_channel_deep", boom)
with pytest.raises(RuntimeError, match="playlistItems 500"):
runner.backfill_channel_now(ch.id, me.id)
assert runner._backfill_in_progress == set() and runner._immediate_running == 0
def test_a_finished_backfill_says_what_it_added(self, db, monkeypatch):
ch, me = self._one_channel(db, monkeypatch, "UC_ok")
monkeypatch.setattr(runner, "backfill_channel_deep", lambda *_a: 7)
monkeypatch.setattr(runner, "reconcile_full_history", lambda _db: 1)
assert runner.backfill_channel_now(ch.id, me.id) == {
"videos_added": 7,
"reconciled_full": 1,
}
def test_scheduler_deep_pass_skips_an_in_flight_channel(self, db, monkeypatch):
# A channel the scheduler's query WOULD pick (deep-requested, recent-synced, not done)…
ch = Channel(
+7 -1
View File
@@ -15,6 +15,7 @@ import pytest
from fastapi import HTTPException
from sqlalchemy import select
from app.apitokens import FilesCaller
from app.db import SessionLocal
from app.downloads import service
from app.models import DownloadJob, MediaAsset, User
@@ -50,11 +51,16 @@ def _asset_held_by_two_jobs(db, status="error"):
return mine, asset, jobs[0]
def delete_file_as(job_id, *, user, db):
"""`delete_file` as the user's own request (it takes the Files caller, not the user)."""
return delete_file(job_id, caller=FilesCaller(user=user), db=db)
@pytest.mark.parametrize(
("status", "transition", "route", "left"),
[
("error", service.cancel, cancel_download, "canceled"),
("error", service.delete, delete_file, None),
("error", service.delete, delete_file_as, None),
("queued", service.cancel, pause_download, "canceled"),
],
)
+397
View File
@@ -0,0 +1,397 @@
"""A `/watch` link to one of its owner's own folders (routes/downloads.py, routes/public.py), DB-lane.
The link shows what is in the folder when it is opened: a file moved in appears, one moved out stops
opening at its next request. Only an own folder can be shared — a source folder lets its files go on
its own clock, the Hub is the channel to a machine. A password, an expiry and stream-only work as on
a file link; a document is never served to open as a page.
"""
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
from alembic import command
from alembic.config import Config
from sqlalchemy import select, text
from app.config import settings
from app.db import engine
from app.downloads import folders, og
from app.downloads import probe as probemod
from app.models import AuditLog, DownloadJob, DownloadLink, FileFolder, MediaAsset
_VIDEO = {
"streams": [
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
{"codec_type": "audio", "codec_name": "aac"},
],
"format": {"format_name": "mov,mp4", "duration": "5"},
}
@pytest.fixture
def probe(monkeypatch):
"""What ffprobe says the next upload is: a phone-safe video, or (None) a file it cannot read."""
said = {"data": _VIDEO}
monkeypatch.setattr(probemod, "probe_json", lambda _p: said["data"])
return said
@pytest.fixture
def shelf(as_user, make_user, tmp_path, monkeypatch, upload_headers, probe):
"""An owner with an own folder "Friends" holding two videos, a third video in their Uploads, and
a function to upload more."""
monkeypatch.setattr(settings, "files_root", str(tmp_path))
user = make_user()
owner = as_user(user)
folder = owner.post("/api/files/folders", json={"name": "Friends"}).json()
def upload(name: str, folder_id: int | None = None, body: bytes = b"fake-video") -> int:
r = owner.post(
"/api/files/upload", content=body, headers=upload_headers(name, folder=folder_id)
)
assert r.status_code == 200, r.text
return r.json()["id"]
inside = [upload("a.mp4", folder["id"]), upload("b.mp4", folder["id"])]
outside = upload("c.mp4")
return {
"user": user,
"owner": owner,
"folder": folder["id"],
"inside": inside,
"outside": outside,
"upload": upload,
}
def _link(shelf, **body) -> dict:
r = shelf["owner"].post(f"/api/downloads/folders/{shelf['folder']}/links", json=body)
assert r.status_code == 200, r.text
return r.json()
def _ids(client, token: str) -> list[int]:
meta = client.get(f"/api/public/watch/{token}").json()
return [f["id"] for f in meta["files"]]
def test_a_folder_link_lists_what_is_in_the_folder(shelf, client):
token = _link(shelf)["token"]
meta = client.get(f"/api/public/watch/{token}").json()
assert meta["kind"] == "folder"
assert meta["name"] == "Friends"
assert sorted(f["id"] for f in meta["files"]) == sorted(shelf["inside"])
# Each one plays and downloads through the link, under its own id.
for f in meta["files"]:
assert f["file_url"] == f"/api/public/watch/{token}/files/{f['id']}/file"
served = client.get(f["file_url"])
assert served.status_code == 200
assert served.content == b"fake-video"
def test_a_file_outside_the_folder_does_not_open_through_it(shelf, client):
token = _link(shelf)["token"]
r = client.get(f"/api/public/watch/{token}/files/{shelf['outside']}/file")
assert r.status_code == 404
# Nor does the folder link open as a file link.
assert client.get(f"/api/public/watch/{token}/file").status_code == 404
def test_the_link_follows_the_folder_as_files_move(shelf, client):
token = _link(shelf)["token"]
owner, moved_out = shelf["owner"], shelf["inside"][0]
other = owner.post("/api/files/folders", json={"name": "Other"}).json()["id"]
def move(job_id: int, folder_id: int) -> None:
r = owner.patch(f"/api/files/{job_id}", json={"folder_id": folder_id})
assert r.status_code == 200, r.text
move(shelf["outside"], shelf["folder"])
move(moved_out, other)
assert sorted(_ids(client, token)) == sorted([shelf["inside"][1], shelf["outside"]])
assert client.get(f"/api/public/watch/{token}/files/{moved_out}/file").status_code == 404
url = f"/api/public/watch/{token}/files/{shelf['outside']}/file"
assert client.get(url).status_code == 200
def test_only_a_folder_of_ones_own_can_be_shared(shelf, db, as_user, make_user):
owner = shelf["owner"]
source = db.execute(
select(FileFolder.id).where(
FileFolder.user_id == shelf["user"].id, FileFolder.kind == "source"
)
).scalar_one()
r = owner.post(f"/api/downloads/folders/{source}/links", json={})
assert r.status_code == 409
assert r.json()["detail"]["reason"] == "share_folder_kind"
# Nor the Hub: whatever a machine drops there would be public the moment it lands.
hub = folders.hub_folder(db, shelf["user"].id).id
db.commit()
r = owner.post(f"/api/downloads/folders/{hub}/links", json={})
assert r.status_code == 409
assert r.json()["detail"]["reason"] == "share_folder_kind"
stranger = as_user(make_user())
assert (
stranger.post(f"/api/downloads/folders/{shelf['folder']}/links", json={}).status_code == 404
)
assert stranger.get(f"/api/downloads/folders/{shelf['folder']}/links").status_code == 404
link_id = _link(shelf)["id"]
patch = stranger.patch(f"/api/downloads/links/{link_id}", json={"allow_download": False})
assert patch.status_code == 404
assert stranger.delete(f"/api/downloads/links/{link_id}").status_code == 404
def test_a_password_folder_link_shows_nothing_until_unlocked(shelf, client):
token = _link(shelf, password="open sesame")["token"]
assert client.get(f"/api/public/watch/{token}").json() == {
"needs_password": True,
"kind": "folder",
}
bare = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/file"
assert client.get(bare).status_code == 403
unlocked = client.post(f"/api/public/watch/{token}/unlock", json={"password": "open sesame"})
assert unlocked.status_code == 200
files = unlocked.json()["files"]
assert len(files) == 2
assert all("?g=" in f["file_url"] for f in files)
assert client.get(files[0]["file_url"]).status_code == 200
def test_a_files_poster_opens_by_the_links_own_rules(shelf, db, client):
"""The poster route on its own: a password link's poster needs the unlock grant, and a file
outside the folder has none through it, grant or not."""
for job_id in (shelf["inside"][0], shelf["outside"]):
asset = db.get(MediaAsset, db.get(DownloadJob, job_id).asset_id)
asset.poster_path = f"poster-{job_id}.jpg"
(Path(settings.files_root) / asset.poster_path).write_bytes(b"a poster frame")
db.commit()
token = _link(shelf, password="open sesame")["token"]
inside = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/poster.jpg"
assert client.get(inside).status_code == 403
unlocked = client.post(f"/api/public/watch/{token}/unlock", json={"password": "open sesame"})
[entry] = [f for f in unlocked.json()["files"] if f["id"] == shelf["inside"][0]]
assert entry["poster_url"].startswith(inside + "?g=")
got = client.get(entry["poster_url"])
assert got.status_code == 200 and got.content == b"a poster frame"
grant = entry["poster_url"].split("?g=", 1)[1]
outside = f"/api/public/watch/{token}/files/{shelf['outside']}/poster.jpg?g={grant}"
assert client.get(outside).status_code == 404
def test_every_open_counts_as_a_view(shelf, client):
made = _link(shelf)
for _ in range(3):
assert client.get(f"/api/public/watch/{made['token']}").status_code == 200
[listed] = shelf["owner"].get(f"/api/downloads/folders/{shelf['folder']}/links").json()
assert listed["view_count"] == 3
def test_the_folder_list_is_capped_like_the_owners_files_list():
from app.downloads import links as linksmod
from app.routes.files import MAX_LIST
assert linksmod.FOLDER_MAX == MAX_LIST
def test_a_document_in_a_shared_folder_only_downloads(shelf, client, probe):
probe["data"] = None
doc = shelf["upload"]("notes.html", shelf["folder"], b"<script>1</script>")
token = _link(shelf, allow_download=False)["token"]
assert doc in _ids(client, token)
served = client.get(f"/api/public/watch/{token}/files/{doc}/file")
assert served.status_code == 200
assert served.headers["content-type"].startswith("application/octet-stream")
assert "attachment" in served.headers["content-disposition"].lower()
def test_a_stream_only_folder_link_plays_inline(shelf, client):
token = _link(shelf, allow_download=False)["token"]
meta = client.get(f"/api/public/watch/{token}").json()
assert meta["allow_download"] is False
served = client.get(meta["files"][0]["file_url"])
assert "inline" in served.headers["content-disposition"].lower()
def test_an_expired_or_deleted_folder_link_is_gone(shelf, client, db):
made = _link(shelf)
db.get(DownloadLink, made["id"]).expires_at = datetime.now(timezone.utc) - timedelta(minutes=1)
db.commit()
assert client.get(f"/api/public/watch/{made['token']}").status_code == 404
token = _link(shelf)["token"]
assert shelf["owner"].delete(f"/api/files/folders/{shelf['folder']}").status_code == 200
assert client.get(f"/api/public/watch/{token}").status_code == 404
db.expire_all()
assert db.execute(select(DownloadLink)).scalars().all() == []
def test_the_folder_list_marks_a_folder_a_live_link_opens(shelf, db):
owner = shelf["owner"]
def marked() -> dict[int, bool]:
return {f["id"]: f["has_link"] for f in owner.get("/api/files/folders").json()}
assert marked()[shelf["folder"]] is False
made = _link(shelf)
assert marked()[shelf["folder"]] is True
# A rename answers with the mark too, so the rail does not lose it.
renamed = owner.patch(f"/api/files/folders/{shelf['folder']}", json={"name": "Pals"}).json()
assert renamed["has_link"] is True
db.get(DownloadLink, made["id"]).expires_at = datetime.now(timezone.utc) - timedelta(minutes=1)
db.commit()
assert marked()[shelf["folder"]] is False
live = _link(shelf)
assert owner.delete(f"/api/downloads/links/{live['id']}").status_code == 200
assert marked()[shelf["folder"]] is False
assert not any(marked()[f] for f in marked() if f != shelf["folder"])
def test_making_and_revoking_a_folder_link_is_audited(shelf, db):
made = _link(shelf)
assert shelf["owner"].delete(f"/api/downloads/links/{made['id']}").status_code == 200
db.expire_all()
rows = {
a.action: a.after
for a in db.execute(
select(AuditLog).where(AuditLog.action.like("download_link_%"))
).scalars()
}
assert rows["download_link_create"]["folder_id"] == shelf["folder"]
assert "job_id" not in rows["download_link_create"]
assert rows["download_link_revoke"] == {"folder_id": shelf["folder"]}
def test_the_unfurl_names_the_folder_and_counts_its_files_only(shelf, db):
token = _link(shelf)["token"]
tags = og._watch_tags(token, db)
assert "Friends" in tags
assert "2 files" in tags
names = db.execute(
select(DownloadJob.display_name).where(DownloadJob.id.in_(shelf["inside"]))
).scalars()
for name in names:
assert not name or name not in tags
# A file whose bytes are gone is off the page, and so off the card's count.
gone = db.get(MediaAsset, db.get(DownloadJob, shelf["inside"][0]).asset_id)
(Path(settings.files_root) / gone.rel_path).unlink()
assert "1 file " in og._watch_tags(token, db)
locked = _link(shelf, password="pw")["token"]
hidden = og._watch_tags(locked, db)
assert "Friends" not in hidden and "Password-protected folder" in hidden
assert 'content="website"' in hidden and "video.other" not in hidden
def test_the_downgrade_drops_folder_links_and_keeps_file_links(shelf, db, client):
_link(shelf)
file_link = shelf["owner"].post(f"/api/downloads/{shelf['outside']}/links", json={}).json()
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
cfg = Config("alembic.ini")
command.downgrade(cfg, "0107_link_token_at_rest")
try:
with engine.connect() as conn:
left = conn.execute(text("SELECT job_id FROM download_links")).scalars().all()
finally:
command.upgrade(cfg, "head")
assert left == [shelf["outside"]]
assert client.get(f"/api/public/watch/{file_link['token']}").status_code == 200
def test_a_password_file_link_says_it_opens_a_file(shelf, client):
made = (
shelf["owner"]
.post(f"/api/downloads/{shelf['outside']}/links", json={"password": "pw"})
.json()
)
meta = client.get(f"/api/public/watch/{made['token']}").json()
assert meta == {"needs_password": True, "kind": "file"}
def test_a_file_whose_bytes_are_gone_is_not_offered(shelf, client, db):
from pathlib import Path
from app.models import MediaAsset
token = _link(shelf)["token"]
gone = shelf["inside"][0]
asset = db.get(MediaAsset, db.get(DownloadJob, gone).asset_id)
(Path(settings.files_root) / asset.rel_path).unlink()
assert _ids(client, token) == [shelf["inside"][1]]
assert client.get(f"/api/public/watch/{token}/files/{gone}/file").status_code == 404
def _meanwhile(work):
"""Run `work(other_session)` in another thread and session, committed; (thread, outcome)."""
import threading
from app.db import SessionLocal
outcome: list[object] = []
def run():
other = SessionLocal()
try:
outcome.append(work(other))
other.commit()
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
outcome.append(e)
finally:
other.close()
t = threading.Thread(target=run)
t.start()
return t, outcome
def test_two_opens_at_once_count_as_two_views(shelf, db):
from app.routes import public
made = _link(shelf)
link = db.get(DownloadLink, made["id"])
public._opened(db, link, made["token"]) # this open's count is not committed yet
t, outcome = _meanwhile(
lambda other: public._opened(other, other.get(DownloadLink, made["id"]), made["token"])
)
t.join(timeout=1)
db.commit()
t.join(timeout=10)
assert not isinstance(outcome[0], Exception), outcome
db.expire_all()
assert db.get(DownloadLink, made["id"]).view_count == 2
def test_a_file_shared_as_it_is_deleted_takes_its_new_link_with_it(shelf, db):
from app.routes import downloads as routes
job_id = shelf["outside"]
routes._shareable_job(db, shelf["user"], job_id) # held from the check until this commits
t, outcome = _meanwhile(lambda other: other.delete(other.get(DownloadJob, job_id)))
t.join(timeout=1)
assert t.is_alive() # the delete waits for the link on its way in
db.add(DownloadLink(job_id=job_id, token_hash="h" * 64, created_by=shelf["user"].id))
db.commit()
t.join(timeout=10)
assert outcome == [None]
assert db.execute(select(DownloadLink.id)).scalars().all() == []
def test_a_folder_shared_as_it_is_deleted_takes_its_new_link_with_it(shelf, db):
from app.routes import downloads as routes
fid = shelf["folder"]
routes._shareable_folder(db, shelf["user"], fid) # held from the check until this commits
t, outcome = _meanwhile(lambda other: folders.delete_own(other, other.get(FileFolder, fid)))
t.join(timeout=1)
assert t.is_alive() # the delete waits for the link on its way in
db.add(DownloadLink(folder_id=fid, token_hash="h" * 64, created_by=shelf["user"].id))
db.commit()
t.join(timeout=10)
assert not isinstance(outcome[0], Exception), outcome
assert db.execute(select(DownloadLink.id)).scalars().all() == []
+227
View File
@@ -0,0 +1,227 @@
"""A `/watch` link's token at rest (downloads/links.py, migration 0107), DB-lane.
The token is the credential, so the database keeps only its SHA-256 (to find the link by) and a
Fernet-encrypted copy (for its owner to copy the same link again). Without a copy this key can read,
the link still opens and only its address cannot be shown again. Making, changing and revoking a
link is audited, never with its token or password.
"""
import json
import pytest
from alembic import command
from alembic.config import Config
from cryptography.fernet import Fernet
from sqlalchemy import select, text
from app import security
from app.config import settings
from app.db import engine
from app.downloads import probe as probemod
from app.models import AuditLog, DownloadLink
_VIDEO = {
"streams": [
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
{"codec_type": "audio", "codec_name": "aac"},
],
"format": {"format_name": "mov,mp4", "duration": "5"},
}
@pytest.fixture
def key(monkeypatch):
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
@pytest.fixture
def owner_job(as_user, make_user, tmp_path, monkeypatch, upload_headers):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
monkeypatch.setattr(probemod, "probe_json", lambda _p: _VIDEO)
owner = as_user(make_user())
job = owner.post("/api/files/upload", content=b"fake-video", headers=upload_headers("clip.mp4"))
assert job.status_code == 200, job.text
return owner, job.json()["id"]
def _row(db, link_id: int) -> dict:
db.expire_all()
row = db.execute(
text("SELECT * FROM download_links WHERE id = :id"), {"id": link_id}
).mappings()
return dict(row.one())
def _audits(db, action: str) -> list[AuditLog]:
db.expire_all()
return list(db.execute(select(AuditLog).where(AuditLog.action == action)).scalars())
def test_the_token_is_kept_only_hashed_and_encrypted(db, key, owner_job, client):
owner, job_id = owner_job
made = owner.post(f"/api/downloads/{job_id}/links", json={}).json()
token = made["token"]
assert made["url"] == f"/watch/{token}"
row = _row(db, made["id"])
assert "token" not in row
assert token not in json.dumps(row, default=str)
assert row["token_hash"] == security.hash_token(token)
assert security.decrypt(row["token_enc"]) == token
# The address it was given opens it, and its owner can copy the same one again.
assert client.get(f"/api/public/watch/{token}").status_code == 200
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [entry["url"] for entry in listed] == [f"/watch/{token}"]
def test_a_token_naming_no_link_finds_none(db, key, owner_job, client):
owner, job_id = owner_job
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
other = token[:-1] + ("A" if token[-1] != "A" else "B")
assert client.get(f"/api/public/watch/{other}").status_code == 404
# Its hash is no key either: the lookup hashes what it is given.
hashed = security.hash_token(token)
assert client.get(f"/api/public/watch/{hashed}").status_code == 404
def test_without_a_key_the_link_opens_but_its_address_is_not_kept(db, owner_job, client):
assert not security.can_encrypt()
owner, job_id = owner_job
made = owner.post(f"/api/downloads/{job_id}/links", json={}).json()
# The one who made it has it in hand…
assert made["url"] == f"/watch/{made['token']}"
assert _row(db, made["id"])["token_enc"] is None
assert client.get(f"/api/public/watch/{made['token']}").status_code == 200
# …but it cannot be shown again.
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [(entry["token"], entry["url"]) for entry in listed] == [(None, None)]
def test_a_changed_key_hides_the_address_and_the_link_still_opens(
db, key, owner_job, client, monkeypatch
):
owner, job_id = owner_job
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [entry["url"] for entry in listed] == [None]
assert client.get(f"/api/public/watch/{token}").status_code == 200
def test_making_changing_and_revoking_a_link_is_audited_without_its_secrets(db, key, owner_job):
owner, job_id = owner_job
pw = "a secret word"
made = owner.post(
f"/api/downloads/{job_id}/links", json={"password": pw, "allow_download": False}
).json()
link_id = made["id"]
patch = f"/api/downloads/links/{link_id}"
assert owner.patch(patch, json={"allow_download": True}).status_code == 200
assert owner.patch(patch, json={"password": "another one"}).status_code == 200
# Nothing changed: nothing to record.
assert owner.patch(patch, json={"allow_download": True}).status_code == 200
gone = owner.delete(f"/api/downloads/links/{link_id}")
assert gone.json() == {"ok": True, "deleted": link_id}
(create,) = _audits(db, "download_link_create")
assert create.target_id == str(link_id)
assert create.after == {
"job_id": job_id,
"allow_download": False,
"has_password": True,
"expires_at": None,
}
updates = _audits(db, "download_link_update")
assert [(u.before["allow_download"], u.after["allow_download"]) for u in updates] == [
(False, True),
(True, True), # the password changed: a rotation is recorded though the flags are equal
]
(revoke,) = _audits(db, "download_link_revoke")
assert revoke.target_id == str(link_id)
assert revoke.after == {"job_id": job_id}
trail = json.dumps(
[[a.summary, a.before, a.after] for a in [create, *updates, revoke]], default=str
)
for secret in (made["token"], pw, "another one"):
assert secret not in trail
def test_the_migration_keeps_every_address_already_sent(db, key, owner_job, client):
owner, job_id = owner_job
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
cfg = Config("alembic.ini")
command.downgrade(cfg, "0106_api_tokens")
try:
with engine.begin() as conn:
conn.execute(
text("INSERT INTO download_links (job_id, token) VALUES (:j, 'an-old-token-sent')"),
{"j": job_id},
)
finally:
command.upgrade(cfg, "head")
(row,) = db.execute(text("SELECT token_hash, token_enc FROM download_links")).mappings().all()
assert row["token_hash"] == security.hash_token("an-old-token-sent")
assert security.decrypt(row["token_enc"]) == "an-old-token-sent"
assert client.get("/api/public/watch/an-old-token-sent").status_code == 200
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [entry["url"] for entry in listed] == ["/watch/an-old-token-sent"]
def test_the_downgrade_writes_the_tokens_back(db, key, owner_job):
owner, job_id = owner_job
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
cfg = Config("alembic.ini")
command.downgrade(cfg, "0106_api_tokens")
try:
with engine.connect() as conn:
back = conn.execute(text("SELECT token FROM download_links")).scalars().all()
finally:
command.upgrade(cfg, "head")
assert back == [token]
assert db.execute(select(DownloadLink.token_hash)).scalars().all() == [
security.hash_token(token)
]
def test_without_a_key_the_migration_keeps_the_address_opening_but_not_shown(db, owner_job, client):
"""No key to seal with: the hash alone is kept, the link still opens, its address is gone."""
assert not security.can_encrypt()
owner, job_id = owner_job
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
cfg = Config("alembic.ini")
command.downgrade(cfg, "0106_api_tokens")
try:
with engine.begin() as conn:
conn.execute(
text("INSERT INTO download_links (job_id, token) VALUES (:j, 'an-old-token-sent')"),
{"j": job_id},
)
finally:
command.upgrade(cfg, "head")
(row,) = db.execute(text("SELECT token_hash, token_enc FROM download_links")).mappings().all()
assert row["token_hash"] == security.hash_token("an-old-token-sent")
assert row["token_enc"] is None
assert client.get("/api/public/watch/an-old-token-sent").status_code == 200
assert [e["url"] for e in owner.get(f"/api/downloads/{job_id}/links").json()] == [None]
def test_a_downgrade_with_no_readable_copy_writes_a_fresh_token(db, key, owner_job, monkeypatch):
"""A key since changed reads no copy: the old column gets a random token, so the link is
revoked rather than left with no token (the column is NOT NULL there)."""
owner, job_id = owner_job
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
cfg = Config("alembic.ini")
command.downgrade(cfg, "0106_api_tokens")
try:
with engine.connect() as conn:
back = conn.execute(text("SELECT token FROM download_links")).scalars().all()
finally:
command.upgrade(cfg, "head")
assert len(back) == 1 and back[0] and back[0] != token
+3
View File
@@ -53,6 +53,9 @@ class TestGrant:
def test_malformed_grants_are_rejected_not_crashed(self):
for bad in [None, "", "no-dot", "notanumber.sig"]:
assert check_grant("tok", bad, 0) is False
# The query string decodes as UTF-8: a signature past ASCII is just a wrong one.
future = int(datetime.now(timezone.utc).timestamp()) + _GRANT_TTL
assert check_grant("tok", f"{future}.é", 0) is False
def _asset(**over):
+50
View File
@@ -0,0 +1,50 @@
"""Migration 0106's downgrade data step, DB-lane: a Hub folder holding a file becomes an own folder
that keeps the file — under a name its user's own folders do not already have (their names are
unique per user, case-blind), and with its token gone with the tokens table."""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def test_a_hub_folder_with_a_file_comes_back_as_an_own_folder_beside_a_namesake(db, make_user):
uid = make_user().id
with engine.begin() as conn:
hub = conn.execute(
text("INSERT INTO file_folders (user_id, kind) VALUES (:u, 'hub') RETURNING id"),
{"u": uid},
).scalar_one()
conn.execute(
text("INSERT INTO file_folders (user_id, kind, name) VALUES (:u, 'own', :n)"),
{"u": uid, "n": f"hub {hub}"}, # the name the downgrade would pick, in another case
)
held = conn.execute(
text(
"INSERT INTO download_jobs (user_id, source_kind, source_ref, profile_snapshot,"
" status, folder_id) VALUES (:u, 'upload', 'x', '{}', 'queued', :f) RETURNING id"
),
{"u": uid, "f": hub},
).scalar_one()
conn.execute(
text(
"INSERT INTO api_tokens (user_id, name, prefix, token_hash) VALUES"
" (:u, 'hub', 'sfl_abcd1234', 'h') "
),
{"u": uid},
)
cfg = Config("alembic.ini")
command.downgrade(cfg, "0105_job_source_key")
try:
with engine.connect() as conn:
kind, name = conn.execute(
text("SELECT kind, name FROM file_folders WHERE id = :i"), {"i": hub}
).one()
folder_of_job = conn.execute(
text("SELECT folder_id FROM download_jobs WHERE id = :j"), {"j": held}
).scalar_one()
finally:
command.upgrade(cfg, "head")
assert (kind, name) == ("own", f"Hub {hub} (2)")
assert folder_of_job == hub
+165
View File
@@ -0,0 +1,165 @@
"""Migration 0109's data step, DB-lane: the audit log's run rows move into `job_runs` with their
outcome intact, and leave the audit log; an admin action stays where it was. The downgrade moves
them back."""
import json
from datetime import timedelta
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def _audit(conn, actor, action, target, after):
conn.execute(
text(
"INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after,"
" created_at) VALUES (:a, :act, 'job', :t, 's', CAST(:after AS json),"
" CAST('2026-10-01T03:00:00+00:00' AS timestamptz))"
),
{"a": actor, "act": action, "t": target, "after": json.dumps(after)},
)
def test_the_audit_logs_runs_move_into_the_run_table_and_back(db, make_user):
uid = make_user().id
cfg = Config("alembic.ini")
command.downgrade(cfg, "0108_folder_links")
try:
with engine.begin() as conn:
_audit(
conn,
None,
"job_run",
"rss_poll",
{"status": "ok", "summary": "new=3", "result": {"new": 3}},
)
_audit(
conn,
uid,
"job_run",
"rss_poll",
{"status": "error", "summary": "boom", "result": {"failure": "rss_all_failed"}},
)
_audit(
conn, uid, "job_run", "backfill", {"status": "skipped", "summary": "sync paused"}
)
_audit(conn, uid, "config_set", None, {"key": "x"})
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
runs = conn.execute(
text(
"SELECT job, trigger, actor_id, status, summary, result, error, failure,"
" to_char(started_at AT TIME ZONE 'UTC', 'MM-DD HH24'), finished_at = started_at"
" FROM job_runs ORDER BY id"
)
).all()
left = conn.execute(text("SELECT action FROM audit_log")).scalars().all()
assert [tuple(r) for r in runs] == [
("rss_poll", "scheduled", None, "ok", "new=3", {"new": 3}, None, None, "10-01 03", True),
(
"rss_poll",
"manual",
uid,
"error",
None,
None,
"boom",
{"failure": "rss_all_failed"},
"10-01 03",
True,
),
("backfill", "manual", uid, "skipped", "sync paused", None, None, None, "10-01 03", True),
]
assert left == ["config_set"]
command.downgrade(cfg, "0108_folder_links")
try:
with engine.connect() as conn:
back = conn.execute(
text(
"SELECT actor_id, target_id, after::jsonb FROM audit_log"
" WHERE action = 'job_run' ORDER BY id"
)
).all()
finally:
command.upgrade(cfg, "head")
# What the audit row has no column for rides along, so the next upgrade restores it.
def run(trigger: str) -> dict:
return {"trigger": trigger, "quota_units": 0, "started_at": "2026-10-01T03:00:00+00:00"}
assert [tuple(r) for r in back] == [
(
None,
"rss_poll",
{"status": "ok", "summary": "new=3", "result": {"new": 3}, **run("scheduled")},
),
(
uid,
"rss_poll",
{
"status": "error",
"summary": "boom",
"result": {"failure": "rss_all_failed"},
**run("manual"),
},
),
(uid, "backfill", {"status": "skipped", "summary": "sync paused", **run("manual")}),
]
def test_a_down_and_up_round_trip_keeps_every_run(db, make_user):
"""A rollback then a redeploy: an interrupted run, a run caught mid-way and a user's run of one
channel come back as runs — not stranded in the audit log under an action no code knows."""
uid = make_user().id
with engine.begin() as conn:
conn.execute(
text(
"INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at,"
" finished_at, quota_units) VALUES"
" ('backfill', 'scheduled', NULL, NULL, 'interrupted', now(), now(), 0),"
" ('maintenance', 'manual', :u, NULL, 'running', now(), NULL, 0),"
" ('channel_backfill', 'ad_hoc', :u, 'UC_x', 'ok', now(), now(), 0),"
# Its admin is gone (SET NULL): only the row itself still says it was manual.
" ('enrich', 'manual', NULL, NULL, 'ok',"
" '2026-10-01 10:00+00', '2026-10-01 10:30+00', 42)"
),
{"u": uid},
)
# What the run caught mid-way had spent: only its quota rows know it yet.
conn.execute(
text(
"INSERT INTO quota_events (action, units, job_run_id)"
" SELECT 'videos.list', 7, id FROM job_runs WHERE job = 'maintenance'"
)
)
cfg = Config("alembic.ini")
command.downgrade(cfg, "0108_folder_links")
command.upgrade(cfg, "head")
with engine.connect() as conn:
runs = conn.execute(
text("SELECT job, trigger, actor_id, target, status FROM job_runs ORDER BY id")
).all()
kept = conn.execute(
text("SELECT quota_units, finished_at - started_at FROM job_runs WHERE job = 'enrich'")
).one()
stopped = conn.execute(
text("SELECT quota_units FROM job_runs WHERE job = 'maintenance'")
).scalar()
stranded = conn.execute(
text("SELECT count(*) FROM audit_log WHERE action = 'job_run'")
).scalar()
assert [tuple(r) for r in runs] == [
("backfill", "scheduled", None, None, "interrupted"),
("maintenance", "manual", uid, None, "interrupted"), # the rollback stopped it
("channel_backfill", "ad_hoc", uid, "UC_x", "ok"),
("enrich", "manual", None, None, "ok"),
]
assert tuple(kept) == (42, timedelta(minutes=30))
assert stopped == 7
assert stranded == 0
+56
View File
@@ -0,0 +1,56 @@
"""Migration 0110's data step, DB-lane: an admin's interval override moves from `scheduler_settings`
into sysconfig and back, a row for a job that no longer exists is not carried over, and the unread
SponsorBlock default leaves the store.
The schema half is checked by test_migrations.py; a value lost on the way is invisible there.
"""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app import scheduler
from app.db import engine
def _config(conn) -> dict[str, str]:
return dict(conn.execute(text("SELECT key, value FROM system_config")).tuples().all())
def test_an_interval_override_moves_into_sysconfig_and_back(db):
cfg = Config("alembic.ini")
command.downgrade(cfg, "0109_job_runs")
try:
with engine.begin() as conn:
conn.execute(
text(
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
" VALUES ('rss_poll', 25), ('log_gc', 90), ('retired_job', 5)"
)
)
conn.execute(
text(
"INSERT INTO system_config (key, value, encrypted) VALUES"
" ('sponsorblock_default', 'true', false), ('quota_daily_budget', '5000', false)"
)
)
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
assert _config(conn) == {
"job_interval_rss_poll": "25",
"job_interval_log_gc": "90",
"quota_daily_budget": "5000",
}
assert conn.execute(text("SELECT to_regclass('scheduler_settings')")).scalar() is None
assert scheduler.load_intervals(db)["rss_poll"] == 25
command.downgrade(cfg, "0109_job_runs")
try:
with engine.connect() as conn:
back = conn.execute(text("SELECT job_id, interval_minutes FROM scheduler_settings"))
assert dict(back.tuples().all()) == {"rss_poll": 25, "log_gc": 90}
assert _config(conn) == {"quota_daily_budget": "5000"}
finally:
command.upgrade(cfg, "head")
-7
View File
@@ -9,7 +9,6 @@ from datetime import datetime, timezone
import httpx
import pytest
from app import scheduler
from app.models import PlexItem, PlexLibrary, PlexPerson, User
from app.plex import imgcache
from app.plex import people as plex_people
@@ -340,12 +339,6 @@ def test_the_people_job_runs_its_halves_without_a_tmdb_key(db, cache_dir, plex_o
}
def test_a_phase_count_is_a_change_worth_an_audit_row():
assert scheduler._run_changed({"bios": {"enriched": 3}, "portraits": {"cached": 0}})
assert scheduler._run_changed({"recent": {"videos_added": 2}, "deep": {}})
assert not scheduler._run_changed({"bios": {"skipped": "x"}, "portraits": {"cached": 0}})
def test_the_person_page_says_a_bio_is_on_its_way(db, monkeypatch, override_dep):
from fastapi.testclient import TestClient
+27
View File
@@ -0,0 +1,27 @@
"""The run words and the migrated database, DB-lane: the `job_runs` CHECKs are frozen in a migration,
and the model's are built from the enums, so nothing else would notice the two drifting apart until a
write of the new word failed in production."""
import re
from sqlalchemy import text
from app.runstatus import RunStatus, RunTrigger
def _words(definition: str) -> set[str]:
return set(re.findall(r"'([a-z_]+)'", definition))
def test_the_databases_checks_allow_exactly_the_run_words(db):
defs = dict(
db.execute(
text(
"SELECT conname, pg_get_constraintdef(oid) FROM pg_constraint"
" WHERE conname LIKE 'ck_job_runs_%'"
)
).all()
)
assert _words(defs["ck_job_runs_status"]) == set(RunStatus)
assert _words(defs["ck_job_runs_trigger"]) == set(RunTrigger)
assert _words(defs["ck_job_runs_finished"]) == {RunStatus.RUNNING}
+534 -47
View File
@@ -1,11 +1,21 @@
"""_claim_running is the single-run guard that both the scheduled fire and the manual trigger go
"""The scheduler's run path, DB-lane: the single-run guard, and every run as a durable `job_runs` row
— its outcome as data, what it spent, the lines it logged — that outlives the process.
_claim_running is the single-run guard that both the scheduled fire and the manual trigger go
through (at the top of _job): holding _activity_lock across the check-and-set means exactly one
caller can hold a job's "running" claim at a time, and it's claimable again once released."""
import logging
import time
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import select
from app import scheduler
from app.models import Notification
from app import jobruns, logs, logsink, quota, scheduler, state, sysconfig
from app.models import AuditLog, Channel, JobRun, Notification, QuotaEvent
from app.routes import scheduler as scheduler_routes
from app.runstatus import RunStatus, RunTrigger
_JOB = "unit_test_claim" # not a real job id — exercises the primitive without touching live jobs
@@ -31,26 +41,38 @@ def test_claim_is_exclusive_until_released():
# ---- the result as data --------------------------------------------------------------------------
def _snapshot_job(job_id):
return next(j for j in scheduler.scheduler_snapshot()["jobs"] if j["id"] == job_id)
def _snapshot_job(db, job_id):
return next(j for j in scheduler.scheduler_snapshot(db)["jobs"] if j["id"] == job_id)
def _runs(db, job=None) -> list[JobRun]:
db.expire_all()
q = select(JobRun).order_by(JobRun.id)
if job is not None:
q = q.where(JobRun.job == job)
return list(db.execute(q).scalars())
def _manual(actor_id, name, fn):
token = scheduler._manual_actor.set(actor_id)
try:
scheduler._job(name, fn)
finally:
scheduler._manual_actor.reset(token)
def test_a_run_keeps_its_result_as_data_for_the_readers_language(db, make_user):
"""The dashboard and the manual-run notice showed `_summarize`'s English one-liner ("new=3,
failed=0", a Python repr for a nested phase). The result rides along as data, so the client names
it; the one-liner stays as the fallback and the audit text."""
it; the one-liner stays as the fallback."""
admin = make_user("sched@example.com", role="admin")
token = scheduler._manual_actor.set(admin.id)
try:
scheduler._job("rss_poll", lambda _db: {"new": 3, "failed": 0, "fallback": 1})
scheduler._job("enrich", lambda _db: 5) # enrich returns a bare count
finally:
scheduler._manual_actor.reset(token)
_manual(admin.id, "rss_poll", lambda _db: {"new": 3, "failed": 0, "fallback": 1})
_manual(admin.id, "enrich", lambda _db: 5) # enrich returns a bare count
rss = _snapshot_job("rss_poll")
rss = _snapshot_job(db, "rss_poll")
assert rss["result"] == {"new": 3, "failed": 0, "fallback": 1}
assert rss["last_result"] == "new=3, failed=0, fallback=1"
assert _snapshot_job("enrich")["result"] == {"enriched": 5}
assert _snapshot_job(db, "enrich")["result"] == {"enriched": 5}
notices = db.execute(
select(Notification).where(Notification.user_id == admin.id).order_by(Notification.id)
@@ -64,53 +86,518 @@ def test_a_run_keeps_its_result_as_data_for_the_readers_language(db, make_user):
def test_a_failed_run_drops_the_previous_result(db):
scheduler._job("rss_poll", lambda _db: {"new": 1})
scheduler._job("rss_poll", lambda _db: (_ for _ in ()).throw(RuntimeError("boom")))
rss = _snapshot_job("rss_poll")
rss = _snapshot_job(db, "rss_poll")
assert rss["status"] == "error"
assert rss["last_error"] == "boom"
assert rss["result"] is None
def test_a_job_runs_audit_row_carries_its_result_as_data(db):
"""The audit log names a run's result in the reader's language, as the inbox does —
from data, not from the English one-liner (a nested phase there was a Python dict repr)."""
from app.models import AuditLog, User
admin = User(email="auditrun@example.com")
db.add(admin)
db.commit()
scheduler._audit_run(db, None, "people_enrich", "ok", "x", {"bios": {"enriched": 2}})
scheduler._audit_run(db, admin.id, "rss_poll", "skipped", "sync paused", scheduler._PAUSED)
scheduler._audit_run(db, admin.id, "rss_poll", "error", "boom", None)
rows = db.execute(select(AuditLog).order_by(AuditLog.id)).scalars().all()
assert [r.after for r in rows] == [
{"status": "ok", "summary": "x", "result": {"bios": {"enriched": 2}}},
{"status": "skipped", "summary": "sync paused", "result": {"skipped": "sync paused"}},
{"status": "error", "summary": "boom"},
]
def test_a_failure_the_job_names_rides_along_as_data(db, make_user):
"""A failed RSS poll reached the Hungarian audit log as the server's English sentence. A
failure the job names (`progress.JobFailed`) is stored as data beside that sentence — on the
dashboard, the manual-run notice and the audit row — so the client says it in the reader's
language. Any other failure keeps only its own text (the test above)."""
"""A failed RSS poll reached the Hungarian UI as the server's English sentence. A failure the job
names (`progress.JobFailed`) is stored as data beside that sentence — on the run's row, the
dashboard and the manual-run notice — so the client says it in the reader's language."""
from app import progress
from app.models import AuditLog
def outage(_db):
raise progress.JobFailed("rss_all_failed", "RSS poll failed", total=2, cause="transport")
admin = make_user("failure@example.com", role="admin")
token = scheduler._manual_actor.set(admin.id)
try:
scheduler._job("rss_poll", outage)
finally:
scheduler._manual_actor.reset(token)
_manual(admin.id, "rss_poll", outage)
named = {"failure": "rss_all_failed", "total": 2, "cause": "transport"}
rss = _snapshot_job("rss_poll")
rss = _snapshot_job(db, "rss_poll")
assert rss["last_error"] == "RSS poll failed"
assert rss["result"] == named
notice = db.execute(select(Notification).where(Notification.user_id == admin.id)).scalar_one()
assert notice.data["result"] == named
row = db.execute(select(AuditLog).order_by(AuditLog.id.desc())).scalars().first()
assert row is not None and row.after["result"] == named
(run,) = _runs(db, "rss_poll")
assert (run.status, run.error, run.failure) == ("error", "RSS poll failed", named)
# ---- the durable row -----------------------------------------------------------------------------
def test_every_run_is_a_row_with_who_started_it_and_none_in_the_audit_log(db, make_user):
"""A no-op scheduled poll was not written anywhere (the audit rule kept only changes), so "did
it run at 03:00?" had no answer. Every run is a row now; the audit log keeps admin actions only."""
admin = make_user("runs@example.com", role="admin")
scheduler._job("rss_poll", lambda _db: {"new": 0})
_manual(admin.id, "rss_poll", lambda _db: {"new": 2})
scheduled, manual = _runs(db, "rss_poll")
assert (scheduled.trigger, scheduled.actor_id, scheduled.status) == ("scheduled", None, "ok")
assert scheduled.result == {"new": 0}
assert (manual.trigger, manual.actor_id, manual.summary) == ("manual", admin.id, "new=2")
assert manual.finished_at is not None and manual.finished_at >= manual.started_at
assert db.execute(select(AuditLog)).scalars().all() == []
def test_the_last_run_survives_a_restart(db):
"""The dashboard's "last run" lived in the process's memory: a deploy blanked every job."""
scheduler._job("plex_sync", lambda _db: {"items": 4})
with scheduler._activity_lock:
scheduler._activity.clear() # what a new process starts with
plex = _snapshot_job(db, "plex_sync")
assert (plex["status"], plex["result"], plex["last_trigger"]) == (
"ok",
{"items": 4},
"scheduled",
)
assert plex["last_finished"] is not None
def test_a_row_shows_its_last_three_runs_newest_first_an_interrupted_one_included(db):
"""The row said only how the LAST run went; an interrupted run read as "no run recorded"."""
oldest = jobruns.start("shorts", RunTrigger.SCHEDULED, None)
jobruns.interrupt_leftovers()
scheduler._job("shorts", lambda _db: {"probed": 1})
scheduler._job("shorts", lambda _db: {"probed": 2})
scheduler._job("shorts", lambda _db: 1 / 0)
scheduler._job("rss_poll", lambda _db: {"new": 0}) # another job's run stays on its own row
recent = _snapshot_job(db, "shorts")["recent"]
assert [r["status"] for r in recent] == ["error", "ok", "ok"]
assert oldest not in [r["id"] for r in recent] # the fourth-newest is not shown
assert recent[0]["trigger"] == "scheduled" and recent[0]["finished_at"] is not None
assert [r["status"] for r in _snapshot_job(db, "rss_poll")["recent"]] == ["ok"]
jobruns.start("shorts", RunTrigger.SCHEDULED, None)
jobruns.interrupt_leftovers()
shorts = _snapshot_job(db, "shorts")
assert shorts["status"] == "interrupted"
assert [r["status"] for r in shorts["recent"]] == ["interrupted", "error", "ok"]
def test_a_paused_sync_is_a_skipped_run(db):
state.set_sync_paused(db, True)
db.commit()
scheduler._job("rss_poll", lambda _db: pytest.fail("a paused run must not do its work"))
(run,) = _runs(db, "rss_poll")
assert (run.status, run.result) == ("skipped", {"skipped": "sync paused"})
def test_a_run_left_running_by_a_stopped_process_is_interrupted_and_stays_so(db):
"""A deploy mid-run left nothing behind before; now the row would say `running` forever. The
next startup closes it, and the stale thread's own finish cannot rewrite it."""
run_id = jobruns.start("backfill", RunTrigger.SCHEDULED, None)
assert jobruns.interrupt_leftovers() == 1
(run,) = _runs(db, "backfill")
assert run.status == "interrupted" and run.finished_at is not None
jobruns.finish(run_id, RunStatus.OK, summary="late")
(run,) = _runs(db, "backfill")
assert (run.status, run.summary) == ("interrupted", None)
def test_a_run_knows_what_it_spent_and_a_request_spend_is_nobodys_run(db):
def spends(_db):
quota.record_usage(5)
quota.record_usage(3)
return {"enriched": 2}
scheduler._job("enrich", spends)
quota.record_usage(11) # a request's own spend, outside any run
(run,) = _runs(db, "enrich")
assert run.quota_units == 8
events = db.execute(select(QuotaEvent.units, QuotaEvent.job_run_id)).all()
assert sorted(events) == [(3, run.id), (5, run.id), (11, None)]
assert _snapshot_job(db, "enrich")["quota_units"] == 8
class _Collect(logging.Handler):
def __init__(self):
super().__init__()
self.addFilter(logs.ContextFilter())
self.records: list[logging.LogRecord] = []
def emit(self, record):
self.records.append(record)
def test_a_runs_log_lines_carry_its_id(db):
"""So a run's row can open exactly the lines it logged (the log table's `run_id`)."""
probe = logging.getLogger("siftlode.test.run")
handler = _Collect()
probe.addHandler(handler)
probe.setLevel(logging.INFO)
try:
scheduler._job("autotag", lambda _db: probe.info("tagging") or {"tagged": 1})
probe.info("after the run")
finally:
probe.removeHandler(handler)
(run,) = _runs(db, "autotag")
inside, after = (logs.record_fields(r).get("run_id") for r in handler.records)
assert (inside, after) == (run.id, None)
# …and it reaches the stored row, whose columns are the model's.
assert logsink.to_row(logs.record_fields(handler.records[0]))["run_id"] == run.id
def test_retention_reaps_finished_runs_only(db):
old = datetime.now(timezone.utc) - timedelta(days=40)
db.add_all(
[
JobRun(
job="rss_poll",
trigger="scheduled",
status="ok",
started_at=old,
finished_at=old,
),
JobRun(job="backfill", trigger="scheduled", status="running", started_at=old),
JobRun(
job="rss_poll",
trigger="scheduled",
status="ok",
started_at=datetime.now(timezone.utc),
finished_at=datetime.now(timezone.utc),
),
]
)
db.commit()
reaped = next(r for r in _runs(db) if r.status == "ok" and r.started_at < old + timedelta(1))
with logs.bind(run_id=reaped.id):
quota.record_usage(7)
assert jobruns.gc(db, 0) == 0 # 0 = keep forever
assert jobruns.gc(db, 30) == 1
db.commit()
assert sorted((r.job, r.status) for r in _runs(db)) == [
("backfill", "running"),
("rss_poll", "ok"),
]
# Its spend stays in the budget's history (kept on its own retention), no longer anyone's run.
assert db.execute(select(QuotaEvent.units, QuotaEvent.job_run_id)).all() == [(7, None)]
# ---- "run now" through the scheduler --------------------------------------------------------------
@pytest.fixture
def live_scheduler(db, monkeypatch):
"""This process's scheduler, started as the API starts it (on the migrated schema `db` brings),
with interval jobs off."""
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", False)
scheduler.start_scheduler()
try:
yield scheduler._scheduler
finally:
scheduler.shutdown_scheduler()
def _wait_finished(db, job, timeout=10.0) -> JobRun:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
runs = [r for r in _runs(db, job) if r.status != "running"]
if runs:
return runs[-1]
time.sleep(0.05)
raise AssertionError(f"{job} did not finish")
def test_run_now_goes_through_the_scheduler_even_with_interval_jobs_off(
db, make_user, live_scheduler, monkeypatch
):
admin = make_user("now@example.com", role="admin")
monkeypatch.setitem(
scheduler.JOB_FUNCS, "shorts", lambda: scheduler._job("shorts", lambda _db: {"probed": 3})
)
assert live_scheduler.get_jobs() == [] # SCHEDULER_ENABLED=false: no interval job
live_scheduler.pause() # hold the one-shot so the second click finds it pending
assert scheduler.trigger_job("shorts", admin.id) == "started"
assert scheduler.trigger_job("shorts", admin.id) == "already_running"
assert [j.id for j in live_scheduler.get_jobs()] == ["shorts@now"]
live_scheduler.resume()
run = _wait_finished(db, "shorts")
assert (run.trigger, run.actor_id, run.result) == ("manual", admin.id, {"probed": 3})
def test_an_ad_hoc_run_is_the_users_run_of_its_target_with_its_spend_and_failure(
db, make_user, live_scheduler
):
"""The one-channel deep backfill ran as a BackgroundTask nothing recorded: its spend counted as
nobody's run and a failure lived only in the log."""
user = make_user("adhoc@example.com")
def work():
quota.record_usage(3)
return {"videos_added": 50}
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, work, actor_id=user.id, target="UC_a")
run = _wait_finished(db, "channel_backfill")
assert (run.trigger, run.actor_id, run.target) == ("ad_hoc", user.id, "UC_a")
assert (run.status, run.result, run.quota_units) == ("ok", {"videos_added": 50}, 3)
def fail():
raise RuntimeError("quota exceeded")
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, fail, actor_id=user.id, target="UC_b")
deadline = time.monotonic() + 10
while time.monotonic() < deadline and len(_finished(db, "channel_backfill")) < 2:
time.sleep(0.05)
failed = next(r for r in _runs(db, "channel_backfill") if r.target == "UC_b")
assert (failed.status, failed.error) == ("error", "quota exceeded")
def test_the_page_shows_the_ad_hoc_runs_beside_the_jobs_with_their_channel(db, make_user):
user = make_user("shown@example.com")
db.add(Channel(id="UC_shown", title="BBC Earth"))
db.commit()
scheduler._ad_hoc("channel_backfill", lambda: {"skipped": "at capacity"}, user.id, "UC_none")
scheduler._ad_hoc("channel_backfill", lambda: {"videos_added": 69}, user.id, "UC_shown")
(row,) = scheduler.scheduler_snapshot(db)["ad_hoc"]
assert (row["id"], row["running"], row["target"], row["target_title"]) == (
"channel_backfill",
False,
"UC_shown",
"BBC Earth",
)
assert (row["status"], row["result"], row["last_trigger"]) == (
"ok",
{"videos_added": 69},
"ad_hoc",
)
assert len(row["recent"]) == 2
assert all(j["id"] != "channel_backfill" for j in scheduler.scheduler_snapshot(db)["jobs"])
seen = []
scheduler._ad_hoc(
"channel_backfill",
lambda: seen.append(scheduler.scheduler_snapshot(db)["ad_hoc"][0]["running"]) or {},
user.id,
"UC_shown",
)
assert seen == [True]
assert scheduler.scheduler_snapshot(db)["ad_hoc"][0]["running"] is False
def _finished(db, job) -> list[JobRun]:
return [r for r in _runs(db, job) if r.status != "running"]
def test_an_ad_hoc_job_outside_the_named_list_is_refused(live_scheduler):
"""The page names every ad-hoc job from AD_HOC_JOBS; one outside it would show as a bare id."""
with pytest.raises(KeyError):
scheduler.run_ad_hoc("unnamed_job", dict, actor_id=1, target="x")
assert live_scheduler.get_jobs() == []
def test_a_second_ad_hoc_request_before_the_first_starts_is_that_same_run(
db, make_user, live_scheduler
):
user = make_user("twice@example.com")
live_scheduler.pause() # hold the one-shots so the second request finds the first pending
for target in ("UC_x", "UC_x", "UC_y"):
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, dict, actor_id=user.id, target=target)
assert sorted(j.id for j in live_scheduler.get_jobs()) == [
"channel_backfill@UC_x",
"channel_backfill@UC_y",
]
live_scheduler.remove_all_jobs() # never run: a run finishing after the test would land in the next
def test_a_manual_run_leaves_no_actor_behind_on_its_thread(db, make_user):
"""The scheduler's pool threads are reused: an actor a manual run left set made the next
scheduled run on that thread an admin's "manual" run, with a notice in their inbox."""
admin = make_user("pool@example.com", role="admin")
seen = []
scheduler._as_actor(lambda: seen.append(scheduler._manual_actor.get()), admin.id)()
assert seen == [admin.id]
assert scheduler._manual_actor.get() is None # this same thread's next task runs as nobody
def test_run_now_refuses_a_job_already_running(live_scheduler):
assert scheduler._claim_running("maintenance") is True
try:
assert scheduler.trigger_job("maintenance") == "already_running"
finally:
scheduler._record("maintenance", running=False)
def test_startup_closes_what_the_previous_process_left_running(db, monkeypatch):
jobruns.start("plex_sync", RunTrigger.MANUAL, None)
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", False)
scheduler.start_scheduler()
scheduler.shutdown_scheduler()
(run,) = _runs(db, "plex_sync")
assert run.status == "interrupted"
def test_an_interrupted_run_keeps_what_it_spent(db):
"""Its spend is in the quota history, tagged with its id: the sweep sums it as `finish` would."""
run_id = jobruns.start("backfill", RunTrigger.SCHEDULED, None)
with logs.bind(run_id=run_id):
quota.record_usage(40)
quota.record_usage(2)
assert jobruns.interrupt_leftovers() == 1
(run,) = _runs(db, "backfill")
assert (run.status, run.quota_units) == ("interrupted", 42)
def _flaky_finish(monkeypatch) -> list:
"""`jobruns.finish` whose first write fails (the database blinked); the statuses it was asked for."""
calls: list = []
real = jobruns.finish
def finish(run_id, status, **outcome):
calls.append(status)
if len(calls) == 1:
raise RuntimeError("db blip")
real(run_id, status, **outcome)
monkeypatch.setattr(jobruns, "finish", finish)
return calls
def test_a_run_whose_outcome_cannot_be_written_is_not_called_failed(db, make_user, monkeypatch):
"""The job succeeded; only its bookkeeping did not. Writing ERROR then would say the work failed."""
calls = _flaky_finish(monkeypatch)
scheduler._job("autotag", lambda _db: {"tagged": 1})
assert calls == ["ok"]
assert not scheduler._is_running("autotag")
(run,) = _runs(db, "autotag")
assert run.status == "running" # left for the next start's sweep, not rewritten as a failure
# Nothing here runs it: the page reads it as the sweep will close it, not as a run going on.
row = _row(db, "autotag")
assert (row["running"], row["status"], row["recent"][0]["status"]) == (
False,
"interrupted",
"interrupted",
)
assert row["last_finished"] is not None
user = make_user("blip@example.com")
calls.clear()
scheduler._ad_hoc("channel_backfill", lambda: {"videos_added": 1}, user.id, "UC_blip")
assert calls == ["ok"]
# Its row is still `running`, but nothing here is: the page does not call it going.
(row,) = scheduler.scheduler_snapshot(db)["ad_hoc"]
assert (row["running"], row["status"], row["recent"][0]["status"]) == (
False,
"interrupted",
"interrupted",
)
def test_a_run_going_on_reads_as_running_on_its_row(db):
seen = []
scheduler._job("autotag", lambda _db: seen.append(_row(db, "autotag")) or {"tagged": 1})
(row,) = seen
assert (row["running"], row["status"], row["recent"][0]["status"]) == (True, None, "running")
assert _row(db, "autotag")["recent"][0]["status"] == "ok"
def _row(db, job_id: str) -> dict:
return next(j for j in scheduler.scheduler_snapshot(db)["jobs"] if j["id"] == job_id)
def test_a_run_that_cannot_be_recorded_does_not_wedge_its_job(db, monkeypatch):
def broken(*_a, **_k):
raise RuntimeError("db down")
monkeypatch.setattr(jobruns, "start", broken)
scheduler._job("autotag", lambda _db: pytest.fail("no row, no run"))
assert not scheduler._is_running("autotag")
monkeypatch.undo()
scheduler._job("autotag", lambda _db: {"tagged": 1})
assert [r.status for r in _runs(db, "autotag")] == ["ok"]
def test_the_page_says_where_the_interval_jobs_do_not_run(db, live_scheduler, monkeypatch):
"""The scheduler starts on every install now; only SCHEDULER_ENABLED says the jobs run here."""
assert scheduler.scheduler_snapshot(db)["running_here"] is False
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", True)
assert scheduler.scheduler_snapshot(db)["running_here"] is True
def test_an_ad_hoc_run_still_going_shows_although_a_newer_one_ended(db, make_user):
"""Two channels can backfill at once; a later run that ended at once must not hide the first."""
user = make_user("two@example.com")
db.add(Channel(id="UC_long", title="Long one"))
db.commit()
rows = []
def long_one():
scheduler._ad_hoc(
"channel_backfill", lambda: {"skipped": "at capacity"}, user.id, "UC_short"
)
rows.extend(scheduler.scheduler_snapshot(db)["ad_hoc"])
return {}
scheduler._ad_hoc("channel_backfill", long_one, user.id, "UC_long")
(row,) = rows
assert (row["running"], row["target"], row["target_title"]) == (True, "UC_long", "Long one")
# The outcome beside it is the long one's (none yet), never the short one's skip.
assert (row["status"], row["result"], row["last_error"]) == (None, None, None)
assert [r["status"] for r in row["recent"]] == ["running", "ok"]
def test_an_interval_change_lands_with_its_audit_row_or_not_at_all(
db, make_user, as_user, live_scheduler, monkeypatch
):
live_scheduler.add_job(lambda: None, "interval", minutes=10, id="backfill")
admin = as_user(make_user(role="admin"))
def broken(*_a, **_k):
raise RuntimeError("audit write failed")
monkeypatch.setattr(scheduler_routes.audit, "record", broken)
with pytest.raises(RuntimeError): # the handler raises: there is no response to check
_ = admin.patch("/api/admin/scheduler/jobs/backfill", json={"interval_minutes": 45})
db.expire_all()
assert sysconfig.get_int(db, "job_interval_backfill") == scheduler.JOB_INTERVALS["backfill"]
assert live_scheduler.get_job("backfill").trigger.interval == timedelta(minutes=10)
def test_every_job_has_an_interval_and_every_interval_a_job():
assert set(scheduler.JOB_FUNCS) == set(scheduler.JOB_INTERVALS)
def test_a_stored_interval_is_what_the_job_starts_with(db, monkeypatch):
sysconfig.set_value(db, "job_interval_rss_poll", 25)
db.commit()
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", True)
scheduler.start_scheduler()
try:
sched = scheduler._scheduler
assert sched is not None
assert sched.get_job("rss_poll").trigger.interval == timedelta(minutes=25)
default = scheduler.JOB_INTERVALS["enrich"]
assert sched.get_job("enrich").trigger.interval == timedelta(minutes=default)
finally:
scheduler.shutdown_scheduler()
def test_an_interval_set_on_the_scheduler_page_is_stored_and_reschedules_the_job(
db, make_user, as_user, live_scheduler
):
live_scheduler.add_job(lambda: None, "interval", minutes=10, id="backfill")
admin = as_user(make_user(role="admin"))
r = admin.patch("/api/admin/scheduler/jobs/backfill", json={"interval_minutes": 45})
assert r.status_code == 200, r.text
assert live_scheduler.get_job("backfill").trigger.interval == timedelta(minutes=45)
db.expire_all()
assert sysconfig.get_int(db, "job_interval_backfill") == 45
assert scheduler.load_intervals(db)["backfill"] == 45
# The Configuration page neither lists an interval nor writes one past the reschedule.
assert (
admin.patch("/api/admin/config/job_interval_backfill", json={"value": 5}).status_code == 404
)
assert admin.delete("/api/admin/config/job_interval_backfill").status_code == 404
assert "scheduler" not in admin.get("/api/admin/config").json()["groups"]
db.expire_all()
assert sysconfig.get_int(db, "job_interval_backfill") == 45
def test_the_setup_wizard_does_not_write_an_interval(db):
from fastapi import HTTPException
from app.routes.setup import setup_config
with pytest.raises(HTTPException) as refused:
setup_config({"job_interval_rss_poll": 5}, db)
assert refused.value.status_code == 400
assert setup_config({"quota_daily_budget": 5000}, db)["saved"] == ["quota_daily_budget"]
+3
View File
@@ -96,6 +96,9 @@ class TestDownloadFilename:
def test_does_not_double_an_extension_already_present(self):
assert download_filename("clip.mp4", "mp4", Path("/x/f.mp4")) == "clip.mp4"
def test_no_extension_is_served_without_a_trailing_dot(self):
assert download_filename("Makefile", None, Path("/x/upload-7")) == "Makefile"
def test_falls_back_to_the_file_suffix_when_no_container(self):
assert download_filename("song", None, Path("/x/f.m4a")) == "song.m4a"
+81 -1
View File
@@ -15,10 +15,14 @@ boot, sysconfig holds a stored override to it, and the admin page renders a pick
The DB lane (`db` fixture) covers the store; the rest is pure logic.
"""
import ast
from pathlib import Path
import pytest
from pydantic import ValidationError
from app import sysconfig
from app.models import SystemConfig
from app.config import Settings, choices_of
CHOICE_KEYS = {s.key for s in sysconfig.SPECS if s.type == "choice"}
@@ -37,7 +41,8 @@ def test_each_group_is_one_run_of_keys():
def test_every_group_sits_in_exactly_one_section():
listed = [g for _, members in sysconfig.SECTIONS for g in members]
assert len(listed) == len(set(listed)), listed
assert set(listed) == {s.group for s in sysconfig.SPECS}
assert set(listed) | set(sysconfig.OWN_PAGE_GROUPS) == {s.group for s in sysconfig.SPECS}
assert not set(listed) & set(sysconfig.OWN_PAGE_GROUPS)
assert [sid for sid, _ in sysconfig.SECTIONS] == ["access", "youtube", "media", "system"]
@@ -112,3 +117,78 @@ def test_set_value_stores_the_canonical_spelling_and_refuses_anything_else(db):
with pytest.raises(ValueError, match="scrape, api"):
sysconfig.set_value(db, "search_source", "bing")
db.rollback()
# --- when a changed value takes effect --------------------------------------------------------------
_APP = Path(__file__).resolve().parent.parent / "app"
_READERS = {"get", "get_str", "get_int", "get_bool"}
def _key_reads() -> list[tuple[str, str, bool]]:
"""Every `sysconfig.get*(db, "<key>")` in the app outside the registry: (key, where, inside a
function)."""
found: list[tuple[str, str, bool]] = []
def walk(node: ast.AST, path: Path, in_fn: bool) -> None:
for child in ast.iter_child_nodes(node):
fn = in_fn or isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda))
if (
isinstance(child, ast.Call)
and isinstance(child.func, ast.Attribute)
and isinstance(child.func.value, ast.Name)
and child.func.value.id == "sysconfig"
and child.func.attr in _READERS
and len(child.args) >= 2
and isinstance(child.args[1], ast.Constant)
and isinstance(child.args[1].value, str)
):
found.append((child.args[1].value, f"{path.name}:{child.lineno}", in_fn))
walk(child, path, fn)
for path in _APP.rglob("*.py"):
if path.name != "sysconfig.py":
walk(ast.parse(path.read_text(encoding="utf-8")), path, False)
return found
def test_every_key_states_when_it_applies_and_none_needs_a_restart_yet():
with pytest.raises(TypeError):
sysconfig.ConfigSpec("k", "int", "quota", "quota_daily_budget") # pyright: ignore[reportCallIssue]
# The Configuration page cannot tell the admin that a setting waits for a restart; a key that
# does needs that first (a badge on it and a banner once it changed).
assert {s.applies for s in sysconfig.SPECS} == {"live"}
def test_a_live_key_is_read_where_it_is_used_never_at_import():
reads = _key_reads()
assert len(reads) > 50, len(
reads
) # the scan sees the reads (it would pass vacuously otherwise)
assert [(k, at) for k, at, in_fn in reads if not in_fn] == []
def test_every_key_on_the_configuration_page_has_a_reader():
read = {k for k, _, _ in _key_reads()}
page = {s.key for s in sysconfig.SPECS if s.group not in sysconfig.OWN_PAGE_GROUPS}
assert sorted(page - read) == []
def test_the_intervals_are_keys_of_the_scheduler_page_only():
keys = {s.key for s in sysconfig.SPECS if s.group == "scheduler"}
assert keys == {sysconfig.interval_key(j) for j in sysconfig.JOB_INTERVAL_DEFAULTS}
assert all(len(k) <= 64 for k in keys) # system_config.key
assert sysconfig.config_page_spec("job_interval_rss_poll") is None
assert sysconfig.config_page_spec("quota_daily_budget") is not None
assert "scheduler" not in sysconfig.describe(_FakeDb())["groups"]
def test_saving_the_value_in_force_still_stores_the_override(db, make_user, as_user):
client = as_user(make_user(role="admin"))
default = sysconfig.get_int(db, "quota_daily_budget")
r = client.patch("/api/admin/config/quota_daily_budget", json={"value": default})
assert r.status_code == 200, r.text
(item,) = [i for i in r.json()["groups"]["quota"] if i["key"] == "quota_daily_budget"]
assert item["is_set"] is True
db.expire_all()
assert db.get(SystemConfig, "quota_daily_budget") is not None
+138 -126
View File
@@ -44,6 +44,10 @@ runs this check before it is called done (E28 S3g):
A rule that holds is not the same as a screen that works: the person page passed every rule above with
half its panel empty.
A **copy-only** change — translated text and nothing else: no markup, class, component or logic — runs step 4
alone: a screenshot of every state the changed text renders in, in Hungarian and in English (Hungarian runs about
a fifth longer, so it is the one that wraps or clips). The key parity is the gate's.
## The design lab
`npm run dev`, then open **`http://localhost:5173/design-lab`**. The lab renders each open decision's variants
@@ -126,7 +130,10 @@ style guide.
to name one of the two. Black and white are the media scrims and their text. Five schemes × dark
and light (B2): **Midnight** (blue), **Forest** (cyan), **Slate** (fuchsia), **Matrix** (moss) each earn their
place with their own hue family; **Fjord** (Midnight's navy with a calm steel-blue accent) is the one kept
exception, the calm twin of the vivid Midnight. A new scheme still needs its own hue family. Normal text ≥ 4.5:1 and a control's edge ≥ 3:1 in **every** scheme and mode.
exception, the calm twin of the vivid Midnight. A new scheme still needs its own hue family. Normal text ≥ 4.5:1 and a control's edge ≥ 3:1 in **every** scheme and mode;
`--muted` keeps ≥ 6:1 on the bare `--bg`, because it mostly sits on glass, which costs up to ~1.5 of the ratio.
A tonal fill under text (a badge, the item count, the current rail row) is an opaque `bg-tint*` (its colour
mixed into `--surface`), never a translucent `bg-fg/8`: the glass and what scrolls under it show through.
[WCAG 1.4.3, 1.4.11] The input edge token is set under Controls (A4). The ≥ 3:1 edge is the field
family's (a field, the form's dropdown trigger), whose edge is the only thing that says "type here". A
labelled pill (a chip, a toolbar trigger, a tab, a segment) keeps the faint `--border`: its text names it,
@@ -268,10 +275,9 @@ Also in force:
- **Page chrome (B4):** a page's tabs and filters sit in ONE sticky band of the page colour, with the fade
under it (below). Below ~600px of height the filter row folds into one "Filters · n" button. Every page has
a visually hidden `h1`; the nav rail and the browser tab title are the visible identity. [WCAG 2.4.6]
Built as: the band is `PageShell`'s toolbar slot (`PageToolbar`), above the one scroller; a console
whose toolbar lives inside its frame (the server log) makes that toolbar `sticky` with `fade-below`
instead. The band holds controls only: an intro paragraph scrolls with the content (in the band it
ate three lines of a short window). The fold is `useFilterFold` + `FilterFoldButton`: the same row
Built as: the band is `PageShell`'s toolbar slot (`PageToolbar`), above the one scroller. The band
holds controls only: an intro paragraph scrolls with the content (in the band it ate three lines of a
short window). The fold is `useFilterFold` + `FilterFoldButton`: the same row
shown or hidden in place, never copied into a popover. The hidden `h1` is `PageShell`'s `title`; a
view with its own visible title (a channel, a Plex show, a playlist) renders it as `PageHeading`
and the hidden one steps aside, so a page has exactly one `h1` (a hand `<h1>` inside the shell is
@@ -642,7 +648,7 @@ of a full-width panel and nothing on the right.
`app/collation.py` on the server, so both sort alike.
- **Hungarian words, not English ones** (E30 D3). Kept as Hungarian users say them: Feed, Shorts, Admin,
Token, Port, and technical tokens (log levels, ffmpeg presets, remux). Everything else is Hungarian:
közvetítés (stream), Prioritás, Feladat (job), Eseménynapló (audit), Demó, Automatikus, Infó,
közvetítés (stream), Prioritás, Feladat (job), Napló (activity), Admin műveletek (audit), Demó, Automatikus, Infó,
Felület / Szerver (frontend / backend), videók behúzása (backfill), átkódolás (transcode), e-mail.
A longer Hungarian word gets its room — a column is sized to it, a pill truncates — never clipped.
- A label says what happens ("Delete playlist"), not "OK"; a destructive confirm names the object.
@@ -661,123 +667,129 @@ of a full-width panel and nothing on the right.
## Decision log
| Date | Decision | Chosen | Over | Why |
| ---------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
| Date | Decision | Chosen | Over | Why |
| ---------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
| 2026-10-17 | Copy-only changes (owner ★) | Step 4 of the Conformance check alone (not 1–3 or 5): every state the changed text renders in, seen in both languages | The whole five-step check; no screenshot at all (DOM test + key parity) | Steps 1–3 and 5 judge controls, patterns and composition, which a text change does not touch; the one risk left is a longer Hungarian string wrapping or clipping, and only a screenshot shows that |
| 2026-10-17 | Text on glass (owner ★) | Light `--muted` 15% toward `--fg` in every scheme (≥ 6:1 on the bare page); the item count, the badges and the current Files row on opaque tints mixed into the surface | Only the two failing components; the item count's number in `--fg` | Pixel-measured over every scheme × mode × backdrop, the count fell to 3.8:1 (dark too: thumbnails scroll under the toolbar's glass) and Channels' tabs to 4.1:1; an `--fg` count outshouts the filters beside it |
| 2026-10-18 | Scheduler run history (owner D4, E1, E2 ★) | A row's last three runs as dots, newest first (interrupted: a hollow warn ring, set apart from skipped and failed), and a history button that opens the job's runs on the Activity page; a user's ad-hoc runs in their own block under the jobs, with no interval and no run button | An inline expandable history per row; a dark-grey interrupted dot; ad-hoc runs only on the Activity page | Three dots answer "did it run, did it work" at a glance; the whole history needs filters, which the Activity page has |
| 2026-10-18 | Activity page (owner D1, D2, G1–G3 ★) | One page, "Napló" / "Activity": four tabs (Runs, Admin actions, Server log, Quota) under one filter bar (text, a period of 1 h · 24 h · 7 d · 30 d · 90 d defaulting to 7 days, an account), the view in the URL; runs and admin actions as dense rows that open in place as a log line does, filtered and paged on the server; a run's or a request's log lines one pivot away, as a history entry | One merged timeline; a page per source; a default period per tab; card rows; the audit log's client-side table | Each source keeps its own table and retention, read in one place; one bar keeps what you narrowed across tabs; about 1.8k runs a day need a log's density and a server-side filter |
+13 -4
View File
@@ -44,12 +44,11 @@ A change is done when this is green, from the repo root:
bash scripts/check.sh # every lane
bash scripts/check.sh lint # the static lanes only (no tests, build or audits) — the quick loop
bash scripts/check.sh --brief # either mode: one line per lane, output only for a red one
bash scripts/lint-changed.sh # ruff + eslint + prettier on the uncommitted files only
bash scripts/check.sh --list # the lane names, one per line (runs nothing)
bash scripts/check.sh --only "vitest" # one lane, by its exact name — never "the gate"
```
It runs every lane even after one fails and names each red one at the end. `lint-changed.sh` is a
convenience, not the gate; with `--hook` it doubles as a Claude Code Stop hook (silent when clean,
the findings handed back otherwise).
It runs every lane even after one fails and names each red one at the end.
The codebase's health numbers — size, complexity, test ratio, gate weight, comment noise — come from
`python scripts/audit_metrics.py` (seconds; it reports, it never fails). `--speed` adds the speed
@@ -85,6 +84,13 @@ docker compose -f backend/docker-compose.test.yml run --rm --no-deps test \
pip-audit -r requirements.lock --no-deps --disable-pip
```
**Shell scripts**
```bash
# every tracked *.sh at shellcheck's default bar (info included); the version is pinned in scripts/check.sh
git ls-files -z -- '*.sh' | xargs -0 shellcheck
```
A missing database or ffmpeg is a hard error, not a skip — so the gate can't print green while the
DB-backed tests were quietly not run (`SIFTLODE_TEST_REQUIRE_DB=0` is the explicit "pure-logic only"
run for a machine without Postgres). Tests that need a database take the `db` fixture
@@ -107,6 +113,9 @@ Coverage is on demand, never enforced and not in the gate (E35): `npm run test -
`frontend/`, and for the backend
`docker compose -f backend/docker-compose.test.yml run --rm test python -m pytest --cov=app` (then
`… down -v`).
The DB lane runs on every core (`pytest -n auto`, pytest-xdist): `tests/db_per_worker.py` gives each worker its own
database (`siftlode_test_gw<N>`, created and migrated on first use), so the same tests run side by side. A serial
run (no `-n`) still works and uses `siftlode_test` itself.
The Playwright suite (`npm run e2e`) runs against the `:8080` stack built from your tree; the release
requires it. Before an image is pushed, it is scanned with Trivy (HIGH/CRITICAL with a fix fails;
exceptions in `.trivyignore.yaml`, each with a reason and an expiry).
+6 -2
View File
@@ -44,7 +44,9 @@ The installer asks for the **public URL** where the instance will be reached (ju
local `.env` file — keep that file private. **Do not change `TOKEN_ENCRYPTION_KEY` once the app has
stored any secret** (Google credentials, API keys, SMTP password, Plex token): those values are
encrypted with it, and a changed key makes them unreadable. If it is ever lost or rotated, clear the
affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page,
affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page.
Share links are kept with it too: after a key change every existing link still opens, but its
address can no longer be shown — its owner makes a new link to share it again,
- pulls the image and starts the app + database,
- prints the **setup wizard URL**, which looks like `…/setup?token=…`.
@@ -163,7 +165,9 @@ then cannot read its own secrets.
2. **`.env`** — in particular `TOKEN_ENCRYPTION_KEY`. Your Google client secret, SMTP password, Plex
token and every stored refresh token are encrypted with it *inside* the database. Restore a dump
against a different key and those values are lost — you would have to clear the affected rows from
`system_config` and re-enter every secret by hand. Keep `.env` beside the dump, and just as private.
`system_config` and re-enter every secret by hand — and no existing share link's address can be
shown again (the links still open; their owners make new ones to share). Keep `.env` beside the
dump, and just as private.
3. **Downloaded media** *(only if you use Files)* — the `siftlode_files` volume, or
whatever `FILES_HOST_PATH` points at. Not needed to bring the instance back; the jobs and their
metadata are in the database, the files themselves are not.
+5
View File
@@ -45,6 +45,11 @@ login-free `/watch/<token>` share link).
PW_BASE_URL=http://localhost:5173 siftlode e2e
```
**Accessibility** — `a11y.spec.ts` runs axe-core's WCAG 2.x A + AA rules (`@axe-core/playwright`)
over every rail page, every admin page and the player, in `chromium` only (the rules read the DOM,
which the engines build alike). It does not judge colour contrast: axe files text on a gradient —
every glass surface — as "incomplete", not as a pass or a hit.
## Account & fixture
Tests run as a dedicated **admin** account `e2e@siftlode.local` (never a real account;
+63
View File
@@ -0,0 +1,63 @@
import AxeBuilder from "@axe-core/playwright";
import { test, expect, type Page } from "@playwright/test";
import { gotoPage, openApp, openFirstVideo } from "./helpers";
// Accessibility, the machine-checkable half: axe-core's WCAG 2.x A + AA rules over every page the
// owner reaches and the player. A hit fails with the rule, its impact and the elements.
//
// What it does NOT cover — colour contrast, mostly: axe cannot compute a contrast over a gradient,
// and the glass surfaces are gradients, so it files most text as "incomplete" instead of judging it
// (measured 2026-10-17: 110 nodes on the feed, 9-32 on the other pages). A green here says nothing
// about contrast; that stays with the visual check.
//
// Chromium only (playwright.config.ts leaves it out of the other projects).
const TAGS = ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"];
/** Wait until nothing is loading: no QueryState spinner and no skeleton bone left. A bone says it is
* one (`data-skeleton`): a pulse alone is not loading — a running job's dot pulses as long as it runs. */
async function settled(page: Page): Promise<void> {
await expect(page.locator('[role="status"] .animate-spin, [data-skeleton]')).toHaveCount(0);
}
async function violations(page: Page): Promise<string[]> {
const result = await new AxeBuilder({ page }).withTags(TAGS).analyze();
return result.violations.map((v) => {
const shown = v.nodes.slice(0, 5).map((n) => n.target.join(" "));
const more = v.nodes.length > 5 ? ` (+${v.nodes.length - 5})` : "";
return `${v.id} (${v.impact ?? "?"}): ${v.help} — ${shown.join(" | ")}${more}`;
});
}
test.describe("accessibility (axe)", () => {
const RAIL = ["feed", "channels", "playlists", "files", "stats", "settings"];
const ADMIN = ["scheduler", "config", "users", "activity"];
for (const id of RAIL) {
test(`page: ${id}`, async ({ page }) => {
await openApp(page);
await gotoPage(page, id);
await settled(page);
expect(await violations(page)).toEqual([]);
});
}
for (const id of ADMIN) {
test(`admin page: ${id}`, async ({ page }) => {
await openApp(page);
await page.getByTestId("nav-admin").click();
await page.getByTestId(`nav-admin-${id}`).click();
await expect(page.getByTestId("nav-admin")).toHaveAttribute("aria-current", "page");
await settled(page);
expect(await violations(page)).toEqual([]);
});
}
test("player", async ({ page }) => {
// hermetic: the YouTube iframe is not loaded
await page.route(/youtube\.com/, (route) => route.abort());
await openApp(page);
await openFirstVideo(page);
await settled(page);
expect(await violations(page)).toEqual([]);
});
});
+1 -1
View File
@@ -11,7 +11,7 @@ test.describe("page switching", () => {
// and it failed on every run for a while without anyone seeing it, because e2e was in no
// required gate (it is now part of `release`).
const RAIL = ["channels", "playlists", "files", "stats", "settings", "feed"];
const ADMIN = ["scheduler", "config", "users", "audit"];
const ADMIN = ["scheduler", "config", "users", "activity"];
test("each rail item activates its page", async ({ page }) => {
await openApp(page);
+26 -13
View File
@@ -17,7 +17,7 @@ import { retryConnect } from "./retryConnect";
const RUNS = 5;
const OUT = "e2e/.speed/latest.json";
const CONTAINER = process.env.E2E_DB_CONTAINER || "siftlode-db-1";
const CONTAINER = process.env.E2E_API_CONTAINER || "siftlode-api-1";
const median = (xs: number[]) => [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)] ?? NaN;
@@ -32,20 +32,33 @@ const scriptBytes = (page: Page) =>
/** Public share links on the local stack that could play: no password, not expired, asset ready.
* The page itself is the judge of the last step (its file must be on THIS disk — a localdev DB
* seeded from prod lists assets whose files never came along), so the caller probes them. */
* seeded from prod lists assets whose files never came along), so the caller probes them. Read
* through the API container: the database keeps a link's token only hashed and encrypted, and only
* the app holds the key to read it back (`links.recover`). */
const SHARE_TOKENS_PY = `
from sqlalchemy import select
from app.db import SessionLocal
from app.downloads import links
from app.models import DownloadJob, DownloadLink, MediaAsset
with SessionLocal() as db:
for link in db.execute(
select(DownloadLink)
.join(DownloadJob, DownloadJob.id == DownloadLink.job_id)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadLink.password_hash.is_(None), MediaAsset.status == "ready")
.order_by(DownloadLink.id)
).scalars():
token = None if links.is_expired(link) else links.recover(link)
if token:
print(token)
`;
function shareTokens(): string[] {
const sql =
"select l.token from download_links l join download_jobs j on j.id = l.job_id" +
" join media_assets a on a.id = j.asset_id" +
" where l.password_hash is null and (l.expires_at is null or l.expires_at > now())" +
" and a.status = 'ready' order by l.id";
// Not caught: a database this cannot reach (no docker, a renamed container) is a failed
// Not caught: an API container this cannot reach (no docker, a renamed container) is a failed
// measurement, not "no playable share link on the local stack".
const out = execFileSync(
"docker",
["exec", CONTAINER, "psql", "-U", "siftlode", "-d", "siftlode", "-tAc", sql],
{ encoding: "utf8" },
);
const out = execFileSync("docker", ["exec", CONTAINER, "python", "-c", SHARE_TOKENS_PY], {
encoding: "utf8",
});
return out
.split("\n")
.map((t) => t.trim())
+7
View File
@@ -627,6 +627,13 @@ export default tseslint.config(
message:
"No native <select>: use ui/SegmentedToggle for a short fixed set, ui/FilterSelect for a longer one or a user's own list.",
},
// A picture that failed to load left the browser's broken-image mark on a card: a bare <img>
// sends a Referer Google's image CDNs refuse, and has no retry and no fallback.
{
selector: "JSXOpeningElement[name.name='img']",
message:
"No bare <img>: use ui/RemoteImage — no Referer, a retry past the browser's negative cache, and a fallback instead of the broken-image mark.",
},
// Keep the rem system from rotting. A px-frozen `text-[Npx]` font size does NOT
// scale with the Settings --font-scale slider while its rem neighbours do; use
// `text-2xs`/`text-3xs` or a `text-[Nrem]` arbitrary instead. Matches both plain strings and
+27 -3
View File
@@ -23,6 +23,7 @@
"react-i18next": "^15.7.4"
},
"devDependencies": {
"@axe-core/playwright": "4.13.0",
"@babel/core": "7.29.7",
"@eslint-community/eslint-plugin-eslint-comments": "4.8.1",
"@eslint/js": "9.39.1",
@@ -91,6 +92,19 @@
"dev": true,
"license": "ISC"
},
"node_modules/@axe-core/playwright": {
"version": "4.13.0",
"resolved": "https://registry.npmjs.org/@axe-core/playwright/-/playwright-4.13.0.tgz",
"integrity": "sha512-6YLx+kxXu5GJceG4ozFg+33a2EMTdjYwWGloJ3sb9Kta5pp+ZNS53uxGVog5JetIY8s++P5UrtX+cri+u0VAVg==",
"dev": true,
"license": "MPL-2.0",
"dependencies": {
"axe-core": "~4.13.0"
},
"peerDependencies": {
"playwright-core": ">= 1.0.0"
}
},
"node_modules/@babel/code-frame": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.7.tgz",
@@ -3631,6 +3645,16 @@
"dev": true,
"license": "MIT"
},
"node_modules/axe-core": {
"version": "4.13.0",
"resolved": "https://registry.npmjs.org/axe-core/-/axe-core-4.13.0.tgz",
"integrity": "sha512-UzGt8zg7Ny8djbYMhxl2zuEevVa7r2gJjYY5Lwr1xM7+XU2nd6CkIWFTVcCIbAP63vSz71NaVyyuSk9lHKcy0A==",
"dev": true,
"license": "MPL-2.0",
"engines": {
"node": ">=4"
}
},
"node_modules/babel-plugin-react-compiler": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/babel-plugin-react-compiler/-/babel-plugin-react-compiler-1.0.0.tgz",
@@ -6163,9 +6187,9 @@
}
},
"node_modules/source-map-js": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
"integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
"version": "1.2.2",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz",
"integrity": "sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==",
"dev": true,
"license": "BSD-3-Clause",
"engines": {
+1
View File
@@ -39,6 +39,7 @@
"react-i18next": "^15.7.4"
},
"devDependencies": {
"@axe-core/playwright": "4.13.0",
"@babel/core": "7.29.7",
"@eslint-community/eslint-plugin-eslint-comments": "4.8.1",
"@eslint/js": "9.39.1",
+5
View File
@@ -17,6 +17,9 @@ import { defineConfig, devices } from "@playwright/test";
const BASE_URL = process.env.PW_BASE_URL || "http://127.0.0.1:8080";
const STORAGE_STATE = "e2e/.auth/state.json";
// The axe scan reads the DOM and the accessibility tree, which the engines build alike: one engine
// answers for all, so the other projects leave it out instead of tripling its time.
const CHROMIUM_ONLY = /a11y\.spec\.ts/;
export default defineConfig({
testDir: "./e2e",
@@ -54,11 +57,13 @@ export default defineConfig({
{
name: "webkit",
use: { ...devices["Desktop Safari"], storageState: STORAGE_STATE },
testIgnore: CHROMIUM_ONLY,
dependencies: ["setup"],
},
{
name: "ipad",
use: { ...devices["iPad (gen 7) landscape"], storageState: STORAGE_STATE },
testIgnore: CHROMIUM_ONLY,
dependencies: ["setup"],
},
// The speed baseline (e2e/speed.measure.ts): readings, not tests. Only when PW_SPEED is set —
+3 -2
View File
@@ -16,7 +16,7 @@ import {
type PanelLayout,
} from "./lib/panelLayout";
import { notify, removeByMetaKind } from "./lib/notifications";
import { LS, setTabRaw } from "./lib/storage";
import { LS } from "./lib/storage";
import { usePref } from "./lib/usePref";
import { useVerifyEntry } from "./lib/useVerifyEntry";
import { useNavigation, useNavigationActions } from "./components/navigationContext";
@@ -46,6 +46,7 @@ import ErrorDialog from "./components/ErrorDialog";
import VersionBanner from "./components/VersionBanner";
import VerifyBanner from "./components/VerifyBanner";
import { focusAccessRequestsTab } from "./lib/adminUsersTab";
import { focusSettingsTab } from "./lib/settingsTab";
// The built app version, NOT imported from releaseNotes — importing anything from that module
// pulls the release-notes loader into the eager App chunk and defeats the lazy ReleaseNotes
// split. FRONTEND_VERSION equals the newest release note's version by release convention (both bump together).
@@ -261,7 +262,7 @@ export default function App() {
if (result === "ok") {
notify({ level: "success", message: t("settings.account.googleLink.linked") });
void refetchMe();
setTabRaw(LS.settingsTab, "account");
focusSettingsTab("account");
setPage("settings");
} else {
const key = result === "conflict" || result === "mismatch" ? result : "error";
+7 -15
View File
@@ -9,13 +9,13 @@ import { displayName } from "../lib/api/account";
import { qk } from "../lib/queryKeys";
import { notify } from "../lib/notifications";
import { notifyActionError } from "../lib/actionErrors";
import { copyText } from "../lib/clipboard";
import { formatDateTime } from "../lib/format";
import { invalidateUserAccess } from "../lib/adminUsers";
import type { AdminUserAccess, AdminUserRow } from "../lib/api/admin";
import { Button } from "./ui/Button";
import { SURFACE } from "./ui/surfaces";
import { SectionLabel } from "./ui/Headings";
import { useCopyFeedback } from "./useCopyFeedback";
// The admin's answer to "why can this person not get in?", and the place every recovery action for
// one account lives.
@@ -59,7 +59,7 @@ export default function AccountPanel({
const lang = i18n.language;
const qc = useQueryClient();
const [link, setLink] = useState<{ url: string; expires_at: string | null } | null>(null);
const [copied, setCopied] = useState(false);
const { copied, copy: copyValue } = useCopyFeedback();
const q = useQuery({
queryKey: qk.adminUserAccess(user.id),
@@ -88,19 +88,11 @@ export default function AccountPanel({
onError: (err) => notifyActionError(err, t("users.pendingAccess.activateFailed")),
});
const copy = async () => {
if (!link) return;
// Via the shared helper, whose synchronous `execCommand` path is what makes copying work on a
// plain-http LAN origin — where `navigator.clipboard` is undefined, and which is exactly the
// minimal self-host this no-email link was built for.
if (await copyText(link.url)) {
setCopied(true);
window.setTimeout(() => setCopied(false), 2000);
return;
}
// The link is on screen and selectable, so a failed copy is a convenience that failed — not the
// action failing. Say so rather than leaving the button looking inert.
notify({ level: "warning", message: t("users.account.copyFailed") });
// Via the shared helper, whose synchronous `execCommand` path is what makes copying work on a
// plain-http LAN origin — where `navigator.clipboard` is undefined, and which is exactly the
// minimal self-host this no-email link was built for.
const copy = () => {
if (link) void copyValue(link.url);
};
const a: AdminUserAccess | undefined = q.data;
@@ -0,0 +1,446 @@
// @vitest-environment jsdom
//
// The activity page around its run and audit lists (the server-log tab has its own file): the view
// is the URL — a link narrows the runs to one job, a status chip narrows them further; a run opens
// to its log lines and an admin action to its request's, each a new history entry that Back undoes;
// the account filter reaches the quota tab; a
// failed read is an error with a retry, never an empty list.
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { act, cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import type { AuditQuery, AuditRow, HistoryPage, LogQuery, RunQuery, RunRow } from "../lib/api";
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
return {
...actual,
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
};
});
const runsMock = vi.fn<(q: RunQuery, before: string | null) => Promise<HistoryPage<RunRow>>>();
const auditMock = vi.fn<(q: AuditQuery, before: string | null) => Promise<HistoryPage<AuditRow>>>();
const logsMock = vi.fn<(q: LogQuery, before: string | null) => Promise<unknown>>();
const quotaMock = vi.fn<(since: string, user: string | null) => Promise<unknown>>();
const runFacetsMock = vi.fn<() => Promise<unknown>>();
const usersMock = vi.fn<() => Promise<unknown>>();
vi.mock("../lib/api", async (importOriginal) => {
const actual = await importOriginal<typeof import("../lib/api")>();
return {
...actual,
api: {
...actual.api,
adminRuns: (q: RunQuery, before: string | null) => runsMock(q, before),
adminRunFacets: () => runFacetsMock(),
adminAudit: (q: AuditQuery, before: string | null) => auditMock(q, before),
adminAuditFacets: () => Promise.resolve({ actions: [] }),
adminLogs: (q: LogQuery, before: string | null) => logsMock(q, before),
adminLogFacets: () =>
Promise.resolve({ levels: [], sources: [], events: [], users: [], accounts: {}, total: 0 }),
adminUsers: () => usersMock(),
adminQuota: (since: string, user: string | null) => quotaMock(since, user),
clearAudit: () => Promise.resolve({ ok: true, cleared: 1 }),
},
};
});
import { qk } from "../lib/queryKeys";
import Activity from "./Activity";
import { ChoiceContext } from "./confirmContext";
import { PageToolbarSlot } from "./ui/pageToolbarSlot";
// The filter bar lives in the toolbar, which portals into the shell's band: give it one.
const slot = document.body.appendChild(document.createElement("div"));
const RUN: RunRow = {
id: 41,
job: "backfill",
trigger: "manual",
actor_id: 1,
target: null,
target_title: null,
status: "error",
started_at: "2026-10-18T02:31:04Z",
finished_at: "2026-10-18T02:31:45Z",
summary: null,
result: null,
error: "quotaExceeded",
failure: null,
quota_units: 212,
log_lines: 9,
};
const ACTION: AuditRow = {
id: 5,
created_at: "2026-10-18T02:27:00Z",
actor_id: 1,
action: "scheduler_interval",
target_type: "job",
target_id: "rss_poll",
summary: "rss_poll → 25 min/run",
before: null,
after: { interval_minutes: 25 },
request_id: "0efdd960c9ad1977",
log_lines: 4,
};
const page = <T,>(items: T[]): HistoryPage<T> => ({
items,
accounts: { "1": "admin@example.com" },
next_cursor: null,
});
function renderPage(qc = new QueryClient({ defaultOptions: { queries: { retry: false } } })) {
return render(
<QueryClientProvider client={qc}>
{/* Every confirmation is answered yes. */}
<ChoiceContext.Provider value={() => Promise.resolve("confirm")}>
<PageToolbarSlot.Provider value={slot}>
<Activity />
</PageToolbarSlot.Provider>
</ChoiceContext.Provider>
</QueryClientProvider>,
);
}
const url = () => new URLSearchParams(window.location.search);
const rowButton = (text: RegExp) =>
screen
.getAllByRole("button")
.find((b) => b.hasAttribute("aria-expanded") && text.test(b.textContent ?? ""))!;
beforeEach(() => {
runsMock.mockResolvedValue(page([RUN]));
auditMock.mockResolvedValue(page([ACTION]));
logsMock.mockResolvedValue({ items: [], accounts: {}, next_cursor: null });
quotaMock.mockResolvedValue({ rows: [], daily: [] });
runFacetsMock.mockResolvedValue({ jobs: [], statuses: [], total: 0 });
usersMock.mockResolvedValue([{ id: 1, email: "admin@example.com" }]);
});
afterEach(() => {
cleanup();
vi.clearAllMocks();
});
describe("Activity", () => {
it("a link narrows the runs to one job, and a status chip narrows them further", async () => {
window.history.replaceState(null, "", "/?page=activity&rjob=backfill");
renderPage();
await waitFor(() =>
expect(runsMock).toHaveBeenCalledWith(
expect.objectContaining({ job: "backfill", status: undefined, since: expect.any(String) }),
null,
),
);
fireEvent.click(screen.getByRole("button", { name: /activity\.runs\.status\.error/ }));
await waitFor(() =>
expect(runsMock).toHaveBeenLastCalledWith(
expect.objectContaining({ job: "backfill", status: "error" }),
null,
),
);
expect([url().get("rjob"), url().get("rst")]).toEqual(["backfill", "error"]);
});
it("Older reads the next page from its cursor, and offers none past the last", async () => {
window.history.replaceState(null, "", "/?page=activity");
runsMock.mockImplementation((_q, before) =>
Promise.resolve(
before === null
? { ...page([RUN]), next_cursor: "c1" }
: page([{ ...RUN, id: 40, error: "the older one" }]),
),
);
renderPage();
fireEvent.click(await screen.findByRole("button", { name: "activity.older" }));
await waitFor(() => expect(screen.getByText("the older one")).toBeTruthy());
expect(runsMock).toHaveBeenLastCalledWith(expect.anything(), "c1");
expect(screen.getByText("quotaExceeded")).toBeTruthy();
expect(screen.queryByRole("button", { name: "activity.older" })).toBeNull();
});
it("an older page that fails says so and keeps what was loaded", async () => {
window.history.replaceState(null, "", "/?page=activity");
runsMock.mockImplementation((_q, before) =>
before === null
? Promise.resolve({ ...page([RUN]), next_cursor: "c1" })
: Promise.reject(new Error("500")),
);
renderPage();
fireEvent.click(await screen.findByRole("button", { name: "activity.older" }));
const alert = await screen.findByRole("alert");
expect(alert.textContent).toContain("activity.olderFailed");
expect(screen.getByText("quotaExceeded")).toBeTruthy();
});
it("offers no older page while a filter change is still loading", async () => {
window.history.replaceState(null, "", "/?page=activity");
// The cursor on screen is the previous filter's: it would continue the wrong list.
runsMock.mockImplementation((q) =>
q.status ? new Promise(() => {}) : Promise.resolve({ ...page([RUN]), next_cursor: "c1" }),
);
renderPage();
await screen.findByRole("button", { name: "activity.older" });
fireEvent.click(screen.getByRole("button", { name: /activity\.runs\.status\.error/ }));
await waitFor(() =>
expect(screen.queryByRole("button", { name: "activity.older" })).toBeNull(),
);
expect(screen.getByText("quotaExceeded")).toBeTruthy(); // the old rows stay, dimmed
});
it("a run opens to its log lines, and Back returns to the runs", async () => {
window.history.replaceState({ sfPage: "activity" }, "", "/?page=activity&aw=24h&aq=quota");
renderPage();
await waitFor(() => expect(screen.getByText("quotaExceeded")).toBeTruthy());
fireEvent.click(rowButton(/quotaExceeded/));
fireEvent.click(screen.getByRole("button", { name: "activity.runs.logLines" }));
await waitFor(() =>
expect(logsMock).toHaveBeenLastCalledWith(
expect.objectContaining({ run_id: 41, q: undefined }),
null,
),
);
// Every other filter is off, and the range is the log's whole retention.
expect([url().get("at"), url().get("lrun"), url().get("aw"), url().get("aq")]).toEqual([
"logs",
"41",
"90d",
null,
]);
await act(async () => {
const landed = new Promise((resolve) =>
window.addEventListener("popstate", resolve, { once: true }),
);
window.history.back();
await landed;
});
await waitFor(() => expect(url().get("at")).toBeNull());
expect([url().get("aw"), url().get("aq"), url().get("lrun")]).toEqual(["24h", "quota", null]);
expect(screen.getByText("quotaExceeded")).toBeTruthy();
});
it("a run's outcome is its result in the reader's words, its failure in the danger colour", async () => {
runsMock.mockResolvedValue(
page<RunRow>([
{ ...RUN, id: 1, status: "ok", error: null, result: { new: 3 }, summary: "new=3" },
// Moved over from the audit log before the result rode along as data: the one-liner only.
{ ...RUN, id: 2, status: "ok", error: null, result: null, summary: "failed=2" },
{ ...RUN, id: 3, status: "interrupted", error: null },
{ ...RUN, id: 4 },
]),
);
renderPage();
await waitFor(() => expect(screen.getByText("scheduler.result.new: 3")).toBeTruthy());
expect(screen.queryByText("new=3")).toBeNull();
expect(screen.getByText("scheduler.result.failed: 2")).toBeTruthy();
expect(screen.getByText("scheduler.interrupted").className).toContain("text-warn");
expect(screen.getByText("quotaExceeded").className).toContain("text-danger");
});
it("filter choices that failed to load say so beside the filters, with a retry", async () => {
runFacetsMock.mockRejectedValueOnce(new Error("500"));
usersMock.mockRejectedValueOnce(new Error("500"));
renderPage();
await waitFor(() =>
expect(screen.getAllByRole("alert").map((a) => a.textContent)).toEqual([
expect.stringContaining("activity.accountsFailed"),
expect.stringContaining("activity.filtersFailed"),
]),
);
for (const retry of screen.getAllByRole("button", { name: "common.retry" }))
fireEvent.click(retry);
await waitFor(() => expect(screen.queryAllByRole("alert")).toEqual([]));
expect(runFacetsMock).toHaveBeenCalledTimes(2);
expect(usersMock).toHaveBeenCalledTimes(2);
});
it("accounts that could not be refreshed, still listed, are not called unloaded", async () => {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: 0 } } });
qc.setQueryData(qk.adminUsers(), [{ id: 1, email: "admin@example.com" }]);
usersMock.mockRejectedValue(new Error("502"));
renderPage(qc);
await waitFor(() => expect(usersMock).toHaveBeenCalled());
await waitFor(() => expect(qc.getQueryState(qk.adminUsers())?.status).toBe("error"));
expect(screen.queryByText("activity.accountsFailed")).toBeNull();
});
it("clearing the admin actions reads the list again", async () => {
window.history.replaceState(null, "", "/?page=activity&at=admin");
renderPage();
await waitFor(() => expect(screen.getByText("auditActions.scheduler_interval")).toBeTruthy());
auditMock.mockResolvedValue(page<AuditRow>([]));
fireEvent.click(screen.getByRole("button", { name: "audit.clear" }));
await waitFor(() => expect(screen.getByText("audit.empty")).toBeTruthy());
// An emptied list is announced, as the error and loading states are.
expect(screen.getByText("audit.empty").closest('[role="status"]')).toBeTruthy();
});
it("closing a dialog over the server log leaves Live on", async () => {
window.history.replaceState(null, "", "/?page=activity&at=logs");
renderPage();
const live = await screen.findByRole("button", { name: "adminLogs.live" });
fireEvent.click(live);
expect(live.getAttribute("aria-pressed")).toBe("true");
// A closing dialog steps back over its own history entry: a popstate onto the same address.
act(() => {
window.dispatchEvent(new PopStateEvent("popstate"));
});
const after = screen.getByRole("button", { name: "adminLogs.live" });
expect(after.getAttribute("aria-pressed")).toBe("true");
expect(after).toBe(live); // the same log, not a fresh one
});
it("Back/Forward on the server log reads only the filters of the entry it lands on", async () => {
// A browser runs the microtasks between two popstate listeners, so each listener's update
// renders on its own; a test dispatch does not. Each listener runs in its own act here.
const listeners: EventListener[] = [];
const addOther = window.addEventListener.bind(window);
const removeOther = window.removeEventListener.bind(window);
const add = vi.spyOn(window, "addEventListener").mockImplementation((type, l, o) => {
if (type === "popstate") listeners.push(l as EventListener);
else addOther(type, l, o);
});
const remove = vi.spyOn(window, "removeEventListener").mockImplementation((type, l, o) => {
if (type === "popstate") listeners.splice(listeners.indexOf(l as EventListener), 1);
else removeOther(type, l, o);
});
window.history.replaceState(null, "", "/?page=activity&at=logs&aw=24h");
renderPage();
await waitFor(() => expect(logsMock).toHaveBeenCalled());
logsMock.mockClear();
// Forward onto a pivot's entry: one run's lines over the log's whole retention.
window.history.pushState(null, "", "/?page=activity&at=logs&aw=90d&lrun=41");
for (const l of [...listeners]) {
await act(async () => l(new PopStateEvent("popstate")));
}
await waitFor(() =>
expect(logsMock).toHaveBeenLastCalledWith(expect.objectContaining({ run_id: 41 }), null),
);
const days = (since?: string) => (Date.now() - Date.parse(since ?? "")) / 86_400_000;
// Never one run's lines over a day, nor the whole log over 90 days.
expect(logsMock.mock.calls.map(([q]) => [q.run_id ?? null, days(q.since) > 30])).toEqual(
logsMock.mock.calls.map(() => [41, true]),
);
add.mockRestore();
remove.mockRestore();
});
it("an action whose request's lines are gone says so instead of linking to an empty list", async () => {
auditMock.mockResolvedValue(page([{ ...ACTION, log_lines: 0 }]));
window.history.replaceState(null, "", "/?page=activity&at=admin");
renderPage();
await waitFor(() => expect(screen.getByText("auditActions.scheduler_interval")).toBeTruthy());
fireEvent.click(rowButton(/scheduler_interval/));
expect(screen.getByText("activity.audit.noRequestLines")).toBeTruthy();
expect(screen.queryByRole("button", { name: "activity.audit.requestLines" })).toBeNull();
});
it("the admin tab lists the actions, and an action opens its request's lines", async () => {
window.history.replaceState(null, "", "/?page=activity&at=admin");
renderPage();
await waitFor(() => expect(screen.getByText("auditActions.scheduler_interval")).toBeTruthy());
expect(runsMock).not.toHaveBeenCalled();
fireEvent.click(rowButton(/auditActions\.scheduler_interval/));
fireEvent.click(screen.getByRole("button", { name: "activity.audit.requestLines" }));
await waitFor(() =>
expect(logsMock).toHaveBeenLastCalledWith(
expect.objectContaining({ request_id: "0efdd960c9ad1977" }),
null,
),
);
expect(url().get("lreq")).toBe("0efdd960c9ad1977");
});
it("the account filter reaches the quota tab, over the page's range", async () => {
window.history.replaceState(null, "", "/?page=activity&at=quota&au=none&aw=30d");
renderPage();
await waitFor(() => expect(quotaMock).toHaveBeenCalled());
const [since, user] = quotaMock.mock.calls[0]!;
expect(user).toBe("none");
const days = (Date.now() - new Date(since).getTime()) / 86_400_000;
expect(Math.round(days)).toBe(30);
});
it("each day's spend is readable without a pointer", async () => {
window.history.replaceState(null, "", "/?page=activity&at=quota");
quotaMock.mockResolvedValue({
rows: [{ user_id: null, email: "System", total: 9, by_action: {} }],
daily: [
{ day: "2026-10-16", total: 4 },
{ day: "2026-10-17", total: 5 },
],
});
renderPage();
const chart = await screen.findByRole("list", { name: "activity.quota.daily" });
expect(
within(chart)
.getAllByRole("listitem")
.map((d) => d.textContent),
).toEqual(["stats.dailyTooltip", "stats.dailyTooltip"]);
});
// The runs moved over from the audit log kept one instant for start and end: their length is
// unknown, and a "1 sec" would be invented.
it("a run whose start and end are one instant has no length", async () => {
window.history.replaceState(null, "", "/?page=activity");
runsMock.mockResolvedValue(
page([
{ ...RUN, id: 1, status: "ok", error: null, finished_at: RUN.started_at, summary: "moved" },
{ ...RUN, id: 2, status: "ok", error: null, summary: "timed" },
]),
);
renderPage();
await waitFor(() => expect(screen.getByText("moved")).toBeTruthy());
const cells = (text: string) =>
[...rowButton(new RegExp(text)).children].map((c) => c.textContent);
expect(cells("moved")).toContain("—");
expect(cells("timed")).not.toContain("—");
});
// A bare "212" in a row reads as nothing in particular: the spend carries its unit, and a run that
// spent none says nothing rather than "0".
it("a run's quota spend reads with its unit, and none spent leaves the cell blank", async () => {
window.history.replaceState(null, "", "/?page=activity");
runsMock.mockResolvedValue(
page([
{ ...RUN, id: 1, status: "ok", error: null, summary: "spent" },
{ ...RUN, id: 2, status: "ok", error: null, summary: "free", quota_units: 0 },
]),
);
renderPage();
await waitFor(() => expect(screen.getByText("spent")).toBeTruthy());
const cells = (text: string) =>
[...rowButton(new RegExp(text)).children].map((c) => c.textContent);
expect(cells("spent")).toContain("activity.runs.units");
expect(cells("spent")).not.toContain("212");
expect(cells("free")).not.toContain("0");
expect(cells("free")).not.toContain("activity.runs.units");
});
// Most runs store no line at all: a link there would only lead to an empty list.
it("a run that logged nothing says so instead of linking to an empty list", async () => {
window.history.replaceState(null, "", "/?page=activity");
runsMock.mockResolvedValue(page([{ ...RUN, log_lines: 0 }]));
renderPage();
await waitFor(() => expect(screen.getByText("quotaExceeded")).toBeTruthy());
fireEvent.click(rowButton(/quotaExceeded/));
expect(screen.getByText("activity.runs.noLogLines")).toBeTruthy();
expect(screen.queryByRole("button", { name: "activity.runs.logLines" })).toBeNull();
});
it("a failed read is an error with a retry, not an empty list", async () => {
window.history.replaceState(null, "", "/?page=activity");
runsMock.mockRejectedValue(new Error("502"));
renderPage();
expect(await screen.findByRole("alert")).toBeTruthy();
expect(screen.queryByText("activity.runs.empty")).toBeNull();
});
});
+214
View File
@@ -0,0 +1,214 @@
import clsx from "clsx";
import { useEffect, useState } from "react";
import { useTranslation } from "react-i18next";
import { useQuery } from "@tanstack/react-query";
import { Search } from "lucide-react";
import { api } from "../lib/api";
import {
ACTIVITY_TABS,
ACTIVITY_WINDOWS,
activityParamsToState,
activityParamUpdates,
type ActivityTab,
type ActivityUrlState,
type ActivityWindow,
} from "../lib/activityUrlState";
import { nextNavId, notifyNav, rememberNavPosition } from "../lib/history";
import {
DEFAULT_LOGS_STATE,
logsParamsToState,
logsParamUpdates,
type LogsUrlState,
} from "../lib/logsUrlState";
import { qk } from "../lib/queryKeys";
import { useDebounced } from "../lib/useDebounced";
import { urlWithParams } from "../lib/urlState";
import { useUrlMirror } from "../lib/useUrlMirror";
import ActivityAudit from "./ActivityAudit";
import ActivityQuota from "./ActivityQuota";
import ActivityRuns from "./ActivityRuns";
import AdminLogs, { type LogPivot } from "./AdminLogs";
import FilterChipGroup from "./ui/FilterChipGroup";
import FilterSelect from "./ui/FilterSelect";
import { InnerInput } from "./ui/Input";
import { fieldClass } from "./ui/fieldStyles";
import { PageToolbar } from "./ui/PageShell";
import { StateMessage } from "./ui/QueryState";
import Tabs from "./ui/Tabs";
import { useTabPanel } from "./ui/tabPanel";
// Activity ("Napló"): what happened, in one place — the background runs, the admins' actions, the
// server's log and the quota spend, each its own table on the server, read here as four tabs under
// one filter bar (time range, account, text). The view lives in the URL, so it is a link; a pivot to
// one run's or one request's log lines is a new history entry, and Back returns to where it was made.
// A pivot shows everything the run or request logged: every other filter would only hide part of it,
// and the longest range is the log's own retention.
const PIVOT_SHARED: Pick<ActivityUrlState, "window" | "user" | "q"> = {
window: "90d",
user: null,
q: "",
};
export default function Activity() {
const { t } = useTranslation();
const [state, setState] = useState<ActivityUrlState>(() =>
activityParamsToState(new URLSearchParams(window.location.search)),
);
// Bumped when the view changes by a jump (a pivot, Back/Forward) rather than by typing: the search
// text then applies at once instead of after the typing debounce.
const [jumps, setJumps] = useState(0);
// Bumped by a pivot only: the server log, already open, takes the pivot's filters from the address
// bar when it mounts again. Back/Forward it follows in place — a popstate is also a dialog closing,
// onto the same address, and a fresh log would drop its Live and its focus.
const [pivots, setPivots] = useState(0);
// The log's half of the entry Back/Forward landed on, read here with this page's half: one listener,
// so both change in one render. Two listeners rendered in between (a browser runs the microtasks
// after each), and that render read the log with one entry's filters and the other's range.
const [landed, setLanded] = useState<{ logs: LogsUrlState } | null>(null);
const q = useDebounced(state.q, 300, jumps);
const panel = useTabPanel();
// `page` too: a link to a page this one replaced (audit, logs) is rewritten to this page's id.
useUrlMirror({ page: "activity", ...activityParamUpdates(state) });
useEffect(() => {
const onPop = () => {
const params = new URLSearchParams(window.location.search);
setJumps((n) => n + 1);
setState(activityParamsToState(params));
setLanded({ logs: logsParamsToState(params) });
};
window.addEventListener("popstate", onPop);
return () => window.removeEventListener("popstate", onPop);
}, []);
const update = (patch: Partial<ActivityUrlState>) => setState((s) => ({ ...s, ...patch }));
const pivot = (target: LogPivot) => {
const next: ActivityUrlState = { ...state, ...PIVOT_SHARED, tab: "logs" };
window.history.pushState(
{ ...rememberNavPosition(), sfScroll: nextNavId() },
"",
urlWithParams({
page: "activity",
...activityParamUpdates(next),
...logsParamUpdates({ ...DEFAULT_LOGS_STATE, ...target }),
}),
);
notifyNav();
setJumps((n) => n + 1);
setPivots((n) => n + 1);
setState(next);
};
const shared = { window: state.window, user: state.user, q };
// The account menu's list (SharedBar reads the same query): without it the menu offers only "Any".
const users = useQuery({ queryKey: qk.adminUsers(), queryFn: api.adminUsers });
return (
<div {...panel.panelProps(state.tab)} className="px-4 pb-4 pt-3 max-w-6xl w-full mx-auto">
<PageToolbar>
<div className="px-4 pt-2 max-w-6xl w-full mx-auto">
<Tabs
idBase={panel.idBase}
active={state.tab}
onChange={(tab) => update({ tab: tab as ActivityTab })}
tabs={ACTIVITY_TABS.map((id) => ({ id, label: t(`activity.tabs.${id}`) }))}
className="flex flex-wrap gap-1 pb-2"
/>
<SharedBar state={state} onChange={update} />
</div>
</PageToolbar>
{/* Only when the menu really lacks them: a refresh that failed keeps the list it had. */}
{users.isError && !users.data && (
<StateMessage tone="error" compact onRetry={() => void users.refetch()}>
{t("activity.accountsFailed")}
</StateMessage>
)}
{/* The quota tab says what its numbers are itself, beside them. */}
{state.tab !== "quota" && (
<p className="mb-3 text-xs leading-relaxed text-muted">
{t(`activity.intro.${state.tab}`)}
</p>
)}
{state.tab === "runs" ? (
<ActivityRuns
shared={shared}
job={state.job}
status={state.status}
onFilter={update}
onLogs={(runId) => pivot({ requestId: null, jobId: null, runId })}
/>
) : state.tab === "admin" ? (
<ActivityAudit
shared={shared}
action={state.action}
onAction={(action) => update({ action })}
onRequest={(requestId) => pivot({ requestId, jobId: null, runId: null })}
/>
) : state.tab === "logs" ? (
// Keyed by the pivots: a pivot hands the log its filters through the address bar, which it
// reads when it mounts.
<AdminLogs key={pivots} shared={shared} landed={landed} onPivot={pivot} />
) : (
<ActivityQuota shared={shared} />
)}
</div>
);
}
/** The filters every tab shares: the text (not on the quota tab — it has nothing to match), the time
* range, and the account. */
function SharedBar({
state,
onChange,
}: {
state: ActivityUrlState;
onChange: (patch: Partial<ActivityUrlState>) => void;
}) {
const { t } = useTranslation();
const users = useQuery({ queryKey: qk.adminUsers(), queryFn: api.adminUsers });
const known = users.data ?? [];
// The account in use stays offered even when the list does not have it (an erased account).
const kept =
state.user && state.user !== "none" && !known.some((u) => String(u.id) === state.user)
? [{ value: state.user, label: `#${state.user}` }]
: [];
return (
<div className="flex flex-wrap items-center gap-2 pb-2">
{state.tab !== "quota" && (
<label
className={clsx(
fieldClass({ size: "sm", pill: true }),
"flex min-w-[12rem] flex-1 items-center gap-1.5",
)}
>
<Search className="h-3.5 w-3.5 shrink-0 text-muted" aria-hidden />
<InnerInput
type="search"
value={state.q}
onChange={(e) => onChange({ q: e.target.value.slice(0, 200) })}
placeholder={t("activity.search")}
aria-label={t("activity.search")}
/>
</label>
)}
<FilterChipGroup
ariaLabel={t("activity.windowLabel")}
value={state.window}
onChange={(window) => onChange({ window: window as ActivityWindow })}
options={ACTIVITY_WINDOWS.map((w) => ({ value: w, label: t(`activity.windows.${w}`) }))}
/>
<FilterSelect
label={t("activity.account")}
options={[
{ value: "", label: t("activity.any") },
{ value: "none", label: t("activity.noAccount") },
...known.map((u) => ({ value: String(u.id), label: u.email })),
...kept,
]}
selected={[state.user ?? ""]}
onChange={([user]) => onChange({ user: user || null })}
testId="activity-user"
/>
</div>
);
}
+229
View File
@@ -0,0 +1,229 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { keepPreviousData, useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Trash2 } from "lucide-react";
import { api, type AuditQuery, type AuditRow } from "../lib/api";
import { notifyActionError } from "../lib/actionErrors";
import { sinceOf } from "../lib/activityUrlState";
import { auditActionLabel, auditText } from "../lib/auditText";
import { formatDateTimeSec } from "../lib/format";
import { notify } from "../lib/notifications";
import { qk } from "../lib/queryKeys";
import { useHistoryList } from "../lib/useHistoryList";
import type { Shared } from "./ActivityRuns";
import { useConfirm } from "./confirmContext";
import { StateMessage } from "./ui/QueryState";
import HistoryList from "./HistoryList";
import HistoryRow from "./HistoryRow";
import { RelativeTime } from "./ui/RelativeTime";
import { Button } from "./ui/Button";
import FilterSelect from "./ui/FilterSelect";
// Activity → Admin actions: the append-only who/what/when trail, newest first, filtered on the
// server. A row opens to its before/after and to the lines its request logged. Clear stays here,
// leaving one row that says who cleared it.
const GRID =
"grid-cols-[minmax(0,1fr)_auto] sm:grid-cols-[minmax(0,12rem)_minmax(0,12rem)_8rem_minmax(10rem,1fr)]";
export default function ActivityAudit({
shared,
action,
onAction,
onRequest,
}: {
shared: Shared;
action: string | null;
onAction: (action: string | null) => void;
/** Open the server log at this request's lines. */
onRequest: (requestId: string) => void;
}) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const qc = useQueryClient();
const confirm = useConfirm();
const [open, setOpen] = useState<number | null>(null);
const query: AuditQuery = {
user: shared.user ?? undefined,
q: shared.q.trim() || undefined,
action: action ?? undefined,
};
const key = { ...query, window: shared.window };
const state = useHistoryList(qk.adminAudit(key), (before) =>
api.adminAudit({ ...query, since: sinceOf(shared.window) }, before),
);
const facets = useQuery({
queryKey: qk.adminAuditFacets(key),
queryFn: () => api.adminAuditFacets({ ...query, since: sinceOf(shared.window) }),
placeholderData: keepPreviousData,
});
const actions = [
...(facets.data?.actions ?? []),
...(action && !facets.data?.actions.some((f) => f.value === action)
? [{ value: action, count: 0 }]
: []),
];
const clear = useMutation({
mutationFn: () => api.clearAudit(),
onSuccess: (r) => {
void qc.invalidateQueries({ queryKey: qk.adminAudit() });
void qc.invalidateQueries({ queryKey: qk.adminAuditFacets() });
notify({ level: "success", message: t("audit.cleared", { count: r.cleared }) });
},
onError: (e) => notifyActionError(e, t("audit.clearFailed")),
});
const onClear = async () => {
if (
await confirm({
title: t("audit.clearTitle"),
message: t("audit.clearConfirm"),
confirmLabel: t("audit.clear"),
danger: true,
})
)
clear.mutate();
};
const actor = (row: AuditRow) =>
row.actor_id === null ? null : (state.accounts[String(row.actor_id)] ?? `#${row.actor_id}`);
return (
<div className="flex flex-col gap-2">
<div className="flex flex-wrap items-center gap-2">
<FilterSelect
label={t("activity.audit.action")}
options={[
{ value: "", label: t("activity.any") },
...actions.map((f) => ({
value: f.value,
label: auditActionLabel(t, f.value),
count: f.count,
})),
]}
selected={[action ?? ""]}
onChange={([value]) => onAction(value || null)}
testId="audit-action"
/>
<Button
variant="outline-danger"
size="xs"
className="ml-auto"
onClick={() => void onClear()}
disabled={clear.isPending}
icon={<Trash2 className="h-3.5 w-3.5" />}
>
{t("audit.clear")}
</Button>
</div>
{/* The action menu above is built from these counts: without them it offers only "Any". */}
{facets.isError && (
<StateMessage tone="error" compact onRetry={() => void facets.refetch()}>
{t("activity.filtersFailed")}
</StateMessage>
)}
<HistoryList
state={state}
empty={t("audit.empty")}
count={t("activity.audit.count", { count: state.rows.length })}
>
{state.rows.map((row, index) => {
const { head, sub } = auditText(t, row, lang);
const who = actor(row);
return (
<HistoryRow
key={row.id}
navIndex={index}
navText={`${auditActionLabel(t, row.action)} ${head ?? ""}`}
grid={GRID}
open={open === row.id}
onToggle={() => setOpen(open === row.id ? null : row.id)}
title={formatDateTimeSec(row.created_at, lang)}
cells={
<>
<span className="truncate font-medium">{auditActionLabel(t, row.action)}</span>
<RelativeTime
at={row.created_at}
seconds
className="tabular-nums text-muted sm:hidden"
/>
<span
className={`hidden truncate sm:block ${who ? "text-muted" : "italic text-muted"}`}
>
{who ?? t("audit.system")}
</span>
<RelativeTime
at={row.created_at}
seconds
className="hidden tabular-nums text-muted sm:block"
/>
<span className="col-span-2 truncate text-muted sm:col-span-1">
{[head, sub].filter(Boolean).join(" · ")}
</span>
</>
}
>
<AuditDetail row={row} who={who} onRequest={onRequest} />
</HistoryRow>
);
})}
</HistoryList>
</div>
);
}
function AuditDetail({
row,
who,
onRequest,
}: {
row: AuditRow;
who: string | null;
onRequest: (requestId: string) => void;
}) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const { head, sub } = auditText(t, row, lang);
return (
<div className="flex flex-col gap-2">
<dl className="grid grid-cols-[max-content_minmax(0,1fr)] gap-x-3 gap-y-0.5">
<dt>{t("audit.cols.actor")}</dt>
<dd>{who ?? t("audit.system")}</dd>
<dt>{t("audit.cols.when")}</dt>
<dd className="tabular-nums">{formatDateTimeSec(row.created_at, lang)}</dd>
{(head || sub) && (
<>
<dt>{t("audit.cols.details")}</dt>
<dd className="wrap-break-word">{[head, sub].filter(Boolean).join(" · ")}</dd>
</>
)}
{row.target_id && (
<>
<dt>{t("activity.audit.target")}</dt>
<dd className="font-mono">{row.target_id}</dd>
</>
)}
</dl>
{(row.before || row.after) && (
<pre className="overflow-x-auto whitespace-pre font-mono">
{JSON.stringify({ before: row.before, after: row.after }, null, 2)}
</pre>
)}
{/* A request's lines are kept days, its row months: a link to none would open an empty list. */}
{row.request_id &&
(row.log_lines > 0 ? (
<div>
<Button
variant="link"
className="underline decoration-dotted underline-offset-2 hover:decoration-solid"
onClick={() => onRequest(row.request_id!)}
>
{t("activity.audit.requestLines")}
</Button>
</div>
) : (
<p>{t("activity.audit.noRequestLines")}</p>
))}
</div>
);
}
+153
View File
@@ -0,0 +1,153 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { keepPreviousData, useQuery } from "@tanstack/react-query";
import { api, type AdminQuotaRow } from "../lib/api";
import { sinceOf } from "../lib/activityUrlState";
import { formatNumber, quotaActionLabel } from "../lib/format";
import { qk } from "../lib/queryKeys";
import type { Shared } from "./ActivityRuns";
import { Pressable } from "./ui/Pressable";
import { LoadingState, StateMessage } from "./ui/QueryState";
import { SURFACE } from "./ui/surfaces";
// Activity → Quota: the shared YouTube API budget's spend over the page's time range — per day
// (Pacific days, as the budget resets), then per account, each opening to its actions. The
// account filter narrows both to one account; the text filter has nothing to match here.
export default function ActivityQuota({ shared }: { shared: Shared }) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const q = useQuery({
queryKey: qk.adminQuota({ window: shared.window, user: shared.user }),
queryFn: () => api.adminQuota(sinceOf(shared.window), shared.user),
placeholderData: keepPreviousData,
});
const data = q.data;
const maxDaily = Math.max(1, ...(data?.daily ?? []).map((d) => d.total));
const grandTotal = (data?.rows ?? []).reduce((s, r) => s + r.total, 0);
return (
<div className={q.isPlaceholderData ? "opacity-60" : ""} aria-busy={q.isPlaceholderData}>
<p className="text-xs text-muted mb-3 leading-relaxed">
{t("stats.introBefore")}
<b className="text-fg/80">{t("stats.introSystem")}</b>
{t("stats.introAfter")}
</p>
{q.isError && !data ? (
<StateMessage tone="error" onRetry={() => void q.refetch()}>
{t("common.loadError")}
</StateMessage>
) : q.isLoading || !data ? (
<LoadingState label={t("stats.loading")} />
) : (
<>
{q.isError && (
<StateMessage tone="error" compact onRetry={() => void q.refetch()} className="mb-3">
{t("activity.stale")}
</StateMessage>
)}
<div className={`${SURFACE.card} p-3 mb-4`}>
<div className="text-xs text-muted mb-2">
{t(shared.window === "all" ? "activity.quota.totalAll" : "activity.quota.total", {
count: grandTotal,
window: t(`activity.windows.${shared.window}`),
units: formatNumber(grandTotal, lang),
})}
</div>
{data.daily.length === 0 ? (
<div className="text-muted text-sm">{t("stats.noUsageInRange")}</div>
) : (
// Each day's figure is in its bar's text too (read out, not shown): the tooltip only
// reaches a pointer.
<div
className="flex items-end gap-0.5 h-24"
role="list"
aria-label={t("activity.quota.daily")}
>
{data.daily.map((d) => (
<div
key={d.day}
role="listitem"
// The day under the pointer is lit by the B3 layer over its whole column, not by
// dimming the bar: an opacity dim reads as "disabled".
className="flex-1 h-full flex items-end bg-card/50 rounded-t state-layer"
title={t("stats.dailyTooltip", {
count: d.total,
day: d.day,
units: formatNumber(d.total, lang),
})}
>
<span className="sr-only">
{t("stats.dailyTooltip", {
count: d.total,
day: d.day,
units: formatNumber(d.total, lang),
})}
</span>
<div
className="w-full bg-accent rounded-t"
style={{ height: `${Math.max(3, (d.total / maxDaily) * 100)}%` }}
/>
</div>
))}
</div>
)}
</div>
<div className="flex flex-col gap-1.5">
{data.rows.length === 0 ? (
<div className="text-muted py-4">{t("stats.noUsageInRange")}</div>
) : (
data.rows.map((r) => (
<UserRow key={r.user_id ?? "system"} row={r} max={data.rows[0]?.total || 1} />
))
)}
</div>
</>
)}
</div>
);
}
function UserRow({ row, max }: { row: AdminQuotaRow; max: number }) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const [open, setOpen] = useState(false);
const actions = Object.entries(row.by_action).sort((a, b) => b[1] - a[1]);
const isSystem = row.user_id === null;
return (
<div className={`${SURFACE.card} p-3`}>
<Pressable
selectable
onClick={() => setOpen((o) => !o)}
aria-expanded={open}
className="w-full flex items-center gap-3"
>
<span className={`text-sm font-medium truncate flex-1 ${isSystem ? "text-muted" : ""}`}>
{isSystem ? t("stats.system") : row.email}
</span>
<span className="text-sm font-semibold tabular-nums shrink-0">
{formatNumber(row.total, lang)}
</span>
</Pressable>
{/* Proportion bar */}
<div className="mt-2 h-1.5 bg-border/40 rounded-full overflow-hidden">
<div
className="h-full bg-accent rounded-full"
style={{ width: `${Math.max(2, (row.total / max) * 100)}%` }}
/>
</div>
{open && actions.length > 0 && (
<div className="mt-2 space-y-1 text-xs text-muted">
{actions.map(([action, units]) => (
<div key={action} className="flex items-center justify-between gap-2">
<span>{quotaActionLabel(t, action)}</span>
<span className="tabular-nums">{formatNumber(units, lang)}</span>
</div>
))}
</div>
)}
</div>
);
}
+282
View File
@@ -0,0 +1,282 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { keepPreviousData, useQuery } from "@tanstack/react-query";
import { api, type LogFacet, type RunQuery, type RunRow } from "../lib/api";
import {
RUN_STATUSES,
sinceOf,
type ActivityUrlState,
type RunStatusFilter,
} from "../lib/activityUrlState";
import { formatDateTimeSec, formatNumber, formatRuntime } from "../lib/format";
import { jobFailureText, jobResultText, parseSummary } from "../lib/jobResult";
import { qk } from "../lib/queryKeys";
import { useHistoryList } from "../lib/useHistoryList";
import HistoryList from "./HistoryList";
import HistoryRow from "./HistoryRow";
import { StateMessage } from "./ui/QueryState";
import { RelativeTime } from "./ui/RelativeTime";
import { StatusDot } from "./RunStatusDot";
import { Button } from "./ui/Button";
import FilterChipGroup from "./ui/FilterChipGroup";
import FilterSelect from "./ui/FilterSelect";
// Activity → Runs: every background run, newest first — the scheduled ones, an admin's "run now", a
// user's work on one channel. A row opens to who started it, when, what it returned or why it
// failed, and two ways on: the lines the run logged, and the other runs of its job.
const GRID =
"grid-cols-[0.75rem_minmax(0,1fr)_auto] sm:grid-cols-[0.75rem_minmax(8rem,12rem)_minmax(0,8rem)_5.5rem_4rem_5.5rem_minmax(10rem,1fr)]";
/** The shared filters, as the page passes them down. */
export interface Shared {
window: ActivityUrlState["window"];
user: string | null;
q: string;
}
export default function ActivityRuns({
shared,
job,
status,
onFilter,
onLogs,
}: {
shared: Shared;
job: string | null;
status: RunStatusFilter | null;
onFilter: (patch: { job?: string | null; status?: RunStatusFilter | null }) => void;
/** Open the server log at this run's lines. */
onLogs: (runId: number) => void;
}) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const [open, setOpen] = useState<number | null>(null);
const query: RunQuery = {
user: shared.user ?? undefined,
q: shared.q.trim() || undefined,
job: job ?? undefined,
status: status ?? undefined,
};
const key = { ...query, window: shared.window };
const state = useHistoryList(qk.adminRuns(key), (before) =>
api.adminRuns({ ...query, since: sinceOf(shared.window) }, before),
);
const facets = useQuery({
queryKey: qk.adminRunFacets(key),
queryFn: () => api.adminRunFacets({ ...query, since: sinceOf(shared.window) }),
placeholderData: keepPreviousData,
});
const count = (list: LogFacet[] | undefined, value: string) =>
list?.find((f) => f.value === value)?.count ?? (facets.data ? 0 : undefined);
const jobName = (id: string) => t(`scheduler.jobs.${id}`, id);
// The job in use stays offered at zero: under the other filters it may count nothing, and a menu
// without it would show no way back.
const jobs = [
...(facets.data?.jobs ?? []),
...(job && !facets.data?.jobs.some((f) => f.value === job) ? [{ value: job, count: 0 }] : []),
];
return (
<div className="flex flex-col gap-2">
<div className="flex flex-wrap items-center gap-2">
<FilterSelect
label={t("activity.runs.job")}
options={[
{ value: "", label: t("activity.any") },
...jobs.map((f) => ({ value: f.value, label: jobName(f.value), count: f.count })),
]}
selected={[job ?? ""]}
onChange={([value]) => onFilter({ job: value || null })}
testId="runs-job"
/>
<FilterChipGroup
ariaLabel={t("activity.runs.statusLabel")}
value={status ?? "all"}
onChange={(value) =>
onFilter({ status: value === "all" ? null : (value as RunStatusFilter) })
}
options={[
{ value: "all", label: t("activity.any"), count: facets.data?.total },
...RUN_STATUSES.map((s) => ({
value: s,
label: t(`activity.runs.status.${s}`),
count: count(facets.data?.statuses, s),
})),
]}
/>
</div>
{/* The menus above are built from these counts: without them they offer only "Any". */}
{facets.isError && (
<StateMessage tone="error" compact onRetry={() => void facets.refetch()}>
{t("activity.filtersFailed")}
</StateMessage>
)}
<HistoryList
state={state}
empty={t("activity.runs.empty")}
count={t("activity.runs.count", { count: state.rows.length })}
>
{state.rows.map((run, index) => (
<HistoryRow
key={run.id}
navIndex={index}
navText={`${jobName(run.job)} ${run.target_title ?? ""}`}
grid={GRID}
open={open === run.id}
onToggle={() => setOpen(open === run.id ? null : run.id)}
title={formatDateTimeSec(run.started_at, lang)}
cells={
<RunCells
run={run}
actor={run.actor_id === null ? null : state.accounts[String(run.actor_id)]}
/>
}
>
<RunDetail
run={run}
actor={run.actor_id === null ? null : state.accounts[String(run.actor_id)]}
onLogs={() => onLogs(run.id)}
onJob={() => onFilter({ job: run.job })}
/>
</HistoryRow>
))}
</HistoryList>
</div>
);
}
/** How long the run took; null while it runs, 0 when its start and end are one instant (the runs
* moved over from the audit log kept only when they were written — the length is unknown). */
function seconds(run: RunRow): number | null {
if (!run.finished_at) return null;
return (new Date(run.finished_at).getTime() - new Date(run.started_at).getTime()) / 1000;
}
/** What the run came to, in the reader's language: its failure, that the process stopped under it,
* or its result. */
function useOutcome(run: RunRow): { text: string; tone: string } {
const { t, i18n } = useTranslation();
const lang = i18n.language;
if (run.status === "error")
return {
text: jobFailureText(t, run.failure, lang) ?? run.error ?? t("scheduler.dot.error"),
tone: "text-danger",
};
if (run.status === "interrupted") return { text: t("scheduler.interrupted"), tone: "text-warn" };
if (run.status === "running") return { text: t("scheduler.runningNow"), tone: "text-accent" };
// A run moved over from the audit log before its result rode along as data holds only the
// server's English one-liner: read back into a result, as the old audit rows were.
const result =
run.result ??
parseSummary(run.summary) ??
(run.status === "skipped" && run.summary ? { skipped: run.summary } : null);
if (result) return { text: jobResultText(t, result, lang), tone: "text-muted" };
return { text: run.summary ?? "", tone: "text-muted" };
}
function RunCells({ run, actor }: { run: RunRow; actor: string | null | undefined }) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const outcome = useOutcome(run);
const secs = seconds(run);
const name = t(`scheduler.jobs.${run.job}`, run.job);
return (
<>
<StatusDot k={run.status} label={t(`scheduler.dot.${run.status}`)} />
<span className="truncate font-medium">
{name}
{run.target_title && <span className="font-normal text-muted"> · {run.target_title}</span>}
</span>
{/* A phone has room for the name and the time only; the row opens to the rest. */}
<RelativeTime at={run.started_at} seconds className="tabular-nums text-muted sm:hidden" />
<span className="hidden truncate text-muted sm:block">
{t(`scheduler.trigger.${run.trigger}`)}
{actor && ` · ${actor}`}
</span>
<RelativeTime
at={run.started_at}
seconds
className="hidden tabular-nums text-muted sm:block"
/>
<span className="hidden tabular-nums text-muted sm:block">
{secs ? formatRuntime(Math.max(1, secs), lang) : "—"}
</span>
{/* The spend carries its unit (a bare figure in a row has no meaning); none spent is blank. */}
<span className="hidden truncate tabular-nums text-muted sm:block">
{run.quota_units > 0 &&
t("activity.runs.units", {
count: run.quota_units,
units: formatNumber(run.quota_units, lang),
})}
</span>
<span className={`col-span-3 truncate sm:col-span-1 ${outcome.tone}`}>{outcome.text}</span>
</>
);
}
function RunDetail({
run,
actor,
onLogs,
onJob,
}: {
run: RunRow;
actor: string | null | undefined;
onLogs: () => void;
onJob: () => void;
}) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const link = "underline decoration-dotted underline-offset-2 hover:decoration-solid";
return (
<div className="flex flex-col gap-2">
<dl className="grid grid-cols-[max-content_minmax(0,1fr)] gap-x-3 gap-y-0.5">
<dt>{t("activity.runs.fields.started")}</dt>
<dd>
{run.actor_id === null
? t(`scheduler.trigger.${run.trigger}`)
: `${actor ?? `#${run.actor_id}`} · ${t(`scheduler.trigger.${run.trigger}`)}`}
</dd>
<dt>{t("activity.runs.fields.when")}</dt>
<dd className="tabular-nums">
{formatDateTimeSec(run.started_at, lang)}
{run.finished_at && ` → ${formatDateTimeSec(run.finished_at, lang)}`}
</dd>
{run.target && (
<>
<dt>{t("activity.runs.fields.target")}</dt>
<dd>
{run.target_title ? `${run.target_title} · ` : ""}
<span className="font-mono">{run.target}</span>
</dd>
</>
)}
<dt>{t("activity.runs.fields.quota")}</dt>
<dd className="tabular-nums">{formatNumber(run.quota_units, lang)}</dd>
</dl>
{run.error && (
<pre className="whitespace-pre-wrap break-words text-danger/90">{run.error}</pre>
)}
{run.result && (
<pre className="overflow-x-auto whitespace-pre font-mono">
{JSON.stringify(run.result, null, 2)}
</pre>
)}
<div className="flex flex-wrap gap-x-4 gap-y-1">
{/* Most runs store no line at all; a link there would only open an empty list. */}
{run.log_lines > 0 ? (
<Button variant="link" className={link} onClick={onLogs}>
{t("activity.runs.logLines", { count: run.log_lines })}
</Button>
) : (
<span>{t("activity.runs.noLogLines")}</span>
)}
<Button variant="link" className={link} onClick={onJob}>
{t("activity.runs.jobRuns")}
</Button>
</div>
</div>
);
}
+75 -17
View File
@@ -1,9 +1,11 @@
// @vitest-environment jsdom
//
// The admin log console. A request id narrows the list to that request (and says so, in the
// filter row and the URL); a filter value the facets no longer count stays pickable; "Live" watches
// the newest page only, and stops on a failed read instead of re-raising it; a failed read is an
// error, not an empty log; Copy takes every loaded line, oldest first.
// The activity page's server-log tab, rendered through the page (which owns the shared filters and
// the pivots). A request or run id narrows the list to that request or run (and says so, in the
// filter row and the URL), and Back returns to the view the pivot was made from; the account in use
// stays pickable; "Live" watches the newest page only, and stops on a failed read instead of
// re-raising it; a failed read is an error, not an empty log; Copy takes every loaded line, oldest
// first.
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { act, cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
@@ -32,6 +34,7 @@ vi.mock("../lib/api", async (importOriginal) => {
...actual.api,
adminLogs: (query: LogQuery, before: string | null) => logsMock(query, before),
adminLogFacets: (query: LogQuery) => facetsMock(query),
adminUsers: () => Promise.resolve([{ id: 1, email: "admin@example.com" }]),
},
};
});
@@ -44,7 +47,11 @@ vi.mock("../lib/clipboard", () => ({
},
}));
import AdminLogs from "./AdminLogs";
import Activity from "./Activity";
import { PageToolbarSlot } from "./ui/pageToolbarSlot";
// The filter bar lives in the toolbar, which portals into the shell's band: give it one.
const slot = document.body.appendChild(document.createElement("div"));
function row(id: number, extra: Partial<ServerLogRow> = {}): ServerLogRow {
return {
@@ -60,6 +67,7 @@ function row(id: number, extra: Partial<ServerLogRow> = {}): ServerLogRow {
request_id: `req-${String(id).padStart(8, "0")}`,
user_id: 1,
job_id: null,
run_id: null,
debug_session_id: null,
version: "0.90.0",
...extra,
@@ -79,7 +87,9 @@ function renderConsole() {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
return render(
<QueryClientProvider client={qc}>
<AdminLogs />
<PageToolbarSlot.Provider value={slot}>
<Activity />
</PageToolbarSlot.Provider>
</QueryClientProvider>,
);
}
@@ -93,7 +103,7 @@ const lines = () =>
.filter((b) => b.hasAttribute("aria-expanded"));
beforeEach(() => {
window.history.replaceState(null, "", "/?page=logs");
window.history.replaceState(null, "", "/?page=activity&at=logs");
facetsMock.mockResolvedValue(FACETS);
});
afterEach(() => {
@@ -104,8 +114,21 @@ afterEach(() => {
});
describe("AdminLogs", () => {
it("a dialog closing onto the same address leaves the open line open", async () => {
logsMock.mockResolvedValue({ items: [row(1)], accounts: {}, next_cursor: null });
renderConsole();
await waitFor(() => expect(lines()).toHaveLength(1));
fireEvent.click(lines()[0]!);
expect(lines()[0]!.getAttribute("aria-expanded")).toBe("true");
// A closing dialog steps back over its own history entry: a popstate onto the same address.
act(() => {
window.dispatchEvent(new PopStateEvent("popstate"));
});
expect(lines()[0]!.getAttribute("aria-expanded")).toBe("true");
});
it("a request id narrows the list to that request, with every other filter off", async () => {
window.history.replaceState(null, "", "/?page=logs&lgrp=error&lq=boom");
window.history.replaceState(null, "", "/?page=activity&at=logs&lgrp=error&aq=boom");
logsMock.mockResolvedValue({ items: [row(1)], accounts: {}, next_cursor: null });
renderConsole();
@@ -121,11 +144,15 @@ describe("AdminLogs", () => {
);
expect(screen.getByText("adminLogs.pivotRequest:req-00000001")).toBeTruthy();
const url = new URLSearchParams(window.location.search);
expect([url.get("lreq"), url.get("lgrp"), url.get("lq")]).toEqual(["req-00000001", null, null]);
expect([url.get("lreq"), url.get("lgrp"), url.get("aq")]).toEqual(["req-00000001", null, null]);
});
it("Back from a request's lines returns to the view the pivot was made from", async () => {
window.history.replaceState({ sfPage: "logs" }, "", "/?page=logs&lgrp=error&lq=boom");
window.history.replaceState(
{ sfPage: "activity" },
"",
"/?page=activity&at=logs&lgrp=error&aq=boom",
);
const depth = window.history.length;
logsMock.mockResolvedValue({ items: [row(1)], accounts: {}, next_cursor: null });
renderConsole();
@@ -157,23 +184,46 @@ describe("AdminLogs", () => {
);
expect(screen.queryByText("adminLogs.pivotRequest:req-00000001")).toBeNull();
const url = new URLSearchParams(window.location.search);
expect([url.get("lreq"), url.get("lgrp"), url.get("lq")]).toEqual([null, "error", "boom"]);
expect([url.get("lreq"), url.get("lgrp"), url.get("aq")]).toEqual([null, "error", "boom"]);
// Neither jump waited out the typing debounce: no read ever mixed one view's filters with the
// other's search text (the request with "boom", or the restored view without it).
const mixed = logsMock.mock.calls.filter(([q]) => (q.request_id != null) === (q.q === "boom"));
expect(mixed).toEqual([]);
});
it("keeps the account in use offered when the facets no longer count it", async () => {
window.history.replaceState(null, "", "/?page=logs&lu=7");
it("a run id narrows the list to that run's lines", async () => {
logsMock.mockResolvedValue({
items: [row(1, { run_id: 77 })],
accounts: {},
next_cursor: null,
});
renderConsole();
await waitFor(() => expect(lines()).toHaveLength(1));
fireEvent.click(lines()[0]!);
fireEvent.click(screen.getByRole("button", { name: "77" }));
await waitFor(() =>
expect(logsMock).toHaveBeenLastCalledWith(
expect.objectContaining({ run_id: 77, request_id: undefined }),
null,
),
);
expect(screen.getByText("adminLogs.pivotRun:77")).toBeTruthy();
expect(new URLSearchParams(window.location.search).get("lrun")).toBe("77");
});
it("keeps the account in use offered when the account list does not have it", async () => {
window.history.replaceState(null, "", "/?page=activity&at=logs&au=7");
logsMock.mockResolvedValue({ items: [], accounts: {}, next_cursor: null });
renderConsole();
await waitFor(() => expect(facetsMock).toHaveBeenCalled());
await waitFor(() =>
expect(logsMock).toHaveBeenCalledWith(expect.objectContaining({ user: "7" }), null),
);
fireEvent.click(screen.getByTestId("logs-user"));
fireEvent.click(screen.getByTestId("activity-user"));
expect(await screen.findByTestId("logs-user-7")).toBeTruthy();
expect(screen.getByTestId("logs-user-1")).toBeTruthy();
expect(await screen.findByTestId("activity-user-7")).toBeTruthy();
expect(screen.getByTestId("activity-user-1")).toBeTruthy();
});
it("Live watches the newest page only", async () => {
@@ -258,6 +308,14 @@ describe("AdminLogs", () => {
expect((await screen.findByRole("alert")).textContent).toContain("adminLogs.olderFailed");
});
it("facets that failed say the filter menus may be short, in the other tabs' words", async () => {
logsMock.mockResolvedValue({ items: [row(1)], accounts: {}, next_cursor: null });
facetsMock.mockRejectedValue(new Error("500"));
renderConsole();
await waitFor(() => expect(lines()).toHaveLength(1));
expect((await screen.findByRole("alert")).textContent).toContain("activity.filtersFailed");
});
it("offers no older page while a filter change loads", async () => {
let landErrors!: (page: LogPage) => void;
logsMock.mockImplementation((q) =>
+184 -258
View File
@@ -1,5 +1,4 @@
import clsx from "clsx";
import { useEffect, useMemo, useRef, useState } from "react";
import { useMemo, useRef, useState } from "react";
import { useTranslation } from "react-i18next";
import {
keepPreviousData,
@@ -7,74 +6,66 @@ import {
useQuery,
useQueryClient,
} from "@tanstack/react-query";
import { Copy, Radio, Search } from "lucide-react";
import { Copy, Radio } from "lucide-react";
import { api, type LogFacet, type LogQuery, type ServerLogRow } from "../lib/api";
import { copyText } from "../lib/clipboard";
import { formatDateTimeSec } from "../lib/format";
import { nextNavId, notifyNav, rememberNavPosition } from "../lib/history";
import { nextPageOnClick } from "../lib/nextPage";
import { LEVEL_GROUPS, type LevelGroup } from "../lib/log";
import {
DEFAULT_LOGS_STATE,
logsParamsToState,
logsParamUpdates,
WINDOW_HOURS,
type LogsUrlState,
type LogWindow,
} from "../lib/logsUrlState";
import { sinceOf } from "../lib/activityUrlState";
import { logsParamsToState, logsParamUpdates, type LogsUrlState } from "../lib/logsUrlState";
import { notify } from "../lib/notifications";
import { qk } from "../lib/queryKeys";
import { useDebounced } from "../lib/useDebounced";
import { urlWithParams } from "../lib/urlState";
import { useUrlMirror } from "../lib/useUrlMirror";
import { HistoryFrame } from "./HistoryList";
import LogLine from "./LogLine";
import NavGrid from "./ui/NavGrid";
import { Button } from "./ui/Button";
import FilterChipGroup from "./ui/FilterChipGroup";
import FilterSelect from "./ui/FilterSelect";
import { FilterFoldButton } from "./ui/FilterFold";
import { useFilterFold } from "../lib/useFilterFold";
import { useStickyTop } from "./ui/stickyTop";
import { Section } from "./ui/form";
import { LoadingState, StateMessage } from "./ui/QueryState";
import { DismissibleChip } from "./ui/Chip";
import { InnerInput } from "./ui/Input";
import { fieldClass } from "./ui/fieldStyles";
import type { Shared } from "./ActivityRuns";
// Admin → Szervernapló: the server's stored log (`log_events`). One list, newest first; a line
// opens in place to its full record, and its request id / job id narrow the list to that request or
// job. Filters live in the URL, so a filtered view is a link. "Élő" re-reads the newest page every
// Activity → Server log: the server's stored log (`log_events`). One list, newest first; a line
// opens in place to its full record, and its request / job / run id narrows the list to that
// request, job or run (the page does that: a pivot is a new history entry with every other filter
// off). The time range, account and text come from the page's shared bar; the rest live in the URL
// (./logsUrlState), so a filtered view is a link. "Élő" re-reads the newest page every
// few seconds while the tab is visible — the newest page only: live is watching the top, so paging
// into older rows turns it off. A failed read stops the polling (the request already reported it
// once) and marks the list as possibly stale, instead of re-raising the failure every few seconds.
const LIVE_MS = 4000;
const GROUPS: readonly LevelGroup[] = ["all", "error", "warning", "info", "detail"];
const WINDOWS: readonly LogWindow[] = ["1h", "24h", "7d"];
// A pivot shows everything about one request or job: the other filters would only hide part of it.
const PIVOT_RESET: Omit<LogsUrlState, "requestId" | "jobId"> = {
...DEFAULT_LOGS_STATE,
window: "7d",
};
function sinceOf(window: LogWindow): string {
return new Date(Date.now() - WINDOW_HOURS[window] * 3_600_000).toISOString();
}
function toQuery(state: LogsUrlState, q: string): LogQuery {
function toQuery(state: LogsUrlState, shared: Shared): LogQuery {
return {
levels: state.group === "all" ? undefined : LEVEL_GROUPS[state.group],
sources: state.sources,
user: state.user ?? undefined,
user: shared.user ?? undefined,
event: state.event ?? undefined,
request_id: state.requestId ?? undefined,
job_id: state.jobId ?? undefined,
q: q.trim() || undefined,
run_id: state.runId ?? undefined,
q: shared.q.trim() || undefined,
};
}
export default function AdminLogs() {
/** The ids a line can narrow the list to. */
export type LogPivot = Pick<LogsUrlState, "requestId" | "jobId" | "runId">;
export default function AdminLogs({
shared,
landed,
onPivot,
}: {
shared: Shared;
/** The filters of the entry Back/Forward landed on (the page reads the address bar, once). */
landed: { logs: LogsUrlState } | null;
/** Show only one request's, job's or run's lines (the page clears every other filter). */
onPivot: (pivot: LogPivot) => void;
}) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const qc = useQueryClient();
@@ -82,14 +73,9 @@ export default function AdminLogs() {
logsParamsToState(new URLSearchParams(window.location.search)),
);
const [live, setLiveState] = useState(false);
// The toolbar sticks at the scroller's top (./ui/stickyTop), and folds its facet row when short.
useStickyTop();
// The facet row folds away when the screen is short.
const fold = useFilterFold();
const facetFilters =
state.sources.length +
(state.user ? 1 : 0) +
(state.event ? 1 : 0) +
(state.window !== DEFAULT_LOGS_STATE.window ? 1 : 0);
const facetFilters = state.sources.length + (state.event ? 1 : 0);
// Read by `getNextPageParam` when a page is fetched, not when it renders: while live, a refetch
// stops after the newest page, however many pages the cache holds — a "Load older" that lands
// after Live was switched on, or a filter set revisited with older pages still cached.
@@ -99,41 +85,37 @@ export default function AdminLogs() {
setLiveState(on);
};
const [open, setOpen] = useState<number | null>(null);
// Bumped when the view changes by a jump (a pivot, Back/Forward) rather than by typing: the
// search box's text then applies at once instead of after the typing debounce.
const [jumps, setJumps] = useState(0);
const q = useDebounced(state.q, 300, jumps);
useUrlMirror(logsParamUpdates(state));
// Back/Forward lands on another view of the console (a pivot pushed one, below): read it back.
useEffect(() => {
const onPop = () => {
// Back/Forward lands on another entry's filters; a dialog closing lands on these same ones, and
// changes nothing (Live and the open line stay). Adopted during the render that brings it, with
// the page's half.
const [seen, setSeen] = useState(landed);
if (landed !== seen) {
setSeen(landed);
if (landed && JSON.stringify(landed.logs) !== JSON.stringify(state)) {
setOpen(null);
setJumps((n) => n + 1);
setState(logsParamsToState(new URLSearchParams(window.location.search)));
};
window.addEventListener("popstate", onPop);
return () => window.removeEventListener("popstate", onPop);
}, []);
setState(landed.logs);
}
}
const update = (patch: Partial<LogsUrlState>) => {
setOpen(null);
setState((s) => ({ ...s, ...patch }));
};
const query = toQuery(state, q);
const key = qk.adminLogs({ ...query, window: state.window });
const query = toQuery(state, shared);
const key = qk.adminLogs({ ...query, window: shared.window });
const logs = useInfiniteQuery({
queryKey: key,
queryFn: ({ pageParam }) =>
api.adminLogs({ ...query, since: sinceOf(state.window) }, pageParam),
api.adminLogs({ ...query, since: sinceOf(shared.window) }, pageParam),
initialPageParam: null as string | null,
getNextPageParam: (last) => (liveRef.current ? undefined : (last.next_cursor ?? undefined)),
placeholderData: keepPreviousData,
refetchInterval: (q) => (live && q.state.status !== "error" ? LIVE_MS : false),
});
const facets = useQuery({
queryKey: qk.adminLogFacets({ ...query, window: state.window }),
queryFn: () => api.adminLogFacets({ ...query, since: sinceOf(state.window) }),
queryKey: qk.adminLogFacets({ ...query, window: shared.window }),
queryFn: () => api.adminLogFacets({ ...query, since: sinceOf(shared.window) }),
placeholderData: keepPreviousData,
refetchInterval: (q) => (live && q.state.status !== "error" ? LIVE_MS * 4 : false),
});
@@ -146,15 +128,16 @@ export default function AdminLogs() {
? null
: (logs.data?.pages[logs.data.pages.length - 1]?.next_cursor ?? null);
// A read that failed while lines were on screen: they are kept, and the message says what the
// failure means for them. (With no lines, the list's own error state says it.)
// failure means for them. (With no lines, the list's own error state says it.) Without the facets
// the Source and Event menus offer only "Any", as on the other tabs, so the same words say it.
const failure = !logs.data
? null
: logs.isFetchNextPageError
? "olderFailed"
? "adminLogs.olderFailed"
: logs.isError || (facets.isError && facets.data)
? "stale"
? "adminLogs.stale"
: facets.isError
? "countsFailed"
? "activity.filtersFailed"
: null;
const retryFailed = () => {
if (logs.isFetchNextPageError)
@@ -217,19 +200,9 @@ export default function AdminLogs() {
return facets.data ? sum(levels.filter((f) => inGroup.includes(f.value))) : undefined;
};
// A pivot follows a link to another view, so it is its own history entry and Back returns to the
// view it was made from. The other filters are adjustments of the view on screen and replace it.
const pivot = (patch: Pick<LogsUrlState, "requestId" | "jobId">) => {
const pivot = (patch: Partial<LogPivot>) => {
setLive(false);
const next = { ...state, ...PIVOT_RESET, ...patch };
window.history.pushState(
{ ...rememberNavPosition(), sfScroll: nextNavId() },
"",
urlWithParams(logsParamUpdates(next)),
);
notifyNav();
setJumps((n) => n + 1);
update(next);
onPivot({ requestId: null, jobId: null, runId: null, ...patch });
};
// "Any", then what the facet counted. The value in use stays offered even at zero — under the
@@ -253,194 +226,130 @@ export default function AdminLogs() {
};
return (
<div className="p-4 max-w-6xl w-full mx-auto">
<Section card title={t("adminLogs.title")}>
<p className="text-xs text-muted leading-relaxed mb-3">{t("adminLogs.intro")}</p>
<HistoryFrame
testId="admin-logs"
list={logs}
rowCount={rows.length}
rowsClassName="font-mono text-2xs"
failure={failure && { text: t(failure), onRetry: retryFailed }}
older={
olderCursor !== null
? { label: t("adminLogs.older"), onClick: loadOlder, disabled: logs.isFetchingNextPage }
: null
}
count={t("adminLogs.count", { count: rows.length })}
empty={t("adminLogs.empty")}
header={
// The page's own band carries the shared filters and stays in reach; this one scrolls.
<div>
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
<FilterChipGroup
ariaLabel={t("adminLogs.levelLabel")}
value={state.group}
onChange={(group) => update({ group })}
options={GROUPS.map((g) => ({
value: g,
label: t(`settings.myLog.filters.${g}`),
count: groupCount(g),
}))}
/>
</div>
<div
data-testid="admin-logs"
// overflow-clip, not -hidden: a hidden box is a scroll container, and the toolbar would stick
// to it instead of to the page.
className="overflow-clip rounded-xl border border-border bg-bg"
>
{/* The console's toolbar is the page's band (B4, B9): it sticks at the top with the fade
under it while the lines scroll, so a filter is in reach at line 500. */}
<div className="sticky top-0 z-base bg-bg fade-below">
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
<label
className={clsx(
fieldClass({ size: "sm", pill: true }),
"flex min-w-[12rem] flex-1 items-center gap-1.5",
)}
>
<Search className="h-3.5 w-3.5 shrink-0 text-muted" aria-hidden />
<InnerInput
type="search"
value={state.q}
onChange={(e) => update({ q: e.target.value.slice(0, 200) })}
placeholder={t("adminLogs.search")}
aria-label={t("adminLogs.search")}
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
<FilterFoldButton fold={fold} label={t("sidebar.filters")} count={facetFilters} />
{!fold.folded && (
<>
<FilterSelect
label={t("adminLogs.source")}
options={facetOptions(facets.data?.sources, state.sources[0] ?? null, (v) => v)}
selected={[state.sources[0] ?? ""]}
onChange={([source]) => update({ sources: source ? [source] : [] })}
testId="logs-source"
/>
</label>
<FilterChipGroup
ariaLabel={t("adminLogs.levelLabel")}
value={state.group}
onChange={(group) => update({ group })}
options={GROUPS.map((g) => ({
value: g,
label: t(`settings.myLog.filters.${g}`),
count: groupCount(g),
}))}
/>
</div>
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
<FilterFoldButton fold={fold} label={t("sidebar.filters")} count={facetFilters} />
{!fold.folded && (
<>
<FilterSelect
label={t("adminLogs.source")}
options={facetOptions(facets.data?.sources, state.sources[0] ?? null, (v) => v)}
selected={[state.sources[0] ?? ""]}
onChange={([source]) => update({ sources: source ? [source] : [] })}
testId="logs-source"
/>
<FilterSelect
label={t("adminLogs.account")}
options={facetOptions(facets.data?.users, state.user, accountLabel)}
selected={[state.user ?? ""]}
onChange={([user]) => update({ user: user || null })}
testId="logs-user"
/>
<FilterSelect
label={t("adminLogs.event")}
options={facetOptions(facets.data?.events, state.event, (v) =>
v === "none" ? t("adminLogs.noEvent") : v,
)}
selected={[state.event ?? ""]}
onChange={([event]) => update({ event: event || null })}
testId="logs-event"
/>
<FilterSelect
label={t("adminLogs.window")}
options={WINDOWS.map((w) => ({ value: w, label: t(`adminLogs.windows.${w}`) }))}
selected={[state.window]}
onChange={([window]) => update({ window: window as LogWindow })}
testId="logs-window"
/>
</>
)}
<span className="ml-auto flex gap-1.5">
<Button
variant="outline"
size="xs"
icon={<Radio className="h-3.5 w-3.5" />}
pressed={polling}
onClick={toggleLive}
>
{t("adminLogs.live")}
</Button>
<Button
variant="outline"
size="xs"
icon={<Copy className="h-3.5 w-3.5" />}
// While a filter change loads, the rows on screen are the previous filter's.
disabled={rows.length === 0 || logs.isPlaceholderData}
onClick={() => void copy()}
>
{t("adminLogs.copy")}
</Button>
</span>
</div>
{(state.requestId || state.jobId !== null) && (
<div className="flex flex-wrap items-center gap-2 border-b border-border px-2 py-1.5 text-xs">
{state.requestId && (
<PivotChip
label={t("adminLogs.pivotRequest", { id: state.requestId })}
clearLabel={t("adminLogs.clearPivot")}
onClear={() => update({ requestId: null })}
/>
)}
{state.jobId !== null && (
<PivotChip
label={t("adminLogs.pivotJob", { id: state.jobId })}
clearLabel={t("adminLogs.clearPivot")}
onClear={() => update({ jobId: null })}
/>
)}
</div>
<FilterSelect
label={t("adminLogs.event")}
options={facetOptions(facets.data?.events, state.event, (v) =>
v === "none" ? t("adminLogs.noEvent") : v,
)}
selected={[state.event ?? ""]}
onChange={([event]) => update({ event: event || null })}
testId="logs-event"
/>
</>
)}
</div>
{failure && (
<StateMessage
tone="error"
compact
onRetry={retryFailed}
className="border-b border-border"
>
{t(`adminLogs.${failure}`)}
</StateMessage>
)}
<div
className={`font-mono text-2xs ${logs.isPlaceholderData ? "opacity-60" : ""}`}
data-testid="admin-logs-lines"
aria-busy={logs.isPlaceholderData}
>
{logs.isError && !logs.data ? (
<StateMessage tone="error" onRetry={() => void logs.refetch()}>
{t("common.loadError")}
</StateMessage>
) : logs.isLoading ? (
<LoadingState shape="table" />
) : rows.length === 0 ? (
<p className="px-3 py-6 text-center font-sans text-sm text-muted">
{t("adminLogs.empty")}
</p>
) : (
<NavGrid>
{rows.map((row, index) => (
<LogLine
navIndex={index}
key={row.id}
ts={row.ts}
level={row.level}
event={row.event ?? "—"}
msg={row.msg}
title={formatDateTimeSec(row.ts, lang)}
open={open === row.id}
onToggle={() => setOpen(open === row.id ? null : row.id)}
>
<RowDetail
row={row}
account={row.user_id === null ? null : accountLabel(String(row.user_id))}
onRequest={(requestId) => pivot({ requestId, jobId: null })}
onJob={(jobId) => pivot({ requestId: null, jobId })}
/>
</LogLine>
))}
</NavGrid>
)}
</div>
<div className="flex flex-wrap items-center justify-between gap-2 border-t border-border px-3 py-1.5 text-xs text-muted">
<span>{t("adminLogs.count", { count: rows.length })}</span>
{olderCursor !== null && (
<span className="ml-auto flex gap-1.5">
<Button
variant="outline"
size="xs"
disabled={logs.isFetchingNextPage}
onClick={loadOlder}
icon={<Radio className="h-3.5 w-3.5" />}
pressed={polling}
onClick={toggleLive}
>
{t("adminLogs.older")}
{t("adminLogs.live")}
</Button>
)}
<Button
variant="outline"
size="xs"
icon={<Copy className="h-3.5 w-3.5" />}
// While a filter change loads, the rows on screen are the previous filter's.
disabled={rows.length === 0 || logs.isPlaceholderData}
onClick={() => void copy()}
>
{t("adminLogs.copy")}
</Button>
</span>
</div>
{(state.requestId || state.jobId !== null || state.runId !== null) && (
<div className="flex flex-wrap items-center gap-2 border-b border-border px-2 py-1.5 text-xs">
{state.requestId && (
<PivotChip
label={t("adminLogs.pivotRequest", { id: state.requestId })}
clearLabel={t("adminLogs.clearPivot")}
onClear={() => update({ requestId: null })}
/>
)}
{state.jobId !== null && (
<PivotChip
label={t("adminLogs.pivotJob", { id: state.jobId })}
clearLabel={t("adminLogs.clearPivot")}
onClear={() => update({ jobId: null })}
/>
)}
{state.runId !== null && (
<PivotChip
label={t("adminLogs.pivotRun", { id: state.runId })}
clearLabel={t("adminLogs.clearPivot")}
onClear={() => update({ runId: null })}
/>
)}
</div>
)}
</div>
</Section>
</div>
}
>
{rows.map((row, index) => (
<LogLine
navIndex={index}
key={row.id}
ts={row.ts}
level={row.level}
event={row.event ?? "—"}
msg={row.msg}
title={formatDateTimeSec(row.ts, lang)}
open={open === row.id}
onToggle={() => setOpen(open === row.id ? null : row.id)}
>
<RowDetail
row={row}
account={row.user_id === null ? null : accountLabel(String(row.user_id))}
onRequest={(requestId) => pivot({ requestId })}
onJob={(jobId) => pivot({ jobId })}
onRun={(runId) => pivot({ runId })}
/>
</LogLine>
))}
</HistoryFrame>
);
}
@@ -465,11 +374,13 @@ function RowDetail({
account,
onRequest,
onJob,
onRun,
}: {
row: ServerLogRow;
account: string | null;
onRequest: (id: string) => void;
onJob: (id: number) => void;
onRun: (id: number) => void;
}) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
@@ -516,6 +427,21 @@ function RowDetail({
</dd>
</>
)}
{row.run_id !== null && (
<>
<dt>{t("adminLogs.fields.run")}</dt>
<dd>
<Button
variant="link"
className={pivotCls}
title={t("adminLogs.showRun")}
onClick={() => onRun(row.run_id!)}
>
{row.run_id}
</Button>
</dd>
</>
)}
{row.version && (
<>
<dt>{t("adminLogs.fields.version")}</dt>
@@ -0,0 +1,269 @@
// @vitest-environment jsdom
//
// Settings → API tokens: a row per token says what it is, what it opens with which rights, when it was
// last used and when it runs out (marked when that is soon); revoking asks first. A new token is the
// Hub exchange in one click, or folders and rights picked one by one — never Write where an upload
// cannot land — and its value is shown once, with a curl line to try it. i18n is key-passthrough
// (a key's options ride along in brackets, so a test can read what was named).
import { afterEach, describe, expect, it, vi } from "vitest";
import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { api, type ApiToken, type FileFolder } from "../lib/api";
const mocks = vi.hoisted(() => ({ confirm: vi.fn(async (_o?: unknown) => true) }));
vi.mock("react-i18next", async (importOriginal) => ({
...(await importOriginal<typeof import("react-i18next")>()),
useTranslation: () => ({
i18n: { language: "en" },
t: (key: string, o?: Record<string, unknown>) =>
o && Object.keys(o).length ? `${key}(${Object.values(o).join(",")})` : key,
}),
}));
vi.mock("./confirmContext", () => ({ useConfirm: () => mocks.confirm }));
vi.mock("../lib/clipboard", () => ({ copyText: vi.fn(async () => true) }));
import ApiTokens from "./ApiTokens";
afterEach(() => {
cleanup();
vi.restoreAllMocks();
mocks.confirm.mockClear();
});
const token = (id: number, extra: Partial<ApiToken> = {}): ApiToken => ({
id,
name: `Token ${id}`,
prefix: `sfl_${id}abcdefg`,
created_at: "2026-10-01T00:00:00Z",
expires_at: "2027-10-01T00:00:00Z",
expired: false,
expiring: false,
last_used_at: null,
grants: [],
...extra,
});
const folder = (id: number, extra: Partial<FileFolder>): FileFolder => ({
id,
kind: "source",
source_key: null,
name: null,
retention_days: 30,
count: 1,
size_bytes: 0,
created_at: null,
has_link: false,
...extra,
});
function mount(tokens: ApiToken[] | Error, folders: FileFolder[] = []) {
if (tokens instanceof Error) vi.spyOn(api, "apiTokens").mockRejectedValue(tokens);
else vi.spyOn(api, "apiTokens").mockResolvedValue(tokens);
vi.spyOn(api, "fileFolders").mockResolvedValue(folders);
render(
<QueryClientProvider
client={new QueryClient({ defaultOptions: { queries: { retry: false } } })}
>
<ApiTokens />
</QueryClientProvider>,
);
}
const rows = () => screen.getAllByTestId("api-token-row");
const dialog = () => screen.getByRole("dialog");
const pressed = (b: HTMLElement) => b.getAttribute("aria-pressed") === "true";
const inRow = (group: string, name: string) =>
[...screen.getByRole("group", { name: group }).querySelectorAll("button")].find(
(b) => b.textContent === name,
);
describe("API tokens · the list", () => {
it("reads what a token opens, with which rights, when it was used and when it runs out", async () => {
mount([
token(1, {
name: "Hub exchange",
last_used_at: new Date().toISOString(),
grants: [
{
folder_id: 7,
kind: "hub",
source_key: null,
name: null,
rights: ["write", "list", "read", "delete"],
},
],
}),
token(2, {
expiring: true,
expires_at: null,
grants: [
{ folder_id: 3, kind: "own", source_key: null, name: "Receptek", rights: ["read"] },
],
}),
token(3, { expired: true }),
]);
await waitFor(() => expect(rows()).toHaveLength(3));
const [hub, soon, gone] = rows();
expect(hub!.textContent).toContain("sfl_1abcdefg…");
// Rights in the server's order, whatever order they came in.
expect(hub!.textContent).toContain(
"files.folder.hub· settings.tokens.rights.list, settings.tokens.rights.read, settings.tokens.rights.write, settings.tokens.rights.delete",
);
expect(hub!.textContent).toContain("settings.tokens.lastUsed");
expect(hub!.textContent).not.toContain("settings.tokens.expiring");
expect(soon!.textContent).toContain("Receptek· settings.tokens.rights.read");
expect(soon!.textContent).toContain("settings.tokens.expiring");
expect(soon!.textContent).toContain("settings.tokens.neverUsed");
expect(soon!.textContent).toContain("settings.tokens.neverExpires");
expect(gone!.textContent).toContain("settings.tokens.expired");
expect(gone!.textContent).toContain("settings.tokens.expiredOn(");
});
it("says when there is none, and when the list could not be read", async () => {
mount([]);
expect(await screen.findByText("settings.tokens.empty")).toBeTruthy();
cleanup();
mount(new Error("down"));
expect(await screen.findByText("common.loadError")).toBeTruthy();
expect(screen.queryByText("settings.tokens.empty")).toBeNull();
});
it("revokes from the row's menu, only once the confirm says yes", async () => {
mount([token(5, { name: "Phone" })]);
const revoke = vi.spyOn(api, "revokeApiToken").mockResolvedValue({ ok: true, deleted: 5 });
await waitFor(() => expect(rows()).toHaveLength(1));
mocks.confirm.mockResolvedValueOnce(false);
const open = async () => {
act(() => {
fireEvent.contextMenu(rows()[0]!);
});
await waitFor(() =>
expect(screen.getAllByRole("menuitem", { hidden: true }).length).toBeGreaterThan(0),
);
fireEvent.click(
screen
.getAllByRole("menuitem", { hidden: true })
.find((b) => b.textContent === "settings.tokens.revoke")!,
);
};
await open();
await waitFor(() => expect(mocks.confirm).toHaveBeenCalledTimes(1));
expect(mocks.confirm.mock.calls[0]![0]).toMatchObject({
title: "settings.tokens.revokeTitle(Phone)",
danger: true,
});
expect(revoke).not.toHaveBeenCalled();
await open();
await waitFor(() => expect(revoke).toHaveBeenCalledWith(5));
});
});
describe("API tokens · making one", () => {
const made = (extra: Partial<ApiToken> = {}) => ({
ok: true as const,
...token(9, extra),
token: "sfl_SECRETVALUE",
});
it("makes the Hub exchange in one click, a year long, and shows the token once with a curl line", async () => {
mount([]);
const create = vi.spyOn(api, "createApiToken").mockResolvedValue(made());
fireEvent.click(await screen.findByText("settings.tokens.new"));
expect(pressed(screen.getByRole("button", { name: "settings.tokens.hubTitle" }))).toBe(true);
expect((screen.getByLabelText("settings.tokens.name") as HTMLInputElement).value).toBe(
"settings.tokens.hubName",
);
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.create" }));
await waitFor(() =>
expect(create).toHaveBeenCalledWith({
name: "settings.tokens.hubName",
expires_days: 365,
preset: "hub",
}),
);
await screen.findByText("settings.tokens.madeTitle");
const codes = [...dialog().querySelectorAll("code")].map((c) => c.textContent);
expect(codes[0]).toBe("sfl_SECRETVALUE");
expect(codes[1]).toBe(
`curl -H "Authorization: Bearer sfl_SECRETVALUE" ${window.location.origin}/api/files/folders`,
);
// The keyboard lands on copying it, not on the closed form's button.
expect(document.activeElement?.textContent).toBe("settings.tokens.copy");
// Two Copy buttons in one dialog: each is named by the value it copies.
expect(
screen.getByRole("button", { name: "settings.tokens.copyField(settings.tokens.token)" }),
).toBe(document.activeElement);
screen.getByRole("button", { name: "settings.tokens.copyField(settings.tokens.tryIt)" });
// Once copied, its name says what it shows, so a voice command saying "Copied" finds it.
fireEvent.click(document.activeElement!);
await screen.findByRole("button", {
name: "settings.tokens.copiedField(settings.tokens.token)",
});
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.done" }));
expect(screen.queryByRole("dialog")).toBeNull();
});
it("a custom token takes the folders and rights picked; no Write where nothing can be uploaded", async () => {
mount(
[],
[
folder(1, { source_key: "youtube" }),
folder(2, { source_key: "upload" }),
folder(3, { source_key: "site:x.com", count: 0 }), // empty: not offered, as the rail hides it
folder(4, { kind: "hub", count: 0 }),
folder(5, { kind: "own", name: "Receptek" }),
],
);
const create = vi.spyOn(api, "createApiToken").mockResolvedValue(made());
fireEvent.click(await screen.findByText("settings.tokens.new"));
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.customTitle" }));
const name = screen.getByLabelText("settings.tokens.name") as HTMLInputElement;
expect(name.value).toBe("");
await screen.findByRole("group", { name: "Receptek" });
expect(
screen
.getAllByRole("group")
.map((g) => g.getAttribute("aria-label"))
.filter((l) => l !== "settings.tokens.kind" && l !== "settings.tokens.expiry"),
).toEqual(["files.source.upload", "files.source.youtube", "files.folder.hub", "Receptek"]);
expect(inRow("files.source.youtube", "settings.tokens.rights.write")).toBeUndefined();
expect(inRow("files.source.upload", "settings.tokens.rights.write")).toBeTruthy();
expect(inRow("files.folder.hub", "settings.tokens.rights.write")).toBeTruthy();
const createBtn = screen.getByRole("button", { name: "settings.tokens.create" });
fireEvent.change(name, { target: { value: " Phone " } });
expect(createBtn.hasAttribute("disabled")).toBe(true); // a name, but no right yet
fireEvent.click(inRow("Receptek", "settings.tokens.rights.list")!);
fireEvent.click(inRow("Receptek", "settings.tokens.rights.write")!);
fireEvent.click(inRow("files.source.youtube", "settings.tokens.rights.read")!);
// Picked and dropped again: no grant for that folder.
fireEvent.click(inRow("files.folder.hub", "settings.tokens.rights.read")!);
fireEvent.click(inRow("files.folder.hub", "settings.tokens.rights.read")!);
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.expiryOption.30" }));
expect(createBtn.hasAttribute("disabled")).toBe(false);
fireEvent.click(createBtn);
await waitFor(() => expect(create).toHaveBeenCalledTimes(1));
const body = create.mock.calls[0]![0];
expect(body).toMatchObject({ name: "Phone", expires_days: 30 });
expect(body.preset).toBeUndefined();
expect(body.grants).toEqual(
expect.arrayContaining([
{ folder_id: 5, rights: ["list", "write"] },
{ folder_id: 1, rights: ["read"] },
]),
);
expect(body.grants).toHaveLength(2);
});
it("a token that never runs out says what that means, and is sent without an expiry", async () => {
mount([]);
const create = vi.spyOn(api, "createApiToken").mockResolvedValue(made());
fireEvent.click(await screen.findByText("settings.tokens.new"));
expect(screen.queryByText("settings.tokens.neverWarning")).toBeNull();
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.expiryOption.never" }));
expect(screen.getByText("settings.tokens.neverWarning")).toBeTruthy();
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.create" }));
await waitFor(() =>
expect(create).toHaveBeenCalledWith(expect.objectContaining({ expires_days: null })),
);
});
});
+504
View File
@@ -0,0 +1,504 @@
import { useId, useState, type ReactNode } from "react";
import { useTranslation } from "react-i18next";
import type { TFunction } from "i18next";
import { useMutation, useQuery, useQueryClient, type UseQueryResult } from "@tanstack/react-query";
import { Check, Copy, KeyRound, Plus, SlidersHorizontal, Trash2 } from "lucide-react";
import {
api,
TOKEN_EXPIRY_DAYS,
TOKEN_EXPIRY_DEFAULT,
TOKEN_RIGHTS,
type ApiToken,
type FileFolder,
type TokenGrant,
type TokenRight,
} from "../lib/api";
import { qk } from "../lib/queryKeys";
import { compareText } from "../lib/collate";
import { formatDate } from "../lib/format";
import { navItemAttrs } from "../lib/listNav";
import { useConfirm } from "./confirmContext";
import { folderIcon, folderLabel, takesUploads } from "./fileFolders";
import { type ActionMenuItem } from "./ui/ActionMenu";
import { Badge } from "./ui/Badge";
import { Button } from "./ui/Button";
import { Chip } from "./ui/Chip";
import { ContextMenu } from "./ui/ContextMenu";
import InlineNotice from "./ui/InlineNotice";
import { Input } from "./ui/Input";
import Modal, { ModalFooter } from "./ui/Modal";
import { MoreButton } from "./ui/MoreButton";
import NavGrid from "./ui/NavGrid";
import { Pressable } from "./ui/Pressable";
import { LoadingState, StateMessage } from "./ui/QueryState";
import { RelativeTime } from "./ui/RelativeTime";
import SegmentedToggle from "./ui/SegmentedToggle";
import { SURFACE } from "./ui/surfaces";
import { useContextMenu } from "./ui/useContextMenu";
import { useCopyFeedback } from "./useCopyFeedback";
// Settings → API tokens: a machine's key to the user's Files library (docs/DESIGN.md decision log,
// "API tokens"). A row per token in three lines — what it is, what it opens, when it was used and
// when it runs out — revoked from its "…" behind a confirm. A new token is made in a dialog that
// asks the kind first (the one-click Hub exchange, or folders and rights of the user's own picking)
// and, once made, shows the token in the same dialog: the one time it can be seen.
/** The rights a token holds in a folder, in the reader's words: "list, read, write". */
function rightsText(rights: readonly TokenRight[], t: TFunction, lang: string): string {
return TOKEN_RIGHTS.filter((r) => rights.includes(r))
.map((r) => t(`settings.tokens.rights.${r}`).toLocaleLowerCase(lang))
.join(", ");
}
/** A folder's glyph (`folderIcon`), drawn at a row's size. */
function Glyph({ icon: Icon }: { icon: ReturnType<typeof folderIcon> }) {
return <Icon className="size-3.5 shrink-0" aria-hidden />;
}
function GrantLine({ grant }: { grant: TokenGrant }) {
const { t, i18n } = useTranslation();
return (
<span className="flex min-w-0 items-center gap-1.5">
<Glyph icon={folderIcon(grant)} />
<span className="text-fg">{folderLabel(grant, t)}</span>
<span>· {rightsText(grant.rights, t, i18n.language)}</span>
</span>
);
}
function TokenRow({
token,
index,
rowProps,
more,
}: {
token: ApiToken;
index: number;
rowProps: Record<string, unknown>;
more: ReactNode;
}) {
const { t, i18n } = useTranslation();
const expires = formatDate(token.expires_at, i18n.language);
return (
<div
{...navItemAttrs(index, token.name, { inset: true })}
{...rowProps}
data-testid="api-token-row"
className="flex items-start gap-3 py-2.5"
>
<KeyRound className="mt-0.5 size-4 shrink-0 text-muted" aria-hidden />
<div className="flex min-w-0 flex-1 flex-col gap-1">
<span className="flex flex-wrap items-center gap-2">
<span className="text-sm font-medium break-all">{token.name}</span>
<code className="text-xs text-muted">{token.prefix}…</code>
{token.expired ? (
<Badge tone="warn">{t("settings.tokens.expired")}</Badge>
) : (
token.expiring && <Badge tone="warn">{t("settings.tokens.expiring")}</Badge>
)}
</span>
<span className="flex flex-wrap items-center gap-x-3 gap-y-1 text-xs text-muted">
{token.grants.map((g) => (
<GrantLine key={g.folder_id} grant={g} />
))}
</span>
<span className="text-xs text-muted">
{token.last_used_at ? (
<>
{t("settings.tokens.lastUsed")} <RelativeTime at={token.last_used_at} />
</>
) : (
t("settings.tokens.neverUsed")
)}
{" · "}
{token.expires_at === null
? t("settings.tokens.neverExpires")
: token.expired
? t("settings.tokens.expiredOn", { date: expires })
: t("settings.tokens.expiresOn", { date: expires })}
</span>
</div>
{more}
</div>
);
}
export default function ApiTokens() {
const { t } = useTranslation();
const qc = useQueryClient();
const confirm = useConfirm();
const [making, setMaking] = useState(false);
const tokensQ = useQuery({ queryKey: qk.apiTokens(), queryFn: api.apiTokens });
const revoke = useMutation({
mutationFn: (id: number) => api.revokeApiToken(id),
onSuccess: () => qc.invalidateQueries({ queryKey: qk.apiTokens() }),
meta: { failTitle: "errors.writeFailed.apiToken.revoke" },
});
const onRevoke = async (token: ApiToken) => {
const ok = await confirm({
title: t("settings.tokens.revokeTitle", { name: token.name }),
message: t("settings.tokens.revokeBody"),
confirmLabel: t("settings.tokens.revokeAction"),
danger: true,
});
if (ok) revoke.mutate(token.id);
};
const menu = useContextMenu<ApiToken>({ keyOf: (tk) => tk.id });
const items = (token: ApiToken): ActionMenuItem[] => [
{
id: "revoke",
label: t("settings.tokens.revoke"),
icon: <Trash2 className="w-4 h-4" />,
danger: true,
onSelect: () => void onRevoke(token),
},
];
const tokens = tokensQ.data;
return (
<div className="flex flex-col gap-3">
<div className="flex items-start gap-3">
<p className="flex-1 text-xs leading-relaxed text-muted">{t("settings.tokens.intro")}</p>
<Button
variant="primary"
size="sm"
icon={<Plus className="w-4 h-4" />}
onClick={() => setMaking(true)}
>
{t("settings.tokens.new")}
</Button>
</div>
{tokensQ.isError && !tokens ? (
<StateMessage tone="error" onRetry={() => void tokensQ.refetch()}>
{t("common.loadError")}
</StateMessage>
) : !tokens ? (
<LoadingState shape="list" />
) : tokens.length === 0 ? (
<p className="py-4 text-sm text-muted">{t("settings.tokens.empty")}</p>
) : (
<NavGrid className="flex flex-col divide-y divide-border/60">
{tokens.map((token, i) => (
<TokenRow
key={token.id}
token={token}
index={i}
rowProps={menu.triggerProps(token)}
more={
<MoreButton
label={t("settings.tokens.rowMenu")}
expanded={menu.open && menu.target?.id === token.id}
menuProps={menu.buttonProps(token)}
/>
}
/>
))}
</NavGrid>
)}
<ContextMenu
menu={menu}
ariaLabel={t("settings.tokens.rowMenu")}
items={menu.target ? items(menu.target) : []}
/>
{making && <NewTokenDialog onClose={() => setMaking(false)} />}
</div>
);
}
// --- making one -----------------------------------------------------------------------------------
type Kind = "hub" | "custom";
type Grants = Record<number, readonly TokenRight[]>;
/** The folders a custom token may be given, as the Files rail shows them: the sources that hold
* something, then the Hub folder, then the user's own. A right a folder cannot take (writing into a
* site's folder) is not offered. */
function grantable(folders: readonly FileFolder[], t: TFunction, lang: string): FileFolder[] {
const byName = (a: FileFolder, b: FileFolder) =>
compareText(folderLabel(a, t), folderLabel(b, t), lang);
return [
...folders.filter((f) => f.kind === "source" && f.count > 0).sort(byName),
...folders.filter((f) => f.kind === "hub"),
...folders.filter((f) => f.kind === "own").sort(byName),
];
}
function RightsGrid({
foldersQ,
grants,
onChange,
}: {
foldersQ: UseQueryResult<FileFolder[]>;
grants: Grants;
onChange: (g: Grants) => void;
}) {
const { t, i18n } = useTranslation();
if (foldersQ.isError && !foldersQ.data)
return (
<StateMessage tone="error" onRetry={() => void foldersQ.refetch()}>
{t("common.loadError")}
</StateMessage>
);
if (!foldersQ.data) return <LoadingState />;
const rows = grantable(foldersQ.data, t, i18n.language);
if (rows.length === 0)
return <p className="text-sm text-muted">{t("settings.tokens.noFolders")}</p>;
const toggle = (folder: FileFolder, right: TokenRight) => {
const had = grants[folder.id] ?? [];
const next = had.includes(right) ? had.filter((r) => r !== right) : [...had, right];
onChange({ ...grants, [folder.id]: next });
};
return (
<div className="flex flex-col divide-y divide-border/50 rounded-lg border border-border/60">
{rows.map((f) => {
const label = folderLabel(f, t);
return (
<div
key={f.id}
role="group"
aria-label={label}
className="flex flex-wrap items-center gap-2 px-2.5 py-2"
>
<span className="flex min-w-28 flex-1 items-center gap-1.5 text-sm">
<Glyph icon={folderIcon(f)} />
<span className="break-all">{label}</span>
</span>
<span className="flex flex-wrap gap-1">
{TOKEN_RIGHTS.map((r) =>
r === "write" && !takesUploads(f) ? null : (
<Chip
key={r}
pressed={(grants[f.id] ?? []).includes(r)}
onClick={() => toggle(f, r)}
>
{t(`settings.tokens.rights.${r}`)}
</Chip>
),
)}
</span>
</div>
);
})}
</div>
);
}
function KindTile({
on,
icon,
title,
text,
onPick,
}: {
on: boolean;
icon: ReactNode;
title: string;
text: string;
onPick: () => void;
}) {
const textId = useId();
return (
<Pressable
aria-pressed={on}
aria-describedby={textId}
onClick={onPick}
className={`${SURFACE.card} flex flex-col items-start gap-1 border p-3 text-left ${
on ? "border-accent" : "border-transparent hover:border-border"
}`}
>
<span className="flex items-center gap-1.5 text-sm font-medium">
{icon}
{title}
</span>
{/* Read after the name, not into it ("Custom" — then what it means). */}
<span id={textId} aria-hidden className="text-xs text-muted">
{text}
</span>
</Pressable>
);
}
const HubIcon = folderIcon({ kind: "hub", source_key: null, name: null });
const expiryValue = (days: number | null) => (days === null ? "never" : String(days));
function NewTokenDialog({ onClose }: { onClose: () => void }) {
const { t } = useTranslation();
const qc = useQueryClient();
const nameId = useId();
const [kind, setKind] = useState<Kind>("hub");
// The name as typed; untouched, the Hub exchange is named for what it is and a custom one is blank.
const [nameDraft, setNameDraft] = useState<string | null>(null);
const name = nameDraft ?? (kind === "hub" ? t("settings.tokens.hubName") : "");
const [grants, setGrants] = useState<Grants>({});
const [expiry, setExpiry] = useState<number | null>(TOKEN_EXPIRY_DEFAULT);
// Read as the dialog opens, not when Custom is picked: the grid is there by the time it is wanted.
const foldersQ = useQuery({ queryKey: qk.fileFolders(), queryFn: api.fileFolders });
const picked = Object.entries(grants)
.filter(([, rights]) => rights.length > 0)
.map(([id, rights]) => ({ folder_id: Number(id), rights: [...rights] }));
const create = useMutation({
mutationFn: () =>
api.createApiToken(
kind === "hub"
? { name: name.trim(), expires_days: expiry, preset: "hub" }
: { name: name.trim(), expires_days: expiry, grants: picked },
),
onSuccess: () => {
void qc.invalidateQueries({ queryKey: qk.apiTokens() });
// The Hub exchange makes the Hub folder the first time.
void qc.invalidateQueries({ queryKey: qk.fileFolders() });
},
meta: { failTitle: "errors.writeFailed.apiToken.create" },
});
const made = create.data;
const ready = name.trim() !== "" && (kind === "hub" || picked.length > 0);
if (made)
return (
<Modal
initialFocus="action"
title={t("settings.tokens.madeTitle")}
onClose={onClose}
maxWidth="max-w-md"
>
<div className="flex flex-col gap-3">
<InlineNotice>{t("settings.tokens.once")}</InlineNotice>
<CopyField label={t("settings.tokens.token")} value={made.token} autoFocus />
<CopyField
label={t("settings.tokens.tryIt")}
value={`curl -H "Authorization: Bearer ${made.token}" ${window.location.origin}/api/files/folders`}
/>
</div>
<ModalFooter>
<Button variant="primary" onClick={onClose}>
{t("settings.tokens.done")}
</Button>
</ModalFooter>
</Modal>
);
return (
<Modal
initialFocus="field"
title={t("settings.tokens.newTitle")}
onClose={onClose}
maxWidth="max-w-md"
>
<form
id={`${nameId}-form`}
className="flex flex-col gap-3"
onSubmit={(e) => {
e.preventDefault();
if (ready && !create.isPending) create.mutate();
}}
>
<div
role="group"
aria-label={t("settings.tokens.kind")}
className="grid grid-cols-1 gap-2 sm:grid-cols-2"
>
<KindTile
on={kind === "hub"}
icon={<HubIcon className="size-4" aria-hidden />}
title={t("settings.tokens.hubTitle")}
text={t("settings.tokens.hubText")}
onPick={() => setKind("hub")}
/>
<KindTile
on={kind === "custom"}
icon={<SlidersHorizontal className="size-4" aria-hidden />}
title={t("settings.tokens.customTitle")}
text={t("settings.tokens.customText")}
onPick={() => setKind("custom")}
/>
</div>
<label htmlFor={nameId} className="flex flex-col gap-1">
<span className="text-xs text-muted">{t("settings.tokens.name")}</span>
<Input
id={nameId}
data-initial-focus=""
className="w-full"
maxLength={80}
value={name}
onChange={(e) => setNameDraft(e.target.value)}
/>
</label>
{kind === "custom" && (
<div className="flex flex-col gap-1">
<span className="text-xs text-muted">{t("settings.tokens.folders")}</span>
<RightsGrid foldersQ={foldersQ} grants={grants} onChange={setGrants} />
</div>
)}
<div className="flex flex-col gap-1">
<span className="text-xs text-muted" aria-hidden>
{t("settings.tokens.expiry")}
</span>
<SegmentedToggle
size="md"
ariaLabel={t("settings.tokens.expiry")}
className="self-start"
options={TOKEN_EXPIRY_DAYS.map((d) => ({
value: expiryValue(d),
label: t(`settings.tokens.expiryOption.${expiryValue(d)}`),
}))}
value={expiryValue(expiry)}
onChange={(v) => setExpiry(v === "never" ? null : Number(v))}
/>
{expiry === null && <InlineNotice>{t("settings.tokens.neverWarning")}</InlineNotice>}
</div>
</form>
<ModalFooter>
<Button variant="outline" onClick={onClose}>
{t("common.cancel")}
</Button>
<Button
variant="primary"
type="submit"
form={`${nameId}-form`}
disabled={!ready || create.isPending}
>
{t("settings.tokens.create")}
</Button>
</ModalFooter>
</Modal>
);
}
/** A value to copy, shown whole (it wraps), with its Copy button. */
function CopyField({
label,
value,
autoFocus = false,
}: {
label: string;
value: string;
autoFocus?: boolean;
}) {
const { t } = useTranslation();
const { copied, copy } = useCopyFeedback();
return (
<div className="flex flex-col gap-1">
<span className="text-xs text-muted">{label}</span>
<div className="flex items-start gap-1.5">
<code className="min-w-0 flex-1 break-all rounded-lg border border-border bg-card px-2.5 py-1.5 font-mono text-xs">
{value}
</code>
<Button
variant="outline"
size="sm"
autoFocus={autoFocus}
icon={copied ? <Check className="w-4 h-4" /> : <Copy className="w-4 h-4" />}
// Two of these sit in one dialog: each is named by the value it copies, and by the word
// it shows, so a voice command saying what is on screen finds it.
aria-label={t(copied ? "settings.tokens.copiedField" : "settings.tokens.copyField", {
field: label,
})}
onClick={() => void copy(value)}
>
{copied ? t("settings.tokens.copied") : t("settings.tokens.copy")}
</Button>
</div>
</div>
);
}
@@ -1,48 +0,0 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import clsx from "clsx";
import { ChevronRight } from "lucide-react";
import type { AuditRow } from "../lib/api";
import { Button } from "./ui/Button";
import { RelativeTime } from "./ui/RelativeTime";
/** A folded run of one job's successful runs (lib/auditGroups): the row shows the newest, this
* opens the earlier ones, each with its moment and its result (`detailOf`, the row's own wording). */
export default function AuditEarlierRuns({
runs,
detailOf,
}: {
runs: AuditRow[];
detailOf: (run: AuditRow) => string | null;
}) {
const { t } = useTranslation();
const [open, setOpen] = useState(false);
const earlier = runs.slice(1);
return (
<div className="mt-1">
<Button
variant="link-muted"
aria-expanded={open}
onClick={() => setOpen(!open)}
icon={<ChevronRight className={clsx("transition-transform", open && "rotate-90")} />}
>
{t("audit.earlierRuns", { count: earlier.length })}
</Button>
{open && (
<ul className="mt-1 space-y-0.5 text-2xs text-muted">
{earlier.map((run) => (
<li key={run.id} className="flex gap-2">
<RelativeTime
at={run.created_at}
seconds
empty="—"
className="shrink-0 tabular-nums"
/>
<span className="min-w-0 wrap-break-word">{detailOf(run)}</span>
</li>
))}
</ul>
)}
</div>
);
}
-107
View File
@@ -1,107 +0,0 @@
import { useMemo } from "react";
import { qk } from "../lib/queryKeys";
import { useTranslation } from "react-i18next";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Trash2 } from "lucide-react";
import { api } from "../lib/api";
import { notify } from "../lib/notifications";
import { LS } from "../lib/storage";
import { Button } from "./ui/Button";
import { Section } from "./ui/form";
import { useConfirm } from "./confirmContext";
import DataTable from "./ui/DataTable";
import { LoadingState, StateMessage } from "./ui/QueryState";
import { auditColumns } from "./auditColumns";
import { notifyActionError } from "../lib/actionErrors";
import { compareText } from "../lib/collate";
import { foldJobRuns } from "../lib/auditGroups";
// Admin-only audit trail: an append-only who/what/when log of admin actions + scheduler run
// summaries (never per-row). Read-only apart from Clear. Client-side sort/filter/paginate via
// DataTable; the API returns the most recent window (capped) — the log stays small by design.
export default function AuditLog() {
const { t, i18n } = useTranslation();
const lang = i18n.language;
const qc = useQueryClient();
const confirm = useConfirm();
const q = useQuery({ queryKey: qk.adminAudit(), queryFn: () => api.adminAudit() });
const data = q.data;
// Back-to-back successful runs of one job fold into one row (lib/auditGroups).
const rows = useMemo(() => foldJobRuns(data?.items ?? []), [data]);
const actionOptions = useMemo(() => {
// By what the reader sees — the named action, in their alphabet — not by the code.
return Array.from(new Set(rows.map((r) => r.action)))
.map((a) => ({ value: a, label: t(`auditActions.${a}`, a) }))
.sort((a, b) => compareText(a.label, b.label, lang));
}, [rows, t, lang]);
const columns = useMemo(() => auditColumns(t, lang, actionOptions), [t, lang, actionOptions]);
const clear = useMutation({
mutationFn: () => api.clearAudit(),
onSuccess: (r) => {
qc.invalidateQueries({ queryKey: qk.adminAudit() });
notify({ level: "success", message: t("audit.cleared", { count: r.cleared }) });
},
onError: (e) => notifyActionError(e, t("audit.clearFailed")),
});
const onClear = async () => {
if (
await confirm({
title: t("audit.clearTitle"),
message: t("audit.clearConfirm"),
confirmLabel: t("audit.clear"),
danger: true,
})
)
clear.mutate();
};
const truncated = !!data && data.total > data.returned;
return (
<div className="p-4 max-w-5xl w-full mx-auto">
<Section card title={t("audit.title")}>
<p className="text-xs text-muted leading-relaxed mb-3">{t("audit.intro")}</p>
{q.isError && !q.data ? (
<StateMessage tone="error" onRetry={() => q.refetch()}>
{t("common.loadError")}
</StateMessage>
) : q.isLoading ? (
<LoadingState shape="table" />
) : rows.length === 0 ? (
<p className="text-sm text-muted">{t("audit.empty")}</p>
) : (
<>
<DataTable
rows={rows}
columns={columns}
rowKey={(r) => String(r.id)}
persistKey={LS.auditTable}
controlsPosition="top"
controlsInBand
controlsBandClassName="px-4 max-w-5xl mx-auto"
controlsLeading={
<Button
variant="outline-danger"
size="xs"
onClick={onClear}
disabled={clear.isPending}
icon={<Trash2 className="w-3.5 h-3.5" />}
>
{t("audit.clear")}
</Button>
}
emptyText={t("audit.empty")}
/>
{truncated && (
<p className="text-2xs text-muted mt-2">
{t("audit.truncated", { returned: data!.returned, total: data!.total })}
</p>
)}
</>
)}
</Section>
</div>
);
}
@@ -99,10 +99,10 @@ describe("ChannelPage — a busy toggle keeps the keyboard", () => {
});
// The banner is retried on error (googleusercontent drops a burst); while an attempt fails and the
// next waits, a visible <img> shows the browser's broken-image glyph (seen live). It stays
// invisible until a source has actually loaded.
// next waits, a visible <img> showed the browser's broken-image glyph (seen live). While a retry
// waits there is no <img> at all — the bar's own ground — and the retry asks past the cache.
describe("ChannelPage — the banner", () => {
it("is invisible until it loads, and again while a failed attempt is retried", async () => {
it("leaves the bar empty while a failed attempt waits, then asks again past the cache", async () => {
vi.useFakeTimers({ shouldAdvanceTime: true });
onTestFinished(() => {
vi.useRealTimers();
@@ -134,16 +134,14 @@ describe("ChannelPage — the banner", () => {
</QueryClientProvider>,
);
const banner = () =>
container.querySelector<HTMLImageElement>('img[src^="https://yt3.example/banner"]')!;
container.querySelector<HTMLImageElement>('img[src^="https://yt3.example/banner"]');
await waitFor(() => expect(banner()).toBeTruthy());
expect(banner().classList.contains("opacity-0")).toBe(true);
fireEvent.error(banner());
expect(banner()!.getAttribute("referrerpolicy")).toBe("no-referrer");
fireEvent.error(banner()!);
expect(banner()).toBeNull();
await act(async () => {
await vi.advanceTimersByTimeAsync(600);
await vi.advanceTimersByTimeAsync(1000);
});
expect(banner().getAttribute("src")).toContain("?r=1");
expect(banner().classList.contains("opacity-0")).toBe(true);
fireEvent.load(banner());
expect(banner().classList.contains("opacity-0")).toBe(false);
expect(banner()!.getAttribute("src")).toContain("?r=1");
});
});
+16 -58
View File
@@ -47,6 +47,7 @@ import { IconButton } from "./ui/IconButton";
import { IconLink } from "./ui/LinkButton";
import { PageHeading } from "./ui/Headings";
import { useTags } from "../lib/useTags";
import RemoteImage from "./ui/RemoteImage";
// A dedicated channel page: an "About"-style header (banner, avatar, stats, subscribe) over the
// channel's videos (the catalog filtered to this channel). For an un-subscribed channel it
@@ -94,17 +95,6 @@ export default function ChannelPage({
// is fixed, so unfolded it kept ~40% of a 133% screen for good. Folded, the name rides in the tab row.
const foldable = useRef<HTMLDivElement>(null);
const folded = useScrollFold(foldable);
// The banner comes from Google's image CDN, which refuses any request carrying a Referer from our
// origin — every time, not only in a burst (measured 2026-09-26) — so the <img> sends none
// (`referrerPolicy`, as ui/Avatar). A load can still drop, and the browser NEGATIVE-CACHES the miss,
// so on error we retry a few times with a cache-buster (forcing a fresh request), and only fall
// back to the blank bar if it keeps failing (a genuinely dead URL — rare).
const MAX_BANNER_RETRIES = 3;
const [bannerAttempt, setBannerAttempt] = useState(0);
// The banner source that has actually loaded. Until then the <img> is invisible: an attempt that
// fails shows the browser's broken-image glyph for as long as the retry waits (seen live).
const [bannerLoaded, setBannerLoaded] = useState<string | null>(null);
// A LIVE query: while THIS user's full-history backfill is running, poll so the stored count (and
// the N/S/L breakdown) climb on their own — the immediate backfill fills it in seconds, and the
// scheduler tops up any tail on its ~10-minute ticks — then stop polling once the back-catalogue is
@@ -113,13 +103,6 @@ export default function ChannelPage({
intervalMs: channelDetailPollMs,
});
const ch = detail.data;
// A fresh channel/banner gets the retries back, in the render that changes it: as an effect the
// new banner was committed with the previous one's exhausted attempt count still in place.
const [bannerFor, setBannerFor] = useState(ch?.banner_url);
if (bannerFor !== ch?.banner_url) {
setBannerFor(ch?.banner_url);
setBannerAttempt(0);
}
// The poll above is also the ONLY signal that the running backfill has written videos, so it
// drives the video grid's refresh too (the "Get full history" mutation below can't: it fires on
// the PATCH response, before the background job has stored anything).
@@ -348,9 +331,7 @@ export default function ChannelPage({
// The channel page owns no scroller of its own (PageShell provides the one app scroller). Its
// whole chrome is fixed — wrapped in <PageToolbar> it portals into the shell's fixed band, so the
// tabs/toolbar stay put and only the video grid scrolls; the banner/identity fold away meanwhile.
const bannerSrc = ch?.banner_url
? `${ch.banner_url}=w1707${bannerAttempt ? `?r=${bannerAttempt}` : ""}`
: null;
const bannerSrc = ch?.banner_url ? `${ch.banner_url}=w1707` : null;
return (
<>
<PageToolbar>
@@ -359,44 +340,21 @@ export default function ChannelPage({
<div ref={foldable}>
{/* Banner — OPTION C: inset, rounded card (not full-bleed) */}
<div className="px-4 sm:px-6 pt-3">
{bannerSrc && bannerAttempt <= MAX_BANNER_RETRIES ? (
// Match YouTube's banner: the stored bannerExternalUrl is the full 16:9 template at a
// low default size, so request a crisp wide version (=w1707) and object-cover the
// desktop "safe area" — the centre 2560×423 (~6:1) band.
<div
className="w-full overflow-hidden rounded-2xl bg-surface"
style={{ aspectRatio: "2560 / 423", maxHeight: "150px" }}
>
<img
// key forces a fresh <img> per attempt; the ?r= cache-buster on a retry bypasses the
// browser's negative cache (googleusercontent accepts an extra query param).
key={bannerAttempt}
src={bannerSrc}
referrerPolicy="no-referrer"
alt=""
className={`w-full h-full object-cover object-center ${
bannerLoaded === bannerSrc ? "" : "opacity-0"
}`}
onLoad={() => setBannerLoaded(bannerSrc)}
onError={() => {
if (bannerAttempt <= MAX_BANNER_RETRIES) {
window.setTimeout(
() => setBannerAttempt((a) => a + 1),
500 * (bannerAttempt + 1),
);
}
}}
/>
</div>
) : (
// No banner (or a dead one): a neutral bar the SAME height as a real banner, so the
// overlapping avatar + the Back button have the same room and don't collide (a short bar
// let the -mt-8 avatar ride up into the Back button).
<div
className="w-full rounded-2xl bg-surface"
style={{ aspectRatio: "2560 / 423", maxHeight: "150px" }}
{/* Match YouTube's banner: the stored bannerExternalUrl is the full 16:9 template at a
low default size, so request a crisp wide version (=w1707) and object-cover the
desktop "safe area" — the centre 2560×423 (~6:1) band. With no banner (or a dead one)
the bar keeps the SAME height, so the overlapping avatar and the Back button keep
their room (a short bar let the -mt-8 avatar ride up into the Back button). */}
<div
className="w-full overflow-hidden rounded-2xl bg-surface"
style={{ aspectRatio: "2560 / 423", maxHeight: "150px" }}
>
<RemoteImage
src={bannerSrc}
alt=""
className="w-full h-full object-cover object-center"
/>
)}
</div>
</div>
<div className="px-4 sm:px-6">
+3 -1
View File
@@ -515,7 +515,9 @@ export default function Channels() {
id: "backfill",
label: t("channels.backfillEverything"),
title: t("channels.backfillEverythingHint"),
icon: <History className={`h-4 w-4 ${deepAll.isPending ? "animate-pulse" : ""}`} />,
icon: (
<History className={`h-4 w-4 ${deepAll.isPending ? "motion-safe:animate-pulse" : ""}`} />
),
onSelect: () => deepAll.mutate(),
disabled: deepAll.isPending,
},
+4 -1
View File
@@ -54,7 +54,10 @@ export default function DownloadButton({
>
{/* The state is in the glyph: downloaded or queued is the accent, a running one pulses. */}
<Download
className={clsx((downloaded || inQueue) && "text-accent", inQueue && "animate-pulse")}
className={clsx(
(downloaded || inQueue) && "text-accent",
inQueue && "motion-safe:animate-pulse",
)}
/>
</IconButton>
{open && (
@@ -70,6 +70,7 @@ vi.mock("../lib/notifications", async (importOriginal) => ({
import DownloadTray from "./DownloadTray";
import { noteQueue, resetQueueWatch, showInTray } from "../lib/downloadQueue";
import { pushLayer, removeLayer } from "../lib/layerStack";
import { qk } from "../lib/queryKeys";
beforeEach(() => {
resetQueueWatch();
@@ -237,6 +238,28 @@ describe("DownloadTray", () => {
);
});
it("a finish refreshes Files at once, so the notice's Open finds the file listed", async () => {
const invalidate = vi.spyOn(QueryClient.prototype, "invalidateQueries");
poll([job(1, "running")]);
const view = mount();
expect(invalidate).not.toHaveBeenCalledWith({ queryKey: qk.files() });
poll([job(1, "done")], view);
await waitFor(() => expect(h.notify).toHaveBeenCalledTimes(1));
expect(invalidate).toHaveBeenCalledWith({ queryKey: qk.files() });
expect(invalidate).toHaveBeenCalledWith({ queryKey: qk.fileFolders() });
});
it("an upload the worker converted first is announced as uploaded, not downloaded", async () => {
poll([job(1, "running", { source_key: "upload" })]);
const view = mount();
poll([job(1, "done", { source_key: "upload" })], view);
await waitFor(() =>
expect(h.notify).toHaveBeenCalledWith(
expect.objectContaining({ title: "tray.done.uploadedTitle:Clip 1" }),
),
);
});
it("closed, it forgets what finished and stays away until the work changes", () => {
poll([job(1, "running"), job(2, "running")]);
const view = mount();

Some files were not shown because too many files have changed in this diff Show More