Compare commits

..
34 Commits
Author SHA1 Message Date
peter e9c60f72cd refactor(capture): the HTTP layer reads through queries and writes through the service
Capture's routes (downloads, files, public watch links) and the token list no longer build SQL, call
the session or set a model's fields: reads are downloads/queries, writes the unit's service (service,
folders, links, quota), and each write flushes for the handler to commit. Whose a row is goes
through userscope (owned_by, owned_row), which owned_or_404 now uses too.

The fence holds Capture below its routes as well: it reads only Catalog's and Plex's tables, writes
no other unit's, filters owners through userscope and commits only in its jobs and the upload
ingest. A test per service use case, among them that the tray lists only the caller's files.
2026-10-19 00:19:03 +02:00
peter 3a688fa0f5 refactor(capture): the unit's tables move into downloads/models
The nine Capture tables (files, jobs, folders, shares, links, quota, cookies, profiles, library
resume points) leave app/models.py for the unit's own module; app.tables registers every unit's
tables for the migrations, the test database and the ownership check. No schema change.
2026-10-18 23:45:53 +02:00
peter d66174ed16 test(arch): a unit's HTTP layer does no database work beyond its pin
The fence is switched on unit by unit: each route file's database work (SQL built, session data
calls, model attributes set, unit tables imported) is pinned at today's count and may only shrink.
A core route may not reach into a unit's tables. Every model has an owner.
2026-10-18 23:01:12 +02:00
peter c85ae42b31 release: 0.108.0 (feature/e40-job-history) 2026-10-18 21:48:23 +02:00
peter cb9a89682a chore(release): 0.108.0 — one Activity page for runs, admin actions, the server log and quota; every background run has a history 2026-10-18 21:40:52 +02:00
peter b842e4e871 fix(ui): a looping pulse stops for reduced motion; the e2e waits for skeleton bones, not for any pulse
- Seven looping pulses (a running job's dot, the scheduler's progress bar, a queued download, the
  download tray, a playlist upload, a channel's full-history action, the setup wizard's test mail)
  ran regardless of prefers-reduced-motion, which docs/DESIGN.md promises to respect: each is
  motion-safe now, held by a contract test.
- The a11y suite's "settled" wait took every aria-hidden pulse for a loading skeleton, so the
  Scheduler legend's running dot, which pulses for good, kept it waiting until it timed out. A
  skeleton bone now says it is one (data-skeleton) and the wait looks for that.
2026-10-18 21:40:52 +02:00
peter 5082636f15 fix(activity): a run nothing here has open reads as interrupted, a rollback sums a stopped run's spend
- The scheduler keeps one registry of the runs this process has open, for the interval jobs and the
  ad-hoc runs alike. A row the table still says is `running` but no run here has open (its outcome
  failed to write) reads as interrupted, as the next start's sweep will close it: no pulsing dot
  beside "No run recorded yet".
- Migration 0109's downgrade sums a still-running run's quota events into its spend before the link
  to them is dropped, as the startup sweep would.
- A test holds that a dialog closing onto the same address leaves an open log line open.
2026-10-18 21:21:19 +02:00
peter 13b7577d9f fix(activity): an ad-hoc run's row is one run's, a rollback keeps a run whole, Back reads one entry's filters
- The channel-backfill row says "running" from the runs this process has open, not from the table:
  a row whose outcome failed to write no longer reads as running until a restart. The run still
  going leads the row, so a newer run's outcome never sits under the older one's channel.
- Migration 0109's downgrade carries every run's trigger, spend and start in the audit row, and the
  upgrade restores them: a manual run whose admin was deleted stays manual.
- Back/Forward is read by the page's one popstate listener, both halves in one render; the server
  log adopts its half during that render, so no read goes out with one entry's filters and the
  other's range.
- The server log's failed facets use the other tabs' words (the menus may be short); the accounts
  message shows only when the menu really lacks them.
2026-10-18 17:32:22 +02:00
peter fad656988a fix(activity): runs keep their true outcome and spend, the page reaches every kept row, a closing dialog leaves the server log alone
Backend
- A run's outcome write sits outside the job's own try: a database blip after a run that
  succeeded no longer records it as failed (the row waits for the next start's sweep).
- An interrupted run keeps what it spent (the sweep sums its quota rows, as finish does).
- The Scheduler's ad-hoc row says running while any backfill runs, not only the newest.
- recent_runs reads each job's last runs off its index instead of ranking the whole table.
- An interval change is stored with its audit row in one transaction, then rescheduled.
- One API instance per database is the stated contract (.env.example, jobruns).
- Turning full history on for a channel already complete records no run.
- Migration 0109 round-trips interrupted and ad-hoc runs; the run words' DB CHECKs are
  test-held to the enums; the account filter is one helper for the four activity reads.
- An audit row says how many of its request's log lines are still kept.

Frontend
- An "All" period reaches every row still kept (runs and admin actions outlive 90 days).
- Back/Forward is followed in place by the server log: a closing dialog no longer turns
  Live off; a pivot still opens a fresh log.
- Runs moved over from the audit log read their one-liner in the reader's language.
- Filter choices or accounts that failed to load say so beside the filters, with a retry.
- An old server-log link keeps its account, text and range; a remembered retired page
  opens its tab.
- A request whose lines are gone says so instead of linking to an empty list.
- The log lines and the history rows share one row frame, and the server log one list
  frame with the other tabs; an empty list is announced; each day's quota is readable.
2026-10-18 15:38:32 +02:00
peter 3021be5166 fix(activity): a run's quota reads with its unit, its outcome keeps room at a large text size, a token's audit rows read in Hungarian
The run list showed the quota spend as a bare figure and, at the largest text size on a 1024-wide
screen, its fixed columns left the outcome column no width at all. The outcome now keeps 10rem and
the trigger column gives way; the audit list's summary keeps the same floor. A run that stored no
log line says so instead of linking to an empty list. An API token revoke and a token's file work
rendered the server's English line under the Hungarian UI; both are rebuilt in the reader's
language, and the test holds the server's wording.
2026-10-18 04:22:52 +02:00
peter c288ba147b fix(activity): an old audit/log link opens its tab, times read relative, one name for admin actions
Seen in the browser, not in the page's own tests: the navigation provider rewrites a retired page id
to `activity` before the page's chunk reads the URL, so `?page=audit` landed on the Runs tab. The
retired-page map now carries the params that open the successor's part (`at`), and the provider
writes them as it rewrites the page — on load and on Back alike.

A list time is relative with the exact second on hover (the DESIGN copy rule); a run moved over from
the audit log, whose start and end are one instant, shows no length instead of an invented "1 sec".
The audit log is called "admin actions" everywhere it is named (the clear button and its confirm, the
Configuration group, the cleanup job, which also prunes the run history its description left out).
2026-10-18 04:00:38 +02:00
peter fe6308d844 refactor(activity): the window's hours stay private to its URL state 2026-10-18 03:48:30 +02:00
peter 24448e7c6f feat(activity): one Activity page — runs, admin actions, server log and quota under one filter bar
The audit log, the server log and the admin quota view were three pages (the quota one a Stats tab),
each with its own filters, and the background runs had no list at all. Now one page ("Napló" /
"Activity") reads all four, each still its own table on the server: a shared bar (text, a period of
1 h to 90 days defaulting to 7, an account) and each tab's own filters, the whole view in the URL.
The old page ids open their tab.

Server side: the audit log and the new run list (`/api/admin/runs`) filter and page like the server
log, through one keyset helper (`routes/_keyset`) the log now shares; an audit row keeps the request
it was made in (migration 0111), so its log lines are a pivot away, as a run's are (`lrun`). The
quota breakdown takes since/until and one account. The Scheduler rows get a history button to their
job's runs.

Also fixed on the way: re-saving the value in force on the Configuration page staged its override
without committing it (the commit sat inside the audit branch), so the save was lost.
2026-10-18 03:45:18 +02:00
peter d13483a593 feat(config): job intervals live in sysconfig, every setting says when it applies, the unread SponsorBlock default goes
An interval override was a row of its own table beside the one store every other admin knob uses; it
is now the `job_interval_<job>` sysconfig key (migration 0110 copies them over and drops the table).
The Scheduler page stays its one editor: the Configuration page and the setup wizard neither list nor
write those keys, since a write there would skip the reschedule.

Each ConfigSpec now states `applies` (required, no default). Every key is `live`, held by a scan that
finds no read at import time; a `restart` key is refused until the page can say so. A second scan
requires a reader for every key on the Configuration page — `sponsorblock_default` had none since it
was added, so it leaves the registry, the translations and the store.
2026-10-18 02:23:15 +02:00
peter 3bab2e7dc7 docs(scheduler): the snapshot docstring names the ad-hoc runs it now carries 2026-10-18 01:29:46 +02:00
peter 1b1e3cdcf2 feat(scheduler): each row shows its last three runs, an interrupted run as such, and the ad-hoc runs beside the jobs
A row's one dot said how the LAST run went, and an interrupted run fell through to "idle" — a grey dot
that read as "no run recorded". The row now leads with its last three runs, newest first, each named
for a screen reader and in its tooltip (how it went, when, who started it); an interrupted run is a
ring, set apart from a skipped run and a failure, and its line says the server restarted under it.

A channel's full-history backfill gets its own "Ad-hoc runs" block: its runs, its channel and what
it fetched, no interval and no run button — it needs its target. AD_HOC_JOBS names every such job;
the backend-id contract holds their names in both languages, and run_ad_hoc refuses any other id.
2026-10-18 01:20:53 +02:00
peter 27c2fcbd2e feat(scheduler): a channel's "get full history" backfill is a recorded run, and a row knows its last three runs
The one-channel deep backfill ran as a FastAPI BackgroundTask: what it fetched, what it spent and
how it failed reached only the log. It is now a one-shot run on the scheduler (`channel_backfill`,
trigger ad_hoc, actor = the user, target = the channel) with its quota tagged to the run. Its
per-channel exclusion with the scheduled backfill and the two-run cap are unchanged, and a run
they turn away says why in its result instead of returning nothing. A failure is no longer
swallowed: the run records it.

The scheduler snapshot carries each job's last three runs (one windowed query), so the page can
show an interrupted or skipped run instead of the latest outcome alone.
2026-10-18 01:05:24 +02:00
peter 911527a548 fix(scheduler): the dashboard no longer says a job has not run "this session" — runs are kept across restarts 2026-10-17 23:22:13 +02:00
peter 8fd91b7b15 fix(scheduler): keep run history 180 days, as long as the audit log kept the runs it moved over 2026-10-17 23:15:17 +02:00
peter 45c0ee7d58 feat(scheduler): every background run is a durable row
What a job last did lived in the API process's memory and was gone at
every restart; the audit log kept a run only when it was manual, failed
or changed something. Now each run is a job_runs row written at its
start and its end in its own session: who started it, its outcome as
data, what it spent of the YouTube quota (quota_events.job_run_id) and
the lines it logged (log_events.run_id, filterable in the log console).

- a run a stopped process left open is marked interrupted at startup
- "run now" is a one-shot job on the scheduler, which now always starts
  (interval jobs only with SCHEDULER_ENABLED); the manual actor is reset
  after the run, since pool threads are reused
- the dashboard's last run comes from job_runs, so it survives a deploy
- runs leave the audit log: migration 0109 moves the old job_run rows
  over (and back on downgrade); the audit page's run folding goes
- job_run_retention_days (default 30) prunes old runs in audit_gc
2026-10-17 23:08:55 +02:00
peter 4b1f6d7ec2 release: 0.107.3 (improvement/dev-tooling) 2026-10-17 21:34:53 +02:00
peter 51bf780100 chore(release): 0.107.3 — text on glass holds 4.5:1; the e2e suite runs axe, the gate lints every shell script 2026-10-17 21:21:37 +02:00
peter 3ada34073b fix(theme): text on glass holds 4.5:1 — light --muted kept at 6:1 on the page, tonal fills opaque
Pixel-measured over every scheme x mode x backdrop: the item count fell to 3.8:1 (dark too: its 8%
fill let the toolbar glass and the thumbnails under it through), Channels' tabs to 4.1:1 and the
current Files row to 4.4:1. Light --muted moves 15% toward --fg in every scheme; the badges, the item
count and the current rail row paint opaque bg-tint* fills mixed into the surface. The token contract
now asks 6:1 of --muted on the bare page and checks those three sites use the tints.
2026-10-17 20:06:54 +02:00
peter d74f2a9ca1 fix(deps): source-map-js 1.2.2 — the indexed-source-map denial-of-service advisory (dev tooling only) 2026-10-17 17:51:10 +02:00
peter a8dddc8c24 test(e2e): axe-core's WCAG A + AA rules over every page and the player (chromium only); contrast stays with the visual check 2026-10-17 17:48:53 +02:00
peter 460bdec406 chore(tooling): drop scripts/lint-changed.sh — the shared gate runner's Stop hook lints the turn's files, shellcheck included 2026-10-17 17:22:35 +02:00
peter 66181d1394 chore(gate): shellcheck joins the gate — every tracked shell script, the host binary held to a pin 2026-10-17 15:31:34 +02:00
peter 0a1fef28ac docs(claude): the YouTube write-error helper, the overlay and z-scale rule, and px-frozen sizes join the canonical patterns 2026-10-17 14:47:57 +02:00
peter fe572d9207 docs: a copy-only change runs the Conformance check's step 4 alone, in both languages 2026-10-17 14:27:45 +02:00
peter c8247c0dde release: 0.107.2 (improvement/gate-runner) 2026-10-17 04:37:30 +02:00
peter 722c37a6d0 chore(release): 0.107.2 — the gate lists and runs single lanes, the backend tests use every core 2026-10-17 04:30:33 +02:00
peter 5ba5679239 test: tests that assumed an idle machine wait for what they assert, or carry a budget for their work
Under a busy machine (the gate's lanes side by side, 12 busy cores) these went red at random: a press landed while the button was still busy, an answer was settled before it was asked for, a page grew past one wait's budget, a lazy chunk was transformed inside a 1 s wait, focus was taken a frame after the press. They now wait for the condition they depend on, preload the chunks they open, and the tests that do heavy work (whole-repo scans, the type-aware linter, module resets) carry an explicit budget naming that work. The whole suite four times under 12 busy-loop processes: red 4/4 before, 0/4 after.
2026-10-17 01:08:15 +02:00
peter 926e5e9899 chore(gate): check.sh can list its lanes and run one of them
--list prints the lane names, --only "<lane>" runs that one lane, so a runner can schedule the lanes itself and a red lane can be re-run on its own. An --only run reports that lane, never the gate, and an unknown name exits 2 instead of passing with nothing run; the contract test holds both.
2026-10-16 02:53:30 +02:00
peter 9251918e08 chore(test): the DB lane runs on every core — each pytest-xdist worker gets its own database 2026-10-16 01:13:31 +02:00
205 changed files with 9153 additions and 3464 deletions
+5 -3
View File
@@ -68,9 +68,11 @@ SMTP_FROM=
# --- Scheduler ---
# The background scheduler (subscription sync, backfill, enrichment) runs inside the app.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. If you ever run
# more than one instance against the same database, keep SCHEDULER_ENABLED=true on exactly one
# of them and false on the rest, to avoid double quota use and write races.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. Run ONE app instance
# per database: a starting instance closes every run still open as interrupted (it assumes the
# previous process is gone), and "is this job already running" is held in the process. A second
# instance would cut the first one's runs short and could run the same job twice.
# SCHEDULER_ENABLED=false keeps the recurring jobs off; "run now" still works.
SCHEDULER_ENABLED=true
# --- Files library & downloads ---
+29
View File
@@ -7,6 +7,35 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.108.0 — 2026-10-18
### New
- Activity: one admin page for background runs, admin actions, the server log and quota, with one time range, account and search. A run opens to its outcome and its log lines, an admin action to its request's lines; old audit and log links open their tab.
- Scheduler: each job shows its last three runs (a run cut short by a restart is marked) and a button opens all of them; turning on a channel's full history is now a run of its own, listed under Ad-hoc runs.
- Configuration: how long the run history is kept (Activity › Run history, 180 days by default). Stats keeps your own numbers; the system quota view moved to Activity › Quota.
### Fixed
- Configuration: saving a setting with the value already in force now keeps it, instead of dropping the change.
- Configuration: the unused SponsorBlock default setting is gone.
## 0.107.3 — 2026-10-17
### New
- Development: the end-to-end suite runs the WCAG A and AA accessibility checks over every page and the player, and every shell script is linted in the gate.
### Fixed
- Readability: muted text in the light themes is a shade darker, and the item count, the labels and the selected Files folder sit on a solid tint, so text on the frosted-glass bars stays readable (4.5:1) in every scheme.
## 0.107.2 — 2026-10-17
### New
- Development: the gate script can list its checks (scripts/check.sh --list) and run just one (--only), and the backend tests run on every CPU core, each worker with its own database.
## 0.107.1 — 2026-10-16
### Fixed
+10 -7
View File
@@ -12,7 +12,7 @@ router-free navigation provider.
## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
The local gate is **ruff (`check` + `format --check`) + shellcheck + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip + vitest + vite build + pytest + pip-audit + npm audit**. It runs
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
@@ -55,14 +55,15 @@ how it gets built. Concretely:
- **Challenge unsound requests.** If a request is technically ill-advised and you can back it with real
arguments, push back or propose an alternative — don't comply blindly.
- **Verify UI in the browser before calling it done.** tsc/unit/DOM tests miss visual and interaction
bugs; do a real visual check of any UI change, and run DESIGN.md's four-step Conformance check.
bugs; do a real visual check of any UI change, and run DESIGN.md's Conformance check (a copy-only
change runs its step 4 alone: every state the new text renders in, in both languages).
- **Be terse everywhere else.** Outside those decision points, lead with the answer or decision. During
implementation and debugging especially: no restating the same point reworded, no filler.
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell (a shared `/watch` link boots from its own
@@ -91,8 +92,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` and `jobruns` own their session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
@@ -101,7 +102,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect | a bare `catch {}` or `notify()` with no error attached |
| FE floating or full-screen layer | `ui/Overlay` (portals to `<body>`) or the popover engine (`AnchoredPopover` portals itself); stacking by the z-index scale (`z-menu` … `z-popover`, docs/DESIGN.md "Layers") | one rendered in place inside a virtualized row or a masked/transformed scroller — clipped or painted over; a magic `z-[n]` |
| FE player on a native media element | `lib/mediaSession` (`useMediaSession`) + `lib/mediaKeys` + `lib/resume` (`useResume`), the element held as STATE | a private `navigator.mediaSession` / `keydown` / position-saving effect per player |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
@@ -114,7 +116,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
Not repeated here, because a lint rule refuses the wrong one and its message names the right one:
buttons (`ui/Button`, `IconButton`, `Pressable`, `Chip`, `DragHandle`), text fields and native
controls (`ui/Input`, `Checkbox`, `Slider`, …), glass surfaces (`SURFACE.*`), a native `<select>`
(`SegmentedToggle` / `FilterSelect`), a picture (`ui/RemoteImage`, never a bare `<img>`), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
(`SegmentedToggle` / `FilterSelect`), a picture (`ui/RemoteImage`, never a bare `<img>`), a box or font size (the spacing scale or rem,
never a px-frozen `w-[Npx]` / `text-[Npx]`: the text-size slider scales the root font), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
## House rules that bite (learned from real bugs)
+1 -1
View File
@@ -1 +1 @@
0.107.1
0.108.0
-76
View File
@@ -630,41 +630,7 @@
}
}
],
"./app/routes/admin_logs.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 15,
"endColumn": 47,
"lineCount": 1
}
}
],
"./app/routes/audit.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 17,
"endColumn": 98,
"lineCount": 1
}
},
{
"code": "reportAssignmentType",
"range": {
"startColumn": 17,
"endColumn": 98,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 22,
"endColumn": 97,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
@@ -726,48 +692,6 @@
}
}
],
"./app/routes/downloads.py": [
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 17,
"endColumn": 19,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 82,
"endColumn": 84,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 87,
"endColumn": 89,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 30,
"endColumn": 32,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 49,
"endColumn": 54,
"lineCount": 1
}
}
],
"./app/routes/playlists.py": [
{
"code": "reportOptionalOperand",
+1 -1
View File
@@ -4,7 +4,7 @@ from sqlalchemy import engine_from_config, pool
from app.config import settings
from app.db import Base
from app.logs import configure_logging
import app.models # noqa: F401 (import so models register on Base.metadata)
import app.tables # noqa: F401 (import so every table registers on Base.metadata)
config = context.config
config.set_main_option("sqlalchemy.url", settings.database_url)
+160
View File
@@ -0,0 +1,160 @@
"""Background runs get a durable history
Revision ID: 0109_job_runs
Revises: 0108_folder_links
Create Date: 2026-10-17
What a scheduler job last did lived in the API process's memory, gone at every restart; the audit
log kept a row for a run only when it was manual, failed, or changed something. Now every run is a
`job_runs` row, written when it starts and when it ends. A run's quota spend points at it
(`quota_events.job_run_id`), and so do the lines it logged (`log_events.run_id`, no foreign key: the
log writer runs behind the event).
Runs are not admin actions, so the audit log stops carrying them: the existing `job_run` audit rows
MOVE into `job_runs` (start = end = when the row was written, the only time it kept), and leave the
audit log. The downgrade moves every run back into the audit log as such a row (one still running as
interrupted, what the audit row has no column for in its `after`), then drops the table.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0109_job_runs"
down_revision: str | None = "0108_folder_links"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"job_runs",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("job", sa.String(40), nullable=False),
sa.Column("trigger", sa.String(16), nullable=False),
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="SET NULL")),
sa.Column("target", sa.String(128)),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True)),
sa.Column("summary", sa.Text()),
sa.Column("result", postgresql.JSONB()),
sa.Column("error", sa.Text()),
sa.Column("failure", postgresql.JSONB()),
sa.Column("quota_units", sa.Integer(), nullable=False, server_default="0"),
sa.CheckConstraint(
"status IN ('running', 'ok', 'error', 'skipped', 'interrupted')",
name="ck_job_runs_status",
),
sa.CheckConstraint(
"trigger IN ('scheduled', 'manual', 'ad_hoc')", name="ck_job_runs_trigger"
),
sa.CheckConstraint(
"(status = 'running') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
op.create_index("ix_job_runs_job_started", "job_runs", ["job", "started_at"])
op.create_index("ix_job_runs_actor_id", "job_runs", ["actor_id"])
op.create_index("ix_job_runs_started_at", "job_runs", ["started_at"])
op.add_column(
"quota_events",
sa.Column("job_run_id", sa.BigInteger(), sa.ForeignKey("job_runs.id", ondelete="SET NULL")),
)
op.create_index("ix_quota_events_job_run_id", "quota_events", ["job_run_id"])
op.add_column("log_events", sa.Column("run_id", sa.BigInteger()))
op.create_index(
"ix_log_events_run",
"log_events",
["run_id"],
postgresql_where=sa.text("run_id IS NOT NULL"),
)
# The audit trail's runs move over: the three statuses the scheduler wrote, and `interrupted`,
# which a downgrade writes back (as it does a run's trigger, target, spend and start, which the
# audit row has no column for: its actor alone cannot say "manual" once that admin is deleted),
# so a rollback and a redeploy keep every run as it was. A status the run table does
# not know would fail the CHECK.
op.execute(
"""
INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at, finished_at,
summary, result, error, failure, quota_units)
SELECT target_id,
coalesce(after::jsonb ->> 'trigger',
CASE WHEN actor_id IS NULL THEN 'scheduled' ELSE 'manual' END),
actor_id,
after::jsonb ->> 'target',
after::jsonb ->> 'status',
coalesce((after::jsonb ->> 'started_at')::timestamptz, created_at),
created_at,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb -> 'result' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb -> 'result' END,
coalesce((after::jsonb ->> 'quota_units')::int, 0)
FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
ORDER BY id
"""
)
op.execute(
"""
DELETE FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
"""
)
def downgrade() -> None:
# A run still `running` is one the rollback stopped: it goes back as interrupted. What the audit
# row's own columns cannot say rides in `after` (its trigger, target, spend and start), so the
# upgrade restores them.
op.execute(
"""
INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after, created_at)
SELECT actor_id,
'job_run',
'job',
job,
left(job || ': ' || st
|| coalesce(' — ' || CASE WHEN st = 'error' THEN error ELSE summary END, ''),
255),
(jsonb_build_object(
'status', st,
'summary', CASE WHEN st = 'error' THEN error ELSE summary END)
|| CASE
WHEN st = 'error' AND failure IS NOT NULL
THEN jsonb_build_object('result', failure)
WHEN st <> 'error' AND result IS NOT NULL
THEN jsonb_build_object('result', result)
ELSE '{}'::jsonb
END
|| jsonb_build_object('trigger', trigger, 'quota_units', units,
'started_at', started_at)
|| CASE WHEN target IS NOT NULL
THEN jsonb_build_object('target', target) ELSE '{}'::jsonb END)::json,
coalesce(finished_at, started_at)
FROM (SELECT *, CASE WHEN status = 'running' THEN 'interrupted' ELSE status END AS st,
-- A run stopped mid-way has not summed its spend yet: sum it as the startup
-- sweep would, before the link to it is dropped.
CASE WHEN status = 'running'
THEN (SELECT coalesce(sum(q.units), 0) FROM quota_events q
WHERE q.job_run_id = job_runs.id)
ELSE quota_units END AS units
FROM job_runs) AS r
ORDER BY id
"""
)
op.drop_index("ix_log_events_run", table_name="log_events")
op.drop_column("log_events", "run_id")
op.drop_index("ix_quota_events_job_run_id", table_name="quota_events")
op.drop_column("quota_events", "job_run_id")
op.drop_index("ix_job_runs_started_at", table_name="job_runs")
op.drop_index("ix_job_runs_actor_id", table_name="job_runs")
op.drop_index("ix_job_runs_job_started", table_name="job_runs")
op.drop_table("job_runs")
@@ -0,0 +1,79 @@
"""Job intervals move into sysconfig; the unread SponsorBlock default goes
Revision ID: 0110_intervals_in_sysconfig
Revises: 0109_job_runs
Create Date: 2026-10-18
An admin's interval override lived in its own table, `scheduler_settings`, beside the one store every
other admin knob uses. Each override becomes the `job_interval_<job>` sysconfig key and the table
goes. Only the jobs that still exist are copied: a row for any other id never did anything.
The `sponsorblock_default` setting was offered on the Configuration page but nothing ever read it, so
its key is gone and its stored value with it — the downgrade cannot bring that value back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0110_intervals_in_sysconfig"
down_revision: str | None = "0109_job_runs"
branch_labels: str | None = None
depends_on: str | None = None
# The interval jobs as of this revision (a migration does not read the live code's list).
_JOBS = (
"rss_poll",
"enrich",
"backfill",
"autotag",
"shorts",
"subscriptions",
"playlist_sync",
"maintenance",
"explore_cleanup",
"download_gc",
"plex_sync",
"plex_watch_sync",
"plex_watch_reconcile",
"tmdb_enrich",
"people_enrich",
"audit_gc",
"log_gc",
)
_PREFIX = "job_interval_"
def upgrade() -> None:
op.execute(
sa.text(
"INSERT INTO system_config (key, value, encrypted)"
" SELECT :prefix || job_id, interval_minutes::text, false FROM scheduler_settings"
" WHERE interval_minutes IS NOT NULL AND job_id IN :jobs"
" ON CONFLICT (key) DO NOTHING"
).bindparams(sa.bindparam("jobs", expanding=True), prefix=_PREFIX, jobs=list(_JOBS))
)
op.drop_table("scheduler_settings")
op.execute("DELETE FROM system_config WHERE key = 'sponsorblock_default'")
def downgrade() -> None:
op.create_table(
"scheduler_settings",
sa.Column("job_id", sa.String(40), primary_key=True),
sa.Column("interval_minutes", sa.Integer(), nullable=False),
)
op.execute(
sa.text(
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
" SELECT substr(key, :start), value::int FROM system_config WHERE key IN :keys"
).bindparams(
sa.bindparam("keys", expanding=True),
start=len(_PREFIX) + 1,
keys=[_PREFIX + j for j in _JOBS],
)
)
op.execute(
sa.text("DELETE FROM system_config WHERE key IN :keys").bindparams(
sa.bindparam("keys", expanding=True), keys=[_PREFIX + j for j in _JOBS]
)
)
@@ -0,0 +1,28 @@
"""An audit row keeps the request that made it
Revision ID: 0111_audit_request_id
Revises: 0110_intervals_in_sysconfig
Create Date: 2026-10-18
The activity page opens an admin action's log lines from its row: `audit_log.request_id` is the
request the change was made in, the same id `log_events.request_id` carries. Older rows have none —
the request that made them was never recorded.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0111_audit_request_id"
down_revision: str | None = "0110_intervals_in_sysconfig"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("audit_log", sa.Column("request_id", sa.String(64)))
op.create_index("ix_audit_log_request_id", "audit_log", ["request_id"])
def downgrade() -> None:
op.drop_index("ix_audit_log_request_id", table_name="audit_log")
op.drop_column("audit_log", "request_id")
+7 -5
View File
@@ -21,7 +21,8 @@ Three layers, by *who* runs the code — not by feature:
`issues/bundle.py` is the one definition of a bug report's files, for the zip export and the tracker.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth` (with `apitokens`, a machine's
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`,
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`, `jobruns` (every
background run's durable row, its words in `runstatus`),
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
@@ -36,7 +37,7 @@ pre-abstract for a single caller; don't leave shared logic duplicated once there
## Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
(only `quota` and `jobruns` own their session)"** — so read it that way, not as an exception to it. The commit belongs
to the **operation owner**; anything called *inside* another transaction only stages its change
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
@@ -45,7 +46,7 @@ mutation it records, so an audit entry can never exist without its change (or vi
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
`jobruns`' run rows — the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
attribute — see `service.decr_ref_count`.
@@ -67,8 +68,9 @@ mutation it records, so an audit entry can never exist without its change (or vi
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
startup, not user-tunable at runtime.
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
slated to fold into `sysconfig` — do not add more like it.)
goes in `SPECS`; don't invent a fourth store. Each spec states when a change takes effect
(`applies`, required): `live` keys are read where they are used, never at import (test-held). The
scheduler's job intervals are `SPECS` keys too, edited on the Scheduler page, not the Configuration page.
## The gate
+7 -7
View File
@@ -1,9 +1,9 @@
"""Admin/scheduler audit trail: an append-only record of who changed what and when.
"""Admin audit trail: an append-only record of who changed what and when.
Design (locked): log at ACTION / job-run granularity, NEVER per-row — a scheduler run that
touches thousands of videos is ONE audit row summarising the outcome. Automatic job runs are
only logged when they actually changed something (or errored); all admin actions and all manual
runs are always logged. Secrets are never stored (config changes log the key name only).
Design (locked): log at ACTION granularity, NEVER per-row — an action that touches thousands of
rows is ONE audit row summarising the outcome. Background runs are not actions: they have their
own history (app/jobruns.py). Secrets are never stored (config changes log the key name only).
Each row keeps the request that made it, so the request's log lines are one click away.
`record()` only db.add()s the row — the caller commits it in the SAME transaction as the
mutation it describes, so the trail can't drift from reality (and a rolled-back mutation drops
@@ -15,6 +15,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from sqlalchemy.orm import Session
from app import logs
from app.models import AuditLog
@@ -48,8 +49,6 @@ class AuditAction:
MAINTENANCE_TUNE = "maintenance_tune"
SYNC_PAUSE = "sync_pause"
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
@@ -104,6 +103,7 @@ def record(
summary=(summary or None) and summary[:255],
before=before,
after=after,
request_id=logs.current_request_id(),
)
)
+4 -2
View File
@@ -268,10 +268,12 @@ class Settings(BaseSettings):
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
quota_retention_days: int = 365
# Background-run history (job_runs): days kept (0 = forever). The audit log's 180, which is how
# long a run lived there before it moved here: ~1.8k rows a day at the default intervals
# (measured 2026-10-17), ~320k at the limit. The same daily cleanup job prunes it.
job_run_retention_days: int = 180
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: DownloadLayout = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
sponsorblock_default: bool = False
# --- Diagnostics / logging ---
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
+1 -1
View File
@@ -22,7 +22,7 @@ from app.db import SessionLocal
from app.downloads import adapters
from app.downloads import edit as editmod
from app.downloads import formats, service
from app.models import MediaAsset
from app.downloads.models import MediaAsset
log = logs.get_logger(__name__)
+4 -3
View File
@@ -33,8 +33,9 @@ from app import logs
from app.downloads import clip as clipmod
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential, MediaAsset
from app.downloads.models import DownloadCredential, MediaAsset
from app.security import decrypt_optional, encrypt
from app.userscope import owned_by
log = logs.get_logger(__name__)
@@ -157,7 +158,7 @@ def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredential | None:
return db.execute(
select(DownloadCredential).where(
DownloadCredential.user_id == user_id, DownloadCredential.host_key == host_key
owned_by(DownloadCredential.user_id, user_id), DownloadCredential.host_key == host_key
)
).scalar_one_or_none()
@@ -214,7 +215,7 @@ def status_list(db: Session, user_id: int) -> list[dict]:
have = {
c.host_key: c
for c in db.execute(
select(DownloadCredential).where(DownloadCredential.user_id == user_id)
select(DownloadCredential).where(owned_by(DownloadCredential.user_id, user_id))
).scalars()
}
out = []
+31 -3
View File
@@ -32,7 +32,8 @@ from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobStatus
from app.models import DownloadJob, FileFolder, MediaAsset
from app.downloads.models import DownloadJob, FileFolder, MediaAsset
from app.userscope import owned_by
class FolderKind(StrEnum):
@@ -106,7 +107,9 @@ def source_key_for(db: Session, job: DownloadJob) -> str:
def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
"""The user's source folder for `key`, made on first use. A race on the unique (user, key) is
settled by re-selecting, inside a SAVEPOINT so the caller's own pending writes survive."""
stmt = select(FileFolder).where(FileFolder.user_id == user_id, FileFolder.source_key == key)
stmt = select(FileFolder).where(
owned_by(FileFolder.user_id, user_id), FileFolder.source_key == key
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
@@ -131,7 +134,7 @@ def hub_folder(db: Session, user_id: int) -> FileFolder:
"""The user's Hub folder, made on first use; a race on its one-per-user index is settled like
`source_folder`'s."""
stmt = select(FileFolder).where(
FileFolder.user_id == user_id, FileFolder.kind == FolderKind.HUB
owned_by(FileFolder.user_id, user_id), FileFolder.kind == FolderKind.HUB
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
@@ -310,3 +313,28 @@ def delete_own(db: Session, folder: FileFolder) -> int:
db.flush() # the files leave before the folder goes: its foreign key RESTRICTs the delete
db.delete(folder)
return len(jobs)
def create_own(db: Session, user_id: int, name: str) -> FileFolder:
"""A new folder of the user's own; FolderError("file_folder_name_taken") when they already have
one by that name (the unique (user, lower(name)))."""
folder = FileFolder(user_id=user_id, kind=FolderKind.OWN, name=name)
try:
with db.begin_nested():
db.add(folder)
db.flush()
except IntegrityError as e:
raise FolderError("file_folder_name_taken") from e
return folder
def rename(db: Session, folder: FileFolder, name: str) -> None:
"""Rename one of the user's own folders (a source folder's name is its origin's)."""
if folder.kind != FolderKind.OWN:
raise FolderError("file_folder_not_own")
folder.name = name
try:
with db.begin_nested():
db.flush()
except IntegrityError as e:
raise FolderError("file_folder_name_taken") from e
+1 -1
View File
@@ -31,7 +31,7 @@ from app import apitokens, logs, sysconfig
from app.config import settings
from app.downloads import quota, service, storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
from app.notifications import create_notification
log = logs.get_logger(__name__)
+1 -1
View File
@@ -6,7 +6,7 @@ from sqlalchemy import text
from app.config import settings
from app.db import SessionLocal
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
# --- small DB helpers (short-lived sessions; the download itself holds no transaction) -------
+124 -6
View File
@@ -15,20 +15,24 @@ grant appended to the file URL, so the raw password never rides in the media URL
import hashlib
import hmac
import secrets
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from sqlalchemy import update as sql_update
from sqlalchemy.orm import Session
from app import security
from app import audit, security
from app.config import settings
from app.downloads import storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.models import User
from app.utils import iso
from app.userscope import owned_by
_GRANT_TTL = 6 * 3600 # a watch session's grant is good for 6h, then the viewer re-unlocks
# The most files a folder link lists: as many as the owner's own Files list shows (`routes.files.
# MAX_LIST`; a test holds the two equal — this layer may not import that one).
# The most files a folder link lists: as many as the owner's own Files list shows
# (`queries.MAX_LIST`; a test holds the two equal — `queries` imports this module).
FOLDER_MAX = 500
@@ -59,7 +63,7 @@ def in_folder(folder: FileFolder) -> tuple:
moved out stops opening through the link at its next request; one moved in starts."""
return (
DownloadJob.folder_id == folder.id,
DownloadJob.user_id == folder.user_id,
owned_by(DownloadJob.user_id, folder.user_id),
DownloadJob.status == JobStatus.DONE,
)
@@ -170,3 +174,117 @@ def public_meta(
"file_url": file_url,
"poster_url": poster_url,
}
# --- the owner's writes --------------------------------------------------------------------
# Each flushes and records its audit row; the handler commits.
def expiry(days: int | None) -> datetime | None:
"""None/0/negative → no expiry; otherwise N days out, capped at ~10 years."""
if not days or days <= 0:
return None
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
def audit_view(link: DownloadLink) -> dict:
"""A link's settings as the audit records them: whether it has a password, never which."""
return {
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": iso(link.expires_at),
}
def create(
db: Session,
user: User,
*,
password: str,
allow_download: bool,
expires_days: int | None,
**target: int,
) -> tuple[DownloadLink, str]:
"""A link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and the
token it is opened by — which is in hand only now."""
token = new_token()
link = DownloadLink(
**target,
created_by=user.id,
allow_download=allow_download,
password_hash=security.hash_password(password) if password else None,
expires_at=expiry(expires_days),
)
seal(link, token)
db.add(link)
db.flush()
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_CREATE,
target_type="download_link",
target_id=link.id,
after={**target, **audit_view(link)},
)
return link, token
def update(
db: Session,
user: User,
link: DownloadLink,
*,
allow_download: bool | None = None,
expires_days: int | None = None,
password: str | None = None,
sent: frozenset[str] = frozenset(),
) -> None:
"""Change what the caller sent (`sent` names the fields, so `expires_days=None` can mean "never
expires"). Setting, clearing or changing the password bumps its version, so any grant issued
under the previous one stops working now rather than living out its TTL."""
before = audit_view(link)
if "allow_download" in sent:
link.allow_download = bool(allow_download)
if "expires_days" in sent:
link.expires_at = expiry(expires_days)
if "password" in sent:
pw = (password or "").strip()
link.password_hash = security.hash_password(pw) if pw else None
link.password_version += 1
after = audit_view(link)
if after != before or "password" in sent:
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
target_type="download_link",
target_id=link.id,
before=before,
after=after,
)
db.flush()
def revoke(db: Session, user: User, link: DownloadLink) -> None:
link_id = link.id
db.delete(link)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
target_type="download_link",
target_id=link_id,
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
)
db.flush()
def count_view(db: Session, link: DownloadLink) -> None:
"""One more opening of the link. In SQL, not `+= 1` on the loaded row: two opens at once would
both write the same count."""
db.execute(
sql_update(DownloadLink)
.where(DownloadLink.id == link.id)
.values(view_count=DownloadLink.view_count + 1)
.execution_options(synchronize_session=False)
)
+385
View File
@@ -0,0 +1,385 @@
"""Capture's tables: the downloaded files (`MediaAsset`), each user's jobs over them and the folders
they sit in, the shares and public links that hand them on, a user's quota, cookies and format
profiles, and where they stopped playing one. Only this unit writes them; another unit reads them
along a declared edge, or asks this unit's service."""
from datetime import date, datetime, timezone
from sqlalchemy import (
BigInteger,
Boolean,
CheckConstraint,
Date,
DateTime,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.orm import Mapped, mapped_column, validates
from app.db import Base
from app.models import DISPLAY_TEXT_MAX, TimestampMixin, UpdatedAtMixin
class LibraryState(Base):
"""Where a user stopped in a library item (their own download, or one shared with them) — the
library player's resume point. Keyed by the JOB, not the file: a job is the user's library entry,
a re-download keeps it, and deleting the job deletes the row; the shared file under it may outlive
both. A revoked share leaves its recipient's row behind, unread — only a job the user can still
reach is ever read or written — and it is back if the item is shared again. Rows exist only for a
position worth keeping (see `routes._common.kept_position`); no row means "from the start"."""
__tablename__ = "library_states"
__table_args__ = (UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
job_id: Mapped[int] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
position_seconds: Mapped[int] = mapped_column(Integer, nullable=False)
progress_updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
class MediaAsset(Base, TimestampMixin):
"""A physical downloaded media file — the SHARED cache layer of the download center.
Mirrors the app's "shared catalog, private per-user state" model: one row per unique
``(source_kind, source_ref, format_sig)`` is the global cache. When a second user requests
the same video with a format that hashes to the same `format_sig`, their `download_job`
links to this ready asset instantly — no re-download, no cost. Only the SOURCE download is
shared; per-user edited derivatives (phase 2) are never cached here.
`rel_path` is relative to `settings.files_root` (the mount), so the physical filename is
system-decided and bound to the source id — never renamed (the user's custom name is display
metadata on `download_job`, applied only when downloading to their own machine). The naming
fields (`title`/`uploader`/`upload_date`) are denormalized so the Plex-tree path can be built
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here) frees
the file with its last holder, and `last_access_at` orders the GC's LRU eviction. How long a file
stays is not the asset's: each holder's job carries its own `expires_at` (its library folder's)."""
__tablename__ = "media_assets"
__table_args__ = (
UniqueConstraint(
"source_kind", "source_ref", "format_sig", name="uq_media_asset_cache_key"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512)) # video id, or full URL for ad-hoc
format_sig: Mapped[str] = mapped_column(String(64)) # hash of the output-affecting spec
status: Mapped[str] = mapped_column(
String(16), default="pending", server_default="pending", index=True
) # "pending" | "ready" | "error"
rel_path: Mapped[str | None] = mapped_column(Text) # relative to files_root
storyboard_path: Mapped[str | None] = mapped_column(Text) # phase-2 filmstrip mosaic
# A locally-generated poster frame (ffmpeg), set only when the source had NO thumbnail so the
# card / watch page / link preview can still show an image. Relative to files_root.
poster_path: Mapped[str | None] = mapped_column(Text)
size_bytes: Mapped[int | None] = mapped_column(BigInteger)
duration_s: Mapped[int | None] = mapped_column(Integer)
width: Mapped[int | None] = mapped_column(Integer)
height: Mapped[int | None] = mapped_column(Integer)
container: Mapped[str | None] = mapped_column(String(16))
vcodec: Mapped[str | None] = mapped_column(String(32))
acodec: Mapped[str | None] = mapped_column(String(32))
# "video" | "audio" | "file" — the media-kind discriminator that drives which library actions
# apply (play/edit for media; download-only for a generic uploaded file). Null while an asset is
# still pending (kind is known once the file is probed at finalize). It is the single stored home
# of the invariant `downloads.mediakind.derive` used to compute — never re-derived in prose (an
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Denormalized naming/metadata for the Plex tree + ad-hoc display.
title: Mapped[str | None] = mapped_column(Text)
uploader: Mapped[str | None] = mapped_column(String(255))
# yt-dlp's channel/uploader page URL (channel_url|uploader_url) — lets an auto-filled channel
# render as a clickable link. Null for sources without one (e.g. reddit) or older rows.
uploader_url: Mapped[str | None] = mapped_column(Text)
upload_date: Mapped[date | None] = mapped_column(Date)
thumbnail_url: Mapped[str | None] = mapped_column(Text)
# yt-dlp's canonical page URL for the source (webpage_url) — the clean "downloaded from"
# reference shown in the Files library. Null until the worker extracts it (older/queued rows
# fall back to a URL derived from source_kind+source_ref).
source_webpage_url: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
ref_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
last_access_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The file's SHA-256 (hex), so a machine fetching it can check what it got. Hashed while an upload
# streams in, otherwise on the first ask (`routes.files`); NULL = not hashed yet.
sha256: Mapped[str | None] = mapped_column(String(64))
@validates("rel_path")
def _forget_hash(self, _key: str, rel_path: str | None) -> str | None:
# A hash names the bytes at `rel_path`: a new file there (a conversion, a retry) is not them.
self.sha256 = None
return rel_path
class DownloadProfile(Base, TimestampMixin):
"""A named download preset — a reusable format profile.
`user_id` NULL + `is_builtin` marks the shipped presets (Best MP4, 1080p, audio-only, …);
a user's saved custom profiles carry their `user_id`. `spec` is the format definition
(mode av/v/a, max_height, container, audio_format, codec prefs, embed subs/chapters/
thumbnail, sponsorblock) that `downloads.formats` turns into a yt-dlp selector + a cache
`format_sig`."""
__tablename__ = "download_profiles"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
) # NULL for a builtin preset
name: Mapped[str] = mapped_column(String(80))
spec: Mapped[dict] = mapped_column(JSON)
is_builtin: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
sort_order: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class FileFolder(Base, TimestampMixin):
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
__tablename__ = "file_folders"
__table_args__ = (
UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
Index(
"uq_file_folder_own_name",
"user_id",
func.lower(text("name")),
unique=True,
postgresql_where=text("name IS NOT NULL"),
),
# One Hub folder per user.
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
kind: Mapped[str] = mapped_column(String(16)) # folders.FolderKind
# "site:" + a host name (DNS caps one at 253 characters).
source_key: Mapped[str | None] = mapped_column(String(260))
name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
retention_days: Mapped[int | None] = mapped_column(Integer)
def _ready_on_insert(context) -> datetime | None:
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
done = context.get_current_parameters().get("status") == "done"
return datetime.now(timezone.utc) if done else None
class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
"""A user's request to download one source — the per-user layer over `MediaAsset`.
The worker claims `queued` jobs with ``FOR UPDATE SKIP LOCKED`` and writes live
`progress`/`speed_bps`/`eta_s`/`phase` here (the frontend polls via useLiveQuery: there is
no server push channel, and the worker is a separate process anyway). `profile_snapshot`
freezes the spec at enqueue so later edits/deletion of the profile don't change this job.
`display_name` is the user's own name for the file (display + the Content-Disposition on
local download); the on-disk asset is never renamed. `asset_id` links to the shared file
(SET NULL if the asset is GC'd — the job then shows as expired).
Phase 2 (editor): `job_kind='edit'` jobs derive a new per-user clip from an already-downloaded
ready job. They still hang off a `MediaAsset` (with `source_kind='edit'`, a per-user cache key)
so all the file-serve/ref-count/GC/quota machinery is reused; the worker branches on the
asset's `source_kind` and runs ffmpeg instead of yt-dlp. `source_job_id`/`source_asset_id`
point at the parent download the clip was cut from; `edit_spec` is the (trim/crop) recipe."""
__tablename__ = "download_jobs"
__table_args__ = (
Index("ix_download_jobs_claim", "status", "queue_pos"),
# Every way a job becomes done stamps when (`folders.stamp_done`, `finalize_if_running`).
CheckConstraint(
"status <> 'done' OR ready_at IS NOT NULL", name="ck_download_jobs_ready_at"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL"), index=True
)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512))
# "download" (default), "edit" (a per-user trim/crop derivative of another job), "upload" (a
# user's own file, converted by the worker when it is not phone-safe — see `downloads.upload`) or
# "clip" (a range cut straight from a YouTube video or a Plex item — see `downloads.clip`).
job_kind: Mapped[str] = mapped_column(String(16), default="download", server_default="download")
# For edit jobs: the parent download this clip is cut from (SET NULL if the parent is deleted).
source_job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="SET NULL")
)
source_asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL")
)
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
edit_spec: Mapped[dict | None] = mapped_column(JSON)
profile_id: Mapped[int | None] = mapped_column(
ForeignKey("download_profiles.id", ondelete="SET NULL")
)
profile_snapshot: Mapped[dict] = mapped_column(JSON)
display_name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
# Per-user display overrides + extras (like display_name — metadata only, never the file).
# `display_uploader`/`display_uploader_url` override the asset's auto channel name/link;
# `extra_links` is an optional list of extra reference URLs shown on the card + watch page.
display_uploader: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
display_uploader_url: Mapped[str | None] = mapped_column(Text)
extra_links: Mapped[list | None] = mapped_column(JSON)
status: Mapped[str] = mapped_column(
String(16), default="queued", server_default="queued", index=True
) # queued | running | paused | done | error | canceled
progress: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
speed_bps: Mapped[int | None] = mapped_column(BigInteger)
eta_s: Mapped[int | None] = mapped_column(Integer)
phase: Mapped[str | None] = mapped_column(String(32))
error: Mapped[str | None] = mapped_column(Text) # raw yt-dlp/ffmpeg text (technical detail)
error_code: Mapped[str | None] = mapped_column(
String(24)
) # category, see downloads.errors.classify
# The host-specific failure-catalog entry on top of the category (`facebook.login_wall`), or NULL.
error_detail: Mapped[str | None] = mapped_column(String(48)) # see downloads.errors.diagnose
queue_pos: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# The Files library: the folder this file sits in (`downloads.folders.place` at creation; an own
# folder is emptied back into the source folders before it is deleted, hence RESTRICT), whether
# the user keeps it past its folder's retention, and when it goes (stamped once done; NULL =
# never). `expiry_warned` marks the one warning the GC sends before it deletes the file.
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="RESTRICT"), index=True
)
# The origin the file came from (the source folder's key), recorded where it is placed: an edit
# traces its origin through its parent, and that parent may be deleted or expire later — the
# file must still know which source folder it may go back to. NULL only for a file placed before
# the column existed outside a source folder; `folders.source_key_for` works those out.
source_key: Mapped[str | None] = mapped_column(String(260))
kept: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# When the file became done — what the library's "Added" means: a download that failed for days
# and then went through is new to its user, so it sorts as new. `created_at` stays when it was
# asked for (the queue's order). Set again when a file is made anew (a retry, a re-download).
ready_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), default=_ready_on_insert
)
class DownloadShare(Base, TimestampMixin):
"""A user→user grant: `to_user` may access `from_user`'s downloaded file (private by
default). Since assets are already deduped/shared on disk, a share is just an ACL row."""
__tablename__ = "download_shares"
__table_args__ = (UniqueConstraint("job_id", "to_user_id", name="uq_download_share"),)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int] = mapped_column(ForeignKey("download_jobs.id", ondelete="CASCADE"))
from_user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
)
to_user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
class DownloadQuota(Base, UpdatedAtMixin):
"""Per-user download limits (admin-managed). A row exists only when an admin customizes a
user; otherwise the sysconfig `download_default_*` values apply. `unlimited` bypasses the
byte cap entirely (e.g. for the admin's own account)."""
__tablename__ = "download_quotas"
user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), primary_key=True
)
max_bytes: Mapped[int] = mapped_column(BigInteger)
max_concurrent: Mapped[int] = mapped_column(Integer)
max_jobs: Mapped[int] = mapped_column(Integer)
unlimited: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class DownloadLink(Base, TimestampMixin):
"""A public capability link to one download, or to one of its owner's own folders — the "share
by link" surface.
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
the folder at the time it is opened. The token itself is never stored as written: a link is
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
grants a *registered* user access via the in-app "Shared with me" list."""
__tablename__ = "download_links"
__table_args__ = (
# A link opens exactly one thing: a file or a folder.
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
password_hash: Mapped[str | None] = mapped_column(Text)
# Bumped whenever the password changes; folded into the signed watch grant so rotating the
# password immediately invalidates any grants issued under the old one (instead of them living
# out the 6h grant TTL).
password_version: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
view_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class DownloadCredential(Base, TimestampMixin, UpdatedAtMixin):
"""A user's stored login cookies for ONE auth-walled download host (Facebook, Instagram, …).
Per-user, per-host: each user pastes their OWN browser cookies.txt (Netscape format) for a host
where they're logged in (e.g. a closed Facebook group they belong to), and a download of a
yt-dlp-supported, auth-walled link on that host is retried with the ACTING user's cookie. The
blob is Fernet-encrypted at rest (`cookies_enc`) and is NEVER returned by any API/log/audit —
the routes are write-only (host + updated_at only). See `downloads.credentials`.
Two load-bearing security controls live around this row, NOT on it:
* **cache privacy** — a cookie-authed download SALTS its MediaAsset `format_sig` with the
user id, so the authed asset is private to that user and can never be cache-served to
another user (`downloads.credentials.cookie_salt`).
* **host scope (SEC-12)** — a cookie is attached ONLY when the download URL's netloc matches
this row's `host_key` (`downloads.credentials.host_key_for_url`), so an arbitrary `?url=`
can't exfiltrate the user's session to a look-alike host.
`host_key` is a short canonical key from the supported-host registry (e.g. "facebook"), not the
raw netloc — one credential can cover several domains of the same site (facebook.com/fb.watch)."""
__tablename__ = "download_credentials"
__table_args__ = (UniqueConstraint("user_id", "host_key", name="uq_download_credential"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
host_key: Mapped[str] = mapped_column(String(64))
cookies_enc: Mapped[str] = mapped_column(Text) # Fernet ciphertext — never returned/logged
+1 -1
View File
@@ -16,7 +16,7 @@ from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import links as linksmod
from app.downloads import storage
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
+2 -1
View File
@@ -28,7 +28,8 @@ from app.downloads.jobdb import (
)
from app.downloads.stated import StatedFailure
from app.downloads.status import JobStatus
from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset, PlexItem
from app.models import DISPLAY_TEXT_MAX, PlexItem
from app.downloads.models import DownloadJob, MediaAsset
from app.plex import lookups as plex_lookups
from app.plex import paths as plex_paths
from app.notifications import create_notification
+406
View File
@@ -0,0 +1,406 @@
"""Capture's reads: what the library, the download tray, the share dialogs and the public watch pages
show. Each returns rows or plain values and writes nothing; whose a row is goes through
`userscope.owned_by` / `owned_row`, so a row of someone else's reads as a missing one.
The HTTP layer (`routes.downloads`, `routes.files`, `routes.public`) asks here and serializes; the
writes are the service's (`service`, `folders`, `links`, `quota`)."""
from collections.abc import Sequence
from datetime import datetime, timezone
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app.apitokens import FilesCaller, Right
from app.collation import collated
from app.downloads import folders, links, quota
from app.downloads.models import (
DownloadJob,
DownloadLink,
DownloadProfile,
DownloadQuota,
DownloadShare,
FileFolder,
LibraryState,
MediaAsset,
)
from app.downloads.status import FINISHED, JobStatus
from app.models import User
from app.userscope import owned_by, owned_row
from app.utils import like_escape
# Cap a listing so a heavy library can't return an unbounded result set (the old Download Center's
# own cap; far beyond any real library).
MAX_LIST = 500
# --- jobs and their files ------------------------------------------------------------------
def own_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
return owned_row(db, DownloadJob, job_id, user)
def job(db: Session, job_id: int) -> DownloadJob | None:
"""A job by id, whoever's — for a public link, which already names its job."""
return db.get(DownloadJob, job_id)
def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""A file the caller may read: their own, or one shared with them."""
row = db.get(DownloadJob, job_id)
if row is None:
return None
if row.user_id == user.id or shared_with(db, job_id, user) is not None:
return row
return None
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob | None:
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
own in a folder it may read."""
if caller.grants is None:
return accessible_job(db, caller.user, job_id)
row = own_job(db, caller.user, job_id)
if row is None or not caller.may(row.folder_id, Right.READ):
return None
return row
def asset_of(db: Session, job: DownloadJob | None) -> MediaAsset | None:
"""The file under a job, if it has one yet."""
return db.get(MediaAsset, job.asset_id) if job is not None and job.asset_id else None
def assets_for(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, MediaAsset]:
ids = {j.asset_id for j in jobs if j.asset_id}
if not ids:
return {}
rows = db.execute(select(MediaAsset).where(MediaAsset.id.in_(ids))).scalars()
return {a.id: a for a in rows}
def positions_for(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, int]:
"""`user`'s resume point in each of `jobs` that has one (the library player's `library_states`)."""
ids = [j.id for j in jobs]
if not ids:
return {}
rows = db.execute(
select(LibraryState.job_id, LibraryState.position_seconds).where(
owned_by(LibraryState.user_id, user), LibraryState.job_id.in_(ids)
)
).all()
return {job_id: pos for job_id, pos in rows}
def source_keys(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, str]:
"""Where each file came from (its source folder's key), for the list's Source column, the
"back to its source folder" move and the download tray's site: a file in a source folder has its
folder's key; one moved into an own folder (or someone else's, shared) is traced back by
`folders.source_key_for`."""
ids = {j.folder_id for j in jobs if j.folder_id is not None}
by_id = (
{f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
if ids
else {}
)
out: dict[int, str] = {}
for j in jobs:
folder = by_id.get(j.folder_id) if j.folder_id is not None else None
if folder is not None and folder.kind == folders.FolderKind.SOURCE and folder.source_key:
out[j.id] = folder.source_key
else:
out[j.id] = folders.source_key_for(db, j)
return out
def queue(db: Session, user: User, recent: datetime) -> list[DownloadJob]:
"""The download tray: what is waiting, running, paused or failed, and what finished (done or
canceled) since `recent`. A done job is timed from when it became done, never from its last
write: a rename, a move or the GC's expiry warning touches an old file's row, and the tray would
announce it as just finished."""
return list(
db.execute(
select(DownloadJob)
.where(
owned_by(DownloadJob.user_id, user),
DownloadJob.status.not_in(FINISHED)
| ((DownloadJob.status == JobStatus.DONE) & (DownloadJob.ready_at >= recent))
| ((DownloadJob.status == JobStatus.CANCELED) & (DownloadJob.updated_at >= recent)),
)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
).scalars()
)
def statuses_by_source(db: Session, user: User, statuses: Sequence[str]) -> list[tuple[str, str]]:
"""(source_ref, status) of each of `user`'s jobs in one of `statuses`."""
return list(
db.execute(
select(DownloadJob.source_ref, DownloadJob.status).where(
owned_by(DownloadJob.user_id, user), DownloadJob.status.in_(tuple(statuses))
)
).tuples()
)
def library(
db: Session,
user: User,
*,
folder_id: int | None,
listable: set[int] | None,
shared: bool,
term: str,
sort: str,
order: str,
lang: str,
) -> list[DownloadJob]:
"""The caller's done files (optionally one folder's, or only in the folders a token may list),
or the ones shared with them; searched by name and channel, sorted by when they were added, by
name (in the reader's alphabet) or by size. A file is added to its owner's list when it is done,
not when it was asked for (a download that failed for days and then went through is new); one
shared with the caller, when it was shared."""
stmt = (
select(DownloadJob)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.status == JobStatus.DONE)
)
if shared:
stmt = stmt.join(DownloadShare, DownloadShare.job_id == DownloadJob.id).where(
owned_by(DownloadShare.to_user_id, user)
)
else:
stmt = stmt.where(owned_by(DownloadJob.user_id, user))
if folder_id is not None:
stmt = stmt.where(DownloadJob.folder_id == folder_id)
elif listable is not None:
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
if term:
like = f"%{like_escape(term)}%"
stmt = stmt.where(
or_(
name.ilike(like),
func.coalesce(DownloadJob.display_uploader, MediaAsset.uploader, "").ilike(like),
)
)
key = {
"added": DownloadShare.created_at if shared else DownloadJob.ready_at,
"name": collated(name, lang),
"size": MediaAsset.size_bytes,
}[sort]
first = key.asc() if order == "asc" else key.desc()
tie = DownloadJob.id.asc() if order == "asc" else DownloadJob.id.desc()
return list(db.execute(stmt.order_by(first.nulls_last(), tie).limit(MAX_LIST)).scalars())
def recent_jobs(db: Session) -> list[DownloadJob]:
"""Everyone's newest jobs, for the admin's list."""
return list(
db.execute(
select(DownloadJob)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
).scalars()
)
# --- shares --------------------------------------------------------------------------------
def shared_with(db: Session, job_id: int, user: User | int) -> DownloadShare | None:
"""The grant that hands this job to `user`, if there is one."""
return db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, owned_by(DownloadShare.to_user_id, user)
)
).scalar_one_or_none()
def shared_at(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, datetime]:
rows = db.execute(
select(DownloadShare.job_id, DownloadShare.created_at).where(
owned_by(DownloadShare.to_user_id, user),
DownloadShare.job_id.in_([j.id for j in jobs]),
)
).all()
return {job_id: at for job_id, at in rows}
# --- folders -------------------------------------------------------------------------------
def own_folder(db: Session, user: User, folder_id: int) -> FileFolder | None:
return owned_row(db, FileFolder, folder_id, user)
def folders_by_id(db: Session, ids: set[int]) -> dict[int, FileFolder]:
"""These folders, by id — for a reader that already knows which ones (a token's grants)."""
if not ids:
return {}
return {f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
def folders_with_totals(
db: Session, user: User, listable: set[int] | None
) -> list[tuple[FileFolder, int, int]]:
"""The user's folders (only `listable` ones, when given) with how many done files each holds and
their size. A source folder its files have all left stays (count 0)."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
stmt = select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
).where(owned_by(FileFolder.user_id, user))
if listable is not None:
stmt = stmt.where(FileFolder.id.in_(listable))
rows = db.execute(
stmt.outerjoin(DownloadJob, done)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.group_by(FileFolder.id)
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
).tuples()
return [(f, count, int(size or 0)) for f, count, size in rows]
def folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
count, size = db.execute(
select(func.count(DownloadJob.id), func.coalesce(func.sum(MediaAsset.size_bytes), 0))
.select_from(DownloadJob)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.folder_id == folder_id, DownloadJob.status == JobStatus.DONE)
).one()
return count, int(size)
def linked_folders(db: Session, folder_ids: list[int]) -> set[int]:
"""Which of these folders a live public link opens (one not past its expiry)."""
now = datetime.now(timezone.utc)
rows = db.execute(
select(DownloadLink.folder_id).where(
DownloadLink.folder_id.in_(folder_ids),
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
)
).scalars()
return {folder_id for folder_id in rows if folder_id is not None}
# --- links ---------------------------------------------------------------------------------
def own_link(db: Session, user: User, link_id: int) -> DownloadLink | None:
"""A link to one of the user's files or folders."""
link = db.get(DownloadLink, link_id)
if link is None:
return None
target = (
db.get(DownloadJob, link.job_id)
if link.job_id is not None
else db.get(FileFolder, link.folder_id)
)
if target is None or target.user_id != user.id:
return None
return link
def links_to(
db: Session, *, job_id: int | None = None, folder_id: int | None = None
) -> list[DownloadLink]:
"""The links to one file or one folder, newest first."""
target = (
DownloadLink.job_id == job_id if job_id is not None else DownloadLink.folder_id == folder_id
)
return list(
db.execute(select(DownloadLink).where(target).order_by(DownloadLink.id.desc())).scalars()
)
def link_folder(db: Session, link: DownloadLink) -> FileFolder | None:
return db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
def folder_job(db: Session, folder: FileFolder, job_id: int) -> DownloadJob | None:
"""One of the files a link to `folder` reaches (`links.in_folder`), while it is in it."""
return db.execute(
select(DownloadJob).where(DownloadJob.id == job_id, *links.in_folder(folder))
).scalar_one_or_none()
def owner_suspended(db: Session, job: DownloadJob) -> bool:
"""The job's owner is suspended (or gone): the worker leaves their queued jobs where they are."""
owner = db.get(User, job.user_id)
return owner is None or owner.is_suspended
# --- profiles ------------------------------------------------------------------------------
def profiles_for(db: Session, user: User) -> list[DownloadProfile]:
"""The built-in format profiles, then the user's own."""
return list(
db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True) | owned_by(DownloadProfile.user_id, user))
.order_by(
DownloadProfile.is_builtin.desc(), DownloadProfile.sort_order, DownloadProfile.id
)
).scalars()
)
def own_profile(db: Session, user: User, profile_id: int) -> DownloadProfile | None:
"""One of the user's own profiles (a built-in is no one's to change)."""
p = owned_row(db, DownloadProfile, profile_id, user)
return None if p is None or p.is_builtin else p
def usable_profile(db: Session, user: User, profile_id: int) -> DownloadProfile | None:
"""A profile the user may enqueue with: a built-in, or their own."""
p = db.get(DownloadProfile, profile_id)
if p is None or (not p.is_builtin and p.user_id != user.id):
return None
return p
def default_profile(db: Session) -> DownloadProfile | None:
"""The first built-in, for an enqueue that named no format."""
return db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True))
.order_by(DownloadProfile.sort_order)
.limit(1)
).scalar_one_or_none()
# --- storage and quota (admin) -------------------------------------------------------------
def storage_totals(db: Session) -> tuple[int, int, int]:
"""(files on disk, their bytes, every asset row). On disk is `quota.ON_DISK`: ready files, and
uploads waiting for or failed in their conversion — the same disk the footprint and the GC cap
count."""
on_disk = db.execute(
select(func.count(), func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).one()
total_assets = db.execute(select(func.count()).select_from(MediaAsset)).scalar_one()
return on_disk[0], int(on_disk[1]), total_assets
def done_files_per_user(db: Session) -> dict[int, int]:
"""How many finished files each user's Files lists."""
return dict(
db.execute(
select(DownloadJob.user_id, func.count())
.where(DownloadJob.status == JobStatus.DONE)
.group_by(DownloadJob.user_id)
)
.tuples()
.all()
)
def custom_quota(db: Session, user_id: int) -> DownloadQuota | None:
"""The admin's own limits for this user, if any were set."""
return db.get(DownloadQuota, user_id)
+47 -3
View File
@@ -26,7 +26,8 @@ from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobKind, JobStatus
from app.models import DownloadJob, DownloadQuota, MediaAsset
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
from app.userscope import owned_by
# Jobs a user is considered to be "holding" (occupy a job slot / footprint). Terminal failures
# (error, canceled) don't count.
@@ -85,7 +86,7 @@ def footprint(db: Session, user_id: int) -> int:
held_assets = (
select(DownloadJob.asset_id)
.where(
DownloadJob.user_id == user_id,
owned_by(DownloadJob.user_id, user_id),
or_(
DownloadJob.status.in_(_HOLDING),
and_(DownloadJob.job_kind == JobKind.UPLOAD, DownloadJob.status == JobStatus.ERROR),
@@ -109,7 +110,7 @@ def _count(db: Session, user_id: int, states) -> int:
db.execute(
select(func.count())
.select_from(DownloadJob)
.where(DownloadJob.user_id == user_id, DownloadJob.status.in_(states))
.where(owned_by(DownloadJob.user_id, user_id), DownloadJob.status.in_(states))
).scalar()
or 0
)
@@ -201,3 +202,46 @@ def usage(db: Session, user_id: int) -> dict:
"max_jobs": lim.max_jobs,
"unlimited": lim.unlimited,
}
def set_custom(
db: Session,
user_id: int,
*,
max_bytes: int | None = None,
max_concurrent: int | None = None,
max_jobs: int | None = None,
unlimited: bool | None = None,
) -> None:
"""The admin's own limits for a user: starts from what applies now, then each value given (a
size floored at 0, a count at 1). The handler commits."""
row = db.get(DownloadQuota, user_id)
if row is None:
cur = resolve(db, user_id)
row = DownloadQuota(
user_id=user_id,
max_bytes=cur.max_bytes,
max_concurrent=cur.max_concurrent,
max_jobs=cur.max_jobs,
unlimited=cur.unlimited,
)
db.add(row)
if max_bytes is not None:
row.max_bytes = max(0, max_bytes)
if max_concurrent is not None:
row.max_concurrent = max(1, max_concurrent)
if max_jobs is not None:
row.max_jobs = max(1, max_jobs)
if unlimited is not None:
row.unlimited = unlimited
db.flush()
def reset_custom(db: Session, user_id: int) -> bool:
"""Back to the defaults; False when the user had no limits of their own."""
row = db.get(DownloadQuota, user_id)
if row is None:
return False
db.delete(row)
db.flush()
return True
+1 -1
View File
@@ -23,7 +23,7 @@ from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import formats
from app.downloads.status import JobKind
from app.models import DownloadJob
from app.downloads.models import DownloadJob
log = logs.get_logger(__name__)
+199 -22
View File
@@ -1,16 +1,19 @@
"""Enqueue + cache resolution for the download center (the per-user layer over MediaAsset).
"""Capture's writes: enqueue + cache resolution, the job transitions, and what the library's
dialogs change (profiles, resume points, shares, a file's display fields).
`enqueue` is the single entry point used by the API: it resolves (or creates) the shared
MediaAsset for the requested source+format, links a per-user DownloadJob to it, and — on a
cache hit (asset already ready) — marks the job done instantly with no re-download. The worker
handles everything else. State transitions used by the routes (pause/resume/cancel/delete) and
the worker live in `worker.py` / the routes; this module owns enqueue + asset bookkeeping.
handles everything else. Every function here flushes and leaves the commit to its caller (the
handler); the folder, link and quota writes are `folders`, `links` and `quota`.
"""
import uuid
from datetime import datetime, timezone
from sqlalchemy import delete as delete_rows
from sqlalchemy import func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -21,17 +24,17 @@ from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import folders, formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobKind, JobStatus
from app.models import (
DISPLAY_TEXT_MAX,
Channel,
from app.models import DISPLAY_TEXT_MAX, Channel, PlexItem, PlexShow, User, Video
from app.downloads.models import (
DownloadJob,
DownloadProfile,
DownloadShare,
LibraryState,
MediaAsset,
PlexItem,
PlexShow,
Video,
)
from app.plex import lookups as plex_lookups
from app.plex import paths as plex_paths
from app.userscope import owned_by, owned_row
class EditError(Exception):
@@ -230,8 +233,7 @@ def enqueue(
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
@@ -310,8 +312,7 @@ def enqueue_edit(
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
@@ -399,8 +400,7 @@ def enqueue_clip(
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
@@ -508,8 +508,8 @@ def release_asset(db: Session, job: DownloadJob) -> None:
# --- Job state transitions ---------------------------------------------------------------------
# The single home for the pause/resume/cancel/delete/redownload transitions + their asset
# bookkeeping (release, failed-asset reset, fresh-asset rebuild), so the routes are thin callers and
# the worker/GC could share the same definitions (B-sub-8). Each commits its own transition; the
# route keeps only auth + HTTP mapping + serialization.
# the worker/GC could share the same definitions (B-sub-8). Each flushes its transition and the caller
# commits it; the route keeps only auth + HTTP mapping + serialization.
def pause(db: Session, job: DownloadJob) -> None:
@@ -517,7 +517,7 @@ def pause(db: Session, job: DownloadJob) -> None:
in any other state."""
if job.status in (JobStatus.QUEUED, JobStatus.RUNNING):
job.status = JobStatus.PAUSED
db.commit()
db.flush()
def resume(db: Session, job: DownloadJob) -> None:
@@ -541,7 +541,7 @@ def resume(db: Session, job: DownloadJob) -> None:
asset.status = "pending"
asset.error = None
job.status = JobStatus.QUEUED
db.commit()
db.flush()
def _follow_the_cookie(db: Session, job: DownloadJob, asset: MediaAsset) -> MediaAsset:
@@ -576,14 +576,14 @@ def cancel(db: Session, job: DownloadJob) -> None:
if job.status not in (JobStatus.DONE, JobStatus.CANCELED):
release_asset(db, job) # release while the job still counts as holding
job.status = JobStatus.CANCELED
db.commit()
db.flush()
def delete(db: Session, job: DownloadJob) -> None:
"""Delete a job, releasing its asset hold first (which frees the file iff this was the last holder)."""
release_asset(db, job)
db.delete(job)
db.commit()
db.flush()
def redownload(db: Session, job: DownloadJob) -> None:
@@ -632,4 +632,181 @@ def redownload(db: Session, job: DownloadJob) -> None:
job.queue_pos = next_queue_pos(db) # tail of the queue, not its stale original slot
job.expires_at = None # stamped again when the rebuild is done
job.expiry_warned = False
db.commit()
db.flush()
# --- the library's writes ------------------------------------------------------------------
# What the HTTP layer changes for a user: each flushes, the handler commits.
def lock_own_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""One of the user's jobs, re-read under a row lock, for a transition that moves its asset's ref
count (resume, cancel, delete, redownload) or writes the status those decide on (pause). Two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds. None: not
theirs, or gone.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`finalize_if_running`), so the two can never deadlock."""
while True:
owned = owned_row(db, DownloadJob, job_id, user)
if owned is None:
return None
asset_id = owned.asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
return None
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
def hold_for_share(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
rather than failing the link's foreign key — or, done first, leaves None."""
if owned_row(db, DownloadJob, job_id, user) is None:
return None
return db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update(read=True)
.execution_options(populate_existing=True)
).scalar_one_or_none()
def create_profile(db: Session, user: User, name: str, spec: dict) -> DownloadProfile:
"""A format profile of the user's own, after their others."""
maxpos = db.execute(
select(func.coalesce(func.max(DownloadProfile.sort_order), 100)).where(
owned_by(DownloadProfile.user_id, user)
)
).scalar_one()
p = DownloadProfile(
user_id=user.id, name=name[:80], spec=spec, is_builtin=False, sort_order=maxpos + 1
)
db.add(p)
db.flush()
return p
def update_profile(profile: DownloadProfile, *, name: str | None, spec: dict | None) -> None:
"""Rename a profile and/or replace its spec; None leaves that one as it is."""
if name is not None:
profile.name = name[:80]
if spec is not None:
profile.spec = spec
def delete_profile(db: Session, profile: DownloadProfile) -> None:
db.delete(profile)
db.flush()
def set_position(db: Session, user: User, job_id: int, position: int) -> None:
"""The library player's resume point in a file the user can play; 0 clears it.
A clear is a statement, not a loaded row's delete: the periodic checkpoint and the pagehide beacon
of one stop may both clear it, and the second then deletes nothing rather than failing. A set is
an upsert, one row per (user, job) even when two checkpoints race, not a read-then-insert that
the second would fail on the unique key."""
if not position:
db.execute(
delete_rows(LibraryState).where(
owned_by(LibraryState.user_id, user), LibraryState.job_id == job_id
)
)
return
now = datetime.now(timezone.utc)
db.execute(
pg_insert(LibraryState)
.values(user_id=user.id, job_id=job_id, position_seconds=position, progress_updated_at=now)
.on_conflict_do_update(
constraint="uq_library_user_job",
set_={"position_seconds": position, "progress_updated_at": now},
)
)
def share(db: Session, job: DownloadJob, sender: User, recipient: User) -> bool:
"""Hand a file to another user, with a notice for them; False when it was already theirs."""
from app.notifications import create_notification
exists = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job.id, owned_by(DownloadShare.to_user_id, recipient)
)
).first()
if exists is not None:
return False
db.add(DownloadShare(job_id=job.id, from_user_id=sender.id, to_user_id=recipient.id))
create_notification(
db,
user_id=recipient.id,
type="download_shared",
title=(job.display_name or "A download") + " was shared with you",
# The client names the notice in the reader's language from `data`; the title is the
# English fallback only.
data={
"kind": "download_shared",
"from": sender.email,
"job_id": job.id,
"title": job.display_name,
},
commit=False,
)
db.flush()
return True
def unshare(db: Session, job_id: int, recipient: User) -> bool:
"""Take a file back from one user (its owner's act, or the recipient's own dismissal); False
when it was not theirs."""
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, owned_by(DownloadShare.to_user_id, recipient)
)
).scalar_one_or_none()
if row is None:
return False
db.delete(row)
db.flush()
return True
# The display fields a user may set on their file; the file on disk keeps its own name.
DISPLAY_FIELDS = frozenset(
{"display_name", "display_uploader", "display_uploader_url", "extra_links"}
)
def set_display(job: DownloadJob, **fields: object) -> None:
"""Set a file's display fields (each already cleaned by the caller); a field not given is left."""
unknown = set(fields) - DISPLAY_FIELDS
if unknown:
raise ValueError(f"not a display field: {sorted(unknown)}")
for name, value in fields.items():
setattr(job, name, value)
def touch(asset: MediaAsset) -> None:
"""The file was just served: the cache's idle clock starts again."""
asset.last_access_at = datetime.now(timezone.utc)
def store_sha256(db: Session, asset_id: int, digest: str) -> None:
"""Keep a file's hash once it is known — only where none is, so a concurrent hash of the same
file never overwrites the first."""
db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id, MediaAsset.sha256.is_(None))
.values(sha256=digest)
)
+2 -1
View File
@@ -37,7 +37,8 @@ from app.config import settings
from app.downloads import edit as editmod
from app.downloads import folders, mediakind, playable, probe, quota, service, storage
from app.downloads.status import JobKind, JobStatus
from app.models import DownloadJob, MediaAsset, display_text
from app.models import display_text
from app.downloads.models import DownloadJob, MediaAsset
def _title_from_filename(filename: str) -> str:
+131
View File
@@ -0,0 +1,131 @@
"""The durable history of background runs — the `job_runs` table.
A run is written twice: a `running` row when it starts, its outcome when it ends. Each write is its
OWN short session, committed at once, for the reason `quota.record_usage` gives: the run's own work
may roll back (its `except: db.rollback()`), and the history of what happened must not go with it —
nor may this bookkeeping commit the job's half-done session.
Only the API process runs jobs (the worker never starts the scheduler), and one API process serves a
database (`.env.example`, the Scheduler section: the single-run guard is in its memory too), so at its
startup nothing can still be running: `interrupt_leftovers` closes what a stopped process left open.
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete, func, select, union_all, update
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import JobRun, QuotaEvent
from app.runstatus import RunStatus, RunTrigger
def start(job: str, trigger: RunTrigger, actor_id: int | None, target: str | None = None) -> int:
"""Open a run's row; returns its id."""
with SessionLocal() as s:
row = JobRun(
job=job,
trigger=trigger,
actor_id=actor_id,
target=target,
status=RunStatus.RUNNING,
started_at=datetime.now(timezone.utc),
)
s.add(row)
s.commit()
return row.id
def finish(
run_id: int,
status: RunStatus,
*,
summary: str | None = None,
result: dict | None = None,
error: str | None = None,
failure: dict | None = None,
) -> None:
"""Close a run with its outcome and what it spent. Conditional on the row still running: a row
the startup sweep already called interrupted is not rewritten by a thread that outlived it."""
with SessionLocal() as s:
units = s.scalar(
select(func.coalesce(func.sum(QuotaEvent.units), 0)).where(
QuotaEvent.job_run_id == run_id
)
)
s.execute(
update(JobRun)
.where(JobRun.id == run_id, JobRun.status == RunStatus.RUNNING)
.values(
status=status,
finished_at=datetime.now(timezone.utc),
summary=summary,
result=result,
error=error,
failure=failure,
quota_units=units or 0,
)
)
s.commit()
def interrupt_leftovers() -> int:
"""Every row a previous process left `running` → `interrupted`, with what it spent until then
(its quota rows outlive it, as `finish` sums them). Called once at startup, before the scheduler
can open a new run. Returns how many."""
spent = (
select(func.coalesce(func.sum(QuotaEvent.units), 0))
.where(QuotaEvent.job_run_id == JobRun.id)
.scalar_subquery()
)
with SessionLocal() as s:
res = s.execute(
update(JobRun)
.where(JobRun.status == RunStatus.RUNNING)
.values(
status=RunStatus.INTERRUPTED,
finished_at=datetime.now(timezone.utc),
quota_units=spent,
)
)
s.commit()
return res.rowcount if isinstance(res, CursorResult) else 0
def recent_runs(db: Session, jobs: list[str], per_job: int) -> dict[str, list[JobRun]]:
"""Each of `jobs`' last `per_job` runs, newest first, by job id — one query, each job's part a
short walk of `ix_job_runs_job_started`, so a poll costs the same at any history length (a window
over the whole table grew with every run kept)."""
if not jobs:
return {}
newest = union_all(
*(
select(JobRun.id)
.where(JobRun.job == job)
.order_by(JobRun.started_at.desc(), JobRun.id.desc())
.limit(per_job)
for job in jobs
)
).subquery()
rows = db.execute(
select(JobRun)
.where(JobRun.id.in_(select(newest.c.id)))
.order_by(JobRun.job, JobRun.started_at.desc(), JobRun.id.desc())
).scalars()
out: dict[str, list[JobRun]] = {}
for r in rows:
out.setdefault(r.job, []).append(r)
return out
def gc(db: Session, retention_days: int) -> int:
"""Delete finished runs older than `retention_days` (0 = keep forever). A run still going is
never reaped. The caller commits."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
res = db.execute(
delete(JobRun).where(JobRun.started_at < cutoff, JobRun.status != RunStatus.RUNNING)
)
return res.rowcount if isinstance(res, CursorResult) else 0
+15 -2
View File
@@ -10,8 +10,8 @@ the logs were both unjoinable and leaky. `configure_logging()` replaces all four
Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
ts · level · sev · source · logger · event · msg · attrs
plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
`job_id`) and the app version. That shape is what the admin log console filters on and what a Gitea
issue attachment will carry, so it is deliberately flat and stable.
`job_id`, `run_id`) and the app version. That shape is what the admin log console filters on and
what a Gitea issue attachment will carry, so it is deliberately flat and stable.
Call sites stay ordinary stdlib logging:
@@ -81,15 +81,28 @@ _debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
"log_debug_session", default=None
)
_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
# The background run (app/jobruns.py) this context is doing; quota's spend is tagged with it too.
_run_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_run_id", default=None)
_CONTEXT_VARS = {
"request_id": _request_id,
"user_id": _user_id,
"debug_session_id": _debug_session,
"job_id": _job_id,
"run_id": _run_id,
}
def current_request_id() -> str | None:
"""The HTTP request this context is serving, if any — the request middleware sets it."""
return _request_id.get()
def current_run_id() -> int | None:
"""The background run this context is doing, if any — `bind(run_id=…)` sets it."""
return _run_id.get()
class _RequestScope:
"""What the auth dependency learns about a request, for every line logged after it.
+1
View File
@@ -362,6 +362,7 @@ def serialize(row: LogEvent) -> dict:
"request_id": row.request_id,
"user_id": row.user_id,
"job_id": row.job_id,
"run_id": row.run_id,
"debug_session_id": row.debug_session_id,
"version": row.version,
}
+2
View File
@@ -52,6 +52,7 @@ from app.routes import ( # noqa: E402 — logging is configured first
plex as plex_routes,
public as public_routes,
quota,
runs as runs_routes,
saved_views,
scheduler as scheduler_routes,
search as search_routes,
@@ -345,6 +346,7 @@ app.include_router(config_routes.router)
app.include_router(scheduler_routes.router)
app.include_router(audit_routes.router)
app.include_router(admin_logs_routes.router)
app.include_router(runs_routes.router)
app.include_router(quota.router)
app.include_router(version.router)
app.include_router(setup_routes.router)
+56 -372
View File
@@ -1,4 +1,4 @@
from datetime import date, datetime, timezone
from datetime import date, datetime
from sqlalchemy import (
BigInteger,
@@ -18,12 +18,12 @@ from sqlalchemy import (
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from app.db import Base
from app.runstatus import RunStatus, RunTrigger, sql_in
from app.utils import canonical_email
# Every user-supplied display string in this schema shares one width, and it lives HERE so the
@@ -464,27 +464,6 @@ class VideoState(Base, UpdatedAtMixin):
progress_updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class LibraryState(Base):
"""Where a user stopped in a library item (their own download, or one shared with them) — the
library player's resume point. Keyed by the JOB, not the file: a job is the user's library entry,
a re-download keeps it, and deleting the job deletes the row; the shared file under it may outlive
both. A revoked share leaves its recipient's row behind, unread — only a job the user can still
reach is ever read or written — and it is back if the item is shared again. Rows exist only for a
position worth keeping (see `routes._common.kept_position`); no row means "from the start"."""
__tablename__ = "library_states"
__table_args__ = (UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
job_id: Mapped[int] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
position_seconds: Mapped[int] = mapped_column(Integer, nullable=False)
progress_updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
class SearchFind(Base, TimestampMixin):
"""Per-user record of a video the user surfaced via their own live YouTube search.
@@ -563,14 +542,20 @@ class QuotaEvent(Base):
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), index=True
)
# The background run that spent it (app/jobruns.py); NULL = a request's own spend. SET NULL: a
# reaped run leaves the spend in the budget's history.
job_run_id: Mapped[int | None] = mapped_column(
ForeignKey("job_runs.id", ondelete="SET NULL"), index=True
)
class AuditLog(Base):
"""Append-only admin/scheduler audit trail: who did what, when, and (where it applies)
the before->after values. `actor_id` is the acting admin; NULL means the scheduler /
background did it (SET NULL on user delete keeps the trail). Logged at ACTION / job-run
granularity — never per-row — so a scheduler run that touches thousands of videos is ONE
row summarising the outcome. `before`/`after` are small JSON snapshots (secrets excluded)."""
"""Append-only admin audit trail: who did what, when, and (where it applies) the before->after
values. `actor_id` is the acting admin; NULL means the background did it (SET NULL on user
delete keeps the trail). Logged per ACTION, never per row; a background run is not an action
and has its own history (`JobRun`). `before`/`after` are small JSON snapshots (secrets
excluded). `request_id` is the request that made the change, so its log lines are one click
away (NULL for rows written outside a request, and for rows older than the column)."""
__tablename__ = "audit_log"
@@ -587,6 +572,7 @@ class AuditLog(Base):
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), index=True
)
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
class LogEvent(Base):
@@ -609,6 +595,8 @@ class LogEvent(Base):
"debug_session_id",
postgresql_where="debug_session_id IS NOT NULL",
),
# A run's lines are `WHERE run_id = …`; request lines carry none.
Index("ix_log_events_run", "run_id", postgresql_where="run_id IS NOT NULL"),
)
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
@@ -624,7 +612,11 @@ class LogEvent(Base):
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
user_id: Mapped[int | None] = mapped_column(Integer)
debug_session_id: Mapped[str | None] = mapped_column(String(64))
# A DOWNLOAD job's id (download_jobs) — not a background run, which is `run_id`.
job_id: Mapped[int | None] = mapped_column(Integer)
# The background run (job_runs) that logged it. No foreign key, as `user_id`: the writer runs
# behind the event, and a run reaped meanwhile must not fail the batch its lines travel in.
run_id: Mapped[int | None] = mapped_column(BigInteger)
version: Mapped[str | None] = mapped_column(String(64))
@@ -689,15 +681,44 @@ class AppState(Base):
setup_token_hash: Mapped[str | None] = mapped_column(String(64))
class SchedulerSetting(Base):
"""Admin override for a scheduler job's run interval (minutes). One row per job id;
absent = use the env/config default. DB-backed so it's editable from the admin UI at
runtime without a redeploy (env stays a fallback)."""
class JobRun(Base):
"""One run of a background job — scheduled, an admin's "run now", or a user's ad-hoc work on one
target — written when it starts and again when it ends (app/jobruns.py), each in its own
committed session, so the history survives the job's own rollback and a restart. A row left
`running` by a process that died is set `interrupted` at the next startup.
__tablename__ = "scheduler_settings"
`quota_units` is what the run spent of the YouTube budget: the `quota_events` rows carrying its
id, summed when it ends. `summary` is the English one-line fallback; `result` / `failure` are the
same outcome as data, which the client names in the reader's language."""
job_id: Mapped[str] = mapped_column(String(40), primary_key=True)
interval_minutes: Mapped[int] = mapped_column(Integer)
__tablename__ = "job_runs"
__table_args__ = (
Index("ix_job_runs_job_started", "job", "started_at"),
CheckConstraint(f"status IN ({sql_in(RunStatus)})", name="ck_job_runs_status"),
CheckConstraint(f"trigger IN ({sql_in(RunTrigger)})", name="ck_job_runs_trigger"),
# Only a run still going has no end.
CheckConstraint(
f"(status = '{RunStatus.RUNNING}') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
job: Mapped[str] = mapped_column(String(40))
trigger: Mapped[str] = mapped_column(String(16))
# Who started it: the admin of a manual run, the user of an ad-hoc one; NULL = the schedule.
actor_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="SET NULL"), index=True
)
# What an ad-hoc run worked on (a channel id); NULL for the recurring jobs.
target: Mapped[str | None] = mapped_column(String(128))
status: Mapped[str] = mapped_column(String(16))
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
summary: Mapped[str | None] = mapped_column(Text)
result: Mapped[dict | None] = mapped_column(JSONB)
error: Mapped[str | None] = mapped_column(Text)
failure: Mapped[dict | None] = mapped_column(JSONB)
quota_units: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class SystemConfig(Base):
@@ -818,141 +839,6 @@ class Notification(Base):
)
class MediaAsset(Base, TimestampMixin):
"""A physical downloaded media file — the SHARED cache layer of the download center.
Mirrors the app's "shared catalog, private per-user state" model: one row per unique
``(source_kind, source_ref, format_sig)`` is the global cache. When a second user requests
the same video with a format that hashes to the same `format_sig`, their `download_job`
links to this ready asset instantly — no re-download, no cost. Only the SOURCE download is
shared; per-user edited derivatives (phase 2) are never cached here.
`rel_path` is relative to `settings.files_root` (the mount), so the physical filename is
system-decided and bound to the source id — never renamed (the user's custom name is display
metadata on `download_job`, applied only when downloading to their own machine). The naming
fields (`title`/`uploader`/`upload_date`) are denormalized so the Plex-tree path can be built
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here) frees
the file with its last holder, and `last_access_at` orders the GC's LRU eviction. How long a file
stays is not the asset's: each holder's job carries its own `expires_at` (its library folder's)."""
__tablename__ = "media_assets"
__table_args__ = (
UniqueConstraint(
"source_kind", "source_ref", "format_sig", name="uq_media_asset_cache_key"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512)) # video id, or full URL for ad-hoc
format_sig: Mapped[str] = mapped_column(String(64)) # hash of the output-affecting spec
status: Mapped[str] = mapped_column(
String(16), default="pending", server_default="pending", index=True
) # "pending" | "ready" | "error"
rel_path: Mapped[str | None] = mapped_column(Text) # relative to files_root
storyboard_path: Mapped[str | None] = mapped_column(Text) # phase-2 filmstrip mosaic
# A locally-generated poster frame (ffmpeg), set only when the source had NO thumbnail so the
# card / watch page / link preview can still show an image. Relative to files_root.
poster_path: Mapped[str | None] = mapped_column(Text)
size_bytes: Mapped[int | None] = mapped_column(BigInteger)
duration_s: Mapped[int | None] = mapped_column(Integer)
width: Mapped[int | None] = mapped_column(Integer)
height: Mapped[int | None] = mapped_column(Integer)
container: Mapped[str | None] = mapped_column(String(16))
vcodec: Mapped[str | None] = mapped_column(String(32))
acodec: Mapped[str | None] = mapped_column(String(32))
# "video" | "audio" | "file" — the media-kind discriminator that drives which library actions
# apply (play/edit for media; download-only for a generic uploaded file). Null while an asset is
# still pending (kind is known once the file is probed at finalize). It is the single stored home
# of the invariant `downloads.mediakind.derive` used to compute — never re-derived in prose (an
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Denormalized naming/metadata for the Plex tree + ad-hoc display.
title: Mapped[str | None] = mapped_column(Text)
uploader: Mapped[str | None] = mapped_column(String(255))
# yt-dlp's channel/uploader page URL (channel_url|uploader_url) — lets an auto-filled channel
# render as a clickable link. Null for sources without one (e.g. reddit) or older rows.
uploader_url: Mapped[str | None] = mapped_column(Text)
upload_date: Mapped[date | None] = mapped_column(Date)
thumbnail_url: Mapped[str | None] = mapped_column(Text)
# yt-dlp's canonical page URL for the source (webpage_url) — the clean "downloaded from"
# reference shown in the Files library. Null until the worker extracts it (older/queued rows
# fall back to a URL derived from source_kind+source_ref).
source_webpage_url: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
ref_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
last_access_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The file's SHA-256 (hex), so a machine fetching it can check what it got. Hashed while an upload
# streams in, otherwise on the first ask (`routes.files`); NULL = not hashed yet.
sha256: Mapped[str | None] = mapped_column(String(64))
@validates("rel_path")
def _forget_hash(self, _key: str, rel_path: str | None) -> str | None:
# A hash names the bytes at `rel_path`: a new file there (a conversion, a retry) is not them.
self.sha256 = None
return rel_path
class DownloadProfile(Base, TimestampMixin):
"""A named download preset — a reusable format profile.
`user_id` NULL + `is_builtin` marks the shipped presets (Best MP4, 1080p, audio-only, …);
a user's saved custom profiles carry their `user_id`. `spec` is the format definition
(mode av/v/a, max_height, container, audio_format, codec prefs, embed subs/chapters/
thumbnail, sponsorblock) that `downloads.formats` turns into a yt-dlp selector + a cache
`format_sig`."""
__tablename__ = "download_profiles"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
) # NULL for a builtin preset
name: Mapped[str] = mapped_column(String(80))
spec: Mapped[dict] = mapped_column(JSON)
is_builtin: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
sort_order: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class FileFolder(Base, TimestampMixin):
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
__tablename__ = "file_folders"
__table_args__ = (
UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
Index(
"uq_file_folder_own_name",
"user_id",
func.lower(text("name")),
unique=True,
postgresql_where=text("name IS NOT NULL"),
),
# One Hub folder per user.
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
kind: Mapped[str] = mapped_column(String(16)) # folders.FolderKind
# "site:" + a host name (DNS caps one at 253 characters).
source_key: Mapped[str | None] = mapped_column(String(260))
name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
retention_days: Mapped[int | None] = mapped_column(Integer)
class ApiToken(Base, TimestampMixin):
"""A user's token for a machine to reach their Files library with (`app.apitokens`): sent as
`Authorization: Bearer sfl_…`, honoured on `/api/files` only, and only in the folders its
@@ -994,208 +880,6 @@ class ApiTokenGrant(Base):
can_delete: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
def _ready_on_insert(context) -> datetime | None:
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
done = context.get_current_parameters().get("status") == "done"
return datetime.now(timezone.utc) if done else None
class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
"""A user's request to download one source — the per-user layer over `MediaAsset`.
The worker claims `queued` jobs with ``FOR UPDATE SKIP LOCKED`` and writes live
`progress`/`speed_bps`/`eta_s`/`phase` here (the frontend polls via useLiveQuery: there is
no server push channel, and the worker is a separate process anyway). `profile_snapshot`
freezes the spec at enqueue so later edits/deletion of the profile don't change this job.
`display_name` is the user's own name for the file (display + the Content-Disposition on
local download); the on-disk asset is never renamed. `asset_id` links to the shared file
(SET NULL if the asset is GC'd — the job then shows as expired).
Phase 2 (editor): `job_kind='edit'` jobs derive a new per-user clip from an already-downloaded
ready job. They still hang off a `MediaAsset` (with `source_kind='edit'`, a per-user cache key)
so all the file-serve/ref-count/GC/quota machinery is reused; the worker branches on the
asset's `source_kind` and runs ffmpeg instead of yt-dlp. `source_job_id`/`source_asset_id`
point at the parent download the clip was cut from; `edit_spec` is the (trim/crop) recipe."""
__tablename__ = "download_jobs"
__table_args__ = (
Index("ix_download_jobs_claim", "status", "queue_pos"),
# Every way a job becomes done stamps when (`folders.stamp_done`, `finalize_if_running`).
CheckConstraint(
"status <> 'done' OR ready_at IS NOT NULL", name="ck_download_jobs_ready_at"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL"), index=True
)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512))
# "download" (default), "edit" (a per-user trim/crop derivative of another job), "upload" (a
# user's own file, converted by the worker when it is not phone-safe — see `downloads.upload`) or
# "clip" (a range cut straight from a YouTube video or a Plex item — see `downloads.clip`).
job_kind: Mapped[str] = mapped_column(String(16), default="download", server_default="download")
# For edit jobs: the parent download this clip is cut from (SET NULL if the parent is deleted).
source_job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="SET NULL")
)
source_asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL")
)
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
edit_spec: Mapped[dict | None] = mapped_column(JSON)
profile_id: Mapped[int | None] = mapped_column(
ForeignKey("download_profiles.id", ondelete="SET NULL")
)
profile_snapshot: Mapped[dict] = mapped_column(JSON)
display_name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
# Per-user display overrides + extras (like display_name — metadata only, never the file).
# `display_uploader`/`display_uploader_url` override the asset's auto channel name/link;
# `extra_links` is an optional list of extra reference URLs shown on the card + watch page.
display_uploader: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
display_uploader_url: Mapped[str | None] = mapped_column(Text)
extra_links: Mapped[list | None] = mapped_column(JSON)
status: Mapped[str] = mapped_column(
String(16), default="queued", server_default="queued", index=True
) # queued | running | paused | done | error | canceled
progress: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
speed_bps: Mapped[int | None] = mapped_column(BigInteger)
eta_s: Mapped[int | None] = mapped_column(Integer)
phase: Mapped[str | None] = mapped_column(String(32))
error: Mapped[str | None] = mapped_column(Text) # raw yt-dlp/ffmpeg text (technical detail)
error_code: Mapped[str | None] = mapped_column(
String(24)
) # category, see downloads.errors.classify
# The host-specific failure-catalog entry on top of the category (`facebook.login_wall`), or NULL.
error_detail: Mapped[str | None] = mapped_column(String(48)) # see downloads.errors.diagnose
queue_pos: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# The Files library: the folder this file sits in (`downloads.folders.place` at creation; an own
# folder is emptied back into the source folders before it is deleted, hence RESTRICT), whether
# the user keeps it past its folder's retention, and when it goes (stamped once done; NULL =
# never). `expiry_warned` marks the one warning the GC sends before it deletes the file.
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="RESTRICT"), index=True
)
# The origin the file came from (the source folder's key), recorded where it is placed: an edit
# traces its origin through its parent, and that parent may be deleted or expire later — the
# file must still know which source folder it may go back to. NULL only for a file placed before
# the column existed outside a source folder; `folders.source_key_for` works those out.
source_key: Mapped[str | None] = mapped_column(String(260))
kept: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# When the file became done — what the library's "Added" means: a download that failed for days
# and then went through is new to its user, so it sorts as new. `created_at` stays when it was
# asked for (the queue's order). Set again when a file is made anew (a retry, a re-download).
ready_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), default=_ready_on_insert
)
class DownloadShare(Base, TimestampMixin):
"""A user→user grant: `to_user` may access `from_user`'s downloaded file (private by
default). Since assets are already deduped/shared on disk, a share is just an ACL row."""
__tablename__ = "download_shares"
__table_args__ = (UniqueConstraint("job_id", "to_user_id", name="uq_download_share"),)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int] = mapped_column(ForeignKey("download_jobs.id", ondelete="CASCADE"))
from_user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
)
to_user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
class DownloadQuota(Base, UpdatedAtMixin):
"""Per-user download limits (admin-managed). A row exists only when an admin customizes a
user; otherwise the sysconfig `download_default_*` values apply. `unlimited` bypasses the
byte cap entirely (e.g. for the admin's own account)."""
__tablename__ = "download_quotas"
user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), primary_key=True
)
max_bytes: Mapped[int] = mapped_column(BigInteger)
max_concurrent: Mapped[int] = mapped_column(Integer)
max_jobs: Mapped[int] = mapped_column(Integer)
unlimited: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class DownloadLink(Base, TimestampMixin):
"""A public capability link to one download, or to one of its owner's own folders — the "share
by link" surface.
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
the folder at the time it is opened. The token itself is never stored as written: a link is
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
grants a *registered* user access via the in-app "Shared with me" list."""
__tablename__ = "download_links"
__table_args__ = (
# A link opens exactly one thing: a file or a folder.
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
password_hash: Mapped[str | None] = mapped_column(Text)
# Bumped whenever the password changes; folded into the signed watch grant so rotating the
# password immediately invalidates any grants issued under the old one (instead of them living
# out the 6h grant TTL).
password_version: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
view_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class DownloadCredential(Base, TimestampMixin, UpdatedAtMixin):
"""A user's stored login cookies for ONE auth-walled download host (Facebook, Instagram, …).
Per-user, per-host: each user pastes their OWN browser cookies.txt (Netscape format) for a host
where they're logged in (e.g. a closed Facebook group they belong to), and a download of a
yt-dlp-supported, auth-walled link on that host is retried with the ACTING user's cookie. The
blob is Fernet-encrypted at rest (`cookies_enc`) and is NEVER returned by any API/log/audit —
the routes are write-only (host + updated_at only). See `downloads.credentials`.
Two load-bearing security controls live around this row, NOT on it:
* **cache privacy** — a cookie-authed download SALTS its MediaAsset `format_sig` with the
user id, so the authed asset is private to that user and can never be cache-served to
another user (`downloads.credentials.cookie_salt`).
* **host scope (SEC-12)** — a cookie is attached ONLY when the download URL's netloc matches
this row's `host_key` (`downloads.credentials.host_key_for_url`), so an arbitrary `?url=`
can't exfiltrate the user's session to a look-alike host.
`host_key` is a short canonical key from the supported-host registry (e.g. "facebook"), not the
raw netloc — one credential can cover several domains of the same site (facebook.com/fb.watch)."""
__tablename__ = "download_credentials"
__table_args__ = (UniqueConstraint("user_id", "host_key", name="uq_download_credential"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
host_key: Mapped[str] = mapped_column(String(64))
cookies_enc: Mapped[str] = mapped_column(Text) # Fernet ciphertext — never returned/logged
# --- Plex integration (optional) -------------------------------------------------------------
# A dedicated, self-contained catalog mirrored from a locally-reachable Plex server. Playback is
# from the LOCAL physical file (Plex is metadata only); per-user watch state lives in plex_states
+11 -3
View File
@@ -16,7 +16,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.db import SessionLocal
from app.models import ApiQuotaUsage, QuotaEvent
@@ -204,6 +204,14 @@ def record_usage(units: int) -> None:
)
with SessionLocal() as qs:
qs.execute(stmt)
# Audit detail: who/what spent it (per-user attribution; NULL actor = system).
qs.add(QuotaEvent(user_id=_actor_id.get(), action=_action.get(), units=units))
# Audit detail: who/what spent it (per-user attribution; NULL actor = system), and the
# background run it belongs to, so the run's row can say what it cost.
qs.add(
QuotaEvent(
user_id=_actor_id.get(),
action=_action.get(),
units=units,
job_run_id=logs.current_run_id(),
)
)
qs.commit()
+17
View File
@@ -0,0 +1,17 @@
"""The account filter the activity page's lists share (runs, admin actions, server log, quota): one
filter bar drives them all, so the value it sends means the same on every tab.
The value is an account id, or `none` — the rows no account made (the background) — a value to filter
on, not the absence of a filter.
"""
from sqlalchemy import ColumnElement
NONE = "none"
# The `user` query parameter's shape on every route that takes it.
PATTERN = r"^(none|[0-9]{1,10})$"
def condition(column, value: str) -> ColumnElement[bool]:
"""`column` (an account id column) narrowed to `value`."""
return column.is_(None) if value == NONE else column == int(value)
+3 -2
View File
@@ -16,6 +16,7 @@ from app.db import Base
from app.models import POSITION_MAX_S, User
from app.issues import gitea
from app.refusal import refused
from app.userscope import owned_row
from app.youtube.client import YouTubeError, refusal_reason
M = TypeVar("M", bound=Base)
@@ -90,8 +91,8 @@ def owned_or_404(
else 404s (NOT 403) on purpose — a 403 would confirm the id exists to a stranger.
It replaced the near-identical per-route ownership helpers, and is called directly — a thin
`_own_x` wrapper around it is the ad-hoc shape CLAUDE.md's patterns table rules out."""
obj = get_or_404(db, model, id_, what=what, lock=lock)
if getattr(obj, owner_attr) != user.id:
obj = owned_row(db, model, id_, user, owner_attr=owner_attr, lock=lock)
if obj is None:
raise not_found(what)
return obj
+66
View File
@@ -0,0 +1,66 @@
"""Keyset paging over `(timestamp desc, id desc)`, shared by the admin history lists (server log,
audit log, background runs).
A timestamp alone is not unique (a browser log row's `ts` is clamped into its consent window; a run
and its siblings can start in the same microsecond), and an offset would skip or repeat rows while
writers add new ones at the top — so a page continues from the last row's `(timestamp, id)`.
"""
import base64
import binascii
import json
from datetime import datetime, timezone
from fastapi import HTTPException
from sqlalchemy import and_, or_, select
from sqlalchemy.orm import Session
from app.models import User
PAGE_MAX = 200
def utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the columns are `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def encode_cursor(ts: datetime, row_id: int) -> str:
return base64.urlsafe_b64encode(json.dumps([ts.isoformat(), row_id]).encode()).decode()
def decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
at = datetime.fromisoformat(ts)
return (at if at.tzinfo is not None else at.replace(tzinfo=timezone.utc)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def older_than(ts_col, id_col, cursor: str):
"""The rows after `cursor` in `(ts desc, id desc)` order. The `ts <=` conjunct is implied by the
OR; it is there so the planner can bound the timestamp index scan instead of filtering every
newer row first."""
ts, row_id = decode_cursor(cursor)
return and_(ts_col <= ts, or_(ts_col < ts, and_(ts_col == ts, id_col < row_id)))
def page(rows: list, limit: int, ts_of) -> tuple[list, str | None]:
"""`rows` was read with `limit + 1` (to know whether there is a next page without a count query):
the page itself, and the cursor that continues after it (None on the last page)."""
more = len(rows) > limit
rows = rows[:limit]
return rows, encode_cursor(ts_of(rows[-1]), rows[-1].id) if more else None
def accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the row
was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
+32 -71
View File
@@ -8,33 +8,29 @@ What a row holds was redacted when it was written (`logs.record_fields`, and the
browser's rows), so this serves the columns as they are; a value the redactor took out reads as
`[redacted]` / `[email]`, which is the point — the console shows that something was there.
Paging is a keyset over `(ts desc, id desc)`: a browser row's `ts` is clamped into its consent
window, so `ts` alone is not unique, and an offset would skip or repeat rows while the writer adds
new ones at the top. The live tail re-reads the first page rather than asking for "ids after N":
Paging is a keyset over `(ts desc, id desc)` (`_keyset`). The live tail re-reads the first page rather than asking for "ids after N":
several processes write this table, and a batch's ids are taken when it inserts, not when it
commits, so a slower batch can land below an id the tail already passed — and would never show.
"""
import base64
import binascii
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import Text, and_, cast, func, or_, select
from fastapi import APIRouter, Depends, Query
from sqlalchemy import Text, cast, func, or_, select
from sqlalchemy.orm import Session
from app import logsink
from app.auth import admin_user
from app.db import get_db
from app.models import LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.utils import like_escape
router = APIRouter(prefix="/api/admin/logs", tags=["admin"])
PAGE_MAX = 200
# A facet list is a chip row, not a report: past this many distinct events the rest are reachable
# through the free-text search, and the response stays small whatever a self-host logs.
FACET_EVENTS_MAX = 100
@@ -42,38 +38,9 @@ FACET_EVENTS_MAX = 100
Level = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
# The filters a facet can leave out of its own count (`LogFilters.conditions`).
Facet = Literal["levels", "sources", "user", "event"]
# `none` selects the rows that have no account / no event code (a scheduler line, an anonymous
# request) — a real value to filter on, not the absence of a filter.
NONE = "none"
def _utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the column is `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def _encode_cursor(row: LogEvent) -> str:
return base64.urlsafe_b64encode(json.dumps([row.ts.isoformat(), row.id]).encode()).decode()
def _decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
return _utc(datetime.fromisoformat(ts)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def _accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the
line was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
# `none` selects the rows that have no event code, as the account filter's `none` (app/routes/_account)
# selects the rows of no account — a real value to filter on, not the absence of a filter.
NONE = account.NONE
@dataclass(frozen=True)
@@ -86,6 +53,7 @@ class LogFilters:
event: str | None
request_id: str | None
job_id: int | None
run_id: int | None
q: str | None
since: datetime | None
until: datetime | None
@@ -100,11 +68,7 @@ class LogFilters:
if self.sources and skip != "sources":
out.append(LogEvent.source.in_(self.sources))
if self.user is not None and skip != "user":
out.append(
LogEvent.user_id.is_(None)
if self.user == NONE
else LogEvent.user_id == int(self.user)
)
out.append(account.condition(LogEvent.user_id, self.user))
if self.event is not None and skip != "event":
out.append(
LogEvent.event.is_(None) if self.event == NONE else LogEvent.event == self.event
@@ -113,10 +77,12 @@ class LogFilters:
out.append(LogEvent.request_id == self.request_id)
if self.job_id is not None:
out.append(LogEvent.job_id == self.job_id)
if self.run_id is not None:
out.append(LogEvent.run_id == self.run_id)
if self.since is not None:
out.append(LogEvent.ts >= _utc(self.since))
out.append(LogEvent.ts >= utc(self.since))
if self.until is not None:
out.append(LogEvent.ts < _utc(self.until))
out.append(LogEvent.ts < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
@@ -134,17 +100,20 @@ class LogFilters:
def log_filters(
levels: list[Level] = Query(default=[]),
sources: list[str] = Query(default=[], max_length=8),
user: str | None = Query(default=None, pattern=r"^(none|[0-9]{1,10})$"),
user: str | None = Query(default=None, pattern=account.PATTERN),
event: str | None = Query(
default=None, max_length=64, pattern=r"^(none|[a-z][a-z0-9_]*(\.[a-z0-9_]+)*)$"
),
request_id: str | None = Query(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$"),
job_id: int | None = Query(default=None, ge=1),
run_id: int | None = Query(default=None, ge=1),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> LogFilters:
return LogFilters(list(levels), list(sources), user, event, request_id, job_id, q, since, until)
return LogFilters(
list(levels), list(sources), user, event, request_id, job_id, run_id, q, since, until
)
@router.get("")
@@ -159,28 +128,20 @@ def list_logs(
continues from a previous page's `next_cursor`."""
query = select(LogEvent).where(*filters.conditions())
if before is not None:
ts, row_id = _decode_cursor(before)
# The `ts <=` conjunct is implied by the OR; it is there so the planner can bound the
# `ts` index scan instead of filtering every newer row first.
query = query.where(
and_(
LogEvent.ts <= ts,
or_(LogEvent.ts < ts, and_(LogEvent.ts == ts, LogEvent.id < row_id)),
)
)
# One more than asked, to know whether there is a next page without a second count query.
rows = (
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
query = query.where(older_than(LogEvent.ts, LogEvent.id, before))
rows, cursor = page(
list(
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
),
limit,
lambda r: r.ts,
)
more = len(rows) > limit
rows = rows[:limit]
return {
"items": [logsink.serialize(r) for r in rows],
"accounts": _accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": _encode_cursor(rows[-1]) if more else None,
"accounts": accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": cursor,
}
@@ -211,6 +172,6 @@ def log_facets(
"sources": counts(LogEvent.source, "sources"),
"events": counts(LogEvent.event, "event", FACET_EVENTS_MAX),
"users": [{**u, "value": str(u["value"])} for u in users],
"accounts": _accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"accounts": accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"total": db.scalar(total) or 0,
}
+106 -29
View File
@@ -1,67 +1,144 @@
"""Admin audit log API: read the append-only who/what/when trail, and clear it.
Admin-only. The client renders these rows in a DataTable (client-side sort/filter/paginate), so
this returns the most recent window of rows (capped) plus the true total, rather than paging
server-side — the log stays small in practice (state-changing runs + admin actions only)."""
Admin-only. The activity page reads it a page at a time, newest first, filtered on the server like
the server log (`_keyset` paging), so the whole trail is reachable however long it grows."""
from fastapi import APIRouter, Depends
from sqlalchemy import delete, func, select
from dataclasses import dataclass
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy import Text, cast, delete, func, or_, select
from sqlalchemy.orm import Session
from app import audit
from app.audit import AuditAction
from app.auth import admin_user
from app.db import get_db
from app.models import AuditLog, User
from app.utils import iso
from app.models import AuditLog, LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.utils import iso, like_escape
router = APIRouter(prefix="/api/admin/audit", tags=["admin"])
# Cap the client-side window. The audit log is small by design; if it ever outgrows this we'd
# switch the DataTable to server-side paging.
MAX_ROWS = 2000
def _serialize(row: AuditLog, actors: dict[int, str]) -> dict:
def _serialize(row: AuditLog, lines: dict[str, int]) -> dict:
return {
"id": row.id,
"created_at": iso(row.created_at),
# actor_id NULL = the scheduler/background did it (client shows "System"); a set id that
# can't be resolved would mean a deleted user, but SET NULL makes that NULL — so present.
"actor": actors.get(row.actor_id) if row.actor_id is not None else None,
# NULL = the background did it (the client shows "System"); the address is in `accounts`.
"actor_id": row.actor_id,
"action": row.action,
"target_type": row.target_type,
"target_id": row.target_id,
"summary": row.summary,
"before": row.before,
"after": row.after,
"request_id": row.request_id,
# How many of that request's log lines are still kept: they go after days, the row after
# months, and a link to none would open an empty list.
"log_lines": lines.get(row.request_id, 0) if row.request_id else 0,
}
@dataclass(frozen=True)
class AuditFilters:
action: str | None
actor: str | None
q: str | None
since: datetime | None
until: datetime | None
def conditions(self, skip: Literal["action"] | None = None) -> list:
"""The WHERE conditions, ANDed. `skip="action"` leaves the action filter out, so the action
menu counts every action under the other filters (the way back from a pick)."""
out = []
if self.action is not None and skip != "action":
out.append(AuditLog.action == self.action)
if self.actor is not None:
out.append(account.condition(AuditLog.actor_id, self.actor))
if self.since is not None:
out.append(AuditLog.created_at >= utc(self.since))
if self.until is not None:
out.append(AuditLog.created_at < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
or_(
AuditLog.summary.ilike(like),
AuditLog.target_id.ilike(like),
AuditLog.action.ilike(like),
AuditLog.request_id.ilike(like),
cast(AuditLog.before, Text).ilike(like),
cast(AuditLog.after, Text).ilike(like),
)
)
return out
def audit_filters(
action: str | None = Query(default=None, max_length=48, pattern=r"^[a-z][a-z0-9_]*$"),
user: str | None = Query(default=None, pattern=account.PATTERN),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> AuditFilters:
return AuditFilters(action, user, q, since, until)
@router.get("")
def list_audit(
limit: int = 500,
filters: AuditFilters = Depends(audit_filters),
before: str | None = Query(default=None, max_length=200),
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
limit = max(1, min(limit, MAX_ROWS))
total = db.scalar(select(func.count()).select_from(AuditLog)) or 0
rows = (
db.execute(select(AuditLog).order_by(AuditLog.created_at.desc()).limit(limit))
.scalars()
.all()
"""One page of rows, newest first, matching every filter given; `before` continues from a
previous page's `next_cursor`."""
query = select(AuditLog).where(*filters.conditions())
if before is not None:
query = query.where(older_than(AuditLog.created_at, AuditLog.id, before))
order = (AuditLog.created_at.desc(), AuditLog.id.desc())
rows, cursor = page(
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
limit,
lambda r: r.created_at,
)
actor_ids = {r.actor_id for r in rows if r.actor_id is not None}
actors: dict[int, str] = {}
if actor_ids:
actors = dict(db.execute(select(User.id, User.email).where(User.id.in_(actor_ids))).all())
requests = {r.request_id for r in rows if r.request_id}
lines: dict[str, int] = {}
if requests:
counted = db.execute(
select(LogEvent.request_id, func.count())
.where(LogEvent.request_id.in_(requests))
.group_by(LogEvent.request_id)
)
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
return {
"items": [_serialize(r, actors) for r in rows],
"total": total,
"returned": len(rows),
"items": [_serialize(r, lines) for r in rows],
"accounts": accounts(db, {r.actor_id for r in rows if r.actor_id is not None}),
"next_cursor": cursor,
}
@router.get("/facets")
def audit_facets(
filters: AuditFilters = Depends(audit_filters),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""The actions present under the other filters, with counts — what the action menu offers."""
n = func.count().label("n")
stmt = (
select(AuditLog.action, n)
.where(*filters.conditions("action"))
.group_by(AuditLog.action)
.order_by(n.desc(), AuditLog.action)
)
return {"actions": [{"value": a, "count": c} for a, c in db.execute(stmt).all()]}
@router.delete("")
def clear_audit(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
"""Clear the audit log. Leaves ONE row recording who cleared it + how many (tamper-evidence)."""
+16 -9
View File
@@ -3,15 +3,16 @@ hidden, tags) and per-channel sync state. Local reads/writes plus YouTube-side w
(subscribe / unsubscribe) that require the optional write scope."""
from datetime import datetime, timezone
from functools import partial
from fastapi import APIRouter, BackgroundTasks, Depends
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import Select, func, select, update
from sqlalchemy.orm import Session
from app.collation import collated, sorted_words
from app.mailtext import lang_for_user
from app import logs, quota
from app import logs, quota, scheduler
from app.auth import admin_user, current_user, require_write_scope
from app.db import get_db
from app.models import (
@@ -481,7 +482,6 @@ class ChannelUpdateIn(BaseModel):
def update_channel(
channel_id: str,
body: ChannelUpdateIn,
background: BackgroundTasks,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
@@ -508,12 +508,19 @@ def update_channel(
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_BACKFILL_RECENT):
run_recent_backfill(db, [channel], max_channels=1)
# …and kick the FULL deep backfill for this one channel immediately in the background, so the
# whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks. The
# request returns right away; the channel page's live poll shows the count climb. The scheduler
# still finishes a very large channel's tail on its normal runs (deep_requested stays set).
if deep_turned_on:
background.add_task(backfill_channel_now, channel_id, user.id)
# …and kick the FULL deep backfill for this one channel immediately, as a recorded run of its own,
# so the whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks.
# The request returns right away; the channel page's live poll shows the count climb. The
# scheduler still finishes a very large channel's tail on its normal runs (deep_requested stays set).
# A channel whose history is all in gets no run: it would only say "nothing to do", one kept row
# per flip of the flag.
if deep_turned_on and not (channel is not None and channel.backfill_done):
scheduler.run_ad_hoc(
scheduler.CHANNEL_BACKFILL,
partial(backfill_channel_now, channel_id, user.id),
actor_id=user.id,
target=channel_id,
)
return {
"ok": True,
+5 -3
View File
@@ -61,7 +61,7 @@ def set_config(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
s = sysconfig.spec(key)
s = sysconfig.config_page_spec(key)
if s is None:
raise not_found("config_key")
if "value" not in body.model_fields_set:
@@ -89,7 +89,9 @@ def set_config(
before=None if s.secret else {"value": _redact(old)},
after=None if s.secret else {"value": _redact(new)},
)
db.commit()
# The override is committed even when the trail skips it: re-saving the value in force still
# pins it as an override, and an uncommitted write would be dropped with the session.
db.commit()
_apply_runtime(db, key)
return sysconfig.describe(db)
@@ -100,7 +102,7 @@ def reset_config(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
s = sysconfig.spec(key)
s = sysconfig.config_page_spec(key)
if s is None:
raise not_found("config_key")
old = None if s.secret else sysconfig.get(db, key)
+106 -337
View File
@@ -3,23 +3,22 @@
All routes are behind `current_user`; admin routes add `admin_user`. Live job state is polled by the
frontend (the worker is a separate process); this module never downloads — it enqueues, manages job
state, and shares finished files. The library itself — listing, renaming, moving, deleting and
serving a done file, and uploading one — is `routes.files`.
serving a done file, and uploading one — is `routes.files`. Reads come from `downloads.queries`,
writes go through Capture's service (`service`, `links`, `quota`); this module validates, answers in
HTTP and commits.
"""
import re
from typing import Literal
from typing import TYPE_CHECKING, Literal
from datetime import datetime, timedelta, timezone
from urllib.parse import parse_qs, urlparse
from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import FileResponse
from pydantic import BaseModel, Field
from sqlalchemy import delete, func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.mailtext import lang_for_user
from app.collation import collated
from app import audit, sysconfig
from app.auth import admin_user, current_user
from app.config import settings
@@ -29,45 +28,32 @@ from app.downloads import credentials as dlcreds
from app.downloads import edit as editmod
from app.downloads import folders
from app.downloads import links as linksmod
from app.downloads import quota, service, storage
from app.downloads import queries, quota, service, storage
from app.downloads import urlguard
from app.downloads.folders import FolderKind
from app.downloads.status import FINISHED, NOT_REFETCHABLE, JobStatus
from app.models import (
DownloadJob,
DownloadLink,
FileFolder,
DownloadProfile,
DownloadQuota,
DownloadShare,
LibraryState,
MediaAsset,
User,
display_text,
)
from app.downloads.status import NOT_REFETCHABLE, JobStatus
from app.models import User, display_text
from app.routes._common import (
ProgressIn,
get_or_404,
kept_position,
not_found,
owned_or_404,
)
from app.routes.files import (
MAX_LIST,
accessible_job,
assets_for,
own_folder,
own_job,
own_job_for_update,
positions_for,
quota_detail,
serialize,
serialize_job,
source_keys,
)
from app.security import hash_password
from app.userscope import is_messageable_user, messageable_clauses
from app.utils import iso
from app.userscope import emails_of, is_messageable_user, messageable_others, user_by_email
from app.refusal import refused
if TYPE_CHECKING:
from app.downloads.models import DownloadJob, DownloadLink, DownloadProfile, FileFolder
router = APIRouter(prefix="/api/downloads", tags=["downloads"])
admin_router = APIRouter(prefix="/api/admin/downloads", tags=["admin-downloads"])
@@ -119,7 +105,7 @@ def resolve_source(raw: str) -> tuple[str, str]:
# --- profiles ------------------------------------------------------------------------------
def _serialize_profile(p: DownloadProfile) -> dict:
def _serialize_profile(p: "DownloadProfile") -> dict:
return {
"id": p.id,
"name": p.name,
@@ -131,18 +117,7 @@ def _serialize_profile(p: DownloadProfile) -> dict:
@router.get("/profiles")
def list_profiles(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
rows = (
db.execute(
select(DownloadProfile)
.where((DownloadProfile.is_builtin.is_(True)) | (DownloadProfile.user_id == user.id))
.order_by(
DownloadProfile.is_builtin.desc(), DownloadProfile.sort_order, DownloadProfile.id
)
)
.scalars()
.all()
)
return [_serialize_profile(p) for p in rows]
return [_serialize_profile(p) for p in queries.profiles_for(db, user)]
class CreateProfileIn(BaseModel):
@@ -158,15 +133,7 @@ def create_profile(
spec = body.spec
if not name:
raise refused(400, "profile_name_required")
maxpos = db.execute(
select(func.coalesce(func.max(DownloadProfile.sort_order), 100)).where(
DownloadProfile.user_id == user.id
)
).scalar_one()
p = DownloadProfile(
user_id=user.id, name=name[:80], spec=spec, is_builtin=False, sort_order=maxpos + 1
)
db.add(p)
p = service.create_profile(db, user, name, spec)
db.commit()
return _serialize_profile(p)
@@ -183,31 +150,32 @@ def update_profile(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
p = db.get(DownloadProfile, profile_id)
if p is None or p.is_builtin or p.user_id != user.id:
raise not_found("profile")
p = _own_profile(db, user, profile_id)
sent = body.model_fields_set
name = None
if "name" in sent:
name = (body.name or "").strip()
if not name:
raise refused(400, "profile_name_required")
p.name = name[:80]
if "spec" in sent:
if body.spec is None:
raise refused(400, "spec_not_object")
p.spec = body.spec
if "spec" in sent and body.spec is None:
raise refused(400, "spec_not_object")
service.update_profile(p, name=name, spec=body.spec if "spec" in sent else None)
db.commit()
return _serialize_profile(p)
def _own_profile(db: Session, user: User, profile_id: int) -> "DownloadProfile":
p = queries.own_profile(db, user, profile_id)
if p is None:
raise not_found("profile")
return p
@router.delete("/profiles/{profile_id}")
def delete_profile(
profile_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
p = db.get(DownloadProfile, profile_id)
if p is None or p.is_builtin or p.user_id != user.id:
raise not_found("profile")
db.delete(p)
service.delete_profile(db, _own_profile(db, user, profile_id))
db.commit()
return {"deleted": profile_id}
@@ -277,18 +245,13 @@ def _resolve_spec(
"""Pick the format spec for an enqueue: an explicit profile_id (builtin or the user's own),
an inline spec, or the first builtin as a fallback."""
if profile_id is not None:
p = db.get(DownloadProfile, profile_id)
if p is None or (not p.is_builtin and p.user_id != user.id):
p = queries.usable_profile(db, user, profile_id)
if p is None:
raise not_found("profile")
return p.spec, p.id
if spec is not None:
return spec, None
fallback = db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True))
.order_by(DownloadProfile.sort_order)
.limit(1)
).scalar_one_or_none()
fallback = queries.default_profile(db)
if fallback is None:
raise refused(400, "no_profile")
return fallback.spec, fallback.id
@@ -326,6 +289,7 @@ def enqueue_download(
job = service.enqueue(db, user.id, source_kind, source_ref, spec, profile_id, display_name)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
db.commit()
return serialize_job(db, job)
@@ -355,6 +319,7 @@ def enqueue_edit(
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
except service.EditError as e:
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason}) from e
db.commit()
return serialize_job(db, job)
@@ -398,38 +363,22 @@ def enqueue_clip(
raise HTTPException(status_code=status, detail=detail) from e
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
db.commit()
return serialize_job(db, job)
# How long a finished job stays in the queue: the tray polls it from every page and must see a job
# turn done (its "Done · Open" row and the done notice) — the finished file itself is the library's.
# A done job is timed from when it became done, never from its last write: a rename, a move or the
# GC's expiry warning touches an old file's row, and the tray would announce it as just finished.
QUEUE_RECENT = timedelta(minutes=10)
@router.get("")
def list_downloads(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
"""The queue: what is waiting, running, paused or failed, and what finished a moment ago."""
recent = datetime.now(timezone.utc) - QUEUE_RECENT
jobs = (
db.execute(
select(DownloadJob)
.where(
DownloadJob.user_id == user.id,
DownloadJob.status.not_in(FINISHED)
| ((DownloadJob.status == JobStatus.DONE) & (DownloadJob.ready_at >= recent))
| ((DownloadJob.status == JobStatus.CANCELED) & (DownloadJob.updated_at >= recent)),
)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
)
.scalars()
.all()
)
assets = assets_for(db, jobs)
positions = positions_for(db, user, jobs)
sources = source_keys(db, jobs)
jobs = queries.queue(db, user, datetime.now(timezone.utc) - QUEUE_RECENT)
assets = queries.assets_for(db, jobs)
positions = queries.positions_for(db, user, jobs)
sources = queries.source_keys(db, jobs)
return [
{
**serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None),
@@ -453,14 +402,8 @@ _INDEX_RANK: dict[str, int] = {
def my_download_index(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
"""source_ref -> best active status, for the feed's per-card "downloaded / queued" badge.
Small + cheap so the feed can poll it without loading the full job list."""
rows = db.execute(
select(DownloadJob.source_ref, DownloadJob.status).where(
DownloadJob.user_id == user.id,
DownloadJob.status.in_(tuple(_INDEX_RANK)),
)
).all()
out: dict[str, str] = {}
for ref, status in rows:
for ref, status in queries.statuses_by_source(db, user, tuple(_INDEX_RANK)):
if _INDEX_RANK.get(status, 0) > _INDEX_RANK.get(out.get(ref, ""), 0):
out[ref] = status
return out
@@ -472,6 +415,7 @@ def pause_download(
) -> dict:
job = own_job_for_update(db, user, job_id)
service.pause(db, job)
db.commit()
return serialize_job(db, job)
@@ -481,6 +425,7 @@ def resume_download(
) -> dict:
job = own_job_for_update(db, user, job_id)
service.resume(db, job)
db.commit()
return serialize_job(db, job)
@@ -508,6 +453,7 @@ def redownload_download(
if job.status in (JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED):
raise refused(409, "still_downloading")
service.redownload(db, job)
db.commit()
return serialize_job(db, job)
@@ -517,6 +463,7 @@ def cancel_download(
) -> dict:
job = own_job_for_update(db, user, job_id)
service.cancel(db, job)
db.commit()
return serialize_job(db, job)
@@ -534,24 +481,7 @@ def set_library_progress(
one shared with them). A position too early or too close to the end clears it instead."""
accessible_job(db, user, job_id)
position = kept_position(body)
mine = (LibraryState.user_id == user.id) & (LibraryState.job_id == job_id)
if not position:
# A statement, not a loaded row's delete: the periodic checkpoint and the pagehide beacon of
# one stop may both clear it, and the second then deletes nothing rather than failing.
db.execute(delete(LibraryState).where(mine))
db.commit()
return {"ok": True, "job_id": job_id, "position_seconds": 0}
now = datetime.now(timezone.utc)
# One row per (user, job) even when two checkpoints race (the same pair): an upsert, not a
# read-then-insert that the second would fail on the unique key.
db.execute(
pg_insert(LibraryState)
.values(user_id=user.id, job_id=job_id, position_seconds=position, progress_updated_at=now)
.on_conflict_do_update(
constraint="uq_library_user_job",
set_={"position_seconds": position, "progress_updated_at": now},
)
)
service.set_position(db, user, job_id, position)
db.commit()
return {"ok": True, "job_id": job_id, "position_seconds": position}
@@ -564,8 +494,7 @@ def storyboard_meta(
Best-effort: for a very long/high-res source the sprite may not generate in time — then
`available` is False and the editor shows a plain timeline."""
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
asset = queries.asset_of(db, accessible_job(db, user, job_id))
if asset is None or asset.status != "ready" or not asset.rel_path:
raise refused(409, "download_not_ready")
meta = editmod.ensure_storyboard(asset, settings.files_root)
@@ -585,8 +514,7 @@ def storyboard_meta(
@router.get("/{job_id}/poster.jpg")
def poster_image(job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)):
"""The locally-generated poster frame for a download whose source had no thumbnail."""
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
asset = queries.asset_of(db, accessible_job(db, user, job_id))
if asset is None or not asset.poster_path:
raise not_found("poster")
path = storage.safe_abs_path(settings.files_root, asset.poster_path)
@@ -599,8 +527,7 @@ def poster_image(job_id: int, user: User = Depends(current_user), db: Session =
def storyboard_image(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
):
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
asset = queries.asset_of(db, accessible_job(db, user, job_id))
if asset is None or not asset.storyboard_path:
raise not_found("filmstrip")
path = storage.safe_abs_path(settings.files_root, asset.storyboard_path)
@@ -623,45 +550,20 @@ def share_download(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
job = own_job(db, user, job_id)
if job.status != JobStatus.DONE or job.asset_id is None:
raise refused(409, "share_unfinished")
target = body.email.strip().lower()
if not target:
raise refused(400, "recipient_required")
recipient = db.execute(
select(User).where(func.lower(User.email) == target)
).scalar_one_or_none()
recipient = user_by_email(db, target)
# Uniform "no user" for a missing OR non-messageable (suspended/deactivated) target, so
# sharing can't be used to probe which addresses exist or to reach a suspended account.
if not is_messageable_user(recipient):
if recipient is None or not is_messageable_user(recipient):
raise not_found("recipient")
if recipient.id == user.id:
raise refused(400, "share_own")
exists = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == recipient.id
)
).first()
if exists is None:
db.add(DownloadShare(job_id=job_id, from_user_id=user.id, to_user_id=recipient.id))
from app.notifications import create_notification
create_notification(
db,
user_id=recipient.id,
type="download_shared",
title=(job.display_name or "A download") + " was shared with you",
# The client names the notice in the reader's language from `data`; the title is the
# English fallback only.
data={
"kind": "download_shared",
"from": user.email,
"job_id": job_id,
"title": job.display_name,
},
commit=False,
)
if service.share(db, job, user, recipient):
db.commit()
return {"ok": True, "shared_with": recipient.email}
@@ -673,22 +575,10 @@ def unshare_download(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
owned_or_404(
db, DownloadJob, job_id, user, what="download"
) # ownership guard (404s if not the caller's job)
recipient = db.execute(
select(User).where(func.lower(User.email) == email.strip().lower())
).scalar_one_or_none()
if recipient is not None:
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id,
DownloadShare.to_user_id == recipient.id,
)
).scalar_one_or_none()
if row is not None:
db.delete(row)
db.commit()
own_job(db, user, job_id) # ownership guard (404s if not the caller's job)
recipient = user_by_email(db, email)
if recipient is not None and service.unshare(db, job_id, recipient):
db.commit()
return {"ok": True}
@@ -701,54 +591,26 @@ def share_recipients(
) -> list[dict]:
"""Registered users this user can share a download with (excludes self + any non-messageable
account — suspended or deactivated — so the picker never leaks those addresses)."""
rows = (
db.execute(
select(User)
.where(User.id != user.id, *messageable_clauses())
.order_by(collated(User.email, lang_for_user(user)))
)
.scalars()
.all()
)
rows = messageable_others(db, user, lang_for_user(user))
return [{"id": u.id, "email": u.email, "display_name": u.display_name} for u in rows]
# --- public watch links (share by link) ----------------------------------------------------
def _own_link(db: Session, user: User, link_id: int) -> DownloadLink:
link = db.get(DownloadLink, link_id)
def _own_link(db: Session, user: User, link_id: int) -> "DownloadLink":
link = queries.own_link(db, user, link_id)
if link is None:
raise not_found("link")
target = (
db.get(DownloadJob, link.job_id)
if link.job_id is not None
else db.get(FileFolder, link.folder_id)
)
if target is None or target.user_id != user.id:
raise not_found("link")
return link
def _link_expiry(days: int | None) -> datetime | None:
# None/0/negative → no expiry; otherwise N days out, capped at ~10 years. The Pydantic model
# already coerced/validated the type, so a non-number is a 422 before we get here.
if not days or days <= 0:
return None
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
def _links_of(db: Session, *where) -> list[dict]:
rows = db.execute(select(DownloadLink).where(*where).order_by(DownloadLink.id.desc()))
return [linksmod.owner_view(row) for row in rows.scalars()]
@router.get("/{job_id}/links")
def list_links(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
owned_or_404(db, DownloadJob, job_id, user, what="download")
return _links_of(db, DownloadLink.job_id == job_id)
own_job(db, user, job_id)
return [linksmod.owner_view(link) for link in queries.links_to(db, job_id=job_id)]
class CreateLinkIn(BaseModel):
@@ -764,30 +626,18 @@ class CreateLinkIn(BaseModel):
def _new_link(
db: Session, user: User, body: CreateLinkIn, **target: int
) -> tuple[DownloadLink, str]:
"""Make a link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and
the token it is opened by — which is in hand only now."""
pw = body.password.strip()
token = linksmod.new_token()
link = DownloadLink(
**target,
created_by=user.id,
allow_download=body.allow_download,
password_hash=hash_password(pw) if pw else None,
expires_at=_link_expiry(body.expires_days),
)
linksmod.seal(link, token)
db.add(link)
db.flush()
audit.record(
) -> tuple["DownloadLink", str]:
"""A link to `target` (`job_id=` or `folder_id=`, already checked to be the user's) and its
token (`links.create`). The Pydantic model already coerced the expiry, so a non-number is a 422
before this runs."""
return linksmod.create(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_CREATE,
target_type="download_link",
target_id=link.id,
after={**target, **_link_audit(link)},
user,
password=body.password.strip(),
allow_download=body.allow_download,
expires_days=body.expires_days,
**target,
)
return link, token
@router.post("/{job_id}/links")
@@ -805,28 +655,20 @@ def create_link(
return linksmod.owner_view(link, token=token)
def _shareable_job(db: Session, user: User, job_id: int) -> DownloadJob:
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
rather than failing the link's foreign key — or, done first, leaves a 404."""
owned_or_404(db, DownloadJob, job_id, user, what="download")
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update(read=True)
.execution_options(populate_existing=True)
).scalar_one_or_none()
def _shareable_job(db: Session, user: User, job_id: int) -> "DownloadJob":
"""One of the user's files, held for the link's transaction (`service.hold_for_share`)."""
job = service.hold_for_share(db, user, job_id)
if job is None:
raise not_found("download")
return job
def _shareable_folder(db: Session, user: User, folder_id: int) -> FileFolder:
def _shareable_folder(db: Session, user: User, folder_id: int) -> "FileFolder":
"""One of the user's own folders. A source folder lets its files go on its own clock, and the Hub
is the user's channel to a machine of theirs: neither is a shelf to hand out. Held (FOR SHARE)
for the rest of the transaction, so a delete of it at the same moment waits and takes the new
link with it, rather than failing the link's foreign key."""
owned_or_404(db, FileFolder, folder_id, user, what="folder")
own_folder(db, user, folder_id)
try:
folder = folders.own_folder(db, user.id, folder_id)
except folders.FolderError as e:
@@ -840,8 +682,8 @@ def _shareable_folder(db: Session, user: User, folder_id: int) -> FileFolder:
def list_folder_links(
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
owned_or_404(db, FileFolder, folder_id, user, what="folder")
return _links_of(db, DownloadLink.folder_id == folder_id)
own_folder(db, user, folder_id)
return [linksmod.owner_view(link) for link in queries.links_to(db, folder_id=folder_id)]
@router.post("/folders/{folder_id}/links")
@@ -857,15 +699,6 @@ def create_folder_link(
return linksmod.owner_view(link, token=token)
def _link_audit(link: DownloadLink) -> dict:
"""A link's settings as the audit records them: whether it has a password, never which."""
return {
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": iso(link.expires_at),
}
class UpdateLinkIn(BaseModel):
allow_download: bool | None = None
expires_days: int | None = None
@@ -880,29 +713,15 @@ def update_link(
db: Session = Depends(get_db),
) -> dict:
link = _own_link(db, user, link_id)
before = _link_audit(link)
sent = body.model_fields_set
if "allow_download" in sent:
link.allow_download = bool(body.allow_download)
if "expires_days" in sent:
link.expires_at = _link_expiry(body.expires_days)
if "password" in sent:
pw = (body.password or "").strip()
link.password_hash = hash_password(pw) if pw else None
# Bump the version so any grant issued under the previous password stops working now, rather
# than living out its 6h TTL (setting/clearing/changing the password all count as a rotation).
link.password_version += 1
after = _link_audit(link)
if after != before or "password" in sent:
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
target_type="download_link",
target_id=link_id,
before=before,
after=after,
)
linksmod.update(
db,
user,
link,
allow_download=body.allow_download,
expires_days=body.expires_days,
password=body.password,
sent=frozenset(body.model_fields_set),
)
db.commit()
return linksmod.owner_view(link)
@@ -911,16 +730,7 @@ def update_link(
def revoke_link(
link_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
link = _own_link(db, user, link_id)
db.delete(link)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
target_type="download_link",
target_id=link_id,
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
)
linksmod.revoke(db, user, _own_link(db, user, link_id))
db.commit()
return {"ok": True, "deleted": link_id}
@@ -932,20 +742,9 @@ def revoke_link(
def admin_list_downloads(
admin: User = Depends(admin_user), db: Session = Depends(get_db)
) -> list[dict]:
jobs = (
db.execute(
select(DownloadJob)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
)
.scalars()
.all()
)
assets = assets_for(db, jobs)
emails = {
u.id: u.email
for u in db.execute(select(User).where(User.id.in_({j.user_id for j in jobs}))).scalars()
}
jobs = queries.recent_jobs(db)
assets = queries.assets_for(db, jobs)
emails = emails_of(db, {j.user_id for j in jobs})
out = []
for j in jobs:
data = serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None)
@@ -957,27 +756,11 @@ def admin_list_downloads(
@admin_router.get("/storage")
def admin_storage(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
# Every file on disk (`quota.ON_DISK`): ready ones, and uploads waiting for or failed in their
# conversion — the same disk the footprint and the GC cap count.
on_disk = db.execute(
select(func.count(), func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).one()
total_assets = db.execute(select(func.count()).select_from(MediaAsset)).scalar()
files_on_disk, total_bytes, total_assets = queries.storage_totals(db)
# EVERY user, not only those holding a file: a quota is set before anyone downloads.
emails: dict[int, str] = {
uid: email
for uid, email in db.execute(select(User.id, User.email).order_by(User.email)).tuples()
}
emails = emails_of(db)
uids = list(emails)
# What each one's Files lists: their finished files.
files: dict[int, int] = {
uid: n
for uid, n in db.execute(
select(DownloadJob.user_id, func.count())
.where(DownloadJob.status == JobStatus.DONE)
.group_by(DownloadJob.user_id)
).tuples()
}
files = queries.done_files_per_user(db)
def row(uid: int) -> dict:
lim = quota.resolve(db, uid)
@@ -991,8 +774,8 @@ def admin_storage(admin: User = Depends(admin_user), db: Session = Depends(get_d
}
return {
"files_on_disk": on_disk[0],
"total_bytes": int(on_disk[1]),
"files_on_disk": files_on_disk,
"total_bytes": total_bytes,
"total_assets": total_assets,
"total_cap_bytes": sysconfig.get_int(db, "download_total_max_bytes"),
"per_user": [row(uid) for uid in uids],
@@ -1004,10 +787,9 @@ def admin_get_quota(
user_id: int, admin: User = Depends(admin_user), db: Session = Depends(get_db)
) -> dict:
lim = quota.resolve(db, user_id)
row = db.get(DownloadQuota, user_id)
return {
"user_id": user_id,
"custom": row is not None,
"custom": queries.custom_quota(db, user_id) is not None,
"max_bytes": lim.max_bytes,
"max_concurrent": lim.max_concurrent,
"max_jobs": lim.max_jobs,
@@ -1032,25 +814,14 @@ def admin_set_quota(
db: Session = Depends(get_db),
) -> dict:
get_or_404(db, User, user_id, what="user")
cur = quota.resolve(db, user_id)
row = db.get(DownloadQuota, user_id)
if row is None:
row = DownloadQuota(
user_id=user_id,
max_bytes=cur.max_bytes,
max_concurrent=cur.max_concurrent,
max_jobs=cur.max_jobs,
unlimited=cur.unlimited,
)
db.add(row)
if body.max_bytes is not None:
row.max_bytes = max(0, body.max_bytes)
if body.max_concurrent is not None:
row.max_concurrent = max(1, body.max_concurrent)
if body.max_jobs is not None:
row.max_jobs = max(1, body.max_jobs)
if body.unlimited is not None:
row.unlimited = body.unlimited
quota.set_custom(
db,
user_id,
max_bytes=body.max_bytes,
max_concurrent=body.max_concurrent,
max_jobs=body.max_jobs,
unlimited=body.unlimited,
)
db.commit()
return admin_get_quota(user_id, admin, db)
@@ -1059,8 +830,6 @@ def admin_set_quota(
def admin_reset_quota(
user_id: int, admin: User = Depends(admin_user), db: Session = Depends(get_db)
) -> dict:
row = db.get(DownloadQuota, user_id)
if row is not None:
db.delete(row)
if quota.reset_custom(db, user_id):
db.commit()
return admin_get_quota(user_id, admin, db)
+91 -244
View File
@@ -5,8 +5,10 @@ the per-user layer (name, folder, kept, expiry, shares and links all hang off it
bytes. Folders are rows (`downloads.folders`), so a rename or a move never touches the disk.
In-flight jobs are not here: they are the download tray's (`routes.downloads`). The helpers above
the routes are shared with that module — serialization, the ownership locks, the upload spool —
because a download and a library file are one row at two moments.
the routes are shared with that module — serialization, the ownership checks, the upload spool —
because a download and a library file are one row at two moments. What is read comes from
`downloads.queries`, what is written goes through Capture's service (`service`, `folders`); this
module validates, answers in HTTP and commits.
A machine with an API token (`app.apitokens`) may use the routes that take `files_caller`: list the
folders and files, fetch a file and its hash, upload, delete — each only in a folder its token was
@@ -19,51 +21,37 @@ import hashlib
import os
import tempfile
import threading
from collections.abc import AsyncIterator, Sequence
from collections.abc import AsyncIterator
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import BinaryIO, Literal
from typing import TYPE_CHECKING, BinaryIO, Literal
from urllib.parse import unquote, urlparse
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import FileResponse, Response
from pydantic import BaseModel, Field
from sqlalchemy import func, or_, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import audit, sysconfig
from app.apitokens import FilesCaller, Right, files_caller
from app.audit import AuditAction
from app.auth import current_user
from app.collation import collated
from app.config import settings
from app.db import get_db, release_read_connection
from app.downloads import folders, preview, quota, service, storage
from app.downloads import folders, preview, queries, quota, service, storage
from app.downloads import upload as uploadmod
from app.downloads.status import JobStatus
from app.mailtext import lang_for_user
from app.models import (
DISPLAY_TEXT_MAX,
DownloadJob,
DownloadLink,
DownloadShare,
FileFolder,
LibraryState,
MediaAsset,
User,
display_text,
)
from app.models import DISPLAY_TEXT_MAX, User, display_text
from app.refusal import refused
from app.routes._common import capped_stream_to, not_found, owned_or_404
from app.utils import iso, like_escape
from app.routes._common import capped_stream_to, not_found
from app.utils import iso
if TYPE_CHECKING:
from app.downloads.models import DownloadJob, FileFolder, MediaAsset
router = APIRouter(prefix="/api/files", tags=["files"])
# Cap the listing so a heavy library can't return an unbounded result set (the old Download Center's
# own cap; far beyond any real library).
MAX_LIST = 500
# The longest a folder may keep a file, in days — ten years is "forever" in all but name.
_MAX_RETENTION_DAYS = 3650
_HASH_CHUNK = 1 << 20
@@ -75,7 +63,7 @@ _HASHING = threading.BoundedSemaphore(_HASHERS)
# --- shared with routes.downloads -------------------------------------------------------
def serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
def serialize(job: "DownloadJob", asset: "MediaAsset | None") -> dict:
ready = asset is not None and asset.status == "ready" and bool(asset.rel_path)
return {
"id": job.id,
@@ -135,58 +123,18 @@ def serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
}
def assets_for(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, MediaAsset]:
ids = {j.asset_id for j in jobs if j.asset_id}
if not ids:
return {}
rows = db.execute(select(MediaAsset).where(MediaAsset.id.in_(ids))).scalars()
return {a.id: a for a in rows}
def positions_for(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, int]:
"""`user`'s resume point in each of `jobs` that has one (the library player's `library_states`)."""
ids = [j.id for j in jobs]
if not ids:
return {}
rows = db.execute(
select(LibraryState.job_id, LibraryState.position_seconds).where(
LibraryState.user_id == user.id, LibraryState.job_id.in_(ids)
)
).all()
return {job_id: pos for job_id, pos in rows}
def serialize_job(db: Session, job: DownloadJob) -> dict:
def serialize_job(db: Session, job: "DownloadJob") -> dict:
"""Resolve a job's cache asset (if any) and serialize the pair — the resolve-then-serialize step
every single-job handler shares."""
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
return serialize(job, asset)
return serialize(job, queries.asset_of(db, job))
def own_job_for_update(db: Session, user: User, job_id: int) -> DownloadJob:
"""The ownership check (`owned_or_404`) for a transition that moves its asset's ref count (resume, cancel, delete, redownload)
or writes the status those decide on (pause). The job is re-read under a row lock, so two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`service.finalize_if_running`), so the two can never deadlock."""
while True:
asset_id = owned_or_404(db, DownloadJob, job_id, user, what="download").asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
raise not_found("download")
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
def own_job_for_update(db: Session, user: User, job_id: int) -> "DownloadJob":
"""One of the user's jobs under its row lock (`service.lock_own_job`), or a 404."""
job = service.lock_own_job(db, user, job_id)
if job is None:
raise not_found("download")
return job
def quota_detail(e: quota.QuotaExceeded) -> dict:
@@ -322,30 +270,32 @@ async def spool_upload(
tmp_path.unlink(missing_ok=True)
def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob:
"""A file the caller may read: their own, or one shared with them."""
job = db.get(DownloadJob, job_id)
def accessible_job(db: Session, user: User, job_id: int) -> "DownloadJob":
"""A file the caller may read (`queries.accessible_job`: their own, or shared with them)."""
job = queries.accessible_job(db, user, job_id)
if job is None:
raise not_found("download")
if job.user_id == user.id:
return job
shared = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == user.id
)
).first()
if shared is None:
raise not_found("download")
return job
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob:
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
own in a folder it may read."""
if caller.grants is None:
return accessible_job(db, caller.user, job_id)
job = db.get(DownloadJob, job_id)
if job is None or job.user_id != caller.user.id or not caller.may(job.folder_id, Right.READ):
def own_job(db: Session, user: User, job_id: int) -> "DownloadJob":
job = queries.own_job(db, user, job_id)
if job is None:
raise not_found("download")
return job
def own_folder(db: Session, user: User, folder_id: int) -> "FileFolder":
folder = queries.own_folder(db, user, folder_id)
if folder is None:
raise not_found("folder")
return folder
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> "DownloadJob":
"""A file `caller` may read (`queries.readable_job`), or a 404."""
job = queries.readable_job(db, caller, job_id)
if job is None:
raise not_found("download")
return job
@@ -353,7 +303,7 @@ def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob:
# --- folders -------------------------------------------------------------------------------
def _serialize_folder(folder: FileFolder, count: int, size: int, linked: bool = False) -> dict:
def _serialize_folder(folder: "FileFolder", count: int, size: int, linked: bool = False) -> dict:
return {
"id": folder.id,
"kind": folder.kind,
@@ -369,18 +319,6 @@ def _serialize_folder(folder: FileFolder, count: int, size: int, linked: bool =
}
def _linked(db: Session, folder_ids: list[int]) -> set[int]:
"""Which of these folders a live public link opens (one not past its expiry)."""
now = datetime.now(timezone.utc)
rows = db.execute(
select(DownloadLink.folder_id).where(
DownloadLink.folder_id.in_(folder_ids),
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
)
).scalars()
return {folder_id for folder_id in rows if folder_id is not None}
@router.get("/folders")
def list_folders(
caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
@@ -388,23 +326,9 @@ def list_folders(
"""The caller's folders with how many files each holds and their size (a token: the ones it may
list). A source folder its files have all left stays (count 0): the client decides whether to
show it."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
stmt = select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
).where(FileFolder.user_id == caller.user.id)
listable = caller.folders_with(Right.LIST)
if listable is not None:
stmt = stmt.where(FileFolder.id.in_(listable))
rows = db.execute(
stmt.outerjoin(DownloadJob, done)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.group_by(FileFolder.id)
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
).all()
linked = _linked(db, [f.id for f, _, _ in rows])
return [_serialize_folder(f, count, int(size), f.id in linked) for f, count, size in rows]
rows = queries.folders_with_totals(db, caller.user, caller.folders_with(Right.LIST))
linked = queries.linked_folders(db, [f.id for f, _, _ in rows])
return [_serialize_folder(f, count, size, f.id in linked) for f, count, size in rows]
class FolderIn(BaseModel):
@@ -422,13 +346,10 @@ def _folder_name(raw: str) -> str:
def create_folder(
body: FolderIn, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
folder = FileFolder(user_id=user.id, kind=folders.FolderKind.OWN, name=_folder_name(body.name))
try:
with db.begin_nested():
db.add(folder)
db.flush()
except IntegrityError as e: # the unique (user, lower(name)): one folder per name
raise refused(409, "file_folder_name_taken") from e
folder = folders.create_own(db, user.id, _folder_name(body.name))
except folders.FolderError as e:
raise refused(409, e.reason) from e
audit.record(
db, user.id, AuditAction.FILE_FOLDER_CREATE, target_type="file_folder", target_id=folder.id
)
@@ -451,17 +372,13 @@ def update_folder(
) -> dict:
"""Rename one of the caller's own folders (a source folder's name is its origin's), and/or set
how long any of their folders keeps a file (`folders.set_retention`)."""
folder = owned_or_404(db, FileFolder, folder_id, user, what="folder")
folder = own_folder(db, user, folder_id)
sent = body.model_fields_set
if "name" in sent:
if folder.kind != folders.FolderKind.OWN:
raise refused(409, "file_folder_not_own")
folder.name = _folder_name(body.name or "")
try:
with db.begin_nested():
db.flush()
except IntegrityError as e:
raise refused(409, "file_folder_name_taken") from e
folders.rename(db, folder, _folder_name(body.name or ""))
except folders.FolderError as e:
raise refused(409, e.reason) from e
audit.record(
db,
user.id,
@@ -486,18 +403,9 @@ def update_folder(
after={"retention_days": body.retention_days},
)
db.commit()
count, size = _folder_totals(db, folder.id)
return _serialize_folder(folder, count, size, folder.id in _linked(db, [folder.id]))
def _folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
count, size = db.execute(
select(func.count(DownloadJob.id), func.coalesce(func.sum(MediaAsset.size_bytes), 0))
.select_from(DownloadJob)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.folder_id == folder_id, DownloadJob.status == JobStatus.DONE)
).one()
return count, int(size)
count, size = queries.folder_totals(db, folder.id)
linked = folder.id in queries.linked_folders(db, [folder.id])
return _serialize_folder(folder, count, size, linked)
@router.delete("/folders/{folder_id}")
@@ -505,7 +413,7 @@ def delete_folder(
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Delete one of the caller's own folders; its files go back to their source folders."""
folder = owned_or_404(db, FileFolder, folder_id, user, what="folder")
folder = own_folder(db, user, folder_id)
try:
moved = folders.delete_own(db, folder)
except folders.FolderError as e:
@@ -547,13 +455,7 @@ def remove_shared_with_me(
"""Remove a file that was shared WITH this user from their 'Shared with me' list. Deletes
only the recipient's share grant — the owner's job and the physical file are untouched (this is
a per-user dismissal, not a delete)."""
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == user.id
)
).scalar_one_or_none()
if row is not None:
db.delete(row)
if service.unshare(db, job_id, user):
db.commit()
return {"ok": True, "removed": job_id}
@@ -582,46 +484,22 @@ def list_files(
raise refused(403, "api_token_scope")
if folder_id is not None and folder_id not in listable:
raise not_found("folder")
stmt = (
select(DownloadJob)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.status == JobStatus.DONE)
)
shared = scope == "shared"
if shared:
stmt = stmt.join(DownloadShare, DownloadShare.job_id == DownloadJob.id).where(
DownloadShare.to_user_id == user.id
)
else:
stmt = stmt.where(DownloadJob.user_id == user.id)
if folder_id is not None:
stmt = stmt.where(DownloadJob.folder_id == folder_id)
elif listable is not None:
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
term = q.strip()
if term:
like = f"%{like_escape(term)}%"
stmt = stmt.where(
or_(
name.ilike(like),
func.coalesce(DownloadJob.display_uploader, MediaAsset.uploader, "").ilike(like),
)
)
# A file is added to its owner's list when it is done, not when it was asked for (a download that
# failed for days and then went through is new); one shared with the caller, when it was shared.
key = {
"added": DownloadShare.created_at if shared else DownloadJob.ready_at,
"name": collated(name, lang_for_user(user)),
"size": MediaAsset.size_bytes,
}[sort]
first = key.asc() if order == "asc" else key.desc()
tie = DownloadJob.id.asc() if order == "asc" else DownloadJob.id.desc()
jobs = db.execute(stmt.order_by(first.nulls_last(), tie).limit(MAX_LIST)).scalars().all()
assets = assets_for(db, jobs)
positions = positions_for(db, user, jobs)
shared_at = _shared_at(db, user, jobs) if shared else {}
sources = source_keys(db, jobs)
jobs = queries.library(
db,
user,
folder_id=folder_id,
listable=listable,
shared=shared,
term=q.strip(),
sort=sort,
order=order,
lang=lang_for_user(user),
)
assets = queries.assets_for(db, jobs)
positions = queries.positions_for(db, user, jobs)
shared_at = queries.shared_at(db, user, jobs) if shared else {}
sources = queries.source_keys(db, jobs)
return [
{
**serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None),
@@ -634,36 +512,6 @@ def list_files(
]
def source_keys(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, str]:
"""Where each file came from (its source folder's key), for the list's Source column, the
"back to its source folder" move and the download tray's site: a file in a source folder has its folder's key; one moved into
an own folder (or someone else's, shared) is traced back by `folders.source_key_for`."""
ids = {j.folder_id for j in jobs if j.folder_id is not None}
by_id = (
{f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
if ids
else {}
)
out: dict[int, str] = {}
for j in jobs:
folder = by_id.get(j.folder_id) if j.folder_id is not None else None
if folder is not None and folder.kind == folders.FolderKind.SOURCE and folder.source_key:
out[j.id] = folder.source_key
else:
out[j.id] = folders.source_key_for(db, j)
return out
def _shared_at(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, datetime]:
rows = db.execute(
select(DownloadShare.job_id, DownloadShare.created_at).where(
DownloadShare.to_user_id == user.id,
DownloadShare.job_id.in_([j.id for j in jobs]),
)
).all()
return {job_id: at for job_id, at in rows}
_MAX_EXTRA_LINKS = 8
@@ -711,18 +559,20 @@ def update_file(
"""Rename a file and its channel, set its reference links (all display-only: the file on disk
keeps its own name), move it to another folder (`folders.move`), or keep it until deleted /
hand it back to its folder's retention."""
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
job = own_job(db, user, job_id)
sent = body.model_fields_set
display: dict[str, object] = {}
if "name" in sent:
job.display_name = display_text(body.name)
display["display_name"] = display_text(body.name)
if "uploader" in sent:
job.display_uploader = display_text(body.uploader)
display["display_uploader"] = display_text(body.uploader)
if "uploader_url" in sent:
job.display_uploader_url = _clean_url(body.uploader_url)
display["display_uploader_url"] = _clean_url(body.uploader_url)
if "extra_links" in sent:
raw = body.extra_links or []
cleaned = [u for u in (_clean_url(x) for x in raw) if u][:_MAX_EXTRA_LINKS]
job.extra_links = cleaned or None
display["extra_links"] = cleaned or None
service.set_display(job, **display)
if "folder_id" in sent:
if body.folder_id is None:
raise refused(400, "file_folder_gone")
@@ -768,7 +618,8 @@ def delete_file(
target_id=job_id,
summary=caller.via,
)
service.delete(db, job) # commits, the audit row with it
service.delete(db, job)
db.commit() # the audit row with it
return {"ok": True, "deleted": job_id}
@@ -787,7 +638,7 @@ def upload_file(
return serialize_job(db, ingest_spooled(db, upload))
def ingest_spooled(db: Session, upload: SpooledUpload) -> DownloadJob:
def ingest_spooled(db: Session, upload: SpooledUpload) -> "DownloadJob":
"""The spooled upload into the library (`upload.ingest_upload`, which commits its own rows),
its refusals answered as HTTP."""
if upload.size == 0:
@@ -811,8 +662,8 @@ def ingest_spooled(db: Session, upload: SpooledUpload) -> DownloadJob:
return job
def _ready_file(db: Session, job: DownloadJob) -> tuple[MediaAsset, Path]:
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
def _ready_file(db: Session, job: "DownloadJob") -> tuple["MediaAsset", Path]:
asset = queries.asset_of(db, job)
if asset is None or asset.status != "ready" or not asset.rel_path:
raise refused(409, "download_not_ready")
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
@@ -821,7 +672,7 @@ def _ready_file(db: Session, job: DownloadJob) -> tuple[MediaAsset, Path]:
return asset, path
def serve_file(db: Session, job: DownloadJob) -> FileResponse:
def serve_file(db: Session, job: "DownloadJob") -> FileResponse:
"""The file of `job` (the caller may read it), range-aware, named by the user's display name.
The served Content-Type comes from the ffprobe-verified media_kind + a container allow-list
@@ -839,7 +690,7 @@ def serve_file(db: Session, job: DownloadJob) -> FileResponse:
if asset.sha256:
digest = base64.b64encode(bytes.fromhex(asset.sha256)).decode()
headers["Repr-Digest"] = f"sha-256=:{digest}:"
asset.last_access_at = datetime.now(timezone.utc) # the handler commits it
service.touch(asset) # the handler commits it
media_type = storage.served_media_type(asset.media_kind, asset.container)
if media_type is None:
return FileResponse(
@@ -894,13 +745,13 @@ def preview_file(
text = preview.text_utf8(head)
if text is None:
raise refused(415, "file_no_preview")
asset.last_access_at = datetime.now(timezone.utc)
service.touch(asset)
db.commit()
return Response(text, media_type=preview.media_type(kind, asset.container), headers=headers)
filename = storage.download_filename(
job.display_name or asset.title or asset.source_ref, asset.container, path
)
asset.last_access_at = datetime.now(timezone.utc)
service.touch(asset)
db.commit()
return FileResponse(
path,
@@ -936,10 +787,6 @@ def get_sha256(
finally:
_HASHING.release()
digest = h.hexdigest()
db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id, MediaAsset.sha256.is_(None))
.values(sha256=digest)
)
service.store_sha256(db, asset_id, digest)
db.commit()
return {"sha256": digest}
+27 -32
View File
@@ -6,23 +6,26 @@ the app or any account data. Password-protected links exchange the password once
a short-lived signed grant that the media URL carries (a `<video src>` can't send a header).
"""
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends, Request
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import select, update
from sqlalchemy.orm import Session
from app.config import settings
from app.db import get_db
from app.downloads import links as linksmod
from app.downloads import storage
from app.downloads import queries, storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset, User
from app.ratelimit import RateLimiter, client_key
from app.security import verify_password
from app.refusal import refused
from app.routes._common import not_found
if TYPE_CHECKING:
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
router = APIRouter(prefix="/api/public", tags=["public"])
# Slow password guessing on a link (the token itself is 192-bit, so it needs no throttle). Counted per
@@ -37,26 +40,25 @@ _unlock_limit = RateLimiter(max_events=10, window_seconds=300)
_unlock_failures = RateLimiter(max_events=50, window_seconds=300)
def _link_or_404(db: Session, token: str) -> DownloadLink:
def _link_or_404(db: Session, token: str) -> "DownloadLink":
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
raise refused(404, "link_gone")
return link
def _being_made(db: Session, job: DownloadJob) -> bool:
def _being_made(db: Session, job: "DownloadJob") -> bool:
"""The job is running, or queued for a worker that will take it: the worker leaves a suspended
owner's queued jobs where they are (`worker._claim_job`), as long as the suspension lasts."""
if job.status == JobStatus.RUNNING:
return True
if job.status != JobStatus.QUEUED:
return False
owner = db.get(User, job.user_id)
return owner is not None and not owner.is_suspended
return not queries.owner_suspended(db, job)
def _ready_job(db: Session, job: DownloadJob | None) -> tuple[DownloadJob, MediaAsset]:
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
def _ready_job(db: Session, job: "DownloadJob | None") -> tuple["DownloadJob", "MediaAsset"]:
asset = queries.asset_of(db, job)
if (
job is not None
and asset is not None
@@ -78,25 +80,24 @@ def _ready_job(db: Session, job: DownloadJob | None) -> tuple[DownloadJob, Media
return job, asset
def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAsset]:
def _ready_target(db: Session, link: "DownloadLink") -> tuple["DownloadJob", "MediaAsset"]:
"""A file link's file. A folder link names no one file: its files are asked for by id."""
if link.job_id is None:
raise refused(404, "video_gone")
return _ready_job(db, db.get(DownloadJob, link.job_id))
return _ready_job(db, queries.job(db, link.job_id))
def _link_folder(db: Session, link: DownloadLink) -> FileFolder:
folder = db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
def _link_folder(db: Session, link: "DownloadLink") -> "FileFolder":
folder = queries.link_folder(db, link)
if folder is None:
raise refused(404, "link_gone")
return folder
def _folder_job(db: Session, link: DownloadLink, job_id: int) -> tuple[DownloadJob, MediaAsset]:
folder = _link_folder(db, link)
job = db.execute(
select(DownloadJob).where(DownloadJob.id == job_id, *linksmod.in_folder(folder))
).scalar_one_or_none()
def _folder_job(
db: Session, link: "DownloadLink", job_id: int
) -> tuple["DownloadJob", "MediaAsset"]:
job = queries.folder_job(db, _link_folder(db, link), job_id)
if job is None:
raise refused(404, "video_gone")
return _ready_job(db, job)
@@ -113,7 +114,7 @@ def _file_url(token: str, grant: str | None = None, job_id: int | None = None) -
def _poster_url(
token: str, asset: MediaAsset, grant: str | None = None, job_id: int | None = None
token: str, asset: "MediaAsset", grant: str | None = None, job_id: int | None = None
) -> str | None:
if not asset.poster_path:
return None
@@ -121,7 +122,7 @@ def _poster_url(
return f"{url}?g={grant}" if grant else url
def _folder_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
def _folder_meta(db: Session, link: "DownloadLink", token: str, grant: str | None) -> dict:
"""A folder link's page: the folder's name and each file's own watch metadata
(`linksmod.folder_files`: ready, on disk, newest first)."""
folder = _link_folder(db, link)
@@ -147,7 +148,7 @@ def _folder_meta(db: Session, link: DownloadLink, token: str, grant: str | None)
}
def _file_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
def _file_meta(db: Session, link: "DownloadLink", token: str, grant: str | None) -> dict:
job, asset = _ready_target(db, link)
return {
"needs_password": False,
@@ -158,20 +159,14 @@ def _file_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -
}
def _opened(db: Session, link: DownloadLink, token: str, grant: str | None = None) -> dict:
def _opened(db: Session, link: "DownloadLink", token: str, grant: str | None = None) -> dict:
"""The page an open link shows, counted as one view (the handler commits the count)."""
meta = (
_folder_meta(db, link, token, grant)
if link.folder_id is not None
else _file_meta(db, link, token, grant)
)
# In SQL, not `+= 1` on the loaded row: two opens at once would both write the same count.
db.execute(
update(DownloadLink)
.where(DownloadLink.id == link.id)
.values(view_count=DownloadLink.view_count + 1)
.execution_options(synchronize_session=False)
)
linksmod.count_view(db, link)
return meta
@@ -217,13 +212,13 @@ def watch_unlock(
return {"ok": True, **meta}
def _granted(link: DownloadLink, token: str, g: str | None) -> None:
def _granted(link: "DownloadLink", token: str, g: str | None) -> None:
"""A password link's files want the grant its unlock handed out."""
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
raise refused(403, "share_password_required")
def _serve(link: DownloadLink, asset: MediaAsset) -> FileResponse:
def _serve(link: "DownloadLink", asset: "MediaAsset") -> FileResponse:
"""Range-aware stream of a shared file. Inline by default (plays in the page); an
`allow_download` link serves it as an attachment."""
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
@@ -252,7 +247,7 @@ def _serve(link: DownloadLink, asset: MediaAsset) -> FileResponse:
)
def _poster(asset: MediaAsset) -> FileResponse:
def _poster(asset: "MediaAsset") -> FileResponse:
if not asset.poster_path:
raise not_found("poster")
path = storage.safe_abs_path(settings.files_root, asset.poster_path)
+20 -9
View File
@@ -6,7 +6,7 @@ The aggregate daily budget lives in ApiQuotaUsage; here we read the per-spend au
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, Query
from sqlalchemy import func, select
from sqlalchemy.orm import Session
@@ -14,6 +14,8 @@ from app import quota
from app.auth import admin_user, current_user
from app.db import get_db
from app.models import QuotaEvent, User
from app.routes import _account as account
from app.routes._keyset import utc
router = APIRouter(prefix="/api/quota", tags=["quota"])
@@ -45,23 +47,32 @@ def my_usage(user: User = Depends(current_user), db: Session = Depends(get_db))
@router.get("/admin")
def admin_usage(
days: int = 30,
since: datetime | None = None,
until: datetime | None = None,
user: str | None = Query(default=None, pattern=account.PATTERN),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
days = max(1, min(days, 365))
since = datetime.now(timezone.utc) - timedelta(days=days)
"""Spend per account (and per action) and per day, between `since` (default: 30 days back) and
`until` (default: now), optionally for one account (`none` = the background)."""
now = datetime.now(timezone.utc)
lo = utc(since) or now - timedelta(days=30)
window = [QuotaEvent.created_at >= lo]
if until is not None:
window.append(QuotaEvent.created_at < utc(until))
if user is not None:
window.append(account.condition(QuotaEvent.user_id, user))
totals = db.execute(
select(QuotaEvent.user_id, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(QuotaEvent.user_id)
).all()
per_action: dict[int | None, dict[str, int]] = {}
for uid, action, units in db.execute(
select(QuotaEvent.user_id, QuotaEvent.action, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(QuotaEvent.user_id, QuotaEvent.action)
).all():
per_action.setdefault(uid, {})[action] = int(units)
@@ -84,16 +95,16 @@ def admin_usage(
]
rows.sort(key=lambda r: r["total"], reverse=True)
# Instance-wide daily series in Pacific days (matches the budget reset).
# The daily series in Pacific days (matches the budget reset), of the one account when filtered.
day_expr = func.date(func.timezone("America/Los_Angeles", QuotaEvent.created_at))
daily = [
{"day": str(day), "total": int(units)}
for day, units in db.execute(
select(day_expr, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(day_expr)
.order_by(day_expr)
).all()
]
return {"range_days": days, "rows": rows, "daily": daily}
return {"rows": rows, "daily": daily}
+156
View File
@@ -0,0 +1,156 @@
"""Admin background-run history: read `job_runs`, a page at a time.
Admin-only and read-only — rows leave the table through the retention job
(`job_run_retention_days`). Filtered on the server and paged by `_keyset`, like the server log and
the audit log beside it on the activity page. A row names what a reader needs without a second
request: the channel a user's ad-hoc run worked on, and how many log lines the run left (only
WARNING and up are stored, so most runs have none — the count says so before the click).
"""
from dataclasses import dataclass
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app import scheduler
from app.auth import admin_user
from app.db import get_db
from app.models import Channel, JobRun, LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.runstatus import RunStatus
from app.utils import iso, like_escape
router = APIRouter(prefix="/api/admin/runs", tags=["admin"])
Facet = Literal["job", "status"]
def _serialize(run: JobRun, titles: dict[str, str], lines: dict[int, int]) -> dict:
return {
"id": run.id,
"job": run.job,
"trigger": run.trigger,
"actor_id": run.actor_id,
"target": run.target,
"target_title": titles.get(run.target) if run.target is not None else None,
"status": run.status,
"started_at": iso(run.started_at),
"finished_at": iso(run.finished_at),
"summary": run.summary,
"result": run.result,
"error": run.error,
"failure": run.failure,
"quota_units": run.quota_units,
"log_lines": lines.get(run.id, 0),
}
@dataclass(frozen=True)
class RunFilters:
job: str | None
status: RunStatus | None
actor: str | None
q: str | None
since: datetime | None
until: datetime | None
def conditions(self, skip: Facet | None = None) -> list:
"""The WHERE conditions, ANDed. `skip` leaves one menu's own filter out, so that menu counts
its values under every other filter (the way back from a pick)."""
out = []
if self.job is not None and skip != "job":
out.append(JobRun.job == self.job)
if self.status is not None and skip != "status":
out.append(JobRun.status == self.status)
if self.actor is not None:
out.append(account.condition(JobRun.actor_id, self.actor))
if self.since is not None:
out.append(JobRun.started_at >= utc(self.since))
if self.until is not None:
out.append(JobRun.started_at < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
or_(
JobRun.job.ilike(like),
JobRun.target.ilike(like),
JobRun.summary.ilike(like),
JobRun.error.ilike(like),
)
)
return out
def run_filters(
job: str | None = Query(default=None, max_length=40, pattern=r"^[a-z][a-z0-9_]*$"),
status: RunStatus | None = None,
user: str | None = Query(default=None, pattern=account.PATTERN),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> RunFilters:
return RunFilters(job, status, user, q, since, until)
@router.get("")
def list_runs(
filters: RunFilters = Depends(run_filters),
before: str | None = Query(default=None, max_length=200),
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""One page of runs, newest first, matching every filter given; `before` continues from a
previous page's `next_cursor`."""
query = select(JobRun).where(*filters.conditions())
if before is not None:
query = query.where(older_than(JobRun.started_at, JobRun.id, before))
order = (JobRun.started_at.desc(), JobRun.id.desc())
runs, cursor = page(
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
limit,
lambda r: r.started_at,
)
channels = {r.target for r in runs if r.job in scheduler.AD_HOC_JOBS and r.target}
titles: dict[str, str] = {}
if channels:
rows = db.execute(select(Channel.id, Channel.title).where(Channel.id.in_(channels)))
titles = {cid: title for cid, title in rows.tuples() if title is not None}
lines: dict[int, int] = {}
if runs:
counted = db.execute(
select(LogEvent.run_id, func.count())
.where(LogEvent.run_id.in_([r.id for r in runs]))
.group_by(LogEvent.run_id)
)
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
return {
"items": [_serialize(r, titles, lines) for r in runs],
"accounts": accounts(db, {r.actor_id for r in runs if r.actor_id is not None}),
"next_cursor": cursor,
}
@router.get("/facets")
def run_facets(
filters: RunFilters = Depends(run_filters),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""What the job menu and the status chips offer, each counted under every filter but its own."""
def counts(col, skip: Facet) -> list[dict]:
n = func.count().label("n")
stmt = select(col, n).where(*filters.conditions(skip)).group_by(col).order_by(n.desc(), col)
return [{"value": v, "count": c} for v, c in db.execute(stmt).all()]
total = select(func.count()).select_from(JobRun).where(*filters.conditions("status"))
return {
"jobs": counts(JobRun.job, "job"),
"statuses": counts(JobRun.status, "status"),
"total": db.scalar(total) or 0,
}
+7 -5
View File
@@ -17,8 +17,9 @@ from app.scheduler import (
JOB_INTERVALS,
MAX_INTERVAL,
MIN_INTERVAL,
apply_interval,
reschedule,
scheduler_snapshot,
stage_interval,
trigger_all,
trigger_job,
)
@@ -42,14 +43,14 @@ def set_job_interval(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Change how often a scheduler job runs (minutes). Persisted as an override and applied
to the live scheduler immediately."""
"""Change how often a scheduler job runs (minutes). Stored as an override together with its
audit row, then applied to the live scheduler."""
if job_id not in JOB_INTERVALS:
raise not_found("job")
minutes = body.interval_minutes
if not (MIN_INTERVAL <= minutes <= MAX_INTERVAL):
raise refused(400, "invalid_interval", min=MIN_INTERVAL, max=MAX_INTERVAL)
applied = apply_interval(db, job_id, minutes)
applied = stage_interval(db, job_id, minutes)
audit.record(
db,
admin.id,
@@ -60,6 +61,7 @@ def set_job_interval(
after={"interval_minutes": applied},
)
db.commit()
reschedule(job_id, applied)
return {"ok": True, "id": job_id, "interval_minutes": applied}
@@ -117,7 +119,7 @@ def set_maintenance_settings(
@router.get("")
def get_scheduler(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
snap = scheduler_snapshot()
snap = scheduler_snapshot(db)
# Work still queued (shared catalog, so these are instance-wide, not per-user).
deep_requested = (
+3 -3
View File
@@ -91,12 +91,12 @@ def setup_admin(body: SetupAdminIn, db: Session = Depends(require_setup)) -> dic
@router.post("/config")
def setup_config(settings_map: dict[str, Any], db: Session = Depends(require_setup)) -> dict:
"""Persist provided DB-backed settings (Google creds, SMTP, quota…). Keys must be in the
sysconfig registry; empty values are skipped (left at the default). Genuinely dynamic
"""Persist provided DB-backed settings (Google creds, SMTP, quota…). Keys must be the
Configuration page's; empty values are skipped (left at the default). Genuinely dynamic
key→value map, so it stays a typed dict rather than a fixed model."""
saved: list[str] = []
for key, value in settings_map.items():
if sysconfig.spec(key) is None:
if sysconfig.config_page_spec(key) is None:
raise refused(400, "unknown_setting", key=key)
if value is None or value == "":
continue
+12 -8
View File
@@ -21,8 +21,8 @@ from app.apitokens import GRANT_COLUMNS, Right
from app.audit import AuditAction
from app.auth import current_user
from app.db import get_db
from app.downloads import folders
from app.models import ApiToken, ApiTokenGrant, FileFolder, User
from app.downloads import folders, queries
from app.models import ApiToken, ApiTokenGrant, User
from app.refusal import refused
from app.routes._common import owned_or_404
from app.security import hash_token
@@ -50,12 +50,16 @@ class TokenIn(BaseModel):
def _serialize(db: Session, token: ApiToken) -> dict:
rows = db.execute(
select(ApiTokenGrant, FileFolder)
.join(FileFolder, FileFolder.id == ApiTokenGrant.folder_id)
.where(ApiTokenGrant.token_id == token.id)
.order_by(FileFolder.id)
).all()
grants = list(
db.execute(
select(ApiTokenGrant)
.where(ApiTokenGrant.token_id == token.id)
.order_by(ApiTokenGrant.folder_id)
).scalars()
)
# The folders are Capture's: read through its queries, not its table.
by_id = queries.folders_by_id(db, {g.folder_id for g in grants})
rows = [(g, by_id[g.folder_id]) for g in grants if g.folder_id in by_id]
now = datetime.now(timezone.utc)
return {
"id": token.id,
+37
View File
@@ -0,0 +1,37 @@
"""The words a background run's row is written with — the `job_runs` table's status and trigger.
One source for the literals: the scheduler writes them, the startup sweep rewrites `running`, the
model's CHECK constraints are built from these sets, and the dashboard reads them back (the frontend's
copies are held to them by `backendIds.contract.test.ts`). The migrated database's CHECKs are frozen
in its migration: a test holds them equal to these sets, so a new word fails until a migration adds
it (`test_runstatus.py`). The download pipeline's `cancelled`/`canceled` drift (`downloads/status.py`)
is why a status is never typed as a bare string.
"""
from enum import StrEnum
class RunStatus(StrEnum):
RUNNING = "running"
OK = "ok"
ERROR = "error"
# Sync was paused: the run was due and deliberately did nothing.
SKIPPED = "skipped"
# The process stopped while the run was going (a deploy, a crash): set at the next startup, so a
# row never claims to be running in a process that no longer exists.
INTERRUPTED = "interrupted"
class RunTrigger(StrEnum):
# The run's interval came round.
SCHEDULED = "scheduled"
# An admin pressed "run now" (or "run all").
MANUAL = "manual"
# A user's own action started work on one target (a channel's full history), outside the
# recurring jobs.
AD_HOC = "ad_hoc"
def sql_in(values: type[StrEnum]) -> str:
"""`'a', 'b'` — the IN-list a CHECK constraint names these values with."""
return ", ".join(f"'{v.value}'" for v in values)
+346 -190
View File
@@ -3,28 +3,31 @@ recent-first / deep backfill. Runs inside the API process (single worker)."""
import contextvars
import threading
from datetime import datetime, timezone
from datetime import timezone
from typing import Callable
from apscheduler.executors.pool import ThreadPoolExecutor
from apscheduler.jobstores.base import ConflictingIdError
from apscheduler.schedulers.background import BackgroundScheduler
from sqlalchemy import select
from app import audit, debugsession, logs, logsink, progress, quota
from app.audit import AuditAction
from app import audit, debugsession, jobruns, logs, logsink, progress, quota, sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads.gc import run_download_gc
from app.models import SchedulerSetting
from app.models import Channel, JobRun
from app.plex.sync import sync as run_plex_sync
from app.plex.tmdb import run_enrich as run_tmdb_enrich
from app.plex.tmdb import run_people_enrich
from app.plex.watch_sync import run_plex_watch_reconcile, run_plex_watch_sync
from app.notifications import create_notification
from app.runstatus import RunStatus, RunTrigger
from app.state import is_sync_paused
from app.sync.autotag import run_autotag_all
from app.sync.explore import purge_ephemeral
from app.sync.maintenance import run_maintenance
from app.sync.playlists import sync_all_playlists
from app.utils import iso
from app.sync.runner import (
run_deep_backfill,
run_enrich,
@@ -37,12 +40,31 @@ from app.sync.runner import (
log = logs.get_logger(__name__)
_scheduler: BackgroundScheduler | None = None
# Per-job run activity, kept in-memory for the admin Scheduler dashboard to read (the
# scheduler lives in the same single-worker process as the API, so a request can read it
# directly). Ephemeral by design — it reflects "what the scheduler is doing right now and
# how the last runs went" since this process started; durable history isn't the goal here.
# What is running RIGHT NOW, per job: the single-run claim and the live progress. In memory on
# purpose — the scheduler lives in the API's one process, and progress changes every second. What a
# run did is durable instead: its `job_runs` row (app/jobruns.py), which outlives a restart.
_activity: dict[str, dict] = {}
_activity_lock = threading.Lock()
# The runs this process has open (run id -> job), under the same lock. A row can stay `running` in
# the table after its outcome failed to write (_close), so the table cannot say what is going; this
# can, as the one process per database (app/jobruns.py) is the one running them.
_open_runs: dict[int, str] = {}
def _opened(run_id: int, job: str) -> None:
with _activity_lock:
_open_runs[run_id] = job
def _closed(run_id: int) -> None:
with _activity_lock:
_open_runs.pop(run_id, None)
def _open_run_ids() -> dict[int, str]:
with _activity_lock:
return dict(_open_runs)
# Set (per-thread) by a manual "run now" trigger to the id of the admin who clicked, so that
# run — and only that run, not the recurring scheduled ones — posts a completion notification
@@ -51,59 +73,36 @@ _manual_actor: contextvars.ContextVar[int | None] = contextvars.ContextVar(
"manual_actor", default=None
)
# Each interval job's env/config default period (minutes). The admin can override any of
# these at runtime (stored in scheduler_settings, applied live); see load_intervals.
# Each interval job's env/config default period (minutes). The admin can override any of these at
# runtime from the Scheduler page — a sysconfig key per job (`sysconfig.interval_key`), applied
# live; see load_intervals.
JOB_INTERVALS: dict[str, int] = {
"rss_poll": settings.rss_poll_minutes,
"enrich": settings.enrich_interval_minutes,
"backfill": settings.backfill_interval_minutes,
"autotag": settings.autotag_interval_minutes,
"shorts": settings.shorts_probe_interval_minutes,
"subscriptions": settings.subscriptions_resync_minutes,
"playlist_sync": settings.playlist_sync_minutes,
"maintenance": settings.maintenance_interval_minutes,
"explore_cleanup": settings.explore_cleanup_minutes,
"download_gc": settings.download_gc_minutes,
"plex_sync": settings.plex_sync_interval_min,
"plex_watch_sync": settings.plex_watch_sync_interval_min,
"plex_watch_reconcile": settings.plex_watch_reconcile_interval_min,
"tmdb_enrich": settings.tmdb_enrich_interval_min,
"people_enrich": settings.people_enrich_interval_min,
"audit_gc": settings.audit_gc_minutes,
"log_gc": settings.log_gc_minutes,
job: getattr(settings, attr) for job, attr in sysconfig.JOB_INTERVAL_DEFAULTS.items()
}
# Sane bounds for an admin-set interval (minutes).
MIN_INTERVAL = 1
MAX_INTERVAL = 1440 # one day
MIN_INTERVAL = sysconfig.MIN_INTERVAL
MAX_INTERVAL = sysconfig.MAX_INTERVAL
def load_intervals(db) -> dict[str, int]:
"""Effective per-job intervals: the env/config defaults overlaid with any admin overrides
saved in scheduler_settings."""
iv = dict(JOB_INTERVALS)
for row in db.execute(select(SchedulerSetting)).scalars():
if row.job_id in iv and row.interval_minutes:
iv[row.job_id] = row.interval_minutes
return iv
"""Effective per-job intervals: each job's sysconfig override, else its env/config default."""
return {job: sysconfig.get_int(db, sysconfig.interval_key(job)) for job in JOB_INTERVALS}
def apply_interval(db, job_id: str, minutes: int) -> int:
"""Persist a job's interval override and reschedule the live job immediately (if the
scheduler runs in this process). Returns the applied value."""
def stage_interval(db, job_id: str, minutes: int) -> int:
"""Stage a job's interval override (clamped) in `db`; returns the value. The caller commits it
with its audit row, then calls `reschedule` — the live job follows only what was stored."""
if job_id not in JOB_INTERVALS:
raise KeyError(job_id)
minutes = max(MIN_INTERVAL, min(MAX_INTERVAL, int(minutes)))
row = db.get(SchedulerSetting, job_id)
if row is None:
db.add(SchedulerSetting(job_id=job_id, interval_minutes=minutes))
else:
row.interval_minutes = minutes
db.commit()
sysconfig.set_value(db, sysconfig.interval_key(job_id), minutes)
return minutes
def reschedule(job_id: str, minutes: int) -> None:
"""Move the live interval job to `minutes` (if this process registered it)."""
if _scheduler is not None and _scheduler.get_job(job_id) is not None:
_scheduler.reschedule_job(job_id, trigger="interval", minutes=minutes)
log.info("rescheduled job %s to every %s min", job_id, minutes)
return minutes
def _record(name: str, **fields) -> None:
@@ -112,7 +111,12 @@ def _record(name: str, **fields) -> None:
def _notify_done(
db, actor_id: int, job_id: str, status: str, summary: str | None, result: dict | None = None
db,
actor_id: int,
job_id: str,
status: RunStatus,
summary: str | None,
result: dict | None = None,
) -> None:
"""Post a completion notice to the triggering admin's inbox (manual runs only). Best
effort — a notification failure must never mask the job's own outcome."""
@@ -130,49 +134,6 @@ def _notify_done(
log.exception("failed to write completion notification for job %s", job_id)
def _run_changed(result) -> bool:
"""Did an automatic job run actually do something worth an audit row? (No-op polls don't.)
A "change" = a truthy NUMERIC count; status-string dicts like {"skipped": "disabled"} (Plex
off — the default) are no-ops, not changes, so they don't flood the trail every interval."""
if result is None:
return False
if isinstance(result, dict):
# A phased job (backfill's recent/deep, people_enrich's bios/portraits) nests a result per
# phase: a count inside a phase is a change too.
return any(
_run_changed(v) if isinstance(v, dict) else isinstance(v, (int, float)) and bool(v)
for v in result.values()
)
if isinstance(result, (int, float)):
return result != 0
return bool(result)
def _audit_run(db, actor_id, job_id, status, summary, result) -> None:
"""Write ONE run-summary audit row per job run (never per-item). Manual runs (an admin
clicked "run now" → actor_id set) and errors are always logged; automatic runs only when
they changed something, so routine no-op polls don't spam the trail. Best-effort."""
if actor_id is None and status == "ok" and not _run_changed(result):
return
# The result as data too: the audit log names it in the reader's language, as the inbox
# does; `summary` stays the English line of the row's own summary column.
data = _result_data(result)
try:
audit.record(
db,
actor_id,
AuditAction.JOB_RUN,
target_type="job",
target_id=job_id,
summary=f"{job_id}: {status}" + (f" — {summary}" if summary else ""),
after={"status": status, "summary": summary, **({"result": data} if data else {})},
)
db.commit()
except Exception:
db.rollback()
log.exception("failed to write audit row for job %s", job_id)
def _job(name: str, fn) -> None:
# The one place both run paths converge (manual trigger AND scheduled fire), so claim the run
# HERE, atomically: whichever reaches this first wins, the other skips. This is the real
@@ -180,71 +141,88 @@ def _job(name: str, fn) -> None:
if not _claim_running(name):
log.info("job %s already running — skipping this trigger", name)
return
db = SessionLocal()
actor_id = _manual_actor.get() # set only for a manual "run now" trigger
_record(
name, last_started=datetime.now(timezone.utc).isoformat(), last_error=None, progress=None
) # running already claimed
trigger = RunTrigger.SCHEDULED if actor_id is None else RunTrigger.MANUAL
_record(name, progress=None) # running already claimed
db = SessionLocal()
run_id = None
try:
try:
run_id = jobruns.start(name, trigger, actor_id)
except Exception:
# No row to tell it in: the database is what failed. The log is the record.
log.exception("job %s not started: its run could not be recorded", name)
return
_opened(run_id, name)
with logs.bind(run_id=run_id):
_run(db, name, fn, run_id, actor_id)
finally:
# ALWAYS release the claim, even if the body raised before its own status update — so a
# failure can never wedge the job as permanently "running".
if run_id is not None:
_closed(run_id)
_record(name, running=False, progress=None)
db.close()
def _run(db, name: str, fn, run_id: int, actor_id: int | None) -> None:
"""The run itself, inside its `job_runs` row: the pause-skip, the work, and its outcome — on
the row, and in the triggering admin's inbox for a manual run."""
def sink(current, total, phase):
_record(name, progress={"current": current, "total": total, "phase": phase})
# Only the work is inside the try: writing its outcome is not the job, and a write that fails
# must not turn a run that succeeded into a failed one (see _close).
result = None
try:
if is_sync_paused(db):
log.info("job %s skipped (sync paused)", name)
_record(
name,
status="skipped",
last_finished=datetime.now(timezone.utc).isoformat(),
last_result="paused",
result=_PAUSED,
progress=None,
)
if actor_id is not None:
_notify_done(db, actor_id, name, "skipped", "sync paused", _PAUSED)
_audit_run(db, actor_id, name, "skipped", "sync paused", _PAUSED)
return
with progress.bind(sink):
result = fn(db)
summary = _summarize(result)
data = _result_data(result)
log.info("job %s -> %s", name, result)
_record(
name,
status="ok",
last_finished=datetime.now(timezone.utc).isoformat(),
last_result=summary,
result=data,
progress=None,
)
if actor_id is not None:
_notify_done(db, actor_id, name, "ok", summary, data)
_audit_run(db, actor_id, name, "ok", summary, result)
paused = is_sync_paused(db)
if not paused:
with progress.bind(sink):
result = fn(db)
except Exception as exc:
db.rollback()
log.exception("job %s failed", name, extra=logs.event("scheduler.job.failed", job=name))
err = str(exc) or exc.__class__.__name__
# A failure the job named (progress.JobFailed) rides along as data, as a result does, so the
# client says it in the reader's language; `err` stays the English line.
failure = (
{"failure": exc.reason, **exc.params} if isinstance(exc, progress.JobFailed) else None
)
_record(
name,
status="error",
last_finished=datetime.now(timezone.utc).isoformat(),
last_error=err,
result=failure,
progress=None,
)
err, failure = _failure(exc)
_close(run_id, RunStatus.ERROR, error=err, failure=failure)
if actor_id is not None:
_notify_done(db, actor_id, name, "error", err, failure)
_audit_run(db, actor_id, name, "error", err, failure)
finally:
# ALWAYS release the claim, even if the body raised before its own status update — so a
# failure can never wedge the job as permanently "running".
_record(name, running=False)
db.close()
_notify_done(db, actor_id, name, RunStatus.ERROR, err, failure)
return
if paused:
log.info("job %s skipped (sync paused)", name)
_close(run_id, RunStatus.SKIPPED, summary="sync paused", result=_PAUSED)
if actor_id is not None:
_notify_done(db, actor_id, name, RunStatus.SKIPPED, "sync paused", _PAUSED)
return
summary, data = _finish_ok(name, run_id, result)
if actor_id is not None:
_notify_done(db, actor_id, name, RunStatus.OK, summary, data)
def _close(run_id: int, status: RunStatus, **outcome) -> None:
"""Write a run's outcome. Its own failure (the database) is the log's to tell, never the job's:
the row then stays `running` until the next start's sweep closes it as interrupted."""
try:
jobruns.finish(run_id, status, **outcome)
except Exception:
log.exception("run %s: its outcome (%s) could not be recorded", run_id, status)
def _finish_ok(name: str, run_id: int, result) -> tuple[str | None, dict | None]:
"""Close a run that returned: its result on the row, as data and as the English one-liner."""
summary = _summarize(result)
data = _result_data(result)
log.info("job %s -> %s", name, result)
_close(run_id, RunStatus.OK, summary=summary, result=data)
return summary, data
def _failure(exc: Exception) -> tuple[str, dict | None]:
"""A raised run's English error line, and the failure the job named (progress.JobFailed) as
data, as a result is, so the client says it in the reader's language."""
err = str(exc) or exc.__class__.__name__
failure = {"failure": exc.reason, **exc.params} if isinstance(exc, progress.JobFailed) else None
return err, failure
# A skipped run's result, in the same shape as a job's own `{"skipped": …}`.
@@ -271,11 +249,13 @@ def _summarize(result) -> str | None:
return str(result)
def scheduler_snapshot() -> dict:
"""What the scheduler is doing, for the admin dashboard. `running_here` is False in
environments where the scheduler is disabled (e.g. local dev shares the prod-ish DB but
runs no scheduler), so the UI can say so while still showing DB-derived queue/quota."""
running_here = _scheduler is not None
def scheduler_snapshot(db) -> dict:
"""What the scheduler is doing, for the admin dashboard: each job's live state from this process
and its last runs from `job_runs`, so a restart no longer blanks "when did it last run, and how
did it go"; and the same for the ad-hoc runs users start on one target. `running_here` is False where the scheduler runs no interval jobs
(SCHEDULER_ENABLED=false), so the UI can say so while still showing the DB-derived history,
queue and quota."""
running_here = _scheduler is not None and settings.scheduler_enabled
next_runs: dict[str, str | None] = {}
live_intervals: dict[str, int] = {}
if _scheduler is not None:
@@ -287,6 +267,12 @@ def scheduler_snapshot() -> dict:
live_intervals[job.id] = round(iv.total_seconds() / 60)
with _activity_lock:
acts = {k: dict(v) for k, v in _activity.items()}
# Open before the read or after it: a run that started or ended while the rows were read is
# still counted as the open run it was, never as a stray one.
open_runs = _open_run_ids()
recent = jobruns.recent_runs(db, [*JOB_INTERVALS, *AD_HOC_JOBS], RECENT_RUNS)
open_runs |= _open_run_ids()
going_id = max((r for r, j in open_runs.items() if j == CHANNEL_BACKFILL), default=None)
jobs = []
for job_id, default_interval in JOB_INTERVALS.items():
a = acts.get(job_id, {})
@@ -296,16 +282,94 @@ def scheduler_snapshot() -> dict:
"interval_minutes": live_intervals.get(job_id, default_interval),
"next_run": next_runs.get(job_id),
"running": a.get("running", False),
"status": a.get("status"),
"last_started": a.get("last_started"),
"last_finished": a.get("last_finished"),
"last_result": a.get("last_result"),
"result": a.get("result"),
"last_error": a.get("last_error"),
"progress": a.get("progress"),
**_last_runs(recent.get(job_id, []), open_runs),
}
)
return {"running_here": running_here, "enabled": settings.scheduler_enabled, "jobs": jobs}
# Work a user starts on one target: no interval, no "run now" (it needs its target), but its
# runs are shown beside the jobs' — it spends the same quota. Its one in-memory guard is the
# target's own (sync/runner), so "running" is the runs this process has open: two channels can
# backfill at once, so a run still going counts even when a newer one already ended. That run then
# leads the row, so its channel, its "no outcome yet" and its dot are one run's, never the
# newer run's outcome under the older one's channel.
backfills = recent.get(CHANNEL_BACKFILL, [])
going = db.get(JobRun, going_id) if going_id is not None else None
if going is not None and going.status != RunStatus.RUNNING:
going = None # its outcome is written; only the bookkeeping has not caught up
if going is not None:
backfills = [going, *(r for r in backfills if r.id != going.id)][:RECENT_RUNS]
shown = backfills[0] if backfills else None
ad_hoc = [
{
"id": CHANNEL_BACKFILL,
"running": going is not None,
"target": shown.target if shown else None,
"target_title": (
db.scalar(select(Channel.title).where(Channel.id == shown.target))
if shown is not None and shown.target
else None
),
**_last_runs(backfills, open_runs),
}
]
return {
"running_here": running_here,
"enabled": settings.scheduler_enabled,
"jobs": jobs,
"ad_hoc": ad_hoc,
}
# How many past runs a Scheduler row shows as dots; the full history is the run list.
RECENT_RUNS = 3
def _seen(run, open_runs: dict[int, str]) -> RunStatus:
"""A run's status as the page tells it. A row `running` that no run here has open is one whose
outcome failed to write: the next start's sweep closes it as interrupted, so it reads so now —
not as a run still going beside a row that says nothing runs."""
if run.status == RunStatus.RUNNING and run.id not in open_runs:
return RunStatus.INTERRUPTED
return RunStatus(run.status)
def _last_runs(runs: list, open_runs: dict[int, str]) -> dict:
"""A row's run fields from its newest runs (newest first): how the last one went, and the few
before it for the history dots."""
run = runs[0] if runs else None
status = _seen(run, open_runs) if run is not None else None
# A run still going has no outcome yet: only an ended run says how it went.
ended = run if status is not None and status != RunStatus.RUNNING else None
return {
"status": status if ended else None,
"last_started": iso(run.started_at) if run else None,
# A stray row has no end of its own: it ended, as far as anyone can tell, where it began.
"last_finished": iso(ended.finished_at or ended.started_at) if ended else None,
"last_result": ended.summary if ended else None,
# The outcome as data: the result, or on a failed run the failure the job named.
"result": _outcome_data(ended),
"last_error": ended.error if ended else None,
"last_trigger": run.trigger if run else None,
"last_run_id": run.id if run else None,
"quota_units": ended.quota_units if ended else None,
"recent": [_run_brief(r, open_runs) for r in runs],
}
def _run_brief(run, open_runs: dict[int, str]) -> dict:
return {
"id": run.id,
"status": _seen(run, open_runs),
"trigger": run.trigger,
"started_at": iso(run.started_at),
"finished_at": iso(run.finished_at),
}
def _outcome_data(run) -> dict | None:
if run is None:
return None
return run.failure if run.status == RunStatus.ERROR else run.result
def _rss_job() -> None:
@@ -408,15 +472,16 @@ def _people_enrich_job() -> None:
def _audit_gc_job() -> None:
# The history cleanup: audit-log rows past audit_retention_days and YouTube quota history past
# quota_retention_days (0 = keep forever, each). Pure DB, no quota.
# The history cleanup: audit-log rows past audit_retention_days, YouTube quota history past
# quota_retention_days and finished runs past job_run_retention_days (0 = keep forever, each).
# Pure DB, no quota.
def work(db):
from app import sysconfig
reaped = audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))
quota_reaped = quota.gc(db, sysconfig.get_int(db, "quota_retention_days"))
runs_reaped = jobruns.gc(db, sysconfig.get_int(db, "job_run_retention_days"))
db.commit()
return {"reaped": reaped, "quota_reaped": quota_reaped}
return {"reaped": reaped, "quota_reaped": quota_reaped, "runs_reaped": runs_reaped}
_job("audit_gc", work)
@@ -425,7 +490,6 @@ def _log_gc_job() -> None:
# Prune log_events rows older than log_retention_days — the limit the privacy policy states, so
# this runs hourly rather than daily. Pure DB, no quota.
def work(db):
from app import sysconfig
days = sysconfig.get_int(db, "log_retention_days")
reaped = logsink.gc(db, days)
@@ -485,15 +549,29 @@ def running_job_ids() -> set[str]:
return {jid for jid, a in _activity.items() if a.get("running")}
def trigger_job(job_id: str, actor_id: int | None = None) -> str:
"""Run a job immediately in a background thread, independent of its interval schedule.
The wrapper handles its own DB session, activity tracking and pause-skip, so the live
dashboard reflects the run. Returns "started", "already_running", or raises KeyError for
an unknown job. Refusing a concurrent run keeps a manual trigger from overlapping a
scheduled run (APScheduler enforces max_instances=1 for the scheduled side).
def _as_actor(fn: Callable[[], None], actor_id: int | None) -> Callable[[], None]:
"""`fn` run as `actor_id`'s manual run. The actor is RESET afterwards: the executor's pool
threads are reused and keep their context from task to task, so an actor left set would make
the next scheduled run that thread picks up look like this admin's manual run."""
`actor_id` (the admin who clicked "run now") is stashed in a contextvar inside the new
thread so the run posts a completion notification to that admin's inbox."""
def run() -> None:
token = _manual_actor.set(actor_id)
try:
fn()
finally:
_manual_actor.reset(token)
return run
def trigger_job(job_id: str, actor_id: int | None = None) -> str:
"""Run a job now, independent of its interval schedule: a one-shot job on the same scheduler,
so a manual run shares the scheduled runs' executor and shutdown — no thread of its own. The
wrapper handles its own DB session, run row and pause-skip. Returns "started",
"already_running", or raises KeyError for an unknown job.
`actor_id` (the admin who clicked "run now") is set in a contextvar inside the run's thread, so
the run is recorded as theirs and posts a completion notification to their inbox."""
fn = JOB_FUNCS.get(job_id)
if fn is None:
raise KeyError(job_id)
@@ -502,13 +580,18 @@ def trigger_job(job_id: str, actor_id: int | None = None) -> str:
# _job claims, sees it's taken, and skips — no double run.
if _is_running(job_id):
return "already_running"
sched = _scheduler
if sched is None:
# Only between shutdown and exit: start_scheduler runs at every API startup.
raise RuntimeError("the scheduler is not running")
def run() -> None:
if actor_id is not None:
_manual_actor.set(actor_id)
fn()
threading.Thread(target=run, name=f"trigger-{job_id}", daemon=True).start()
try:
# One id per job: a second click before the first one-shot has even started is the same
# run, not a queued second one. misfire_grace_time=None: a busy pool delays it, never drops it.
sched.add_job(_as_actor(fn, actor_id), "date", id=f"{job_id}@now", misfire_grace_time=None)
# silent-ok: not a failure — the answer is the 409 the admin sees ("already running").
except ConflictingIdError:
return "already_running"
return "started"
@@ -521,22 +604,95 @@ def trigger_all(actor_id: int | None = None) -> list[str]:
return started
def start_scheduler() -> None:
global _scheduler
if not settings.scheduler_enabled or _scheduler is not None:
return
# Effective intervals = env defaults overlaid with any admin overrides from the DB.
db = SessionLocal()
# Work a user starts on one target outside the job list — today the "get full history" backfill of
# one channel. It runs on the same scheduler as a recorded run (trigger `ad_hoc`), so what it did and
# spent is in `job_runs` like any job's, not a background task only the log ever heard of.
CHANNEL_BACKFILL = "channel_backfill"
# Every ad-hoc job id: the page names each one (a frontend contract test reads this tuple).
AD_HOC_JOBS = (CHANNEL_BACKFILL,)
def run_ad_hoc(job: str, fn: Callable[[], dict], *, actor_id: int, target: str) -> None:
"""Run `fn` now as `actor_id`'s run of `job` on `target`, a one-shot job on the scheduler. Its
own exclusion (if any) lives in `fn`, which says so in its result; one id per job and target, so
a second request before the first has even started is that same run."""
if job not in AD_HOC_JOBS:
raise KeyError(job) # an unnamed job would show on the page as its bare id
sched = _scheduler
if sched is None:
# Only between shutdown and exit: start_scheduler runs at every API startup.
raise RuntimeError("the scheduler is not running")
try:
iv = load_intervals(db)
sched.add_job(
_ad_hoc,
"date",
args=(job, fn, actor_id, target),
id=f"{job}@{target}",
misfire_grace_time=None,
)
# silent-ok: not a failure — the run for this target is already queued, and it does this work.
except ConflictingIdError:
log.info("ad-hoc %s for %s already queued", job, target)
def _ad_hoc(job: str, fn: Callable[[], dict], actor_id: int, target: str) -> None:
try:
run_id = jobruns.start(job, RunTrigger.AD_HOC, actor_id, target)
except Exception:
# No row to tell it in: the database is what failed. The log is the record.
log.exception("ad-hoc %s for %s not started: its run could not be recorded", job, target)
return
_opened(run_id, job)
try:
_ad_hoc_work(job, fn, run_id)
finally:
db.close()
scheduler = BackgroundScheduler(timezone="UTC")
for job_id, fn in JOB_FUNCS.items():
scheduler.add_job(fn, "interval", minutes=iv[job_id], id=job_id)
_closed(run_id)
def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int) -> None:
with logs.bind(run_id=run_id):
# As in _run: only the work is inside the try.
try:
result = fn()
except Exception as exc:
log.exception("job %s failed", job, extra=logs.event("scheduler.job.failed", job=job))
err, failure = _failure(exc)
_close(run_id, RunStatus.ERROR, error=err, failure=failure)
return
_finish_ok(job, run_id, result)
# Room for every job at once (a "run all") beside a few ad-hoc runs: APScheduler's default pool of
# 10 would queue a scheduled fire behind long manual runs.
_POOL_SIZE = len(JOB_FUNCS) + 8
def start_scheduler() -> None:
"""Start this process's scheduler. It always starts, so "run now" works on every install; only
with SCHEDULER_ENABLED does it also register the recurring interval jobs. Before anything can
run, every run a previous process left open is closed as interrupted — one API process serves a
database (app/jobruns.py), so an open run here is a dead process's."""
global _scheduler
if _scheduler is not None:
return
left = jobruns.interrupt_leftovers()
if left:
log.warning("%s run(s) left running by the previous process marked interrupted", left)
scheduler = BackgroundScheduler(
timezone="UTC", executors={"default": ThreadPoolExecutor(_POOL_SIZE)}
)
if settings.scheduler_enabled:
# Effective intervals = env defaults overlaid with any admin overrides from the DB.
db = SessionLocal()
try:
iv = load_intervals(db)
finally:
db.close()
for job_id, fn in JOB_FUNCS.items():
scheduler.add_job(fn, "interval", minutes=iv[job_id], id=job_id)
scheduler.start()
_scheduler = scheduler
log.info("scheduler started")
log.info("scheduler started (interval jobs %s)", "on" if settings.scheduler_enabled else "off")
def shutdown_scheduler() -> None:
+22 -21
View File
@@ -437,7 +437,7 @@ def run_deep_backfill(db: Session, max_channels: int = 10, max_pages: int = 5) -
# Channels being deep-backfilled RIGHT NOW by EITHER path — the interactive backfill_channel_now (a
# FastAPI BackgroundTasks threadpool thread) OR the scheduler's run_deep_backfill (its own thread).
# one-shot run on the scheduler's pool, see scheduler.run_ad_hoc) OR the scheduler's run_deep_backfill.
# BOTH register the channel they're paging (under the lock) so the other skips it: no two runs page
# the same channel's `backfill_cursor` concurrently, whichever started first. `_immediate_running`
# counts only the interactive path, capped so one user's burst can't drain the shared budget (the
@@ -448,9 +448,9 @@ _backfill_lock = threading.Lock()
_MAX_CONCURRENT_IMMEDIATE = 2
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> None:
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, in a
background task with its OWN session, so the full catalogue lands in seconds instead of over the
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> dict:
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, as its
own recorded run with its OWN session, so the full catalogue lands in seconds instead of over the
paced scheduler's 10-minute ticks (max_pages=5/channel/tick). max_pages is high enough to finish
any realistic channel in a single run (400 pages = 20 000 videos; e.g. BBC Earth's ~3400 = 69
pages — the old 60-page cap stopped one page short of large channels, so their tail dribbled in
@@ -459,34 +459,35 @@ def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) ->
Quota-guarded, capped to _MAX_CONCURRENT_IMMEDIATE interactive runs so one user can't drain the
shared backfill budget by opting many channels in at once (the overflow, and any channel too large
to finish under the reserve, is finished by the scheduler since deep_requested stays set), and
mutually excluded per-channel with the scheduler via _backfill_in_progress. Best-effort — logs and
swallows failures."""
mutually excluded per-channel with the scheduler via _backfill_in_progress.
Returns what the run did, as a job result: the counts, or `{"skipped": <why>}` when it left the
channel to the scheduler. A failure raises — the run that wraps this records it."""
global _immediate_running
with _backfill_lock:
if channel_id in _backfill_in_progress or _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
return # already backfilling this channel, or too many interactive runs — leave it to the scheduler
if channel_id in _backfill_in_progress:
return {"skipped": "channel busy"}
if _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
return {"skipped": "at capacity"}
_backfill_in_progress.add(channel_id)
_immediate_running += 1
try:
with SessionLocal() as db:
user = get_service_user(db)
if user is None:
return
return {"skipped": "no service user"}
channel = db.get(Channel, channel_id)
if channel is None or channel.backfill_done or not channel.uploads_playlist_id:
return
return {"skipped": "nothing to do"}
if quota.backfill_reserve_reached():
return
try:
# Attribute the spend to the requesting user under the full-backfill action, so it
# shows up correctly in the usage breakdown (not the default OTHER/system bucket).
with quota.attribute(user_id, quota.QuotaAction.VIDEOS_BACKFILL_FULL):
with YouTubeClient(db, user) as yt:
backfill_channel_deep(db, yt, channel, max_pages)
reconcile_full_history(db)
except Exception:
db.rollback()
log.exception("Immediate backfill failed for channel %s", channel_id)
return {"skipped": "low quota"}
# Attribute the spend to the requesting user under the full-backfill action, so it
# shows up correctly in the usage breakdown (not the default OTHER/system bucket).
with quota.attribute(user_id, quota.QuotaAction.VIDEOS_BACKFILL_FULL):
with YouTubeClient(db, user) as yt:
added = backfill_channel_deep(db, yt, channel, max_pages)
reconciled = reconcile_full_history(db)
return {"videos_added": added, "reconciled_full": reconciled}
finally:
with _backfill_lock:
_backfill_in_progress.discard(channel_id)
+251 -61
View File
@@ -8,7 +8,8 @@ stored Fernet-encrypted at rest (requires TOKEN_ENCRYPTION_KEY; without it, secr
stored in the DB and stay env-only).
"""
from dataclasses import dataclass
from dataclasses import dataclass, field
from typing import Literal
from sqlalchemy import select
from sqlalchemy.orm import Session
@@ -27,6 +28,16 @@ _decrypt_warned: set[str] = set()
_coerce_warned: set[tuple[str, str]] = set()
# When a changed value takes effect. "live": the next read obeys it — every consumer reads the key
# where it uses it, never once at import (test-held: a read outside a function fails). A value read
# once at startup is live only when its writer also applies it (the job intervals reschedule, the log
# level is set as it is saved). "restart": the process has to start
# again. No key needs a restart today, and the Configuration page has no way to say so yet, so a
# "restart" key is refused until the page can (test-held).
Applies = Literal["live", "restart"]
LIVE: Applies = "live"
@dataclass(frozen=True)
class ConfigSpec:
key: str
@@ -37,6 +48,8 @@ class ConfigSpec:
min: int | None = None
max: int | None = None
allow_empty: bool = False # str keys: a stored "" is an explicit empty value, not "unset"
# Required, with no default: each key states it, so a new one cannot be live by omission.
applies: Applies = field(kw_only=True)
@property
def choices(self) -> tuple[str, ...]:
@@ -45,30 +58,78 @@ class ConfigSpec:
return choices_of(Settings.model_fields[self.default_attr].annotation)
# Each interval job of the scheduler and the `settings` attribute holding its default period
# (minutes). Its override is the `job_interval_<job>` key, set from the Scheduler page.
JOB_INTERVAL_DEFAULTS: dict[str, str] = {
"rss_poll": "rss_poll_minutes",
"enrich": "enrich_interval_minutes",
"backfill": "backfill_interval_minutes",
"autotag": "autotag_interval_minutes",
"shorts": "shorts_probe_interval_minutes",
"subscriptions": "subscriptions_resync_minutes",
"playlist_sync": "playlist_sync_minutes",
"maintenance": "maintenance_interval_minutes",
"explore_cleanup": "explore_cleanup_minutes",
"download_gc": "download_gc_minutes",
"plex_sync": "plex_sync_interval_min",
"plex_watch_sync": "plex_watch_sync_interval_min",
"plex_watch_reconcile": "plex_watch_reconcile_interval_min",
"tmdb_enrich": "tmdb_enrich_interval_min",
"people_enrich": "people_enrich_interval_min",
"audit_gc": "audit_gc_minutes",
"log_gc": "log_gc_minutes",
}
# Sane bounds for an admin-set interval (minutes).
MIN_INTERVAL = 1
MAX_INTERVAL = 1440 # one day
def interval_key(job: str) -> str:
return f"job_interval_{job}"
# Registry of DB-overridable keys. Add a key here + wire its read through get_*() to move it
# off env into the admin UI; the admin page renders it automatically from this list. A group's keys
# stay together, in the order its section lists them; a NEW group also needs its place in SECTIONS.
SPECS: tuple[ConfigSpec, ...] = (
# --- Access / registration ---
ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
ConfigSpec("allow_registration", "bool", "access", "allow_registration", applies=LIVE),
# --- Email / SMTP (one logical group; the password is the only secret) ---
ConfigSpec("smtp_host", "str", "email", "smtp_host"),
ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535),
ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True),
ConfigSpec("smtp_from", "str", "email", "smtp_from"),
ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True),
ConfigSpec("smtp_host", "str", "email", "smtp_host", applies=LIVE),
ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535, applies=LIVE),
ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True, applies=LIVE),
ConfigSpec("smtp_from", "str", "email", "smtp_from", applies=LIVE),
ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True, applies=LIVE),
# --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True, applies=LIVE),
# Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
ConfigSpec(
"youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True, applies=LIVE
),
# --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
# install wizard / Configuration page so no restart is needed when the creds change. ---
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
# --- Quota (shared YouTube Data API daily budget) ---
ConfigSpec("quota_daily_budget", "int", "quota", "quota_daily_budget", min=1, max=1_000_000),
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True, applies=LIVE),
ConfigSpec(
"backfill_quota_reserve", "int", "quota", "backfill_quota_reserve", min=0, max=1_000_000
"google_client_secret", "str", "google", "google_client_secret", secret=True, applies=LIVE
),
# --- Quota (shared YouTube Data API daily budget) ---
ConfigSpec(
"quota_daily_budget",
"int",
"quota",
"quota_daily_budget",
min=1,
max=1_000_000,
applies=LIVE,
),
ConfigSpec(
"backfill_quota_reserve",
"int",
"quota",
"backfill_quota_reserve",
min=0,
max=1_000_000,
applies=LIVE,
),
ConfigSpec(
"search_daily_limit_per_user",
@@ -77,10 +138,19 @@ SPECS: tuple[ConfigSpec, ...] = (
"search_daily_limit_per_user",
min=0,
max=100_000,
applies=LIVE,
),
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
ConfigSpec("search_source", "choice", "quota", "search_source"),
ConfigSpec("quota_retention_days", "int", "quota", "quota_retention_days", min=0, max=3650),
ConfigSpec("search_source", "choice", "quota", "search_source", applies=LIVE),
ConfigSpec(
"quota_retention_days",
"int",
"quota",
"quota_retention_days",
min=0,
max=3650,
applies=LIVE,
),
# --- Backfill (recent-first first pass per channel) ---
ConfigSpec(
"backfill_recent_max_videos",
@@ -89,19 +159,46 @@ SPECS: tuple[ConfigSpec, ...] = (
"backfill_recent_max_videos",
min=1,
max=100_000,
applies=LIVE,
),
ConfigSpec(
"backfill_recent_max_days", "int", "backfill", "backfill_recent_max_days", min=1, max=36_500
"backfill_recent_max_days",
"int",
"backfill",
"backfill_recent_max_days",
min=1,
max=36_500,
applies=LIVE,
),
ConfigSpec(
"rss_fallback_hours", "int", "backfill", "rss_fallback_hours", min=0, max=168, applies=LIVE
),
ConfigSpec("rss_fallback_hours", "int", "backfill", "rss_fallback_hours", min=0, max=168),
# --- Shorts probe ---
ConfigSpec(
"shorts_probe_max_seconds", "int", "shorts", "shorts_probe_max_seconds", min=1, max=3_600
"shorts_probe_max_seconds",
"int",
"shorts",
"shorts_probe_max_seconds",
min=1,
max=3_600,
applies=LIVE,
),
ConfigSpec(
"shorts_probe_batch", "int", "shorts", "shorts_probe_batch", min=1, max=10_000, applies=LIVE
),
ConfigSpec("shorts_probe_batch", "int", "shorts", "shorts_probe_batch", min=1, max=10_000),
# --- Batch sizes ---
ConfigSpec("enrich_batch_size", "int", "batch", "enrich_batch_size", min=1, max=50),
ConfigSpec("autotag_title_sample", "int", "batch", "autotag_title_sample", min=1, max=1_000),
ConfigSpec(
"enrich_batch_size", "int", "batch", "enrich_batch_size", min=1, max=50, applies=LIVE
),
ConfigSpec(
"autotag_title_sample",
"int",
"batch",
"autotag_title_sample",
min=1,
max=1_000,
applies=LIVE,
),
# --- Maintenance (rolling re-validation of already-known videos) ---
# Videos re-checked per maintenance run. Folded here from app_state (B-core-11) so every admin
# knob lives in one store; read by sync/maintenance.py, tuned from the Scheduler dashboard.
@@ -112,16 +209,33 @@ SPECS: tuple[ConfigSpec, ...] = (
"maintenance_revalidate_batch",
min=100,
max=100_000,
applies=LIVE,
),
# --- Discovery cleanup (search + explore ephemeral content) ---
# Days an explored-but-unsubscribed channel is kept before the cleanup job purges it.
ConfigSpec("explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650),
# Days an un-kept live-search result video is kept before the cleanup job removes it.
ConfigSpec("search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650),
# --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
ConfigSpec("download_total_max_bytes", "int", "downloads", "download_total_max_bytes", min=0),
ConfigSpec(
"download_default_max_bytes", "int", "downloads", "download_default_max_bytes", min=0
"explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650, applies=LIVE
),
# Days an un-kept live-search result video is kept before the cleanup job removes it.
ConfigSpec(
"search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650, applies=LIVE
),
# --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
ConfigSpec(
"download_total_max_bytes",
"int",
"downloads",
"download_total_max_bytes",
min=0,
applies=LIVE,
),
ConfigSpec(
"download_default_max_bytes",
"int",
"downloads",
"download_default_max_bytes",
min=0,
applies=LIVE,
),
ConfigSpec(
"download_default_max_concurrent",
@@ -130,6 +244,7 @@ SPECS: tuple[ConfigSpec, ...] = (
"download_default_max_concurrent",
min=1,
max=20,
applies=LIVE,
),
ConfigSpec(
"download_default_max_jobs",
@@ -138,6 +253,7 @@ SPECS: tuple[ConfigSpec, ...] = (
"download_default_max_jobs",
min=1,
max=100_000,
applies=LIVE,
),
# min=1, NOT 0: a stored 0 used to read as "no per-file cap" (the cap was enforced as
# `if cap and size > cap`, so a falsy value disabled it). Now the route passes this number
@@ -161,14 +277,27 @@ SPECS: tuple[ConfigSpec, ...] = (
"download_upload_max_bytes",
min=1,
max=settings.download_upload_max_bytes,
applies=LIVE,
),
ConfigSpec(
"download_retention_days", "int", "downloads", "download_retention_days", min=0, max=3_650
"download_retention_days",
"int",
"downloads",
"download_retention_days",
min=0,
max=3_650,
applies=LIVE,
),
ConfigSpec(
"download_gc_grace_days", "int", "downloads", "download_gc_grace_days", min=0, max=3_650
"download_gc_grace_days",
"int",
"downloads",
"download_gc_grace_days",
min=0,
max=3_650,
applies=LIVE,
),
ConfigSpec("download_layout", "choice", "downloads", "download_layout"),
ConfigSpec("download_layout", "choice", "downloads", "download_layout", applies=LIVE),
ConfigSpec(
"download_worker_concurrency",
"int",
@@ -176,60 +305,112 @@ SPECS: tuple[ConfigSpec, ...] = (
"download_worker_concurrency",
min=1,
max=16,
applies=LIVE,
),
ConfigSpec("sponsorblock_default", "bool", "downloads", "sponsorblock_default"),
# --- Audit log ---
# Days to keep admin audit-log rows before the audit_gc job prunes them (0 = keep forever).
ConfigSpec("audit_retention_days", "int", "audit", "audit_retention_days", min=0, max=3650),
ConfigSpec(
"audit_retention_days",
"int",
"audit",
"audit_retention_days",
min=0,
max=3650,
applies=LIVE,
),
# Days the job_runs table keeps a finished background run before the audit_gc job prunes it.
ConfigSpec(
"job_run_retention_days",
"int",
"audit",
"job_run_retention_days",
min=0,
max=3650,
applies=LIVE,
),
# --- Diagnostics ---
# Runtime log level for the API process. Applied without a restart — see routes/config.py and
# main.py's lifespan.
ConfigSpec("log_level", "choice", "diagnostics", "log_level"),
ConfigSpec("log_level", "choice", "diagnostics", "log_level", applies=LIVE),
# Days the log_events table keeps a row (the log_gc job prunes). No "0 = forever": stored logs
# carry account-linked request history, and the privacy policy names a limit.
ConfigSpec("log_retention_days", "int", "diagnostics", "log_retention_days", min=1, max=90),
ConfigSpec(
"log_retention_days",
"int",
"diagnostics",
"log_retention_days",
min=1,
max=90,
applies=LIVE,
),
# --- Plex integration (optional; local-file playback + Plex metadata) ---
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled"),
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url"),
ConfigSpec("plex_server_token", "str", "plex", "plex_server_token", secret=True),
ConfigSpec("plex_path_map", "str", "plex", "plex_path_map"),
ConfigSpec("plex_libraries", "str", "plex", "plex_libraries"),
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
# (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
ConfigSpec("plex_max_sessions", "int", "plex", "plex_max_sessions", min=1, max=64),
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled", applies=LIVE),
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url", applies=LIVE),
ConfigSpec("plex_server_token", "str", "plex", "plex_server_token", secret=True, applies=LIVE),
ConfigSpec("plex_path_map", "str", "plex", "plex_path_map", applies=LIVE),
ConfigSpec("plex_libraries", "str", "plex", "plex_libraries", applies=LIVE),
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map", applies=LIVE),
ConfigSpec(
"plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16, applies=LIVE
),
ConfigSpec(
"plex_max_sessions", "int", "plex", "plex_max_sessions", min=1, max=64, applies=LIVE
),
# Transcode re-encode tuning. The libx264 knobs apply only to the software fallback.
# crf 0–51 (x264 range); max_height 0 = source.
ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw"),
ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset"),
ConfigSpec("plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51),
ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw", applies=LIVE),
ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset", applies=LIVE),
ConfigSpec(
"plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
"plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51, applies=LIVE
),
ConfigSpec(
"plex_transcode_max_height",
"int",
"plex",
"plex_transcode_max_height",
min=0,
max=4320,
applies=LIVE,
),
# Audio loudness normalisation on the HLS path (remux + transcode alike, not direct play).
ConfigSpec("plex_audio_loudnorm", "bool", "plex", "plex_audio_loudnorm"),
# NOTE: plex_watch_sync_interval_min is a config.py default read by the watch-sync scheduler job
# (scheduler.py); it is intentionally NOT a runtime admin knob, so it stays out of SPECS.
ConfigSpec("plex_audio_loudnorm", "bool", "plex", "plex_audio_loudnorm", applies=LIVE),
# --- TMDB enrichment (optional) — the API key is the only secret ---
ConfigSpec("tmdb_api_key", "str", "tmdb", "tmdb_api_key", secret=True),
ConfigSpec("tmdb_language", "str", "tmdb", "tmdb_language"),
ConfigSpec("tmdb_api_key", "str", "tmdb", "tmdb_api_key", secret=True, applies=LIVE),
ConfigSpec("tmdb_language", "str", "tmdb", "tmdb_language", applies=LIVE),
# --- Bug reports: the Gitea issue tracker a debug session's issue goes to ---
ConfigSpec("issues_gitea_url", "str", "issues", "issues_gitea_url"),
ConfigSpec("issues_gitea_repo", "str", "issues", "issues_gitea_repo"),
ConfigSpec("issues_gitea_token", "str", "issues", "issues_gitea_token", secret=True),
# (tmdb_enrich_interval_min is a scheduler job, tuned from the Scheduler dashboard; tmdb_enrich_batch
# is a config.py default — both stay out of SPECS, like plex_sync_interval_min.)
ConfigSpec("issues_gitea_url", "str", "issues", "issues_gitea_url", applies=LIVE),
ConfigSpec("issues_gitea_repo", "str", "issues", "issues_gitea_repo", applies=LIVE),
ConfigSpec(
"issues_gitea_token", "str", "issues", "issues_gitea_token", secret=True, applies=LIVE
),
# --- Scheduler: how often each interval job runs (minutes), one key per job ---
*(
ConfigSpec(
interval_key(job),
"int",
"scheduler",
attr,
min=MIN_INTERVAL,
max=MAX_INTERVAL,
applies=LIVE,
)
for job, attr in JOB_INTERVAL_DEFAULTS.items()
),
)
# The admin page's sections (the owner's pick): sixteen groups as tabs wrapped to three rows at 133%
# zoom, so they are a vertical list under four headings, in this order. Every group of SPECS sits in
# exactly one section (test-held), so a new group cannot fall off the page.
# exactly one section, or is edited on its own page (test-held), so a new group cannot fall off.
SECTIONS: tuple[tuple[str, tuple[str, ...]], ...] = (
("access", ("access", "google", "email")),
("youtube", ("youtube", "quota", "backfill", "shorts", "batch", "explore")),
("media", ("downloads", "plex", "tmdb")),
("system", ("maintenance", "audit", "diagnostics", "issues")),
)
# Groups another admin page edits, so the Configuration page neither lists nor writes them: an
# interval is changed on the Scheduler page, which also reschedules the running job — a write that
# skipped it would leave the job on its old period while the page showed the new one.
OWN_PAGE_GROUPS: tuple[str, ...] = ("scheduler",)
_BY_KEY: dict[str, ConfigSpec] = {s.key: s for s in SPECS}
@@ -238,6 +419,12 @@ def spec(key: str) -> ConfigSpec | None:
return _BY_KEY.get(key)
def config_page_spec(key: str) -> ConfigSpec | None:
"""`spec`, for the Configuration page: None for a key edited on its own page as well."""
s = _BY_KEY.get(key)
return None if s is None or s.group in OWN_PAGE_GROUPS else s
def secrets_manageable() -> bool:
"""Secret values can only be stored when Fernet is configured (TOKEN_ENCRYPTION_KEY)."""
return bool(settings.token_encryption_key)
@@ -426,13 +613,16 @@ def reset(db: Session, key: str) -> None:
def describe(db: Session) -> dict:
"""Admin-UI metadata + current values, grouped. Secret values are never echoed —
only whether an override is set and whether an env default exists."""
"""Admin-UI metadata + current values, grouped — the keys of the Configuration page, not those
edited on their own page. Secret values are never echoed — only whether an override is set and
whether an env default exists."""
groups: dict[str, list[dict]] = {}
# Every override in one query: a db.get per key was one round trip each (two for a key with no
# row, which the session cannot cache), ~40 ms of the page's ~55 on prod.
rows = {r.key: r for r in db.scalars(select(SystemConfig))}
for s in SPECS:
if s.group in OWN_PAGE_GROUPS:
continue
row = rows.get(s.key)
is_set = row is not None
item: dict = {
+9
View File
@@ -0,0 +1,9 @@
"""Every table, registered on `Base.metadata`.
Each unit declares its tables in its own `models` module, so importing `app.models` alone no longer
names the whole schema. Whatever needs all of it — the migrations, the test database, a check over
every mapper — imports this module.
"""
import app.downloads.models # Capture's tables
import app.models # noqa: F401 — the core tables
+73 -5
View File
@@ -1,13 +1,54 @@
"""Shared "who is an active, non-suspended account" predicate.
"""Whose a row is, and who an account is to the others — one place for both.
Used wherever a user must be filtered out of a recipient list or rejected as a target: the download
share-by-email and recipient-picker endpoints. Kept in one place so the two forms (SQL clauses for a
query, Python check for a loaded row) can't drift apart, and so a route module doesn't have to import
another route module just to reuse the rule.
**Ownership.** A user-owned table says whose a row is in one column (`user_id`, or `to_user_id` for
something handed to them). A unit's reads and writes name that column through `owned_by` and fetch a
row by id through `owned_row`, never with an inline `Model.user_id == …`: the rule "a row owned by
someone else is answered like a missing one" then lives here, and a contract test
(`test_unit_fences.py`) holds a switched-on unit to it.
**Recipients.** "An active, non-suspended account" — the download share-by-email and the recipient
picker filter on it. Kept here so the two forms (SQL clauses for a query, a Python check for a loaded
row) can't drift apart.
"""
from typing import Any, TypeVar
from sqlalchemy import ColumnElement, func, select
from sqlalchemy.orm import InstrumentedAttribute, Session
from app.collation import collated
from app.db import Base
from app.models import User
M = TypeVar("M", bound=Base)
def _owner_id(owner: User | int) -> int:
return owner if isinstance(owner, int) else owner.id
def owned_by(column: InstrumentedAttribute, owner: User | int) -> ColumnElement[bool]:
"""`column` — a table's owner column — names `owner`."""
return column == _owner_id(owner)
def owned_row(
db: Session,
model: type[M],
id_: Any,
owner: User | int,
*,
owner_attr: str = "user_id",
lock: bool = False,
) -> M | None:
"""The row of `model` with this primary key when `owner` owns it, else None — a row of someone
else's is the same answer as a missing one, so a caller can never tell the two apart. `lock`
takes it FOR UPDATE (past the identity map)."""
row = db.get(model, id_, with_for_update=True if lock else None)
if row is None or getattr(row, owner_attr) != _owner_id(owner):
return None
return row
def messageable_clauses() -> list:
"""SQL clauses for "an active, non-suspended account" — splat into a recipient query."""
@@ -20,3 +61,30 @@ def messageable_clauses() -> list:
def is_messageable_user(u: User | None) -> bool:
"""Python mirror of `messageable_clauses()` for an already-loaded row (a share recipient)."""
return bool(u and u.is_active and not u.is_suspended)
def user_by_email(db: Session, email: str) -> User | None:
"""The account with this address, matched without regard to case."""
return db.execute(
select(User).where(func.lower(User.email) == email.strip().lower())
).scalar_one_or_none()
def messageable_others(db: Session, user: User, lang: str) -> list[User]:
"""Every account `user` may hand something to: active, not suspended, not themselves; in the
reader's alphabet by address."""
return list(
db.execute(
select(User)
.where(User.id != user.id, *messageable_clauses())
.order_by(collated(User.email, lang))
).scalars()
)
def emails_of(db: Session, ids: set[int] | None = None) -> dict[int, str]:
"""Each account's address by id, in address order: these ids, or every account."""
stmt = select(User.id, User.email).order_by(User.email)
if ids is not None:
stmt = stmt.where(User.id.in_(ids))
return dict(db.execute(stmt).tuples().all())
+2 -1
View File
@@ -46,7 +46,8 @@ from app.downloads.jobdb import (
_staging_dir,
)
from app.downloads.status import STOPPED_BY_ROUTE, JobKind, JobStatus
from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset
from app.models import DISPLAY_TEXT_MAX
from app.downloads.models import DownloadJob, MediaAsset
log = logs.get_logger(__name__)
_stop = threading.Event()
+4 -1
View File
@@ -16,6 +16,9 @@ services:
POSTGRES_DB: siftlode_test
tmpfs:
- /var/lib/postgresql/data # ephemeral + fast; the schema is rebuilt per run
# One database per pytest-xdist worker, each through the app's own pool (5 + 10 overflow): the default 100
# connections would run out at 8 workers.
command: ["postgres", "-c", "max_connections=300"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U siftlode -d siftlode_test"]
interval: 1s
@@ -37,4 +40,4 @@ services:
- ./:/app
working_dir: /app
# Coverage is on demand, not in the gate: see docs/development.md.
command: ["python", "-m", "pytest"]
command: ["python", "-m", "pytest", "-n", "auto"]
+6 -1
View File
@@ -7,4 +7,9 @@
# not an unknown marker that quietly does nothing.
testpaths = tests
pythonpath = .
addopts = -q -rs --strict-markers
# `-p tests.db_per_worker`: under pytest-xdist (`-n`) every worker gets its own database — DATABASE_URL + `_gw<N>`,
# created on first use — because the `db` fixture's TRUNCATE-per-test isolation assumes one process per database.
# It is a plugin so it runs before conftest imports the app (whose settings read DATABASE_URL once). Serial: no-op.
# The file is a verbatim copy of a shared tool (see its docstring); change the source, not the copy.
addopts = -q -rs --strict-markers -p tests.db_per_worker
db_per_worker_env = DATABASE_URL
+3
View File
@@ -16,3 +16,6 @@ pytest-cov==7.1.0
ruff==0.15.21
httpx2==2.13.0
pip-audit==2.10.1
# pytest-xdist: the DB lane runs on every core (`-n auto` in docker-compose.test.yml); tests/db_per_worker.py gives
# each worker its own database. Same tests, same strictness — only side by side.
pytest-xdist==3.8.0
+1 -1
View File
@@ -25,7 +25,7 @@ from fastapi.testclient import TestClient
from sqlalchemy import create_engine, event, text
from starlette.requests import HTTPConnection
import app.models # register every model on Base.metadata
import app.tables # every table on Base.metadata
import app.state as app_state
from app import apitokens, security
from app.auth import current_user
+94
View File
@@ -0,0 +1,94 @@
"""Give every pytest-xdist worker its own database, so a suite that shares one test database can run in parallel.
Generic (source of truth: the memory repo's tools/pytest-parallel/db_per_worker.py); a project carries a verbatim copy
next to its conftest — tests often run in a container that cannot see this repo — and
`tools/pytest-parallel/sync-check.sh <copy>` reports drift. Keep the copy byte-identical; change it here.
Use, as a pytest plugin (it then runs before any conftest, i.e. before the app reads its settings):
pytest.ini: addopts = -p tests.db_per_worker (the copy's import path; `python -m pytest` puts cwd on sys.path)
db_per_worker_env = DATABASE_URL (the environment variable that holds the test database)
or call `apply("<ENV VAR>")` yourself before the app is imported. The variable may hold:
- a Postgres URL (`postgresql://…/name`, `postgresql+psycopg://…`): the worker gets `name_gw<N>`, created from the
base database's server if missing (empty — the project's own schema fixture migrates it, as for one process);
- a SQLite URL (`sqlite:///path.db`) or a plain file path: the worker gets `path-gw<N>.db`.
Without xdist (no PYTEST_XDIST_WORKER) or with the variable unset it changes nothing, so a serial run is unaffected.
"""
from __future__ import annotations
import os
import re
from urllib.parse import urlsplit, urlunsplit
import pytest
_PG = re.compile(r"^postgres(?:ql)?(?:\+\w+)?://", re.I)
def pytest_addoption(parser: pytest.Parser) -> None:
parser.addini(
"db_per_worker_env",
"env var holding the test database (a Postgres URL, SQLite URL or file path)",
)
@pytest.hookimpl(tryfirst=True)
def pytest_load_initial_conftests(early_config: pytest.Config) -> None:
var = early_config.getini("db_per_worker_env")
if var:
apply(str(var))
def apply(var: str) -> str | None:
worker = os.environ.get("PYTEST_XDIST_WORKER")
value = os.environ.get(var)
if not worker or not value:
return value
if _PG.match(value):
new = _postgres(value, worker)
else:
new = _file(value, worker)
os.environ[var] = new
return new
def _postgres(url: str, worker: str) -> str:
parts = urlsplit(url)
base = parts.path.lstrip("/")
name = f"{base}_{worker}"
if not re.fullmatch(r"\w+", name):
raise ValueError(f"db_per_worker: refusing an unusual database name {name!r}")
# The server, through the BASE database (it exists; `postgres` may not be reachable for this role).
plain = urlunsplit(
(re.sub(r"\+\w+$", "", parts.scheme), parts.netloc, parts.path, parts.query, "")
)
_create_if_missing(plain, name)
return urlunsplit((parts.scheme, parts.netloc, f"/{name}", parts.query, parts.fragment))
def _create_if_missing(dsn: str, name: str) -> None:
try:
import psycopg # psycopg 3
conn = psycopg.connect(dsn, autocommit=True)
except ImportError:
import psycopg2 # fallback for projects on psycopg2
conn = psycopg2.connect(dsn)
conn.autocommit = True
try:
cur = conn.cursor()
cur.execute("SELECT 1 FROM pg_database WHERE datname = %s", (name,))
if cur.fetchone() is None:
cur.execute(f'CREATE DATABASE "{name}"') # name checked to be \w+ above
finally:
conn.close()
def _file(value: str, worker: str) -> str:
prefix = ""
m = re.match(r"^(sqlite(?:\+\w+)?:///)(.*)$", value, re.I)
if m:
prefix, value = m.group(1), m.group(2)
root, ext = os.path.splitext(value)
return f"{prefix}{root}-{worker}{ext}"
+205
View File
@@ -0,0 +1,205 @@
"""The activity page's reads: the audit log and the background runs, filtered on the server and
paged by a keyset, and the quota breakdown over a time range and one account.
Each list is asked for the rows a filter must keep AND must drop, and a page boundary is crossed
with rows that share a timestamp — the case an offset or a timestamp-only cursor gets wrong.
"""
from datetime import datetime, timedelta, timezone
from app.auth import admin_user
from app.main import app
from app.models import AuditLog, Channel, JobRun, LogEvent, QuotaEvent
from tests.test_body_limit import _route_objects
T0 = datetime(2026, 10, 1, 12, 0, tzinfo=timezone.utc)
def _audit(db, action, actor=None, at=T0, summary=None, request_id=None) -> int:
row = AuditLog(
actor_id=actor, action=action, summary=summary, created_at=at, request_id=request_id
)
db.add(row)
db.flush()
return row.id
def _run(db, job, status="ok", actor=None, at=T0, target=None, trigger="scheduled") -> int:
row = JobRun(
job=job,
trigger=trigger,
actor_id=actor,
target=target,
status=status,
started_at=at,
finished_at=None if status == "running" else at + timedelta(seconds=4),
quota_units=0,
)
db.add(row)
db.flush()
return row.id
def _ids(body) -> list[int]:
return [r["id"] for r in body["items"]]
def test_the_audit_list_filters_on_the_server_and_pages_across_a_shared_timestamp(
db, make_user, as_user
):
admin = make_user(role="admin")
a = _audit(db, "config_set", admin.id, T0, "quota_daily_budget = 9000")
b = _audit(db, "config_set", admin.id, T0, "log_level = DEBUG")
c = _audit(db, "config_set", admin.id, T0, "smtp_host = mail")
bg = _audit(db, "user_purged", None, T0 - timedelta(days=3))
db.commit()
client = as_user(admin)
first = client.get("/api/admin/audit", params={"limit": 2, "action": "config_set"}).json()
assert _ids(first) == [c, b]
second = client.get(
"/api/admin/audit",
params={"limit": 2, "action": "config_set", "before": first["next_cursor"]},
).json()
assert (_ids(second), second["next_cursor"]) == ([a], None)
# A page that ends exactly at the last row says there is nothing older.
whole = client.get("/api/admin/audit", params={"limit": 3, "action": "config_set"}).json()
assert (_ids(whole), whole["next_cursor"]) == ([c, b, a], None)
assert first["accounts"] == {str(admin.id): admin.email}
def ids(**params) -> list[int]:
return _ids(client.get("/api/admin/audit", params=params).json())
assert ids(user="none") == [bg]
assert ids(user=str(admin.id)) == [c, b, a]
assert ids(q="DEBUG") == [b]
assert ids(since=(T0 - timedelta(days=1)).isoformat()) == [c, b, a]
assert ids(until=(T0 - timedelta(days=1)).isoformat()) == [bg]
# The action menu counts every action under the other filters, its own pick included.
facets = client.get("/api/admin/audit/facets", params={"action": "config_set"}).json()
assert facets["actions"] == [
{"value": "config_set", "count": 3},
{"value": "user_purged", "count": 1},
]
def test_an_admin_action_keeps_the_request_it_was_made_in(db, make_user, as_user):
client = as_user(make_user(role="admin"))
r = client.patch("/api/admin/config/quota_daily_budget", json={"value": 9500})
assert r.status_code == 200, r.text
(row,) = client.get("/api/admin/audit", params={"action": "config_set"}).json()["items"]
assert row["request_id"] == r.headers["x-request-id"]
def test_an_admin_action_says_how_many_of_its_requests_lines_are_still_kept(db, make_user, as_user):
"""A request's lines are kept days, its audit row months: a link to lines gone would open an
empty list, so the row says how many are left, as a run does."""
admin = make_user(role="admin")
kept = _audit(db, "config_set", actor=admin.id, request_id="req-kept-0001")
gone = _audit(db, "config_set", actor=admin.id, request_id="req-gone-0001")
db.add_all(
[
LogEvent(
ts=T0,
level="INFO",
sev=20,
source="api",
logger="t",
msg="m",
request_id="req-kept-0001",
)
for _ in range(3)
]
)
db.commit()
items = as_user(admin).get("/api/admin/audit").json()["items"]
lines = {r["id"]: r["log_lines"] for r in items}
assert (lines[kept], lines[gone]) == (3, 0)
def test_the_run_list_filters_pages_and_names_the_channel_and_the_log_lines(db, make_user, as_user):
admin = make_user(role="admin")
db.add(Channel(id="UCchan", title="Aileen Wang"))
ok1 = _run(db, "rss_poll", at=T0)
ok2 = _run(db, "rss_poll", at=T0)
bad = _run(db, "backfill", "error", admin.id, T0, trigger="manual")
adhoc = _run(
db, "channel_backfill", "ok", admin.id, T0 - timedelta(hours=1), "UCchan", "ad_hoc"
)
for _ in range(2):
db.add(
LogEvent(ts=T0, level="WARNING", sev=30, source="api", logger="t", msg="m", run_id=bad)
)
db.commit()
client = as_user(admin)
first = client.get("/api/admin/runs", params={"limit": 2}).json()
rest = client.get("/api/admin/runs", params={"limit": 2, "before": first["next_cursor"]}).json()
assert _ids(first) + _ids(rest) == [bad, ok2, ok1, adhoc]
assert rest["next_cursor"] is None
rows = {r["id"]: r for r in first["items"] + rest["items"]}
assert (rows[bad]["log_lines"], rows[ok2]["log_lines"]) == (2, 0)
assert (rows[adhoc]["target_title"], rows[ok1]["target_title"]) == ("Aileen Wang", None)
def ids(**params) -> list[int]:
return _ids(client.get("/api/admin/runs", params=params).json())
assert ids(status="error") == [bad]
assert ids(job="rss_poll") == [ok2, ok1]
assert ids(user="none") == [ok2, ok1]
assert ids(user=str(admin.id)) == [bad, adhoc]
assert ids(q="UCchan") == [adhoc]
assert ids(since=(T0 - timedelta(minutes=30)).isoformat()) == [bad, ok2, ok1]
assert client.get("/api/admin/runs", params={"status": "finished"}).status_code == 422
facets = client.get("/api/admin/runs/facets", params={"status": "error"}).json()
assert facets["statuses"] == [{"value": "ok", "count": 3}, {"value": "error", "count": 1}]
assert facets["jobs"] == [{"value": "backfill", "count": 1}]
assert facets["total"] == 4
def test_the_quota_breakdown_covers_its_range_and_one_account(db, make_user, as_user):
admin = make_user(role="admin")
now = datetime.now(timezone.utc)
db.add_all(
[
QuotaEvent(user_id=admin.id, action="search", units=100, created_at=now),
QuotaEvent(user_id=None, action="backfill", units=7, created_at=now),
QuotaEvent(
user_id=admin.id, action="search", units=50, created_at=now - timedelta(days=10)
),
]
)
db.commit()
client = as_user(admin)
def totals(**params) -> dict:
body = client.get("/api/quota/admin", params=params).json()
# The chart is the same filter's: its days add up to the table's total.
assert sum(d["total"] for d in body["daily"]) == sum(r["total"] for r in body["rows"])
return {r["user_id"]: r["total"] for r in body["rows"]}
assert totals() == {admin.id: 150, None: 7} # the last 30 days by default
assert totals(since=(now - timedelta(days=1)).isoformat()) == {admin.id: 100, None: 7}
assert totals(until=(now - timedelta(days=1)).isoformat()) == {admin.id: 50}
assert totals(user="none") == {None: 7}
assert totals(user=str(admin.id)) == {admin.id: 150}
def test_every_admin_read_and_write_is_admin_only():
"""The route table, not a list kept by hand: every /api/admin route and the quota breakdown
resolve `admin_user`, so a dropped dependency cannot hand an account's runs, errors and e-mail
addresses to any signed-in user."""
routes = _route_objects(app)
admin = {k: r for k, r in routes.items() if k[1].startswith(("/api/admin", "/api/quota/admin"))}
assert ("GET", "/api/admin/runs") in admin and ("GET", "/api/quota/admin") in admin
open_ = []
for key, route in admin.items():
stack, calls = [route.dependant], set()
while stack:
d = stack.pop()
calls.add(d.call)
stack.extend(d.dependencies)
if admin_user not in calls:
open_.append(key)
assert open_ == []
+10 -1
View File
@@ -24,6 +24,7 @@ def _row(**kw) -> dict:
"request_id": None,
"user_id": None,
"job_id": None,
"run_id": None,
"version": "0.90.0",
}
row.update(kw)
@@ -80,6 +81,7 @@ def test_newest_first_and_every_row_once_across_pages_even_on_equal_timestamps(d
({"event": "none"}, {"warn"}),
({"request_id": "req-abcdef12"}, {"mine"}),
({"job_id": 42}, {"err"}),
({"run_id": 7}, {"warn"}), # a background run's lines, not a download job's
({"q": "FORBIDDEN"}, {"err"}), # in attrs, any case
({"q": "KeyError"}, {"warn"}), # in the stack
({"q": "100%"}, {"plain"}), # a literal, not a wildcard
@@ -102,7 +104,14 @@ def test_each_filter_selects_exactly_its_rows(db, admin, make_user, params, expe
job_id=42,
attrs={"code": "forbidden"},
),
_row(msg="warn", level="WARNING", sev=13, event=None, exc="Traceback …\nKeyError: 'x'"),
_row(
msg="warn",
level="WARNING",
sev=13,
event=None,
exc="Traceback …\nKeyError: 'x'",
run_id=7,
),
_row(msg="plain", ts=T0 - timedelta(hours=2), attrs={"note": "100% done"}),
# Matched by no filter above but `until`: another account, and `100%` read as a pattern.
_row(msg="decoy", ts=T0 - timedelta(hours=3), attrs={"note": "1000 done"}, user_id=99),
+2 -10
View File
@@ -25,16 +25,8 @@ from app.config import settings
from app.db import SessionLocal, get_db
from app.main import app
from app.notifications import create_notification
from app.models import (
ApiToken,
ApiTokenGrant,
AuditLog,
DownloadJob,
FileFolder,
MediaAsset,
Notification,
User,
)
from app.models import ApiToken, ApiTokenGrant, AuditLog, Notification, User
from app.downloads.models import DownloadJob, FileFolder, MediaAsset
from app.ratelimit import RateLimiter
from app.routes import tokens as tokens_routes
from app.routes.auth import bump_session_epoch
+1 -1
View File
@@ -1,6 +1,6 @@
"""An admin action that counts something keeps the count as data in its audit row's `after`, under
the key the audit log names in the reader's language (`scheduler.result.<key>`, frontend
auditColumns.tsx) — not only in the English `summary`. A renamed key would fall back to the raw word.
lib/jobResult.ts, called from lib/auditText.ts) — not only in the English `summary`. A renamed key would fall back to the raw word.
DB lane (Postgres).
"""
+295
View File
@@ -0,0 +1,295 @@
"""Capture's service and queries, one use case each, below the HTTP layer.
The routes are thin callers now, and their own tests stay the contract a client sees. These hold
what each write does and leaves on its own: that it flushes and leaves the commit to its caller,
that a row of someone else's reads as a missing one, and the rule each write carries (a hash kept
once, a password change that retires the old grants, a quota floored). DB-backed (real Postgres).
"""
import pytest
from sqlalchemy import select
from app.audit import AuditAction
from app.downloads import folders, links, queries, quota, service
from app.downloads.models import (
DownloadJob,
DownloadLink,
DownloadQuota,
DownloadShare,
FileFolder,
LibraryState,
MediaAsset,
)
from app.models import AuditLog, Notification
from app.userscope import owned_by, owned_row
def _file(db, user, *, sha256=None) -> DownloadJob:
asset = MediaAsset(
source_kind="url",
source_ref=f"https://example.com/{user.id}.mp4",
format_sig=f"sig-{user.id}",
status="ready",
rel_path=f"f-{user.id}.mp4",
ref_count=1,
media_kind="video",
container="mp4",
sha256=sha256,
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=user.id,
asset_id=asset.id,
source_kind="url",
source_ref=asset.source_ref,
status="done",
profile_snapshot={},
display_name="Clip",
)
db.add(job)
db.commit()
return job
def _uncommitted(db):
"""What the caller still has to commit: the session holds pending work."""
return bool(db.new or db.dirty or db.deleted) or db.in_transaction()
# --- whose a row is ------------------------------------------------------------------------
def test_a_row_of_someone_elses_reads_as_a_missing_one(db, make_user):
owner, stranger = make_user(), make_user()
job = _file(db, owner)
assert owned_row(db, DownloadJob, job.id, owner) is job
assert owned_row(db, DownloadJob, job.id, stranger) is None
assert owned_row(db, DownloadJob, job.id + 1000, owner) is None
mine = db.execute(select(DownloadJob.id).where(owned_by(DownloadJob.user_id, owner.id)))
assert mine.scalars().all() == [job.id]
def test_a_file_is_readable_by_its_owner_and_whoever_it_was_shared_with(db, make_user):
owner, friend, stranger = make_user(), make_user(), make_user()
job = _file(db, owner)
assert service.share(db, job, owner, friend) is True
db.commit()
assert queries.accessible_job(db, owner, job.id) is job
assert queries.accessible_job(db, friend, job.id) is job
assert queries.accessible_job(db, stranger, job.id) is None
assert queries.own_job(db, friend, job.id) is None # shared is not owned
def test_the_tray_and_the_library_list_only_the_callers_own_files(db, make_user):
from datetime import datetime, timedelta, timezone
mine, theirs = make_user(), make_user()
own, other = _file(db, mine), _file(db, theirs)
since = datetime.now(timezone.utc) - timedelta(days=1)
for job in (own, other):
job.ready_at = datetime.now(timezone.utc)
db.commit()
assert [j.id for j in queries.queue(db, mine, since)] == [own.id]
library = queries.library(
db,
mine,
folder_id=None,
listable=None,
shared=False,
term="",
sort="added",
order="desc",
lang="en",
)
assert [j.id for j in library] == [own.id]
assert [r for r, _ in queries.statuses_by_source(db, mine, ["done"])] == [own.source_ref]
# --- locks -----------------------------------------------------------------------------------
def test_the_job_lock_answers_none_for_a_job_not_the_callers(db, make_user):
owner, stranger = make_user(), make_user()
job = _file(db, owner)
assert service.lock_own_job(db, stranger, job.id) is None
assert service.lock_own_job(db, owner, job.id) is job
assert service.hold_for_share(db, stranger, job.id) is None
assert service.hold_for_share(db, owner, job.id) is job
# --- profiles --------------------------------------------------------------------------------
def test_a_profile_is_made_after_the_users_others_and_only_theirs_to_change(db, make_user):
user, other = make_user(), make_user()
first = service.create_profile(db, user, "x" * 100, {"kind": "video"})
second = service.create_profile(db, user, "Audio", {"kind": "audio"})
assert _uncommitted(db)
db.commit()
assert first.name == "x" * 80 # names are capped
assert second.sort_order == first.sort_order + 1
assert queries.own_profile(db, other, first.id) is None
service.update_profile(first, name="Video", spec=None)
db.commit()
assert (first.name, first.spec) == ("Video", {"kind": "video"})
service.delete_profile(db, first)
db.commit()
assert [p.id for p in queries.profiles_for(db, user)] == [second.id]
# --- resume points ---------------------------------------------------------------------------
def test_a_resume_point_is_one_row_per_user_and_file_and_a_zero_clears_it(db, make_user):
user = make_user()
job = _file(db, user)
service.set_position(db, user, job.id, 40)
service.set_position(db, user, job.id, 90)
db.commit()
rows = db.execute(select(LibraryState.position_seconds)).scalars().all()
assert rows == [90]
assert queries.positions_for(db, user, [job]) == {job.id: 90}
service.set_position(db, user, job.id, 0)
service.set_position(db, user, job.id, 0) # the beacon's second clear deletes nothing
db.commit()
assert db.execute(select(LibraryState)).first() is None
# --- shares ----------------------------------------------------------------------------------
def test_a_share_is_made_once_with_one_notice_and_undone_by_either_side(db, make_user):
owner, friend = make_user(), make_user()
job = _file(db, owner)
assert service.share(db, job, owner, friend) is True
assert service.share(db, job, owner, friend) is False
assert _uncommitted(db)
db.commit()
notices = db.execute(select(Notification).where(Notification.user_id == friend.id)).scalars()
assert [n.data["job_id"] for n in notices] == [job.id]
assert service.unshare(db, job.id, friend) is True
assert service.unshare(db, job.id, friend) is False
db.commit()
assert db.execute(select(DownloadShare)).first() is None
# --- a file's display fields, its access clock, its hash ------------------------------------
def test_only_display_fields_can_be_set_on_a_file(db, make_user):
job = _file(db, make_user())
service.set_display(job, display_name="New", extra_links=None)
assert job.display_name == "New"
with pytest.raises(ValueError):
service.set_display(job, status="error")
assert job.status == "done"
def test_a_hash_is_kept_once_and_never_overwritten(db, make_user):
job = _file(db, make_user())
service.store_sha256(db, job.asset_id, "a" * 64)
service.store_sha256(db, job.asset_id, "b" * 64)
db.commit()
assert db.get(MediaAsset, job.asset_id).sha256 == "a" * 64
def test_serving_a_file_restarts_its_idle_clock(db, make_user):
job = _file(db, make_user())
asset = queries.asset_of(db, job)
before = asset.last_access_at
service.touch(asset)
assert asset.last_access_at is not None and asset.last_access_at != before
# --- links -----------------------------------------------------------------------------------
def _actions(db) -> list[str]:
return db.execute(select(AuditLog.action).order_by(AuditLog.id)).scalars().all()
def test_a_link_is_made_changed_and_revoked_with_an_audit_row_each(db, make_user):
user = make_user()
job = _file(db, user)
link, token = links.create(
db, user, password="pw", allow_download=False, expires_days=3, job_id=job.id
)
db.commit()
assert links.find(db, token) is link
assert link.expires_at is not None and link.password_hash is not None
version = link.password_version
links.update(db, user, link, password="other", sent=frozenset({"password"}))
db.commit()
assert link.password_version == version + 1 # the old grants stop working
links.update(db, user, link, allow_download=True, sent=frozenset()) # nothing was sent
assert link.allow_download is False
assert [link.id for link in queries.links_to(db, job_id=job.id)] == [link.id]
links.revoke(db, user, link)
db.commit()
assert db.execute(select(DownloadLink)).first() is None
assert _actions(db) == [
AuditAction.DOWNLOAD_LINK_CREATE,
AuditAction.DOWNLOAD_LINK_UPDATE,
AuditAction.DOWNLOAD_LINK_REVOKE,
]
def test_a_link_is_the_callers_only_through_a_file_or_folder_of_theirs(db, make_user):
owner, stranger = make_user(), make_user()
job = _file(db, owner)
link, _ = links.create(
db, owner, password="", allow_download=True, expires_days=None, job_id=job.id
)
db.commit()
assert queries.own_link(db, owner, link.id) is link
assert queries.own_link(db, stranger, link.id) is None
def test_a_view_is_counted_in_sql(db, make_user):
user = make_user()
job = _file(db, user)
link, _ = links.create(
db, user, password="", allow_download=True, expires_days=None, job_id=job.id
)
db.commit()
links.count_view(db, link)
links.count_view(db, link)
db.commit()
db.expire_all()
assert db.get(DownloadLink, link.id).view_count == 2
# --- folders ---------------------------------------------------------------------------------
def test_a_folder_name_is_one_per_user_and_a_source_folder_keeps_its_own(db, make_user):
user, other = make_user(), make_user()
mine = folders.create_own(db, user.id, "Trips")
with pytest.raises(folders.FolderError, match="file_folder_name_taken"):
folders.create_own(db, user.id, "trips")
folders.create_own(db, other.id, "Trips") # another user's namesake is theirs
db.commit()
source = folders.source_folder(db, user.id, folders.SourceKey.UPLOAD)
with pytest.raises(folders.FolderError, match="file_folder_not_own"):
folders.rename(db, source, "Mine now")
folders.rename(db, mine, "Holidays")
db.commit()
assert db.get(FileFolder, mine.id).name == "Holidays"
assert queries.own_folder(db, other, mine.id) is None
# --- quota -----------------------------------------------------------------------------------
def test_a_custom_quota_starts_from_what_applies_and_floors_each_value(db, make_user):
user = make_user()
defaults = quota.resolve(db, user.id)
quota.set_custom(db, user.id, max_concurrent=0, max_bytes=-5)
db.commit()
row = queries.custom_quota(db, user.id)
assert (row.max_concurrent, row.max_bytes, row.max_jobs) == (1, 0, defaults.max_jobs)
assert quota.reset_custom(db, user.id) is True
assert quota.reset_custom(db, user.id) is False
db.commit()
assert db.get(DownloadQuota, user.id) is None
+64 -7
View File
@@ -107,25 +107,48 @@ def _setup_channel_and_sub(db, *, email):
def _patch_deep_requested(db, monkeypatch, override_dep, user):
submitted: list[tuple] = []
calls: list[tuple] = []
# background.add_task runs after the response in TestClient; record what would be scheduled.
# Record the run the route submits, and what its work would call, instead of running it.
monkeypatch.setattr(
channels_route.scheduler,
"run_ad_hoc",
lambda job, fn, **kw: submitted.append((job, kw)) or fn(),
)
monkeypatch.setattr(channels_route, "backfill_channel_now", lambda *a, **k: calls.append(a))
monkeypatch.setattr(state, "_configured_cache", True)
override_dep(get_db, db.info["request"])
override_dep(current_user, lambda: user)
resp = TestClient(app).patch("/api/channels/UC_wire", json={"deep_requested": True})
return resp, calls
return resp, submitted, calls
def test_deep_request_schedules_immediate_backfill(db, monkeypatch, override_dep):
def test_deep_request_starts_the_backfill_as_the_users_run_of_that_channel(
db, monkeypatch, override_dep
):
"""It was a BackgroundTask: what it did, spent or failed with reached only the log."""
ch, me = _setup_channel_and_sub(db, email="wire@example.com")
resp, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
resp, submitted, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
assert resp.status_code == 200, resp.text
assert resp.json()["deep_requested"] is True
# scheduled exactly once, with (channel_id, user_id) — the args backfill_channel_now needs.
assert submitted == [("channel_backfill", {"actor_id": me.id, "target": ch.id})]
# its work runs exactly once, with (channel_id, user_id) — the args backfill_channel_now needs.
assert calls == [(ch.id, me.id)]
def test_turning_full_history_on_for_a_finished_channel_records_no_run(
db, monkeypatch, override_dep
):
"""Its whole history is in already: a run would only say "nothing to do", and switching the flag
off and on again would otherwise write one such row each time, kept for months."""
ch, me = _setup_channel_and_sub(db, email="done@example.com")
ch.backfill_done = True
db.commit()
resp, submitted, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
assert resp.status_code == 200, resp.text
assert (submitted, calls) == ([], [])
class TestImmediateBackfillGuard:
"""backfill_channel_now coordinates with the paced scheduler through _backfill_in_progress (per-
channel mutual exclusion, both directions) + _immediate_running (a cap on interactive runs so a
@@ -148,15 +171,49 @@ class TestImmediateBackfillGuard:
def test_a_second_call_for_an_in_flight_channel_is_a_noop(self, monkeypatch):
runner._backfill_in_progress.add("UC_busy")
monkeypatch.setattr(runner, "SessionLocal", self._boom_session)
runner.backfill_channel_now("UC_busy", 1) # returns immediately; _boom_session never runs
# returns at once — _boom_session never runs — and the run says why it did nothing.
assert runner.backfill_channel_now("UC_busy", 1) == {"skipped": "channel busy"}
assert runner._backfill_in_progress == {"UC_busy"} # the in-flight entry is untouched
def test_over_the_concurrency_cap_falls_back_to_the_scheduler(self, monkeypatch):
runner._immediate_running = runner._MAX_CONCURRENT_IMMEDIATE
monkeypatch.setattr(runner, "SessionLocal", self._boom_session)
runner.backfill_channel_now("UC_new", 1) # over the cap → no-op, scheduler will handle it
# over the cap → no-op, the scheduler will handle it
assert runner.backfill_channel_now("UC_new", 1) == {"skipped": "at capacity"}
assert "UC_new" not in runner._backfill_in_progress
@staticmethod
def _one_channel(db, monkeypatch, cid):
ch = Channel(id=cid, title=cid, uploads_playlist_id="UU" + cid, backfill_done=False)
me = User(email=f"{cid}@example.com")
db.add_all([ch, me])
db.commit()
monkeypatch.setattr(runner, "get_service_user", lambda _db: me)
monkeypatch.setattr(runner.quota, "backfill_reserve_reached", lambda: False)
monkeypatch.setattr(runner, "YouTubeClient", lambda *a, **k: _NullYt())
return ch, me
def test_a_failure_reaches_its_run_and_still_frees_the_channel(self, db, monkeypatch):
"""The failure was logged and swallowed, so the run that now wraps it would read "ok"."""
ch, me = self._one_channel(db, monkeypatch, "UC_fail")
def boom(*_a):
raise RuntimeError("playlistItems 500")
monkeypatch.setattr(runner, "backfill_channel_deep", boom)
with pytest.raises(RuntimeError, match="playlistItems 500"):
runner.backfill_channel_now(ch.id, me.id)
assert runner._backfill_in_progress == set() and runner._immediate_running == 0
def test_a_finished_backfill_says_what_it_added(self, db, monkeypatch):
ch, me = self._one_channel(db, monkeypatch, "UC_ok")
monkeypatch.setattr(runner, "backfill_channel_deep", lambda *_a: 7)
monkeypatch.setattr(runner, "reconcile_full_history", lambda _db: 1)
assert runner.backfill_channel_now(ch.id, me.id) == {
"videos_added": 7,
"reconciled_full": 1,
}
def test_scheduler_deep_pass_skips_an_in_flight_channel(self, db, monkeypatch):
# A channel the scheduler's query WOULD pick (deep-requested, recent-synced, not done)…
ch = Channel(
+2 -1
View File
@@ -18,7 +18,8 @@ from app.downloads import errors as dlerrors
from app.downloads import formats, pipeline, quota, runner, service
from app.downloads import probe as probemod
from app.downloads.edit import INPUT_GUARD
from app.models import Channel, DownloadJob, MediaAsset, PlexItem, PlexLibrary, PlexShow, Video
from app.models import Channel, PlexItem, PlexLibrary, PlexShow, Video
from app.downloads.models import DownloadJob, MediaAsset
from app.plex import paths as plex_paths
# --- the spec -----------------------------------------------------------------------------------
+2 -1
View File
@@ -18,7 +18,8 @@ from cryptography.fernet import Fernet
from app import security
from app.downloads import credentials, formats, service
from app.models import DownloadCredential, DownloadJob, MediaAsset, User
from app.models import User
from app.downloads.models import DownloadCredential, DownloadJob, MediaAsset
@pytest.fixture(autouse=True)
@@ -24,7 +24,8 @@ from app import security
from app.config import settings
from app.db import engine
from app.downloads import credentials, formats, pipeline, service
from app.models import DownloadJob, DownloadQuota, MediaAsset, User
from app.models import User
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
from app.worker import _process_job
_REEL = "https://www.facebook.com/reel/1736360027626346/?fs=e&fs=e"
+3 -1
View File
@@ -18,7 +18,8 @@ from sqlalchemy import select
from app.apitokens import FilesCaller
from app.db import SessionLocal
from app.downloads import service
from app.models import DownloadJob, MediaAsset, User
from app.models import User
from app.downloads.models import DownloadJob, MediaAsset
from app.routes.downloads import cancel_download, pause_download
from app.routes.files import delete_file, own_job_for_update
@@ -88,6 +89,7 @@ def test_overlapping_requests_for_one_job_release_its_hold_once(
# The second request waits for the first one's locks instead of deciding on the same row.
assert not finished.wait(0.5), "the second request did not wait for the first"
transition(first, locked)
first.commit() # the handler's commit: the service only flushes
other.join(10)
assert finished.is_set()
finally:
+4 -2
View File
@@ -13,7 +13,8 @@ its OWN session, so every test COMMITS its setup before calling it.
from app.config import settings
from app.downloads import errors as dlerrors
from app.downloads import formats, pipeline, quota
from app.models import DownloadJob, DownloadQuota, MediaAsset, User
from app.models import User
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
from app.worker import _claim_job
@@ -179,6 +180,7 @@ def test_a_suspended_users_queued_jobs_wait(db):
def test_list_downloads_is_capped(db, monkeypatch):
from app.downloads import queries
from app.routes import downloads as dl
u = User(email="heavy@example.com")
@@ -188,7 +190,7 @@ def test_list_downloads_is_capped(db, monkeypatch):
_job(db, u.id, queue_pos=i, status="done")
db.commit()
monkeypatch.setattr(dl, "MAX_LIST", 2)
monkeypatch.setattr(queries, "MAX_LIST", 2)
rows = dl.list_downloads(user=u, db=db)
assert len(rows) == 2 # bounded by the cap, not the 3 rows on disk
+2 -1
View File
@@ -13,7 +13,8 @@ DB lane (`db` fixture) — skipped without Postgres.
from sqlalchemy import select, text
from app.downloads import service
from app.models import DownloadJob, MediaAsset, User
from app.models import User
from app.downloads.models import DownloadJob, MediaAsset
def _user(db, email="ref@example.com"):
+2 -1
View File
@@ -4,7 +4,8 @@ you") — the only place the title lived, so a Hungarian reader got the English.
from sqlalchemy import select
from app.models import DownloadJob, MediaAsset, Notification
from app.models import Notification
from app.downloads.models import DownloadJob, MediaAsset
def test_a_share_notice_carries_the_title_and_the_sharer(db, make_user, as_user):
+2 -1
View File
@@ -8,7 +8,8 @@ behaviour-equivalently. DB lane (`db` fixture) — skipped without Postgres.
from sqlalchemy import text
from app.models import DownloadJob, MediaAsset, User
from app.models import User
from app.downloads.models import DownloadJob, MediaAsset
def _seed(db, *, job_status, asset_status="ready", ref_count=1, rel_path=None):
+2 -1
View File
@@ -11,7 +11,8 @@ from sqlalchemy import text
from app.downloads import service
from app.downloads.status import JobStatus
from app.models import DownloadJob, User
from app.models import User
from app.downloads.models import DownloadJob
def _running_job(db) -> DownloadJob:
+1 -1
View File
@@ -13,7 +13,7 @@ from app.config import settings
from app.downloads import errors as dlerrors
from app.downloads import pipeline, quota, service
from app.downloads import upload as uploadmod
from app.models import DownloadJob, DownloadQuota, MediaAsset
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
pytestmark = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
+1 -1
View File
@@ -20,7 +20,7 @@ import yt_dlp
from app import logs, sysconfig, worker
from app.downloads import pipeline
from app.downloads.stated import StatedFailure
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
from app.plex import stream as plex_stream
+2 -1
View File
@@ -17,7 +17,8 @@ from app.downloads import folders, formats, service
from app.downloads import gc as gcmod
from app.db import SessionLocal
from app.downloads.status import JobStatus
from app.models import DownloadJob, FileFolder, MediaAsset, Notification
from app.models import Notification
from app.downloads.models import DownloadJob, FileFolder, MediaAsset
NOW = datetime.now(timezone.utc)
+2 -1
View File
@@ -18,7 +18,8 @@ from app.config import settings
from app.downloads import folders, service
from app.downloads import upload as uploadmod
from app.downloads.status import JobStatus
from app.models import AuditLog, DownloadJob, DownloadShare, FileFolder, MediaAsset
from app.models import AuditLog
from app.downloads.models import DownloadJob, DownloadShare, FileFolder, MediaAsset
NOW = datetime.now(timezone.utc)
+1 -1
View File
@@ -11,7 +11,7 @@ from sqlalchemy import select
from app.config import settings
from app.downloads import folders, preview, service
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadShare, MediaAsset
from app.downloads.models import DownloadJob, DownloadShare, MediaAsset
from app.routes.files import TRUNCATED_HEADER
+4 -3
View File
@@ -18,7 +18,8 @@ from app.config import settings
from app.db import engine
from app.downloads import folders, og
from app.downloads import probe as probemod
from app.models import AuditLog, DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.models import AuditLog
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_VIDEO = {
"streams": [
@@ -193,7 +194,7 @@ def test_every_open_counts_as_a_view(shelf, client):
def test_the_folder_list_is_capped_like_the_owners_files_list():
from app.downloads import links as linksmod
from app.routes.files import MAX_LIST
from app.downloads.queries import MAX_LIST
assert linksmod.FOLDER_MAX == MAX_LIST
@@ -317,7 +318,7 @@ def test_a_password_file_link_says_it_opens_a_file(shelf, client):
def test_a_file_whose_bytes_are_gone_is_not_offered(shelf, client, db):
from pathlib import Path
from app.models import MediaAsset
from app.downloads.models import MediaAsset
token = _link(shelf)["token"]
gone = shelf["inside"][0]
+1 -1
View File
@@ -9,7 +9,7 @@ cannot reach, is ever read or written. DB-backed (real Postgres).
import pytest
from sqlalchemy import select
from app.models import DownloadJob, DownloadShare, LibraryState, MediaAsset
from app.downloads.models import DownloadJob, DownloadShare, LibraryState, MediaAsset
from app.routes._common import ProgressIn, kept_position
+2 -1
View File
@@ -18,7 +18,8 @@ from app import security
from app.config import settings
from app.db import engine
from app.downloads import probe as probemod
from app.models import AuditLog, DownloadLink
from app.models import AuditLog
from app.downloads.models import DownloadLink
_VIDEO = {
"streams": [
+165
View File
@@ -0,0 +1,165 @@
"""Migration 0109's data step, DB-lane: the audit log's run rows move into `job_runs` with their
outcome intact, and leave the audit log; an admin action stays where it was. The downgrade moves
them back."""
import json
from datetime import timedelta
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def _audit(conn, actor, action, target, after):
conn.execute(
text(
"INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after,"
" created_at) VALUES (:a, :act, 'job', :t, 's', CAST(:after AS json),"
" CAST('2026-10-01T03:00:00+00:00' AS timestamptz))"
),
{"a": actor, "act": action, "t": target, "after": json.dumps(after)},
)
def test_the_audit_logs_runs_move_into_the_run_table_and_back(db, make_user):
uid = make_user().id
cfg = Config("alembic.ini")
command.downgrade(cfg, "0108_folder_links")
try:
with engine.begin() as conn:
_audit(
conn,
None,
"job_run",
"rss_poll",
{"status": "ok", "summary": "new=3", "result": {"new": 3}},
)
_audit(
conn,
uid,
"job_run",
"rss_poll",
{"status": "error", "summary": "boom", "result": {"failure": "rss_all_failed"}},
)
_audit(
conn, uid, "job_run", "backfill", {"status": "skipped", "summary": "sync paused"}
)
_audit(conn, uid, "config_set", None, {"key": "x"})
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
runs = conn.execute(
text(
"SELECT job, trigger, actor_id, status, summary, result, error, failure,"
" to_char(started_at AT TIME ZONE 'UTC', 'MM-DD HH24'), finished_at = started_at"
" FROM job_runs ORDER BY id"
)
).all()
left = conn.execute(text("SELECT action FROM audit_log")).scalars().all()
assert [tuple(r) for r in runs] == [
("rss_poll", "scheduled", None, "ok", "new=3", {"new": 3}, None, None, "10-01 03", True),
(
"rss_poll",
"manual",
uid,
"error",
None,
None,
"boom",
{"failure": "rss_all_failed"},
"10-01 03",
True,
),
("backfill", "manual", uid, "skipped", "sync paused", None, None, None, "10-01 03", True),
]
assert left == ["config_set"]
command.downgrade(cfg, "0108_folder_links")
try:
with engine.connect() as conn:
back = conn.execute(
text(
"SELECT actor_id, target_id, after::jsonb FROM audit_log"
" WHERE action = 'job_run' ORDER BY id"
)
).all()
finally:
command.upgrade(cfg, "head")
# What the audit row has no column for rides along, so the next upgrade restores it.
def run(trigger: str) -> dict:
return {"trigger": trigger, "quota_units": 0, "started_at": "2026-10-01T03:00:00+00:00"}
assert [tuple(r) for r in back] == [
(
None,
"rss_poll",
{"status": "ok", "summary": "new=3", "result": {"new": 3}, **run("scheduled")},
),
(
uid,
"rss_poll",
{
"status": "error",
"summary": "boom",
"result": {"failure": "rss_all_failed"},
**run("manual"),
},
),
(uid, "backfill", {"status": "skipped", "summary": "sync paused", **run("manual")}),
]
def test_a_down_and_up_round_trip_keeps_every_run(db, make_user):
"""A rollback then a redeploy: an interrupted run, a run caught mid-way and a user's run of one
channel come back as runs — not stranded in the audit log under an action no code knows."""
uid = make_user().id
with engine.begin() as conn:
conn.execute(
text(
"INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at,"
" finished_at, quota_units) VALUES"
" ('backfill', 'scheduled', NULL, NULL, 'interrupted', now(), now(), 0),"
" ('maintenance', 'manual', :u, NULL, 'running', now(), NULL, 0),"
" ('channel_backfill', 'ad_hoc', :u, 'UC_x', 'ok', now(), now(), 0),"
# Its admin is gone (SET NULL): only the row itself still says it was manual.
" ('enrich', 'manual', NULL, NULL, 'ok',"
" '2026-10-01 10:00+00', '2026-10-01 10:30+00', 42)"
),
{"u": uid},
)
# What the run caught mid-way had spent: only its quota rows know it yet.
conn.execute(
text(
"INSERT INTO quota_events (action, units, job_run_id)"
" SELECT 'videos.list', 7, id FROM job_runs WHERE job = 'maintenance'"
)
)
cfg = Config("alembic.ini")
command.downgrade(cfg, "0108_folder_links")
command.upgrade(cfg, "head")
with engine.connect() as conn:
runs = conn.execute(
text("SELECT job, trigger, actor_id, target, status FROM job_runs ORDER BY id")
).all()
kept = conn.execute(
text("SELECT quota_units, finished_at - started_at FROM job_runs WHERE job = 'enrich'")
).one()
stopped = conn.execute(
text("SELECT quota_units FROM job_runs WHERE job = 'maintenance'")
).scalar()
stranded = conn.execute(
text("SELECT count(*) FROM audit_log WHERE action = 'job_run'")
).scalar()
assert [tuple(r) for r in runs] == [
("backfill", "scheduled", None, None, "interrupted"),
("maintenance", "manual", uid, None, "interrupted"), # the rollback stopped it
("channel_backfill", "ad_hoc", uid, "UC_x", "ok"),
("enrich", "manual", None, None, "ok"),
]
assert tuple(kept) == (42, timedelta(minutes=30))
assert stopped == 7
assert stranded == 0
+56
View File
@@ -0,0 +1,56 @@
"""Migration 0110's data step, DB-lane: an admin's interval override moves from `scheduler_settings`
into sysconfig and back, a row for a job that no longer exists is not carried over, and the unread
SponsorBlock default leaves the store.
The schema half is checked by test_migrations.py; a value lost on the way is invisible there.
"""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app import scheduler
from app.db import engine
def _config(conn) -> dict[str, str]:
return dict(conn.execute(text("SELECT key, value FROM system_config")).tuples().all())
def test_an_interval_override_moves_into_sysconfig_and_back(db):
cfg = Config("alembic.ini")
command.downgrade(cfg, "0109_job_runs")
try:
with engine.begin() as conn:
conn.execute(
text(
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
" VALUES ('rss_poll', 25), ('log_gc', 90), ('retired_job', 5)"
)
)
conn.execute(
text(
"INSERT INTO system_config (key, value, encrypted) VALUES"
" ('sponsorblock_default', 'true', false), ('quota_daily_budget', '5000', false)"
)
)
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
assert _config(conn) == {
"job_interval_rss_poll": "25",
"job_interval_log_gc": "90",
"quota_daily_budget": "5000",
}
assert conn.execute(text("SELECT to_regclass('scheduler_settings')")).scalar() is None
assert scheduler.load_intervals(db)["rss_poll"] == 25
command.downgrade(cfg, "0109_job_runs")
try:
with engine.connect() as conn:
back = conn.execute(text("SELECT job_id, interval_minutes FROM scheduler_settings"))
assert dict(back.tuples().all()) == {"rss_poll": 25, "log_gc": 90}
assert _config(conn) == {"quota_daily_budget": "5000"}
finally:
command.upgrade(cfg, "head")
+1 -1
View File
@@ -7,7 +7,7 @@ the shared one, so every other owner check kept the English. Needs the DB.
"""
from app.auth import require_write_scope
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
def test_a_gone_view_tag_or_pin_says_what_is_gone(db, make_user, as_user):
+1 -1
View File
@@ -17,7 +17,7 @@ import pytest
from app.config import settings
from app.downloads import pipeline, playable, probe, service, storage
from app.downloads import upload as uploadmod
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
pytestmark = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
+1 -1
View File
@@ -17,7 +17,7 @@ import pytest
from app.config import settings
from app.downloads import pipeline, playable, probe
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
APP = Path(__file__).resolve().parents[1] / "app"
-7
View File
@@ -9,7 +9,6 @@ from datetime import datetime, timezone
import httpx
import pytest
from app import scheduler
from app.models import PlexItem, PlexLibrary, PlexPerson, User
from app.plex import imgcache
from app.plex import people as plex_people
@@ -340,12 +339,6 @@ def test_the_people_job_runs_its_halves_without_a_tmdb_key(db, cache_dir, plex_o
}
def test_a_phase_count_is_a_change_worth_an_audit_row():
assert scheduler._run_changed({"bios": {"enriched": 3}, "portraits": {"cached": 0}})
assert scheduler._run_changed({"recent": {"videos_added": 2}, "deep": {}})
assert not scheduler._run_changed({"bios": {"skipped": "x"}, "portraits": {"cached": 0}})
def test_the_person_page_says_a_bio_is_on_its_way(db, monkeypatch, override_dep):
from fastapi.testclient import TestClient
+27
View File
@@ -0,0 +1,27 @@
"""The run words and the migrated database, DB-lane: the `job_runs` CHECKs are frozen in a migration,
and the model's are built from the enums, so nothing else would notice the two drifting apart until a
write of the new word failed in production."""
import re
from sqlalchemy import text
from app.runstatus import RunStatus, RunTrigger
def _words(definition: str) -> set[str]:
return set(re.findall(r"'([a-z_]+)'", definition))
def test_the_databases_checks_allow_exactly_the_run_words(db):
defs = dict(
db.execute(
text(
"SELECT conname, pg_get_constraintdef(oid) FROM pg_constraint"
" WHERE conname LIKE 'ck_job_runs_%'"
)
).all()
)
assert _words(defs["ck_job_runs_status"]) == set(RunStatus)
assert _words(defs["ck_job_runs_trigger"]) == set(RunTrigger)
assert _words(defs["ck_job_runs_finished"]) == {RunStatus.RUNNING}
+534 -47
View File
@@ -1,11 +1,21 @@
"""_claim_running is the single-run guard that both the scheduled fire and the manual trigger go
"""The scheduler's run path, DB-lane: the single-run guard, and every run as a durable `job_runs` row
— its outcome as data, what it spent, the lines it logged — that outlives the process.
_claim_running is the single-run guard that both the scheduled fire and the manual trigger go
through (at the top of _job): holding _activity_lock across the check-and-set means exactly one
caller can hold a job's "running" claim at a time, and it's claimable again once released."""
import logging
import time
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import select
from app import scheduler
from app.models import Notification
from app import jobruns, logs, logsink, quota, scheduler, state, sysconfig
from app.models import AuditLog, Channel, JobRun, Notification, QuotaEvent
from app.routes import scheduler as scheduler_routes
from app.runstatus import RunStatus, RunTrigger
_JOB = "unit_test_claim" # not a real job id — exercises the primitive without touching live jobs
@@ -31,26 +41,38 @@ def test_claim_is_exclusive_until_released():
# ---- the result as data --------------------------------------------------------------------------
def _snapshot_job(job_id):
return next(j for j in scheduler.scheduler_snapshot()["jobs"] if j["id"] == job_id)
def _snapshot_job(db, job_id):
return next(j for j in scheduler.scheduler_snapshot(db)["jobs"] if j["id"] == job_id)
def _runs(db, job=None) -> list[JobRun]:
db.expire_all()
q = select(JobRun).order_by(JobRun.id)
if job is not None:
q = q.where(JobRun.job == job)
return list(db.execute(q).scalars())
def _manual(actor_id, name, fn):
token = scheduler._manual_actor.set(actor_id)
try:
scheduler._job(name, fn)
finally:
scheduler._manual_actor.reset(token)
def test_a_run_keeps_its_result_as_data_for_the_readers_language(db, make_user):
"""The dashboard and the manual-run notice showed `_summarize`'s English one-liner ("new=3,
failed=0", a Python repr for a nested phase). The result rides along as data, so the client names
it; the one-liner stays as the fallback and the audit text."""
it; the one-liner stays as the fallback."""
admin = make_user("sched@example.com", role="admin")
token = scheduler._manual_actor.set(admin.id)
try:
scheduler._job("rss_poll", lambda _db: {"new": 3, "failed": 0, "fallback": 1})
scheduler._job("enrich", lambda _db: 5) # enrich returns a bare count
finally:
scheduler._manual_actor.reset(token)
_manual(admin.id, "rss_poll", lambda _db: {"new": 3, "failed": 0, "fallback": 1})
_manual(admin.id, "enrich", lambda _db: 5) # enrich returns a bare count
rss = _snapshot_job("rss_poll")
rss = _snapshot_job(db, "rss_poll")
assert rss["result"] == {"new": 3, "failed": 0, "fallback": 1}
assert rss["last_result"] == "new=3, failed=0, fallback=1"
assert _snapshot_job("enrich")["result"] == {"enriched": 5}
assert _snapshot_job(db, "enrich")["result"] == {"enriched": 5}
notices = db.execute(
select(Notification).where(Notification.user_id == admin.id).order_by(Notification.id)
@@ -64,53 +86,518 @@ def test_a_run_keeps_its_result_as_data_for_the_readers_language(db, make_user):
def test_a_failed_run_drops_the_previous_result(db):
scheduler._job("rss_poll", lambda _db: {"new": 1})
scheduler._job("rss_poll", lambda _db: (_ for _ in ()).throw(RuntimeError("boom")))
rss = _snapshot_job("rss_poll")
rss = _snapshot_job(db, "rss_poll")
assert rss["status"] == "error"
assert rss["last_error"] == "boom"
assert rss["result"] is None
def test_a_job_runs_audit_row_carries_its_result_as_data(db):
"""The audit log names a run's result in the reader's language, as the inbox does —
from data, not from the English one-liner (a nested phase there was a Python dict repr)."""
from app.models import AuditLog, User
admin = User(email="auditrun@example.com")
db.add(admin)
db.commit()
scheduler._audit_run(db, None, "people_enrich", "ok", "x", {"bios": {"enriched": 2}})
scheduler._audit_run(db, admin.id, "rss_poll", "skipped", "sync paused", scheduler._PAUSED)
scheduler._audit_run(db, admin.id, "rss_poll", "error", "boom", None)
rows = db.execute(select(AuditLog).order_by(AuditLog.id)).scalars().all()
assert [r.after for r in rows] == [
{"status": "ok", "summary": "x", "result": {"bios": {"enriched": 2}}},
{"status": "skipped", "summary": "sync paused", "result": {"skipped": "sync paused"}},
{"status": "error", "summary": "boom"},
]
def test_a_failure_the_job_names_rides_along_as_data(db, make_user):
"""A failed RSS poll reached the Hungarian audit log as the server's English sentence. A
failure the job names (`progress.JobFailed`) is stored as data beside that sentence — on the
dashboard, the manual-run notice and the audit row — so the client says it in the reader's
language. Any other failure keeps only its own text (the test above)."""
"""A failed RSS poll reached the Hungarian UI as the server's English sentence. A failure the job
names (`progress.JobFailed`) is stored as data beside that sentence — on the run's row, the
dashboard and the manual-run notice — so the client says it in the reader's language."""
from app import progress
from app.models import AuditLog
def outage(_db):
raise progress.JobFailed("rss_all_failed", "RSS poll failed", total=2, cause="transport")
admin = make_user("failure@example.com", role="admin")
token = scheduler._manual_actor.set(admin.id)
try:
scheduler._job("rss_poll", outage)
finally:
scheduler._manual_actor.reset(token)
_manual(admin.id, "rss_poll", outage)
named = {"failure": "rss_all_failed", "total": 2, "cause": "transport"}
rss = _snapshot_job("rss_poll")
rss = _snapshot_job(db, "rss_poll")
assert rss["last_error"] == "RSS poll failed"
assert rss["result"] == named
notice = db.execute(select(Notification).where(Notification.user_id == admin.id)).scalar_one()
assert notice.data["result"] == named
row = db.execute(select(AuditLog).order_by(AuditLog.id.desc())).scalars().first()
assert row is not None and row.after["result"] == named
(run,) = _runs(db, "rss_poll")
assert (run.status, run.error, run.failure) == ("error", "RSS poll failed", named)
# ---- the durable row -----------------------------------------------------------------------------
def test_every_run_is_a_row_with_who_started_it_and_none_in_the_audit_log(db, make_user):
"""A no-op scheduled poll was not written anywhere (the audit rule kept only changes), so "did
it run at 03:00?" had no answer. Every run is a row now; the audit log keeps admin actions only."""
admin = make_user("runs@example.com", role="admin")
scheduler._job("rss_poll", lambda _db: {"new": 0})
_manual(admin.id, "rss_poll", lambda _db: {"new": 2})
scheduled, manual = _runs(db, "rss_poll")
assert (scheduled.trigger, scheduled.actor_id, scheduled.status) == ("scheduled", None, "ok")
assert scheduled.result == {"new": 0}
assert (manual.trigger, manual.actor_id, manual.summary) == ("manual", admin.id, "new=2")
assert manual.finished_at is not None and manual.finished_at >= manual.started_at
assert db.execute(select(AuditLog)).scalars().all() == []
def test_the_last_run_survives_a_restart(db):
"""The dashboard's "last run" lived in the process's memory: a deploy blanked every job."""
scheduler._job("plex_sync", lambda _db: {"items": 4})
with scheduler._activity_lock:
scheduler._activity.clear() # what a new process starts with
plex = _snapshot_job(db, "plex_sync")
assert (plex["status"], plex["result"], plex["last_trigger"]) == (
"ok",
{"items": 4},
"scheduled",
)
assert plex["last_finished"] is not None
def test_a_row_shows_its_last_three_runs_newest_first_an_interrupted_one_included(db):
"""The row said only how the LAST run went; an interrupted run read as "no run recorded"."""
oldest = jobruns.start("shorts", RunTrigger.SCHEDULED, None)
jobruns.interrupt_leftovers()
scheduler._job("shorts", lambda _db: {"probed": 1})
scheduler._job("shorts", lambda _db: {"probed": 2})
scheduler._job("shorts", lambda _db: 1 / 0)
scheduler._job("rss_poll", lambda _db: {"new": 0}) # another job's run stays on its own row
recent = _snapshot_job(db, "shorts")["recent"]
assert [r["status"] for r in recent] == ["error", "ok", "ok"]
assert oldest not in [r["id"] for r in recent] # the fourth-newest is not shown
assert recent[0]["trigger"] == "scheduled" and recent[0]["finished_at"] is not None
assert [r["status"] for r in _snapshot_job(db, "rss_poll")["recent"]] == ["ok"]
jobruns.start("shorts", RunTrigger.SCHEDULED, None)
jobruns.interrupt_leftovers()
shorts = _snapshot_job(db, "shorts")
assert shorts["status"] == "interrupted"
assert [r["status"] for r in shorts["recent"]] == ["interrupted", "error", "ok"]
def test_a_paused_sync_is_a_skipped_run(db):
state.set_sync_paused(db, True)
db.commit()
scheduler._job("rss_poll", lambda _db: pytest.fail("a paused run must not do its work"))
(run,) = _runs(db, "rss_poll")
assert (run.status, run.result) == ("skipped", {"skipped": "sync paused"})
def test_a_run_left_running_by_a_stopped_process_is_interrupted_and_stays_so(db):
"""A deploy mid-run left nothing behind before; now the row would say `running` forever. The
next startup closes it, and the stale thread's own finish cannot rewrite it."""
run_id = jobruns.start("backfill", RunTrigger.SCHEDULED, None)
assert jobruns.interrupt_leftovers() == 1
(run,) = _runs(db, "backfill")
assert run.status == "interrupted" and run.finished_at is not None
jobruns.finish(run_id, RunStatus.OK, summary="late")
(run,) = _runs(db, "backfill")
assert (run.status, run.summary) == ("interrupted", None)
def test_a_run_knows_what_it_spent_and_a_request_spend_is_nobodys_run(db):
def spends(_db):
quota.record_usage(5)
quota.record_usage(3)
return {"enriched": 2}
scheduler._job("enrich", spends)
quota.record_usage(11) # a request's own spend, outside any run
(run,) = _runs(db, "enrich")
assert run.quota_units == 8
events = db.execute(select(QuotaEvent.units, QuotaEvent.job_run_id)).all()
assert sorted(events) == [(3, run.id), (5, run.id), (11, None)]
assert _snapshot_job(db, "enrich")["quota_units"] == 8
class _Collect(logging.Handler):
def __init__(self):
super().__init__()
self.addFilter(logs.ContextFilter())
self.records: list[logging.LogRecord] = []
def emit(self, record):
self.records.append(record)
def test_a_runs_log_lines_carry_its_id(db):
"""So a run's row can open exactly the lines it logged (the log table's `run_id`)."""
probe = logging.getLogger("siftlode.test.run")
handler = _Collect()
probe.addHandler(handler)
probe.setLevel(logging.INFO)
try:
scheduler._job("autotag", lambda _db: probe.info("tagging") or {"tagged": 1})
probe.info("after the run")
finally:
probe.removeHandler(handler)
(run,) = _runs(db, "autotag")
inside, after = (logs.record_fields(r).get("run_id") for r in handler.records)
assert (inside, after) == (run.id, None)
# …and it reaches the stored row, whose columns are the model's.
assert logsink.to_row(logs.record_fields(handler.records[0]))["run_id"] == run.id
def test_retention_reaps_finished_runs_only(db):
old = datetime.now(timezone.utc) - timedelta(days=40)
db.add_all(
[
JobRun(
job="rss_poll",
trigger="scheduled",
status="ok",
started_at=old,
finished_at=old,
),
JobRun(job="backfill", trigger="scheduled", status="running", started_at=old),
JobRun(
job="rss_poll",
trigger="scheduled",
status="ok",
started_at=datetime.now(timezone.utc),
finished_at=datetime.now(timezone.utc),
),
]
)
db.commit()
reaped = next(r for r in _runs(db) if r.status == "ok" and r.started_at < old + timedelta(1))
with logs.bind(run_id=reaped.id):
quota.record_usage(7)
assert jobruns.gc(db, 0) == 0 # 0 = keep forever
assert jobruns.gc(db, 30) == 1
db.commit()
assert sorted((r.job, r.status) for r in _runs(db)) == [
("backfill", "running"),
("rss_poll", "ok"),
]
# Its spend stays in the budget's history (kept on its own retention), no longer anyone's run.
assert db.execute(select(QuotaEvent.units, QuotaEvent.job_run_id)).all() == [(7, None)]
# ---- "run now" through the scheduler --------------------------------------------------------------
@pytest.fixture
def live_scheduler(db, monkeypatch):
"""This process's scheduler, started as the API starts it (on the migrated schema `db` brings),
with interval jobs off."""
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", False)
scheduler.start_scheduler()
try:
yield scheduler._scheduler
finally:
scheduler.shutdown_scheduler()
def _wait_finished(db, job, timeout=10.0) -> JobRun:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
runs = [r for r in _runs(db, job) if r.status != "running"]
if runs:
return runs[-1]
time.sleep(0.05)
raise AssertionError(f"{job} did not finish")
def test_run_now_goes_through_the_scheduler_even_with_interval_jobs_off(
db, make_user, live_scheduler, monkeypatch
):
admin = make_user("now@example.com", role="admin")
monkeypatch.setitem(
scheduler.JOB_FUNCS, "shorts", lambda: scheduler._job("shorts", lambda _db: {"probed": 3})
)
assert live_scheduler.get_jobs() == [] # SCHEDULER_ENABLED=false: no interval job
live_scheduler.pause() # hold the one-shot so the second click finds it pending
assert scheduler.trigger_job("shorts", admin.id) == "started"
assert scheduler.trigger_job("shorts", admin.id) == "already_running"
assert [j.id for j in live_scheduler.get_jobs()] == ["shorts@now"]
live_scheduler.resume()
run = _wait_finished(db, "shorts")
assert (run.trigger, run.actor_id, run.result) == ("manual", admin.id, {"probed": 3})
def test_an_ad_hoc_run_is_the_users_run_of_its_target_with_its_spend_and_failure(
db, make_user, live_scheduler
):
"""The one-channel deep backfill ran as a BackgroundTask nothing recorded: its spend counted as
nobody's run and a failure lived only in the log."""
user = make_user("adhoc@example.com")
def work():
quota.record_usage(3)
return {"videos_added": 50}
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, work, actor_id=user.id, target="UC_a")
run = _wait_finished(db, "channel_backfill")
assert (run.trigger, run.actor_id, run.target) == ("ad_hoc", user.id, "UC_a")
assert (run.status, run.result, run.quota_units) == ("ok", {"videos_added": 50}, 3)
def fail():
raise RuntimeError("quota exceeded")
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, fail, actor_id=user.id, target="UC_b")
deadline = time.monotonic() + 10
while time.monotonic() < deadline and len(_finished(db, "channel_backfill")) < 2:
time.sleep(0.05)
failed = next(r for r in _runs(db, "channel_backfill") if r.target == "UC_b")
assert (failed.status, failed.error) == ("error", "quota exceeded")
def test_the_page_shows_the_ad_hoc_runs_beside_the_jobs_with_their_channel(db, make_user):
user = make_user("shown@example.com")
db.add(Channel(id="UC_shown", title="BBC Earth"))
db.commit()
scheduler._ad_hoc("channel_backfill", lambda: {"skipped": "at capacity"}, user.id, "UC_none")
scheduler._ad_hoc("channel_backfill", lambda: {"videos_added": 69}, user.id, "UC_shown")
(row,) = scheduler.scheduler_snapshot(db)["ad_hoc"]
assert (row["id"], row["running"], row["target"], row["target_title"]) == (
"channel_backfill",
False,
"UC_shown",
"BBC Earth",
)
assert (row["status"], row["result"], row["last_trigger"]) == (
"ok",
{"videos_added": 69},
"ad_hoc",
)
assert len(row["recent"]) == 2
assert all(j["id"] != "channel_backfill" for j in scheduler.scheduler_snapshot(db)["jobs"])
seen = []
scheduler._ad_hoc(
"channel_backfill",
lambda: seen.append(scheduler.scheduler_snapshot(db)["ad_hoc"][0]["running"]) or {},
user.id,
"UC_shown",
)
assert seen == [True]
assert scheduler.scheduler_snapshot(db)["ad_hoc"][0]["running"] is False
def _finished(db, job) -> list[JobRun]:
return [r for r in _runs(db, job) if r.status != "running"]
def test_an_ad_hoc_job_outside_the_named_list_is_refused(live_scheduler):
"""The page names every ad-hoc job from AD_HOC_JOBS; one outside it would show as a bare id."""
with pytest.raises(KeyError):
scheduler.run_ad_hoc("unnamed_job", dict, actor_id=1, target="x")
assert live_scheduler.get_jobs() == []
def test_a_second_ad_hoc_request_before_the_first_starts_is_that_same_run(
db, make_user, live_scheduler
):
user = make_user("twice@example.com")
live_scheduler.pause() # hold the one-shots so the second request finds the first pending
for target in ("UC_x", "UC_x", "UC_y"):
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, dict, actor_id=user.id, target=target)
assert sorted(j.id for j in live_scheduler.get_jobs()) == [
"channel_backfill@UC_x",
"channel_backfill@UC_y",
]
live_scheduler.remove_all_jobs() # never run: a run finishing after the test would land in the next
def test_a_manual_run_leaves_no_actor_behind_on_its_thread(db, make_user):
"""The scheduler's pool threads are reused: an actor a manual run left set made the next
scheduled run on that thread an admin's "manual" run, with a notice in their inbox."""
admin = make_user("pool@example.com", role="admin")
seen = []
scheduler._as_actor(lambda: seen.append(scheduler._manual_actor.get()), admin.id)()
assert seen == [admin.id]
assert scheduler._manual_actor.get() is None # this same thread's next task runs as nobody
def test_run_now_refuses_a_job_already_running(live_scheduler):
assert scheduler._claim_running("maintenance") is True
try:
assert scheduler.trigger_job("maintenance") == "already_running"
finally:
scheduler._record("maintenance", running=False)
def test_startup_closes_what_the_previous_process_left_running(db, monkeypatch):
jobruns.start("plex_sync", RunTrigger.MANUAL, None)
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", False)
scheduler.start_scheduler()
scheduler.shutdown_scheduler()
(run,) = _runs(db, "plex_sync")
assert run.status == "interrupted"
def test_an_interrupted_run_keeps_what_it_spent(db):
"""Its spend is in the quota history, tagged with its id: the sweep sums it as `finish` would."""
run_id = jobruns.start("backfill", RunTrigger.SCHEDULED, None)
with logs.bind(run_id=run_id):
quota.record_usage(40)
quota.record_usage(2)
assert jobruns.interrupt_leftovers() == 1
(run,) = _runs(db, "backfill")
assert (run.status, run.quota_units) == ("interrupted", 42)
def _flaky_finish(monkeypatch) -> list:
"""`jobruns.finish` whose first write fails (the database blinked); the statuses it was asked for."""
calls: list = []
real = jobruns.finish
def finish(run_id, status, **outcome):
calls.append(status)
if len(calls) == 1:
raise RuntimeError("db blip")
real(run_id, status, **outcome)
monkeypatch.setattr(jobruns, "finish", finish)
return calls
def test_a_run_whose_outcome_cannot_be_written_is_not_called_failed(db, make_user, monkeypatch):
"""The job succeeded; only its bookkeeping did not. Writing ERROR then would say the work failed."""
calls = _flaky_finish(monkeypatch)
scheduler._job("autotag", lambda _db: {"tagged": 1})
assert calls == ["ok"]
assert not scheduler._is_running("autotag")
(run,) = _runs(db, "autotag")
assert run.status == "running" # left for the next start's sweep, not rewritten as a failure
# Nothing here runs it: the page reads it as the sweep will close it, not as a run going on.
row = _row(db, "autotag")
assert (row["running"], row["status"], row["recent"][0]["status"]) == (
False,
"interrupted",
"interrupted",
)
assert row["last_finished"] is not None
user = make_user("blip@example.com")
calls.clear()
scheduler._ad_hoc("channel_backfill", lambda: {"videos_added": 1}, user.id, "UC_blip")
assert calls == ["ok"]
# Its row is still `running`, but nothing here is: the page does not call it going.
(row,) = scheduler.scheduler_snapshot(db)["ad_hoc"]
assert (row["running"], row["status"], row["recent"][0]["status"]) == (
False,
"interrupted",
"interrupted",
)
def test_a_run_going_on_reads_as_running_on_its_row(db):
seen = []
scheduler._job("autotag", lambda _db: seen.append(_row(db, "autotag")) or {"tagged": 1})
(row,) = seen
assert (row["running"], row["status"], row["recent"][0]["status"]) == (True, None, "running")
assert _row(db, "autotag")["recent"][0]["status"] == "ok"
def _row(db, job_id: str) -> dict:
return next(j for j in scheduler.scheduler_snapshot(db)["jobs"] if j["id"] == job_id)
def test_a_run_that_cannot_be_recorded_does_not_wedge_its_job(db, monkeypatch):
def broken(*_a, **_k):
raise RuntimeError("db down")
monkeypatch.setattr(jobruns, "start", broken)
scheduler._job("autotag", lambda _db: pytest.fail("no row, no run"))
assert not scheduler._is_running("autotag")
monkeypatch.undo()
scheduler._job("autotag", lambda _db: {"tagged": 1})
assert [r.status for r in _runs(db, "autotag")] == ["ok"]
def test_the_page_says_where_the_interval_jobs_do_not_run(db, live_scheduler, monkeypatch):
"""The scheduler starts on every install now; only SCHEDULER_ENABLED says the jobs run here."""
assert scheduler.scheduler_snapshot(db)["running_here"] is False
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", True)
assert scheduler.scheduler_snapshot(db)["running_here"] is True
def test_an_ad_hoc_run_still_going_shows_although_a_newer_one_ended(db, make_user):
"""Two channels can backfill at once; a later run that ended at once must not hide the first."""
user = make_user("two@example.com")
db.add(Channel(id="UC_long", title="Long one"))
db.commit()
rows = []
def long_one():
scheduler._ad_hoc(
"channel_backfill", lambda: {"skipped": "at capacity"}, user.id, "UC_short"
)
rows.extend(scheduler.scheduler_snapshot(db)["ad_hoc"])
return {}
scheduler._ad_hoc("channel_backfill", long_one, user.id, "UC_long")
(row,) = rows
assert (row["running"], row["target"], row["target_title"]) == (True, "UC_long", "Long one")
# The outcome beside it is the long one's (none yet), never the short one's skip.
assert (row["status"], row["result"], row["last_error"]) == (None, None, None)
assert [r["status"] for r in row["recent"]] == ["running", "ok"]
def test_an_interval_change_lands_with_its_audit_row_or_not_at_all(
db, make_user, as_user, live_scheduler, monkeypatch
):
live_scheduler.add_job(lambda: None, "interval", minutes=10, id="backfill")
admin = as_user(make_user(role="admin"))
def broken(*_a, **_k):
raise RuntimeError("audit write failed")
monkeypatch.setattr(scheduler_routes.audit, "record", broken)
with pytest.raises(RuntimeError): # the handler raises: there is no response to check
_ = admin.patch("/api/admin/scheduler/jobs/backfill", json={"interval_minutes": 45})
db.expire_all()
assert sysconfig.get_int(db, "job_interval_backfill") == scheduler.JOB_INTERVALS["backfill"]
assert live_scheduler.get_job("backfill").trigger.interval == timedelta(minutes=10)
def test_every_job_has_an_interval_and_every_interval_a_job():
assert set(scheduler.JOB_FUNCS) == set(scheduler.JOB_INTERVALS)
def test_a_stored_interval_is_what_the_job_starts_with(db, monkeypatch):
sysconfig.set_value(db, "job_interval_rss_poll", 25)
db.commit()
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", True)
scheduler.start_scheduler()
try:
sched = scheduler._scheduler
assert sched is not None
assert sched.get_job("rss_poll").trigger.interval == timedelta(minutes=25)
default = scheduler.JOB_INTERVALS["enrich"]
assert sched.get_job("enrich").trigger.interval == timedelta(minutes=default)
finally:
scheduler.shutdown_scheduler()
def test_an_interval_set_on_the_scheduler_page_is_stored_and_reschedules_the_job(
db, make_user, as_user, live_scheduler
):
live_scheduler.add_job(lambda: None, "interval", minutes=10, id="backfill")
admin = as_user(make_user(role="admin"))
r = admin.patch("/api/admin/scheduler/jobs/backfill", json={"interval_minutes": 45})
assert r.status_code == 200, r.text
assert live_scheduler.get_job("backfill").trigger.interval == timedelta(minutes=45)
db.expire_all()
assert sysconfig.get_int(db, "job_interval_backfill") == 45
assert scheduler.load_intervals(db)["backfill"] == 45
# The Configuration page neither lists an interval nor writes one past the reschedule.
assert (
admin.patch("/api/admin/config/job_interval_backfill", json={"value": 5}).status_code == 404
)
assert admin.delete("/api/admin/config/job_interval_backfill").status_code == 404
assert "scheduler" not in admin.get("/api/admin/config").json()["groups"]
db.expire_all()
assert sysconfig.get_int(db, "job_interval_backfill") == 45
def test_the_setup_wizard_does_not_write_an_interval(db):
from fastapi import HTTPException
from app.routes.setup import setup_config
with pytest.raises(HTTPException) as refused:
setup_config({"job_interval_rss_poll": 5}, db)
assert refused.value.status_code == 400
assert setup_config({"quota_daily_budget": 5000}, db)["saved"] == ["quota_daily_budget"]
+81 -1
View File
@@ -15,10 +15,14 @@ boot, sysconfig holds a stored override to it, and the admin page renders a pick
The DB lane (`db` fixture) covers the store; the rest is pure logic.
"""
import ast
from pathlib import Path
import pytest
from pydantic import ValidationError
from app import sysconfig
from app.models import SystemConfig
from app.config import Settings, choices_of
CHOICE_KEYS = {s.key for s in sysconfig.SPECS if s.type == "choice"}
@@ -37,7 +41,8 @@ def test_each_group_is_one_run_of_keys():
def test_every_group_sits_in_exactly_one_section():
listed = [g for _, members in sysconfig.SECTIONS for g in members]
assert len(listed) == len(set(listed)), listed
assert set(listed) == {s.group for s in sysconfig.SPECS}
assert set(listed) | set(sysconfig.OWN_PAGE_GROUPS) == {s.group for s in sysconfig.SPECS}
assert not set(listed) & set(sysconfig.OWN_PAGE_GROUPS)
assert [sid for sid, _ in sysconfig.SECTIONS] == ["access", "youtube", "media", "system"]
@@ -112,3 +117,78 @@ def test_set_value_stores_the_canonical_spelling_and_refuses_anything_else(db):
with pytest.raises(ValueError, match="scrape, api"):
sysconfig.set_value(db, "search_source", "bing")
db.rollback()
# --- when a changed value takes effect --------------------------------------------------------------
_APP = Path(__file__).resolve().parent.parent / "app"
_READERS = {"get", "get_str", "get_int", "get_bool"}
def _key_reads() -> list[tuple[str, str, bool]]:
"""Every `sysconfig.get*(db, "<key>")` in the app outside the registry: (key, where, inside a
function)."""
found: list[tuple[str, str, bool]] = []
def walk(node: ast.AST, path: Path, in_fn: bool) -> None:
for child in ast.iter_child_nodes(node):
fn = in_fn or isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda))
if (
isinstance(child, ast.Call)
and isinstance(child.func, ast.Attribute)
and isinstance(child.func.value, ast.Name)
and child.func.value.id == "sysconfig"
and child.func.attr in _READERS
and len(child.args) >= 2
and isinstance(child.args[1], ast.Constant)
and isinstance(child.args[1].value, str)
):
found.append((child.args[1].value, f"{path.name}:{child.lineno}", in_fn))
walk(child, path, fn)
for path in _APP.rglob("*.py"):
if path.name != "sysconfig.py":
walk(ast.parse(path.read_text(encoding="utf-8")), path, False)
return found
def test_every_key_states_when_it_applies_and_none_needs_a_restart_yet():
with pytest.raises(TypeError):
sysconfig.ConfigSpec("k", "int", "quota", "quota_daily_budget") # pyright: ignore[reportCallIssue]
# The Configuration page cannot tell the admin that a setting waits for a restart; a key that
# does needs that first (a badge on it and a banner once it changed).
assert {s.applies for s in sysconfig.SPECS} == {"live"}
def test_a_live_key_is_read_where_it_is_used_never_at_import():
reads = _key_reads()
assert len(reads) > 50, len(
reads
) # the scan sees the reads (it would pass vacuously otherwise)
assert [(k, at) for k, at, in_fn in reads if not in_fn] == []
def test_every_key_on_the_configuration_page_has_a_reader():
read = {k for k, _, _ in _key_reads()}
page = {s.key for s in sysconfig.SPECS if s.group not in sysconfig.OWN_PAGE_GROUPS}
assert sorted(page - read) == []
def test_the_intervals_are_keys_of_the_scheduler_page_only():
keys = {s.key for s in sysconfig.SPECS if s.group == "scheduler"}
assert keys == {sysconfig.interval_key(j) for j in sysconfig.JOB_INTERVAL_DEFAULTS}
assert all(len(k) <= 64 for k in keys) # system_config.key
assert sysconfig.config_page_spec("job_interval_rss_poll") is None
assert sysconfig.config_page_spec("quota_daily_budget") is not None
assert "scheduler" not in sysconfig.describe(_FakeDb())["groups"]
def test_saving_the_value_in_force_still_stores_the_override(db, make_user, as_user):
client = as_user(make_user(role="admin"))
default = sysconfig.get_int(db, "quota_daily_budget")
r = client.patch("/api/admin/config/quota_daily_budget", json={"value": default})
assert r.status_code == 200, r.text
(item,) = [i for i in r.json()["groups"]["quota"] if i["key"] == "quota_daily_budget"]
assert item["is_set"] is True
db.expire_all()
assert db.get(SystemConfig, "quota_daily_budget") is not None
+507
View File
@@ -0,0 +1,507 @@
"""The HTTP layer of a unit does no database work of its own, held in code rather than prose.
A route validates, calls its unit's `queries` (reads) or `service` (writes), serializes and commits.
What it may not do is build SQL, call the session's data methods, change a model's attributes or
import a unit's tables: when a route did those, ownership, the commit and the conditional `WHERE`
lived in a different place in each handler, and a fix in one never reached its siblings.
The rule is switched on unit by unit. `ALLOWED` pins what each HTTP file still does today, and the
pin is exact: a count that rises fails (new database work in a route), and one that falls fails
until its entry is lowered (the fence only tightens, as `baseline.json` does for basedpyright). An
empty `ALLOWED` is the end state — a file listed there is a debt, not an exemption.
A core route (identity, admin, ops) is not a unit's HTTP layer and may use the session; it may
still not reach into a unit's tables. A route may NAME a unit's model as a type (under
`TYPE_CHECKING`), never use one as a value.
Below the HTTP layer, a fenced unit's own code keeps to its edges (`FENCED`, `READS`): see the
second half of this file.
"""
import ast
from pathlib import Path
import app.tables # noqa: F401 — every unit's tables, for the ownership check
from app.db import Base
APP = Path(__file__).parent.parent / "app"
ROUTES = APP / "routes"
# Each unit's tables. A new model is placed here (or in CORE_MODELS) by the test below, so a table
# never joins the schema without an owner.
UNIT_MODELS = {
"capture": {
"MediaAsset",
"DownloadProfile",
"FileFolder",
"DownloadJob",
"DownloadShare",
"DownloadQuota",
"DownloadLink",
"DownloadCredential",
"LibraryState",
},
"reader": {
"VideoState",
"Playlist",
"PlaylistItem",
"SavedView",
"ComfortPin",
"Tag",
"ChannelTag",
},
"catalog": {
"Channel",
"Subscription",
"Video",
"SearchFind",
"ExploredChannel",
"BlockedChannel",
},
"plex": {
"PlexLibrary",
"PlexShow",
"PlexSeason",
"PlexItem",
"PlexPerson",
"PlexCollection",
"PlexState",
"PlexLink",
"PlexPlaylist",
"PlexPlaylistItem",
},
}
CORE_MODELS = {
"User",
"OAuthToken",
"AuthToken",
"Invite",
"ApiQuotaUsage",
"QuotaEvent",
"AuditLog",
"LogEvent",
"DebugSession",
"DebugScreenshot",
"AppState",
"JobRun",
"SystemConfig",
"Notification",
"ApiToken",
"ApiTokenGrant",
}
# Which unit each HTTP file belongs to (path relative to app/routes; a folder covers its files).
# Anything not listed is a core route.
UNIT_ROUTES = {
"downloads.py": "capture",
"files.py": "capture",
"public.py": "capture",
"feed.py": "reader",
"playlists.py": "reader",
"saved_views.py": "reader",
"discover.py": "reader",
"tags.py": "reader",
"channels.py": "catalog",
"search.py": "catalog",
"sync.py": "catalog",
"plex/": "plex",
}
# What each HTTP file still does, by file: the debt a unit's sprint pays off. Exact counts. A core
# route's count is only the unit tables it imports; it is paid by the sprint of the unit it reaches.
ALLOWED = {
"channels.py": 83,
"discover.py": 7,
"feed.py": 36,
"playlists.py": 72,
"plex/_shared.py": 1,
"plex/admin.py": 12,
"plex/browse.py": 93,
"plex/collections.py": 33,
"plex/detail.py": 29,
"plex/discover.py": 44,
"plex/people.py": 20,
"plex/playback.py": 42,
"plex/playlists.py": 80,
"runs.py": 1,
"saved_views.py": 17,
"scheduler.py": 3,
"search.py": 46,
"sync.py": 38,
"tags.py": 10,
}
# The session methods that read or write data. `commit` and `rollback` stay with the handler.
SESSION_DATA = {
"add",
"add_all",
"delete",
"execute",
"get",
"merge",
"flush",
"refresh",
"query",
"scalar",
"scalars",
"expunge",
}
# What a route may name from SQLAlchemy: the session's type, for its `db: Session` parameter, and
# the errors its commit raises (the commit is the handler's, so is catching what it throws).
SQLALCHEMY_ALLOWED = {"Session"}
SQLALCHEMY_ALLOWED_MODULES = {"sqlalchemy.exc"}
# Objects a route owns and may set attributes on; anything else it writes to is a model.
OWN_OBJECTS = {"self", "cls", "request", "response"}
UNIT_MODEL_NAMES = {name: unit for unit, names in UNIT_MODELS.items() for name in names}
def _unit_of(rel: str) -> str | None:
for key, unit in UNIT_ROUTES.items():
if rel == key or (key.endswith("/") and rel.startswith(key)):
return unit
return None
def _session_names(tree: ast.AST) -> set[str]:
"""The names a module binds to a database session: a parameter typed `Session`, or a name
assigned (or `with`-bound) from `SessionLocal()`."""
names = set()
for node in ast.walk(tree):
if isinstance(node, ast.arg) and isinstance(node.annotation, ast.Name):
if node.annotation.id == "Session":
names.add(node.arg)
made = None
if isinstance(node, ast.Assign) and len(node.targets) == 1:
made, target = node.value, node.targets[0]
elif isinstance(node, ast.withitem):
made, target = node.context_expr, node.optional_vars
if (
isinstance(made, ast.Call)
and isinstance(made.func, ast.Name)
and made.func.id == "SessionLocal"
and isinstance(target, ast.Name)
):
names.add(target.id)
return names
def _type_only_imports(tree: ast.AST) -> set[ast.AST]:
"""The imports under `if TYPE_CHECKING:` — they name a type for an annotation and are never
loaded at run time, so a model imported there cannot be queried or written."""
out: set[ast.AST] = set()
for node in ast.walk(tree):
if isinstance(node, ast.If):
test = node.test
name = test.id if isinstance(test, ast.Name) else getattr(test, "attr", None)
if name == "TYPE_CHECKING":
for child in node.body:
out |= set(ast.walk(child))
return out
def _root(node: ast.expr) -> ast.expr:
while isinstance(node, (ast.Attribute, ast.Subscript)):
node = node.value
return node
def violations(source: str, unit: str | None) -> list[str]:
"""Each piece of database work in an HTTP file, as `line: what`. A core route (`unit` None) is
held only to the last rule: no unit's tables."""
tree = ast.parse(source)
found = []
sql_names = set()
typing_only = _type_only_imports(tree)
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node not in typing_only:
module = node.module or ""
for alias in node.names:
bound = alias.asname or alias.name
if (
module.split(".")[0] == "sqlalchemy"
and module not in SQLALCHEMY_ALLOWED_MODULES
and alias.name not in SQLALCHEMY_ALLOWED
):
sql_names.add(bound)
if (
alias.name in UNIT_MODEL_NAMES
and module.startswith("app.")
and "models" in module
):
found.append(
f"{node.lineno}: imports {UNIT_MODEL_NAMES[alias.name]}'s {alias.name}"
)
elif isinstance(node, ast.Import):
for alias in node.names:
if alias.name.split(".")[0] == "sqlalchemy":
sql_names.add((alias.asname or alias.name).split(".")[0])
if unit is None:
return found
sessions = _session_names(tree)
for node in ast.walk(tree):
if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Load) and node.id in sql_names:
found.append(f"{node.lineno}: uses sqlalchemy's {node.id}")
if isinstance(node, ast.Call):
f = node.func
if isinstance(f, ast.Attribute) and isinstance(f.value, ast.Name):
if f.value.id in sessions and f.attr in SESSION_DATA:
found.append(f"{node.lineno}: {f.value.id}.{f.attr}()")
targets = []
if isinstance(node, ast.Assign):
targets = node.targets
elif isinstance(node, (ast.AugAssign, ast.AnnAssign)):
targets = [node.target]
for target in targets:
if isinstance(target, ast.Attribute):
base = _root(target)
if not (isinstance(base, ast.Name) and base.id in OWN_OBJECTS):
found.append(f"{node.lineno}: sets .{target.attr}")
return sorted(found, key=lambda v: int(v.split(":")[0]))
def _http_files():
for path in sorted(ROUTES.rglob("*.py")):
rel = path.relative_to(ROUTES).as_posix()
yield rel, path.read_text(encoding="utf-8")
def test_every_table_has_an_owner():
tables = {mapper.class_.__name__ for mapper in Base.registry.mappers}
owned = set(UNIT_MODEL_NAMES) | CORE_MODELS
assert tables - owned == set(), "a model with no unit: place it in UNIT_MODELS or CORE_MODELS"
assert owned - tables == set(), "UNIT_MODELS/CORE_MODELS name a model that no longer exists"
assert not set(UNIT_MODEL_NAMES) & CORE_MODELS
def test_every_listed_route_exists():
rels = [rel for rel, _ in _http_files()]
for key in UNIT_ROUTES:
assert any(rel == key or (key.endswith("/") and rel.startswith(key)) for rel in rels), key
assert set(ALLOWED) <= set(rels), set(ALLOWED) - set(rels)
def test_no_http_file_does_database_work_beyond_its_pin():
drift = {}
for rel, src in _http_files():
found = violations(src, _unit_of(rel))
pinned = ALLOWED.get(rel, 0)
if len(found) != pinned:
drift[rel] = (pinned, len(found), found if len(found) > pinned else "lower its pin")
assert drift == {}, (
"an HTTP file's database work moved off its pin (pinned, now, what): move new work into"
" the unit's queries/service; when a count fell, lower ALLOWED to it"
)
# The detector itself: each kind of database work it must see, and what it must leave alone.
DETECTOR_CASES = {
"from sqlalchemy import select\nq = select(1)": 1,
"import sqlalchemy as sa\nsa.select(1).where(sa.true())": 2,
"from sqlalchemy import func\nn = func.count()": 1,
"from sqlalchemy.orm import Session\ndef f(db: Session): db.commit(); db.rollback()": 0,
"from sqlalchemy.orm import Session\ndef f(db: Session): db.add(x)": 1,
"def f(conn: Session): conn.execute(q)": 1,
"from app.db import SessionLocal\nwith SessionLocal() as s:\n s.get(X, 1)": 1,
"from app.db import SessionLocal\ns = SessionLocal()\ns.scalars(q)": 1,
"def f(job): job.status = 'x'": 1,
"def f(job): job.meta['k'] = 1": 0,
"def f(job): job.parts.count += 1": 1,
"def f(self, request, response): self.x = 1; request.state.y = 2; response.status_code = 3": 0,
"from app.models import DownloadJob, User": 1,
"from app.capture.models import DownloadJob as J": 1,
"def f(cache): cache.get('k')": 0,
"from sqlalchemy.exc import IntegrityError\ntry:\n pass\nexcept IntegrityError:\n pass": 0,
"from typing import TYPE_CHECKING\nif TYPE_CHECKING:\n from app.downloads.models import Job": 0,
"import typing\nif typing.TYPE_CHECKING:\n from app.models import DownloadJob": 0,
"if TYPE_CHECKING:\n pass\nfrom app.downloads.models import DownloadJob": 1,
}
def test_the_detector_sees_each_kind_of_database_work():
for source, expected in DETECTOR_CASES.items():
assert len(violations(source, "capture")) == expected, source
assert violations("from app.models import Video\nfrom sqlalchemy import select", None) == [
"1: imports catalog's Video"
]
# --- inside a unit: its package --------------------------------------------------------------
# A unit whose sprint has landed is fenced beyond its HTTP layer too. Its own code reads another
# unit's tables only along a declared edge (the feed joins Video with VideoState in one query; a
# boundary that split it would cost the speed the app is for), and writes another unit's tables
# never — that goes through the owner's service. Whose a row is, it says through `userscope`, and
# it commits only where a job owns its session.
# Each unit's code outside the HTTP layer: its package, and the entry point that runs its jobs.
UNIT_PACKAGES = {"capture": ("downloads/", "worker.py")}
# The units fenced so far; the rest join with their sprint.
FENCED = {"capture"}
# Whose tables a unit's own code may READ. Core code reads a fenced unit's tables through its queries.
READS = {"capture": {"catalog", "plex"}}
# The columns that say whose a row is.
OWNER_COLUMNS = {"user_id", "to_user_id", "from_user_id", "created_by"}
# Where a fenced unit's code may commit: the jobs that own their session (the worker and what it
# runs, the GC, the backfill — none of them inside a request), and the one write that must commit
# where it stands: an upload's rows and the file it placed on disk land together or not at all, so
# a failed commit can remove the file (`upload.ingest_upload`). Everything else flushes; the handler
# commits.
UNIT_COMMITS = {
"capture": {
"worker.py",
"downloads/backfill.py",
"downloads/gc.py",
"downloads/jobdb.py",
"downloads/pipeline.py",
"downloads/upload.py:ingest_upload",
}
}
SQL_WRITES = {"insert", "update", "delete", "pg_insert"}
def _package_unit(rel: str) -> str | None:
for unit, prefixes in UNIT_PACKAGES.items():
if any(rel == p or (p.endswith("/") and rel.startswith(p)) for p in prefixes):
return unit
return None
def _app_files():
for path in sorted(APP.rglob("*.py")):
rel = path.relative_to(APP).as_posix()
if not rel.startswith("routes/"):
yield rel, path.read_text(encoding="utf-8")
def _imported_models(tree: ast.AST) -> dict[str, str]:
"""The unit models a module imports at run time (a `TYPE_CHECKING` import names a type only),
by the name it binds them to."""
typing_only = _type_only_imports(tree)
out = {}
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node not in typing_only:
module = node.module or ""
if module.startswith("app.") and "models" in module:
for alias in node.names:
if alias.name in UNIT_MODEL_NAMES:
out[alias.asname or alias.name] = alias.name
return out
def _enclosing_functions(tree: ast.AST) -> dict[ast.AST, str]:
out = {}
for fn in ast.walk(tree):
if isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)):
for node in ast.walk(fn):
out.setdefault(node, fn.name)
return out
def package_violations(rel: str, source: str) -> list[str]:
"""What a module outside the HTTP layer does against a fenced unit's boundary, as `line: what`."""
tree = ast.parse(source)
mine = _package_unit(rel)
found = []
models = _imported_models(tree)
for name in models.values():
theirs = UNIT_MODEL_NAMES[name]
if theirs == mine or (mine not in FENCED and theirs not in FENCED):
continue
if theirs not in READS.get(mine or "", set()):
found.append(f"0: {mine or 'core'} imports {theirs}'s {name}")
for node in ast.walk(tree):
if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Name):
continue
written = None
if node.func.id in models:
written = models[node.func.id]
elif node.func.id in SQL_WRITES and node.args and isinstance(node.args[0], ast.Name):
written = models.get(node.args[0].id)
if written is None:
continue
theirs = UNIT_MODEL_NAMES[written]
if theirs != mine and (mine in FENCED or theirs in FENCED):
found.append(f"{node.lineno}: {mine or 'core'} writes {theirs}'s {written}")
if mine in FENCED:
found += _owner_filters(tree)
functions = _enclosing_functions(tree)
allowed = UNIT_COMMITS.get(mine, set())
for node in ast.walk(tree):
if (
isinstance(node, ast.Call)
and isinstance(node.func, ast.Attribute)
and node.func.attr == "commit"
and rel not in allowed
and f"{rel}:{functions.get(node)}" not in allowed
):
found.append(f"{node.lineno}: commits in {functions.get(node)}")
return sorted(found, key=lambda v: int(v.split(":")[0]))
def _owner_filters(tree: ast.AST) -> list[str]:
"""A comparison on a table's owner column written inline (`Model.user_id == …`), not through
`userscope.owned_by`."""
found = []
for node in ast.walk(tree):
if isinstance(node, ast.Compare):
for side in (node.left, *node.comparators):
if (
isinstance(side, ast.Attribute)
and isinstance(side.value, ast.Name)
and side.value.id in set(UNIT_MODEL_NAMES) | CORE_MODELS
and side.attr in OWNER_COLUMNS
):
found.append(f"{node.lineno}: filters {side.value.id}.{side.attr} inline")
return found
def test_a_fenced_units_code_keeps_to_its_edges():
hits = {rel: found for rel, src in _app_files() if (found := package_violations(rel, src))}
assert hits == {}, (
"read another unit's tables only along READS, write them only through their service, say"
" whose a row is through userscope, commit only where UNIT_COMMITS says"
)
def test_a_fenced_units_routes_filter_owners_through_userscope():
hits = {
rel: found
for rel, src in _http_files()
if _unit_of(rel) in FENCED and (found := _owner_filters(ast.parse(src)))
}
assert hits == {}
def test_every_unit_package_exists():
for prefixes in UNIT_PACKAGES.values():
for p in prefixes:
assert (APP / p).exists(), p
for commits in UNIT_COMMITS.values():
for entry in commits:
assert (APP / entry.split(":")[0]).exists(), entry
PACKAGE_CASES = {
# (module path, source): violations
("downloads/x.py", "from app.models import Video"): 0, # capture reads catalog
("downloads/x.py", "from app.models import Playlist"): 1, # no capture→reader edge
("downloads/x.py", "from app.models import Video\nVideo(id=1)"): 1, # a write across
("downloads/x.py", "from app.models import Video\nupdate(Video)"): 1,
("downloads/x.py", "from app.downloads.models import DownloadJob\nDownloadJob()"): 0,
("search.py", "from app.downloads.models import DownloadJob"): 1, # core reads capture
("search.py", "from app.models import Video\nVideo()"): 0, # neither side fenced yet
("downloads/x.py", "q = DownloadJob.user_id == 1"): 1,
("downloads/x.py", "q = owned_by(DownloadJob.user_id, 1)"): 0,
("downloads/x.py", "def f(db):\n db.commit()"): 1,
("downloads/gc.py", "def f(db):\n db.commit()"): 0,
("downloads/upload.py", "def ingest_upload(db):\n db.commit()"): 0,
("downloads/upload.py", "def other(db):\n db.commit()"): 1,
(
"downloads/x.py",
"from typing import TYPE_CHECKING\nif TYPE_CHECKING:\n from app.models import Playlist",
): 0,
}
def test_the_package_detector_sees_each_kind():
for (rel, source), expected in PACKAGE_CASES.items():
assert len(package_violations(rel, source)) == expected, (rel, source)
+2 -1
View File
@@ -33,7 +33,8 @@ from app.downloads import mediakind, pipeline, playable, quota, service, storage
from app.downloads import probe as probemod
from app.downloads import upload as uploadmod
from app.downloads.stated import StatedFailure
from app.models import DISPLAY_TEXT_MAX, DownloadJob, DownloadQuota, MediaAsset
from app.models import DISPLAY_TEXT_MAX
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
# --- pure: the kind invariant ---------------------------------------------------------------
+2 -1
View File
@@ -21,7 +21,8 @@ from app.downloads import pipeline, quota, storage
from app.downloads import probe as probemod
from app.downloads import upload as uploadmod
from app.downloads.stated import StatedFailure
from app.models import DownloadJob, DownloadQuota, MediaAsset, Notification
from app.models import Notification
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
from sqlalchemy import select, update
from sqlalchemy.exc import OperationalError
from sqlalchemy.orm import Session
+1 -1
View File
@@ -12,7 +12,7 @@ from app.config import settings
from app.downloads import pipeline, runner, service, storage
from app.downloads import probe as probemod
from app.downloads import upload as uploadmod
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
@pytest.mark.parametrize(
+138 -128
View File
@@ -44,6 +44,10 @@ runs this check before it is called done (E28 S3g):
A rule that holds is not the same as a screen that works: the person page passed every rule above with
half its panel empty.
A **copy-only** change — translated text and nothing else: no markup, class, component or logic — runs step 4
alone: a screenshot of every state the changed text renders in, in Hungarian and in English (Hungarian runs about
a fifth longer, so it is the one that wraps or clips). The key parity is the gate's.
## The design lab
`npm run dev`, then open **`http://localhost:5173/design-lab`**. The lab renders each open decision's variants
@@ -126,7 +130,10 @@ style guide.
to name one of the two. Black and white are the media scrims and their text. Five schemes × dark
and light (B2): **Midnight** (blue), **Forest** (cyan), **Slate** (fuchsia), **Matrix** (moss) each earn their
place with their own hue family; **Fjord** (Midnight's navy with a calm steel-blue accent) is the one kept
exception, the calm twin of the vivid Midnight. A new scheme still needs its own hue family. Normal text ≥ 4.5:1 and a control's edge ≥ 3:1 in **every** scheme and mode.
exception, the calm twin of the vivid Midnight. A new scheme still needs its own hue family. Normal text ≥ 4.5:1 and a control's edge ≥ 3:1 in **every** scheme and mode;
`--muted` keeps ≥ 6:1 on the bare `--bg`, because it mostly sits on glass, which costs up to ~1.5 of the ratio.
A tonal fill under text (a badge, the item count, the current rail row) is an opaque `bg-tint*` (its colour
mixed into `--surface`), never a translucent `bg-fg/8`: the glass and what scrolls under it show through.
[WCAG 1.4.3, 1.4.11] The input edge token is set under Controls (A4). The ≥ 3:1 edge is the field
family's (a field, the form's dropdown trigger), whose edge is the only thing that says "type here". A
labelled pill (a chip, a toolbar trigger, a tab, a segment) keeps the faint `--border`: its text names it,
@@ -268,10 +275,9 @@ Also in force:
- **Page chrome (B4):** a page's tabs and filters sit in ONE sticky band of the page colour, with the fade
under it (below). Below ~600px of height the filter row folds into one "Filters · n" button. Every page has
a visually hidden `h1`; the nav rail and the browser tab title are the visible identity. [WCAG 2.4.6]
Built as: the band is `PageShell`'s toolbar slot (`PageToolbar`), above the one scroller; a console
whose toolbar lives inside its frame (the server log) makes that toolbar `sticky` with `fade-below`
instead. The band holds controls only: an intro paragraph scrolls with the content (in the band it
ate three lines of a short window). The fold is `useFilterFold` + `FilterFoldButton`: the same row
Built as: the band is `PageShell`'s toolbar slot (`PageToolbar`), above the one scroller. The band
holds controls only: an intro paragraph scrolls with the content (in the band it ate three lines of a
short window). The fold is `useFilterFold` + `FilterFoldButton`: the same row
shown or hidden in place, never copied into a popover. The hidden `h1` is `PageShell`'s `title`; a
view with its own visible title (a channel, a Plex show, a playlist) renders it as `PageHeading`
and the hidden one steps aside, so a page has exactly one `h1` (a hand `<h1>` inside the shell is
@@ -642,7 +648,7 @@ of a full-width panel and nothing on the right.
`app/collation.py` on the server, so both sort alike.
- **Hungarian words, not English ones** (E30 D3). Kept as Hungarian users say them: Feed, Shorts, Admin,
Token, Port, and technical tokens (log levels, ffmpeg presets, remux). Everything else is Hungarian:
közvetítés (stream), Prioritás, Feladat (job), Eseménynapló (audit), Demó, Automatikus, Infó,
közvetítés (stream), Prioritás, Feladat (job), Napló (activity), Admin műveletek (audit), Demó, Automatikus, Infó,
Felület / Szerver (frontend / backend), videók behúzása (backfill), átkódolás (transcode), e-mail.
A longer Hungarian word gets its room — a column is sized to it, a pill truncates — never clipped.
- A label says what happens ("Delete playlist"), not "OK"; a destructive confirm names the object.
@@ -661,125 +667,129 @@ of a full-width panel and nothing on the right.
## Decision log
| Date | Decision | Chosen | Over | Why |
| ---------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
| Date | Decision | Chosen | Over | Why |
| ---------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
| 2026-10-17 | Copy-only changes (owner ★) | Step 4 of the Conformance check alone (not 1–3 or 5): every state the changed text renders in, seen in both languages | The whole five-step check; no screenshot at all (DOM test + key parity) | Steps 1–3 and 5 judge controls, patterns and composition, which a text change does not touch; the one risk left is a longer Hungarian string wrapping or clipping, and only a screenshot shows that |
| 2026-10-17 | Text on glass (owner ★) | Light `--muted` 15% toward `--fg` in every scheme (≥ 6:1 on the bare page); the item count, the badges and the current Files row on opaque tints mixed into the surface | Only the two failing components; the item count's number in `--fg` | Pixel-measured over every scheme × mode × backdrop, the count fell to 3.8:1 (dark too: thumbnails scroll under the toolbar's glass) and Channels' tabs to 4.1:1; an `--fg` count outshouts the filters beside it |
| 2026-10-18 | Scheduler run history (owner D4, E1, E2 ★) | A row's last three runs as dots, newest first (interrupted: a hollow warn ring, set apart from skipped and failed), and a history button that opens the job's runs on the Activity page; a user's ad-hoc runs in their own block under the jobs, with no interval and no run button | An inline expandable history per row; a dark-grey interrupted dot; ad-hoc runs only on the Activity page | Three dots answer "did it run, did it work" at a glance; the whole history needs filters, which the Activity page has |
| 2026-10-18 | Activity page (owner D1, D2, G1–G3 ★) | One page, "Napló" / "Activity": four tabs (Runs, Admin actions, Server log, Quota) under one filter bar (text, a period of 1 h · 24 h · 7 d · 30 d · 90 d defaulting to 7 days, an account), the view in the URL; runs and admin actions as dense rows that open in place as a log line does, filtered and paged on the server; a run's or a request's log lines one pivot away, as a history entry | One merged timeline; a page per source; a default period per tab; card rows; the audit log's client-side table | Each source keeps its own table and retention, read in one place; one bar keeps what you narrowed across tabs; about 1.8k runs a day need a log's density and a server-side filter |
+13 -4
View File
@@ -44,12 +44,11 @@ A change is done when this is green, from the repo root:
bash scripts/check.sh # every lane
bash scripts/check.sh lint # the static lanes only (no tests, build or audits) — the quick loop
bash scripts/check.sh --brief # either mode: one line per lane, output only for a red one
bash scripts/lint-changed.sh # ruff + eslint + prettier on the uncommitted files only
bash scripts/check.sh --list # the lane names, one per line (runs nothing)
bash scripts/check.sh --only "vitest" # one lane, by its exact name — never "the gate"
```
It runs every lane even after one fails and names each red one at the end. `lint-changed.sh` is a
convenience, not the gate; with `--hook` it doubles as a Claude Code Stop hook (silent when clean,
the findings handed back otherwise).
It runs every lane even after one fails and names each red one at the end.
The codebase's health numbers — size, complexity, test ratio, gate weight, comment noise — come from
`python scripts/audit_metrics.py` (seconds; it reports, it never fails). `--speed` adds the speed
@@ -85,6 +84,13 @@ docker compose -f backend/docker-compose.test.yml run --rm --no-deps test \
pip-audit -r requirements.lock --no-deps --disable-pip
```
**Shell scripts**
```bash
# every tracked *.sh at shellcheck's default bar (info included); the version is pinned in scripts/check.sh
git ls-files -z -- '*.sh' | xargs -0 shellcheck
```
A missing database or ffmpeg is a hard error, not a skip — so the gate can't print green while the
DB-backed tests were quietly not run (`SIFTLODE_TEST_REQUIRE_DB=0` is the explicit "pure-logic only"
run for a machine without Postgres). Tests that need a database take the `db` fixture
@@ -107,6 +113,9 @@ Coverage is on demand, never enforced and not in the gate (E35): `npm run test -
`frontend/`, and for the backend
`docker compose -f backend/docker-compose.test.yml run --rm test python -m pytest --cov=app` (then
`… down -v`).
The DB lane runs on every core (`pytest -n auto`, pytest-xdist): `tests/db_per_worker.py` gives each worker its own
database (`siftlode_test_gw<N>`, created and migrated on first use), so the same tests run side by side. A serial
run (no `-n`) still works and uses `siftlode_test` itself.
The Playwright suite (`npm run e2e`) runs against the `:8080` stack built from your tree; the release
requires it. Before an image is pushed, it is scanned with Trivy (HIGH/CRITICAL with a fix fails;
exceptions in `.trivyignore.yaml`, each with a reason and an expiry).
+1
View File
@@ -113,6 +113,7 @@ project checks needs a row.
| CVE gate | HIGH/CRITICAL with a fix available only | `--ignore-unfixed` + `--severity HIGH,CRITICAL`: a finding with no fix cannot be acted on by an upgrade; it shows in the scan output. A fixable one fails the publish, and the only way past it is `.trivyignore.yaml`. | 0 open (E32 S1) |
| npm audit | `--omit=dev --audit-level=high` | What ships to the browser, at high or above. The dev toolchain's advisories were burned down by hand in E32 S1 (18 → 0). | 0 |
| knip | `entry` — one file added | Not a narrowing: knip's Vite plugin reads entries from index.html only, so the shared link's page (`watch.html` → `src/watchMain.tsx`) is named here, or the lane reports that whole entry unused. It adds files to the lane, never takes any out. | 1 entry |
| unit fence | the database work the HTTP layer still does, pinned | `backend/tests/test_unit_fences.py`: a unit's route file may not build SQL, call the session's data methods, set a model's attributes or import a unit's tables; a core route may not import a unit's tables. Switched on unit by unit: `ALLOWED` pins each file's count exactly, so a rise fails and a fall fails until the pin is lowered — the list may only shrink. End state: empty, and the pin mechanism goes. A unit in `FENCED` is held below its HTTP layer too: its code reads another unit's tables only along `READS`, writes them never, filters owners through `userscope`, and commits only where `UNIT_COMMITS` names a job's own session. | 826 in 23 files (E42 S0) → 667 in 19 (S1: Capture 0, fenced) |
| React Compiler | the functions it skips, pinned | `components/compilerCoverage.contract.test.ts` — the list may only shrink. End state: only VirtualGrid (TanStack Virtual, an IncompatibleLibrary) remains. | 93 (E32 S1) → 29 (E32 S3) → 1, VirtualGrid (E32 S4) |
## Accepted findings
+5
View File
@@ -45,6 +45,11 @@ login-free `/watch/<token>` share link).
PW_BASE_URL=http://localhost:5173 siftlode e2e
```
**Accessibility** — `a11y.spec.ts` runs axe-core's WCAG 2.x A + AA rules (`@axe-core/playwright`)
over every rail page, every admin page and the player, in `chromium` only (the rules read the DOM,
which the engines build alike). It does not judge colour contrast: axe files text on a gradient —
every glass surface — as "incomplete", not as a pass or a hit.
## Account & fixture
Tests run as a dedicated **admin** account `e2e@siftlode.local` (never a real account;
+63
View File
@@ -0,0 +1,63 @@
import AxeBuilder from "@axe-core/playwright";
import { test, expect, type Page } from "@playwright/test";
import { gotoPage, openApp, openFirstVideo } from "./helpers";
// Accessibility, the machine-checkable half: axe-core's WCAG 2.x A + AA rules over every page the
// owner reaches and the player. A hit fails with the rule, its impact and the elements.
//
// What it does NOT cover — colour contrast, mostly: axe cannot compute a contrast over a gradient,
// and the glass surfaces are gradients, so it files most text as "incomplete" instead of judging it
// (measured 2026-10-17: 110 nodes on the feed, 9-32 on the other pages). A green here says nothing
// about contrast; that stays with the visual check.
//
// Chromium only (playwright.config.ts leaves it out of the other projects).
const TAGS = ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"];
/** Wait until nothing is loading: no QueryState spinner and no skeleton bone left. A bone says it is
* one (`data-skeleton`): a pulse alone is not loading — a running job's dot pulses as long as it runs. */
async function settled(page: Page): Promise<void> {
await expect(page.locator('[role="status"] .animate-spin, [data-skeleton]')).toHaveCount(0);
}
async function violations(page: Page): Promise<string[]> {
const result = await new AxeBuilder({ page }).withTags(TAGS).analyze();
return result.violations.map((v) => {
const shown = v.nodes.slice(0, 5).map((n) => n.target.join(" "));
const more = v.nodes.length > 5 ? ` (+${v.nodes.length - 5})` : "";
return `${v.id} (${v.impact ?? "?"}): ${v.help} — ${shown.join(" | ")}${more}`;
});
}
test.describe("accessibility (axe)", () => {
const RAIL = ["feed", "channels", "playlists", "files", "stats", "settings"];
const ADMIN = ["scheduler", "config", "users", "activity"];
for (const id of RAIL) {
test(`page: ${id}`, async ({ page }) => {
await openApp(page);
await gotoPage(page, id);
await settled(page);
expect(await violations(page)).toEqual([]);
});
}
for (const id of ADMIN) {
test(`admin page: ${id}`, async ({ page }) => {
await openApp(page);
await page.getByTestId("nav-admin").click();
await page.getByTestId(`nav-admin-${id}`).click();
await expect(page.getByTestId("nav-admin")).toHaveAttribute("aria-current", "page");
await settled(page);
expect(await violations(page)).toEqual([]);
});
}
test("player", async ({ page }) => {
// hermetic: the YouTube iframe is not loaded
await page.route(/youtube\.com/, (route) => route.abort());
await openApp(page);
await openFirstVideo(page);
await settled(page);
expect(await violations(page)).toEqual([]);
});
});

Some files were not shown because too many files have changed in this diff Show More