peter
45c0ee7d58
What a job last did lived in the API process's memory and was gone at every restart; the audit log kept a run only when it was manual, failed or changed something. Now each run is a job_runs row written at its start and its end in its own session: who started it, its outcome as data, what it spent of the YouTube quota (quota_events.job_run_id) and the lines it logged (log_events.run_id, filterable in the log console). - a run a stopped process left open is marked interrupted at startup - "run now" is a one-shot job on the scheduler, which now always starts (interval jobs only with SCHEDULER_ENABLED); the manual actor is reset after the run, since pool threads are reused - the dashboard's last run comes from job_runs, so it survives a deploy - runs leave the audit log: migration 0109 moves the old job_run rows over (and back on downgrade); the audit page's run folding goes - job_run_retention_days (default 30) prunes old runs in audit_gc
Backend conventions
The rules below are already followed across backend/app — they lived only in a few module
docstrings (userscope.py, routes/_common.py) until this file collected them (E3 S3,
B-core-20). When you add code, match these so the next reader doesn't have to guess "which pattern do
I copy".
Module layering
Three layers, by who runs the code — not by feature:
-
routes/…— request-only. A route module holds HTTP concerns: auth dependencies, request/ response models, status-code mapping, serialization. It owns logic that runs only on a request. Business logic that a request happens to trigger does not live here if anything else runs it too. -
plex/,downloads/,sync/,issues/— domain logic shared by more than one caller. Anything the scheduler or the worker runs (not just a request) lives in one of these domain packages, so the route, the worker, and the scheduler share one definition. Examples from this codebase:downloads/service.pyowns enqueue + the job state transitions (routes call it);plex/sync.pyis run by both the admin route and the scheduler;downloads/pipeline.pyis the worker's child pipeline;issues/bundle.pyis the one definition of a bug report's files, for the zip export and the tracker. -
Flat
app/*.pycore modules — cross-cutting primitives.auth(withapitokens, a machine's folder-scoped way into the Files library),sysconfig,quota,audit,jobruns(every background run's durable row, its words inrunstatus),notifications,userscope,ratelimit,progress,logs(the one logging configuration),logsink(itslog_eventstable),diagnostics(a user's consented log level),debugsession(a recording that tags its user's lines, E16) andcollation(A→Z in the reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core module must never import aroutes/…module).
Extraction trigger: the SECOND consumer. Keep new logic inline in its first caller. The moment a second caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down a layer — into the relevant domain package, or a flat core module if it's cross-cutting. Don't pre-abstract for a single caller; don't leave shared logic duplicated once there are two.
Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — "the handler/job-wrapper commits; helpers flush
(only quota and jobruns own their session)" — so read it that way, not as an exception to it. The commit belongs
to the operation owner; anything called inside another transaction only stages its change
(db.add / flush) and lets the owner commit, so the whole operation is one transaction.
audit.record() is the reference: it adds the audit row and the caller commits it together with the
mutation it records, so an audit entry can never exist without its change (or vice-versa).
- The operation owner is usually the route handler. A
service.*operation the handler delegates a whole transition to (e.g.service.cancel) is that owner and commits its one transition — it isn't a mid-operation helper. A helper that owns its OWN short-lived session (the worker'sjobdbwrites,jobruns' run rows — the same carve-out CLAUDE.md grantsquota) commits that session. Everything else flushes. SessionLocalisexpire_on_commit=False: after a rawUPDATE+ commit, an ORM object you already loaded is stale. Read the fresh value from the row (RETURNING, or a re-SELECT), not the ORM attribute — seeservice.decr_ref_count.
Shared counters and status: do it atomically at the DB
- Never read-modify-write a shared counter in Python (
x.count = x.count + 1). Concurrent sessions lose an update. Use a DB-atomic statement —col = col + 1, orgreatest(col - 1, 0)withRETURNINGso the caller branches on the post-value under the row lock (service.incr_ref_count/decr_ref_count). - Download job status is a
JobStatusStrEnum (downloads/status.py), not bare strings. TheHOLDINGset defines which states hold aMediaAsset(so release accounting stays correct);STOPPED_BY_ROUTEis the route-initiated stop set. A worker child's terminal write goes throughservice.finalize_if_running(an atomicUPDATE … WHERE status='running') so a finishing child can't resurrect a job a route just stopped.
Config: one home per kind
app.config.settings— boot/env config (DB URL, download root, worker concurrency): read once at startup, not user-tunable at runtime.sysconfig.SPECS— the runtime admin knobs (DB-override-or-default). A new admin-tunable value goes inSPECS; don't invent a fourth store. (state.py'smaintenance_batchpredates this and is slated to fold intosysconfig— do not add more like it.)
The gate
Everything below must be green before a change is done (run via siftlode check):
ruff + basedpyright (backend) · the lock check + tsc ×3 + eslint + prettier + knip +
vite build (frontend) · vitest · pytest. The pytest lane runs in backend/docker-compose.test.yml
— an ephemeral Postgres + the backend tree bind-mounted — so the DB-backed integration tests (the db
fixture) actually run; a missing database is a hard failure, not a silent skip, unless a hand run asks
for pure-logic only with SIFTLODE_TEST_REQUIRE_DB=0. See docs/development.md, and docs/gates.md
for what each gate deliberately leaves out.