peter
04b58e5c6f
The gap this sprint was scoped against turned out to be smaller than the survey said, and in a way worth writing down: `log_requests` already records every HTTP call with its route template, status, duration, request id and user id, and the audit log already records every admin act on someone else's account. Putting a line in each of routes/downloads.py's twenty mutating endpoints would have added a second copy of the first and nothing else. So this instruments what neither of them can see. yt-dlp now explains itself to our log. `quiet: True` was doing more than quieting: with no `logger` its whole account of a download went to the child's stdout, so "Requested format is not available", a signature-extraction failure or a POT refusal reached us only as whatever text ended up in the raised exception — the real reason a download failed was routinely one line the app never kept. Measured against the pinned yt-dlp 2026.07.04 before wiring it: with `logger` set it calls `debug` for extraction and progress lines and `warning`/`error` for the rest, and neither `quiet` nor `no_warnings` gates those calls. `debug` lands at TRACE, being several lines a second, so the noisy half is the half the user's own consent gates. The full source URL is trimmed to its host on the way in — it is a row of someone's download history, and `logs.set_job` already puts the job id on every one of these lines. The option is set AFTER the adapter merge, like `cookiefile` and the size cap: a source adapter must not be able to take away the one account of what went wrong that a failed job leaves behind. An HLS session now says which pipeline it got (transcode/remux, hw, tonemap, variants, offset) — everything downstream is about that choice, and it was only inferable from an ffmpeg command line nobody keeps. A session culled by the concurrent-transcode cap says so: somebody's playback stops there, the route turns it into a retryable 404, and "my film keeps rebuffering" had no other explanation anywhere. The HW→SW fallback keeps its line and gains an event code. A live search refused by a quota cap says WHICH cap: the 429 is in the request line either way, but the per-user daily cap is answered by tomorrow and the shared budget by the admin. The download and edit job lifecycle lines gain event codes, which is what E22.6's console filters on — the job id was already a structured field. tests/test_ytdlp_logging.py; three of its four cases mutation-checked. A fourth (a `%` in a progress line) survived its mutation — stdlib only applies `%` formatting to a record that has args — so it is gone rather than kept as a case nothing can break; the reasoning moved into the comment on `_emit`.
Backend conventions
The rules below are already followed across backend/app — they lived only in a few module
docstrings (demo.py, userscope.py, routes/_common.py) until this file collected them (E3 S3,
B-core-20). When you add code, match these so the next reader doesn't have to guess "which pattern do
I copy".
Module layering
Three layers, by who runs the code — not by feature:
-
routes/…— request-only. A route module holds HTTP concerns: auth dependencies, request/ response models, status-code mapping, serialization. It owns logic that runs only on a request. Business logic that a request happens to trigger does not live here if anything else runs it too. -
plex/,downloads/,sync/— domain logic shared by more than one caller. Anything the scheduler or the worker runs (not just a request) lives in one of these domain packages, so the route, the worker, and the scheduler share one definition. Examples from this codebase:downloads/service.pyowns enqueue + the job state transitions (routes call it);plex/sync.pyis run by both the admin route and the scheduler;downloads/pipeline.pyis the worker's child pipeline. -
Flat
app/*.pycore modules — cross-cutting primitives.auth,sysconfig,quota,audit,notifications,userscope,realtime,ratelimit,progress,logs(the one logging configuration),logsink(itslog_eventstable) anddiagnostics(a user's consented log level), … are single-file primitives many layers import. They depend on nothing above them (a core module must never import aroutes/…module).
Extraction trigger: the SECOND consumer. Keep new logic inline in its first caller. The moment a second caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down a layer — into the relevant domain package, or a flat core module if it's cross-cutting. Don't pre-abstract for a single caller; don't leave shared logic duplicated once there are two.
Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — "the handler/job-wrapper commits; helpers flush
(only quota owns its session)" — so read it that way, not as an exception to it. The commit belongs
to the operation owner; anything called inside another transaction only stages its change
(db.add / flush) and lets the owner commit, so the whole operation is one transaction.
audit.record() is the reference: it adds the audit row and the caller commits it together with the
mutation it records, so an audit entry can never exist without its change (or vice-versa).
- The operation owner is usually the route handler. A
service.*operation the handler delegates a whole transition to (e.g.service.cancel) is that owner and commits its one transition — it isn't a mid-operation helper. A helper that owns its OWN short-lived session (the worker'sjobdbwrites, the same carve-out CLAUDE.md grantsquota) commits that session. Everything else flushes. SessionLocalisexpire_on_commit=False: after a rawUPDATE+ commit, an ORM object you already loaded is stale. Read the fresh value from the row (RETURNING, or a re-SELECT), not the ORM attribute — seeservice.decr_ref_count.
Shared counters and status: do it atomically at the DB
- Never read-modify-write a shared counter in Python (
x.count = x.count + 1). Concurrent sessions lose an update. Use a DB-atomic statement —col = col + 1, orgreatest(col - 1, 0)withRETURNINGso the caller branches on the post-value under the row lock (service.incr_ref_count/decr_ref_count). - Download job status is a
JobStatusStrEnum (downloads/status.py), not bare strings. TheHOLDINGset defines which states hold aMediaAsset(so release accounting stays correct);STOPPED_BY_ROUTEis the route-initiated stop set. A worker child's terminal write goes throughservice.finalize_if_running(an atomicUPDATE … WHERE status='running') so a finishing child can't resurrect a job a route just stopped.
Config: one home per kind
app.config.settings— boot/env config (DB URL, download root, worker concurrency): read once at startup, not user-tunable at runtime.sysconfig.SPECS— the runtime admin knobs (DB-override-or-default). A new admin-tunable value goes inSPECS; don't invent a fourth store. (state.py'smaintenance_batchpredates this and is slated to fold intosysconfig— do not add more like it.)
The gate
Everything below must be green before a change is done (run via siftlode check):
ruff (backend) · tsc ×3 + eslint + prettier + knip (frontend) · vitest · pytest. The
pytest lane runs in backend/docker-compose.test.yml — an ephemeral Postgres + the backend tree
bind-mounted — so the DB-backed integration tests (the db fixture) actually run; SIFTLODE_TEST_ REQUIRE_DB=1 there turns a would-be silent skip into a hard failure. See docs/development.md.