peter
b59fd38a9c
The log now has a stored, queryable copy for the diagnostics console: every WARNING and above plus the request line of every request, kept for log_retention_days (default 7, no "forever") and pruned hourly by the new log_gc job. Rows are built on the logging thread from the same redacted fields as the stdout line, queued without blocking and written in batches by one background thread on its own connection. A full queue or a failed write drops and reports instead of retrying. Flood control keeps 30 of any one message per minute plus a count of the rest; anonymous request lines are capped at 60 a minute per client, and the client address is never stored. Deleting an account deletes its stored events. Measured on prod first: a YouTube RSS outage wrote 11k warnings in four hours, one per channel per poll. The poll now writes one warning with per-reason counts and a few examples, and a failure on our side gets one traceback per poll instead of one per channel.
77 lines
4.9 KiB
Markdown
77 lines
4.9 KiB
Markdown
# Backend conventions
|
||
|
||
The rules below are already followed across `backend/app` — they lived only in a few module
|
||
docstrings (`demo.py`, `userscope.py`, `routes/_common.py`) until this file collected them (E3 S3,
|
||
B-core-20). When you add code, match these so the next reader doesn't have to guess "which pattern do
|
||
I copy".
|
||
|
||
## Module layering
|
||
|
||
Three layers, by *who* runs the code — not by feature:
|
||
|
||
1. **`routes/…` — request-only.** A route module holds HTTP concerns: auth dependencies, request/
|
||
response models, status-code mapping, serialization. It owns logic that runs *only* on a request.
|
||
Business logic that a request happens to trigger does **not** live here if anything else runs it too.
|
||
|
||
2. **`plex/`, `downloads/`, `sync/` — domain logic shared by more than one caller.** Anything the
|
||
**scheduler or the worker** runs (not just a request) lives in one of these domain packages, so the
|
||
route, the worker, and the scheduler share one definition. Examples from this codebase:
|
||
`downloads/service.py` owns enqueue + the job state transitions (routes call it); `plex/sync.py` is
|
||
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline.
|
||
|
||
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
|
||
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, `logs` (the one logging
|
||
configuration) and `logsink` (its `log_events` table), … are single-file primitives many
|
||
layers import. They depend on nothing above them (a core module must never import a `routes/…`
|
||
module).
|
||
|
||
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
|
||
*second* caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down
|
||
a layer — into the relevant domain package, or a flat core module if it's cross-cutting. Don't
|
||
pre-abstract for a single caller; don't leave shared logic duplicated once there are two.
|
||
|
||
## Transaction ownership
|
||
|
||
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
|
||
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
|
||
to the **operation owner**; anything called *inside* another transaction only stages its change
|
||
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
|
||
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
|
||
mutation it records, so an audit entry can never exist without its change (or vice-versa).
|
||
|
||
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
|
||
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
|
||
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
|
||
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
|
||
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
|
||
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
|
||
attribute — see `service.decr_ref_count`.
|
||
|
||
## Shared counters and status: do it atomically at the DB
|
||
|
||
- **Never read-modify-write a shared counter in Python** (`x.count = x.count + 1`). Concurrent sessions
|
||
lose an update. Use a DB-atomic statement — `col = col + 1`, or `greatest(col - 1, 0)` with
|
||
`RETURNING` so the caller branches on the post-value under the row lock (`service.incr_ref_count` /
|
||
`decr_ref_count`).
|
||
- **Download job status is a `JobStatus` StrEnum** (`downloads/status.py`), not bare strings. The
|
||
`HOLDING` set defines which states hold a `MediaAsset` (so release accounting stays correct);
|
||
`STOPPED_BY_ROUTE` is the route-initiated stop set. A worker child's terminal write goes through
|
||
`service.finalize_if_running` (an atomic `UPDATE … WHERE status='running'`) so a finishing child can't
|
||
resurrect a job a route just stopped.
|
||
|
||
## Config: one home per kind
|
||
|
||
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
|
||
startup, not user-tunable at runtime.
|
||
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
|
||
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
|
||
slated to fold into `sysconfig` — do not add more like it.)
|
||
|
||
## The gate
|
||
|
||
Everything below must be green before a change is done (run via `siftlode check`):
|
||
`ruff` (backend) · `tsc` ×3 + `eslint` + `prettier` + `knip` (frontend) · `vitest` · `pytest`. The
|
||
pytest lane runs in `backend/docker-compose.test.yml` — an ephemeral Postgres + the backend tree
|
||
bind-mounted — so the DB-backed integration tests (the `db` fixture) actually run; `SIFTLODE_TEST_
|
||
REQUIRE_DB=1` there turns a would-be silent skip into a hard failure. See `docs/development.md`.
|