siftlode/backend/app

Backend conventions

The rules below are already followed across backend/app — they lived only in a few module docstrings (userscope.py, routes/_common.py) until this file collected them. When you add code, match these so the next reader doesn't have to guess "which pattern do I copy".

Module layering

Three layers, by who runs the code — not by feature:

1. routes/… — request-only. A route module holds HTTP concerns: auth dependencies, request/ response models, status-code mapping, serialization. It owns logic that runs only on a request. Business logic that a request happens to trigger does not live here if anything else runs it too.

2. The units and the other domain packages — logic shared by more than one caller. A unit's database work (below), and anything the scheduler or the worker runs (not just a request), lives in a domain package, so the route, the worker, and the scheduler share one definition. Examples from this codebase: downloads/service.py owns enqueue + the job state transitions (routes call it); plex/sync.py is run by both the admin route and the scheduler; downloads/pipeline.py is the worker's child pipeline; issues/bundle.py is the one definition of a bug report's files, for the zip export and the tracker.

3. Flat app/*.py core modules — cross-cutting primitives. auth (with apitokens, a machine's folder-scoped way into the Files library), sysconfig, quota, audit, jobruns (every background run's durable row, its words in runstatus), notifications, userscope, ratelimit, progress, logs (the one logging configuration), logsink (its log_events table), diagnostics (a user's consented log level), debugsession (a recording that tags its user's lines, E16) and collation (A→Z in the reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core module must never import a routes/… module).

Extraction trigger: the SECOND consumer. Keep new logic inline in its first caller. The moment a second caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down a layer — into the relevant domain package, or a flat core module if it's cross-cutting. Don't pre-abstract for a single caller; don't leave shared logic duplicated once there are two.

Units

The domain is four units, each owning its tables (models.py in its package):

Unit	Package	Its routes
Capture	downloads/, the worker	downloads.py, files.py, public.py
Reader	reader/, discover/, feedquery.py, the playlist sync and tag suggestions in sync/	feed.py, playlists.py, saved_views.py, discover.py, tags.py
Catalog	catalog/, the ingests in sync/	channels.py, search.py, sync.py
Plex	plex/	plex/

• A unit's route does no database work. It validates, reads through the unit's queries.py, writes through its service.py, serializes, and commits. It may name a unit's model as a type (under TYPE_CHECKING), never use one as a value, and it passes a kind ("movie"), not a model, to a query.
• A unit's code keeps to its edges. It reads another unit's tables only along a declared edge (Capture and Reader read Catalog's and Plex's: the feed joins Video with VideoState in one query) and writes them only through the owner's service. Core code reaches a unit's tables through that unit's queries.
• Whose a row is, userscope says — in every module, core included: owned_by(Model.user_id, user) for a filter, owned_row(db, Model, id, user) for a row by id (someone else's row is the same answer as a missing one). A unit's queries.own_x returns None and its route answers not_found; a core route calls owned_or_404. A rule owned_row cannot say — a job its owner shared, a built-in profile nobody owns, a folder read under FOR SHARE — is written out in the unit's own code, beside the reason.

All of it is held by tests/test_unit_fences.py, with no exemptions: the unit map, each unit's packages and edges (UNIT_PACKAGES, READS) and what may commit (UNIT_COMMITS, CORE_COMMITS) are listed there. It reads the code's AST, so it holds the forms written here: an owner column compared inline (a model's, an import alias's or an aliased() one's, by ==, !=, .in_(), or a filter_by(user_id=…)), a model built or an insert/update/delete (under any import name, of a model or its __table__) across units, and a .commit() call off the lists all fail it. What an AST cannot tell is review's to catch: an ORM write on another unit's loaded row (an attribute, a db.delete), SQL in text(), a column or model reached another way (bound to a variable, through __table__.c or a module path), and a commit made by a begin() block or a commit handed in.

Transaction ownership

This follows CLAUDE.md's canonical rule — the handler commits; a unit's service and every helper flush — so read it that way, not as an exception to it. The commit belongs to the operation owner; anything called inside another transaction only stages its change (db.add / flush) and lets the owner commit, so the whole operation is one transaction. audit.record() is the reference: it adds the audit row and the caller commits it together with the mutation it records, so an audit entry can never exist without its change (or vice-versa).

• The operation owner is the route handler, or a job. A unit's service flushes even when it makes a whole transition (service.cancel): the handler commits it, together with whatever else the request changed. What owns its OWN session commits it: quota and jobruns' run rows, the scheduler's jobs, a unit's jobs (the worker's jobdb writes, the syncs and ingests that commit per page or batch so a later failure loses nothing already stored), and a few core steps that must land whatever their caller does next (a token's last-used stamp, a refreshed YouTube grant) — the fence's UNIT_COMMITS and CORE_COMMITS name each with its reason, test_route_conventions a route's own committing helpers. Everything else flushes.
• SessionLocal is expire_on_commit=False: after a raw UPDATE + commit, an ORM object you already loaded is stale. Read the fresh value from the row (RETURNING, or a re-SELECT), not the ORM attribute — see service.decr_ref_count.

Shared counters and status: do it atomically at the DB

• Never read-modify-write a shared counter in Python (x.count = x.count + 1). Concurrent sessions lose an update. Use a DB-atomic statement — col = col + 1, or greatest(col - 1, 0) with RETURNING so the caller branches on the post-value under the row lock (service.incr_ref_count / decr_ref_count).
• Download job status is a JobStatus StrEnum (downloads/status.py), not bare strings. The HOLDING set defines which states hold a MediaAsset (so release accounting stays correct); STOPPED_BY_ROUTE is the route-initiated stop set. A worker child's terminal write goes through service.finalize_if_running (an atomic UPDATE … WHERE status='running') so a finishing child can't resurrect a job a route just stopped.

Config: one home per kind

• app.config.settings — boot/env config (DB URL, download root, worker concurrency): read once at startup, not user-tunable at runtime.
• sysconfig.SPECS — the runtime admin knobs (DB-override-or-default). A new admin-tunable value goes in SPECS; don't invent a fourth store. Each spec states when a change takes effect (applies, required): live keys are read where they are used, never at import (test-held). The scheduler's job intervals are SPECS keys too, edited on the Scheduler page, not the Configuration page.

The gate

Everything below must be green before a change is done (run via siftlode check): ruff + basedpyright (backend) · the lock check + tsc ×3 + eslint + prettier + knip + vite build (frontend) · vitest · pytest. The pytest lane runs in backend/docker-compose.test.yml — an ephemeral Postgres + the backend tree bind-mounted — so the DB-backed integration tests (the db fixture) actually run; a missing database is a hard failure, not a silent skip, unless a hand run asks for pure-logic only with SIFTLODE_TEST_REQUIRE_DB=0. See docs/development.md, and docs/gates.md for what each gate deliberately leaves out.