siftlode/docs/development.md

Development

How to run Siftlode locally and pass the gate. Conventions for the backend code itself live in backend/app/README.md.

Node

One major, everywhere. .nvmrc pins it; the Dockerfile's frontend build stage, frontend/package.json's engines and @types/node must all match it — and so must the machine you run the gate on. frontend/src/lib/nodeEngine.contract.test.ts fails the gate the moment any of them drift apart, including when yours is the odd one out. nvm use in the repo root picks it up.

This is a contract, not a preference: tsc and vitest read the engine's own Intl and web APIs, and those change between majors — the same currency format renders $160M on one and $160.0M on the next. A gate that is green on a major you do not ship is green about nothing.

Run it locally

The whole stack (Postgres + API + worker) comes up from the base compose file plus the localdev override:

docker compose -f docker-compose.yml -f docker-compose.localdev.yml up --build

Browse at http://localhost:8080 (Google login works — the OAuth redirect is http://localhost:8080/auth/callback). The API runs alembic upgrade head on startup, so the schema is always current.

For a fast frontend edit loop, run the Vite dev server with HMR against that API:

cd frontend && npm install && npm run dev   # http://localhost:5173, proxying the API on :8080

The dev server also serves the design lab at http://localhost:5173/design-lab: the open design decisions and the chosen patterns, rendered with the app's real CSS. It never reaches a production build (see DESIGN.md).

Adding a page

A frontend page is one folder, frontend/src/modules/<id>/ (the layering is in CLAUDE.md § Architecture shape):

1. The page itself, default-exported, and anything only it uses. Header content (a search box, a toggle) the page renders into the floating header itself through PageHeader; the shell knows no page.
2. module.ts: its ModuleDef (lib/moduleDef.ts): id, label keys, icon, an optional gate, the page as a lazyWithPreload(() => import(…)), and optionally a rail and a Store. Of its own folder it imports statically only its store (a …Provider / …Store file); the page and the rail load lazily. urlKeys names every address-bar key its view writes (the URL is the view: a bookmark reopens it), its list kept in lib/ beside its codec; the shell clears them on a page switch and opens a page without another page's keys. Every setting the page shows — a tab, a sort, a filter, a search, the item open in it — goes there: a codec in lib/<id>UrlState.ts (read the params into the view, write the view as updates, a default left out — or always written, when an address naming none falls back to a view remembered elsewhere), written with useUrlMirror (a sub-view through useHistorySubview's url, an overlay through useBackToClose's), always as the codec's result: a writer outside lib/ computes no key of its own. Read back first on mount (a link wins over anything remembered). A key that carries typed text — a search box's, a column filter map with a text filter — is also in typedUrlKeys (the client log keeps what was typed out). Add its round trip to e2e/bookmark.spec.ts, tagged with the page ({ tag: "@<id>" }). Only what an address cannot promise to reopen stays out, said in its codec (Home's draw, lib/homeUrlState).
3. index.ts, the module's public face: the definition, and only what other folders must read (its store's hooks). Another folder reaches the module through this file or not at all.
4. The id in lib/urlState's PAGES, and the definition in features/navigation/moduleRegistry.ts, in nav-rail order. The rail, the palette, the g chords, the tab title and the root's stores all read from there.
5. Its strings in both locales (src/i18n/locales).
6. A page with a side rail also names its panel below the module, in lib/ (rails share one layout and collapse machinery, which lib may not learn from a module): the PanelId and its default group order, storage and pref keys in lib/panelLayout.ts, its collapse flag in lib/useRail.ts (COLLAPSE), and those keys in lib/storage.ts. The rail calls useRail("<id>") and hands what it returns to ui/FilterRail; tsc names each table still missing the new id.

What holds it: moduleRegistry.contract.test.ts (every id defined once, from its own folder, lazily), moduleStores.contract.test.tsx (no store reads another module), moduleUrl.contract.test.ts (each URL key belongs to one module and matches what its codec writes; a writer outside lib/ is handed a literal or a lib/ function's result, never a key of its own; no key literal written or read from the live query undeclared; every key sorted as typed or not; every page with keys in a tagged bookmark round trip), the local/layers lint rule (imports only go down, a module only through its face) and chunkGuard in the build (no lazy page on the boot path), plus the i18n parity test.

The gate

A change is done when this is green, from the repo root:

bash scripts/check.sh        # every lane
bash scripts/check.sh lint   # the static lanes only (no tests, build or audits) — the quick loop
bash scripts/check.sh --brief          # either mode: one line per lane, output only for a red one
bash scripts/check.sh --list           # the lane names, one per line (runs nothing)
bash scripts/check.sh --only "vitest"  # one lane, by its exact name — never "the gate"

It runs every lane even after one fails and names each red one at the end.

The codebase's health numbers — size, complexity, test ratio, gate weight, comment noise — come from python scripts/audit_metrics.py (seconds; it reports, it never fails). --speed adds the speed baseline (frontend/e2e/speed.measure.ts: the feed's first card and a shared /watch link's time-to-play against the :8080 stack, with E2E_PASSWORD set). An epic that changes structure compares its before and after. What it runs, and why each lane runs where it does, is in the script's comments; frontend/src/lib/gates.contract.test.ts holds its lines, so a lane dropped or narrowed there fails a test. What each gate deliberately does NOT check — and why — is the gate register. There is no CI: the gate runs here, and a release runs it (plus e2e and the CVE scan) before anything is merged.

The lanes, for running one by hand:

Backend

# lint + format check (ruff is pinned in backend/requirements-dev.txt)
python -m ruff check --config backend/ruff.toml backend scripts
python -m ruff format --check --config backend/ruff.toml backend scripts

# type check — in the test image (requirements.lock does not install on Windows); new errors only,
# the pre-existing ones are in backend/.basedpyright/baseline.json
docker compose -f backend/docker-compose.test.yml run --rm --no-deps --build test \
  python -m basedpyright --project pyrightconfig.json

# tests — pure-logic + the DB-backed integration lane, against an ephemeral Postgres. The backend
# tree is bind-mounted, so it always tests your working files.
docker compose -f backend/docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from test
docker compose -f backend/docker-compose.test.yml down -v

# the lock against the advisory databases, at any severity
docker compose -f backend/docker-compose.test.yml run --rm --no-deps test \
  pip-audit -r requirements.lock --no-deps --disable-pip

Shell scripts

# every tracked *.sh at shellcheck's default bar (info included); the version is pinned in scripts/check.sh
git ls-files -z -- '*.sh' | xargs -0 shellcheck

A missing database or ffmpeg is a hard error, not a skip — so the gate can't print green while the DB-backed tests were quietly not run (SIFTLODE_TEST_REQUIRE_DB=0 is the explicit "pure-logic only" run for a machine without Postgres). Tests that need a database take the db fixture (backend/tests/conftest.py); the rest are pure-logic and run without one.

Frontend (from frontend/)

node ../scripts/check-node-modules.mjs   # node_modules == package-lock.json (else: npm ci)
npm run typecheck     # tsc x3 (app / node / e2e)
npm run lint          # eslint (typed rules included; warnings fail)
npm run format:check  # prettier
npm run knip          # unused files/exports/deps
npm run test          # vitest (incl. the compiler and gate-register contracts)
npx vite build        # the production bundle (devOnlyGuard fails it on dev-only code, chunkGuard on a lazy page loaded at boot)
npm audit --omit=dev --audit-level=high   # what ships to the browser, at high or above

Coverage is on demand, never enforced and not in the gate (E35): npm run test -- --coverage in frontend/, and for the backend docker compose -f backend/docker-compose.test.yml run --rm test python -m pytest --cov=app (then … down -v). The DB lane runs on every core (pytest -n auto, pytest-xdist): tests/db_per_worker.py gives each worker its own database (siftlode_test_gw<N>, created and migrated on first use), so the same tests run side by side. A serial run (no -n) still works and uses siftlode_test itself. The Playwright suite (npm run e2e) runs against the :8080 stack built from your tree; the release requires it. Before an image is pushed, it is scanned with Trivy (HIGH/CRITICAL with a fix fails; exceptions in .trivyignore.yaml, each with a reason and an expiry).

Branching

Trunk-based: branch off main, keep the branch gate-green, and land the whole change back to main in one reviewed step. Migrations are Alembic revisions under backend/alembic/versions; the API applies them on startup, and the DB-lane rebuilds the schema via alembic upgrade head, so a broken migration chain fails the tests.