Compare commits

...
9 Commits
Author SHA1 Message Date
peter 0dfb4a984b release: 0.99.2 (bug/b1-e34-defects) 2026-10-04 18:28:53 +02:00
peter 5bcd1d8d8a chore(release): 0.99.2 2026-10-04 18:19:37 +02:00
peter f49789665e fix(i18n): label the history cleanup's quota_reaped result
The scheduler row read 'quota reaped 0' — the raw key, humanised — because the new result key had no
label (found in the browser check; nothing tests result-key labels).
2026-10-04 18:03:19 +02:00
peter 0e077b84c8 fix(search): the live-search tooltip says what a search costs
It always said a live search spends the shared API quota, while the default source (the website's
own endpoint, scrape) spends none — the owner avoided a free search because of it. /api/me now names
the running search_source, and the tooltip names the cost under it: none for scrape (the enrich step
costs a unit or two), 100 units a page for the official API.
2026-10-04 17:53:19 +02:00
peter ce8ccfc3c0 fix(quota): prune quota history past a retention window
quota_events had no retention: one row per API spend plus a zero-unit row per live search, kept
forever. The daily history cleanup (audit_gc) now also prunes it past quota_retention_days (a new
Quota setting, default 365 so the admin usage view's longest window stays whole; 0 = forever). The
floor of two days keeps today's rows, which the per-user search cap counts.
2026-10-04 17:53:19 +02:00
peter 33dec56e45 fix(worker): obey the stored concurrency and log level, live
The admin page's download_worker_concurrency and log_level were stored but never reached the worker:
it read the env once at boot (worker.py) and never looked again. The worker now re-reads both every
10 s (_sync_knobs): extra loop threads start at once, surplus ones retire after their current job, and
a changed level is applied to the process. An unreadable database keeps what already runs.
2026-10-04 17:53:19 +02:00
peter 2a699e26ba release: 0.99.1 (improvement/drop-ci) 2026-10-04 06:45:48 +02:00
peter 13c8c05661 chore(release): 0.99.1 2026-10-04 06:36:06 +02:00
peter cc414de952 build: the gate is scripts/check.sh, and CI goes
The Gitea Actions gate repeated, after the merge, what a release already runs
before it, and its runner shared the git host's container: one run there
starved Gitea of memory until it stopped answering. The lanes now live in
scripts/check.sh, run the same by the release dispatcher and by hand, and the
gate-contract test holds its lines (the ruff and pip-audit checks read it
instead of the workflow). The two audits only CI ran join the gate: pip-audit
over the lock (pinned in the test image, blocking at any severity) and npm
audit of the production dependencies. The engine contracts drop the CI
containers; the register records that pushes have no server-side gate.
2026-10-04 06:20:58 +02:00
36 changed files with 626 additions and 276 deletions
-128
View File
@@ -1,128 +0,0 @@
# Server-side mirror of the local `siftlode check` gate (E1 S3 / DX-1), so `main` is always green
# independent of any one machine. Two jobs match the two local lanes:
# frontend — tsc (3 projects) + eslint + prettier --check + knip (both modes) + vitest + vite build
# + npm audit of the production dependencies
# backend — ruff + basedpyright + the full pytest suite (pure-logic + the DB-backed lane) against
# a real Postgres + pip-audit of the lock
#
# They are declared side by side but they DO NOT run in parallel: the lab has a single act_runner
# with one slot, so they queue. Measured across runs #86-#91 — the backend job starts at the exact
# second the frontend one ends, every time, and the pair costs ~4 + ~6 = **~10 minutes from the
# push**. Anything that waits on this gate has to budget for the sum, not the max; the ops
# `release` read-back timed out on its first real run for believing otherwise.
#
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
# failed to come up is a hard error, never a silent skip.
#
# The frontend container's node major is NOT free: it must equal the prod build image's and the dev
# machine's, because vitest/tsc read the engine's own Intl and web-API implementations and those split
# between majors. `.nvmrc` is the single source of truth and
# `frontend/src/lib/nodeEngine.contract.test.ts` fails this gate if any of them drift apart.
name: gate
# Branch pushes only. A release pushes `main` and then the `v<ver>` tag, and a bare `on: [push]`
# treated the tag as a second push of the IDENTICAL commit — every release ran this whole gate
# twice, to the same verdict, for nothing (runs #81-#90 are five such pairs). `branches: ['**']`
# keeps every branch covered and drops tag refs.
#
# It has to be `branches`, NOT the `tags-ignore` this obviously wants to be. Gitea's matchPushEvent
# requires EVERY named filter to match, and each one bails on the wrong ref kind: `tags-ignore`
# starts with `if !refName.IsTag() { break }`, so a filter meant to skip tags rejects every BRANCH
# push and the gate stops running entirely. `branches` bails the same way on a tag, which is the
# direction we want. (Measured in go-gitea v1.27.3 modules/actions/workflows.go, not assumed — it
# differs from GitHub, where `tags-ignore` alone leaves branches unfiltered.)
#
# If that filter were ever wrong the gate would not run at all — which is why `siftlode release`
# now ends by reading the verdict back for the sha it pushed, and says so out loud when there is
# no run for it. A gate that silently stops running is the failure this repo already lived through.
on:
push:
branches: ["**"]
pull_request:
jobs:
frontend:
runs-on: ubuntu-latest
container: node:24-bookworm
defaults:
run:
working-directory: frontend
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run typecheck
- run: npm run lint
- run: npm run format:check
- run: npm run knip
- run: npm run test
# The production bundle (E32): before this, `vite build` — and devOnlyGuard, which fails it if
# dev-only code reaches the bundle — first ran inside the image build, after main was pushed.
- run: npx vite build
# What ships to the browser: a known CVE at high or above in a production dependency fails.
# (The dev tooling's own advisories are burned down by hand; see docs/gates.md.)
- run: npm audit --omit=dev --audit-level=high
backend:
runs-on: ubuntu-latest
# A combined python+node image: python 3.12 (matches the prod Dockerfile) for the tests, and node
# so the JS actions (checkout) run. actions/setup-python is NOT usable here — the runner can't reach
# GitHub's python-versions download, so it must be baked into the image.
#
# This node runs the checkout action, never our code, so it is deliberately OUTSIDE the engine
# contract the frontend job is held to — it tracks `.nvmrc` as hygiene (node 20 went EOL in April
# 2026), but a missing upstream tag here must never be able to block the gate.
container: nikolaik/python-nodejs:python3.12-nodejs24
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: siftlode
POSTGRES_PASSWORD: siftlode
POSTGRES_DB: siftlode_test
options: >-
--health-cmd "pg_isready -U siftlode -d siftlode_test"
--health-interval 2s
--health-timeout 3s
--health-retries 30
env:
DATABASE_URL: postgresql+psycopg://siftlode:siftlode@postgres:5432/siftlode_test
SIFTLODE_TEST_REQUIRE_DB: "1"
steps:
- uses: actions/checkout@v4
# ffmpeg for the poster/thumbnail tests; postgresql-client for the readiness wait below.
- run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client
- name: Wait for Postgres
run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
- name: ruff
run: python -m ruff check --config backend/ruff.toml backend scripts
- name: ruff format --check
run: python -m ruff format --check --config backend/ruff.toml backend scripts
# The ninth lane (E27 S3). Run from backend/ so it resolves exactly as the local lane does —
# pyrightconfig.json's include/extraPaths are relative to the config, but the baseline is
# found relative to the working directory. New errors fail; the ~200 pre-existing ones are in
# .basedpyright/baseline.json and burn down epic by epic. Matching is per (file, rule, column
# span) plus a count, so a new error of the SAME shape in a file where a sibling was just
# fixed is absorbed — burn the hot files down first.
#
# The second step is not redundant. basedpyright TIGHTENS the baseline in place when the count
# drops, with no --writebaseline, and CI throws its workspace away — so without this a
# burn-down is green here and red on the developer's `siftlode check`, which has the same
# guard. Two gates over one tree must not answer differently.
- name: basedpyright
run: python -m basedpyright --project pyrightconfig.json
working-directory: backend
- name: basedpyright baseline is committed
run: git diff --exit-code -- backend/.basedpyright/baseline.json
- name: pytest
run: python -m pytest
working-directory: backend
# Dependency CVE audit over the pinned lock. BLOCKING since E32: it ran advisory-only, and when
# it was finally read it had three urllib3 advisories with a fix available (bumped in the same
# change). An advisory without a fix is answered with `--ignore-vuln <id>` here, a reason and a
# date in the comment, and a line in docs/gates.md.
- name: pip-audit
run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock
+14
View File
@@ -7,6 +7,20 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading). Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.99.2 — 2026-10-04
### Fixed
- Settings: the download worker's concurrency and the log level now take effect within seconds — before, the worker ignored both until it was rebuilt.
- The YouTube quota history no longer grows forever: the daily cleanup prunes it after a year (Configuration → Quota → Quota history kept).
- The live-search button no longer says a search spends the API quota: with the default website source the search itself is free. It names the cost when the official API is selected.
## 0.99.1 — 2026-10-04
### Fixed
- Behind the scenes: every release is now checked by one script on the build machine, including the security audit of every dependency.
## 0.99.0 — 2026-10-04 ## 0.99.0 — 2026-10-04
### New ### New
+4 -4
View File
@@ -13,14 +13,14 @@ router-free navigation provider.
## The gate — never merge or ship red ## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) + The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip (both modes) + vitest + vite build + pytest**, then a coverage report (no threshold). It runs eslint + prettier --check + knip (both modes) + vitest + vite build + pytest + pip-audit + npm audit**, then a coverage report (no threshold). It runs
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip` before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
green **before** any merge, release, or handing work to review. green **before** any merge, release, or handing work to review.
- Full gate (agents with the private ops dispatcher): `siftlode check`. - Full gate: `bash scripts/check.sh` (`lint` for the static lanes only); the private ops dispatcher's `siftlode check` runs the same script. Single lanes: [`docs/development.md`](docs/development.md) § The gate.
- Directly, without the dispatcher: the commands in [`docs/development.md`](docs/development.md) § The gate (frontend lanes, `ruff` over `backend scripts`, basedpyright in the test image, and `pytest` — the DB-lane needs Postgres). - **There is no CI** (owner, 2026-10-04): this gate, run here, is the only one — green before every hand-over, and `release` refuses to merge without it.
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every file and the CI/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: a new code file or any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped). - **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every file and the gate/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: a new code file or any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` and burn down epic by epic. Fixed some? Regenerate with `--writebaseline`. - **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` and burn down epic by epic. Fixed some? Regenerate with `--writebaseline`.
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`. - New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
+3 -3
View File
@@ -11,7 +11,7 @@ ARG BUILD_DATE=
# Stage 1: build the frontend SPA # Stage 1: build the frontend SPA
# The major is pinned to `.nvmrc` and held there by frontend/src/lib/nodeEngine.contract.test.ts: # The major is pinned to `.nvmrc` and held there by frontend/src/lib/nodeEngine.contract.test.ts:
# this image, the CI gate's image and the dev machine must be ONE engine (see that test for why). # this image and the dev machine the gate runs on must be ONE engine (see that test for why).
FROM node:24-alpine AS frontend FROM node:24-alpine AS frontend
WORKDIR /fe WORKDIR /fe
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules. # Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
@@ -79,8 +79,8 @@ RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nCo
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is # Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats # byte-reproducible and the exact versions are the ones the gate's pip-audit scanned — not whatever
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header). # floats up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# pip itself is removed once it has installed the lock (E32): nothing runs it at runtime, and it carries # pip itself is removed once it has installed the lock (E32): nothing runs it at runtime, and it carries
# its own VENDORED copies of urllib3, msgpack and setuptools, which Trivy reports as the image's own — # its own VENDORED copies of urllib3, msgpack and setuptools, which Trivy reports as the image's own —
# an attack surface and a CVE stream for a tool the running app never calls. # an attack surface and a CVE stream for a tool the running app never calls.
+1 -1
View File
@@ -1 +1 @@
0.99.0 0.99.2
+3
View File
@@ -260,6 +260,9 @@ class Settings(BaseSettings):
# Admin audit log: how long to keep rows (days; 0 = keep forever) + how often the GC runs. # Admin audit log: how long to keep rows (days; 0 = keep forever) + how often the GC runs.
audit_retention_days: int = 180 audit_retention_days: int = 180
audit_gc_minutes: int = 1440 audit_gc_minutes: int = 1440
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
quota_retention_days: int = 365
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat". # On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: DownloadLayout = "plex" download_layout: DownloadLayout = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles. # Default SponsorBlock (skip sponsor segments) for new custom profiles.
+18 -2
View File
@@ -7,11 +7,13 @@ always be allowed; expensive backfill must yield to the remaining budget.
import contextlib import contextlib
import contextvars import contextvars
from datetime import datetime, timezone from datetime import datetime, timedelta, timezone
from typing import Any, cast
from zoneinfo import ZoneInfo from zoneinfo import ZoneInfo
from sqlalchemy import func, select from sqlalchemy import delete, func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app import sysconfig from app import sysconfig
@@ -96,6 +98,20 @@ def pacific_today():
return datetime.now(_PACIFIC).date() return datetime.now(_PACIFIC).date()
def gc(db: Session, retention_days: int) -> int:
"""Delete quota-history rows older than `retention_days` (<= 0 keeps them forever) and return how
many went. Flushes only; the cleanup job commits. Today's rows are never in reach: the per-user
search cap counts them, and the floor of two days outlasts any Pacific day (25 h at most)."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=max(2, retention_days))
# A DELETE answers with a CursorResult; `execute` is typed as the general Result, without rowcount.
result = cast(
CursorResult[Any], db.execute(delete(QuotaEvent).where(QuotaEvent.created_at < cutoff))
)
return result.rowcount or 0
def pacific_day_start_utc() -> datetime: def pacific_day_start_utc() -> datetime:
"""UTC instant of the current Pacific day's midnight — aligns 'today' with the budget reset.""" """UTC instant of the current Pacific day's midnight — aligns 'today' with the budget reset."""
start = datetime.now(_PACIFIC).replace(hour=0, minute=0, second=0, microsecond=0) start = datetime.now(_PACIFIC).replace(hour=0, minute=0, second=0, microsecond=0)
+3
View File
@@ -148,6 +148,9 @@ def get_me(
"google_enabled": google_enabled(db), "google_enabled": google_enabled(db),
# Whether the optional Plex module is enabled (UI shows the Plex feed source when so). # Whether the optional Plex module is enabled (UI shows the Plex feed source when so).
"plex_enabled": sysconfig.get_bool(db, "plex_enabled"), "plex_enabled": sysconfig.get_bool(db, "plex_enabled"),
# Which live-search source runs, so the search button can say what a search costs: "scrape"
# spends no API quota, "api" spends 100 units a page.
"search_source": sysconfig.get_str(db, "search_source"),
"can_read": has_read_scope(user), "can_read": has_read_scope(user),
"can_write": has_write_scope(user), "can_write": has_write_scope(user),
"pending_invites": pending_invites, "pending_invites": pending_invites,
+6 -3
View File
@@ -425,12 +425,15 @@ def _people_enrich_job() -> None:
def _audit_gc_job() -> None: def _audit_gc_job() -> None:
# Prune audit-log rows older than the admin-set retention (audit_retention_days; 0 = keep # The history cleanup: audit-log rows past audit_retention_days and YouTube quota history past
# forever). Pure DB, no quota. # quota_retention_days (0 = keep forever, each). Pure DB, no quota.
def work(db): def work(db):
from app import sysconfig from app import sysconfig
return {"reaped": audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))} reaped = audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))
quota_reaped = quota.gc(db, sysconfig.get_int(db, "quota_retention_days"))
db.commit()
return {"reaped": reaped, "quota_reaped": quota_reaped}
_job("audit_gc", work) _job("audit_gc", work)
+1
View File
@@ -80,6 +80,7 @@ SPECS: tuple[ConfigSpec, ...] = (
), ),
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page). # Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
ConfigSpec("search_source", "choice", "quota", "search_source"), ConfigSpec("search_source", "choice", "quota", "search_source"),
ConfigSpec("quota_retention_days", "int", "quota", "quota_retention_days", min=0, max=3650),
# --- Backfill (recent-first first pass per channel) --- # --- Backfill (recent-first first pass per channel) ---
ConfigSpec( ConfigSpec(
"backfill_recent_max_videos", "backfill_recent_max_videos",
+59 -10
View File
@@ -5,7 +5,8 @@ from the API so long yt-dlp downloads + ffmpeg postprocessing never block web re
Design: Design:
* N loop threads (download_worker_concurrency) each claim a `queued` DownloadJob with * N loop threads (download_worker_concurrency) each claim a `queued` DownloadJob with
FOR UPDATE SKIP LOCKED and process it. FOR UPDATE SKIP LOCKED and process it. N and the log level follow the settings page while the
worker runs (`_sync_knobs`), not only the env it booted with.
* The shared MediaAsset is the dedup unit. A job whose asset is already `ready` finishes * The shared MediaAsset is the dedup unit. A job whose asset is already `ready` finishes
instantly (no re-download); a fresh asset is claimed pending→downloading by exactly one instantly (no re-download); a fresh asset is claimed pending→downloading by exactly one
worker (a DB compare-and-set), so two jobs for the same video+format never double-download. worker (a DB compare-and-set), so two jobs for the same video+format never double-download.
@@ -48,12 +49,60 @@ from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset
log = logs.get_logger(__name__) log = logs.get_logger(__name__)
_stop = threading.Event() _stop = threading.Event()
# How many loop threads may claim jobs. A thread whose index is at or above it stops after its
# current job, so lowering the knob never abandons a download halfway.
_target = 1
# How often the worker re-reads the knobs it obeys; the settings page says nothing about a restart.
_KNOB_SECONDS = 10.0
def _handle_signal(signum, frame): def _handle_signal(signum, frame):
_stop.set() _stop.set()
def _read_knobs() -> tuple[int, str] | None:
"""The stored concurrency and log level, or None when the database cannot answer — the caller
then keeps what already runs."""
try:
with SessionLocal() as db:
return (
max(1, sysconfig.get_int(db, "download_worker_concurrency")),
sysconfig.get_str(db, "log_level"),
)
except Exception: # noqa: BLE001 — any DB or config failure keeps the knobs already in force
log.warning(
"could not read the worker's settings; keeping the ones in force",
exc_info=True,
extra=logs.event("download.worker.knobs_unreadable"),
)
return None
def _sync_knobs(threads: dict[int, threading.Thread], level: str | None) -> str | None:
"""Apply the stored knobs: run loop threads up to the concurrency (surplus ones retire after
their job, see `_target`) and re-apply the log level when it changed. Returns the level in force."""
global _target
knobs = _read_knobs()
if knobs is not None:
target, stored_level = knobs
if target != _target:
log.info(
"download concurrency is now %d",
target,
extra=logs.event("download.worker.concurrency", before=_target, after=target),
)
_target = target
if stored_level != level:
logs.set_level(stored_level)
level = stored_level
for i in range(_target):
t = threads.get(i)
if t is None or not t.is_alive():
threads[i] = threading.Thread(target=_worker_loop, args=(i,), daemon=True)
threads[i].start()
return level
# --- claim + recovery ----------------------------------------------------------------------- # --- claim + recovery -----------------------------------------------------------------------
@@ -576,7 +625,7 @@ def _finish_from_cache(job_id: int, asset_id: int) -> None:
def _worker_loop(idx: int) -> None: def _worker_loop(idx: int) -> None:
while not _stop.is_set(): while not _stop.is_set() and idx < _target:
try: try:
job_id = _claim_job() job_id = _claim_job()
except Exception: except Exception:
@@ -634,15 +683,15 @@ def main() -> None:
# has migrated — those few seconds of batches fail and say so on stderr; nothing else breaks. # has migrated — those few seconds of batches fail and say so on stderr; nothing else breaks.
logsink.start() logsink.start()
_recover_orphans() _recover_orphans()
n = max(1, settings.download_worker_concurrency) global _target
log.info("Download worker started (concurrency=%d, root=%s).", n, settings.download_root) _target = max(1, settings.download_worker_concurrency) # used only if the DB can't answer
threads = [threading.Thread(target=_worker_loop, args=(i,), daemon=True) for i in range(n)] threads: dict[int, threading.Thread] = {}
for t in threads: level = _sync_knobs(threads, None)
t.start() log.info("Download worker started (concurrency=%d, root=%s).", _target, settings.download_root)
while not _stop.is_set(): while not _stop.wait(_KNOB_SECONDS):
_stop.wait(1.0) level = _sync_knobs(threads, level)
log.info("Download worker stopping…") log.info("Download worker stopping…")
for t in threads: for t in threads.values():
t.join(timeout=5) t.join(timeout=5)
+2
View File
@@ -9,8 +9,10 @@
# writes is keyed to the diagnostics THIS version produces. # writes is keyed to the diagnostics THIS version produces.
# pytest-cov: the coverage REPORT the gate prints (E32 D4) — a number to read, deliberately with no # pytest-cov: the coverage REPORT the gate prints (E32 D4) — a number to read, deliberately with no
# threshold; "every regression test seen red" is the bar, not a percentage. # threshold; "every regression test seen red" is the bar, not a percentage.
# pip-audit: the gate's audit of requirements.lock (scripts/check.sh), blocking at any severity.
basedpyright==1.40.1 basedpyright==1.40.1
pytest==8.4.2 pytest==8.4.2
pytest-cov==7.1.0 pytest-cov==7.1.0
ruff==0.15.21 ruff==0.15.21
httpx2==2.13.0 httpx2==2.13.0
pip-audit==2.10.1
+3 -3
View File
@@ -1,9 +1,9 @@
# Pinned dependency lock for reproducible builds (SEC-5). GENERATED — do not hand-edit. # Pinned dependency lock for reproducible builds (SEC-5). GENERATED — do not hand-edit.
# Source of truth is requirements.txt (the floor ranges); regenerate after editing it with: # Source of truth is requirements.txt (the floor ranges); regenerate after editing it with:
# docker run --rm -v "$PWD":/b python:3.12-slim sh -c 'pip install -q -r /b/requirements.txt && pip freeze --exclude pip --exclude setuptools --exclude wheel' | sort # docker run --rm -v "$PWD":/b python:3.12-slim sh -c 'pip install -q -r /b/requirements.txt && pip freeze --exclude pip --exclude setuptools --exclude wheel' | sort
# The prod Dockerfile + Dockerfile.test + CI install THIS file; pip-audit scans it in CI (blocking # The prod Dockerfile + Dockerfile.test install THIS file; the gate's pip-audit (scripts/check.sh)
# since E32). A security fix may be ONE pin bumped in place instead of a full regeneration, which # scans it, blocking since E32. A security fix may be ONE pin bumped in place instead of a full
# would move every other package too — then `pip install -r requirements.lock && pip check` must # regeneration, which would move every other package too — then `pip install -r requirements.lock && pip check` must
# pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177, E32). # pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177, E32).
APScheduler==3.11.3 APScheduler==3.11.3
Authlib==1.7.2 Authlib==1.7.2
+1 -1
View File
@@ -39,7 +39,7 @@ _MISSING = object()
# A would-be silent skip (no DB, or no ffmpeg) is a hard error unless the run says otherwise — the # A would-be silent skip (no DB, or no ffmpeg) is a hard error unless the run says otherwise — the
# suite must not print green while the DB-backed / ffmpeg tests were quietly not run. ON by default # suite must not print green while the DB-backed / ffmpeg tests were quietly not run. ON by default
# since E32: as an opt-in only the compose lane and CI set, a hand-run `pytest` skipped ~60 DB tests # since E32: as an opt-in only the compose lane set, a hand-run `pytest` skipped ~60 DB tests
# and still passed. `SIFTLODE_TEST_REQUIRE_DB=0` is the explicit "pure-logic only" run. # and still passed. `SIFTLODE_TEST_REQUIRE_DB=0` is the explicit "pure-logic only" run.
_REQUIRE_FULL = os.getenv("SIFTLODE_TEST_REQUIRE_DB", "1") != "0" _REQUIRE_FULL = os.getenv("SIFTLODE_TEST_REQUIRE_DB", "1") != "0"
+26
View File
@@ -0,0 +1,26 @@
"""`/api/me` names the live-search source, so the search button's tooltip can say what a search costs
under the source that actually runs (the default website source spends no API quota)."""
from app import sysconfig
_PW = "correct horse battery"
def _me(client):
r = client.get("/api/me")
assert r.status_code == 200 and r.json()["authenticated"] is True
return r.json()
def test_me_names_the_running_search_source(client, db, make_user):
# /api/me resolves its user through optional_current_user, which the as_user override does not
# cover — so sign in for real (see test_me_email_flag).
user = make_user(password=_PW)
login = client.post("/auth/password-login", json={"email": user.email, "password": _PW})
assert login.status_code == 200
assert _me(client)["search_source"] == "scrape" # the default
sysconfig.set_value(db, "search_source", "api")
db.commit()
assert _me(client)["search_source"] == "api"
+2 -2
View File
@@ -9,7 +9,7 @@ independent literal, there was no `.python-version`, no `requires-python`, and r
**The contract is a chain, and it is split across two test files because no single runner can see **The contract is a chain, and it is split across two test files because no single runner can see
every link.** This suite runs inside the test image, which mounts `backend/` and nothing else, so it every link.** This suite runs inside the test image, which mounts `backend/` and nothing else, so it
cannot read the repo-root Dockerfile or the workflow. It therefore holds the link it CAN see, and the cannot read the repo-root Dockerfile. It therefore holds the link it CAN see, and the
one that matters most here: one that matters most here:
the interpreter actually running == backend/pyrightconfig.json == backend/ruff.toml the interpreter actually running == backend/pyrightconfig.json == backend/ruff.toml
@@ -17,7 +17,7 @@ one that matters most here:
and `frontend/src/lib/pythonEngine.contract.test.ts` — which runs on the host, with the whole tree — and `frontend/src/lib/pythonEngine.contract.test.ts` — which runs on the host, with the whole tree —
holds the rest of the chain: holds the rest of the chain:
/.python-version == pyrightconfig.json == ruff.toml == Dockerfile == Dockerfile.test == check.yml /.python-version == pyrightconfig.json == ruff.toml == Dockerfile == Dockerfile.test
Together they close the loop. To move the version: edit `/.python-version`, then follow the failures Together they close the loop. To move the version: edit `/.python-version`, then follow the failures
in both files. Every place they name is a place that would otherwise have been forgotten. in both files. Every place they name is a place that would otherwise have been forgotten.
+46
View File
@@ -0,0 +1,46 @@
"""`quota_events` has a retention: the daily history-cleanup job (`audit_gc`) prunes rows older than
`quota_retention_days`, as it does the audit log. Without it the table only ever grew — one row per
API spend plus a zero-unit row per live search."""
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from app import scheduler, sysconfig
from app.models import QuotaEvent
def _event(days_ago: int) -> QuotaEvent:
return QuotaEvent(
user_id=None,
action="videos.list",
units=1,
created_at=datetime.now(timezone.utc) - timedelta(days=days_ago),
)
def _remaining(db) -> list[int]:
db.expire_all()
return sorted(e.units for e in db.execute(select(QuotaEvent)).scalars())
def test_the_history_cleanup_prunes_quota_events_past_the_retention(db):
old, recent = _event(400), _event(10)
recent.units = 7
db.add_all([old, recent])
db.commit()
scheduler.JOB_FUNCS["audit_gc"]()
assert _remaining(db) == [7] # the default keeps a year, which the admin usage view can show
def test_a_zero_retention_keeps_quota_history_forever(db):
db.add(_event(4000))
db.commit()
sysconfig.set_value(db, "quota_retention_days", 0)
db.commit()
scheduler.JOB_FUNCS["audit_gc"]()
assert _remaining(db) == [1]
+109
View File
@@ -0,0 +1,109 @@
"""The worker obeys the two admin knobs that concern it — `download_worker_concurrency` and
`log_level` — as stored in the settings page, not only the env it booted with. It used to read
`settings.download_worker_concurrency` and `settings.log_level` once at start, so the admin page
showed values the worker never applied. No DB or real threads of work here: the stored values come
from a patched `sysconfig`, and the loop threads are stand-ins that only record their index."""
import threading
import pytest
from app import worker
class _NoSession:
def __enter__(self):
return None
def __exit__(self, *exc):
return False
@pytest.fixture
def stored(monkeypatch):
"""The settings page's values, served to the worker instead of a database."""
values = {"download_worker_concurrency": 3, "log_level": "DEBUG"}
monkeypatch.setattr(worker, "SessionLocal", _NoSession)
monkeypatch.setattr(worker.sysconfig, "get_int", lambda _db, key: int(values[key]))
monkeypatch.setattr(worker.sysconfig, "get_str", lambda _db, key: str(values[key]))
return values
@pytest.fixture
def levels(monkeypatch):
applied: list[str] = []
monkeypatch.setattr(worker.logs, "set_level", applied.append)
return applied
@pytest.fixture
def loops(monkeypatch):
"""Stand-in loop threads: each records its index and parks until the worker stops."""
started: list[int] = []
lock = threading.Lock()
def fake_loop(idx: int) -> None:
with lock:
started.append(idx)
worker._stop.wait()
monkeypatch.setattr(worker, "_worker_loop", fake_loop)
yield started
worker._stop.set()
@pytest.fixture(autouse=True)
def _clean_stop():
worker._stop.clear()
yield
worker._stop.clear()
def test_worker_boots_with_the_stored_concurrency_and_log_level(monkeypatch, stored, levels, loops):
monkeypatch.setattr(worker.settings, "download_worker_concurrency", 2)
monkeypatch.setattr(worker.settings, "worker_enabled", True)
monkeypatch.setattr(worker.logs, "configure_logging", lambda _source: None)
monkeypatch.setattr(worker.signal, "signal", lambda *_a: None)
monkeypatch.setattr(worker, "_wait_for_schema", lambda: True)
monkeypatch.setattr(worker.logsink, "start", lambda: None)
monkeypatch.setattr(worker, "_recover_orphans", lambda: None)
threading.Timer(0.5, worker._stop.set).start()
worker.main()
assert sorted(loops) == [0, 1, 2] # the stored 3, not the env's 2
assert levels and levels[-1] == "DEBUG"
def test_a_changed_knob_applies_without_a_restart(stored, levels, loops):
threads: dict[int, threading.Thread] = {}
level = worker._sync_knobs(threads, None)
assert level == "DEBUG"
assert worker._target == 3
stored["download_worker_concurrency"] = 5
stored["log_level"] = "WARNING"
level = worker._sync_knobs(threads, level)
assert worker._target == 5
assert sorted(threads) == [0, 1, 2, 3, 4]
assert level == "WARNING" and levels[-1] == "WARNING"
# Lowering it retires the surplus threads after their current job: they see the lower target.
stored["download_worker_concurrency"] = 1
worker._sync_knobs(threads, level)
assert worker._target == 1
def test_an_unreadable_database_keeps_what_runs(monkeypatch, levels, loops):
def boom():
raise RuntimeError("db down")
monkeypatch.setattr(worker, "SessionLocal", boom)
monkeypatch.setattr(worker, "_target", 2)
threads: dict[int, threading.Thread] = {}
assert worker._sync_knobs(threads, "INFO") == "INFO"
assert worker._target == 2
assert sorted(threads) == [0, 1] # the loops still run on the boot value
assert levels == []
+20 -4
View File
@@ -5,7 +5,7 @@ How to run Siftlode locally and pass the gate. Conventions for the backend code
## Node ## Node
One major, everywhere. `.nvmrc` pins it; the CI container, the `Dockerfile`'s frontend build stage, One major, everywhere. `.nvmrc` pins it; the `Dockerfile`'s frontend build stage,
`frontend/package.json`'s `engines` and `@types/node` must all match it — and so must the machine you `frontend/package.json`'s `engines` and `@types/node` must all match it — and so must the machine you
run the gate on. `frontend/src/lib/nodeEngine.contract.test.ts` fails the gate the moment any of them run the gate on. `frontend/src/lib/nodeEngine.contract.test.ts` fails the gate the moment any of them
drift apart, including when yours is the odd one out. `nvm use` in the repo root picks it up. drift apart, including when yours is the odd one out. `nvm use` in the repo root picks it up.
@@ -38,9 +38,20 @@ decisions and the chosen patterns, rendered with the app's real CSS. It never re
## The gate ## The gate
A change is done when all of this is green. CI runs the same checks server-side A change is done when this is green, from the repo root:
(`.gitea/workflows/check.yml`), so "main is always green". What each gate deliberately does NOT
check — and why — is the [gate register](gates.md); a test fails when a narrowing is missing from it. ```bash
bash scripts/check.sh # every lane
bash scripts/check.sh lint # the static lanes only (no tests, build or audits) — the quick loop
```
It runs every lane even after one fails and names each red one at the end. What it runs, and why
each lane runs where it does, is in the script's comments; `frontend/src/lib/gates.contract.test.ts`
holds its lines, so a lane dropped or narrowed there fails a test. What each gate deliberately does
NOT check — and why — is the [gate register](gates.md). There is no CI: the gate runs here, and a
release runs it (plus e2e and the CVE scan) before anything is merged.
The lanes, for running one by hand:
**Backend** **Backend**
@@ -58,6 +69,10 @@ docker compose -f backend/docker-compose.test.yml run --rm --no-deps --build tes
# coverage total. The backend tree is bind-mounted, so it always tests your working files. # coverage total. The backend tree is bind-mounted, so it always tests your working files.
docker compose -f backend/docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from test docker compose -f backend/docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from test
docker compose -f backend/docker-compose.test.yml down -v docker compose -f backend/docker-compose.test.yml down -v
# the lock against the advisory databases, at any severity
docker compose -f backend/docker-compose.test.yml run --rm --no-deps test \
pip-audit -r requirements.lock --no-deps --disable-pip
``` ```
A missing database or ffmpeg is a hard error, not a skip — so the gate can't print green while the A missing database or ffmpeg is a hard error, not a skip — so the gate can't print green while the
@@ -75,6 +90,7 @@ npm run format:check # prettier
npm run knip # unused files/exports/deps — and unused by what SHIPS (--production) npm run knip # unused files/exports/deps — and unused by what SHIPS (--production)
npm run test # vitest (incl. the compiler, size/complexity and gate-register contracts) npm run test # vitest (incl. the compiler, size/complexity and gate-register contracts)
npx vite build # the production bundle (devOnlyGuard fails it on dev-only code) npx vite build # the production bundle (devOnlyGuard fails it on dev-only code)
npm audit --omit=dev --audit-level=high # what ships to the browser, at high or above
``` ```
Coverage is reported, never enforced: `npm run test -- --coverage`, and the pytest lane's last line. Coverage is reported, never enforced: `npm run test -- --coverage`, and the pytest lane's last line.
+5 -5
View File
@@ -14,10 +14,10 @@ name its invariant. Whatever is left narrowed is listed here, and
not an oversight; a narrowing missing from here is a test failure. not an oversight; a narrowing missing from here is a test failure.
**The snapshot.** ESLint's effective config for every code file `eslint .` reaches, and the lines **The snapshot.** ESLint's effective config for every code file `eslint .` reaches, and the lines
that run the lanes (`.gitea/workflows/check.yml`, `backend/docker-compose.test.yml`, that run the lanes (`scripts/check.sh`, `backend/docker-compose.test.yml`,
`backend/Dockerfile.test`, the npm scripts, `vitest.config.ts`, `.prettierignore`), are committed in `backend/Dockerfile.test`, the npm scripts, `vitest.config.ts`, `.prettierignore`), are committed in
[`frontend/src/lib/gates.snapshot.json`](../frontend/src/lib/gates.snapshot.json). The test fails on [`frontend/src/lib/gates.snapshot.json`](../frontend/src/lib/gates.snapshot.json). The test fails on
ANY difference — a rule, an option, a glob that moves one file to another config, a new ignore, a CI ANY difference — a rule, an option, a glob that moves one file to another config, a new ignore, a gate
flag. So a change to a gate is always a diff of that file in review, never a shape a detector failed flag. So a change to a gate is always a diff of that file in review, never a shape a detector failed
to model (E32 review: three rounds of a cleverer detector each found a bypass of the last one's to model (E32 review: three rounds of a cleverer detector each found a bypass of the last one's
fixes). Regenerate it with `GATE_SNAPSHOT=update npx vitest run src/lib/gates.contract.test.ts` fixes). Regenerate it with `GATE_SNAPSHOT=update npx vitest run src/lib/gates.contract.test.ts`
@@ -117,17 +117,17 @@ project checks needs a row.
| Scope | Narrowed | Why | Measured | | Scope | Narrowed | Why | Measured |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| check | e2e (Playwright) not in `check` | It needs the :8080 stack built from the tree (minutes). It runs in `check-full`, and `release` rebuilds the stack and requires it (E32 D2) — nothing ships without it. | 5 specs × 3 engines | | check | e2e (Playwright) not in `check` | It needs the :8080 stack built from the tree (minutes). It runs in `check-full`, and `release` rebuilds the stack and requires it (E32 D2) — nothing ships without it. | 5 specs × 3 engines |
| CI | no e2e, no Trivy | The lab runner has no browser stack, and Trivy's DB download is not reachable from it. Both run in `release` / `publish` on the dev machine, before anything is pushed. | — | | pushes | no server-side gate (no CI) | Owner decision 2026-10-04. The gate (`scripts/check.sh`) runs on the dev machine before work is handed over, and `release` runs it, then e2e, before anything is merged and Trivy before the image is pushed — the CI only repeated that, after the merge. Its runner shared the git host's container, and one run there starved Gitea of memory until it stopped answering. What it added — a fresh `npm ci` on Linux — the image build does at publish. | — |
| coverage | no threshold | Owner decision E32 D4: reported on both sides, never enforced. "Every regression test seen red" is the bar; a percentage invites tests written to move it. | frontend 80% / backend 78% of lines (E32 S1) | | coverage | no threshold | Owner decision E32 D4: reported on both sides, never enforced. "Every regression test seen red" is the bar; a percentage invites tests written to move it. | frontend 80% / backend 78% of lines (E32 S1) |
| CVE gate | HIGH/CRITICAL with a fix available only | `--ignore-unfixed` + `--severity HIGH,CRITICAL`: a finding with no fix cannot be acted on by an upgrade; it shows in the scan output. A fixable one fails the publish, and the only way past it is `.trivyignore.yaml`. | 0 open (E32 S1) | | CVE gate | HIGH/CRITICAL with a fix available only | `--ignore-unfixed` + `--severity HIGH,CRITICAL`: a finding with no fix cannot be acted on by an upgrade; it shows in the scan output. A fixable one fails the publish, and the only way past it is `.trivyignore.yaml`. | 0 open (E32 S1) |
| npm audit (CI) | `--omit=dev --audit-level=high` | What ships to the browser, at high or above. The dev toolchain's advisories were burned down by hand in E32 S1 (18 → 0). | 0 | | npm audit | `--omit=dev --audit-level=high` | What ships to the browser, at high or above. The dev toolchain's advisories were burned down by hand in E32 S1 (18 → 0). | 0 |
| React Compiler | the functions it skips, pinned | `components/compilerCoverage.contract.test.ts` — the list may only shrink. End state: only VirtualGrid (TanStack Virtual, an IncompatibleLibrary) remains. | 93 (E32 S1) → 29 (E32 S3) → 1, VirtualGrid (E32 S4) | | React Compiler | the functions it skips, pinned | `components/compilerCoverage.contract.test.ts` — the list may only shrink. End state: only VirtualGrid (TanStack Virtual, an IncompatibleLibrary) remains. | 93 (E32 S1) → 29 (E32 S3) → 1, VirtualGrid (E32 S4) |
| size / complexity | today's offenders as ceilings | `src/lib/sourceSize.contract.test.ts`, `backend/tests/test_source_size.py`: listed files and functions may not grow, nothing new may cross the limit, an entry back under it must be deleted. A function is held by its name, not its rank in the file. | files >800 lines: 13 FE / 9 BE; functions over the limit: 64 FE / 32 BE (E32 review) | | size / complexity | today's offenders as ceilings | `src/lib/sourceSize.contract.test.ts`, `backend/tests/test_source_size.py`: listed files and functions may not grow, nothing new may cross the limit, an entry back under it must be deleted. A function is held by its name, not its rank in the file. | files >800 lines: 13 FE / 9 BE; functions over the limit: 64 FE / 32 BE (E32 review) |
| size / complexity | shipped code only — not tests, test kits, e2e or `scripts/` | The ratchet guards what runs in production. A test file is long because it is a list of cases, and a test kit is test code; splitting either moves lines without making anything easier to change. `scripts/` is release and seed tooling, not in the image. | outside it: tests >800 lines 2 FE / 5 BE; functions over the limit 2 FE test code (`gates.contract.test.ts` `eslintNarrowings` 23, E32 review round 3; `compilerSkips.testkit.ts` 25), 1 BE test (11), 1 script (`e2e_seed.py` `main` 13) | | size / complexity | shipped code only — not tests, test kits, e2e or `scripts/` | The ratchet guards what runs in production. A test file is long because it is a list of cases, and a test kit is test code; splitting either moves lines without making anything easier to change. `scripts/` is release and seed tooling, not in the image. | outside it: tests >800 lines 2 FE / 5 BE; functions over the limit 2 FE test code (`gates.contract.test.ts` `eslintNarrowings` 23, E32 review round 3; `compilerSkips.testkit.ts` 25), 1 BE test (11), 1 script (`e2e_seed.py` `main` 13) |
## Accepted findings ## Accepted findings
Entries in `.trivyignore.yaml` (CVE gate) and `--ignore-vuln` flags on CI's `pip-audit` step. Each Entries in `.trivyignore.yaml` (CVE gate) and `--ignore-vuln` flags on the gate's `pip-audit` lane (`scripts/check.sh`). Each
must have a row here (Scope `accepted`) with its reason and its expiry; the test fails on an entry must have a row here (Scope `accepted`) with its reason and its expiry; the test fails on an entry
without one, and on a `.trivyignore.yaml` entry with no `statement` or no `expired_at`. without one, and on a `.trivyignore.yaml` entry with no `statement` or no `expired_at`.
@@ -0,0 +1,47 @@
// @vitest-environment jsdom
//
// The live-search button's tooltip says what a search costs under the search source the instance
// actually runs. It used to say "uses shared API quota" always, while the default source (the
// website's own endpoint) spends none — so the owner avoided a free search for months.
import { describe, it, expect, vi, afterEach } from "vitest";
import { render, screen, cleanup } from "@testing-library/react";
vi.mock("react-i18next", async (importOriginal) => ({
...(await importOriginal<typeof import("react-i18next")>()),
useTranslation: () => ({ t: (k: string) => k, i18n: { language: "en" } }),
}));
vi.mock("./navigationContext", () => ({ useNavigation: () => ({ page: "feed" }) }));
vi.mock("./feedFiltersContext", () => ({
useFeedFilters: () => ({ q: "cats", sort: "newest" }),
useFeedFiltersActions: () => ({ setFilters: vi.fn() }),
}));
vi.mock("./playlistsContext", () => ({
usePlaylists: () => ({ search: "" }),
usePlaylistsActions: () => ({ setSearch: vi.fn() }),
}));
vi.mock("./channelsContext", () => ({
useChannels: () => ({ search: "" }),
useChannelsActions: () => ({ setSearch: vi.fn(), goToFullHistory: vi.fn() }),
}));
vi.mock("./SyncStatus", () => ({ default: () => null }));
import Header from "./Header";
import type { Me } from "../lib/api";
afterEach(cleanup);
function tipFor(source: Me["search_source"]): string | null {
const me = { is_demo: false, plex_enabled: false, search_source: source } as Me;
render(<Header me={me} onYtSearch={() => {}} />);
return screen.getByRole("button", { name: /header\.searchYoutube$/ }).getAttribute("title");
}
describe("the live-search tooltip", () => {
it("does not claim a quota cost under the free website source", () => {
expect(tipFor("scrape")).toBe("header.searchYoutubeTip");
});
it("names the quota cost when the instance searches through the official API", () => {
expect(tipFor("api")).toBe("header.searchYoutubeTipApi");
});
});
+3 -1
View File
@@ -60,7 +60,9 @@ const Header = forwardRef<
}} }}
onGo={() => onYtSearch(trimmed)} onGo={() => onYtSearch(trimmed)}
goLabel={t("header.searchYoutube")} goLabel={t("header.searchYoutube")}
goTitle={t("header.searchYoutubeTip")} goTitle={t(
me.search_source === "api" ? "header.searchYoutubeTipApi" : "header.searchYoutubeTip",
)}
goIcon={<YouTubeIcon className="w-4 h-4" />} goIcon={<YouTubeIcon className="w-4 h-4" />}
goDisabled={!trimmed || me.is_demo} goDisabled={!trimmed || me.is_demo}
/> />
@@ -1,4 +1,22 @@
[ [
{
"version": "0.99.2",
"date": "2026-10-04",
"new": [],
"fixed": [
"Settings: the download worker's concurrency and the log level now take effect within seconds — before, the worker ignored both until it was rebuilt.",
"The YouTube quota history no longer grows forever: the daily cleanup prunes it after a year (Configuration → Quota → Quota history kept).",
"The live-search button no longer says a search spends the API quota: with the default website source the search itself is free. It names the cost when the official API is selected."
]
},
{
"version": "0.99.1",
"date": "2026-10-04",
"new": [],
"fixed": [
"Behind the scenes: every release is now checked by one script on the build machine, including the security audit of every dependency."
]
},
{ {
"version": "0.99.0", "version": "0.99.0",
"date": "2026-10-04", "date": "2026-10-04",
@@ -1,4 +1,22 @@
[ [
{
"version": "0.99.2",
"date": "2026-10-04",
"new": [],
"fixed": [
"Beállítások: a letöltő párhuzamossága és a naplószint pár másodpercen belül érvénybe lép — eddig a letöltő mindkettőt figyelmen kívül hagyta az újraépítéséig.",
"A YouTube-kvótaelőzmény többé nem nő a végtelenségig: a napi takarítás egy év után törli (Konfiguráció → Kvóta → Kvótaelőzmények megőrzése).",
"Az élő keresés gombja többé nem állítja, hogy a keresés API-kvótát fogyaszt: az alapértelmezett, weboldalas forrással maga a keresés ingyenes. A hivatalos API kiválasztásakor kiírja a költséget."
]
},
{
"version": "0.99.1",
"date": "2026-10-04",
"new": [],
"fixed": [
"A háttérben: minden kiadást egyetlen ellenőrző script vizsgál a build gépen, minden függőség biztonsági auditjával együtt."
]
},
{ {
"version": "0.99.0", "version": "0.99.0",
"date": "2026-10-04", "date": "2026-10-04",
+4
View File
@@ -62,6 +62,10 @@
"label": "Live search source", "label": "Live search source",
"hint": "Where live YouTube search results come from. “Website” reads YouTube's own search endpoint and spends no API quota (recommended). “Official API” uses search.list (100 units/page) — switch to it if the website source ever breaks or is blocked." "hint": "Where live YouTube search results come from. “Website” reads YouTube's own search endpoint and spends no API quota (recommended). “Official API” uses search.list (100 units/page) — switch to it if the website source ever breaks or is blocked."
}, },
"quota_retention_days": {
"label": "Quota history kept (days)",
"hint": "How long the per-action YouTube quota history (who spent what) is kept before the history cleanup job prunes it. 0 = keep forever. Default 365, so the usage view's longest window stays whole."
},
"backfill_recent_max_videos": { "backfill_recent_max_videos": {
"label": "Recent backfill — max videos", "label": "Recent backfill — max videos",
"hint": "Newest videos fetched per channel on the first pass." "hint": "Newest videos fetched per channel on the first pass."
+2 -1
View File
@@ -2,7 +2,8 @@
"searchPlaceholder": "Search your subscriptions…", "searchPlaceholder": "Search your subscriptions…",
"clearSearch": "Clear search", "clearSearch": "Clear search",
"searchYoutube": "Search YouTube", "searchYoutube": "Search YouTube",
"searchYoutubeTip": "Search live on YouTube for this term (uses shared API quota). You can also press Enter.", "searchYoutubeTip": "Search live on YouTube for this term — the search itself spends no API quota; adding the new videos it finds costs a unit or two. You can also press Enter.",
"searchYoutubeTipApi": "Search live on YouTube for this term — uses the shared API quota (100 units a page). You can also press Enter.",
"go": "Go", "go": "Go",
"goTip": "Search (or press Enter)", "goTip": "Search (or press Enter)",
"channelManager": "Channels", "channelManager": "Channels",
+3 -2
View File
@@ -76,7 +76,7 @@
"plex_sync": "Mirrors the enabled Plex libraries into the app's catalogue — the metadata behind Plex browsing, search, filters and the info pages (genres, cast, ratings, artwork). If it stops, newly added or changed Plex titles don't appear until the next sync.", "plex_sync": "Mirrors the enabled Plex libraries into the app's catalogue — the metadata behind Plex browsing, search, filters and the info pages (genres, cast, ratings, artwork). If it stops, newly added or changed Plex titles don't appear until the next sync.",
"plex_watch_sync": "Pulls recent Plex watch changes (finished episodes, resume positions) into Siftlode and re-pushes any local watch changes that didn't reach Plex. The cheap two-way keep-in-sync between full reconciles. If it stops, watched/resume made in the Plex app takes longer to show here.", "plex_watch_sync": "Pulls recent Plex watch changes (finished episodes, resume positions) into Siftlode and re-pushes any local watch changes that didn't reach Plex. The cheap two-way keep-in-sync between full reconciles. If it stops, watched/resume made in the Plex app takes longer to show here.",
"plex_watch_reconcile": "A full watch-state reconcile that rescans every library — catches what the incremental pass can't, notably an episode un-marked as watched on the Plex side. Heavier, so it runs about once a day. If it stops, un-watches done in Plex aren't mirrored back.", "plex_watch_reconcile": "A full watch-state reconcile that rescans every library — catches what the incremental pass can't, notably an episode un-marked as watched on the Plex side. Heavier, so it runs about once a day. If it stops, un-watches done in Plex aren't mirrored back.",
"audit_gc": "Prunes admin audit-log entries older than the configured retention window (Configuration → Audit log). If it stops, the audit log just keeps growing — harmless but unbounded.", "audit_gc": "Prunes admin audit-log entries and YouTube quota history older than their retention windows (Configuration → Audit log, Quota). If it stops, both just keep growing — harmless but unbounded.",
"log_gc": "Deletes stored log events older than the retention window (Configuration → Diagnostics). Runs hourly so nothing outlives the promised limit by more than an hour.", "log_gc": "Deletes stored log events older than the retention window (Configuration → Diagnostics). Runs hourly so nothing outlives the promised limit by more than an hour.",
"tmdb_enrich": "Enriches the mirrored Plex catalogue from TMDB (keywords, crew, budget/revenue, TMDb score, recommendations, artwork), matched by each title's canonical id and served from the app's own database. Optional — a no-op until a TMDB API key is set. If it stops, new Plex titles just miss their TMDB extras until it runs again.", "tmdb_enrich": "Enriches the mirrored Plex catalogue from TMDB (keywords, crew, budget/revenue, TMDb score, recommendations, artwork), matched by each title's canonical id and served from the app's own database. Optional — a no-op until a TMDB API key is set. If it stops, new Plex titles just miss their TMDB extras until it runs again.",
"people_enrich": "Makes the people in the Plex library local: fetches each person's biography from TMDB and caches their portrait, resized through Plex, the most-credited people first. A person page and a cast strip then open from the app's own data. The TMDB half does nothing without a TMDB API key, the portrait half nothing without Plex. If it stops, a person page fetches its own bio when opened, a moment later." "people_enrich": "Makes the people in the Plex library local: fetches each person's biography from TMDB and caches their portrait, resized through Plex, the most-credited people first. A person page and a cast strip then open from the app's own data. The TMDB half does nothing without a TMDB API key, the portrait half nothing without Plex. If it stops, a person page fetches its own bio when opened, a moment later."
@@ -97,7 +97,7 @@
"plex_watch_sync": "Plex watch sync (incremental)", "plex_watch_sync": "Plex watch sync (incremental)",
"plex_watch_reconcile": "Plex watch reconcile (full)", "plex_watch_reconcile": "Plex watch reconcile (full)",
"tmdb_enrich": "TMDB enrichment (Plex)", "tmdb_enrich": "TMDB enrichment (Plex)",
"audit_gc": "Audit-log cleanup", "audit_gc": "Audit-log and quota-history cleanup",
"log_gc": "Log cleanup", "log_gc": "Log cleanup",
"people_enrich": "People details and portraits (Plex)" "people_enrich": "People details and portraits (Plex)"
}, },
@@ -172,6 +172,7 @@
"countries_coded": "countries coded", "countries_coded": "countries coded",
"sessions_closed": "sessions closed", "sessions_closed": "sessions closed",
"sessions_reaped": "sessions cleaned up", "sessions_reaped": "sessions cleaned up",
"quota_reaped": "quota history cleaned up",
"recent": "recent", "recent": "recent",
"deep": "deep", "deep": "deep",
"bios": "biographies", "bios": "biographies",
+4
View File
@@ -62,6 +62,10 @@
"label": "Élő keresés forrása", "label": "Élő keresés forrása",
"hint": "Honnan jönnek az élő YouTube-keresés találatai. A „Weboldal” a YouTube saját keresővégpontját olvassa, és nem fogyaszt API-kvótát (ajánlott). A „Hivatalos API” a search.list-et használja (100 egység/oldal) — válts rá, ha a weboldalas forrás elromlik vagy letiltják." "hint": "Honnan jönnek az élő YouTube-keresés találatai. A „Weboldal” a YouTube saját keresővégpontját olvassa, és nem fogyaszt API-kvótát (ajánlott). A „Hivatalos API” a search.list-et használja (100 egység/oldal) — válts rá, ha a weboldalas forrás elromlik vagy letiltják."
}, },
"quota_retention_days": {
"label": "Kvótaelőzmények megőrzése (nap)",
"hint": "Meddig marad meg a YouTube-kvóta tételes előzménye (ki mit fogyasztott), mielőtt az előzmény-takarító feladat törli. 0 = örökre. Alapérték 365, így a használati nézet leghosszabb ablaka is teljes marad."
},
"backfill_recent_max_videos": { "backfill_recent_max_videos": {
"label": "Friss behúzás — max videó", "label": "Friss behúzás — max videó",
"hint": "Csatornánként az első körben behúzott legújabb videók száma." "hint": "Csatornánként az első körben behúzott legújabb videók száma."
+2 -1
View File
@@ -2,7 +2,8 @@
"searchPlaceholder": "Keresés a feliratkozásaid között…", "searchPlaceholder": "Keresés a feliratkozásaid között…",
"clearSearch": "Keresés törlése", "clearSearch": "Keresés törlése",
"searchYoutube": "Keresés a YouTube-on", "searchYoutube": "Keresés a YouTube-on",
"searchYoutubeTip": "Élő keresés a YouTube-on erre a kifejezésre (a közös API-kvótát fogyasztja). Entert is nyomhatsz.", "searchYoutubeTip": "Élő keresés a YouTube-on erre a kifejezésre — maga a keresés nem fogyaszt API-kvótát, a talált új videók behúzása is csak egy-két egységet. Entert is nyomhatsz.",
"searchYoutubeTipApi": "Élő keresés a YouTube-on erre a kifejezésre — a közös API-kvótát fogyasztja (oldalanként 100 egység). Entert is nyomhatsz.",
"go": "Mehet", "go": "Mehet",
"goTip": "Keresés (vagy Enter)", "goTip": "Keresés (vagy Enter)",
"channelManager": "Csatornák", "channelManager": "Csatornák",
+3 -2
View File
@@ -76,7 +76,7 @@
"plex_sync": "Tükrözi az engedélyezett Plex-könyvtárakat az app katalógusába — ez adja a Plex böngészés, keresés, szűrők és infóoldalak metaadatait (műfaj, szereplők, pontszám, borítók). Ha leáll, az újonnan hozzáadott vagy módosított Plex-címek csak a következő szinkron után jelennek meg.", "plex_sync": "Tükrözi az engedélyezett Plex-könyvtárakat az app katalógusába — ez adja a Plex böngészés, keresés, szűrők és infóoldalak metaadatait (műfaj, szereplők, pontszám, borítók). Ha leáll, az újonnan hozzáadott vagy módosított Plex-címek csak a következő szinkron után jelennek meg.",
"plex_watch_sync": "Áthozza a Plex-oldali friss megtekintés-változásokat (befejezett epizódok, folytatás-pozíciók) a Siftlode-ba, és felnyomja a Plexet el nem érő helyi változásokat. Ez az olcsó kétirányú szinkron a teljes egyeztetések között. Ha leáll, a Plex-appban jelölt megtekintés/folytatás lassabban jelenik meg itt.", "plex_watch_sync": "Áthozza a Plex-oldali friss megtekintés-változásokat (befejezett epizódok, folytatás-pozíciók) a Siftlode-ba, és felnyomja a Plexet el nem érő helyi változásokat. Ez az olcsó kétirányú szinkron a teljes egyeztetések között. Ha leáll, a Plex-appban jelölt megtekintés/folytatás lassabban jelenik meg itt.",
"plex_watch_reconcile": "A megtekintési állapot teljes egyeztetése, amely minden könyvtárat újraszken­nel — elkapja, amit az inkrementális menet nem, főleg ha egy epizódról a Plex oldalán levették a megtekintve jelet. Nehezebb, ezért kb. naponta fut. Ha leáll, a Plexben levett megtekintve jel nem tükröződik vissza.", "plex_watch_reconcile": "A megtekintési állapot teljes egyeztetése, amely minden könyvtárat újraszken­nel — elkapja, amit az inkrementális menet nem, főleg ha egy epizódról a Plex oldalán levették a megtekintve jelet. Nehezebb, ezért kb. naponta fut. Ha leáll, a Plexben levett megtekintve jel nem tükröződik vissza.",
"audit_gc": "Törli a beállított megőrzési időnél (Konfiguráció → Eseménynapló) régebbi admin-naplóbejegyzéseket. Ha leáll, az eseménynapló csak nő — ártalmatlan, de korlátlanul.", "audit_gc": "Törli a megőrzési időnél régebbi admin-naplóbejegyzéseket és YouTube-kvótaelőzményeket (Konfiguráció → Eseménynapló, Kvóta). Ha leáll, mindkettő csak nő — ártalmatlan, de korlátlanul.",
"log_gc": "Törli a megőrzési időnél (Konfiguráció → Diagnosztika) régebbi tárolt naplóeseményeket. Óránként fut, így semmi sem marad meg egy óránál tovább az ígért határidőn túl.", "log_gc": "Törli a megőrzési időnél (Konfiguráció → Diagnosztika) régebbi tárolt naplóeseményeket. Óránként fut, így semmi sem marad meg egy óránál tovább az ígért határidőn túl.",
"tmdb_enrich": "A tükrözött Plex-katalógust dúsítja a TMDB-ből (kulcsszavak, stáb, költségvetés/bevétel, TMDb-pontszám, ajánlások, artwork), a címek kanonikus azonosítója alapján párosítva, és az app saját adatbázisából szolgálja ki. Opcionális — TMDB API-kulcs nélkül nem csinál semmit. Ha leáll, az új Plex-címek csak a következő futásig maradnak TMDB-extrák nélkül.", "tmdb_enrich": "A tükrözött Plex-katalógust dúsítja a TMDB-ből (kulcsszavak, stáb, költségvetés/bevétel, TMDb-pontszám, ajánlások, artwork), a címek kanonikus azonosítója alapján párosítva, és az app saját adatbázisából szolgálja ki. Opcionális — TMDB API-kulcs nélkül nem csinál semmit. Ha leáll, az új Plex-címek csak a következő futásig maradnak TMDB-extrák nélkül.",
"people_enrich": "Helyivé teszi a Plex-könyvtár szereplőit: a TMDB-ből letölti az életrajzukat, a portréjukat pedig a Plexszel kicsinyítve elmenti, a legtöbbet szereplőkkel kezdve. A személyoldal és a szereplősáv így az app saját adataiból nyílik meg. A TMDB-rész TMDB API-kulcs nélkül nem csinál semmit, a portrérész Plex nélkül. Ha leáll, a személyoldal megnyitáskor maga kéri le az életrajzot, egy pillanattal később." "people_enrich": "Helyivé teszi a Plex-könyvtár szereplőit: a TMDB-ből letölti az életrajzukat, a portréjukat pedig a Plexszel kicsinyítve elmenti, a legtöbbet szereplőkkel kezdve. A személyoldal és a szereplősáv így az app saját adataiból nyílik meg. A TMDB-rész TMDB API-kulcs nélkül nem csinál semmit, a portrérész Plex nélkül. Ha leáll, a személyoldal megnyitáskor maga kéri le az életrajzot, egy pillanattal később."
@@ -97,7 +97,7 @@
"plex_watch_sync": "Plex megtekintés-szinkron (inkrementális)", "plex_watch_sync": "Plex megtekintés-szinkron (inkrementális)",
"plex_watch_reconcile": "Plex megtekintés-egyeztetés (teljes)", "plex_watch_reconcile": "Plex megtekintés-egyeztetés (teljes)",
"tmdb_enrich": "TMDB-dúsítás (Plex)", "tmdb_enrich": "TMDB-dúsítás (Plex)",
"audit_gc": "Eseménynapló-takarítás", "audit_gc": "Eseménynapló- és kvótaelőzmény-takarítás",
"log_gc": "Naplótakarítás", "log_gc": "Naplótakarítás",
"people_enrich": "Személyadatok és portrék (Plex)" "people_enrich": "Személyadatok és portrék (Plex)"
}, },
@@ -172,6 +172,7 @@
"countries_coded": "országkód pótolva", "countries_coded": "országkód pótolva",
"sessions_closed": "lezárt munkamenet", "sessions_closed": "lezárt munkamenet",
"sessions_reaped": "eltakarított munkamenet", "sessions_reaped": "eltakarított munkamenet",
"quota_reaped": "eltakarított kvótaelőzmény",
"recent": "friss", "recent": "friss",
"deep": "mély", "deep": "mély",
"bios": "életrajzok", "bios": "életrajzok",
+1
View File
@@ -20,6 +20,7 @@ export interface Me {
has_password: boolean; has_password: boolean;
google_enabled: boolean; google_enabled: boolean;
plex_enabled: boolean; plex_enabled: boolean;
search_source: "scrape" | "api";
can_read: boolean; can_read: boolean;
can_write: boolean; can_write: boolean;
pending_invites: number; pending_invites: number;
+13 -15
View File
@@ -105,7 +105,7 @@ function restrictedEntries(entry: Linter.RuleEntry | undefined): Set<string> {
// the lines that run the lanes. Any change fails until the snapshot is regenerated, so a narrowing // the lines that run the lanes. Any change fails until the snapshot is regenerated, so a narrowing
// arrives as a diff of `gates.snapshot.json` in review instead of slipping past a detector. (Three // arrives as a diff of `gates.snapshot.json` in review instead of slipping past a detector. (Three
// review rounds kept finding shapes the detector this replaced did not model — a `basePath` global // review rounds kept finding shapes the detector this replaced did not model — a `basePath` global
// ignore, a widened glob filed under a narrow row, a substring check on a CI line. A comparison // ignore, a widened glob filed under a narrow row, a substring check on a gate line. A comparison
// against everything has no shape to miss.) // against everything has no shape to miss.)
// //
// Regenerate: GATE_SNAPSHOT=update npx vitest run src/lib/gates.contract.test.ts // Regenerate: GATE_SNAPSHOT=update npx vitest run src/lib/gates.contract.test.ts
@@ -285,7 +285,7 @@ function meaningful(path: string): string[] {
}); });
} }
const lanes = () => ({ const lanes = () => ({
".gitea/workflows/check.yml": meaningful(".gitea/workflows/check.yml"), "scripts/check.sh": meaningful("scripts/check.sh"),
"backend/docker-compose.test.yml": meaningful("backend/docker-compose.test.yml"), "backend/docker-compose.test.yml": meaningful("backend/docker-compose.test.yml"),
"backend/Dockerfile.test": meaningful("backend/Dockerfile.test"), "backend/Dockerfile.test": meaningful("backend/Dockerfile.test"),
"frontend/package.json scripts": ( "frontend/package.json scripts": (
@@ -485,19 +485,17 @@ describe("the gate register (docs/gates.md)", () => {
}); });
// Every ruff lane passes `--config backend/ruff.toml`, which makes ruff ignore any other config // Every ruff lane passes `--config backend/ruff.toml`, which makes ruff ignore any other config
// file it would discover (measured: a nested `ruff.toml` selecting the `D` rules changed nothing). // file it would discover (measured: a nested `ruff.toml` selecting the `D` rules changed nothing).
// So that flag is what makes this file the whole configuration. The snapshot holds every CI line; // So that flag is what makes this file the whole configuration. The snapshot holds every line of
// these two are spelled out here because the reason is (round 3: a substring check let an extra // the gate script; these two are spelled out because the reason is (round 3: a substring check let
// `--config "lint.extend-ignore=…"` override through). // an extra `--config "lint.extend-ignore=…"` override through).
it("runs every CI ruff step on backend/ruff.toml and nothing else", () => { it("runs both ruff lanes on backend/ruff.toml and nothing else", () => {
const steps = read(".gitea/workflows/check.yml") const lanes = read("scripts/check.sh")
.split("\n") .split("\n")
.filter((l) => /\bruff\b/.test(l) && !l.trim().startsWith("#")) .filter((l) => /python -m ruff (check|format)/.test(l))
.map((l) => l.trim()); .map((l) => l.trim());
expect(steps).toEqual([ expect(lanes).toEqual([
"- name: ruff", "ruff_check() { ruff_pinned && python -m ruff check --config backend/ruff.toml backend scripts; }",
"run: python -m ruff check --config backend/ruff.toml backend scripts", "ruff_format() { ruff_pinned && python -m ruff format --check --config backend/ruff.toml backend scripts; }",
"- name: ruff format --check",
"run: python -m ruff format --check --config backend/ruff.toml backend scripts",
]); ]);
}); });
@@ -577,9 +575,9 @@ describe("the gate register (docs/gates.md)", () => {
expires: /\n\s*expired_at:\s*\d{4}-\d{2}-\d{2}/.test(chunk), expires: /\n\s*expired_at:\s*\d{4}-\d{2}-\d{2}/.test(chunk),
})); }));
expect(entries.filter((e) => !e.statement || !e.expires).map((e) => e.id)).toEqual([]); expect(entries.filter((e) => !e.statement || !e.expires).map((e) => e.id)).toEqual([]);
const pipIgnores = read(".gitea/workflows/check.yml") const pipIgnores = read("scripts/check.sh")
.split("\n") .split("\n")
.filter((l) => !l.trim().startsWith("#")) // the step's comment explains the flag; it is not one .filter((l) => !l.trim().startsWith("#")) // the lane's comment explains the flag; it is not one
.flatMap((l) => [...l.matchAll(/--ignore-vuln[ =](\S+)/g)].map((m) => m[1]!)); .flatMap((l) => [...l.matchAll(/--ignore-vuln[ =](\S+)/g)].map((m) => m[1]!));
const found = [...entries.map((e) => e.id), ...pipIgnores].map((id): [string, string] => [ const found = [...entries.map((e) => e.id), ...pipIgnores].map((id): [string, string] => [
"accepted", "accepted",
+72 -62
View File
@@ -1794,6 +1794,7 @@
"src/components/Feed.ytsort.dom.test.tsx", "src/components/Feed.ytsort.dom.test.tsx",
"src/components/FeedFiltersProvider.dom.test.ts", "src/components/FeedFiltersProvider.dom.test.ts",
"src/components/FeedFiltersProvider.seed.dom.test.tsx", "src/components/FeedFiltersProvider.seed.dom.test.tsx",
"src/components/Header.searchTip.dom.test.tsx",
"src/components/Home.dom.test.tsx", "src/components/Home.dom.test.tsx",
"src/components/Home.pin.dom.test.tsx", "src/components/Home.pin.dom.test.tsx",
"src/components/IssueComposer.dom.test.tsx", "src/components/IssueComposer.dom.test.tsx",
@@ -2222,68 +2223,6 @@
} }
}, },
"lanes": { "lanes": {
".gitea/workflows/check.yml": [
"name: gate",
"on:",
" push:",
" branches: [\"**\"]",
" pull_request:",
"jobs:",
" frontend:",
" runs-on: ubuntu-latest",
" container: node:24-bookworm",
" defaults:",
" run:",
" working-directory: frontend",
" steps:",
" - uses: actions/checkout@v4",
" - run: npm ci",
" - run: npm run typecheck",
" - run: npm run lint",
" - run: npm run format:check",
" - run: npm run knip",
" - run: npm run test",
" - run: npx vite build",
" - run: npm audit --omit=dev --audit-level=high",
" backend:",
" runs-on: ubuntu-latest",
" container: nikolaik/python-nodejs:python3.12-nodejs24",
" services:",
" postgres:",
" image: postgres:16-alpine",
" env:",
" POSTGRES_USER: siftlode",
" POSTGRES_PASSWORD: siftlode",
" POSTGRES_DB: siftlode_test",
" options: >-",
" --health-cmd \"pg_isready -U siftlode -d siftlode_test\"",
" --health-interval 2s",
" --health-timeout 3s",
" --health-retries 30",
" env:",
" DATABASE_URL: postgresql+psycopg://siftlode:siftlode@postgres:5432/siftlode_test",
" SIFTLODE_TEST_REQUIRE_DB: \"1\"",
" steps:",
" - uses: actions/checkout@v4",
" - run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client",
" - name: Wait for Postgres",
" run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done",
" - run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt",
" - name: ruff",
" run: python -m ruff check --config backend/ruff.toml backend scripts",
" - name: ruff format --check",
" run: python -m ruff format --check --config backend/ruff.toml backend scripts",
" - name: basedpyright",
" run: python -m basedpyright --project pyrightconfig.json",
" working-directory: backend",
" - name: basedpyright baseline is committed",
" run: git diff --exit-code -- backend/.basedpyright/baseline.json",
" - name: pytest",
" run: python -m pytest",
" working-directory: backend",
" - name: pip-audit",
" run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock"
],
"backend/Dockerfile.test": [ "backend/Dockerfile.test": [
"FROM python:3.12-slim", "FROM python:3.12-slim",
"WORKDIR /app", "WORKDIR /app",
@@ -2366,6 +2305,77 @@
" },", " },",
" },", " },",
"});" "});"
],
"scripts/check.sh": [
"set -u",
"cd \"$(dirname \"$0\")/..\" || exit 2",
"mode=\"${1:-all}\"",
"case \"$mode\" in all | lint) ;; *) echo \"usage: scripts/check.sh [lint]\" >&2; exit 2 ;; esac",
"export MSYS_NO_PATHCONV=1 # Git Bash: do not rewrite the in-container paths below",
"TESTCOMPOSE=backend/docker-compose.test.yml",
"failed=()",
"lane() { # lane <name> <command…> — run it, remember a failure, never stop",
" local name=\"$1\"",
" shift",
" echo \">> $name…\"",
" \"$@\" || failed+=(\"$name\")",
"}",
"ruff_pinned() {",
" local pin host",
" pin=\"$(sed -n 's/^ruff==\\([0-9.]*\\).*/\\1/p' backend/requirements-dev.txt)\"",
" host=\"$(python -m ruff --version 2>/dev/null | awk '{print $2}')\"",
" [ -n \"$pin\" ] && [ \"$host\" = \"$pin\" ] && return 0",
" echo \"ruff ${host:-missing} on the host, ${pin:-nothing} pinned — python -m pip install ruff==$pin\" >&2",
" return 1",
"}",
"ruff_check() { ruff_pinned && python -m ruff check --config backend/ruff.toml backend scripts; }",
"ruff_format() { ruff_pinned && python -m ruff format --check --config backend/ruff.toml backend scripts; }",
"basedpyright() {",
" local before",
" before=\"$(git hash-object backend/.basedpyright/baseline.json)\"",
" docker compose -f \"$TESTCOMPOSE\" run --rm --no-deps --build test \\",
" python -m basedpyright --project pyrightconfig.json || return 1",
" if [ \"$(git hash-object backend/.basedpyright/baseline.json)\" != \"$before\" ]; then",
" echo \"this run rewrote backend/.basedpyright/baseline.json (errors were fixed) — commit it\" >&2",
" return 1",
" fi",
" git diff HEAD --quiet -- backend/.basedpyright/baseline.json && return 0",
" echo \"backend/.basedpyright/baseline.json is uncommitted — commit it\" >&2",
" return 1",
"}",
"frontend() { (cd frontend && \"$@\"); }",
"lane \"ruff\" ruff_check",
"lane \"ruff format --check\" ruff_format",
"lane \"basedpyright (new errors only)\" basedpyright",
"lane \"lock check (frontend/node_modules == package-lock.json)\" node scripts/check-node-modules.mjs",
"lane \"tsc (src + configs + e2e)\" frontend npm run --silent typecheck",
"lane \"eslint (warnings fail too)\" frontend npm run --silent lint",
"lane \"prettier --check\" frontend npm run --silent format:check",
"lane \"knip (unused, and unused by what ships)\" frontend npm run --silent knip",
"pytest_lane() { # the full suite against a throwaway Postgres; --pull keeps the base image current",
" docker compose -f \"$TESTCOMPOSE\" build --pull -q test || return 1",
" local rc=0",
" docker compose -f \"$TESTCOMPOSE\" up --build --abort-on-container-exit --exit-code-from test || rc=1",
" docker compose -f \"$TESTCOMPOSE\" down -v >/dev/null 2>&1",
" return $rc",
"}",
"pip_audit() {",
" docker compose -f \"$TESTCOMPOSE\" run --rm --no-deps test \\",
" pip-audit -r requirements.lock --no-deps --disable-pip",
"}",
"if [ \"$mode\" = all ]; then",
" lane \"vitest (+ coverage report)\" frontend npm run --silent test -- --coverage",
" lane \"vite build\" frontend npx vite build --logLevel warn",
" lane \"pytest (pure-logic + DB lane)\" pytest_lane",
" lane \"pip-audit (backend lock)\" pip_audit",
" lane \"npm audit (production dependencies)\" frontend npm audit --omit=dev --audit-level=high",
"fi",
"if [ ${#failed[@]} -eq 0 ]; then",
" echo \">> gate ($mode): green ✓\"",
" exit 0",
"fi",
"echo \">> gate ($mode): RED — ${failed[*]}\" >&2",
"exit 1"
] ]
} }
} }
@@ -48,18 +48,6 @@ it("builds the shipped bundle on the pinned engine", () => {
).toBe(pinned); ).toBe(pinned);
}); });
it("runs the server gate on the pinned engine", () => {
// The frontend job only — the backend job's node runs the checkout action, never our code, so it is
// deliberately not part of this contract (see the note in check.yml).
expect(
majorIn(
readRoot(".gitea/workflows/check.yml"),
/^\s*container: node:(\d+)-bookworm$/m,
"check.yml",
),
).toBe(pinned);
});
it("declares the pinned engine to npm and to tsc", () => { it("declares the pinned engine to npm and to tsc", () => {
expect(pkg.engines?.node).toBe(`${pinned}.x`); expect(pkg.engines?.node).toBe(`${pinned}.x`);
// Types a major ahead of the runtime make tsc green over APIs the shipped engine does not have — // Types a major ahead of the runtime make tsc green over APIs the shipped engine does not have —
+2 -14
View File
@@ -11,12 +11,12 @@ import { expect, it } from "vitest";
// //
// It lives in the FRONTEND suite for one boring reason: this is the only runner that can see the // It lives in the FRONTEND suite for one boring reason: this is the only runner that can see the
// whole repository. The backend suite runs inside the test image, which mounts `backend/` alone, so // whole repository. The backend suite runs inside the test image, which mounts `backend/` alone, so
// it cannot read the repo-root Dockerfile or the workflow. It holds the other half of the chain — // it cannot read the repo-root Dockerfile. It holds the other half of the chain —
// including "the interpreter running these tests IS the pinned one", which only it can assert: // including "the interpreter running these tests IS the pinned one", which only it can assert:
// //
// backend/tests/test_python_engine_contract.py : running interpreter == pyrightconfig == ruff.toml // backend/tests/test_python_engine_contract.py : running interpreter == pyrightconfig == ruff.toml
// here : .python-version == pyrightconfig == ruff.toml // here : .python-version == pyrightconfig == ruff.toml
// == Dockerfile == Dockerfile.test == check.yml // == Dockerfile == Dockerfile.test
// //
// To move the version: edit `.python-version`, then follow the failures in both files. // To move the version: edit `.python-version`, then follow the failures in both files.
const readRoot = (rel: string) => readFileSync(new URL(`../../../${rel}`, import.meta.url), "utf8"); const readRoot = (rel: string) => readFileSync(new URL(`../../../${rel}`, import.meta.url), "utf8");
@@ -55,18 +55,6 @@ it("tests on the pinned version", () => {
).toBe(pinned); ).toBe(pinned);
}); });
it("runs the server gate on the pinned version", () => {
// The backend job's container. Its NODE is deliberately outside the Node contract (it only runs
// the checkout action), but its PYTHON runs our code and our tests, so it is inside this one.
expect(
versionIn(
readRoot(".gitea/workflows/check.yml"),
/^\s*container: nikolaik\/python-nodejs:python(\d+)\.(\d+)-nodejs\d+$/m,
"check.yml",
),
).toBe(pinned);
});
it("type-checks against the pinned version", () => { it("type-checks against the pinned version", () => {
// A type checker aimed at another version green-lights APIs the shipped interpreter does not have // A type checker aimed at another version green-lights APIs the shipped interpreter does not have
// — the same defect as the image split, moved to check time. // — the same defect as the image split, moved to check time.
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env bash
# The gate — every lane, in one place, run from the repo root (E32, after CI went: the lab runner
# duplicated what this already ran before every release, and one CI run starved the git host of
# memory until it stopped answering). The release dispatcher calls this; so can anyone with Docker
# and the pinned toolchain. `frontend/src/lib/gates.contract.test.ts` holds its lines, so a lane
# dropped or narrowed here fails a test like any other gate change.
#
# bash scripts/check.sh every lane (what a release requires, before e2e and the CVE scan)
# bash scripts/check.sh lint the static lanes only: no tests, no build, no audits
#
# Each lane runs even when an earlier one failed, and the summary names every red one: a gate that
# stops at the first failure hides the second until the first is fixed. The backend lanes run in the
# test image (backend/Dockerfile.test): requirements.lock will not install on Windows (uvloop), and a
# type checker or auditor without the real packages reports nonsense.
set -u
cd "$(dirname "$0")/.." || exit 2
mode="${1:-all}"
case "$mode" in all | lint) ;; *) echo "usage: scripts/check.sh [lint]" >&2; exit 2 ;; esac
export MSYS_NO_PATHCONV=1 # Git Bash: do not rewrite the in-container paths below
TESTCOMPOSE=backend/docker-compose.test.yml
failed=()
lane() { # lane <name> <command…> — run it, remember a failure, never stop
local name="$1"
shift
echo ">> $name…"
"$@" || failed+=("$name")
}
# --- static lanes ------------------------------------------------------------------------------------
# Host ruff, held to the pin in requirements-dev.txt: a newer ruff silently changes what "formatted"
# means. `--config backend/ruff.toml` on both: it makes that file the WHOLE configuration (ruff then
# ignores every other config it would discover) and gives scripts/ the backend's rule set.
ruff_pinned() {
local pin host
pin="$(sed -n 's/^ruff==\([0-9.]*\).*/\1/p' backend/requirements-dev.txt)"
host="$(python -m ruff --version 2>/dev/null | awk '{print $2}')"
[ -n "$pin" ] && [ "$host" = "$pin" ] && return 0
echo "ruff ${host:-missing} on the host, ${pin:-nothing} pinned — python -m pip install ruff==$pin" >&2
return 1
}
ruff_check() { ruff_pinned && python -m ruff check --config backend/ruff.toml backend scripts; }
ruff_format() { ruff_pinned && python -m ruff format --check --config backend/ruff.toml backend scripts; }
# New errors only: the pre-existing ones are in backend/.basedpyright/baseline.json. basedpyright
# TIGHTENS that file by itself when errors go, and the tree is bind-mounted read-write, so a green run
# can leave it rewritten: that fails here until it is committed, or the next run's stack reads stale.
basedpyright() {
local before
before="$(git hash-object backend/.basedpyright/baseline.json)"
docker compose -f "$TESTCOMPOSE" run --rm --no-deps --build test \
python -m basedpyright --project pyrightconfig.json || return 1
if [ "$(git hash-object backend/.basedpyright/baseline.json)" != "$before" ]; then
echo "this run rewrote backend/.basedpyright/baseline.json (errors were fixed) — commit it" >&2
return 1
fi
git diff HEAD --quiet -- backend/.basedpyright/baseline.json && return 0
echo "backend/.basedpyright/baseline.json is uncommitted — commit it" >&2
return 1
}
frontend() { (cd frontend && "$@"); }
lane "ruff" ruff_check
lane "ruff format --check" ruff_format
lane "basedpyright (new errors only)" basedpyright
# every frontend lane judges the HOST node_modules: this proves it is the tree the lock pins
lane "lock check (frontend/node_modules == package-lock.json)" node scripts/check-node-modules.mjs
lane "tsc (src + configs + e2e)" frontend npm run --silent typecheck
lane "eslint (warnings fail too)" frontend npm run --silent lint
lane "prettier --check" frontend npm run --silent format:check
lane "knip (unused, and unused by what ships)" frontend npm run --silent knip
# --- tests, build, audits --------------------------------------------------------------------------
pytest_lane() { # the full suite against a throwaway Postgres; --pull keeps the base image current
docker compose -f "$TESTCOMPOSE" build --pull -q test || return 1
local rc=0
docker compose -f "$TESTCOMPOSE" up --build --abort-on-container-exit --exit-code-from test || rc=1
docker compose -f "$TESTCOMPOSE" down -v >/dev/null 2>&1
return $rc
}
# Every pinned backend package against the advisory databases, at any severity — the lock is a full
# `pip freeze`, so it is audited as written, without a resolver. An advisory with no fix is answered
# with `--ignore-vuln <id>` here and a row in docs/gates.md (the register test reads this line).
pip_audit() {
docker compose -f "$TESTCOMPOSE" run --rm --no-deps test \
pip-audit -r requirements.lock --no-deps --disable-pip
}
if [ "$mode" = all ]; then
# --coverage prints the summary: read, never a threshold (E32 D4)
lane "vitest (+ coverage report)" frontend npm run --silent test -- --coverage
# the production bundle, and devOnlyGuard, which fails it when dev-only code reaches it
lane "vite build" frontend npx vite build --logLevel warn
lane "pytest (pure-logic + DB lane)" pytest_lane
lane "pip-audit (backend lock)" pip_audit
# what ships to the browser, at high or above; the dev toolchain is burned down by hand
lane "npm audit (production dependencies)" frontend npm audit --omit=dev --audit-level=high
fi
if [ ${#failed[@]} -eq 0 ]; then
echo ">> gate ($mode): green ✓"
exit 0
fi
echo ">> gate ($mode): RED — ${failed[*]}" >&2
exit 1