Compare commits
9
Commits
0528b35f2d
...
v0.99.2
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0dfb4a984b | ||
|
|
5bcd1d8d8a | ||
|
|
f49789665e | ||
|
|
0e077b84c8 | ||
|
|
ce8ccfc3c0 | ||
|
|
33dec56e45 | ||
|
|
2a699e26ba | ||
|
|
13c8c05661 | ||
|
|
cc414de952 |
@@ -1,128 +0,0 @@
|
||||
# Server-side mirror of the local `siftlode check` gate (E1 S3 / DX-1), so `main` is always green
|
||||
# independent of any one machine. Two jobs match the two local lanes:
|
||||
# frontend — tsc (3 projects) + eslint + prettier --check + knip (both modes) + vitest + vite build
|
||||
# + npm audit of the production dependencies
|
||||
# backend — ruff + basedpyright + the full pytest suite (pure-logic + the DB-backed lane) against
|
||||
# a real Postgres + pip-audit of the lock
|
||||
#
|
||||
# They are declared side by side but they DO NOT run in parallel: the lab has a single act_runner
|
||||
# with one slot, so they queue. Measured across runs #86-#91 — the backend job starts at the exact
|
||||
# second the frontend one ends, every time, and the pair costs ~4 + ~6 = **~10 minutes from the
|
||||
# push**. Anything that waits on this gate has to budget for the sum, not the max; the ops
|
||||
# `release` read-back timed out on its first real run for believing otherwise.
|
||||
#
|
||||
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
|
||||
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
|
||||
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
|
||||
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
|
||||
# failed to come up is a hard error, never a silent skip.
|
||||
#
|
||||
# The frontend container's node major is NOT free: it must equal the prod build image's and the dev
|
||||
# machine's, because vitest/tsc read the engine's own Intl and web-API implementations and those split
|
||||
# between majors. `.nvmrc` is the single source of truth and
|
||||
# `frontend/src/lib/nodeEngine.contract.test.ts` fails this gate if any of them drift apart.
|
||||
name: gate
|
||||
|
||||
# Branch pushes only. A release pushes `main` and then the `v<ver>` tag, and a bare `on: [push]`
|
||||
# treated the tag as a second push of the IDENTICAL commit — every release ran this whole gate
|
||||
# twice, to the same verdict, for nothing (runs #81-#90 are five such pairs). `branches: ['**']`
|
||||
# keeps every branch covered and drops tag refs.
|
||||
#
|
||||
# It has to be `branches`, NOT the `tags-ignore` this obviously wants to be. Gitea's matchPushEvent
|
||||
# requires EVERY named filter to match, and each one bails on the wrong ref kind: `tags-ignore`
|
||||
# starts with `if !refName.IsTag() { break }`, so a filter meant to skip tags rejects every BRANCH
|
||||
# push and the gate stops running entirely. `branches` bails the same way on a tag, which is the
|
||||
# direction we want. (Measured in go-gitea v1.27.3 modules/actions/workflows.go, not assumed — it
|
||||
# differs from GitHub, where `tags-ignore` alone leaves branches unfiltered.)
|
||||
#
|
||||
# If that filter were ever wrong the gate would not run at all — which is why `siftlode release`
|
||||
# now ends by reading the verdict back for the sha it pushed, and says so out loud when there is
|
||||
# no run for it. A gate that silently stops running is the failure this repo already lived through.
|
||||
on:
|
||||
push:
|
||||
branches: ["**"]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
frontend:
|
||||
runs-on: ubuntu-latest
|
||||
container: node:24-bookworm
|
||||
defaults:
|
||||
run:
|
||||
working-directory: frontend
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: npm ci
|
||||
- run: npm run typecheck
|
||||
- run: npm run lint
|
||||
- run: npm run format:check
|
||||
- run: npm run knip
|
||||
- run: npm run test
|
||||
# The production bundle (E32): before this, `vite build` — and devOnlyGuard, which fails it if
|
||||
# dev-only code reaches the bundle — first ran inside the image build, after main was pushed.
|
||||
- run: npx vite build
|
||||
# What ships to the browser: a known CVE at high or above in a production dependency fails.
|
||||
# (The dev tooling's own advisories are burned down by hand; see docs/gates.md.)
|
||||
- run: npm audit --omit=dev --audit-level=high
|
||||
|
||||
backend:
|
||||
runs-on: ubuntu-latest
|
||||
# A combined python+node image: python 3.12 (matches the prod Dockerfile) for the tests, and node
|
||||
# so the JS actions (checkout) run. actions/setup-python is NOT usable here — the runner can't reach
|
||||
# GitHub's python-versions download, so it must be baked into the image.
|
||||
#
|
||||
# This node runs the checkout action, never our code, so it is deliberately OUTSIDE the engine
|
||||
# contract the frontend job is held to — it tracks `.nvmrc` as hygiene (node 20 went EOL in April
|
||||
# 2026), but a missing upstream tag here must never be able to block the gate.
|
||||
container: nikolaik/python-nodejs:python3.12-nodejs24
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
env:
|
||||
POSTGRES_USER: siftlode
|
||||
POSTGRES_PASSWORD: siftlode
|
||||
POSTGRES_DB: siftlode_test
|
||||
options: >-
|
||||
--health-cmd "pg_isready -U siftlode -d siftlode_test"
|
||||
--health-interval 2s
|
||||
--health-timeout 3s
|
||||
--health-retries 30
|
||||
env:
|
||||
DATABASE_URL: postgresql+psycopg://siftlode:siftlode@postgres:5432/siftlode_test
|
||||
SIFTLODE_TEST_REQUIRE_DB: "1"
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
# ffmpeg for the poster/thumbnail tests; postgresql-client for the readiness wait below.
|
||||
- run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client
|
||||
- name: Wait for Postgres
|
||||
run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done
|
||||
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
|
||||
- name: ruff
|
||||
run: python -m ruff check --config backend/ruff.toml backend scripts
|
||||
- name: ruff format --check
|
||||
run: python -m ruff format --check --config backend/ruff.toml backend scripts
|
||||
# The ninth lane (E27 S3). Run from backend/ so it resolves exactly as the local lane does —
|
||||
# pyrightconfig.json's include/extraPaths are relative to the config, but the baseline is
|
||||
# found relative to the working directory. New errors fail; the ~200 pre-existing ones are in
|
||||
# .basedpyright/baseline.json and burn down epic by epic. Matching is per (file, rule, column
|
||||
# span) plus a count, so a new error of the SAME shape in a file where a sibling was just
|
||||
# fixed is absorbed — burn the hot files down first.
|
||||
#
|
||||
# The second step is not redundant. basedpyright TIGHTENS the baseline in place when the count
|
||||
# drops, with no --writebaseline, and CI throws its workspace away — so without this a
|
||||
# burn-down is green here and red on the developer's `siftlode check`, which has the same
|
||||
# guard. Two gates over one tree must not answer differently.
|
||||
- name: basedpyright
|
||||
run: python -m basedpyright --project pyrightconfig.json
|
||||
working-directory: backend
|
||||
- name: basedpyright baseline is committed
|
||||
run: git diff --exit-code -- backend/.basedpyright/baseline.json
|
||||
- name: pytest
|
||||
run: python -m pytest
|
||||
working-directory: backend
|
||||
# Dependency CVE audit over the pinned lock. BLOCKING since E32: it ran advisory-only, and when
|
||||
# it was finally read it had three urllib3 advisories with a fix available (bumped in the same
|
||||
# change). An advisory without a fix is answered with `--ignore-vuln <id>` here, a reason and a
|
||||
# date in the comment, and a line in docs/gates.md.
|
||||
- name: pip-audit
|
||||
run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock
|
||||
@@ -7,6 +7,20 @@ edit the release notes and re-run the generator.
|
||||
|
||||
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
|
||||
|
||||
## 0.99.2 — 2026-10-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- Settings: the download worker's concurrency and the log level now take effect within seconds — before, the worker ignored both until it was rebuilt.
|
||||
- The YouTube quota history no longer grows forever: the daily cleanup prunes it after a year (Configuration → Quota → Quota history kept).
|
||||
- The live-search button no longer says a search spends the API quota: with the default website source the search itself is free. It names the cost when the official API is selected.
|
||||
|
||||
## 0.99.1 — 2026-10-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- Behind the scenes: every release is now checked by one script on the build machine, including the security audit of every dependency.
|
||||
|
||||
## 0.99.0 — 2026-10-04
|
||||
|
||||
### New
|
||||
|
||||
@@ -13,14 +13,14 @@ router-free navigation provider.
|
||||
## The gate — never merge or ship red
|
||||
|
||||
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
|
||||
eslint + prettier --check + knip (both modes) + vitest + vite build + pytest**, then a coverage report (no threshold). It runs
|
||||
eslint + prettier --check + knip (both modes) + vitest + vite build + pytest + pip-audit + npm audit**, then a coverage report (no threshold). It runs
|
||||
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
|
||||
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
|
||||
green **before** any merge, release, or handing work to review.
|
||||
|
||||
- Full gate (agents with the private ops dispatcher): `siftlode check`.
|
||||
- Directly, without the dispatcher: the commands in [`docs/development.md`](docs/development.md) § The gate (frontend lanes, `ruff` over `backend scripts`, basedpyright in the test image, and `pytest` — the DB-lane needs Postgres).
|
||||
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every file and the CI/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: a new code file or any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
|
||||
- Full gate: `bash scripts/check.sh` (`lint` for the static lanes only); the private ops dispatcher's `siftlode check` runs the same script. Single lanes: [`docs/development.md`](docs/development.md) § The gate.
|
||||
- **There is no CI** (owner, 2026-10-04): this gate, run here, is the only one — green before every hand-over, and `release` refuses to merge without it.
|
||||
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every file and the gate/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: a new code file or any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
|
||||
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` and burn down epic by epic. Fixed some? Regenerate with `--writebaseline`.
|
||||
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
|
||||
|
||||
|
||||
+3
-3
@@ -11,7 +11,7 @@ ARG BUILD_DATE=
|
||||
|
||||
# Stage 1: build the frontend SPA
|
||||
# The major is pinned to `.nvmrc` and held there by frontend/src/lib/nodeEngine.contract.test.ts:
|
||||
# this image, the CI gate's image and the dev machine must be ONE engine (see that test for why).
|
||||
# this image and the dev machine the gate runs on must be ONE engine (see that test for why).
|
||||
FROM node:24-alpine AS frontend
|
||||
WORKDIR /fe
|
||||
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
|
||||
@@ -79,8 +79,8 @@ RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nCo
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
|
||||
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
|
||||
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
|
||||
# byte-reproducible and the exact versions are the ones the gate's pip-audit scanned — not whatever
|
||||
# floats up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
|
||||
# pip itself is removed once it has installed the lock (E32): nothing runs it at runtime, and it carries
|
||||
# its own VENDORED copies of urllib3, msgpack and setuptools, which Trivy reports as the image's own —
|
||||
# an attack surface and a CVE stream for a tool the running app never calls.
|
||||
|
||||
@@ -260,6 +260,9 @@ class Settings(BaseSettings):
|
||||
# Admin audit log: how long to keep rows (days; 0 = keep forever) + how often the GC runs.
|
||||
audit_retention_days: int = 180
|
||||
audit_gc_minutes: int = 1440
|
||||
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
|
||||
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
|
||||
quota_retention_days: int = 365
|
||||
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
|
||||
download_layout: DownloadLayout = "plex"
|
||||
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
|
||||
|
||||
+18
-2
@@ -7,11 +7,13 @@ always be allowed; expensive backfill must yield to the remaining budget.
|
||||
|
||||
import contextlib
|
||||
import contextvars
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, cast
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import delete, func, select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.engine import CursorResult
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import sysconfig
|
||||
@@ -96,6 +98,20 @@ def pacific_today():
|
||||
return datetime.now(_PACIFIC).date()
|
||||
|
||||
|
||||
def gc(db: Session, retention_days: int) -> int:
|
||||
"""Delete quota-history rows older than `retention_days` (<= 0 keeps them forever) and return how
|
||||
many went. Flushes only; the cleanup job commits. Today's rows are never in reach: the per-user
|
||||
search cap counts them, and the floor of two days outlasts any Pacific day (25 h at most)."""
|
||||
if retention_days <= 0:
|
||||
return 0
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=max(2, retention_days))
|
||||
# A DELETE answers with a CursorResult; `execute` is typed as the general Result, without rowcount.
|
||||
result = cast(
|
||||
CursorResult[Any], db.execute(delete(QuotaEvent).where(QuotaEvent.created_at < cutoff))
|
||||
)
|
||||
return result.rowcount or 0
|
||||
|
||||
|
||||
def pacific_day_start_utc() -> datetime:
|
||||
"""UTC instant of the current Pacific day's midnight — aligns 'today' with the budget reset."""
|
||||
start = datetime.now(_PACIFIC).replace(hour=0, minute=0, second=0, microsecond=0)
|
||||
|
||||
@@ -148,6 +148,9 @@ def get_me(
|
||||
"google_enabled": google_enabled(db),
|
||||
# Whether the optional Plex module is enabled (UI shows the Plex feed source when so).
|
||||
"plex_enabled": sysconfig.get_bool(db, "plex_enabled"),
|
||||
# Which live-search source runs, so the search button can say what a search costs: "scrape"
|
||||
# spends no API quota, "api" spends 100 units a page.
|
||||
"search_source": sysconfig.get_str(db, "search_source"),
|
||||
"can_read": has_read_scope(user),
|
||||
"can_write": has_write_scope(user),
|
||||
"pending_invites": pending_invites,
|
||||
|
||||
@@ -425,12 +425,15 @@ def _people_enrich_job() -> None:
|
||||
|
||||
|
||||
def _audit_gc_job() -> None:
|
||||
# Prune audit-log rows older than the admin-set retention (audit_retention_days; 0 = keep
|
||||
# forever). Pure DB, no quota.
|
||||
# The history cleanup: audit-log rows past audit_retention_days and YouTube quota history past
|
||||
# quota_retention_days (0 = keep forever, each). Pure DB, no quota.
|
||||
def work(db):
|
||||
from app import sysconfig
|
||||
|
||||
return {"reaped": audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))}
|
||||
reaped = audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))
|
||||
quota_reaped = quota.gc(db, sysconfig.get_int(db, "quota_retention_days"))
|
||||
db.commit()
|
||||
return {"reaped": reaped, "quota_reaped": quota_reaped}
|
||||
|
||||
_job("audit_gc", work)
|
||||
|
||||
|
||||
@@ -80,6 +80,7 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
),
|
||||
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
|
||||
ConfigSpec("search_source", "choice", "quota", "search_source"),
|
||||
ConfigSpec("quota_retention_days", "int", "quota", "quota_retention_days", min=0, max=3650),
|
||||
# --- Backfill (recent-first first pass per channel) ---
|
||||
ConfigSpec(
|
||||
"backfill_recent_max_videos",
|
||||
|
||||
+59
-10
@@ -5,7 +5,8 @@ from the API so long yt-dlp downloads + ffmpeg postprocessing never block web re
|
||||
|
||||
Design:
|
||||
* N loop threads (download_worker_concurrency) each claim a `queued` DownloadJob with
|
||||
FOR UPDATE SKIP LOCKED and process it.
|
||||
FOR UPDATE SKIP LOCKED and process it. N and the log level follow the settings page while the
|
||||
worker runs (`_sync_knobs`), not only the env it booted with.
|
||||
* The shared MediaAsset is the dedup unit. A job whose asset is already `ready` finishes
|
||||
instantly (no re-download); a fresh asset is claimed pending→downloading by exactly one
|
||||
worker (a DB compare-and-set), so two jobs for the same video+format never double-download.
|
||||
@@ -48,12 +49,60 @@ from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
_stop = threading.Event()
|
||||
# How many loop threads may claim jobs. A thread whose index is at or above it stops after its
|
||||
# current job, so lowering the knob never abandons a download halfway.
|
||||
_target = 1
|
||||
# How often the worker re-reads the knobs it obeys; the settings page says nothing about a restart.
|
||||
_KNOB_SECONDS = 10.0
|
||||
|
||||
|
||||
def _handle_signal(signum, frame):
|
||||
_stop.set()
|
||||
|
||||
|
||||
def _read_knobs() -> tuple[int, str] | None:
|
||||
"""The stored concurrency and log level, or None when the database cannot answer — the caller
|
||||
then keeps what already runs."""
|
||||
try:
|
||||
with SessionLocal() as db:
|
||||
return (
|
||||
max(1, sysconfig.get_int(db, "download_worker_concurrency")),
|
||||
sysconfig.get_str(db, "log_level"),
|
||||
)
|
||||
except Exception: # noqa: BLE001 — any DB or config failure keeps the knobs already in force
|
||||
log.warning(
|
||||
"could not read the worker's settings; keeping the ones in force",
|
||||
exc_info=True,
|
||||
extra=logs.event("download.worker.knobs_unreadable"),
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _sync_knobs(threads: dict[int, threading.Thread], level: str | None) -> str | None:
|
||||
"""Apply the stored knobs: run loop threads up to the concurrency (surplus ones retire after
|
||||
their job, see `_target`) and re-apply the log level when it changed. Returns the level in force."""
|
||||
global _target
|
||||
knobs = _read_knobs()
|
||||
if knobs is not None:
|
||||
target, stored_level = knobs
|
||||
if target != _target:
|
||||
log.info(
|
||||
"download concurrency is now %d",
|
||||
target,
|
||||
extra=logs.event("download.worker.concurrency", before=_target, after=target),
|
||||
)
|
||||
_target = target
|
||||
if stored_level != level:
|
||||
logs.set_level(stored_level)
|
||||
level = stored_level
|
||||
for i in range(_target):
|
||||
t = threads.get(i)
|
||||
if t is None or not t.is_alive():
|
||||
threads[i] = threading.Thread(target=_worker_loop, args=(i,), daemon=True)
|
||||
threads[i].start()
|
||||
return level
|
||||
|
||||
|
||||
# --- claim + recovery -----------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -576,7 +625,7 @@ def _finish_from_cache(job_id: int, asset_id: int) -> None:
|
||||
|
||||
|
||||
def _worker_loop(idx: int) -> None:
|
||||
while not _stop.is_set():
|
||||
while not _stop.is_set() and idx < _target:
|
||||
try:
|
||||
job_id = _claim_job()
|
||||
except Exception:
|
||||
@@ -634,15 +683,15 @@ def main() -> None:
|
||||
# has migrated — those few seconds of batches fail and say so on stderr; nothing else breaks.
|
||||
logsink.start()
|
||||
_recover_orphans()
|
||||
n = max(1, settings.download_worker_concurrency)
|
||||
log.info("Download worker started (concurrency=%d, root=%s).", n, settings.download_root)
|
||||
threads = [threading.Thread(target=_worker_loop, args=(i,), daemon=True) for i in range(n)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
while not _stop.is_set():
|
||||
_stop.wait(1.0)
|
||||
global _target
|
||||
_target = max(1, settings.download_worker_concurrency) # used only if the DB can't answer
|
||||
threads: dict[int, threading.Thread] = {}
|
||||
level = _sync_knobs(threads, None)
|
||||
log.info("Download worker started (concurrency=%d, root=%s).", _target, settings.download_root)
|
||||
while not _stop.wait(_KNOB_SECONDS):
|
||||
level = _sync_knobs(threads, level)
|
||||
log.info("Download worker stopping…")
|
||||
for t in threads:
|
||||
for t in threads.values():
|
||||
t.join(timeout=5)
|
||||
|
||||
|
||||
|
||||
@@ -9,8 +9,10 @@
|
||||
# writes is keyed to the diagnostics THIS version produces.
|
||||
# pytest-cov: the coverage REPORT the gate prints (E32 D4) — a number to read, deliberately with no
|
||||
# threshold; "every regression test seen red" is the bar, not a percentage.
|
||||
# pip-audit: the gate's audit of requirements.lock (scripts/check.sh), blocking at any severity.
|
||||
basedpyright==1.40.1
|
||||
pytest==8.4.2
|
||||
pytest-cov==7.1.0
|
||||
ruff==0.15.21
|
||||
httpx2==2.13.0
|
||||
pip-audit==2.10.1
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Pinned dependency lock for reproducible builds (SEC-5). GENERATED — do not hand-edit.
|
||||
# Source of truth is requirements.txt (the floor ranges); regenerate after editing it with:
|
||||
# docker run --rm -v "$PWD":/b python:3.12-slim sh -c 'pip install -q -r /b/requirements.txt && pip freeze --exclude pip --exclude setuptools --exclude wheel' | sort
|
||||
# The prod Dockerfile + Dockerfile.test + CI install THIS file; pip-audit scans it in CI (blocking
|
||||
# since E32). A security fix may be ONE pin bumped in place instead of a full regeneration, which
|
||||
# would move every other package too — then `pip install -r requirements.lock && pip check` must
|
||||
# The prod Dockerfile + Dockerfile.test install THIS file; the gate's pip-audit (scripts/check.sh)
|
||||
# scans it, blocking since E32. A security fix may be ONE pin bumped in place instead of a full
|
||||
# regeneration, which would move every other package too — then `pip install -r requirements.lock && pip check` must
|
||||
# pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177, E32).
|
||||
APScheduler==3.11.3
|
||||
Authlib==1.7.2
|
||||
|
||||
@@ -39,7 +39,7 @@ _MISSING = object()
|
||||
|
||||
# A would-be silent skip (no DB, or no ffmpeg) is a hard error unless the run says otherwise — the
|
||||
# suite must not print green while the DB-backed / ffmpeg tests were quietly not run. ON by default
|
||||
# since E32: as an opt-in only the compose lane and CI set, a hand-run `pytest` skipped ~60 DB tests
|
||||
# since E32: as an opt-in only the compose lane set, a hand-run `pytest` skipped ~60 DB tests
|
||||
# and still passed. `SIFTLODE_TEST_REQUIRE_DB=0` is the explicit "pure-logic only" run.
|
||||
_REQUIRE_FULL = os.getenv("SIFTLODE_TEST_REQUIRE_DB", "1") != "0"
|
||||
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
"""`/api/me` names the live-search source, so the search button's tooltip can say what a search costs
|
||||
under the source that actually runs (the default website source spends no API quota)."""
|
||||
|
||||
from app import sysconfig
|
||||
|
||||
_PW = "correct horse battery"
|
||||
|
||||
|
||||
def _me(client):
|
||||
r = client.get("/api/me")
|
||||
assert r.status_code == 200 and r.json()["authenticated"] is True
|
||||
return r.json()
|
||||
|
||||
|
||||
def test_me_names_the_running_search_source(client, db, make_user):
|
||||
# /api/me resolves its user through optional_current_user, which the as_user override does not
|
||||
# cover — so sign in for real (see test_me_email_flag).
|
||||
user = make_user(password=_PW)
|
||||
login = client.post("/auth/password-login", json={"email": user.email, "password": _PW})
|
||||
assert login.status_code == 200
|
||||
|
||||
assert _me(client)["search_source"] == "scrape" # the default
|
||||
|
||||
sysconfig.set_value(db, "search_source", "api")
|
||||
db.commit()
|
||||
assert _me(client)["search_source"] == "api"
|
||||
@@ -9,7 +9,7 @@ independent literal, there was no `.python-version`, no `requires-python`, and r
|
||||
|
||||
**The contract is a chain, and it is split across two test files because no single runner can see
|
||||
every link.** This suite runs inside the test image, which mounts `backend/` and nothing else, so it
|
||||
cannot read the repo-root Dockerfile or the workflow. It therefore holds the link it CAN see, and the
|
||||
cannot read the repo-root Dockerfile. It therefore holds the link it CAN see, and the
|
||||
one that matters most here:
|
||||
|
||||
the interpreter actually running == backend/pyrightconfig.json == backend/ruff.toml
|
||||
@@ -17,7 +17,7 @@ one that matters most here:
|
||||
and `frontend/src/lib/pythonEngine.contract.test.ts` — which runs on the host, with the whole tree —
|
||||
holds the rest of the chain:
|
||||
|
||||
/.python-version == pyrightconfig.json == ruff.toml == Dockerfile == Dockerfile.test == check.yml
|
||||
/.python-version == pyrightconfig.json == ruff.toml == Dockerfile == Dockerfile.test
|
||||
|
||||
Together they close the loop. To move the version: edit `/.python-version`, then follow the failures
|
||||
in both files. Every place they name is a place that would otherwise have been forgotten.
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
"""`quota_events` has a retention: the daily history-cleanup job (`audit_gc`) prunes rows older than
|
||||
`quota_retention_days`, as it does the audit log. Without it the table only ever grew — one row per
|
||||
API spend plus a zero-unit row per live search."""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import scheduler, sysconfig
|
||||
from app.models import QuotaEvent
|
||||
|
||||
|
||||
def _event(days_ago: int) -> QuotaEvent:
|
||||
return QuotaEvent(
|
||||
user_id=None,
|
||||
action="videos.list",
|
||||
units=1,
|
||||
created_at=datetime.now(timezone.utc) - timedelta(days=days_ago),
|
||||
)
|
||||
|
||||
|
||||
def _remaining(db) -> list[int]:
|
||||
db.expire_all()
|
||||
return sorted(e.units for e in db.execute(select(QuotaEvent)).scalars())
|
||||
|
||||
|
||||
def test_the_history_cleanup_prunes_quota_events_past_the_retention(db):
|
||||
old, recent = _event(400), _event(10)
|
||||
recent.units = 7
|
||||
db.add_all([old, recent])
|
||||
db.commit()
|
||||
|
||||
scheduler.JOB_FUNCS["audit_gc"]()
|
||||
|
||||
assert _remaining(db) == [7] # the default keeps a year, which the admin usage view can show
|
||||
|
||||
|
||||
def test_a_zero_retention_keeps_quota_history_forever(db):
|
||||
db.add(_event(4000))
|
||||
db.commit()
|
||||
sysconfig.set_value(db, "quota_retention_days", 0)
|
||||
db.commit()
|
||||
|
||||
scheduler.JOB_FUNCS["audit_gc"]()
|
||||
|
||||
assert _remaining(db) == [1]
|
||||
@@ -0,0 +1,109 @@
|
||||
"""The worker obeys the two admin knobs that concern it — `download_worker_concurrency` and
|
||||
`log_level` — as stored in the settings page, not only the env it booted with. It used to read
|
||||
`settings.download_worker_concurrency` and `settings.log_level` once at start, so the admin page
|
||||
showed values the worker never applied. No DB or real threads of work here: the stored values come
|
||||
from a patched `sysconfig`, and the loop threads are stand-ins that only record their index."""
|
||||
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
from app import worker
|
||||
|
||||
|
||||
class _NoSession:
|
||||
def __enter__(self):
|
||||
return None
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stored(monkeypatch):
|
||||
"""The settings page's values, served to the worker instead of a database."""
|
||||
values = {"download_worker_concurrency": 3, "log_level": "DEBUG"}
|
||||
monkeypatch.setattr(worker, "SessionLocal", _NoSession)
|
||||
monkeypatch.setattr(worker.sysconfig, "get_int", lambda _db, key: int(values[key]))
|
||||
monkeypatch.setattr(worker.sysconfig, "get_str", lambda _db, key: str(values[key]))
|
||||
return values
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def levels(monkeypatch):
|
||||
applied: list[str] = []
|
||||
monkeypatch.setattr(worker.logs, "set_level", applied.append)
|
||||
return applied
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def loops(monkeypatch):
|
||||
"""Stand-in loop threads: each records its index and parks until the worker stops."""
|
||||
started: list[int] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def fake_loop(idx: int) -> None:
|
||||
with lock:
|
||||
started.append(idx)
|
||||
worker._stop.wait()
|
||||
|
||||
monkeypatch.setattr(worker, "_worker_loop", fake_loop)
|
||||
yield started
|
||||
worker._stop.set()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_stop():
|
||||
worker._stop.clear()
|
||||
yield
|
||||
worker._stop.clear()
|
||||
|
||||
|
||||
def test_worker_boots_with_the_stored_concurrency_and_log_level(monkeypatch, stored, levels, loops):
|
||||
monkeypatch.setattr(worker.settings, "download_worker_concurrency", 2)
|
||||
monkeypatch.setattr(worker.settings, "worker_enabled", True)
|
||||
monkeypatch.setattr(worker.logs, "configure_logging", lambda _source: None)
|
||||
monkeypatch.setattr(worker.signal, "signal", lambda *_a: None)
|
||||
monkeypatch.setattr(worker, "_wait_for_schema", lambda: True)
|
||||
monkeypatch.setattr(worker.logsink, "start", lambda: None)
|
||||
monkeypatch.setattr(worker, "_recover_orphans", lambda: None)
|
||||
|
||||
threading.Timer(0.5, worker._stop.set).start()
|
||||
worker.main()
|
||||
|
||||
assert sorted(loops) == [0, 1, 2] # the stored 3, not the env's 2
|
||||
assert levels and levels[-1] == "DEBUG"
|
||||
|
||||
|
||||
def test_a_changed_knob_applies_without_a_restart(stored, levels, loops):
|
||||
threads: dict[int, threading.Thread] = {}
|
||||
level = worker._sync_knobs(threads, None)
|
||||
assert level == "DEBUG"
|
||||
assert worker._target == 3
|
||||
|
||||
stored["download_worker_concurrency"] = 5
|
||||
stored["log_level"] = "WARNING"
|
||||
level = worker._sync_knobs(threads, level)
|
||||
|
||||
assert worker._target == 5
|
||||
assert sorted(threads) == [0, 1, 2, 3, 4]
|
||||
assert level == "WARNING" and levels[-1] == "WARNING"
|
||||
|
||||
# Lowering it retires the surplus threads after their current job: they see the lower target.
|
||||
stored["download_worker_concurrency"] = 1
|
||||
worker._sync_knobs(threads, level)
|
||||
assert worker._target == 1
|
||||
|
||||
|
||||
def test_an_unreadable_database_keeps_what_runs(monkeypatch, levels, loops):
|
||||
def boom():
|
||||
raise RuntimeError("db down")
|
||||
|
||||
monkeypatch.setattr(worker, "SessionLocal", boom)
|
||||
monkeypatch.setattr(worker, "_target", 2)
|
||||
threads: dict[int, threading.Thread] = {}
|
||||
|
||||
assert worker._sync_knobs(threads, "INFO") == "INFO"
|
||||
assert worker._target == 2
|
||||
assert sorted(threads) == [0, 1] # the loops still run on the boot value
|
||||
assert levels == []
|
||||
+20
-4
@@ -5,7 +5,7 @@ How to run Siftlode locally and pass the gate. Conventions for the backend code
|
||||
|
||||
## Node
|
||||
|
||||
One major, everywhere. `.nvmrc` pins it; the CI container, the `Dockerfile`'s frontend build stage,
|
||||
One major, everywhere. `.nvmrc` pins it; the `Dockerfile`'s frontend build stage,
|
||||
`frontend/package.json`'s `engines` and `@types/node` must all match it — and so must the machine you
|
||||
run the gate on. `frontend/src/lib/nodeEngine.contract.test.ts` fails the gate the moment any of them
|
||||
drift apart, including when yours is the odd one out. `nvm use` in the repo root picks it up.
|
||||
@@ -38,9 +38,20 @@ decisions and the chosen patterns, rendered with the app's real CSS. It never re
|
||||
|
||||
## The gate
|
||||
|
||||
A change is done when all of this is green. CI runs the same checks server-side
|
||||
(`.gitea/workflows/check.yml`), so "main is always green". What each gate deliberately does NOT
|
||||
check — and why — is the [gate register](gates.md); a test fails when a narrowing is missing from it.
|
||||
A change is done when this is green, from the repo root:
|
||||
|
||||
```bash
|
||||
bash scripts/check.sh # every lane
|
||||
bash scripts/check.sh lint # the static lanes only (no tests, build or audits) — the quick loop
|
||||
```
|
||||
|
||||
It runs every lane even after one fails and names each red one at the end. What it runs, and why
|
||||
each lane runs where it does, is in the script's comments; `frontend/src/lib/gates.contract.test.ts`
|
||||
holds its lines, so a lane dropped or narrowed there fails a test. What each gate deliberately does
|
||||
NOT check — and why — is the [gate register](gates.md). There is no CI: the gate runs here, and a
|
||||
release runs it (plus e2e and the CVE scan) before anything is merged.
|
||||
|
||||
The lanes, for running one by hand:
|
||||
|
||||
**Backend**
|
||||
|
||||
@@ -58,6 +69,10 @@ docker compose -f backend/docker-compose.test.yml run --rm --no-deps --build tes
|
||||
# coverage total. The backend tree is bind-mounted, so it always tests your working files.
|
||||
docker compose -f backend/docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from test
|
||||
docker compose -f backend/docker-compose.test.yml down -v
|
||||
|
||||
# the lock against the advisory databases, at any severity
|
||||
docker compose -f backend/docker-compose.test.yml run --rm --no-deps test \
|
||||
pip-audit -r requirements.lock --no-deps --disable-pip
|
||||
```
|
||||
|
||||
A missing database or ffmpeg is a hard error, not a skip — so the gate can't print green while the
|
||||
@@ -75,6 +90,7 @@ npm run format:check # prettier
|
||||
npm run knip # unused files/exports/deps — and unused by what SHIPS (--production)
|
||||
npm run test # vitest (incl. the compiler, size/complexity and gate-register contracts)
|
||||
npx vite build # the production bundle (devOnlyGuard fails it on dev-only code)
|
||||
npm audit --omit=dev --audit-level=high # what ships to the browser, at high or above
|
||||
```
|
||||
|
||||
Coverage is reported, never enforced: `npm run test -- --coverage`, and the pytest lane's last line.
|
||||
|
||||
+5
-5
@@ -14,10 +14,10 @@ name its invariant. Whatever is left narrowed is listed here, and
|
||||
not an oversight; a narrowing missing from here is a test failure.
|
||||
|
||||
**The snapshot.** ESLint's effective config for every code file `eslint .` reaches, and the lines
|
||||
that run the lanes (`.gitea/workflows/check.yml`, `backend/docker-compose.test.yml`,
|
||||
that run the lanes (`scripts/check.sh`, `backend/docker-compose.test.yml`,
|
||||
`backend/Dockerfile.test`, the npm scripts, `vitest.config.ts`, `.prettierignore`), are committed in
|
||||
[`frontend/src/lib/gates.snapshot.json`](../frontend/src/lib/gates.snapshot.json). The test fails on
|
||||
ANY difference — a rule, an option, a glob that moves one file to another config, a new ignore, a CI
|
||||
ANY difference — a rule, an option, a glob that moves one file to another config, a new ignore, a gate
|
||||
flag. So a change to a gate is always a diff of that file in review, never a shape a detector failed
|
||||
to model (E32 review: three rounds of a cleverer detector each found a bypass of the last one's
|
||||
fixes). Regenerate it with `GATE_SNAPSHOT=update npx vitest run src/lib/gates.contract.test.ts`
|
||||
@@ -117,17 +117,17 @@ project checks needs a row.
|
||||
| Scope | Narrowed | Why | Measured |
|
||||
| --- | --- | --- | --- |
|
||||
| check | e2e (Playwright) not in `check` | It needs the :8080 stack built from the tree (minutes). It runs in `check-full`, and `release` rebuilds the stack and requires it (E32 D2) — nothing ships without it. | 5 specs × 3 engines |
|
||||
| CI | no e2e, no Trivy | The lab runner has no browser stack, and Trivy's DB download is not reachable from it. Both run in `release` / `publish` on the dev machine, before anything is pushed. | — |
|
||||
| pushes | no server-side gate (no CI) | Owner decision 2026-10-04. The gate (`scripts/check.sh`) runs on the dev machine before work is handed over, and `release` runs it, then e2e, before anything is merged and Trivy before the image is pushed — the CI only repeated that, after the merge. Its runner shared the git host's container, and one run there starved Gitea of memory until it stopped answering. What it added — a fresh `npm ci` on Linux — the image build does at publish. | — |
|
||||
| coverage | no threshold | Owner decision E32 D4: reported on both sides, never enforced. "Every regression test seen red" is the bar; a percentage invites tests written to move it. | frontend 80% / backend 78% of lines (E32 S1) |
|
||||
| CVE gate | HIGH/CRITICAL with a fix available only | `--ignore-unfixed` + `--severity HIGH,CRITICAL`: a finding with no fix cannot be acted on by an upgrade; it shows in the scan output. A fixable one fails the publish, and the only way past it is `.trivyignore.yaml`. | 0 open (E32 S1) |
|
||||
| npm audit (CI) | `--omit=dev --audit-level=high` | What ships to the browser, at high or above. The dev toolchain's advisories were burned down by hand in E32 S1 (18 → 0). | 0 |
|
||||
| npm audit | `--omit=dev --audit-level=high` | What ships to the browser, at high or above. The dev toolchain's advisories were burned down by hand in E32 S1 (18 → 0). | 0 |
|
||||
| React Compiler | the functions it skips, pinned | `components/compilerCoverage.contract.test.ts` — the list may only shrink. End state: only VirtualGrid (TanStack Virtual, an IncompatibleLibrary) remains. | 93 (E32 S1) → 29 (E32 S3) → 1, VirtualGrid (E32 S4) |
|
||||
| size / complexity | today's offenders as ceilings | `src/lib/sourceSize.contract.test.ts`, `backend/tests/test_source_size.py`: listed files and functions may not grow, nothing new may cross the limit, an entry back under it must be deleted. A function is held by its name, not its rank in the file. | files >800 lines: 13 FE / 9 BE; functions over the limit: 64 FE / 32 BE (E32 review) |
|
||||
| size / complexity | shipped code only — not tests, test kits, e2e or `scripts/` | The ratchet guards what runs in production. A test file is long because it is a list of cases, and a test kit is test code; splitting either moves lines without making anything easier to change. `scripts/` is release and seed tooling, not in the image. | outside it: tests >800 lines 2 FE / 5 BE; functions over the limit 2 FE test code (`gates.contract.test.ts` `eslintNarrowings` 23, E32 review round 3; `compilerSkips.testkit.ts` 25), 1 BE test (11), 1 script (`e2e_seed.py` `main` 13) |
|
||||
|
||||
## Accepted findings
|
||||
|
||||
Entries in `.trivyignore.yaml` (CVE gate) and `--ignore-vuln` flags on CI's `pip-audit` step. Each
|
||||
Entries in `.trivyignore.yaml` (CVE gate) and `--ignore-vuln` flags on the gate's `pip-audit` lane (`scripts/check.sh`). Each
|
||||
must have a row here (Scope `accepted`) with its reason and its expiry; the test fails on an entry
|
||||
without one, and on a `.trivyignore.yaml` entry with no `statement` or no `expired_at`.
|
||||
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
// @vitest-environment jsdom
|
||||
//
|
||||
// The live-search button's tooltip says what a search costs under the search source the instance
|
||||
// actually runs. It used to say "uses shared API quota" always, while the default source (the
|
||||
// website's own endpoint) spends none — so the owner avoided a free search for months.
|
||||
import { describe, it, expect, vi, afterEach } from "vitest";
|
||||
import { render, screen, cleanup } from "@testing-library/react";
|
||||
|
||||
vi.mock("react-i18next", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("react-i18next")>()),
|
||||
useTranslation: () => ({ t: (k: string) => k, i18n: { language: "en" } }),
|
||||
}));
|
||||
vi.mock("./navigationContext", () => ({ useNavigation: () => ({ page: "feed" }) }));
|
||||
vi.mock("./feedFiltersContext", () => ({
|
||||
useFeedFilters: () => ({ q: "cats", sort: "newest" }),
|
||||
useFeedFiltersActions: () => ({ setFilters: vi.fn() }),
|
||||
}));
|
||||
vi.mock("./playlistsContext", () => ({
|
||||
usePlaylists: () => ({ search: "" }),
|
||||
usePlaylistsActions: () => ({ setSearch: vi.fn() }),
|
||||
}));
|
||||
vi.mock("./channelsContext", () => ({
|
||||
useChannels: () => ({ search: "" }),
|
||||
useChannelsActions: () => ({ setSearch: vi.fn(), goToFullHistory: vi.fn() }),
|
||||
}));
|
||||
vi.mock("./SyncStatus", () => ({ default: () => null }));
|
||||
|
||||
import Header from "./Header";
|
||||
import type { Me } from "../lib/api";
|
||||
|
||||
afterEach(cleanup);
|
||||
|
||||
function tipFor(source: Me["search_source"]): string | null {
|
||||
const me = { is_demo: false, plex_enabled: false, search_source: source } as Me;
|
||||
render(<Header me={me} onYtSearch={() => {}} />);
|
||||
return screen.getByRole("button", { name: /header\.searchYoutube$/ }).getAttribute("title");
|
||||
}
|
||||
|
||||
describe("the live-search tooltip", () => {
|
||||
it("does not claim a quota cost under the free website source", () => {
|
||||
expect(tipFor("scrape")).toBe("header.searchYoutubeTip");
|
||||
});
|
||||
|
||||
it("names the quota cost when the instance searches through the official API", () => {
|
||||
expect(tipFor("api")).toBe("header.searchYoutubeTipApi");
|
||||
});
|
||||
});
|
||||
@@ -60,7 +60,9 @@ const Header = forwardRef<
|
||||
}}
|
||||
onGo={() => onYtSearch(trimmed)}
|
||||
goLabel={t("header.searchYoutube")}
|
||||
goTitle={t("header.searchYoutubeTip")}
|
||||
goTitle={t(
|
||||
me.search_source === "api" ? "header.searchYoutubeTipApi" : "header.searchYoutubeTip",
|
||||
)}
|
||||
goIcon={<YouTubeIcon className="w-4 h-4" />}
|
||||
goDisabled={!trimmed || me.is_demo}
|
||||
/>
|
||||
|
||||
@@ -1,4 +1,22 @@
|
||||
[
|
||||
{
|
||||
"version": "0.99.2",
|
||||
"date": "2026-10-04",
|
||||
"new": [],
|
||||
"fixed": [
|
||||
"Settings: the download worker's concurrency and the log level now take effect within seconds — before, the worker ignored both until it was rebuilt.",
|
||||
"The YouTube quota history no longer grows forever: the daily cleanup prunes it after a year (Configuration → Quota → Quota history kept).",
|
||||
"The live-search button no longer says a search spends the API quota: with the default website source the search itself is free. It names the cost when the official API is selected."
|
||||
]
|
||||
},
|
||||
{
|
||||
"version": "0.99.1",
|
||||
"date": "2026-10-04",
|
||||
"new": [],
|
||||
"fixed": [
|
||||
"Behind the scenes: every release is now checked by one script on the build machine, including the security audit of every dependency."
|
||||
]
|
||||
},
|
||||
{
|
||||
"version": "0.99.0",
|
||||
"date": "2026-10-04",
|
||||
|
||||
@@ -1,4 +1,22 @@
|
||||
[
|
||||
{
|
||||
"version": "0.99.2",
|
||||
"date": "2026-10-04",
|
||||
"new": [],
|
||||
"fixed": [
|
||||
"Beállítások: a letöltő párhuzamossága és a naplószint pár másodpercen belül érvénybe lép — eddig a letöltő mindkettőt figyelmen kívül hagyta az újraépítéséig.",
|
||||
"A YouTube-kvótaelőzmény többé nem nő a végtelenségig: a napi takarítás egy év után törli (Konfiguráció → Kvóta → Kvótaelőzmények megőrzése).",
|
||||
"Az élő keresés gombja többé nem állítja, hogy a keresés API-kvótát fogyaszt: az alapértelmezett, weboldalas forrással maga a keresés ingyenes. A hivatalos API kiválasztásakor kiírja a költséget."
|
||||
]
|
||||
},
|
||||
{
|
||||
"version": "0.99.1",
|
||||
"date": "2026-10-04",
|
||||
"new": [],
|
||||
"fixed": [
|
||||
"A háttérben: minden kiadást egyetlen ellenőrző script vizsgál a build gépen, minden függőség biztonsági auditjával együtt."
|
||||
]
|
||||
},
|
||||
{
|
||||
"version": "0.99.0",
|
||||
"date": "2026-10-04",
|
||||
|
||||
@@ -62,6 +62,10 @@
|
||||
"label": "Live search source",
|
||||
"hint": "Where live YouTube search results come from. “Website” reads YouTube's own search endpoint and spends no API quota (recommended). “Official API” uses search.list (100 units/page) — switch to it if the website source ever breaks or is blocked."
|
||||
},
|
||||
"quota_retention_days": {
|
||||
"label": "Quota history kept (days)",
|
||||
"hint": "How long the per-action YouTube quota history (who spent what) is kept before the history cleanup job prunes it. 0 = keep forever. Default 365, so the usage view's longest window stays whole."
|
||||
},
|
||||
"backfill_recent_max_videos": {
|
||||
"label": "Recent backfill — max videos",
|
||||
"hint": "Newest videos fetched per channel on the first pass."
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
"searchPlaceholder": "Search your subscriptions…",
|
||||
"clearSearch": "Clear search",
|
||||
"searchYoutube": "Search YouTube",
|
||||
"searchYoutubeTip": "Search live on YouTube for this term (uses shared API quota). You can also press Enter.",
|
||||
"searchYoutubeTip": "Search live on YouTube for this term — the search itself spends no API quota; adding the new videos it finds costs a unit or two. You can also press Enter.",
|
||||
"searchYoutubeTipApi": "Search live on YouTube for this term — uses the shared API quota (100 units a page). You can also press Enter.",
|
||||
"go": "Go",
|
||||
"goTip": "Search (or press Enter)",
|
||||
"channelManager": "Channels",
|
||||
|
||||
@@ -76,7 +76,7 @@
|
||||
"plex_sync": "Mirrors the enabled Plex libraries into the app's catalogue — the metadata behind Plex browsing, search, filters and the info pages (genres, cast, ratings, artwork). If it stops, newly added or changed Plex titles don't appear until the next sync.",
|
||||
"plex_watch_sync": "Pulls recent Plex watch changes (finished episodes, resume positions) into Siftlode and re-pushes any local watch changes that didn't reach Plex. The cheap two-way keep-in-sync between full reconciles. If it stops, watched/resume made in the Plex app takes longer to show here.",
|
||||
"plex_watch_reconcile": "A full watch-state reconcile that rescans every library — catches what the incremental pass can't, notably an episode un-marked as watched on the Plex side. Heavier, so it runs about once a day. If it stops, un-watches done in Plex aren't mirrored back.",
|
||||
"audit_gc": "Prunes admin audit-log entries older than the configured retention window (Configuration → Audit log). If it stops, the audit log just keeps growing — harmless but unbounded.",
|
||||
"audit_gc": "Prunes admin audit-log entries and YouTube quota history older than their retention windows (Configuration → Audit log, Quota). If it stops, both just keep growing — harmless but unbounded.",
|
||||
"log_gc": "Deletes stored log events older than the retention window (Configuration → Diagnostics). Runs hourly so nothing outlives the promised limit by more than an hour.",
|
||||
"tmdb_enrich": "Enriches the mirrored Plex catalogue from TMDB (keywords, crew, budget/revenue, TMDb score, recommendations, artwork), matched by each title's canonical id and served from the app's own database. Optional — a no-op until a TMDB API key is set. If it stops, new Plex titles just miss their TMDB extras until it runs again.",
|
||||
"people_enrich": "Makes the people in the Plex library local: fetches each person's biography from TMDB and caches their portrait, resized through Plex, the most-credited people first. A person page and a cast strip then open from the app's own data. The TMDB half does nothing without a TMDB API key, the portrait half nothing without Plex. If it stops, a person page fetches its own bio when opened, a moment later."
|
||||
@@ -97,7 +97,7 @@
|
||||
"plex_watch_sync": "Plex watch sync (incremental)",
|
||||
"plex_watch_reconcile": "Plex watch reconcile (full)",
|
||||
"tmdb_enrich": "TMDB enrichment (Plex)",
|
||||
"audit_gc": "Audit-log cleanup",
|
||||
"audit_gc": "Audit-log and quota-history cleanup",
|
||||
"log_gc": "Log cleanup",
|
||||
"people_enrich": "People details and portraits (Plex)"
|
||||
},
|
||||
@@ -172,6 +172,7 @@
|
||||
"countries_coded": "countries coded",
|
||||
"sessions_closed": "sessions closed",
|
||||
"sessions_reaped": "sessions cleaned up",
|
||||
"quota_reaped": "quota history cleaned up",
|
||||
"recent": "recent",
|
||||
"deep": "deep",
|
||||
"bios": "biographies",
|
||||
|
||||
@@ -62,6 +62,10 @@
|
||||
"label": "Élő keresés forrása",
|
||||
"hint": "Honnan jönnek az élő YouTube-keresés találatai. A „Weboldal” a YouTube saját keresővégpontját olvassa, és nem fogyaszt API-kvótát (ajánlott). A „Hivatalos API” a search.list-et használja (100 egység/oldal) — válts rá, ha a weboldalas forrás elromlik vagy letiltják."
|
||||
},
|
||||
"quota_retention_days": {
|
||||
"label": "Kvótaelőzmények megőrzése (nap)",
|
||||
"hint": "Meddig marad meg a YouTube-kvóta tételes előzménye (ki mit fogyasztott), mielőtt az előzmény-takarító feladat törli. 0 = örökre. Alapérték 365, így a használati nézet leghosszabb ablaka is teljes marad."
|
||||
},
|
||||
"backfill_recent_max_videos": {
|
||||
"label": "Friss behúzás — max videó",
|
||||
"hint": "Csatornánként az első körben behúzott legújabb videók száma."
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
"searchPlaceholder": "Keresés a feliratkozásaid között…",
|
||||
"clearSearch": "Keresés törlése",
|
||||
"searchYoutube": "Keresés a YouTube-on",
|
||||
"searchYoutubeTip": "Élő keresés a YouTube-on erre a kifejezésre (a közös API-kvótát fogyasztja). Entert is nyomhatsz.",
|
||||
"searchYoutubeTip": "Élő keresés a YouTube-on erre a kifejezésre — maga a keresés nem fogyaszt API-kvótát, a talált új videók behúzása is csak egy-két egységet. Entert is nyomhatsz.",
|
||||
"searchYoutubeTipApi": "Élő keresés a YouTube-on erre a kifejezésre — a közös API-kvótát fogyasztja (oldalanként 100 egység). Entert is nyomhatsz.",
|
||||
"go": "Mehet",
|
||||
"goTip": "Keresés (vagy Enter)",
|
||||
"channelManager": "Csatornák",
|
||||
|
||||
@@ -76,7 +76,7 @@
|
||||
"plex_sync": "Tükrözi az engedélyezett Plex-könyvtárakat az app katalógusába — ez adja a Plex böngészés, keresés, szűrők és infóoldalak metaadatait (műfaj, szereplők, pontszám, borítók). Ha leáll, az újonnan hozzáadott vagy módosított Plex-címek csak a következő szinkron után jelennek meg.",
|
||||
"plex_watch_sync": "Áthozza a Plex-oldali friss megtekintés-változásokat (befejezett epizódok, folytatás-pozíciók) a Siftlode-ba, és felnyomja a Plexet el nem érő helyi változásokat. Ez az olcsó kétirányú szinkron a teljes egyeztetések között. Ha leáll, a Plex-appban jelölt megtekintés/folytatás lassabban jelenik meg itt.",
|
||||
"plex_watch_reconcile": "A megtekintési állapot teljes egyeztetése, amely minden könyvtárat újraszkennel — elkapja, amit az inkrementális menet nem, főleg ha egy epizódról a Plex oldalán levették a megtekintve jelet. Nehezebb, ezért kb. naponta fut. Ha leáll, a Plexben levett megtekintve jel nem tükröződik vissza.",
|
||||
"audit_gc": "Törli a beállított megőrzési időnél (Konfiguráció → Eseménynapló) régebbi admin-naplóbejegyzéseket. Ha leáll, az eseménynapló csak nő — ártalmatlan, de korlátlanul.",
|
||||
"audit_gc": "Törli a megőrzési időnél régebbi admin-naplóbejegyzéseket és YouTube-kvótaelőzményeket (Konfiguráció → Eseménynapló, Kvóta). Ha leáll, mindkettő csak nő — ártalmatlan, de korlátlanul.",
|
||||
"log_gc": "Törli a megőrzési időnél (Konfiguráció → Diagnosztika) régebbi tárolt naplóeseményeket. Óránként fut, így semmi sem marad meg egy óránál tovább az ígért határidőn túl.",
|
||||
"tmdb_enrich": "A tükrözött Plex-katalógust dúsítja a TMDB-ből (kulcsszavak, stáb, költségvetés/bevétel, TMDb-pontszám, ajánlások, artwork), a címek kanonikus azonosítója alapján párosítva, és az app saját adatbázisából szolgálja ki. Opcionális — TMDB API-kulcs nélkül nem csinál semmit. Ha leáll, az új Plex-címek csak a következő futásig maradnak TMDB-extrák nélkül.",
|
||||
"people_enrich": "Helyivé teszi a Plex-könyvtár szereplőit: a TMDB-ből letölti az életrajzukat, a portréjukat pedig a Plexszel kicsinyítve elmenti, a legtöbbet szereplőkkel kezdve. A személyoldal és a szereplősáv így az app saját adataiból nyílik meg. A TMDB-rész TMDB API-kulcs nélkül nem csinál semmit, a portrérész Plex nélkül. Ha leáll, a személyoldal megnyitáskor maga kéri le az életrajzot, egy pillanattal később."
|
||||
@@ -97,7 +97,7 @@
|
||||
"plex_watch_sync": "Plex megtekintés-szinkron (inkrementális)",
|
||||
"plex_watch_reconcile": "Plex megtekintés-egyeztetés (teljes)",
|
||||
"tmdb_enrich": "TMDB-dúsítás (Plex)",
|
||||
"audit_gc": "Eseménynapló-takarítás",
|
||||
"audit_gc": "Eseménynapló- és kvótaelőzmény-takarítás",
|
||||
"log_gc": "Naplótakarítás",
|
||||
"people_enrich": "Személyadatok és portrék (Plex)"
|
||||
},
|
||||
@@ -172,6 +172,7 @@
|
||||
"countries_coded": "országkód pótolva",
|
||||
"sessions_closed": "lezárt munkamenet",
|
||||
"sessions_reaped": "eltakarított munkamenet",
|
||||
"quota_reaped": "eltakarított kvótaelőzmény",
|
||||
"recent": "friss",
|
||||
"deep": "mély",
|
||||
"bios": "életrajzok",
|
||||
|
||||
@@ -20,6 +20,7 @@ export interface Me {
|
||||
has_password: boolean;
|
||||
google_enabled: boolean;
|
||||
plex_enabled: boolean;
|
||||
search_source: "scrape" | "api";
|
||||
can_read: boolean;
|
||||
can_write: boolean;
|
||||
pending_invites: number;
|
||||
|
||||
@@ -105,7 +105,7 @@ function restrictedEntries(entry: Linter.RuleEntry | undefined): Set<string> {
|
||||
// the lines that run the lanes. Any change fails until the snapshot is regenerated, so a narrowing
|
||||
// arrives as a diff of `gates.snapshot.json` in review instead of slipping past a detector. (Three
|
||||
// review rounds kept finding shapes the detector this replaced did not model — a `basePath` global
|
||||
// ignore, a widened glob filed under a narrow row, a substring check on a CI line. A comparison
|
||||
// ignore, a widened glob filed under a narrow row, a substring check on a gate line. A comparison
|
||||
// against everything has no shape to miss.)
|
||||
//
|
||||
// Regenerate: GATE_SNAPSHOT=update npx vitest run src/lib/gates.contract.test.ts
|
||||
@@ -285,7 +285,7 @@ function meaningful(path: string): string[] {
|
||||
});
|
||||
}
|
||||
const lanes = () => ({
|
||||
".gitea/workflows/check.yml": meaningful(".gitea/workflows/check.yml"),
|
||||
"scripts/check.sh": meaningful("scripts/check.sh"),
|
||||
"backend/docker-compose.test.yml": meaningful("backend/docker-compose.test.yml"),
|
||||
"backend/Dockerfile.test": meaningful("backend/Dockerfile.test"),
|
||||
"frontend/package.json scripts": (
|
||||
@@ -485,19 +485,17 @@ describe("the gate register (docs/gates.md)", () => {
|
||||
});
|
||||
// Every ruff lane passes `--config backend/ruff.toml`, which makes ruff ignore any other config
|
||||
// file it would discover (measured: a nested `ruff.toml` selecting the `D` rules changed nothing).
|
||||
// So that flag is what makes this file the whole configuration. The snapshot holds every CI line;
|
||||
// these two are spelled out here because the reason is (round 3: a substring check let an extra
|
||||
// `--config "lint.extend-ignore=…"` override through).
|
||||
it("runs every CI ruff step on backend/ruff.toml and nothing else", () => {
|
||||
const steps = read(".gitea/workflows/check.yml")
|
||||
// So that flag is what makes this file the whole configuration. The snapshot holds every line of
|
||||
// the gate script; these two are spelled out because the reason is (round 3: a substring check let
|
||||
// an extra `--config "lint.extend-ignore=…"` override through).
|
||||
it("runs both ruff lanes on backend/ruff.toml and nothing else", () => {
|
||||
const lanes = read("scripts/check.sh")
|
||||
.split("\n")
|
||||
.filter((l) => /\bruff\b/.test(l) && !l.trim().startsWith("#"))
|
||||
.filter((l) => /python -m ruff (check|format)/.test(l))
|
||||
.map((l) => l.trim());
|
||||
expect(steps).toEqual([
|
||||
"- name: ruff",
|
||||
"run: python -m ruff check --config backend/ruff.toml backend scripts",
|
||||
"- name: ruff format --check",
|
||||
"run: python -m ruff format --check --config backend/ruff.toml backend scripts",
|
||||
expect(lanes).toEqual([
|
||||
"ruff_check() { ruff_pinned && python -m ruff check --config backend/ruff.toml backend scripts; }",
|
||||
"ruff_format() { ruff_pinned && python -m ruff format --check --config backend/ruff.toml backend scripts; }",
|
||||
]);
|
||||
});
|
||||
|
||||
@@ -577,9 +575,9 @@ describe("the gate register (docs/gates.md)", () => {
|
||||
expires: /\n\s*expired_at:\s*\d{4}-\d{2}-\d{2}/.test(chunk),
|
||||
}));
|
||||
expect(entries.filter((e) => !e.statement || !e.expires).map((e) => e.id)).toEqual([]);
|
||||
const pipIgnores = read(".gitea/workflows/check.yml")
|
||||
const pipIgnores = read("scripts/check.sh")
|
||||
.split("\n")
|
||||
.filter((l) => !l.trim().startsWith("#")) // the step's comment explains the flag; it is not one
|
||||
.filter((l) => !l.trim().startsWith("#")) // the lane's comment explains the flag; it is not one
|
||||
.flatMap((l) => [...l.matchAll(/--ignore-vuln[ =](\S+)/g)].map((m) => m[1]!));
|
||||
const found = [...entries.map((e) => e.id), ...pipIgnores].map((id): [string, string] => [
|
||||
"accepted",
|
||||
|
||||
@@ -1794,6 +1794,7 @@
|
||||
"src/components/Feed.ytsort.dom.test.tsx",
|
||||
"src/components/FeedFiltersProvider.dom.test.ts",
|
||||
"src/components/FeedFiltersProvider.seed.dom.test.tsx",
|
||||
"src/components/Header.searchTip.dom.test.tsx",
|
||||
"src/components/Home.dom.test.tsx",
|
||||
"src/components/Home.pin.dom.test.tsx",
|
||||
"src/components/IssueComposer.dom.test.tsx",
|
||||
@@ -2222,68 +2223,6 @@
|
||||
}
|
||||
},
|
||||
"lanes": {
|
||||
".gitea/workflows/check.yml": [
|
||||
"name: gate",
|
||||
"on:",
|
||||
" push:",
|
||||
" branches: [\"**\"]",
|
||||
" pull_request:",
|
||||
"jobs:",
|
||||
" frontend:",
|
||||
" runs-on: ubuntu-latest",
|
||||
" container: node:24-bookworm",
|
||||
" defaults:",
|
||||
" run:",
|
||||
" working-directory: frontend",
|
||||
" steps:",
|
||||
" - uses: actions/checkout@v4",
|
||||
" - run: npm ci",
|
||||
" - run: npm run typecheck",
|
||||
" - run: npm run lint",
|
||||
" - run: npm run format:check",
|
||||
" - run: npm run knip",
|
||||
" - run: npm run test",
|
||||
" - run: npx vite build",
|
||||
" - run: npm audit --omit=dev --audit-level=high",
|
||||
" backend:",
|
||||
" runs-on: ubuntu-latest",
|
||||
" container: nikolaik/python-nodejs:python3.12-nodejs24",
|
||||
" services:",
|
||||
" postgres:",
|
||||
" image: postgres:16-alpine",
|
||||
" env:",
|
||||
" POSTGRES_USER: siftlode",
|
||||
" POSTGRES_PASSWORD: siftlode",
|
||||
" POSTGRES_DB: siftlode_test",
|
||||
" options: >-",
|
||||
" --health-cmd \"pg_isready -U siftlode -d siftlode_test\"",
|
||||
" --health-interval 2s",
|
||||
" --health-timeout 3s",
|
||||
" --health-retries 30",
|
||||
" env:",
|
||||
" DATABASE_URL: postgresql+psycopg://siftlode:siftlode@postgres:5432/siftlode_test",
|
||||
" SIFTLODE_TEST_REQUIRE_DB: \"1\"",
|
||||
" steps:",
|
||||
" - uses: actions/checkout@v4",
|
||||
" - run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client",
|
||||
" - name: Wait for Postgres",
|
||||
" run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done",
|
||||
" - run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt",
|
||||
" - name: ruff",
|
||||
" run: python -m ruff check --config backend/ruff.toml backend scripts",
|
||||
" - name: ruff format --check",
|
||||
" run: python -m ruff format --check --config backend/ruff.toml backend scripts",
|
||||
" - name: basedpyright",
|
||||
" run: python -m basedpyright --project pyrightconfig.json",
|
||||
" working-directory: backend",
|
||||
" - name: basedpyright baseline is committed",
|
||||
" run: git diff --exit-code -- backend/.basedpyright/baseline.json",
|
||||
" - name: pytest",
|
||||
" run: python -m pytest",
|
||||
" working-directory: backend",
|
||||
" - name: pip-audit",
|
||||
" run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock"
|
||||
],
|
||||
"backend/Dockerfile.test": [
|
||||
"FROM python:3.12-slim",
|
||||
"WORKDIR /app",
|
||||
@@ -2366,6 +2305,77 @@
|
||||
" },",
|
||||
" },",
|
||||
"});"
|
||||
],
|
||||
"scripts/check.sh": [
|
||||
"set -u",
|
||||
"cd \"$(dirname \"$0\")/..\" || exit 2",
|
||||
"mode=\"${1:-all}\"",
|
||||
"case \"$mode\" in all | lint) ;; *) echo \"usage: scripts/check.sh [lint]\" >&2; exit 2 ;; esac",
|
||||
"export MSYS_NO_PATHCONV=1 # Git Bash: do not rewrite the in-container paths below",
|
||||
"TESTCOMPOSE=backend/docker-compose.test.yml",
|
||||
"failed=()",
|
||||
"lane() { # lane <name> <command…> — run it, remember a failure, never stop",
|
||||
" local name=\"$1\"",
|
||||
" shift",
|
||||
" echo \">> $name…\"",
|
||||
" \"$@\" || failed+=(\"$name\")",
|
||||
"}",
|
||||
"ruff_pinned() {",
|
||||
" local pin host",
|
||||
" pin=\"$(sed -n 's/^ruff==\\([0-9.]*\\).*/\\1/p' backend/requirements-dev.txt)\"",
|
||||
" host=\"$(python -m ruff --version 2>/dev/null | awk '{print $2}')\"",
|
||||
" [ -n \"$pin\" ] && [ \"$host\" = \"$pin\" ] && return 0",
|
||||
" echo \"ruff ${host:-missing} on the host, ${pin:-nothing} pinned — python -m pip install ruff==$pin\" >&2",
|
||||
" return 1",
|
||||
"}",
|
||||
"ruff_check() { ruff_pinned && python -m ruff check --config backend/ruff.toml backend scripts; }",
|
||||
"ruff_format() { ruff_pinned && python -m ruff format --check --config backend/ruff.toml backend scripts; }",
|
||||
"basedpyright() {",
|
||||
" local before",
|
||||
" before=\"$(git hash-object backend/.basedpyright/baseline.json)\"",
|
||||
" docker compose -f \"$TESTCOMPOSE\" run --rm --no-deps --build test \\",
|
||||
" python -m basedpyright --project pyrightconfig.json || return 1",
|
||||
" if [ \"$(git hash-object backend/.basedpyright/baseline.json)\" != \"$before\" ]; then",
|
||||
" echo \"this run rewrote backend/.basedpyright/baseline.json (errors were fixed) — commit it\" >&2",
|
||||
" return 1",
|
||||
" fi",
|
||||
" git diff HEAD --quiet -- backend/.basedpyright/baseline.json && return 0",
|
||||
" echo \"backend/.basedpyright/baseline.json is uncommitted — commit it\" >&2",
|
||||
" return 1",
|
||||
"}",
|
||||
"frontend() { (cd frontend && \"$@\"); }",
|
||||
"lane \"ruff\" ruff_check",
|
||||
"lane \"ruff format --check\" ruff_format",
|
||||
"lane \"basedpyright (new errors only)\" basedpyright",
|
||||
"lane \"lock check (frontend/node_modules == package-lock.json)\" node scripts/check-node-modules.mjs",
|
||||
"lane \"tsc (src + configs + e2e)\" frontend npm run --silent typecheck",
|
||||
"lane \"eslint (warnings fail too)\" frontend npm run --silent lint",
|
||||
"lane \"prettier --check\" frontend npm run --silent format:check",
|
||||
"lane \"knip (unused, and unused by what ships)\" frontend npm run --silent knip",
|
||||
"pytest_lane() { # the full suite against a throwaway Postgres; --pull keeps the base image current",
|
||||
" docker compose -f \"$TESTCOMPOSE\" build --pull -q test || return 1",
|
||||
" local rc=0",
|
||||
" docker compose -f \"$TESTCOMPOSE\" up --build --abort-on-container-exit --exit-code-from test || rc=1",
|
||||
" docker compose -f \"$TESTCOMPOSE\" down -v >/dev/null 2>&1",
|
||||
" return $rc",
|
||||
"}",
|
||||
"pip_audit() {",
|
||||
" docker compose -f \"$TESTCOMPOSE\" run --rm --no-deps test \\",
|
||||
" pip-audit -r requirements.lock --no-deps --disable-pip",
|
||||
"}",
|
||||
"if [ \"$mode\" = all ]; then",
|
||||
" lane \"vitest (+ coverage report)\" frontend npm run --silent test -- --coverage",
|
||||
" lane \"vite build\" frontend npx vite build --logLevel warn",
|
||||
" lane \"pytest (pure-logic + DB lane)\" pytest_lane",
|
||||
" lane \"pip-audit (backend lock)\" pip_audit",
|
||||
" lane \"npm audit (production dependencies)\" frontend npm audit --omit=dev --audit-level=high",
|
||||
"fi",
|
||||
"if [ ${#failed[@]} -eq 0 ]; then",
|
||||
" echo \">> gate ($mode): green ✓\"",
|
||||
" exit 0",
|
||||
"fi",
|
||||
"echo \">> gate ($mode): RED — ${failed[*]}\" >&2",
|
||||
"exit 1"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -48,18 +48,6 @@ it("builds the shipped bundle on the pinned engine", () => {
|
||||
).toBe(pinned);
|
||||
});
|
||||
|
||||
it("runs the server gate on the pinned engine", () => {
|
||||
// The frontend job only — the backend job's node runs the checkout action, never our code, so it is
|
||||
// deliberately not part of this contract (see the note in check.yml).
|
||||
expect(
|
||||
majorIn(
|
||||
readRoot(".gitea/workflows/check.yml"),
|
||||
/^\s*container: node:(\d+)-bookworm$/m,
|
||||
"check.yml",
|
||||
),
|
||||
).toBe(pinned);
|
||||
});
|
||||
|
||||
it("declares the pinned engine to npm and to tsc", () => {
|
||||
expect(pkg.engines?.node).toBe(`${pinned}.x`);
|
||||
// Types a major ahead of the runtime make tsc green over APIs the shipped engine does not have —
|
||||
|
||||
@@ -11,12 +11,12 @@ import { expect, it } from "vitest";
|
||||
//
|
||||
// It lives in the FRONTEND suite for one boring reason: this is the only runner that can see the
|
||||
// whole repository. The backend suite runs inside the test image, which mounts `backend/` alone, so
|
||||
// it cannot read the repo-root Dockerfile or the workflow. It holds the other half of the chain —
|
||||
// it cannot read the repo-root Dockerfile. It holds the other half of the chain —
|
||||
// including "the interpreter running these tests IS the pinned one", which only it can assert:
|
||||
//
|
||||
// backend/tests/test_python_engine_contract.py : running interpreter == pyrightconfig == ruff.toml
|
||||
// here : .python-version == pyrightconfig == ruff.toml
|
||||
// == Dockerfile == Dockerfile.test == check.yml
|
||||
// == Dockerfile == Dockerfile.test
|
||||
//
|
||||
// To move the version: edit `.python-version`, then follow the failures in both files.
|
||||
const readRoot = (rel: string) => readFileSync(new URL(`../../../${rel}`, import.meta.url), "utf8");
|
||||
@@ -55,18 +55,6 @@ it("tests on the pinned version", () => {
|
||||
).toBe(pinned);
|
||||
});
|
||||
|
||||
it("runs the server gate on the pinned version", () => {
|
||||
// The backend job's container. Its NODE is deliberately outside the Node contract (it only runs
|
||||
// the checkout action), but its PYTHON runs our code and our tests, so it is inside this one.
|
||||
expect(
|
||||
versionIn(
|
||||
readRoot(".gitea/workflows/check.yml"),
|
||||
/^\s*container: nikolaik\/python-nodejs:python(\d+)\.(\d+)-nodejs\d+$/m,
|
||||
"check.yml",
|
||||
),
|
||||
).toBe(pinned);
|
||||
});
|
||||
|
||||
it("type-checks against the pinned version", () => {
|
||||
// A type checker aimed at another version green-lights APIs the shipped interpreter does not have
|
||||
// — the same defect as the image split, moved to check time.
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
# The gate — every lane, in one place, run from the repo root (E32, after CI went: the lab runner
|
||||
# duplicated what this already ran before every release, and one CI run starved the git host of
|
||||
# memory until it stopped answering). The release dispatcher calls this; so can anyone with Docker
|
||||
# and the pinned toolchain. `frontend/src/lib/gates.contract.test.ts` holds its lines, so a lane
|
||||
# dropped or narrowed here fails a test like any other gate change.
|
||||
#
|
||||
# bash scripts/check.sh every lane (what a release requires, before e2e and the CVE scan)
|
||||
# bash scripts/check.sh lint the static lanes only: no tests, no build, no audits
|
||||
#
|
||||
# Each lane runs even when an earlier one failed, and the summary names every red one: a gate that
|
||||
# stops at the first failure hides the second until the first is fixed. The backend lanes run in the
|
||||
# test image (backend/Dockerfile.test): requirements.lock will not install on Windows (uvloop), and a
|
||||
# type checker or auditor without the real packages reports nonsense.
|
||||
set -u
|
||||
cd "$(dirname "$0")/.." || exit 2
|
||||
mode="${1:-all}"
|
||||
case "$mode" in all | lint) ;; *) echo "usage: scripts/check.sh [lint]" >&2; exit 2 ;; esac
|
||||
|
||||
export MSYS_NO_PATHCONV=1 # Git Bash: do not rewrite the in-container paths below
|
||||
TESTCOMPOSE=backend/docker-compose.test.yml
|
||||
failed=()
|
||||
lane() { # lane <name> <command…> — run it, remember a failure, never stop
|
||||
local name="$1"
|
||||
shift
|
||||
echo ">> $name…"
|
||||
"$@" || failed+=("$name")
|
||||
}
|
||||
|
||||
# --- static lanes ------------------------------------------------------------------------------------
|
||||
|
||||
# Host ruff, held to the pin in requirements-dev.txt: a newer ruff silently changes what "formatted"
|
||||
# means. `--config backend/ruff.toml` on both: it makes that file the WHOLE configuration (ruff then
|
||||
# ignores every other config it would discover) and gives scripts/ the backend's rule set.
|
||||
ruff_pinned() {
|
||||
local pin host
|
||||
pin="$(sed -n 's/^ruff==\([0-9.]*\).*/\1/p' backend/requirements-dev.txt)"
|
||||
host="$(python -m ruff --version 2>/dev/null | awk '{print $2}')"
|
||||
[ -n "$pin" ] && [ "$host" = "$pin" ] && return 0
|
||||
echo "ruff ${host:-missing} on the host, ${pin:-nothing} pinned — python -m pip install ruff==$pin" >&2
|
||||
return 1
|
||||
}
|
||||
ruff_check() { ruff_pinned && python -m ruff check --config backend/ruff.toml backend scripts; }
|
||||
ruff_format() { ruff_pinned && python -m ruff format --check --config backend/ruff.toml backend scripts; }
|
||||
|
||||
# New errors only: the pre-existing ones are in backend/.basedpyright/baseline.json. basedpyright
|
||||
# TIGHTENS that file by itself when errors go, and the tree is bind-mounted read-write, so a green run
|
||||
# can leave it rewritten: that fails here until it is committed, or the next run's stack reads stale.
|
||||
basedpyright() {
|
||||
local before
|
||||
before="$(git hash-object backend/.basedpyright/baseline.json)"
|
||||
docker compose -f "$TESTCOMPOSE" run --rm --no-deps --build test \
|
||||
python -m basedpyright --project pyrightconfig.json || return 1
|
||||
if [ "$(git hash-object backend/.basedpyright/baseline.json)" != "$before" ]; then
|
||||
echo "this run rewrote backend/.basedpyright/baseline.json (errors were fixed) — commit it" >&2
|
||||
return 1
|
||||
fi
|
||||
git diff HEAD --quiet -- backend/.basedpyright/baseline.json && return 0
|
||||
echo "backend/.basedpyright/baseline.json is uncommitted — commit it" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
frontend() { (cd frontend && "$@"); }
|
||||
|
||||
lane "ruff" ruff_check
|
||||
lane "ruff format --check" ruff_format
|
||||
lane "basedpyright (new errors only)" basedpyright
|
||||
# every frontend lane judges the HOST node_modules: this proves it is the tree the lock pins
|
||||
lane "lock check (frontend/node_modules == package-lock.json)" node scripts/check-node-modules.mjs
|
||||
lane "tsc (src + configs + e2e)" frontend npm run --silent typecheck
|
||||
lane "eslint (warnings fail too)" frontend npm run --silent lint
|
||||
lane "prettier --check" frontend npm run --silent format:check
|
||||
lane "knip (unused, and unused by what ships)" frontend npm run --silent knip
|
||||
|
||||
# --- tests, build, audits --------------------------------------------------------------------------
|
||||
|
||||
pytest_lane() { # the full suite against a throwaway Postgres; --pull keeps the base image current
|
||||
docker compose -f "$TESTCOMPOSE" build --pull -q test || return 1
|
||||
local rc=0
|
||||
docker compose -f "$TESTCOMPOSE" up --build --abort-on-container-exit --exit-code-from test || rc=1
|
||||
docker compose -f "$TESTCOMPOSE" down -v >/dev/null 2>&1
|
||||
return $rc
|
||||
}
|
||||
# Every pinned backend package against the advisory databases, at any severity — the lock is a full
|
||||
# `pip freeze`, so it is audited as written, without a resolver. An advisory with no fix is answered
|
||||
# with `--ignore-vuln <id>` here and a row in docs/gates.md (the register test reads this line).
|
||||
pip_audit() {
|
||||
docker compose -f "$TESTCOMPOSE" run --rm --no-deps test \
|
||||
pip-audit -r requirements.lock --no-deps --disable-pip
|
||||
}
|
||||
|
||||
if [ "$mode" = all ]; then
|
||||
# --coverage prints the summary: read, never a threshold (E32 D4)
|
||||
lane "vitest (+ coverage report)" frontend npm run --silent test -- --coverage
|
||||
# the production bundle, and devOnlyGuard, which fails it when dev-only code reaches it
|
||||
lane "vite build" frontend npx vite build --logLevel warn
|
||||
lane "pytest (pure-logic + DB lane)" pytest_lane
|
||||
lane "pip-audit (backend lock)" pip_audit
|
||||
# what ships to the browser, at high or above; the dev toolchain is burned down by hand
|
||||
lane "npm audit (production dependencies)" frontend npm audit --omit=dev --audit-level=high
|
||||
fi
|
||||
|
||||
if [ ${#failed[@]} -eq 0 ]; then
|
||||
echo ">> gate ($mode): green ✓"
|
||||
exit 0
|
||||
fi
|
||||
echo ">> gate ($mode): RED — ${failed[*]}" >&2
|
||||
exit 1
|
||||
Reference in New Issue
Block a user