- GC: the expiry warning marks each file conditionally on the expiry it saw, one commit each (a file
deleted meanwhile no longer fails the run; one re-timed keeps its fresh expiry unwarned); eviction
counts the disk again when a candidate was freed meanwhile, so it never evicts for space already free.
- Folders: a retention change locks the folder and its files, and a finishing file reads the retention
under a share lock, so neither misses the other; a folder delete locks the files it takes home (a
file moved out meanwhile stays put, one deleted meanwhile no longer fails the delete).
- Upload: the folder is checked before the probe and held only once the job names it.
- Dedup enqueue locks the cached asset, so a release meanwhile yields a fresh one instead of a 500.
- Mount check reads the kernel's mount table instead of comparing devices (the vfs storage driver gave
a mounted volume the device of /), and only inside a container; the API also logs a fresh, empty
volume in the library's place.
- Compose files fall back to DOWNLOAD_HOST_PATH; the upgrade notes cover the from-source pull.
- Folder routes call owned_or_404 directly.
- The queue keeps a done job by when it became done (ready_at), not by its last write: renaming,
keeping or moving an old file, or the GC's expiry warning, no longer brings it back as just finished.
- GC expiry and eviction re-read each file under the asset→job locks the user routes take, and commit
one file at a time: a delete, a re-download or a keep made while the GC runs is honoured, never a
second release that frees a file another user still holds.
- A file records the origin it was placed under (download_jobs.source_key, migration 0105): an edit
whose parent expired still goes back to its folder, not to Uploads.
- A folder a file is being put in is held (FOR SHARE) until it lands; a delete of it waits and takes
that file home too, instead of failing on the foreign key.
- The hash route reads at most two files at once (429 past that) and holds no connection meanwhile.
- 0104's backfill dates old files by their request date only: no notice prod wrote names its job.
- The old DOWNLOAD_ROOT still finds the files; a files root that is not a mounted volume stops the
worker and is logged by the API; the volume-copy recipe looks the project-prefixed names up.
- Stale comments about the /downloads mount and the Downloads page.
The Files library starts here. A user's every download, clip and upload now
lands in a source folder made per origin (YouTube, Plex clips, uploads, one per
other site), decided once in downloads/folders.py at every place a job is made.
How long a file stays moves from the shared asset to the user's job: stamped
from its folder when it is done, never for a file the user keeps. The GC warns
each user once, then takes the expired file out of their library; the cache cap
never evicts a kept file, and an evicted one leaves every holder's library.
The media mount is /files now (FILES_ROOT, FILES_HOST_PATH, volume
siftlode_files); docs/self-hosting.md says how to carry an old volume over.
Migration 0102 places every existing job and carries each done file's expiry
over from its asset.
safe_abs_path takes a missing path and answers None, which drops one baselined
type error.
The code cited its plan instead of saying why: `E28 review J4`, `(R6 S3b)`, `S1 justified…`,
`deep-review round 5`. That plan lives in the private ops repo, so for every reader of this one the
citation was noise, and a sentence whose subject was the plan step ("round 2 showed…") hid its
lesson behind an id. Each one is gone and the "why" kept: a bare citation is deleted, a sentence
carrying a lesson is reworded around it ("a later attempt showed…").
Measured (`scripts/audit_metrics.py`, product code): 652 lines in 270 files carried an epic code by
the definition the next commit holds (507 / 241 by the old, two-digit-only one) -> 0. Across all
code, tests, scripts and configs: ~1100 epic-code lines plus ~320 sprint/round ids.
Behaviour unchanged, checked mechanically: every changed .py file has the same AST once docstrings
are blanked; css/sh/toml/yml/json files are identical once comments are stripped; the only
non-comment changes in TS/JS are test titles, one description string in a contract table, and five
ESLint messages, whose text is all the gates snapshot diff holds. Not touched: Markdown (docs and
the changelog keep their history), Alembic migrations, TV episode notation (`S2 · E5`), DESIGN.md
ids (`C1`, `B4`, `S3h`), tracker ids (`SEC-16`, `DX-17`).
One stale claim fixed on the way: test_hls_reaper's docstring said the player checkpoints "every
10 s whether or not it is playing", which plex/stream.py calls wrong; it now says what the code does.
A deployment printed four different log formats and none of them could be
joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in
main.py for anything not started by uvicorn, logging.basicConfig() in the worker
and in the download child (no timestamp at all), and alembic's fileConfig.
app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log
data model, no SDK): ts, level, severity, source, logger, event code, message,
attributes, plus the correlation the context carries — request id, account,
download job.
Redaction is applied at the HANDLER, not at the call sites: every message, every
attribute and every traceback is scrubbed on the way out, so a secret cannot
leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies,
secret config values and email addresses never reach a line.
The access log is ours now (`http.request`): the route template instead of the
concrete path, the status, the duration and the account — and never the query
string, which uvicorn's access log printed on every single request (`?account=`,
`?source=`, the `?token=` of a share link). A well-formed client request id is
accepted and echoed in X-Request-Id, so a browser line and a server line join up.
The account comes from `request.state`, filled by the auth dependency: the
middleware is the outermost one, so it runs outside SessionMiddleware and in a
different task context.
`log_level` becomes an admin knob (sysconfig), applied without a restart and
re-applied at startup — best-effort on purpose, since a logging knob must never
be what stops the app from booting. Every compose service gets bounded json-file
logs (3×50 MB): Docker keeps every line forever by default, and the API now
writes one line per request.
Verified on the localdev stack: JSON everywhere including alembic and the
worker, the query strings gone, a supplied X-Request-Id echoed and logged, and
three /healthz hits of which only the one made while the level was DEBUG
produced a line.
DX-10: the repo invited people to clone, build and self-host while carrying
no LICENSE at all — which legally means all rights reserved, so none of that
was permitted. Adds the verbatim GNU AGPL-3.0 text, the SPDX id in
package.json and a README section. AGPL rather than MIT deliberately: this is
a server application, so the case worth covering is someone running a
modified copy as a network service.
DX-12: .env.example documented PLEX_MEDIA_HOST_PATH -> /plex-media and the
README advertised the Plex library, but only the localdev and home compose
files carried the mount — on both PUBLIC files the module could read Plex's
catalogue and then fail to open a single file, because it streams the media
itself rather than proxying Plex. Both api and worker now mount it read-only
with an empty named volume as the fallback, and self-hosting.md gains the
section that explains the server URL/token, the path map and the mount
together (it had never mentioned Plex).
DX-15: the worker had no init: true on either public file, though localdev
and home both carry tini with the reason written out — without PID 1 reaping,
a watchdog hard-kill of a download's process group orphans its ffmpeg/yt-dlp
grandchildren.
Verified: 'compose config' resolves on both files, the Plex mount comes out
read_only: true on api and worker, and init lands on worker (not just the
bgutil-pot sidecar). Gate green: vitest 671, pytest 896.
The worker and API start in parallel; the API applies migrations on its own startup, so on a fresh
deploy the worker hit a missing download_jobs table and crash-looped until migrations landed. The
worker now polls for the schema before recovering orphans, and (belt-and-suspenders) the compose
files gate it on the API being healthy. Fixes the observed prod restart-loop on the v0.22.0 deploy.
Prep the Download Center epic (Phase 1 + editor + share) for prod/self-host:
- Dockerfile: create /downloads owned by appuser so a named-volume mount is writable (prod Linux).
- docker-compose.{home,selfhost,yml}: add the 'worker' (yt-dlp/ffmpeg job loop) + 'bgutil-pot'
(PO-token) services + a downloads mount (DOWNLOAD_ROOT, WORKER_ENABLED). Media defaults to a
named volume; DOWNLOAD_HOST_PATH points it at a host dir (e.g. a Plex-readable folder).
- README / docs/self-hosting.md / .env.example / install.{sh,ps1}: document the Download Center,
the two extra containers, and DOWNLOAD_HOST_PATH.
- VERSION 0.22.0 + releaseNotes entry.
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
- docker-compose.selfhost.yml pulls the published image (forge.b1fr0st.eu/peter/siftlode) with a
bundled Postgres — no source build on the operator's host.
- install.sh / install.ps1 generate the four required secrets (.env: POSTGRES_PASSWORD, SECRET_KEY,
a valid Fernet TOKEN_ENCRYPTION_KEY, OAUTH_REDIRECT_URL), start the stack, and print the first-run
setup-wizard URL. Everything else is configured in the web wizard.
- docs/self-hosting.md: the copy-paste playbook (get the files, run the installer, finish in the
wizard, optional Google/SMTP, HTTPS/reverse-proxy, updates + backups).
- Verified end-to-end: a fresh install from the published image boots into setup mode and serves
the wizard.
peter
peter