ruff.toml ignored E402 everywhere for main.py's logging setup. Measured: 28 hits, of which 11
had nothing to do with it — runner.py's logger line between its imports, models.py importing
canonical_email mid-file, a test constant above its imports, and main.py's own third-party and
mimetypes imports. Those are moved up. What stays after configure_logging in main.py is marked
per statement (noqa E402, logging is configured first), its comment now says why the order
matters (a line logged while importing, not the logger's creation). The register row goes, the
ruff key whitelist loses ignore, and a late import is red again. The T20 comment counts the
third deliberate print (scripts/e2e_seed.py).
ruff gains RUF100, PLW1510 and flake8-bugbear, every hit fixed rather than
ignored. B008 stays on with FastAPI's Depends/Query declared immutable.
basedpyright moves to standard mode, pytest to --strict-markers, and a hand
run of pytest now requires the database unless SIFTLODE_TEST_REQUIRE_DB=0.
PLW1510 found a real bug: the three ffprobe probes in plex/stream.py never
looked at the exit status, so a probe that failed was read as the file's
truth (an HDR source sent down the HW-safe path, a multi-audio file collapsed
to one track, a subtitle offset taken from garbage) and the warning each
except branch promises never fired. They now run with check=True.
The views, duration and title sorts had no index: every page read the whole videos heap and kept a
top-N (prod 350 ms a page, 550-780 for title). Migration 0096 adds one per direction and, for title,
per reader collation, in the exact ORDER BY feedquery builds. Localdev prod copy: plain sorts
180-310 ms -> ~1 ms, page 20 3-5 ms, narrow filters unchanged. view_count/duration are only written
by the enrich, which already rewrites indexed columns, so no HOT update is lost.
The title indexes are the first on an ICU collation, and postgres:16-alpine floats: collation.heal_drift
rebuilds them (CONCURRENTLY) at startup when Postgres reports the ICU version moved.
test_feed_sort_indexes holds every videos sort to an index and checks the rebuild.
A second review round over the first round's fixes, which is where the remaining
real bugs usually are. All four are the same failure as the round it reviewed: a
claim stated in a comment that the code does not hold.
The middleware moved inside CORS, but the setup gate is registered LATER and is
therefore OUTSIDE it — so the comment's "the setup gate never runs for a body that
is already too large" was false the moment it was written. On an unconfigured
instance that gate opens a DB session per request, which is exactly the anonymous
pre-auth work SEC-20 exists to remove. The cap now sits between them, and the
order is asserted against `app.user_middleware` rather than described: prose about
a stack that `add_middleware` builds inside-out is how this got backwards.
The edge ceiling and the route's own cap were pinned EQUAL, but they measure
different things — the edge counts raw body bytes, the route counts the decoded
file. A file of exactly the advertised maximum was refused one layer before the
route that would have taken it. The edge now allows for the multipart envelope,
and the test asserts the relation and the gap instead of equality.
`sysconfig`'s new `max` was enforced on write only. That bound did not exist
before this epic, so a value that was legal when it was written can be sitting in
the database — served unclamped it produces exactly the drift the comment says is
impossible, with the admin page showing a number outside its own range. Int
overrides are clamped on read now, which also survives an env default being
lowered later.
And `DOWNLOAD_UPLOAD_MAX_BYTES=0` meant "no per-file cap" to the route but a hard
zero to the edge: every upload refused, including the admin page's own way back,
recoverable only by editing env and restarting. The setting takes `ge=1`.
Two test gaps from the same round: the header assertion that is the ONLY check of
the middleware's position was DB-gated and skipped on a host without Postgres, and
the headroom "ratio" the docblock calls load-bearing was held as `<=`, i.e. no
ratio at all. Both fixed, plus a sweep that every `capped_body` call site in the
routers is known and fits under its ceiling — the point of that list is the third
entry, not the two that exist today.
The module admitted its two opt-outs under a stated rule — "both enforce their own
limit on the bytes they actually read, which is what makes opting them out safe" —
and that rule is false for one of them. `POST /api/downloads/upload` declares
`file: UploadFile = File(...)`, so FastAPI parses and spools the whole multipart
part to a temp file BEFORE the first dependency runs, `require_human` included:
its in-handler cap measures bytes that have already landed. Measured on the
installed fastapi 0.141.1 / starlette 1.6.0 — starlette's max_part_size guards
non-file parts only, so an anonymous chunked upload was bounded by nothing. In the
shipped self-host compose that lands in a RAM-backed tmpfs.
The exposure predates SEC-20 (no route was capped before it); what this branch
added was the claim. So the list becomes a mapping of per-route CEILINGS instead
of a set of exemptions: every request now gets a finite number, the upload route's
being the same one the route itself enforces. `sysconfig`'s knob is bounded by it
so an admin cannot raise the handler's cap past the edge and start collecting 413s
from a layer the admin page never mentions — held by a test, not by prose.
Two more things the review caught in the same module:
The 413 was not byte-identical to `capped_body`'s after all. `json.dumps` puts
spaces where JSONResponse does not, and the test that promised to hold them
together compared PARSED JSON, so it could never have seen it. Built through the
same JSONResponse now, and compared as raw bytes.
And the middleware sat outside CORS and the security headers, so its 413 carried
neither — on the split-origin dev setup the browser drops it and the user gets a
generic network failure instead of the translated too_large, while the identical
refusal from capped_body one layer down renders fine. It moves to just inside
CORS: still ahead of the session, the setup gate and every auth dependency.
Also corrected: the docblock's "largest non-exempt body is 64 KiB, ~16x headroom"
(it is 512 KiB and 2x — the diagnostics sink, named eleven lines further down),
and CLAUDE.md's Request SIZE row, which forbade the one correct use of
capped_body that is actually in the tree.
FastAPI reads a declared body IN FULL before the first dependency, so
`POST /auth/password-login` drained an arbitrary upload from an anonymous
caller and only then answered 401. 70 of the 72 body-declaring routes
behaved the same way: one unauthenticated client could spend the
process's memory and time at will.
A raw ASGI middleware, not `@app.middleware("http")` — BaseHTTPMiddleware
cannot wrap `receive`, so it could only trust `Content-Length`, which a
client may omit (chunked) or understate. This refuses an oversized
declaration outright and counts the bytes of everything else as they
arrive, the same rule `capped_body` already applies per-route. Registered
directly inside the access log: a refusal is still logged with its
request id, but the session, CORS, the setup gate and every auth
dependency never run for a body that is already too large.
Two routes opt out, under one stated rule: an exemption is ONLY for a
route whose own cap is larger than the global one. That is why
`/api/diagnostics/events` is absent — it caps itself at 512 KiB, below
the ceiling, so exempting it could only widen what it accepts. The list
names route templates and a test resolves each against the live route
table, so a rename fails the gate instead of silently dropping an
exemption in prod.
The 413 is byte-identical to `capped_body`'s and the test compares the
two against each other rather than against a literal — two shapes for one
fact is how the `cancelled`/`canceled` bug happened.
A session is one recording, capped at 60 minutes by the server. While it is
open, a request naming the caller's own session in X-Debug-Session is logged
at the session's level with every line tagged; a header naming anything else
is ignored. The browser's batches count a session as consent to its level and
keep the tag only on its own events. Screenshots are judged by their bytes,
capped in size and count, and served back to their owner only.
- debug_sessions / debug_screenshots tables, one open session per user
(partial unique index), log_events.debug_session_id indexed
- /api/debug-sessions: start (joins a running one), stop, list, get, events,
screenshots; /api/me reports the live session
- the hourly log job closes expired sessions and drops them with the logs
- logsink.serialize and routes._common.capped_body shared by their readers
GET /api/admin/logs pages `log_events` newest first on a (ts, id) keyset:
a browser row's ts is clamped into its consent window, so ts alone is not
unique. Filters AND together: level, source, account (or `none`), event
code (or `none`), request id, job id, a free-text search over the
message, code, request id, stack and attributes (typed text is a literal),
and a time window. Each page names the accounts it mentions.
GET /api/admin/logs/facets counts what the filter chips offer over the
window alone, so a chip row does not shrink as filters are picked.
Admin-only and read-only; reading is not audited (the console's live tail
polls). There is deliberately no "ids after N" tail: several processes
write this table and a batch takes its ids at insert, not at commit, so a
tail keyed on ids could pass a row that commits late.
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.
What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
degraded to a software transcode with nothing saying why; a failed audio probe
collapsed a multi-audio file to one track and the player showed no language
picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
own root, leaving every crashed session's segments on disk forever.
Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.
ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.
logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.
tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
25 literal names covered 34 modules. plex/client.py, plex/stream.py,
plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves
`siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`;
routes/channels.py called itself `siftlode.api`. The `logger` column is in every
stored row so E22.6's console can filter on it, and so an operator can pin a level
to one part of the system — neither works when six modules share a name.
`logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync.
One name per module, and it cannot drift from the file it sits in.
tests/test_logger_names.py pins the derivation, that everything stays under the
one `siftlode` tree the handler and the tests attach to, and that no module
hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`).
Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic
it carries rather than the module it lives in — it replaced uvicorn's access log
and is what an operator pins when they want request lines and nothing else.
app/scheduler.py called its logger `logger`; every other module calls it `log`.
ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them:
worker.py logged `str(exc)[:120]` where the traceback was the whole question, and
three prints/f-strings that are deliberate now say so on the line. S110 and BLE001
stay off until S3 — 23 handlers are outstanding and each is a judgement about one
of them; enabling the rules first would just mean 23 blanket noqas.
The E22 redactor is a safety net over addresses and credential params, and every
one of these call sites was leaning on it to do the whole job. What it does not
cover went in whole, and what it does cover came out as a constant.
Not covered, so a real leak:
- routes/search.py logged the raw search term at ERROR. The privacy policy shipped
in 0.88.0 says a search string is kept only at Teljes/TRACE, with the user's own
consent. The ERROR line now carries the shape of the failure (source, candidate
and collected counts) and the term follows on a TRACE line, which nothing emits
unless that consent is in force.
- downloads/backfill.py logged the full source URL at INFO — a row of the user's
download history. Now the asset id and the host.
- youtube/client.py logged the upstream error body, which is prose that quotes the
request back. Every failing YouTube call now goes through one _upstream_failure()
that logs the API's `reason` code, keeps the body for the same consented TRACE
line, and raises a YouTubeError carrying `reason` as a field — callers put that
error in log lines of their own (routes/channels.py) and in a 422 answered to the
user, so the body must not travel in it either.
Covered, so merely useless: routes/auth.py named the account by an address that
reached both sinks as `[email]`, and app/email.py printed recipient lists that read
as `['[email]']`. Both now carry the uid (or a count) and a reason code, which is
also the first half of E22.5 S4's failed-sign-in coverage.
routes/channels.py branched on the text of that error message to recognise an
already-existing subscription. It now branches on `exc.reason`, with the substring
check kept beside it for a body that carried no reason at all.
tests/test_log_pii.py pins the call sites and the family: a source-level contract
test walks every log call in app/ and fails on an argument naming user-typed
content unless the line says why it is safe (`# log-pii-ok: <why>`). Eight sites
carry that marker today, each with its reason. Gate 8/8 green; all five cases
mutation-checked.
A user can raise their diagnostics level to Részletes (DEBUG) or Teljes (TRACE) for 24 hours from a
new Settings tab, after a consent box that says what is recorded, that the server keeps it and for
how long (the instance's own retention), and that it lapses. Alap is the base and needs no consent.
- users.diagnostics_level/_until (migration 0087), written only by PUT /api/me/diagnostics, which
computes the expiry itself; the demo account cannot raise it. /api/me reports the level in force.
- The server logs an elevated user's requests at that level and stores every line of them
(logs.LevelGate on the stdout handler and the sink; the root is lowered only while an elevation
is active). Teljes adds the redacted query string to the request line.
- POST /api/diagnostics/events stores the browser's log as source=web rows: account from the
session, body capped while streaming, events/attrs capped, re-redacted, events under the
consented level dropped, timestamps clamped, 30 batches a minute, 409 once the level lapsed.
- lib/diagnostics.ts applies the level to the tab and streams only what happens after consent, in
batches, never its own request lines, with a keepalive flush on pagehide; stops on lapse (its
own timer), on withdrawal and on 409.
- Detail hooks at DEBUG: which control was clicked (test id, icon, app copy in English — never user
text, input values or a link's query), visibility/bfcache/resize, filter changes
(lib/filterLog.ts; typed search only at TRACE), player events (lib/mediaLog.ts; media errors are
warnings at every level). Offline/online at the base level.
- "Saját napló" viewer: live, newest first, level groups with counts, search, row details, copy
the filtered events as JSON lines, clear.
- Privacy policy and Terms exist in Hungarian and English (a contract test holds both languages to
the same sections); the policy now describes the server logs and diagnostic logging, self-service
deletion and password sign-in.
Fixed on the way:
- A sync auth dependency ran logs.set_user on a copied context, so the account never reached a
handler's own log lines (measured); a per-request mutable scope carries it now.
- The Settings tab rail left a phone no room (clipped labels, buttons over text): below `sm` it is
a scrolling row above the content.
- PUT /api/me/preferences took a bare dict body; it is a Pydantic model now.
The log now has a stored, queryable copy for the diagnostics console:
every WARNING and above plus the request line of every request, kept
for log_retention_days (default 7, no "forever") and pruned hourly by
the new log_gc job.
Rows are built on the logging thread from the same redacted fields as
the stdout line, queued without blocking and written in batches by one
background thread on its own connection. A full queue or a failed write
drops and reports instead of retrying. Flood control keeps 30 of any
one message per minute plus a count of the rest; anonymous request
lines are capped at 60 a minute per client, and the client address is
never stored. Deleting an account deletes its stored events.
Measured on prod first: a YouTube RSS outage wrote 11k warnings in four
hours, one per channel per poll. The poll now writes one warning with
per-reason counts and a few examples, and a failure on our side gets
one traceback per poll instead of one per channel.
A deployment printed four different log formats and none of them could be
joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in
main.py for anything not started by uvicorn, logging.basicConfig() in the worker
and in the download child (no timestamp at all), and alembic's fileConfig.
app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log
data model, no SDK): ts, level, severity, source, logger, event code, message,
attributes, plus the correlation the context carries — request id, account,
download job.
Redaction is applied at the HANDLER, not at the call sites: every message, every
attribute and every traceback is scrubbed on the way out, so a secret cannot
leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies,
secret config values and email addresses never reach a line.
The access log is ours now (`http.request`): the route template instead of the
concrete path, the status, the duration and the account — and never the query
string, which uvicorn's access log printed on every single request (`?account=`,
`?source=`, the `?token=` of a share link). A well-formed client request id is
accepted and echoed in X-Request-Id, so a browser line and a server line join up.
The account comes from `request.state`, filled by the auth dependency: the
middleware is the outermost one, so it runs outside SessionMiddleware and in a
different task context.
`log_level` becomes an admin knob (sysconfig), applied without a restart and
re-applied at startup — best-effort on purpose, since a logging knob must never
be what stops the app from booting. Every compose service gets bounded json-file
logs (3×50 MB): Docker keeps every line forever by default, and the API now
writes one line per request.
Verified on the localdev stack: JSON everywhere including alembic and the
worker, the query strings gone, a supplied X-Request-Id echoed and logged, and
three /healthz hits of which only the one made while the level was DEBUG
produced a line.
Deep-review round 5, over round 4's own fixes.
- Sessions are keyed by rating_key alone while a session's DIRECTORY is per (key, offset, tag), and
the artifact URL carried no session identity — so once a seek-restart or a second viewer of the
same item took the registry slot, the old viewer's request for `seg_12.ts` was answered 200 with
the new session's twelfth segment: bytes from a different `-ss` offset under `-copyts`, on a
timeline the client is not on. Wrong content with a success status is the one failure a player
cannot notice, and round 4's reclaim-on-404 could never fire on it. The id is in the path now (so
the playlist's relative names resolve back into the same session), a mismatch is a 404, and the
player reclaims at the playhead. The entry playlist's auto-start is gone with it: it could only
ever begin at 0, silently discarding the viewer's position.
- Round 4 exempted paused viewers from the reaper by keeping EVERY finished session for the full
ten-minute idle window — one whole segmented copy of a film per item, on the volume the download
center shares. The signal it needed already existed: an open player checkpoints its position every
10 s whether or not it is playing, so that route now touches the session and the periodic pass is
back on the short rule.
- POST /session's opportunistic reap was never scoped to the key being started: it walked the whole
registry with the 30 s finished rule, so pressing play on ANY item — even a direct-play one, the
call ran before that early return — killed another viewer's paused session. It is gone; the
periodic pass owns reaping.
- A segment that can never arrive no longer polls out the full 20 s: the producer's handle is passed
to `wait_for`, so a finished ffmpeg answers at once. This is a sync route, so each such wait parked
a threadpool worker AND a pooled DB connection, and an ordinary seek past the produced region was
enough to do it.
- The DataError handler's re-raise branch — the half that keeps a genuine DataError a 500 — is
pinned, so simplifying it to an unconditional 422 cannot pass in silence.
Six regression tests (two route-level, in the DB lane), each mutation-checked red on the assertion
it is named after. pytest 1124, vitest 999.
Deep-review round 4, backend half.
- The periodic HLS reaper (round 3) applied the "finished and unfetched for 30 s" rule that only a
NEW session request may apply. A remux finishes long before the film, and a paused player with a
full buffer fetches nothing, so a one-minute pause killed the session and playback died ~30 s after
resuming. The periodic pass is `reap_abandoned` now (the long idle limit, finished or not).
- A subtitle `publish` that could not keep its output (full / read-only cache volume) returned
quietly, so every poll queued the window and the whole-container pass again. It reports failure
and the pass is backed off like a failed read; a cached VTT that cannot be READ is answered
`unavailable` (`cache_unreadable`) instead of scheduling reads that can never land.
- The subtitle routes bound `rating_key` / `sub_ord` (they become a cache file name: a 250-digit
one was ENAMETOOLONG, which pathlib's exists() re-raises -> a 500), and the cache helpers no longer
throw on such a name.
- One app-wide handler turns psycopg's "text fields cannot contain NUL" DataError into a 422: round 3
guarded only the by-rating_key lookup, and `contains=`, `contains_group=`, reorder/bulk key lists
and the collection search still 500'd.
- The hand-written `db.get(...) is None -> 404 "Unknown ..."` siblings use `get_or_404` with a
`what` (channel / video / user), so the Hungarian UI no longer shows English for a gone row.
Deep-review round 3 of bug hunt 2026-09.
- The YouTube and Plex progress bodies are bounded by POSITION_MAX_S (the int4 position columns): a
finite position of 2^31 or more was a DataError, i.e. a 500 — the sibling of round 2's non-finite
fix.
- plex/stream.py promised "a periodic reaper", but reap_idle ran only inside the NEXT session
request: after the last playback an abandoned session (its ffmpeg still working, its directory a
whole segmented copy of the film) lived until someone pressed play again. A lifespan task reaps
every 60 s.
- Tests for what round 2 left untested: the 422's body keeps the list shape the UI localizes, and
stream_session refuses non-finite start/aoff.
The JSON parser under FastAPI accepts `Infinity`/`NaN`, and both progress checkpoints truncate the
position with `int()` — a hand-written checkpoint came back as a 500 with a traceback. Their bodies
now refuse non-finite floats, and the app's validation handler — FastAPI's own shape — echoes a
non-finite input by name: the default one failed to serialise its own error (JSON has no NaN), which
was a 500 by another route. Sibling of the subtitle routes' non-finite query parameters.
Deep-review of the bug-hunt round (R1). The background extractor from ed4ab4e1 held, but its seams
did not:
- Window + full pass doubled every cue in the first hour. merge_vtt de-duplicated on the raw timing
text, and the two passes spell a timestamp differently (shift_vtt writes HH:MM:SS.mmm, ffmpeg's
webvtt muxer drops the hour below 1 h; shift_vtt(-0.0) never normalised the full pass). Timing is
now parsed and re-emitted canonically, keyed with the cue text so two cues sharing a time survive.
- A full pass that exited non-zero was published as complete: "ready" forever, subtitles ending
where ffmpeg died, and no eviction to ever retry. It now covers only up to its last cue, is never
complete, and records the failure memo.
- The window's dedup identity held the live playhead, so every 1 s poll queued another read of the
file. It is keyed on the track now, and a queued job re-checks coverage before reading anything.
- A sidecar track waited behind an unrelated embedded full pass (the full job is per item). Its
window job now fetches it whole — one small request.
- The <track> fetch ran the scheduler at position 0: an unrequested window over the head of the file,
and a refusal of a window that was on disk. It is a pure read of the cache now.
- publish() is serialised (a lost update became permanent once complete was set); the pool is shut
down and running ffmpeg children killed at app shutdown (they held interpreter exit for minutes);
?at=inf is a 422, not a 500; and both subtitle routes use current_user instead of the cookie-only
poster gate, which let a suspended or epoch-revoked session keep scheduling reads (pre-existing).
_extract_and_publish finally has tests of its own — every earlier test stubbed it out. 19 mutations,
each red on the assertion that names it.
E24 S1. The anti-paralysis answer layer, served from our own Postgres with no
upstream call: one hero suggestion, one CROSS-SOURCE resume rail, and the user's
own comfort shelf.
Survey first, because the roadmap line was wrong about the starting point: Plex
already ships the whole trio (E5's Discover rails, including a reshuffling
Surprise me), the YouTube side ships none of it, and nothing anywhere answers
the question across both. The data already existed on both sides — VideoState
and PlexState are structurally identical, the feed has an `in_progress` filter
and a seeded shuffle, and the YT player already resumes position.
- app/discover/budget.py — the time bands. "How much time do you have"
(10 perc / 30 perc / egy este / mindegy) replaces 15/D's mood questionnaire:
deterministic, answerable from a column we have, and it picks the source for
free. Matched against REMAINING time, so a film with 8 minutes left is a
ten-minute answer. An unknown runtime fits only the unbounded band.
- app/discover/cards.py — a thin unified card that carries the NATIVE feed /
Plex payload, so the page renders with the existing card components instead of
a third one. `aspect` is the one new field: a rail needs to know poster from
16:9, and that is a property of the artwork, not of the module.
- app/discover/queries.py — the SQL, reusing feedquery and plex.rollup rather
than the Plex Discover ROUTE, whose LibraryFilters narrowing belongs to a
surface that has a sidebar. One seed drives both halves so a reroll moves
them together.
- app/discover/hero.py — resume -> comfort -> surprise, with the chosen tier in
the payload so the client labels it honestly instead of guessing.
- comfort_pins (0083): five nullable CASCADE-ing FKs under num_nonnulls(...) = 1
rather than a flag or a polymorphic string ref. A flag cannot order a shelf or
hold a channel; a string ref would let a deleted library item leave a dead tile
on the one surface whose whole job is to be reliable.
Two honest-state rules the tests pin: the surprise draw excludes anything
already started (the resume tier may have skipped it for not fitting, and
offering it back as "Meglepetés" would contradict the question), and which
source leads the draw follows the seed, so the Plex library can actually answer
"egy este".
Gate green: ruff, ruff format, 3x tsc, eslint, prettier, knip, vitest 770,
pytest 987.
Review round on the whole E1 stack (main..HEAD). 6 findings, all low/medium,
none blocking; fixed for a clean gate-green handoff.
- plex/sync (correctness): the un-enriched cast guard compared the stored cast
against THIS listing's length, so a trimmed Plex cast on a not-yet-enriched
row was never applied. Compare against the listing CAP (_LISTING_CAST_CAP=20)
instead: >cap = enriched (keep, survives a watermark reset), <=cap =
listing-sourced (track edits in both directions). +2 regression tests.
- main (security): add X-Frame-Options: DENY as the legacy-browser fallback for
the CSP frame-ancestors clickjacking guard.
- downloads/service (altitude): finalize_if_running now uses a SQLAlchemy 2.0
update() construct instead of a hand-built text() UPDATE string — same atomic
WHERE status='running', idiomatic, no identifier interpolation, no manual
StrEnum coercion. Drops the now-unused text import.
- auth/admin (reuse): extract the SEC-10 "drop untrusted preset password + bump
epoch" invariant into one helper (revoke_preset_password_if_unverified) so the
Google-callback and admin-activation paths can't drift.
- tests/migrations (robustness): guarantee the shared schema is restored to head
in a finally, so a downgrade/re-upgrade failure can't cascade into later tests.
- worker (simplification): align _finish_from_cache's display_name fill with the
_download path's guard (no behavior change).
Gate green: ruff clean, 334 pytest (0 skip), frontend unchanged.
A small, safe set applied to every response via middleware: X-Content-Type-Options
nosniff, Referrer-Policy strict-origin-when-cross-origin, CSP frame-ancestors
'none' (anti-clickjacking; unrelated to the YouTube/Plex embeds we host), and
Permissions-Policy disabling camera/mic/geolocation. A full CSP and HSTS are
deferred (HSTS belongs on the Caddy edge).
R12 S2. app/auth.py (1154 lines) mixed the core auth surface with 18 HTTP
endpoints on one /auth router. Split by dependency graph:
- app/auth.py (223 lines) keeps ONLY the core surface other modules import
(current_user, resolved_user_id, admin_user, require_*, has_*_scope,
purge_user, count_admins, get_or_create_demo_user, google_enabled,
is_allowed, operator_contact, optional_current_user) plus everything it
transitively depends on.
- app/routes/auth.py (883 lines, new) gets the router + all 18 endpoints +
their endpoint-only helpers, importing the core surface from app.auth.
- main.py includes auth_routes.router (was app.auth.router).
- test_oauth_redirect imports _oauth_redirect_uri from its new home.
The STAY/MOVE partition is driven by an AST scan of the COMPLETE external
surface (every `from app.auth import` incl. multiline + `from app import
auth; auth.X`), so no still-referenced symbol was moved. Behaviour-equivalent:
every symbol relocated verbatim (get_source_segment equality), route set
proven identical, no undefined names (symtable sweep).
Gate: ruff clean (backend + scripts), pytest 193 passed.
The worst one was, again, the previous round's own fix.
- Round 7 stopped awaiting the HLS sweep so startup could serve immediately — which
also made it run CONCURRENTLY with playback, and `sweep_orphans` snapshotted the
live-session set BEFORE listing the root and rmtree'd outside the lock. Since
`start_session` reuses a directory path for the same (key, offset, tag), a stale
name can become a live session at any moment. Two guards now: nothing whose mtime
is newer than this process's start is ever deleted, and the live check + rmtree
happen together under the lock, one directory at a time.
- The sweep task is awaited after cancel, so shutdown can't leave it pending.
- `_rate_limiter` used a truthiness test on its timestamp, so a `now()` of exactly
0.0 read as "never ran" — unreachable in production, but `now` exists to be
injected and a test clock starting at 0 is the obvious choice.
- The RSS total-outage error now carries the first underlying error: "all N channels
failed" is also what ONE dead feed id looks like on a small instance, and a red
card the user can't act on is its own kind of dishonest.
- The WebKit fullscreen handling moved into `lib/fullscreen` and PlexPlayer uses it
too — it had the same unprefixed-only reads in three places, including an Escape
branch that would have closed the whole player instead of leaving fullscreen.
- The `--max` CSS comment claimed `inset: 0` did the sizing; Tailwind's `w-full` is
still on the element and wins. Comment now states the real mechanism.
Verified in real Chrome (the pane can't composite or do native fullscreen):
- F fills the 1920x889 viewport; the exit pill renders top-right on the letterbox
band and overlaps NO YouTube control — the collision this round suspected does not
reproduce in this embed (its own buttons sit bottom-left / bottom-right).
- Four real ArrowDown presses while maximised: persisted volume 100 -> 80, with the
level flash visible on screen.
- A real click on the exit pill un-maximises and returns focus to the card.
- lib/fullscreen driven from a real click: enter (innerHeight 889 -> 1024), the
change listener fires exactly once per transition, exit returns to 889.
Backend suite 150 -> 158; the new sweep tests are mutation-checked, and they read
their cutoff from the filesystem rather than a clock (the container VM and the
bind-mounted host disagree by enough to make a clock-based assertion flaky).
Five of them sit where round 6 turned F from a Fullscreen API call into a CSS
maximise: the guards around it still asked `isFullscreen`, which that pivot made
permanently false.
- Arrow-key volume was dead in the maximised view: the scroll-deference checked
only `isFullscreenRef`, so the (now hidden but still scrollable) card kept
swallowing Up/Down. Both flags now count as "the video fills the screen".
Measured: 87px of scroll room on the focused card, ArrowDown moves 80 -> 75
and scrolls nothing.
- The maximised view had no way out once the pointer touched the video. Reaching
YouTube's controls means clicking into the cross-origin iframe, which takes
keyboard focus with it, and the re-arm paths (stage mouseleave, window focus)
need the pointer to leave the browser entirely while the stage fills it. So
the exit now lives on screen, in our DOM: a button that un-maximises AND pulls
focus back, re-arming every shortcut.
- WebKit fires only `webkitfullscreenchange`/`webkitFullscreenElement`, so on
Safari/iPadOS YouTube's own fullscreen never registered — no overlay yield, no
focus reclaim. One `fullscreenEl()` reader, both event spellings.
- The stage claimed `z-index: 9999`, the glass tuner's reserved level, for a job
that only needs to beat the modal chrome it is nested in (its siblings top out
at z-menu). Dropped to rail-level; nothing escapes the modal's stacking
context either way.
- `inset: 0` already sizes a fixed layer — the inherited `100vw`/`100vh`
over-constrained it, which is how a classic scrollbar gets a horizontal
overflow and a mobile URL bar hides the bottom edge (YouTube's control bar).
- The shortcut hint still promised "F: fullscreen" in both locales.
Backend, both "the fix stopped one layer short":
- The startup HLS sweep was awaited inside lifespan, so uvicorn still served
nothing until it finished — a thread doesn't help when nothing is listening
yet. Fire-and-forget task instead, cancelled on shutdown. Measured with 200
planted segments: "Application startup complete" now precedes "swept 1".
- An RSS poll where every channel failed returned normally, so the scheduler
recorded `ok` with the outage buried in the summary string. `failed == total`
now raises: a run that reached nothing is a failed run.
Suites: backend 147 -> 150.
- HLS sweep only deletes directories matching the session-name shape; the root
is admin-configurable and may be a shared path
- playlist unwrap picks the first entry that actually downloaded, not entries[0]
- concat quoting keeps backslashes literal (ffmpeg treats them so inside '')
- ArrowUp/Down defer while the focused element can still scroll that way
- the staging fallback skips yt-dlp working files (.part-Frag/.ytdl/.temp/.fNNN)
- the boot HLS sweep runs off the event loop
- volume persistence is debounced (was a setItem per wheel notch) and a spin no
longer loses notches to the iframe's lagging getVolume
- _Breaker.run owns the whole open/call/record protocol; four call sites shrink
- should_prune extracted and unit-tested, plus RSS error-vs-empty tests
- the worker exits non-zero when the schema never lands
- pick the produced file safely: Path("") is Path("."), which passes .exists(),
so a missing yt-dlp filepath skipped the staging fallback and crashed in
with_name ("PosixPath('.') has an empty name") on non-YouTube sources
- unwrap a playlist/multi_video extraction to the entry that carries the media
- run_ffmpeg: drain stderr while it runs (a full 64KB pipe blocked ffmpeg
forever and hung the worker thread), 1s cancel polling, stall watchdog and a
bounded wait on exit
- escape apostrophes/backslashes in the ffmpeg concat list (pre-0.51.0 names)
- sweep orphaned HLS segment dirs at startup (nothing else ever reaped them)
A shared /watch/{token} link is a client-rendered SPA, so a social crawler only
saw the generic index.html — a blank link card. The server now injects per-video
Open Graph / Twitter tags (title, channel, thumbnail) into the served HTML for
that route, so links unfurl richly in Messenger and other chat apps; real
browsers ignore the extra tags and hydrate the page as usual. Password /
expired / invalid links fall back to the generic card with no metadata leak.
Also shortens the generic site description used for search engines and link
previews.
The 3 landing screenshots were 2.3 MB of PNG (feed.png alone 1.6 MB, decoded
at 2400x1350 for a ~400px slot) — the biggest LCP/transfer cost on the public
page. Re-encoded to WebP capped at 1600px (~284 KB total, feed 91% smaller).
Also set Cache-Control: content-hashed /assets/* are immutable for a year,
other SPA-root static files (welcome images, favicon, robots) get a 7-day TTL;
index.html stays no-cache. Register image/webp+avif mimetypes so FileResponse
serves the right Content-Type. Lighthouse (dev): Perf 80->93, LCP 1.8s->1.2s.
Share a download by a capability URL (/watch/{token}) that anyone can play on a login-free page —
distinct from the registered-user ACL share. Per-link controls: optional expiry, allow-download
toggle (stream-only vs downloadable), optional argon2 password. Revoke = delete.
- migration 0042 + DownloadLink model; app/downloads/links.py (token, HMAC grant sign/verify)
- owner endpoints (POST/GET/PATCH/DELETE links) + /recipients for the internal user picker
- public router (no auth): watch meta / password unlock→signed grant / range-aware file serve
(inline vs attachment); unlock is rate-limited; meta verifies the file exists on disk
Verified end-to-end: 206 range, inline vs attachment, wrong-password 403, tampered-grant 403.
Two tabs in one browser can now run two different signed-in accounts at once.
- The signed session cookie stays the browser's account WALLET (account_ids). Which account a
given tab acts as is a per-tab choice held in sessionStorage and sent per-request via the
X-Siftlode-Account header; current_user honours it only for an account already in the wallet,
without mutating the cookie's default account. Switching accounts sets the header + reloads
THIS tab only, instead of the old cookie-wide switch that changed every tab.
- WebSocket can't send headers, so the per-tab account rides in the ?account= query param
(validated against the wallet).
- Logout is per-tab aware: it signs the requesting tab's active account out of the wallet
(promoting a new default only if the removed one was the default), and the tab drops its
override. A stale per-tab header account 401s just that tab instead of clearing the session.
- Serve index.html with Cache-Control: no-cache so a deploy's new hashed bundle is picked up
immediately instead of the browser running a heuristically-cached stale index.html.
Per-user named snapshots of the feed's FeedFilters. New saved_views table
(migration 0035, unique per user+name, position, is_default) + SavedView
model, and a /api/saved-views router (list/create/patch/delete/reorder)
gated behind require_human so the shared demo account can't pollute it.
Setting is_default clears the flag on the user's other views (one default).
Add a live YouTube search that materialises results into the shared catalog so
they render with the normal feed cards + in-app player and gain per-user state.
- YouTubeClient.search_videos(): search.list (100 units), embeddable-only, returns
flat stubs + nextPageToken; surfaces liveBroadcastContent for live filtering.
- routes/search.py GET /api/search/youtube: require_human + per-user daily cap
(search_daily_limit_per_user, default 70) + can_spend pre-check (429 on either);
drops live/upcoming, upserts channel stubs (channels.list) + video stubs, enriches
(videos.list), runs the youtube.com/shorts probe, then excludes Shorts/live and
returns feed cards in relevance order with the YouTube pageToken as the cursor.
- Provenance: videos.via_search / channels.from_search (migration 0028) flag
search-discovered rows; the feed hides them from the Library (scope=all) by default
via exclude_search_discovered, leaving the Mine feed untouched.
- quota.actions_today() counts a user's per-action events today for the cap; only the
search.list call is attributed VIDEOS_SEARCH so the counter is exactly 1 per search.
Private user-to-user messages are end-to-end encrypted: the server only ever
stores ciphertext + iv and acts as a key directory (public keys) plus an opaque
store for each user's private key, wrapped client-side with a passphrase the
server never sees — so not even an admin can read a conversation. A separate
kind=system message (plaintext, no sender) powers a server-authored Siftlode
welcome shown on first open, reusable later for announcements.
- models: rework Message (kind, nullable sender/body, ciphertext+iv) + MessageKey;
migrations 0026 (table) + 0027 (E2EE rework).
- routes/messages.py: key directory/blob endpoints, ciphertext send, conversations
+ threads (system + user), lazy welcome, all gated by require_human.
- realtime.py: in-process WebSocket connection registry; /ws delivers sent
messages to a user's open tabs instantly (sync-callable push, single-process).
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
- app_state gains configured + setup_token_hash (migration 0025); existing installs (any users)
are marked configured so they never see the wizard, a fresh DB starts in setup mode.
- On first boot the app mints a one-time setup token and logs the wizard URL (only the hash is
stored); the token gates the setup steps (added in 6c).
- A middleware locks all /api + /auth routes (except /api/setup, /api/version, /healthz, static)
to 503 until configured, with a DB-free cached fast-path once setup completes.
- GET /api/setup/status tells the SPA whether to show the wizard.
- Serve real files at the SPA root (e.g. /welcome/*.png, favicon) from the built app — the
landing screenshots were only ever shown by the Vite dev server, never in production.
- Add the Channel-manager screenshot; the two secondary previews are smaller thumbnails that
open in a custom full-size lightbox (fit-to-viewport, Esc/backdrop/✕ to close).
- Drop a suspended/deleted user to the login page on the next 401 (and poll the session only
while signed in, so the public login page no longer flickers); 'suspended' and 'account
deleted' confirmation banners; strip redirect query params for a clean address bar.
Add a generic admin-editable config layer (epic 4a): system_config KV table
(migration 0021), a sysconfig registry that is the single source of truth for
DB-overridable keys (type/group/default/bounds/secret) + a DB-override-or-env
resolver, and admin endpoints (GET/PATCH/DELETE /api/admin/config + a test-email
probe). Secrets are Fernet-encrypted at rest (TOKEN_ENCRYPTION_KEY); without it
they stay env-only. First group wired end-to-end: Email/SMTP — email.py now
reads host/port/user/from/password via the resolver (own session, since email
is sent from background tasks).
Add a server-backed notification center that coexists with the client-side
transient bell: a per-user `notifications` table (type/title/body/data JSON/
read/dismissed), a `/api/me/notifications` CRUD API (list, unread_count, read,
read_all, dismiss, clear), and a left-nav inbox module with a live unread badge
polled via useLiveQuery. Known types render trilingual text from type+data
(English stored text is the fallback); read rows are trimmed past a soft cap.
Also adds the schema the maintenance job builds on: videos.list?part=status
columns (embeddable/privacy_status/upload_status) and the validation lifecycle
columns (last_checked_at, unavailable_since, unavailable_reason).
Page-id validation is centralized in one PAGES source of truth (isPage) so the
new page survives reload without a second allowlist to keep in sync.
Record each scheduler job's run activity in-process (running / last result /
error / timestamps) and expose a snapshot. New admin-only GET
/api/admin/scheduler returns the per-job activity (with APScheduler next-run
times), the DB-derived work still queued (channels/videos pending, deep ETA,
live-refresh backlog) and the shared quota picture.
Add per-user local playlists: Playlist + PlaylistItem models (with forward-looking
kind/source/yt_playlist_id/dirty columns for the later YouTube-sync phases) and
migration 0011. New /api/playlists routes: list (with optional contains=<video_id>
membership flags for the add-to-playlist popover), create, get detail (items
serialized via the feed serializer, with per-user watch state), rename, delete,
add/remove item, and reorder. Watch later (built-in) is protected from deletion.
Add a VERSION file (0.1.0) and inject APP_VERSION/GIT_SHA/BUILD_DATE as Docker
build-args (both stages; Vite inlines them into the SPA). New public GET
/api/version returns app_version, git_sha, build_date and the Alembic head as the
database revision. deploy.sh and the localdev build pass the args.
Base login now requests only openid/email/profile (non-sensitive), so a new user
gets a clean Google consent with no "unverified app" warning and no 7-day refresh
token expiry. YouTube read (youtube.readonly) and write (youtube) are granted later
by the onboarding wizard via a parameterized /auth/upgrade?access=read|write.
Security fixes folded in from the baseline audit:
- config: refuse to boot in production (https OAUTH_REDIRECT_URL) with the
placeholder/short SECRET_KEY or a missing TOKEN_ENCRYPTION_KEY, closing a
session-forgery / admin-impersonation hole.
- main: mark the session cookie Secure when served over HTTPS.
- me: expose can_read; sync/subscriptions returns a friendly 403 (not a 500)
until YouTube read access is granted.
Track who burned how much YouTube API quota. A QuotaEvent audit log (migration
0009) records every spend with the triggering user (NULL = background/system) and
an action label, set via a request/job-scoped contextvar (quota.attribute) so no
call signatures change. User-initiated work (sync subscriptions, unsubscribe,
opt-in recent backfill, manual enrich) attributes to the user; scheduler work to
System, split by action.
- backend: QuotaEvent model + migration 0009; quota.attribute() contextvar;
record_usage logs events; entry points wrapped (routes/sync, routes/channels,
scheduler); GET /api/quota/my-usage + GET /api/quota/admin
- frontend: admin-only Stats page (header nav, page=stats) with daily bars +
per-user breakdown by action and range picker; 'Your API usage' in Settings ->
Sync for every user
Verified: attribution + endpoints compute correctly; events are per-user vs System.
Move the access whitelist from the ALLOWED_EMAILS env var into a DB Invite table
(env kept as bootstrap fallback), and add a self-service request + admin approval
flow with fail-soft email.
- models: Invite(email, status pending|approved|denied, requested_at, decided_*)
- migration 0008: invites table; seed env ALLOWED_EMAILS u ADMIN_EMAILS as approved
- auth: is_allowed() (DB-first, env fallback); a denied Google login records a pending
request and bounces to /?access=requested instead of a raw 403; public POST
/auth/request-access; upsert is idempotent so repeats don't re-spam admins
- routes/admin.py (admin-only): list/approve/deny invites + manual add
- email.py: smtplib + Gmail App Password, fail-soft (skips if SMTP unset)
- /api/me exposes pending_invites; config + .env.example gain SMTP_*
- UI: Login 'Request access' form + access=requested/denied handling; Settings ->
Access requests (approve/deny + add); admin nudge toast on pending requests
Verified locally: request-access creates a pending invite and emails the admin;
seed approved npeter83; guinea-pig yt.trash2023 denied until approved.
peter