E2EE direct messages, the chat dock, the Messages module, the message
WebSocket and its realtime registry, and the messages / message_keys
tables (migration 0097; the downgrade restores the empty schema). Prod
held two welcome notes, no message between people and one key — never
used (owner decision, E34). WebSocket now has no owner module: the lint
bans it everywhere. Privacy policy updated: it no longer promises E2EE
for a feature that is gone.
POST /auth/request-access had no caller left: the login page's form was
gone, and registration and a refused Google sign-in already land in the
same admin approval. A third public write path with no protection of
its own (owner decision, E34).
Its one guard moves to where every path passes: upsert_pending_invite no
longer opens a pending row for an address the env already lets in. The
registration path lacked it — registering b.ob@gmail.com beside an
env-granted bob@gmail.com opened a pending row, and is_allowed answers
from the invite rows once any exist, so the real owner was locked out.
The tests that drove the endpoint now drive registration or the shared
function.
It always said a live search spends the shared API quota, while the default source (the website's
own endpoint, scrape) spends none — the owner avoided a free search because of it. /api/me now names
the running search_source, and the tooltip names the cost under it: none for scrape (the enrich step
costs a unit or two), 100 units a page for the official API.
quota_events had no retention: one row per API spend plus a zero-unit row per live search, kept
forever. The daily history cleanup (audit_gc) now also prunes it past quota_retention_days (a new
Quota setting, default 365 so the admin usage view's longest window stays whole; 0 = forever). The
floor of two days keeps today's rows, which the per-user search cap counts.
The admin page's download_worker_concurrency and log_level were stored but never reached the worker:
it read the env once at boot (worker.py) and never looked again. The worker now re-reads both every
10 s (_sync_knobs): extra loop threads start at once, surplus ones retire after their current job, and
a changed level is applied to the process. An unreadable database keeps what already runs.
YouTube answers the uploads list of a channel with no uploads as
playlistNotFound. The recent and deep backfills raised on it, so the channel
was never stamped and was picked again, charged a quota unit and logged an
ERROR on every run. Both now read it as an empty last page, as explore already
does; any other failure still raises.
A mirrored library that answers "0 items" is not believed (its rows stay), but
that verdict stopped the whole reconcile: one library emptied on Plex with its
mirrored items kept made every run incomplete, and un-watches in every other
library never synced again. Each section now settles on its own walk, and only
the settled sections' rows are cleared; the unsettled ones are named in the
warning.
A library left out of the allowlist is never scanned, so its watch states were cleared on
every reconcile; clearing is now limited to the sections walked. A section we mirror items
from that says it holds 0 is not believed complete. The page size is no longer read as the
section total when totalSize is missing (measured: the server always sends totalSize on a
paged request, 0 for an empty one).
playlistNotFound was mapped to "gone" everywhere: exploring a channel without uploads (its
uploads list answers not found) and a list deleted mid bulk pull both said a playlist no
longer exists. The reason is now "gone" only on single-playlist routes; explore reads it
as an empty back-catalogue, and the bulk pull skips that list and syncs the rest.
ruff.toml ignored E402 everywhere for main.py's logging setup. Measured: 28 hits, of which 11
had nothing to do with it — runner.py's logger line between its imports, models.py importing
canonical_email mid-file, a test constant above its imports, and main.py's own third-party and
mimetypes imports. Those are moved up. What stays after configure_logging in main.py is marked
per statement (noqa E402, logging is configured first), its comment now says why the order
matters (a line logged while importing, not the logger's creation). The register row goes, the
ruff key whitelist loses ignore, and a late import is red again. The T20 comment counts the
third deliberate print (scripts/e2e_seed.py).
E32 review round 1 (B7). Each phase took its own now; phase 2's "not checked this run" test then
let a just-recovered video through and spent another videos.list call on it.
E32 review round 1 (B5, B6).
- The autotag orphan cleanup reads every user's feed views. A view whose tag pick was not a list
("tags": 5) raised for everyone's job, True passed as tag id 1, and ~66k ids pushed the NOT IN
past psycopg's 65535 bind parameters. The views are now read defensively and the kept ids
filtered in Python; the saved-views route holds filters.tags to a list of at most 500 ids, on
create and on update.
- tags_by_category resolved any tag id, so another user's private tag landed in its real category
and the feed summary answered differently than for an id that does not exist, which told the
caller which foreign ids exist. It is owner-scoped now (pre-existing; S2's missing-tag branch
made the difference visible).
E32 review round 1 (B1, B2, B3, B8).
- Plex watch reconcile: a section the server SAYS is empty (totalSize 0, nothing listed) is a
complete listing. It read as "cut short" forever, so one empty library stopped un-watches from
ever syncing. A listing with no total at all stays incomplete (the client now returns None for
it instead of 0), and a whole scan that listed nothing still clears nothing. Tests now cover
completeness across two sections, both orders.
- Plex collections: no members from a collection whose listing counts some (childCount) keeps
the membership we hold, in the full sync and in the targeted resync after a write.
- YouTube playlists: no items for a playlist whose listing counts some (itemCount) keeps the
mirror in the bulk pull, and is refused as retryable in "reset to YouTube" and in the push
plan/merge, where it read as "YouTube deleted them all". The rule sits beside vanished().
- A playlist deleted on YouTube answered "reconnect your account"; it now says it is gone
(errors.youtube.gone, HU+EN).
- The pure 3-way merge moved to sync/playlist_merge.py; playlists.py is back under 800 lines and
off the size list.
The missing identifier became server:/// in the write URI. Client tests over a MockTransport
(errors, paging, cutoff, write URIs, image transcode) and admin-route tests (connection test,
enabled-section sampling, no server URL in a 422, the one-time watch import).
A failed config fetch or a keyless consent page cached the fallback InnerTube key for the life
of the process; once it went stale, search returned nothing with no error. The fallback is now
used for that call only. A 200 JSON without a result frame returned an empty page that read as
no results; it is a ScrapeError now. Parser, cursor and Shorts-probe tests added.
A saved view or the stored filter outlives its tag. feedquery dropped the unknown id, so the
view showed the whole feed under its old name. The id now lands in a MISSING_TAG_CATEGORY that
filters to nothing in the SQL and in feed_summary's mirror alike; the feed chip reads "Deleted
tag" (a placeholder while the tag list loads, the id if it failed). Channels' persisted tag
filter gets the same chip, first in the row so the +N overflow cannot fold it away.
The YouTube playlist guard from the previous commit put sync/playlists.py over its size ceiling;
the rule now lives beside should_prune as vanished(existing, listed), which the mirror calls in
place of its own list. The pyright baseline also drops by one (187), from the plex/overlay split.
Feed views store tag ids. A system tag no channel carried for a while was deleted and, once a
channel earned it again, recreated under a new id — the view's filter then pointed at nothing, the
feed dropped the unknown id, and the view showed the whole feed under its old name. The cleanup now
skips tags a saved feed view references (test seen red without it).
The catalog prune and the subscription import have long refused to reconcile against an empty
listing that contradicts what we hold (should_prune: a 200 with no items is a blip far more often
than a user who removed everything). Two mirrors did not: the YouTube playlist mirrors (dropped
every mirror) and the Plex collections (dropped every collection with its user-set `editable`
flag, and every member's keys — the guard landed in plex/sync.py with the previous commit). Both
now hold the same rule; test_empty_listing_guards covers each way (seen red).
sync/maintenance (19% covered) gets a lifecycle test: recover, grace, delete + one notification
per user, live stream kept, abandoned premiere. It found the rolling revalidation re-fetching the
same videos until `checked` reached the batch (15000 by default) whenever fewer videos were
available — quota spent on answers it already had; a run now skips what it has checked.
The daily watch reconcile reads "absent from Plex's scan" as "Plex un-watched it" and deletes the
row — but it walked the sections without asking _paginate whether the walk reached the server's
total, so a Plex hiccup mid-walk (an empty page early) un-watched every title it failed to reach
until the next complete run. The scan now reports completeness; an incomplete one still pulls in
what it saw, re-pushes dirty rows, and clears nothing (logged as plex.watch.reconcile_incomplete).
The HU overlay had the same shape: clear every row's HU text, then refill from the listing. It now
reads the listing first and clears only after a complete one. The overlay moved to plex/overlay.py
(plex/sync.py drops under the 800-line limit and off the size ratchet's list).
Tests: plex/paths (23% -> 100%: prefix boundary, longest prefix, dead mount), the reconcile's
clear/keep/re-push rules, the cut-short scan and listing (each seen red with the guard removed).
The 7 baselined Optional accesses in plex/stream.py were one fact the type did not carry: the
initial spawn (replace_only=None) always supersedes, so only the fallback respawn can get None.
An assert at the call site says so; the pyright baseline drops 195 -> 188.
test_abuse_limits: the public-URL and YouTube enqueue tests stopped at the profile lookup
(400 no_profile) and passed on that unrelated refusal. They now send an inline spec and assert
the job is queued. The same shape — a request whose response is discarded — sat at 53 more
places across the suite (all returned what they should, measured with a TestClient probe);
each now asserts its status, and test_response_checked keeps it that way (seen red).
docs/gates.md lists every place a gate deliberately checks less than it
could, each with its reason and the count measured when the row was
written:
- an ESLint rule off, at warn, or not applied, by scope;
- a weak compiler flag, a ruff ignore, the basedpyright mode and baseline;
- a knip exclusion, a lane that does not run everywhere;
- an accepted CVE.
gates.contract.test.ts reads the EFFECTIVE configuration and fails when a
narrowing has no row:
- ESLint's computed config per scope;
- the three tsconfigs, ruff.toml, pyrightconfig, pytest.ini and knip.jsonc;
- .trivyignore.yaml, where an entry also needs a statement and an expiry;
- CI's pip-audit ignores.
It also pins `--max-warnings 0`, which is what makes a warn-level rule fail.
Seen red on a missing row, a new `off` rule and an allowlist entry without
an expiry.
CLAUDE.md gains the rule E32 exists for: a gate with any real hit is never
switched off or narrowed, and an "enforced by" comment is checked by running
the check. The gate descriptions there, in docs/development.md and in
backend/app/README.md name the lanes as they now are, and the "directly"
recipe points at the full list.
ruff gains RUF100, PLW1510 and flake8-bugbear, every hit fixed rather than
ignored. B008 stays on with FastAPI's Depends/Query declared immutable.
basedpyright moves to standard mode, pytest to --strict-markers, and a hand
run of pytest now requires the database unless SIFTLODE_TEST_REQUIRE_DB=0.
PLW1510 found a real bug: the three ffprobe probes in plex/stream.py never
looked at the exit status, so a probe that failed was read as the file's
truth (an HDR source sent down the HW-safe path, a multi-audio file collapsed
to one track, a subtitle offset taken from garbage) and the warning each
except branch promises never fired. They now run with check=True.
The page halves compare the cursor's timestamp with the window's in Python;
a null or naive one raised there, past the route's 400. decode_cursor now
refuses a null non-nullable key and a naive timestamp.
A broad word matched most of the library and ts_rank read every match:
"a" took 4.4 s a page on prod. The window's start is read once (0.1 s) and
rides the cursor, so a video ingested mid-scroll cannot repeat a row; the
page is read as its two halves, each on an index (80 ms + 5 ms).
The e2e asserted the channel chrome never moves; it folds once now. The
ui catalog lists scrollFold and progressiveRows, Tabs names the
per-tab memory its callers use, and the welcome note says a removed
language orphans its stored copies too.
The playlist rows, the search box and the collection picker still
fetched the 400px poster for a 32-80px picture. A contract test pins the
set of bare Plex pictures, and the srcset widths to imgcache.SIZES.
A 1-2 letter prefix matched most of the library and ts_rank scored every
match: "m" took ~4-8 s on prod against ~150 ms now. The prefix match
starts at three characters (MIN_PREFIX_LEN); shared by the YouTube feed
and the Plex library search.
A watch-state scope (watched / in progress / unwatched) joined the per-user episode rollup into
each show facet query: five passes over the user's episodes per call, half of a filtered facets
request. The shows in scope are now resolved once and every group filters on their ids; the
grid-parity test holds the counts. Localdev: 99 -> 61 ms.
Cards fetched the full cuts: a 400x600 poster (~38 KB) for a ~144px picture and the 1280x720
backdrop (~162 KB) for a ~212px shelf picture. The image proxy now also cuts thumb_sm (240x360)
and art_sm (480x270), cached apart; card pictures (grid, shelves, collection tiles, Up next) carry
both in srcset with sizes=auto, so the browser picks by laid-out width and DPR. Localdev:
17 KB a poster, 56 KB a backdrop. The detail page keeps the full cuts.
The views, duration and title sorts had no index: every page read the whole videos heap and kept a
top-N (prod 350 ms a page, 550-780 for title). Migration 0096 adds one per direction and, for title,
per reader collation, in the exact ORDER BY feedquery builds. Localdev prod copy: plain sorts
180-310 ms -> ~1 ms, page 20 3-5 ms, narrow filters unchanged. view_count/duration are only written
by the enrich, which already rewrites indexed columns, so no HOT update is lost.
The title indexes are the first on an ICU collation, and postgres:16-alpine floats: collation.heal_drift
rebuilds them (CONCURRENTLY) at startup when Postgres reports the ICU version moved.
test_feed_sort_indexes holds every videos sort to an index and checks the rebuild.
GET /api/channels summed every subscribed channel's videos by reading each video's heap page
(681 MB on prod): 51-61 ms warm, but 2.25 s on a first open and 6.4 s under EXPLAIN once the
table had left memory. /feed/summary and /feed/years did the same (~140 ms warm, ~800 ms cold).
Migration 0095 replaces ix_videos_channel_pub_desc with the same key INCLUDING the columns
those queries read, so they are Index Only Scans over a 25 MB index. A test holds the queries
and the index's column list together.
The admin Configuration page did a db.get per setting (two for an unset one): it now reads
every override in one query (~40 ms of ~55 on prod).
- NavRailGroup: one "Admin" rail row opening a panel of the admin pages (links, so a
middle-click still opens a tab); the rail fits 133% zoom without a hidden scroll
- sysconfig.SECTIONS orders the config groups under four headings (every group in exactly
one, test-held); the page shows them as Settings' side list (Tabs `group` headings);
the lone registration group reads "Registration" under "Access"
- StatTiles / StatTile / StatBars: the Statistics overview's figures as tiles, the
per-operation quota use as horizontal bars (a zero draws none)
The audit log showed a failed RSS poll as the server's English sentence
under a Hungarian UI. A job can now raise progress.JobFailed(reason,
message, **params): the scheduler stores the reason and params as the
run's data (dashboard, manual-run notice, audit row) beside the English
line, and the client renders scheduler.failure.<reason>. The RSS outage
names its first channel and the reason code (HTTP 404 vs unreachable).
An upstream's own error text stays as raised (diagnostic).
E31 S1: the welcome is rendered once, at account creation, so an account that
switched language kept it in the old one (the owner's, English under a
Hungarian UI). A stored copy is recognised by its opening line (derived from
WELCOME) and served in the reader's language; other system messages stay as
stored.
- warm_portraits: only 404/410 mark a headshot dead (measured: Plex answers a dead CDN file 404);
a refused token or a 5xx is Plex failing, ends the run after three in a row and marks nothing.
'failed' counts every failure, not only the current streak.
- is_portrait_url: plain path segments only — no dot or empty segment an HTTP client folds onto
one file; person_thumb_url uses the same predicate (27 852 stored headshots all pass).
- Tests: Plex refusing (401/403/429/5xx), failures between successes (portraits and bios), the kept
original served from cache, the audit_clear / demo_reset / discovery_purge rows' counts as data,
and the 0094 backfill on real rows.
- people_enrich: a failed TMDB lookup or a dead headshot is skipped, not the end of the run (a dead
file is marked and left for a week; three failures in a row end it); portraits wait for the Plex
module switch; a no-Plex original keeps its own cache key, so the resize still replaces it; the job
entry lives in the plex domain (run_people_enrich)
- person-image takes only a headshot-shaped CDN url (no query variants of one file)
- every synced row gets Hungarian genres, and a HU genre filter matches the word as picked too;
migration 0094 fills the rows no overlay reached
- a job run's audit row carries its result as data; audit_clear / demo_reset store their counts
- one credit-column list for the prune and the enrichment order; _overlay_pass typed
- tests: batch skip + stop, HU bio stored, dead headshots, module switch, the person-image route,
the job body, genre filter + facet write + twin fallback, tagline/summary fallbacks, audit data,
the reader's alphabet at every remaining collated() call site
A person's TMDB bio was fetched only when their page was first opened, off the request path, so the first
open showed no bio and the page never asked again (2 of 18,202 linked people were ever enriched).
- people_enrich scheduler job (every 10 min, 300 per run): linked people's bios from TMDB and their
portraits into the image cache, the most-credited people first. A failed TMDB lookup ends the batch.
- Portraits are resized through Plex's photo transcoder to a 320 box (~15 KB instead of ~170 KB; the
person-image proxy uses the same cache key, and a full-size original is removed as its small twin lands).
- The person response says `enriching` while a bio is on its way (a person the job hasn't reached yet).
- "Known for" is ranked by vote count, without guest visits as themselves (talk shows had crowded out the
films: Tom Cruise had 14 of 15), with directed works, from a 40-deep pool; the page shows up to 15.
Migration 0093 un-stamps enriched people so the job refetches them.
- A phased job result (backfill, people_enrich) now counts as a change for the audit trail; before, an
automatic backfill run that added videos was never audited.
- HU: the watch reconcile job reads "egyeztetés" (a leftover from the S5 sweep).
The /person fetch appends translations and keeps TMDB's Hungarian biography beside the instance-language
one (new plex_people.biography_hu, migration 0092). A Hungarian reader is served it, the English one when
TMDB has none. Already-enriched people are un-stamped, so their next open refetches them once.
"Full history: done / requested" counted every subscribed channel whose
back-catalog was complete in the numerator - also the ones another user asked
for, or short enough to finish on their own - against only the channels this
user requested, so the Stats page read "342/329". The numerator now counts the
requested channels that are done; "channels without full history yet" still
counts every subscribed one.
The Postgres image is Alpine, whose en_US.utf8 is byte order: "Életrajzi" and
"Ötödik" sorted after "Zorró" in the feed, the channel lists, the Plex grid,
its genres and collections. `app/collation.py` names the ICU collation of the
reader's language (hu-x-icu puts "Ö" after "O" and "Cs" after "Cu"), wraps
every text ORDER BY with it - the feed's keyset cursor compares with the same
expression, so paging walks the order it sorted in - and orders the lists built
in Python (genre and studio facets, collection members, the discovery re-sort)
through the same collation. Plex title tiebreaks use the shown (HU) title.
- lib/vocab + the `vocab` locale area: TMDB crew jobs and departments, auto-tags (D1 A: the data
stays English, the catalog is named at its one source, useTags), built-in download profiles,
YouTube topics; held key-for-key to the backend's own lists by backendIds.contract.
- Countries by ISO code: TMDB parse stores {code, name}; rows stored before are coded from TMDB's
own country list in one request (run_enrich); Intl names them, historic codes (YU, SU, XI…)
from the catalog.
- Player tracks named from the language tag, forced/SDH flags and codec tail, not Plex's
server-language displayTitle; no invented "Audio N" on the server.
- Download lifecycle notices (shared / expiring / expired / evicted) named from data; the share
notice carries the title.
- Scheduler: the job result rides along as data and is named per key (dashboard + run notice).
- Invite status named; a render crash's notice no longer shows the engine's English message.
- Fix: YouTube links …/wiki/Humour, the tagger looked for "Humor" — Comedy never came from a topic.
Found in the S1b UAT: 'Mert megnézted: Batman Forever' kept the English title, and
the info page's TMDB recommendations kept TMDB's English titles even for titles we
hold. A held recommendation is now named as the library names it, in the reader's
language; one we lack keeps TMDB's title.
The grid served Hungarian titles, but every other card surface (Discover rails, the
show page's episodes, search, collections, the person page, playlists, Home's resume
rail, hero and comfort shelf) served English. The card serializers now take a
required keyword-only lang (a forgotten call site fails instead of going English), and
each of those reads takes ?lang=hu, with the language in its query key. A Hungarian
search finds an episode by its Hungarian title too.
An episode on a poster rail showed its 16:9 still cropped into a poster, titled only
with the episode name; it now shows the show's poster and name, with S·E · title under
it (option A). Plex's generic season names (Season 1, Specials) are rebuilt from the
number in the reader's language; a season with a name of its own keeps it.
Home's comfort-shelf move wrote its optimistic order to the key without the language,
which nothing read any more; it writes the query's own key.
The HU TV section lists every episode with its Hungarian title and summary, but the
overlay read only movies and shows, so every episode stayed English. It now overlays
the episodes too, and the HU tagline (new tagline_hu column, served on ?lang=hu).
Plex's HU agent returns a mixed genre vocabulary (Action beside Dráma), and a title
with no HU twin kept its English list, so the HU genre list showed Drama and Dráma.
plex/genres.py maps every genre word to one Hungarian word; the overlay writes it
into genres_hu for every row, and a HU genre filter is read through the same map,
so a saved filter still holding Action keeps matching.
A delete landing while a push read YouTube left a plan over no items, so the push emptied the
YouTube playlist a local-only delete keeps: the push now stops at a vanished list (PlaylistGone,
404), and a delete is refused while a sync of the list runs. The closing baseline fingerprints the
title YouTube got, not the one its lock re-reads, so a rename made during the writes stays dirty.
The other-tab test helper bounds its lock wait, so a lock-order regression fails instead of hanging.
The mirror refresh re-ingested inside its row lock, and that ingest's commit let go of the lock
before the replace: a remove made then was undone and the list read as synced. Push, re-pull and a
delete touched items before the list row, a deadlock beside the playlist-first route edits. A push
now re-plans from the items under the lock and derives dirty after its YouTube writes.
lock_playlists moves to the sync package. Tests pin both ids of a move's lock and the claim's
own connection.
- Every playlist item write (add, remove, move, reorder, rename; Plex remove and remove-bulk) row-locks
the playlist before it touches an item, a move both lists in id order: remove beside reorder was a
deadlock, and two adds of one video a unique-key 500.
- The scheduled mirror refresh re-reads dirty under that lock, so an edit made during its YouTube calls
is not replaced by YouTube's order.
- A user's bulk pull holds a per-user advisory claim: two at once created duplicate mirrors. The manual
sync answers 409 sync_all_busy instead of '0 synced', and like the re-pull spends its cooldown only
on a pull that runs.
- Deleted mirrors are committed at once, not only when a later list commits.
- The claims open an unpooled connection of their own instead of a third pooled one per push.
peter