The audit log showed a failed RSS poll as the server's English sentence
under a Hungarian UI. A job can now raise progress.JobFailed(reason,
message, **params): the scheduler stores the reason and params as the
run's data (dashboard, manual-run notice, audit row) beside the English
line, and the client renders scheduler.failure.<reason>. The RSS outage
names its first channel and the reason code (HTTP 404 vs unreachable).
An upstream's own error text stays as raised (diagnostic).
- lib/vocab + the `vocab` locale area: TMDB crew jobs and departments, auto-tags (D1 A: the data
stays English, the catalog is named at its one source, useTags), built-in download profiles,
YouTube topics; held key-for-key to the backend's own lists by backendIds.contract.
- Countries by ISO code: TMDB parse stores {code, name}; rows stored before are coded from TMDB's
own country list in one request (run_enrich); Intl names them, historic codes (YU, SU, XI…)
from the catalog.
- Player tracks named from the language tag, forced/SDH flags and codec tail, not Plex's
server-language displayTitle; no invented "Audio N" on the server.
- Download lifecycle notices (shared / expiring / expired / evicted) named from data; the share
notice carries the title.
- Scheduler: the job result rides along as data and is named per key (dashboard + run notice).
- Invite status named; a render crash's notice no longer shows the engine's English message.
- Fix: YouTube links …/wiki/Humour, the tagger looked for "Humor" — Comedy never came from a topic.
A delete landing while a push read YouTube left a plan over no items, so the push emptied the
YouTube playlist a local-only delete keeps: the push now stops at a vanished list (PlaylistGone,
404), and a delete is refused while a sync of the list runs. The closing baseline fingerprints the
title YouTube got, not the one its lock re-reads, so a rename made during the writes stays dirty.
The other-tab test helper bounds its lock wait, so a lock-order regression fails instead of hanging.
The mirror refresh re-ingested inside its row lock, and that ingest's commit let go of the lock
before the replace: a remove made then was undone and the list read as synced. Push, re-pull and a
delete touched items before the list row, a deadlock beside the playlist-first route edits. A push
now re-plans from the items under the lock and derives dirty after its YouTube writes.
lock_playlists moves to the sync package. Tests pin both ids of a move's lock and the claim's
own connection.
- Every playlist item write (add, remove, move, reorder, rename; Plex remove and remove-bulk) row-locks
the playlist before it touches an item, a move both lists in id order: remove beside reorder was a
deadlock, and two adds of one video a unique-key 500.
- The scheduled mirror refresh re-reads dirty under that lock, so an edit made during its YouTube calls
is not replaced by YouTube's order.
- A user's bulk pull holds a per-user advisory claim: two at once created duplicate mirrors. The manual
sync answers 409 sync_all_busy instead of '0 synced', and like the re-pull spends its cooldown only
on a pull that runs.
- Deleted mirrors are committed at once, not only when a later list commits.
- The claims open an unpooled connection of their own instead of a third pooled one per push.
The client's in-flight flag covers the menus of one page only. A push reads YouTube, then writes
both sides for seconds and commits step by step, so a row lock ends at its first commit: two tabs,
or a push and the scheduled mirror refresh, could interleave into a doubled YouTube item or a
baseline recorded over the other's writes. `sync_claim` takes a session-level advisory lock on a
connection of its own (it survives those commits; a crashed worker's lock dies with its
connection, so no claim column is left stuck). Push and revert answer 409 `sync_busy`; the
scheduled refresh skips a list being synced and picks it up next run.
channels.list simply omits a terminated channel, so nothing was stamped and the
channel page spent a quota unit on it every time. The miss is stamped like a
fetch; the published_at sentinel that kept it stale is gone.
Channel metadata (banner, avatar, description, counts) was fetched only the first time a
channel was seen: the subscription import asked only for channels with no details, and the
channel page only for ones missing a join date. 394 of 397 subscribed channels had July
metadata. Both now also refresh details older than DETAILS_MAX_AGE (7 days), through one
shared staleness rule; channels.list costs 1 unit per 50 channels.
A failed connection or token step answered nothing about the channel, yet stamped it: one proxy blip
during an RSS blackout delayed every due channel by a whole window. YouTubeError now names the
endpoint that answered; a failure no endpoint gave, or a spent quota, ends the round and stamps
nothing.
A channel whose uploads playlist also failed was never stamped, so a terminated channel was charged
again on every 20-minute poll instead of once per rss_fallback_hours. An upstream quotaExceeded now
stops the round instead of charging every remaining channel the same refusal.
An INFO line outside a request never reaches log_events, so the fallback's
own summary was invisible in the Server log. Its counts ride on the
sync.rss.failed warning, the one line of a poll the console keeps.
Measured on localdev: 375 of 375 dark channels read, 403 new videos.
The reads the frontend polls (unread counts, /api/me, the download index,
sync status, subtitle progress) are logged at DEBUG: at INFO they were ~95%
of the stored log on prod. A scheduler job failure and a playlist sync
failure carry event codes, and a Google account without a YouTube channel
has an empty playlist list instead of a warning every six hours.
YouTube's RSS goes dark per channel for hours (prod 2026-09-18/19: 393 of
396 feeds answered 404 for four hours, sticky per channel, so a retry does
not help), and nothing else read a followed channel's new uploads. A
channel whose feed has not answered for `rss_fallback_hours` (admin knob,
default 2, 0 = off) now gets one page of its uploads playlist through the
Data API (1 unit), at most once per window, yielding to the backfill
reserve. Spend is attributed to its own quota action.
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
`formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
file paths, and the job-failed, refused, edit-failed and backfill lines use
it. Done lines log the asset id, not the title-built path. The PII contract
now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
a process's last batch closes the repeat window, so a short-lived download
child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
`download.job.failed` with a cause; they left no line at all before. The
crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
carry reason `transport`. Subscribe recognises a duplicate by reason code
only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.
What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
degraded to a software transcode with nothing saying why; a failed audio probe
collapsed a multi-audio file to one track and the player showed no language
picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
own root, leaving every crashed session's segments on disk forever.
Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.
ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.
logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.
tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
25 literal names covered 34 modules. plex/client.py, plex/stream.py,
plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves
`siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`;
routes/channels.py called itself `siftlode.api`. The `logger` column is in every
stored row so E22.6's console can filter on it, and so an operator can pin a level
to one part of the system — neither works when six modules share a name.
`logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync.
One name per module, and it cannot drift from the file it sits in.
tests/test_logger_names.py pins the derivation, that everything stays under the
one `siftlode` tree the handler and the tests attach to, and that no module
hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`).
Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic
it carries rather than the module it lives in — it replaced uvicorn's access log
and is what an operator pins when they want request lines and nothing else.
app/scheduler.py called its logger `logger`; every other module calls it `log`.
ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them:
worker.py logged `str(exc)[:120]` where the traceback was the whole question, and
three prints/f-strings that are deliberate now say so on the line. S110 and BLE001
stay off until S3 — 23 handlers are outstanding and each is a judgement about one
of them; enabling the rules first would just mean 23 blanket noqas.
The explore purge rebuilt followed_channel's two EXISTS clauses inline, so what is polled and what may be deleted could drift; it now uses the predicate, which moved to sync/explore.py. The env validator and sysconfig each folded a choice value on their own; both call config.canonical_choice. The deep-backfill comment no longer calls recent backfill unconditional.
The scheduled RSS poll and recent backfill covered every channel row,
but rows also come from search results, playlist sync and dropped
subscriptions, and nothing deletes them. On prod 582 of 979 polled
channels had no follower, three of them dead feeds answering 404 on
every poll. Both jobs now take subscribed or explored channels only,
and the scheduler card counts the same set. As a deliberate trade-off,
the shared Library no longer receives new uploads from channels nobody
follows.
The log now has a stored, queryable copy for the diagnostics console:
every WARNING and above plus the request line of every request, kept
for log_retention_days (default 7, no "forever") and pruned hourly by
the new log_gc job.
Rows are built on the logging thread from the same redacted fields as
the stdout line, queued without blocking and written in batches by one
background thread on its own connection. A full queue or a failed write
drops and reports instead of retrying. Flood control keeps 30 of any
one message per minute plus a count of the rest; anonymous request
lines are capped at 60 a minute per client, and the client address is
never stored. Deleting an account deletes its stored events.
Measured on prod first: a YouTube RSS outage wrote 11k warnings in four
hours, one per channel per poll. The poll now writes one warning with
per-reason counts and a few examples, and a failure on our side gets
one traceback per poll instead of one per channel.
- Button: `loading` always disables (|| not ??) — stops a double-submit
when a call-site passes an explicit boolean `disabled` (VideoEditor).
- playlists: track un-ingestable-but-live YouTube ids in the sync baseline
so a linked playlist containing a private/deleted video converges instead
of re-classifying it as a pending pull forever; re-attempt ingestion each
sync so a restored (private->public) video heals, and prune ids that
leave YouTube.
- SettingsPanel: download-auth section renders an honest error/loading
state (StateMessage + retry) instead of a bare empty section on a failed
credentials fetch; HU+EN loadError string.
- tests: reorder-order, partial-failure baseline hold, un-ingestable
converge/heal/prune, Button loading-disables, download-auth error/retry.
"Sync to YouTube" was a one-directional destructive push (local wins, deletes
from YouTube every item not in the local list), so items added on YouTube were
lost and a dirty pull was skipped entirely. Replace it with a real 3-way merge:
each item is classified against the last-synced baseline as a local-add, a
YouTube-add, or a genuine delete, so an add or delete on either side survives.
- Persist the ordered baseline id set (new playlists.synced_video_ids, migration
0075); the fingerprint hash alone couldn't drive a merge. NULL baseline (fresh
export / pre-S7 link) unions losslessly on the first sync and records one.
- Pure merge_plan(baseline, local, youtube) -> add/remove per side + merged order
(local order, YouTube-only additions appended). merge_playlist applies the
local side directly and the YouTube side through the failure breaker; the
baseline only advances on full success so a partial failure re-converges.
- plan_merge/push-plan now report what each side gains and loses; the confirm
dialog shows the two-way plan before applying (pushPlanMerge, HU+EN).
- Tests: 10 new (pure classification incl. delete-vs-stale-copy; DB apply proving
a YouTube-add and a Siftlode-add both survive on both sides, a genuine delete
propagates without collateral loss, no-baseline never deletes, create path).
Fixes from several multi-agent deep-review passes over the E8 feed-filtering &
channel-manager epic (S1-S4):
- feed_years buckets by UTC year to match the UTC-pinned year-filter bounds
- tag suggestions: examples ∩ subscribed (drop tagged-then-unsubscribed channels
from the profile denominator); exclude hidden channels as suggestion targets
- channel manager: restore ghost chips + surface an error toast on a failed tag accept
- year filter: debounce the search term so the per-year facet doesn't refetch per keystroke
- channel tags: aria-label the "+N" overflow pills
- extract useTagSuggestions hook; add backend + hook tests (profile boundaries, cap,
language exclusion, unsubscribed/hidden handling, UTC bucketing, bulk-accept)
Bump VERSION to 0.73.0 + release notes.
Learn, per user tag, which auto TOPIC tags are characteristic of the channels
you've tagged it, then suggest that tag for still-untagged subscribed channels
whose topics cover enough of the profile. Language auto-tags are excluded from
the signal — near-universal within a language, they flooded every channel.
New GET /api/tags/suggestions (read-only, deterministic) backed by
app/sync/tagsuggest.py. Surfaced in the Channel Manager as dashed "ghost" chips
in the tags cell — one tap accepts (reuses the attach-tag endpoint), × dismisses
for the session — plus a header badge and a bulk "Accept N confident" (the
full-profile matches) behind a confirm. No silent writes (D4 = suggest-and-confirm).
Also reflow a pre-existing prettier drift in dateFilter.ts from E8·S2.
The scrape source exposes no publish date, so its search stub carried
published_at=None. Migration 0068 (PERF-15, E2/0.68.0) made
videos.published_at NOT NULL, so insert_stubs raised IntegrityError —
outside the route's page-fetch try/except — and the whole
/api/search/youtube request 500'd ("YouTube search failed."). Every
scrape search (the default source) had been broken since 0.68.0.
- routes/search._ingest_candidates: stub published_at falls back to
now() (mirrors stub_from_playlist_item), satisfying the constraint.
- sync.videos.apply_video_details: prefer the enrich snippet's real
date whenever present, so the now() placeholder is corrected instead
of becoming the video's permanent date (the guard only filled a NULL,
which a non-null placeholder no longer is). Never clobbers a real
date with now().
Regression tests (DB lane): dateless scrape stub ingests and takes the
real enrich date; an enrich miss keeps a non-null placeholder, unkept.
YouTube channel descriptions pad their contact/link blocks with runs of blank
lines, which the About popover renders verbatim (whitespace-pre-wrap) — wasted
space and storage, and noise in the search_vector. Add normalize_description()
and apply it at both channel store sites (full details + initial import) so the
DB holds the clean text. New channels are clean immediately; existing rows heal
on their next metadata re-sync. Unit-tested in test_utils.
The admin-tunable maintenance re-validation batch lived in its own app_state
column with a hand-rolled override-or-default reader in app.state — a fourth
config store duplicating the sysconfig SPECS pattern. Move it to a sysconfig key
(`maintenance` group, min=100 max=100_000): the reader (sync/maintenance.py) and
the Scheduler dashboard (routes/scheduler.py PATCH + GET) now go through
sysconfig.get_int / set_value / spec, and app.state loses get/set_maintenance_batch
+ the MIN/MAX constants + the settings import. Migration 0070 carries any existing
override into the system_config store before dropping the app_state column
(reversible; restores + copies back on downgrade). Reuses the existing
sysconfig.spec() for the dashboard's min/max, so the bounds have one source.
Full lane 365 pass (migration drift + reversibility exercised), ruff clean.
This codebase leans on "why" comments as its architecture doc, so a header that
claims the opposite of the code below it is worse than none. Fix the seven the
review flagged: channels.py (subscribe/unsubscribe shipped), playlists.py +
sync/playlists.py (YouTube two-way sync + write-back shipped), setup.py (drop the
"epic 6c" ref — the wizard steps exist), config.py ("auth._client_ip" ->
"routes.auth._client_ip"), db.py ("routes/plex.py" -> "routes/plex/" after the
package split), sysconfig.py (the "Phase C" note — the watch-sync job that reads
plex_watch_sync_interval_min has landed; clarified it's a boot default, not a knob).
Docs only.
Two scheduler jobs full-seq-scanned the 257k-row videos table every run:
- PERF-17 enrich_pending (`WHERE enriched_at IS NULL LIMIT N`, every few minutes):
44,680 buffers even with 0 pending -> a partial ix_videos_enrich_pending makes
it a 1-buffer index scan (empty when nothing pending).
- PERF-18 maintenance revalidate (`WHERE unavailable_since IS NULL ORDER BY
last_checked_at NULLS FIRST LIMIT N`, ~300 pages/run): 44,754 buffers + top-N
sort -> a partial ix_videos_recheck_order in that exact order = a 42-buffer
index scan. The order_by is rewritten to `asc().nulls_first()` (equivalent to
the old `(IS NULL) DESC, ASC`) so it rides the index; the unused plain
ix_videos_last_checked_at is dropped.
PERF-3: drop the never-picked boolean btrees (is_short, via_explore) and slim the
useful ones to partials over the queried subset (via_search=true, NOT
shorts_probed, unavailable_since IS NOT NULL) — less write amplification on the
biggest table, same lookups. Model: those 6 columns lose their inline index=True
(the partials are migration-defined, like the search_vector GIN); live_status
left alone.
Measured on the prod-sized localdev reseed: enrich 44,680 -> 1 buffer (25.9 ->
0.02 ms), maintenance 44,754 -> 42 buffers (35.5 -> 0.17 ms); all 11 CONCURRENTLY
index ops on 257k rows in 0.75s. Gate green (346 pytest incl. drift + downgrade
round-trip).
A nullable published_at forced the feed keyset into a non-sargable
`... OR published_at IS NULL` disjunction: with the 0067 index the planner still
applied it as a FILTER and scanned from the top of the index, so a deep `newest`
page discarded every earlier row (O(depth)).
- Migration 0068: backfill any NULLs (0 in prod) -> SET NOT NULL + DEFAULT now().
- Model: Video.published_at is now non-Optional, nullable=False, server_default now().
- feedquery: the `pub` key is nullable=False, and keyset_predicate is rebuilt
recursively so a non-nullable leading key gets a non-strict range bound
(`published_at <= v`) as an AND conjunct — an index range Cond instead of a
filter. Nullable keys keep the NULL-aware OR form (a `<= v` bound would drop
their NULL group). Equivalence proven by test_feed_keyset (all 13 sorts,
cursor-walk == unpaged).
- Ingest paths (stub_from_playlist_item, apply_video_details, playlists sync)
fall back to now() so they never write an explicit NULL.
Measured on the prod-sized localdev reseed at a ~50k-deep cursor (0067 index +
prod planner): OLD `Rows Removed by Filter: 50001`, 25,687 buffers / 9.16 ms ->
NEW `Index Cond: published_at <= v`, 1 row filtered, 47 buffers / 0.037 ms. SET
NOT NULL on 257k rows: 0.32s. Gate green (334 pytest).
Addresses the code-review findings on the immediate full-history change:
- Extract quota.backfill_reserve_reached(): the remaining/reserve compare was
open-coded at seven sites (recent/deep/immediate backfill, explore, rolling
revalidation, and the new per-page guard). It reads in its OWN short-lived
session (like record_usage/log_action), so a per-page guard no longer holds
the caller's transaction open across the YouTube fetch that follows — the
400-page immediate run no longer pins a pooled connection idle-in-transaction
through each network round-trip. Migrated all call sites; dropped the now-dead
sysconfig imports in channels.py and maintenance.py.
- Simplify the deep-backfill loop: channel.backfill_done = not page_token (done
is False on entry) plus one break, replacing the doubled `if not page_token`.
- Tests patch the backfill_reserve_reached seam directly instead of mocking
remaining_today internals, removing the fragile dead-value budget iterator.
Full backend suite green; ruff clean.
The user-triggered "get full history" (backfill_channel_now) capped at
max_pages=60 = 3000 videos. A channel just over that (BBC Earth: 3403
uploads = 69 pages) stopped one page short, so its tail dribbled in over
the scheduler's paced 10-minute ticks (250 videos/tick) and the button
spinner looked stuck for ~18 minutes with no visible progress.
- Raise the immediate cap to max_pages=400 (20 000 videos) so any
realistic channel completes in the single background run.
- Make a high cap safe: backfill_channel_deep now re-checks the shared
quota before each page and yields at the backfill reserve (previously
quota was checked once by the caller before the whole run), and commits
the advanced cursor after every page so a restart resumes instead of
re-fetching — and re-charging quota for — pages already stored.
- Channel page: while loading, the button shows the live stored/total
count and a tooltip explaining the tail finishes in the background, so a
very large channel's paced remainder reads as progress, not a hang.
Adds deep-backfill paging tests (per-page cursor commit + resume, and the
per-page quota yield).
- Make the immediate-backfill guard SYMMETRIC: run_deep_backfill now also
registers the channel it pages in _backfill_in_progress, so it and
backfill_channel_now mutually exclude per-channel and never advance the
same cursor concurrently (whichever started first). The interactive cap
moves to a separate _immediate_running counter so the scheduler's channels
don't false-trip it.
- Attribute the immediate backfill's spend: wrap it in
quota.attribute(user_id, VIDEOS_BACKFILL_FULL) (route passes user.id), so
it shows under the user's full-backfill action, not the OTHER/system bucket.
- Tests: the update_channel->background-task wiring + the demo guard
(scheduled for a real user, NOT for demo); distinct N/S/L counts (2 short
+ 1 live) so a short/live swap fails; extracted channelDetailPollMs with a
pure unit test of the poll-or-stop predicate.
- a11y: role="img" on the fully-synced charm so screen readers announce it.
- list_channels no longer computes the Normal/Shorts/Live aggregation the
frontend stopped reading when R14 dropped the N/S/L table column; the dead
ManagedChannel.count_* fields are gone too.
- Immediate "get full history" backfill now coordinates with the paced
scheduler via a small in-flight set: capped at 2 concurrent + per-channel
dedup, and run_deep_backfill skips a channel an immediate backfill owns —
killing the cursor race and bounding the quota-drain burst, without losing
the speed win.
- Add test_channel_backfill.py: the channelDetail N/S/L counts + the guard's
no-op / cap / scheduler-skip behaviour.
- Inline the syncedChannels useMemo (its dep is a fresh array each render, so
it never cached); fix the stale "~2-min ticks" comment (it's 10 minutes).
Clicking "get full history" now kicks an immediate background deep
backfill of that one channel (backfill_channel_now: own session, up to
~3000 videos in one pass, quota-guarded) instead of waiting for the
scheduler's paced 10-minute ticks (5 pages/channel/tick). The request
returns instantly and the channel page's live poll shows the count fill
in seconds. A very large channel's tail still falls to the scheduler
(deep_requested stays set).
Layering cleanup, no behaviour change (ruff clean backend+scripts, pytest 193):
- Invert the scheduler->routes dependency: reset_demo_state (+ its demo-seed
helpers) move from routes/admin.py to a new service module app/demo.py.
Both routes/admin.py and scheduler.py now import it from app.demo, so the
scheduler no longer reaches into the routes layer — its lazy import (the
routes<->scheduler cycle workaround) is gone, hoisted to a top-level import.
- Publish the private symbols that were imported across module boundaries (the
underscore convention was lying): feed._serialize -> serialize,
feed._to_tsquery_str -> to_tsquery_str, videos._insert_stubs -> insert_stubs,
videos._stub_from_playlist_item -> stub_from_playlist_item,
worker._uploader_url -> uploader_url, plex._shared._item_or_404 ->
item_or_404. Scoped renames — the 5 other route modules keep their own truly
private _serialize helpers.
- _release_asset moves from routes/downloads.py to app/downloads/service.py as
the public release_asset (asset bookkeeping is a service concern); the routes
call it via service.release_asset.
The worst one was, again, the previous round's own fix.
- Round 7 stopped awaiting the HLS sweep so startup could serve immediately — which
also made it run CONCURRENTLY with playback, and `sweep_orphans` snapshotted the
live-session set BEFORE listing the root and rmtree'd outside the lock. Since
`start_session` reuses a directory path for the same (key, offset, tag), a stale
name can become a live session at any moment. Two guards now: nothing whose mtime
is newer than this process's start is ever deleted, and the live check + rmtree
happen together under the lock, one directory at a time.
- The sweep task is awaited after cancel, so shutdown can't leave it pending.
- `_rate_limiter` used a truthiness test on its timestamp, so a `now()` of exactly
0.0 read as "never ran" — unreachable in production, but `now` exists to be
injected and a test clock starting at 0 is the obvious choice.
- The RSS total-outage error now carries the first underlying error: "all N channels
failed" is also what ONE dead feed id looks like on a small instance, and a red
card the user can't act on is its own kind of dishonest.
- The WebKit fullscreen handling moved into `lib/fullscreen` and PlexPlayer uses it
too — it had the same unprefixed-only reads in three places, including an Escape
branch that would have closed the whole player instead of leaving fullscreen.
- The `--max` CSS comment claimed `inset: 0` did the sizing; Tailwind's `w-full` is
still on the element and wins. Comment now states the real mechanism.
Verified in real Chrome (the pane can't composite or do native fullscreen):
- F fills the 1920x889 viewport; the exit pill renders top-right on the letterbox
band and overlaps NO YouTube control — the collision this round suspected does not
reproduce in this embed (its own buttons sit bottom-left / bottom-right).
- Four real ArrowDown presses while maximised: persisted volume 100 -> 80, with the
level flash visible on screen.
- A real click on the exit pill un-maximises and returns focus to the card.
- lib/fullscreen driven from a real click: enter (innerHeight 889 -> 1024), the
change listener fires exactly once per transition, exit returns to 889.
Backend suite 150 -> 158; the new sweep tests are mutation-checked, and they read
their cutoff from the filesystem rather than a clock (the container VM and the
bind-mounted host disagree by enough to make a clock-based assertion flaky).
Five of them sit where round 6 turned F from a Fullscreen API call into a CSS
maximise: the guards around it still asked `isFullscreen`, which that pivot made
permanently false.
- Arrow-key volume was dead in the maximised view: the scroll-deference checked
only `isFullscreenRef`, so the (now hidden but still scrollable) card kept
swallowing Up/Down. Both flags now count as "the video fills the screen".
Measured: 87px of scroll room on the focused card, ArrowDown moves 80 -> 75
and scrolls nothing.
- The maximised view had no way out once the pointer touched the video. Reaching
YouTube's controls means clicking into the cross-origin iframe, which takes
keyboard focus with it, and the re-arm paths (stage mouseleave, window focus)
need the pointer to leave the browser entirely while the stage fills it. So
the exit now lives on screen, in our DOM: a button that un-maximises AND pulls
focus back, re-arming every shortcut.
- WebKit fires only `webkitfullscreenchange`/`webkitFullscreenElement`, so on
Safari/iPadOS YouTube's own fullscreen never registered — no overlay yield, no
focus reclaim. One `fullscreenEl()` reader, both event spellings.
- The stage claimed `z-index: 9999`, the glass tuner's reserved level, for a job
that only needs to beat the modal chrome it is nested in (its siblings top out
at z-menu). Dropped to rail-level; nothing escapes the modal's stacking
context either way.
- `inset: 0` already sizes a fixed layer — the inherited `100vw`/`100vh`
over-constrained it, which is how a classic scrollbar gets a horizontal
overflow and a mobile URL bar hides the bottom edge (YouTube's control bar).
- The shortcut hint still promised "F: fullscreen" in both locales.
Backend, both "the fix stopped one layer short":
- The startup HLS sweep was awaited inside lifespan, so uvicorn still served
nothing until it finished — a thread doesn't help when nothing is listening
yet. Fire-and-forget task instead, cancelled on shutdown. Measured with 200
planted segments: "Application startup complete" now precedes "swept 1".
- An RSS poll where every channel failed returned normally, so the scheduler
recorded `ok` with the outage buried in the summary string. `failed == total`
now raises: a run that reached nothing is a failed run.
Suites: backend 147 -> 150.
- run_ffmpeg stops the process from ONE exit path (`except BaseException`), so
an exception out of on_progress (a DB write) can no longer orphan a live
ffmpeg after a 10s join on its pipes; _stop is now a no-op on a dead process
- both run_ffmpeg callbacks are throttled to ~1s (they were a DB round-trip per
output line: ~10^5 queries on a long re-encode), mirroring the download hook
- volume arithmetic extracted to lib/playerVolume with unit tests; a corrupt
level now falls back to the LAST GOOD one instead of full blast, and the
sanitising happens where the stored value enters
- RSS is back to two except clauses (the shared tail already sits after the try)
- new tests: callback-exception cleanup, the two throttles; the fake Popen grew
poll() and its wait count is now asserted
Mutation-checked: each new group goes red when its fix is reverted.
- Feed memoizes onClose (like Playlists already did): the inline closure re-ran
PlayerModal's keydown effect on every Feed render, which since the debounce
landed also flushed the volume write and re-stole focus
- the player reads its volume with readAccountMerged again — a partial stored
object made the level undefined -> NaN -> a persisted null; applyVolume also
refuses a non-finite level outright
- that read moved into a lazy useState initializer (was running per render)
- the two RSS except branches share one tail, so a counter can't drift again
- run_ffmpeg tests cover the SIGTERM->SIGKILL escalation and the lingering-
process path; dropped an assertion that only restated the fixture
- storage.ts documents the actual { volume } shape (0 IS muted)
- the sweep guard anchors the rating_key to digits: the round-2 widening had
degenerated to "anything ending in _<digits>" (backup_2024, release_10)
- RSS counts apply-side failures too, so a read-only database no longer reports
a wholly failed poll as "ok — new=0, failed=0"
- the player reads/writes its volume directly (readAccount/writeAccount): the
persisted-object hook rendered nothing and cost a second write per settle
- new tests: the sweep guard's accept/reject table, and run_ffmpeg's threading
(drain, cancel while silent, stall watchdog, stderr tail on a nonzero exit)
- the breaker sentinel is checked one way (isinstance) at all four sites
- run_rss_poll is typed dict[str, int]
All three new test groups were mutation-checked: reverting each fix turns them
red (or, for the cancel/stall pair, hangs — which is the bug itself).
- the volume keys no longer defer to a scrollable card while in fullscreen
(mirrored into a ref: the keydown handler is bound once and state would be stale)
- the debounced volume flush writes straight to storage; persisting inside a
setState updater is not guaranteed to run for an unmounting component
- the HLS sweep also matches the original {key}_{start} directory naming, which
is what the oldest leftovers on disk actually use
- run_rss_poll returns {new, failed} so an egress outage stops rendering as
"ok — 0" in the scheduler card and the audit log; /api/sync/rss reports it too
- drop the unused poll_rss_channel (it silently gained a new exception)
- _Breaker.run is generic, so a call site keeps its item-id type
- test helper instead of the generator-throw idiom
- HLS sweep only deletes directories matching the session-name shape; the root
is admin-configurable and may be a shared path
- playlist unwrap picks the first entry that actually downloaded, not entries[0]
- concat quoting keeps backslashes literal (ffmpeg treats them so inside '')
- ArrowUp/Down defer while the focused element can still scroll that way
- the staging fallback skips yt-dlp working files (.part-Frag/.ytdl/.temp/.fNNN)
- the boot HLS sweep runs off the event loop
- volume persistence is debounced (was a setItem per wheel notch) and a spin no
longer loses notches to the iframe's lagging getVolume
- _Breaker.run owns the whole open/call/record protocol; four call sites shrink
- should_prune extracted and unit-tested, plus RSS error-vs-empty tests
- the worker exits non-zero when the schema never lands
- never prune subscriptions on an empty fetch: one anomalous empty-but-200
response wiped every subscription (and its deep_requested flag) in one run
- push_playlist: stop after 5 consecutive write failures — a quota-dead or
scope-revoked account otherwise burns 50 units per remaining item
- RSS: raise RssError instead of returning [] on a transport/HTTP failure, so a
failed poll no longer stamps last_rss_at and a 404 feed stops looking healthy
- worker: exit when the schema never appears instead of "continuing" into a
crash one call later
- apply_channel_details: (branding.get("channel") or {}) so a "channel": null
in brandingSettings can't AttributeError-abort the enrich batch.
- topicLabels: drop empty labels so a malformed topic URL can't render a blank chip.
- fix: reserve the scrollbar gutter (scrollbar-gutter:stable) on the channel
page's scroll container, so switching between the tall Videos tab and the
short About tab no longer shifts the banner/avatar/buttons a few px sideways
(the vertical scrollbar was appearing/disappearing between the two tabs).
- About tab now shows Country (flag + localized name), Language, Topics
(topicCategories → readable chips), and Keywords (brandingSettings keywords
parsed into chips, quoted multi-word tags kept whole). country/language/topics
were already stored; keywords is new (migration 0055_channel_keywords, mapped
in apply_channel_details, returned by channel_detail).
- Discovery → channel page: the Channel-manager "Discover from playlists" tab now
links each channel name to its in-app channel page (ChannelLink onView), so a
discovered channel's About/videos can be inspected BEFORE subscribing (was
subscribe-only, plain text before).
Note: the channel info-card epic itself was already delivered in v0.19.0; this is
the About-tab enrichment follow-up + the header-shift fix. external_links stays
empty by design (YouTube removed the field from the Data API ~2023).
Verified each deferred per-subsystem item against current source, then fixed the
real ones (tradeoffs left as-is, see siftlode-ops/CODE-HYGIENE.md closeout):
- search BUG-3: don't finalise shorts_probed on un-enriched stubs (enrichment
failure/omitted rows) — restores the scheduler's enriched_at guard so a real
Short can't leak into the feed and never get reclassified.
- downloads GC: 4th pass reaps orphaned errored, fileless, unreferenced assets
(they carry no TTL, so the expiry passes never cleared them).
- downloads B6: quota/edit errors now return a structured {code,reason,limit}
the trilingual client localises, instead of hardcoded English + dot-decimal GB.
- channels BB1: reset-backfill re-arms deep sync on every subscriber (bulk update)
instead of 404ing when the admin isn't personally subscribed.
- channels BB2: enrich the stub on BOTH the normal and "already exists" desync
path (nest the insert try inside the client `with`).
- channels CB3: drop the redundant second _discovery_rows read (identity-mapped
rows are already enriched; re-sort in Python for the title tie-break).
- playlists PC1: read the live playlist once in push() and share it with plan_push
+ push_playlist (halves read-quota, closes the second TOCTOU window).
- playlists PC2/PC5: batch the cover thumbnails in one window query (was N+1);
rename combines count+duration into one aggregate + a single cover lookup.
- messages MB2: get_thread only resolves a messageable partner OR one we already
share a thread with (closes the user-enumeration oracle).
- messages MB3: push a live unread-count to the user's other tabs after mark-read.
- messages MC4: list_conversations uses DISTINCT ON + grouped COUNT instead of
loading the whole message history into memory.
- config AB3: per-spec allow_empty so smtp_user/youtube_api_proxy can be blanked
to disable them rather than snapping back to the env default.
- Extract runner.token_users(db): the "all users with a stored refresh token"
query was spelled 3× (get_service_user, run_subscription_resync,
sync_all_playlists). get_service_user now reuses it ([0]); sync_all_playlists
imports it from runner (no cycle — runner doesn't import playlists); the now-
unused OAuthToken import is dropped there.
- scheduler.py: _job("rss_poll", lambda db: run_rss_poll(db)) → _job("rss_poll",
run_rss_poll) — the lambda was dead indirection; all sibling jobs pass the
callable directly and _job calls fn(db).
Behavior-neutral. ruff clean, localdev boots, re-review clean.
- feed.py _filtered_query: drop the dead 2nd return element (status_expr was
used only internally for WHERE filters; all 3 callers discarded it as _status).
Now returns (query, rank_expr); fixed the stale docstring.
- youtube/client.py: extract _iter_playlist_items() — iter_my_playlist_video_ids
and iter_playlist_items_with_ids were near-identical playlistItems paging loops
(jscpd [173-187]≈[267-281]); both now map over the shared generator.
- sync/videos.py: extract _apply_video_batch() with an on_missing callback —
enrich_pending and refresh_live shared the fetch-map-apply skeleton, differing
only in the query and how they retire rows YouTube omits.
- models.py: add LIVE_OR_UPCOMING = ("live","upcoming"); replace the 4 duplicated
copies (feed.HIDDEN_LIVE, search._LIVE_HIDDEN, videos.py + channels.py inline).
Behavior-neutral. ruff clean on touched files, localdev boots, feed/worker healthy.
peter