- Make the immediate-backfill guard SYMMETRIC: run_deep_backfill now also
registers the channel it pages in _backfill_in_progress, so it and
backfill_channel_now mutually exclude per-channel and never advance the
same cursor concurrently (whichever started first). The interactive cap
moves to a separate _immediate_running counter so the scheduler's channels
don't false-trip it.
- Attribute the immediate backfill's spend: wrap it in
quota.attribute(user_id, VIDEOS_BACKFILL_FULL) (route passes user.id), so
it shows under the user's full-backfill action, not the OTHER/system bucket.
- Tests: the update_channel->background-task wiring + the demo guard
(scheduled for a real user, NOT for demo); distinct N/S/L counts (2 short
+ 1 live) so a short/live swap fails; extracted channelDetailPollMs with a
pure unit test of the poll-or-stop predicate.
- a11y: role="img" on the fully-synced charm so screen readers announce it.
- list_channels no longer computes the Normal/Shorts/Live aggregation the
frontend stopped reading when R14 dropped the N/S/L table column; the dead
ManagedChannel.count_* fields are gone too.
- Immediate "get full history" backfill now coordinates with the paced
scheduler via a small in-flight set: capped at 2 concurrent + per-channel
dedup, and run_deep_backfill skips a channel an immediate backfill owns —
killing the cursor race and bounding the quota-drain burst, without losing
the speed win.
- Add test_channel_backfill.py: the channelDetail N/S/L counts + the guard's
no-op / cap / scheduler-skip behaviour.
- Inline the syncedChannels useMemo (its dep is a fresh array each render, so
it never cached); fix the stale "~2-min ticks" comment (it's 10 minutes).
Clicking "get full history" now kicks an immediate background deep
backfill of that one channel (backfill_channel_now: own session, up to
~3000 videos in one pass, quota-guarded) instead of waiting for the
scheduler's paced 10-minute ticks (5 pages/channel/tick). The request
returns instantly and the channel page's live poll shows the count fill
in seconds. A very large channel's tail still falls to the scheduler
(deep_requested stays set).
The worst one was, again, the previous round's own fix.
- Round 7 stopped awaiting the HLS sweep so startup could serve immediately — which
also made it run CONCURRENTLY with playback, and `sweep_orphans` snapshotted the
live-session set BEFORE listing the root and rmtree'd outside the lock. Since
`start_session` reuses a directory path for the same (key, offset, tag), a stale
name can become a live session at any moment. Two guards now: nothing whose mtime
is newer than this process's start is ever deleted, and the live check + rmtree
happen together under the lock, one directory at a time.
- The sweep task is awaited after cancel, so shutdown can't leave it pending.
- `_rate_limiter` used a truthiness test on its timestamp, so a `now()` of exactly
0.0 read as "never ran" — unreachable in production, but `now` exists to be
injected and a test clock starting at 0 is the obvious choice.
- The RSS total-outage error now carries the first underlying error: "all N channels
failed" is also what ONE dead feed id looks like on a small instance, and a red
card the user can't act on is its own kind of dishonest.
- The WebKit fullscreen handling moved into `lib/fullscreen` and PlexPlayer uses it
too — it had the same unprefixed-only reads in three places, including an Escape
branch that would have closed the whole player instead of leaving fullscreen.
- The `--max` CSS comment claimed `inset: 0` did the sizing; Tailwind's `w-full` is
still on the element and wins. Comment now states the real mechanism.
Verified in real Chrome (the pane can't composite or do native fullscreen):
- F fills the 1920x889 viewport; the exit pill renders top-right on the letterbox
band and overlaps NO YouTube control — the collision this round suspected does not
reproduce in this embed (its own buttons sit bottom-left / bottom-right).
- Four real ArrowDown presses while maximised: persisted volume 100 -> 80, with the
level flash visible on screen.
- A real click on the exit pill un-maximises and returns focus to the card.
- lib/fullscreen driven from a real click: enter (innerHeight 889 -> 1024), the
change listener fires exactly once per transition, exit returns to 889.
Backend suite 150 -> 158; the new sweep tests are mutation-checked, and they read
their cutoff from the filesystem rather than a clock (the container VM and the
bind-mounted host disagree by enough to make a clock-based assertion flaky).
Five of them sit where round 6 turned F from a Fullscreen API call into a CSS
maximise: the guards around it still asked `isFullscreen`, which that pivot made
permanently false.
- Arrow-key volume was dead in the maximised view: the scroll-deference checked
only `isFullscreenRef`, so the (now hidden but still scrollable) card kept
swallowing Up/Down. Both flags now count as "the video fills the screen".
Measured: 87px of scroll room on the focused card, ArrowDown moves 80 -> 75
and scrolls nothing.
- The maximised view had no way out once the pointer touched the video. Reaching
YouTube's controls means clicking into the cross-origin iframe, which takes
keyboard focus with it, and the re-arm paths (stage mouseleave, window focus)
need the pointer to leave the browser entirely while the stage fills it. So
the exit now lives on screen, in our DOM: a button that un-maximises AND pulls
focus back, re-arming every shortcut.
- WebKit fires only `webkitfullscreenchange`/`webkitFullscreenElement`, so on
Safari/iPadOS YouTube's own fullscreen never registered — no overlay yield, no
focus reclaim. One `fullscreenEl()` reader, both event spellings.
- The stage claimed `z-index: 9999`, the glass tuner's reserved level, for a job
that only needs to beat the modal chrome it is nested in (its siblings top out
at z-menu). Dropped to rail-level; nothing escapes the modal's stacking
context either way.
- `inset: 0` already sizes a fixed layer — the inherited `100vw`/`100vh`
over-constrained it, which is how a classic scrollbar gets a horizontal
overflow and a mobile URL bar hides the bottom edge (YouTube's control bar).
- The shortcut hint still promised "F: fullscreen" in both locales.
Backend, both "the fix stopped one layer short":
- The startup HLS sweep was awaited inside lifespan, so uvicorn still served
nothing until it finished — a thread doesn't help when nothing is listening
yet. Fire-and-forget task instead, cancelled on shutdown. Measured with 200
planted segments: "Application startup complete" now precedes "swept 1".
- An RSS poll where every channel failed returned normally, so the scheduler
recorded `ok` with the outage buried in the summary string. `failed == total`
now raises: a run that reached nothing is a failed run.
Suites: backend 147 -> 150.
- run_ffmpeg stops the process from ONE exit path (`except BaseException`), so
an exception out of on_progress (a DB write) can no longer orphan a live
ffmpeg after a 10s join on its pipes; _stop is now a no-op on a dead process
- both run_ffmpeg callbacks are throttled to ~1s (they were a DB round-trip per
output line: ~10^5 queries on a long re-encode), mirroring the download hook
- volume arithmetic extracted to lib/playerVolume with unit tests; a corrupt
level now falls back to the LAST GOOD one instead of full blast, and the
sanitising happens where the stored value enters
- RSS is back to two except clauses (the shared tail already sits after the try)
- new tests: callback-exception cleanup, the two throttles; the fake Popen grew
poll() and its wait count is now asserted
Mutation-checked: each new group goes red when its fix is reverted.
- Feed memoizes onClose (like Playlists already did): the inline closure re-ran
PlayerModal's keydown effect on every Feed render, which since the debounce
landed also flushed the volume write and re-stole focus
- the player reads its volume with readAccountMerged again — a partial stored
object made the level undefined -> NaN -> a persisted null; applyVolume also
refuses a non-finite level outright
- that read moved into a lazy useState initializer (was running per render)
- the two RSS except branches share one tail, so a counter can't drift again
- run_ffmpeg tests cover the SIGTERM->SIGKILL escalation and the lingering-
process path; dropped an assertion that only restated the fixture
- storage.ts documents the actual { volume } shape (0 IS muted)
- the sweep guard anchors the rating_key to digits: the round-2 widening had
degenerated to "anything ending in _<digits>" (backup_2024, release_10)
- RSS counts apply-side failures too, so a read-only database no longer reports
a wholly failed poll as "ok — new=0, failed=0"
- the player reads/writes its volume directly (readAccount/writeAccount): the
persisted-object hook rendered nothing and cost a second write per settle
- new tests: the sweep guard's accept/reject table, and run_ffmpeg's threading
(drain, cancel while silent, stall watchdog, stderr tail on a nonzero exit)
- the breaker sentinel is checked one way (isinstance) at all four sites
- run_rss_poll is typed dict[str, int]
All three new test groups were mutation-checked: reverting each fix turns them
red (or, for the cancel/stall pair, hangs — which is the bug itself).
- the volume keys no longer defer to a scrollable card while in fullscreen
(mirrored into a ref: the keydown handler is bound once and state would be stale)
- the debounced volume flush writes straight to storage; persisting inside a
setState updater is not guaranteed to run for an unmounting component
- the HLS sweep also matches the original {key}_{start} directory naming, which
is what the oldest leftovers on disk actually use
- run_rss_poll returns {new, failed} so an egress outage stops rendering as
"ok — 0" in the scheduler card and the audit log; /api/sync/rss reports it too
- drop the unused poll_rss_channel (it silently gained a new exception)
- _Breaker.run is generic, so a call site keeps its item-id type
- test helper instead of the generator-throw idiom
- never prune subscriptions on an empty fetch: one anomalous empty-but-200
response wiped every subscription (and its deep_requested flag) in one run
- push_playlist: stop after 5 consecutive write failures — a quota-dead or
scope-revoked account otherwise burns 50 units per remaining item
- RSS: raise RssError instead of returning [] on a transport/HTTP failure, so a
failed poll no longer stamps last_rss_at and a 404 feed stops looking healthy
- worker: exit when the schema never appears instead of "continuing" into a
crash one call later
- Extract runner.token_users(db): the "all users with a stored refresh token"
query was spelled 3× (get_service_user, run_subscription_resync,
sync_all_playlists). get_service_user now reuses it ([0]); sync_all_playlists
imports it from runner (no cycle — runner doesn't import playlists); the now-
unused OAuthToken import is dropped there.
- scheduler.py: _job("rss_poll", lambda db: run_rss_poll(db)) → _job("rss_poll",
run_rss_poll) — the lambda was dead indirection; all sibling jobs pass the
callable directly and _job calls fn(db).
Behavior-neutral. ruff clean, localdev boots, re-review clean.
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
Register 8 more keys in the sysconfig registry (quota daily budget + backfill
reserve, recent-backfill window, shorts-probe params, enrich/autotag batch sizes)
and route their reads through the DB-override-or-env resolver at every call site
(quota.py, sync/runner.py, sync/videos.py, sync/autotag.py, sync/maintenance.py,
routes/scheduler.py). All read sites already had a db session in scope. Reads are
read-through (no cache), matching app.state; admin can tune them live, no redeploy.
Every background job now reports progress, so the dashboard can show a live
bar for any run (not only enrich/backfill/shorts/maintenance): rss_poll and
subscriptions (runner), autotag, and playlist_sync gained progress.report
calls over their loops.
RSS polling now fetches the channel feeds concurrently (16 workers) instead
of one-at-a-time — a slow/unreachable feed no longer blocks the rest, cutting
a full poll of the catalogue from minutes to seconds. Feed fetches are
network-bound and run in the pool; DB inserts stay on the session's thread
(SQLAlchemy sessions aren't thread-safe), applied as each fetch completes.
Split the DB-write half of poll_rss_channel into apply_rss_feed so the fetch
and apply phases compose cleanly.
Add per-job "Run now" buttons and a "Start all now" button to the admin Scheduler
dashboard (admin-gated endpoints). Triggers run the job in a background thread
independent of its interval, refusing a concurrent run (409). While running, the
long jobs (maintenance, enrich, backfill, shorts) report live progress through a
decoupled contextvar sink, shown as a progress bar on the job row via the existing
4s poll. A manually-triggered run posts a completion notification to the triggering
admin's inbox (scheduled runs stay silent to avoid spam); the inbox renders the
"scheduler" type trilingually from type+data. While here, give the maintenance job
its missing dashboard label/description in all three languages.
enrich_pending only touches enriched_at IS NULL, so a video first seen
while live was stamped live + duration-null and never revisited — staying
'live' with no duration forever after the broadcast ended. Add refresh_live
(run after each enrich pass) that re-fetches anything still live/upcoming,
plus just-ended was_live videos that haven't got their duration yet, until
they settle. Cheap: videos.list is 1 unit per 50 ids.
A channel whose stored uploads already meet or exceed YouTube's advertised
video_count holds its whole history, but backfill_done could stay false forever
when the deep cursor never reached the end (e.g. a small channel that was never
deep-requested, so the demand-driven deep job never ran). Such channels nagged
as 'needs full history' despite having every video. Add reconcile_full_history()
(idempotent, no quota) and run it at the end of each recent/deep backfill cycle
so backfill_done self-heals.
Per-user opt-in to full-history (deep) backfill so a new user's unique
channels no longer trigger a big shared-quota burst.
- migration 0007: Subscription.deep_requested (default false); seed admins'
existing subscriptions to preserve today's global-backfill behaviour
- run_deep_backfill is now demand-driven: only channels at least one user has
requested are deep-backfilled; recent backfill stays unconditional (cheap)
- estimate_deep_backfill ETA helper (quota-bound) surfaced in /api/sync/my-status
- POST /api/sync/deep-all to opt all my channels in; PATCH channels accepts
deep_requested
- UI: per-channel Full history toggle, Backfill everything action, deep
progress + ETA in Channels header and Settings - Sync
- uvicorn --log-config (log_config.json): timestamped formatters for uvicorn
access/error and the app's own "subfeed" loggers (ms precision)
- Log key activity: startup/shutdown, login/denied, token refresh, YouTube API
errors, subscription imports, sync pause/resume
- Background sync jobs no longer swallow errors silently — failures in RSS poll,
backfill, enrichment, autotag and resync are logged with tracebacks
- Language detection: classify one cleaned+concatenated blob (strip emoji,
@mentions, #tags, numbers, punctuation); fixes caps/emoji-heavy channels
(e.g. Nessaj -> Hungarian, no more bogus Chinese/Korean)
- Feed now joins the user's subscriptions, so unsubscribing on YouTube removes a
channel from the feed; periodic subscription re-sync job picks up changes
- Watched/Saved/Hidden views ignore the Shorts/live default-hiding so the full
set is visible (fixes hidden videos missing from the Hidden view)
- Persist feed filters + search across reloads (localStorage)
- 3D polish: cards lift with shadow on hover; chips/buttons get depth and a
press effect; undo toast lasts longer
- Shorts: confirm via youtube.com/shorts/<id> probe (SOCS cookie bypasses the
consent redirect) instead of a 60s heuristic; concurrent probing, shorts_probed
flag, scheduled refinement (migration 0005)
- Search: match title + channel name only (descriptions caused noisy results)
- Faceted tag filtering: AND across categories (language AND topic narrows),
OR within a category; any/all toggle applies to topics
- Language detection: majority vote over individual titles (fixes misdetections
like multipoleguy -> English; drops bogus Polish/Romanian)
- Login: drop forced consent so returning sign-in is quick (select_account)
- Feed cards: clickable channel name (opens channel), persistent saved badge,
undo toast on hide, Hidden view to restore; tag chips show counts in tooltip
- Free per-channel RSS reader for quota-less fresh-video detection
- Recent-first backfill (configurable: 100 videos / 1 year) plus resumable deep
backfill from the uploads playlist
- Enrichment via videos.list: duration, view/like counts, category, topics,
language, Shorts heuristic and livestream/premiere classification
- Reusable sync runners + APScheduler jobs (rss / enrich / backfill), all
quota-aware with a reserve so backfill never starves fresh enrichment
- Manual triggers: POST /api/sync/{rss,backfill,enrich}
- Exact insert counting via RETURNING with in-batch de-duplication
peter