Commit Graph
18 Commits
Author SHA1 Message Date
peter a5336d20e0 fix(r5): address the seventh /code-review high round — 7 findings
Five of them sit where round 6 turned F from a Fullscreen API call into a CSS
maximise: the guards around it still asked `isFullscreen`, which that pivot made
permanently false.

- Arrow-key volume was dead in the maximised view: the scroll-deference checked
  only `isFullscreenRef`, so the (now hidden but still scrollable) card kept
  swallowing Up/Down. Both flags now count as "the video fills the screen".
  Measured: 87px of scroll room on the focused card, ArrowDown moves 80 -> 75
  and scrolls nothing.
- The maximised view had no way out once the pointer touched the video. Reaching
  YouTube's controls means clicking into the cross-origin iframe, which takes
  keyboard focus with it, and the re-arm paths (stage mouseleave, window focus)
  need the pointer to leave the browser entirely while the stage fills it. So
  the exit now lives on screen, in our DOM: a button that un-maximises AND pulls
  focus back, re-arming every shortcut.
- WebKit fires only `webkitfullscreenchange`/`webkitFullscreenElement`, so on
  Safari/iPadOS YouTube's own fullscreen never registered — no overlay yield, no
  focus reclaim. One `fullscreenEl()` reader, both event spellings.
- The stage claimed `z-index: 9999`, the glass tuner's reserved level, for a job
  that only needs to beat the modal chrome it is nested in (its siblings top out
  at z-menu). Dropped to rail-level; nothing escapes the modal's stacking
  context either way.
- `inset: 0` already sizes a fixed layer — the inherited `100vw`/`100vh`
  over-constrained it, which is how a classic scrollbar gets a horizontal
  overflow and a mobile URL bar hides the bottom edge (YouTube's control bar).
- The shortcut hint still promised "F: fullscreen" in both locales.

Backend, both "the fix stopped one layer short":
- The startup HLS sweep was awaited inside lifespan, so uvicorn still served
  nothing until it finished — a thread doesn't help when nothing is listening
  yet. Fire-and-forget task instead, cancelled on shutdown. Measured with 200
  planted segments: "Application startup complete" now precedes "swept 1".
- An RSS poll where every channel failed returned normally, so the scheduler
  recorded `ok` with the outage buried in the summary string. `failed == total`
  now raises: a run that reached nothing is a failed run.

Suites: backend 147 -> 150.
2026-07-24 03:41:29 +02:00
peter c4942eb590 fix(r5): address the fifth /code-review high round — 6 findings
- run_ffmpeg stops the process from ONE exit path (`except BaseException`), so
  an exception out of on_progress (a DB write) can no longer orphan a live
  ffmpeg after a 10s join on its pipes; _stop is now a no-op on a dead process
- both run_ffmpeg callbacks are throttled to ~1s (they were a DB round-trip per
  output line: ~10^5 queries on a long re-encode), mirroring the download hook
- volume arithmetic extracted to lib/playerVolume with unit tests; a corrupt
  level now falls back to the LAST GOOD one instead of full blast, and the
  sanitising happens where the stored value enters
- RSS is back to two except clauses (the shared tail already sits after the try)
- new tests: callback-exception cleanup, the two throttles; the fake Popen grew
  poll() and its wait count is now asserted

Mutation-checked: each new group goes red when its fix is reverted.
2026-07-24 01:50:50 +02:00
peter fc82c8b47a fix(r5): address the fourth /code-review high round — 6 findings
- Feed memoizes onClose (like Playlists already did): the inline closure re-ran
  PlayerModal's keydown effect on every Feed render, which since the debounce
  landed also flushed the volume write and re-stole focus
- the player reads its volume with readAccountMerged again — a partial stored
  object made the level undefined -> NaN -> a persisted null; applyVolume also
  refuses a non-finite level outright
- that read moved into a lazy useState initializer (was running per render)
- the two RSS except branches share one tail, so a counter can't drift again
- run_ffmpeg tests cover the SIGTERM->SIGKILL escalation and the lingering-
  process path; dropped an assertion that only restated the fixture
- storage.ts documents the actual { volume } shape (0 IS muted)
2026-07-24 00:45:42 +02:00
peter c0db80d333 fix(r5): address the third /code-review high round — 7 findings
- the sweep guard anchors the rating_key to digits: the round-2 widening had
  degenerated to "anything ending in _<digits>" (backup_2024, release_10)
- RSS counts apply-side failures too, so a read-only database no longer reports
  a wholly failed poll as "ok — new=0, failed=0"
- the player reads/writes its volume directly (readAccount/writeAccount): the
  persisted-object hook rendered nothing and cost a second write per settle
- new tests: the sweep guard's accept/reject table, and run_ffmpeg's threading
  (drain, cancel while silent, stall watchdog, stderr tail on a nonzero exit)
- the breaker sentinel is checked one way (isinstance) at all four sites
- run_rss_poll is typed dict[str, int]

All three new test groups were mutation-checked: reverting each fix turns them
red (or, for the cancel/stall pair, hangs — which is the bug itself).
2026-07-24 00:17:09 +02:00
peter 53ff61653d fix(r5): address the second /code-review high round — 7 findings
- the volume keys no longer defer to a scrollable card while in fullscreen
  (mirrored into a ref: the keydown handler is bound once and state would be stale)
- the debounced volume flush writes straight to storage; persisting inside a
  setState updater is not guaranteed to run for an unmounting component
- the HLS sweep also matches the original {key}_{start} directory naming, which
  is what the oldest leftovers on disk actually use
- run_rss_poll returns {new, failed} so an egress outage stops rendering as
  "ok — 0" in the scheduler card and the audit log; /api/sync/rss reports it too
- drop the unused poll_rss_channel (it silently gained a new exception)
- _Breaker.run is generic, so a call site keeps its item-id type
- test helper instead of the generator-throw idiom
2026-07-23 23:59:58 +02:00
peter 4209f6bb24 fix(sync): R5 S2 — sync guards
- never prune subscriptions on an empty fetch: one anomalous empty-but-200
  response wiped every subscription (and its deep_requested flag) in one run
- push_playlist: stop after 5 consecutive write failures — a quota-dead or
  scope-revoked account otherwise burns 50 units per remaining item
- RSS: raise RssError instead of returning [] on a transport/HTTP failure, so a
  failed poll no longer stamps last_rss_at and a 404 feed stops looking healthy
- worker: exit when the schema never appears instead of "continuing" into a
  crash one call later
2026-07-23 22:36:33 +02:00
peter 1fc8214f8f chore(scheduler): dedup token-user query + drop pointless rss lambda
- Extract runner.token_users(db): the "all users with a stored refresh token"
  query was spelled 3× (get_service_user, run_subscription_resync,
  sync_all_playlists). get_service_user now reuses it ([0]); sync_all_playlists
  imports it from runner (no cycle — runner doesn't import playlists); the now-
  unused OAuthToken import is dropped there.
- scheduler.py: _job("rss_poll", lambda db: run_rss_poll(db)) → _job("rss_poll",
  run_rss_poll) — the lambda was dead indirection; all sibling jobs pass the
  callable directly and _job calls fn(db).

Behavior-neutral. ruff clean, localdev boots, re-review clean.
2026-07-11 19:44:53 +02:00
peter 74830eb0eb chore: rename remaining subfeed references to siftlode
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
2026-06-21 06:53:12 +02:00
peter e6bcf5ba1e feat(config): move operational params to DB (quota/backfill/shorts/batch)
Register 8 more keys in the sysconfig registry (quota daily budget + backfill
reserve, recent-backfill window, shorts-probe params, enrich/autotag batch sizes)
and route their reads through the DB-override-or-env resolver at every call site
(quota.py, sync/runner.py, sync/videos.py, sync/autotag.py, sync/maintenance.py,
routes/scheduler.py). All read sites already had a db session in scope. Reads are
read-through (no cache), matching app.state; admin can tune them live, no redeploy.
2026-06-19 13:07:45 +02:00
peter e0971a23ec feat(scheduler): report progress from every job + parallelize RSS polling
Every background job now reports progress, so the dashboard can show a live
bar for any run (not only enrich/backfill/shorts/maintenance): rss_poll and
subscriptions (runner), autotag, and playlist_sync gained progress.report
calls over their loops.

RSS polling now fetches the channel feeds concurrently (16 workers) instead
of one-at-a-time — a slow/unreachable feed no longer blocks the rest, cutting
a full poll of the catalogue from minutes to seconds. Feed fetches are
network-bound and run in the pool; DB inserts stay on the session's thread
(SQLAlchemy sessions aren't thread-safe), applied as each fetch completes.
Split the DB-write half of poll_rss_channel into apply_rss_feed so the fetch
and apply phases compose cleanly.
2026-06-19 02:43:37 +02:00
peter ed4194a8d3 feat(scheduler): admin run-now/run-all triggers + live progress + completion notices
Add per-job "Run now" buttons and a "Start all now" button to the admin Scheduler
dashboard (admin-gated endpoints). Triggers run the job in a background thread
independent of its interval, refusing a concurrent run (409). While running, the
long jobs (maintenance, enrich, backfill, shorts) report live progress through a
decoupled contextvar sink, shown as a progress bar on the job row via the existing
4s poll. A manually-triggered run posts a completion notification to the triggering
admin's inbox (scheduled runs stay silent to avoid spam); the inbox renders the
"scheduler" type trilingually from type+data. While here, give the maintenance job
its missing dashboard label/description in all three languages.
2026-06-18 04:01:10 +02:00
peter b114dae39b fix(sync): re-enrich live videos so ended streams gain duration
enrich_pending only touches enriched_at IS NULL, so a video first seen
while live was stamped live + duration-null and never revisited — staying
'live' with no duration forever after the broadcast ended. Add refresh_live
(run after each enrich pass) that re-fetches anything still live/upcoming,
plus just-ended was_live videos that haven't got their duration yet, until
they settle. Cheap: videos.list is 1 unit per 50 ids.
2026-06-16 10:06:58 +02:00
peter 75d122de75 fix(backfill): auto-complete full history when stored >= video_count
A channel whose stored uploads already meet or exceed YouTube's advertised
video_count holds its whole history, but backfill_done could stay false forever
when the deep cursor never reached the end (e.g. a small channel that was never
deep-requested, so the demand-driven deep job never ran). Such channels nagged
as 'needs full history' despite having every video. Add reconcile_full_history()
(idempotent, no quota) and run it at the end of each recent/deep backfill cycle
so backfill_done self-heals.
2026-06-14 19:01:04 +02:00
peter beb961c021 feat(m5d): demand-driven deep backfill + per-user ETA
Per-user opt-in to full-history (deep) backfill so a new user's unique
channels no longer trigger a big shared-quota burst.

- migration 0007: Subscription.deep_requested (default false); seed admins'
  existing subscriptions to preserve today's global-backfill behaviour
- run_deep_backfill is now demand-driven: only channels at least one user has
  requested are deep-backfilled; recent backfill stays unconditional (cheap)
- estimate_deep_backfill ETA helper (quota-bound) surfaced in /api/sync/my-status
- POST /api/sync/deep-all to opt all my channels in; PATCH channels accepts
  deep_requested
- UI: per-channel Full history toggle, Backfill everything action, deep
  progress + ETA in Channels header and Settings - Sync
2026-06-11 23:07:09 +02:00
peter f6c5488566 chore: structured timestamped logging across the backend
- uvicorn --log-config (log_config.json): timestamped formatters for uvicorn
  access/error and the app's own "subfeed" loggers (ms precision)
- Log key activity: startup/shutdown, login/denied, token refresh, YouTube API
  errors, subscription imports, sync pause/resume
- Background sync jobs no longer swallow errors silently — failures in RSS poll,
  backfill, enrichment, autotag and resync are logged with tracebacks
2026-06-11 04:26:18 +02:00
peter e07a37622d fix: feedback round 2 — language, subscriptions, feed scope, UX
- Language detection: classify one cleaned+concatenated blob (strip emoji,
  @mentions, #tags, numbers, punctuation); fixes caps/emoji-heavy channels
  (e.g. Nessaj -> Hungarian, no more bogus Chinese/Korean)
- Feed now joins the user's subscriptions, so unsubscribing on YouTube removes a
  channel from the feed; periodic subscription re-sync job picks up changes
- Watched/Saved/Hidden views ignore the Shorts/live default-hiding so the full
  set is visible (fixes hidden videos missing from the Hidden view)
- Persist feed filters + search across reloads (localStorage)
- 3D polish: cards lift with shadow on hover; chips/buttons get depth and a
  press effect; undo toast lasts longer
2026-06-11 03:28:45 +02:00
peter 8c245e986f fix: address reader-UI feedback (shorts, search, tags, login, UX)
- Shorts: confirm via youtube.com/shorts/<id> probe (SOCS cookie bypasses the
  consent redirect) instead of a 60s heuristic; concurrent probing, shorts_probed
  flag, scheduled refinement (migration 0005)
- Search: match title + channel name only (descriptions caused noisy results)
- Faceted tag filtering: AND across categories (language AND topic narrows),
  OR within a category; any/all toggle applies to topics
- Language detection: majority vote over individual titles (fixes misdetections
  like multipoleguy -> English; drops bogus Polish/Romanian)
- Login: drop forced consent so returning sign-in is quick (select_account)
- Feed cards: clickable channel name (opens channel), persistent saved badge,
  undo toast on hide, Hidden view to restore; tag chips show counts in tooltip
2026-06-11 03:07:49 +02:00
peter cff1d23071 feat: M2 (part 2) — RSS poller, backfill, enrichment, scheduler
- Free per-channel RSS reader for quota-less fresh-video detection
- Recent-first backfill (configurable: 100 videos / 1 year) plus resumable deep
  backfill from the uploads playlist
- Enrichment via videos.list: duration, view/like counts, category, topics,
  language, Shorts heuristic and livestream/premiere classification
- Reusable sync runners + APScheduler jobs (rss / enrich / backfill), all
  quota-aware with a reserve so backfill never starves fresh enrichment
- Manual triggers: POST /api/sync/{rss,backfill,enrich}
- Exact insert counting via RETURNING with in-batch de-duplication
2026-06-11 01:36:41 +02:00