feat(scheduler): a channel's "get full history" backfill is a recorded run, and a row knows its last three runs

The one-channel deep backfill ran as a FastAPI BackgroundTask: what it fetched, what it spent and
how it failed reached only the log. It is now a one-shot run on the scheduler (`channel_backfill`,
trigger ad_hoc, actor = the user, target = the channel) with its quota tagged to the run. Its
per-channel exclusion with the scheduled backfill and the two-run cap are unchanged, and a run
they turn away says why in its result instead of returning nothing. A failure is no longer
swallowed: the run records it.

The scheduler snapshot carries each job's last three runs (one windowed query), so the page can
show an interrupted or skipped run instead of the latest outcome alone.
This commit is contained in:
2026-10-18 01:05:24 +02:00
parent 911527a548
commit 27c2fcbd2e
8 changed files with 256 additions and 55 deletions
+22 -21
View File
@@ -437,7 +437,7 @@ def run_deep_backfill(db: Session, max_channels: int = 10, max_pages: int = 5) -
# Channels being deep-backfilled RIGHT NOW by EITHER path — the interactive backfill_channel_now (a
# FastAPI BackgroundTasks threadpool thread) OR the scheduler's run_deep_backfill (its own thread).
# one-shot run on the scheduler's pool, see scheduler.run_ad_hoc) OR the scheduler's run_deep_backfill.
# BOTH register the channel they're paging (under the lock) so the other skips it: no two runs page
# the same channel's `backfill_cursor` concurrently, whichever started first. `_immediate_running`
# counts only the interactive path, capped so one user's burst can't drain the shared budget (the
@@ -448,9 +448,9 @@ _backfill_lock = threading.Lock()
_MAX_CONCURRENT_IMMEDIATE = 2
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> None:
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, in a
background task with its OWN session, so the full catalogue lands in seconds instead of over the
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> dict:
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, as its
own recorded run with its OWN session, so the full catalogue lands in seconds instead of over the
paced scheduler's 10-minute ticks (max_pages=5/channel/tick). max_pages is high enough to finish
any realistic channel in a single run (400 pages = 20 000 videos; e.g. BBC Earth's ~3400 = 69
pages — the old 60-page cap stopped one page short of large channels, so their tail dribbled in
@@ -459,34 +459,35 @@ def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) ->
Quota-guarded, capped to _MAX_CONCURRENT_IMMEDIATE interactive runs so one user can't drain the
shared backfill budget by opting many channels in at once (the overflow, and any channel too large
to finish under the reserve, is finished by the scheduler since deep_requested stays set), and
mutually excluded per-channel with the scheduler via _backfill_in_progress. Best-effort — logs and
swallows failures."""
mutually excluded per-channel with the scheduler via _backfill_in_progress.
Returns what the run did, as a job result: the counts, or `{"skipped": <why>}` when it left the
channel to the scheduler. A failure raises — the run that wraps this records it."""
global _immediate_running
with _backfill_lock:
if channel_id in _backfill_in_progress or _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
return # already backfilling this channel, or too many interactive runs — leave it to the scheduler
if channel_id in _backfill_in_progress:
return {"skipped": "channel busy"}
if _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
return {"skipped": "at capacity"}
_backfill_in_progress.add(channel_id)
_immediate_running += 1
try:
with SessionLocal() as db:
user = get_service_user(db)
if user is None:
return
return {"skipped": "no service user"}
channel = db.get(Channel, channel_id)
if channel is None or channel.backfill_done or not channel.uploads_playlist_id:
return
return {"skipped": "nothing to do"}
if quota.backfill_reserve_reached():
return
try:
# Attribute the spend to the requesting user under the full-backfill action, so it
# shows up correctly in the usage breakdown (not the default OTHER/system bucket).
with quota.attribute(user_id, quota.QuotaAction.VIDEOS_BACKFILL_FULL):
with YouTubeClient(db, user) as yt:
backfill_channel_deep(db, yt, channel, max_pages)
reconcile_full_history(db)
except Exception:
db.rollback()
log.exception("Immediate backfill failed for channel %s", channel_id)
return {"skipped": "low quota"}
# Attribute the spend to the requesting user under the full-backfill action, so it
# shows up correctly in the usage breakdown (not the default OTHER/system bucket).
with quota.attribute(user_id, quota.QuotaAction.VIDEOS_BACKFILL_FULL):
with YouTubeClient(db, user) as yt:
added = backfill_channel_deep(db, yt, channel, max_pages)
reconciled = reconcile_full_history(db)
return {"videos_added": added, "reconciled_full": reconciled}
finally:
with _backfill_lock:
_backfill_in_progress.discard(channel_id)