fix(backfill): finish large channels in one immediate full-history run

The user-triggered "get full history" (backfill_channel_now) capped at
max_pages=60 = 3000 videos. A channel just over that (BBC Earth: 3403
uploads = 69 pages) stopped one page short, so its tail dribbled in over
the scheduler's paced 10-minute ticks (250 videos/tick) and the button
spinner looked stuck for ~18 minutes with no visible progress.

- Raise the immediate cap to max_pages=400 (20 000 videos) so any
  realistic channel completes in the single background run.
- Make a high cap safe: backfill_channel_deep now re-checks the shared
  quota before each page and yields at the backfill reserve (previously
  quota was checked once by the caller before the whole run), and commits
  the advanced cursor after every page so a restart resumes instead of
  re-fetching — and re-charging quota for — pages already stored.
- Channel page: while loading, the button shows the live stored/total
  count and a tooltip explaining the tail finishes in the background, so a
  very large channel's paced remainder reads as progress, not a hang.

Adds deep-backfill paging tests (per-page cursor commit + resume, and the
per-page quota yield).
This commit is contained in:
2026-08-15 17:13:52 +02:00
parent a7b4b2ed79
commit e3ee1baa94
6 changed files with 146 additions and 14 deletions
+11 -6
View File
@@ -267,14 +267,19 @@ _backfill_lock = threading.Lock()
_MAX_CONCURRENT_IMMEDIATE = 2
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 60) -> None:
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> None:
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, in a
background task with its OWN session, so the full catalogue lands in seconds instead of over the
paced scheduler's 10-minute ticks (max_pages=5/channel/tick). Quota-guarded, capped to
_MAX_CONCURRENT_IMMEDIATE interactive runs so one user can't drain the shared backfill budget by
opting many channels in at once (the overflow, and any very large channel's tail, is finished by
the scheduler since deep_requested stays set), and mutually excluded per-channel with the scheduler
via _backfill_in_progress. Best-effort — logs and swallows failures."""
paced scheduler's 10-minute ticks (max_pages=5/channel/tick). max_pages is high enough to finish
any realistic channel in a single run (400 pages = 20 000 videos; e.g. BBC Earth's ~3400 = 69
pages — the old 60-page cap stopped one page short of large channels, so their tail dribbled in
over the scheduler's 10-minute ticks and looked stuck). backfill_channel_deep re-checks the shared
budget before each page and yields at the reserve, so a high cap still can't drain the day's quota.
Quota-guarded, capped to _MAX_CONCURRENT_IMMEDIATE interactive runs so one user can't drain the
shared backfill budget by opting many channels in at once (the overflow, and any channel too large
to finish under the reserve, is finished by the scheduler since deep_requested stays set), and
mutually excluded per-channel with the scheduler via _backfill_in_progress. Best-effort — logs and
swallows failures."""
global _immediate_running
with _backfill_lock:
if channel_id in _backfill_in_progress or _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
+15 -4
View File
@@ -10,7 +10,7 @@ from sqlalchemy import and_, func, or_, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app import progress, sysconfig
from app import progress, quota, sysconfig
from app.models import LIVE_OR_UPCOMING, Channel, Video
from app.titles import normalize_title
from app.youtube.client import YouTubeClient, best_thumbnail
@@ -156,12 +156,22 @@ def backfill_channel_recent(db: Session, yt: YouTubeClient, channel: Channel) ->
def backfill_channel_deep(
db: Session, yt: YouTubeClient, channel: Channel, max_pages: int = 5
) -> int:
"""Continue paging older uploads from the stored cursor (bounded per run)."""
"""Continue paging older uploads from the stored cursor (bounded per run).
Commits the advanced cursor after every page (not just at the end) so progress
is durable: a restart resumes from the last stored page instead of re-fetching —
and re-charging quota for — pages already ingested. Re-checks the shared budget
before each page and yields once it hits the backfill reserve, so a single large
channel can't drain the day's quota in one run even with a high max_pages; the
paced scheduler finishes the tail on later ticks (deep_requested stays set)."""
if channel.backfill_done or not channel.uploads_playlist_id:
return 0
page_token = channel.backfill_cursor
total = 0
reserve = sysconfig.get_int(db, "backfill_quota_reserve")
for _ in range(max_pages):
if quota.remaining_today(db) <= reserve:
break
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
rows = [
s
@@ -175,9 +185,10 @@ def backfill_channel_deep(
channel.backfill_cursor = page_token
if not page_token:
channel.backfill_done = True
db.add(channel)
db.commit()
if not page_token:
break
db.add(channel)
db.commit()
return total