fix(backfill): finish large channels in one immediate full-history run
The user-triggered "get full history" (backfill_channel_now) capped at max_pages=60 = 3000 videos. A channel just over that (BBC Earth: 3403 uploads = 69 pages) stopped one page short, so its tail dribbled in over the scheduler's paced 10-minute ticks (250 videos/tick) and the button spinner looked stuck for ~18 minutes with no visible progress. - Raise the immediate cap to max_pages=400 (20 000 videos) so any realistic channel completes in the single background run. - Make a high cap safe: backfill_channel_deep now re-checks the shared quota before each page and yields at the backfill reserve (previously quota was checked once by the caller before the whole run), and commits the advanced cursor after every page so a restart resumes instead of re-fetching — and re-charging quota for — pages already stored. - Channel page: while loading, the button shows the live stored/total count and a tooltip explaining the tail finishes in the background, so a very large channel's paced remainder reads as progress, not a hang. Adds deep-backfill paging tests (per-page cursor commit + resume, and the per-page quota yield).
This commit is contained in:
@@ -10,7 +10,7 @@ from sqlalchemy import and_, func, or_, select, update
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import progress, sysconfig
|
||||
from app import progress, quota, sysconfig
|
||||
from app.models import LIVE_OR_UPCOMING, Channel, Video
|
||||
from app.titles import normalize_title
|
||||
from app.youtube.client import YouTubeClient, best_thumbnail
|
||||
@@ -156,12 +156,22 @@ def backfill_channel_recent(db: Session, yt: YouTubeClient, channel: Channel) ->
|
||||
def backfill_channel_deep(
|
||||
db: Session, yt: YouTubeClient, channel: Channel, max_pages: int = 5
|
||||
) -> int:
|
||||
"""Continue paging older uploads from the stored cursor (bounded per run)."""
|
||||
"""Continue paging older uploads from the stored cursor (bounded per run).
|
||||
|
||||
Commits the advanced cursor after every page (not just at the end) so progress
|
||||
is durable: a restart resumes from the last stored page instead of re-fetching —
|
||||
and re-charging quota for — pages already ingested. Re-checks the shared budget
|
||||
before each page and yields once it hits the backfill reserve, so a single large
|
||||
channel can't drain the day's quota in one run even with a high max_pages; the
|
||||
paced scheduler finishes the tail on later ticks (deep_requested stays set)."""
|
||||
if channel.backfill_done or not channel.uploads_playlist_id:
|
||||
return 0
|
||||
page_token = channel.backfill_cursor
|
||||
total = 0
|
||||
reserve = sysconfig.get_int(db, "backfill_quota_reserve")
|
||||
for _ in range(max_pages):
|
||||
if quota.remaining_today(db) <= reserve:
|
||||
break
|
||||
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
|
||||
rows = [
|
||||
s
|
||||
@@ -175,9 +185,10 @@ def backfill_channel_deep(
|
||||
channel.backfill_cursor = page_token
|
||||
if not page_token:
|
||||
channel.backfill_done = True
|
||||
db.add(channel)
|
||||
db.commit()
|
||||
if not page_token:
|
||||
break
|
||||
db.add(channel)
|
||||
db.commit()
|
||||
return total
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user