fix(backfill): finish large channels in one immediate full-history run

The user-triggered "get full history" (backfill_channel_now) capped at
max_pages=60 = 3000 videos. A channel just over that (BBC Earth: 3403
uploads = 69 pages) stopped one page short, so its tail dribbled in over
the scheduler's paced 10-minute ticks (250 videos/tick) and the button
spinner looked stuck for ~18 minutes with no visible progress.

- Raise the immediate cap to max_pages=400 (20 000 videos) so any
  realistic channel completes in the single background run.
- Make a high cap safe: backfill_channel_deep now re-checks the shared
  quota before each page and yields at the backfill reserve (previously
  quota was checked once by the caller before the whole run), and commits
  the advanced cursor after every page so a restart resumes instead of
  re-fetching — and re-charging quota for — pages already stored.
- Channel page: while loading, the button shows the live stored/total
  count and a tooltip explaining the tail finishes in the background, so a
  very large channel's paced remainder reads as progress, not a hang.

Adds deep-backfill paging tests (per-page cursor commit + resume, and the
per-page quota yield).
This commit is contained in:
2026-08-15 17:13:52 +02:00
parent a7b4b2ed79
commit e3ee1baa94
6 changed files with 146 additions and 14 deletions
+15 -3
View File
@@ -422,13 +422,18 @@ export default function ChannelPage({
</div>
)}
{/* Only the recent subset is stored → offer to pull the channel's full back-catalogue.
Once queued (deep_requested) it shows as loading; hidden once the backfill is done. */}
Once queued (deep_requested) it shows as loading; hidden once the backfill is done.
While loading, the label carries the live stored/total count (the detail poll refreshes
it) so a large channel's paced tail reads as visible progress rather than a stuck
spinner. */}
{ch?.subscribed && !ch.backfill_done && !me.is_demo && (
<button
onClick={() => !ch.deep_requested && deepBackfill.mutate()}
disabled={ch.deep_requested || deepBackfill.isPending}
title={
ch.deep_requested ? t("channel.fullHistoryLoading") : t("channel.fullHistory")
ch.deep_requested
? t("channel.fullHistoryLoadingHint")
: t("channel.fullHistory")
}
className="inline-flex items-center gap-1.5 px-3 py-2 rounded-lg text-sm border border-border text-muted hover:text-accent hover:border-accent disabled:opacity-60 disabled:hover:text-muted disabled:hover:border-border transition"
>
@@ -438,7 +443,14 @@ export default function ChannelPage({
<History className="w-4 h-4" />
)}
<span className="hidden sm:inline whitespace-nowrap">
{ch.deep_requested ? t("channel.fullHistoryLoading") : t("channel.fullHistory")}
{!ch.deep_requested
? t("channel.fullHistory")
: ch.video_count != null && ch.stored_videos < ch.video_count
? t("channel.fullHistoryProgress", {
stored: ch.stored_videos,
total: ch.video_count,
})
: t("channel.fullHistoryLoading")}
</span>
</button>
)}
@@ -29,6 +29,8 @@
"showInFeed": "Show in my feed",
"fullHistory": "Full history",
"fullHistoryLoading": "Fetching history…",
"fullHistoryProgress": "Fetching history… {{stored}} / {{total}}",
"fullHistoryLoadingHint": "Fetching the full back-catalogue — videos appear as they're found; a very large channel's tail is finished in the background.",
"fullHistoryQueued": "Fetching this channel's full history — new videos will appear as they're found.",
"fullHistoryFailed": "Couldn't start the full-history fetch."
}
@@ -29,6 +29,8 @@
"showInFeed": "Mutasd a hírfolyamomban",
"fullHistory": "Teljes előzmény",
"fullHistoryLoading": "Előzmény letöltése…",
"fullHistoryProgress": "Előzmény letöltése… {{stored}} / {{total}}",
"fullHistoryLoadingHint": "A teljes előzmény letöltése folyamatban — a videók feltűnnek, ahogy előkerülnek; egy nagyon nagy csatorna maradékát a háttérben fejezi be.",
"fullHistoryQueued": "A csatorna teljes előzményének letöltése elindult — az új videók feltűnnek, ahogy előkerülnek.",
"fullHistoryFailed": "Nem sikerült elindítani a teljes előzmény letöltését."
}