refactor(backfill): centralize the reserve-yield guard and drop the per-page connection hold
Addresses the code-review findings on the immediate full-history change: - Extract quota.backfill_reserve_reached(): the remaining/reserve compare was open-coded at seven sites (recent/deep/immediate backfill, explore, rolling revalidation, and the new per-page guard). It reads in its OWN short-lived session (like record_usage/log_action), so a per-page guard no longer holds the caller's transaction open across the YouTube fetch that follows — the 400-page immediate run no longer pins a pooled connection idle-in-transaction through each network round-trip. Migrated all call sites; dropped the now-dead sysconfig imports in channels.py and maintenance.py. - Simplify the deep-backfill loop: channel.backfill_done = not page_token (done is False on entry) plus one break, replacing the doubled `if not page_token`. - Tests patch the backfill_reserve_reached seam directly instead of mocking remaining_today internals, removing the fragile dead-value budget iterator. Full backend suite green; ruff clean.
This commit is contained in:
@@ -168,9 +168,8 @@ def backfill_channel_deep(
|
||||
return 0
|
||||
page_token = channel.backfill_cursor
|
||||
total = 0
|
||||
reserve = sysconfig.get_int(db, "backfill_quota_reserve")
|
||||
for _ in range(max_pages):
|
||||
if quota.remaining_today(db) <= reserve:
|
||||
if quota.backfill_reserve_reached():
|
||||
break
|
||||
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
|
||||
rows = [
|
||||
@@ -183,8 +182,7 @@ def backfill_channel_deep(
|
||||
total += insert_stubs(db, rows)
|
||||
page_token = data.get("nextPageToken")
|
||||
channel.backfill_cursor = page_token
|
||||
if not page_token:
|
||||
channel.backfill_done = True
|
||||
channel.backfill_done = not page_token
|
||||
db.add(channel)
|
||||
db.commit()
|
||||
if not page_token:
|
||||
|
||||
Reference in New Issue
Block a user