refactor(backfill): centralize the reserve-yield guard and drop the per-page connection hold
Addresses the code-review findings on the immediate full-history change: - Extract quota.backfill_reserve_reached(): the remaining/reserve compare was open-coded at seven sites (recent/deep/immediate backfill, explore, rolling revalidation, and the new per-page guard). It reads in its OWN short-lived session (like record_usage/log_action), so a per-page guard no longer holds the caller's transaction open across the YouTube fetch that follows — the 400-page immediate run no longer pins a pooled connection idle-in-transaction through each network round-trip. Migrated all call sites; dropped the now-dead sysconfig imports in channels.py and maintenance.py. - Simplify the deep-backfill loop: channel.backfill_done = not page_token (done is False on entry) plus one break, replacing the doubled `if not page_token`. - Tests patch the backfill_reserve_reached seam directly instead of mocking remaining_today internals, removing the fragile dead-value budget iterator. Full backend suite green; ruff clean.
This commit is contained in:
@@ -183,7 +183,7 @@ def run_recent_backfill(
|
||||
videos_added = 0
|
||||
with YouTubeClient(db, user) as yt:
|
||||
for channel in channels:
|
||||
if quota.remaining_today(db) <= sysconfig.get_int(db, "backfill_quota_reserve"):
|
||||
if quota.backfill_reserve_reached():
|
||||
break
|
||||
try:
|
||||
videos_added += backfill_channel_recent(db, yt, channel)
|
||||
@@ -229,7 +229,7 @@ def run_deep_backfill(db: Session, max_channels: int = 10, max_pages: int = 5) -
|
||||
videos_added = 0
|
||||
with YouTubeClient(db, user) as yt:
|
||||
for channel in channels:
|
||||
if quota.remaining_today(db) <= sysconfig.get_int(db, "backfill_quota_reserve"):
|
||||
if quota.backfill_reserve_reached():
|
||||
break
|
||||
# Claim the channel (or skip it if an interactive backfill already owns it) so the two
|
||||
# never advance the same cursor concurrently — symmetric with backfill_channel_now.
|
||||
@@ -294,7 +294,7 @@ def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) ->
|
||||
channel = db.get(Channel, channel_id)
|
||||
if channel is None or channel.backfill_done or not channel.uploads_playlist_id:
|
||||
return
|
||||
if quota.remaining_today(db) <= sysconfig.get_int(db, "backfill_quota_reserve"):
|
||||
if quota.backfill_reserve_reached():
|
||||
return
|
||||
try:
|
||||
# Attribute the spend to the requesting user under the full-backfill action, so it
|
||||
|
||||
Reference in New Issue
Block a user