fix(downloads): exclude in-flight assets from the claim, drop the busy-wait (E3 S2, B-sub-3)
Two jobs on the same shared asset (same video+format queued twice, or by two users) starved the worker pool: worker A downloads the asset; worker B claims the sibling job, finds the asset `downloading`, requeues it, sleep(2)s, and re-claims it on the next tick — burning a whole worker slot behind one in-flight asset for the entire download (at concurrency=2, the queue stalls). Fix: the claim SQL now excludes any job whose asset is already `downloading` (`NOT EXISTS … media_assets.status='downloading'`), so a competing sibling simply waits until the asset settles and the workers pick up unrelated jobs meanwhile. The `_process_job` busy/lost-race branches drop their `time.sleep(2)` — the exclusion keeps a requeued job out of the claim window, so there's no spin. TDD: the new test failed on the old claim (claimed the sibling), passes now. Full lane green, ruff clean.
This commit is contained in:
+16
-4
@@ -176,7 +176,14 @@ def _recover_orphans() -> None:
|
||||
|
||||
|
||||
def _claim_job() -> int | None:
|
||||
"""Claim the oldest queued job whose user is under their per-user concurrency limit.
|
||||
"""Claim the oldest queued job whose user is under their per-user concurrency limit AND whose
|
||||
shared asset isn't already being downloaded by another worker.
|
||||
|
||||
The asset exclusion (`NOT EXISTS … media_assets.status='downloading'`) stops the same-asset ping-
|
||||
pong (B-sub-3): without it, a second worker claims a queued job that shares an in-flight asset,
|
||||
finds it busy, requeues it, and immediately re-claims it — burning a worker slot behind one
|
||||
in-flight asset and starving everyone else. Excluded here, that job simply waits until its asset
|
||||
settles, so `_process_job` no longer needs to sleep on the busy/lost-race paths.
|
||||
|
||||
Users already AT their concurrency limit are excluded IN THE SQL, so one user flooding the
|
||||
queue can't monopolize the fetch window and starve everyone behind them. (The old fixed
|
||||
@@ -193,6 +200,10 @@ def _claim_job() -> int | None:
|
||||
"""
|
||||
SELECT id, user_id FROM download_jobs
|
||||
WHERE status='queued'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM media_assets a
|
||||
WHERE a.id = download_jobs.asset_id AND a.status = 'downloading'
|
||||
)
|
||||
AND user_id NOT IN (
|
||||
SELECT r.user_id
|
||||
FROM download_jobs r
|
||||
@@ -463,15 +474,16 @@ def _process_job(job_id: int) -> None:
|
||||
_set_job(job_id, status="error", error=asset_error or "Download failed", phase=None)
|
||||
return
|
||||
if asset_status == "downloading":
|
||||
# Another worker owns this asset; wait and let the job be reclaimed later.
|
||||
# Another worker won this asset between our claim and here (a narrow race — the claim SQL
|
||||
# excludes downloading-asset jobs). Requeue and let it be reclaimed once the asset settles;
|
||||
# no sleep — the claim exclusion keeps it out of the window until then, so no busy-loop.
|
||||
_requeue_if_running(job_id, "waiting")
|
||||
time.sleep(2)
|
||||
return
|
||||
|
||||
# asset_status == "pending": try to win the download / edit.
|
||||
if not _claim_asset(asset_id):
|
||||
# Lost the pending->downloading race; requeue (same reasoning — the exclusion prevents a spin).
|
||||
_requeue_if_running(job_id, "waiting")
|
||||
time.sleep(2)
|
||||
return
|
||||
|
||||
# The asset is now ours ('downloading'): run the heavy work in a supervised child process.
|
||||
|
||||
Reference in New Issue
Block a user