fix(downloads): worker claim fairness + bounded job listings (R12 S5)
#2 — _claim_job excluded users at their concurrency limit only in Python, AFTER a fixed LIMIT-50 fetch window. One user flooding the queue with >50 queued jobs while at their cap filled the whole window, chosen came back None, and every other user's work past position 50 was never seen. Exclude capped users IN THE SQL (per-user max_concurrent from the quota row, else the sysconfig default; unlimited users never capped), keeping the Python re-check as a race guard that can only reject, never over-claim. #3 — list_downloads and shared_with_me loaded every job for a user with no bound. Cap both at 500 newest-first (a shared _MAX_LIST, matching the admin listing's existing cap); the Download Center renders newest-first and 500 is far beyond any real manual-download history, so no frontend change is needed. Tests: DB-backed — a capped user's 60-job flood no longer starves a user behind the window; oldest-by-queue_pos claim, all-capped->None, unlimited never excluded; and the list cap bounds the result set.
This commit is contained in:
+20
-3
@@ -177,19 +177,36 @@ def _recover_orphans() -> None:
|
||||
def _claim_job() -> int | None:
|
||||
"""Claim the oldest queued job whose user is under their per-user concurrency limit.
|
||||
|
||||
Fetches a window of locked candidates (FOR UPDATE SKIP LOCKED) and picks the first eligible
|
||||
one, so one user flooding the queue can't starve others beyond their max_concurrent."""
|
||||
Users already AT their concurrency limit are excluded IN THE SQL, so one user flooding the
|
||||
queue can't monopolize the fetch window and starve everyone behind them. (The old fixed
|
||||
LIMIT-50 window did exactly that: 50 queued jobs from a single capped user filled the window,
|
||||
`chosen` came back None, and every other user's work — sitting past position 50 — was never
|
||||
seen.) The per-user limit is the quota row's `max_concurrent` (else the sysconfig default);
|
||||
`unlimited` users are never capped. A final Python `at_concurrency_limit` re-check still guards
|
||||
the tiny race between the exclusion subquery and the claim (it can only ever reject, never
|
||||
over-claim), so correctness never depends on the two observations agreeing."""
|
||||
with SessionLocal() as db:
|
||||
default_mc = sysconfig.get_int(db, "download_default_max_concurrent")
|
||||
rows = db.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT id, user_id FROM download_jobs
|
||||
WHERE status='queued'
|
||||
AND user_id NOT IN (
|
||||
SELECT r.user_id
|
||||
FROM download_jobs r
|
||||
LEFT JOIN download_quotas q ON q.user_id = r.user_id
|
||||
WHERE r.status='running' AND r.user_id IS NOT NULL
|
||||
GROUP BY r.user_id, q.max_concurrent, q.unlimited
|
||||
HAVING COALESCE(q.unlimited, false) = false
|
||||
AND count(*) >= COALESCE(q.max_concurrent, :dmc)
|
||||
)
|
||||
ORDER BY queue_pos, id
|
||||
FOR UPDATE SKIP LOCKED
|
||||
LIMIT 50
|
||||
"""
|
||||
)
|
||||
),
|
||||
{"dmc": default_mc},
|
||||
).fetchall()
|
||||
chosen = next(
|
||||
(jid for jid, uid in rows if not quota.at_concurrency_limit(db, uid)), None
|
||||
|
||||
Reference in New Issue
Block a user