refactor(downloads): route worker terminal writes through an atomic finalize_if_running (B-sub-1 review)

Follow-up to the /code-review high on the S1 branch. The per-site
`if job.status == 'running'` guards were a non-atomic read-then-write (TOCTOU):
a cancel committing between the child's status read and its commit could still
be overwritten with done/error, re-opening the double-release window.

Introduce service.finalize_if_running(db, job_id, **fields): one atomic
`UPDATE download_jobs SET ... WHERE status='running'` that returns whether it
fired. It flushes pending asset writes first so the asset-then-job row-lock
order matches the cancel route (no deadlock), and runs in the caller's session
so it commits with the asset writes in one transaction. All five worker sites
(download/edit completion, both error paths, cache finish) now go through it,
so the invariant lives in one place. The 'job done' logs move inside the
finalized branch so a job stopped mid-completion no longer logs a false 'done'.
Add tests/test_downloads_finalize.py: running -> finalizes; canceled/paused ->
no-op.
This commit is contained in:
2026-08-18 00:58:13 +02:00
parent 9845c8d0f7
commit 4ce0a702f4
3 changed files with 135 additions and 47 deletions
+29 -2
View File
@@ -8,7 +8,7 @@ the worker live in `worker.py` / the routes; this module owns enqueue + asset bo
"""
from datetime import datetime, timezone
from sqlalchemy import func, select
from sqlalchemy import func, select, text
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -16,7 +16,7 @@ from app.config import settings
from app.downloads import adapters
from app.downloads import edit as editmod
from app.downloads import formats, quota, storage
from app.downloads.status import HOLDING
from app.downloads.status import HOLDING, JobStatus
from app.models import Channel, DownloadJob, MediaAsset, Video
@@ -224,6 +224,33 @@ def enqueue_edit(
return job
def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
"""Write terminal `fields` to a job in ONE atomic ``UPDATE ... WHERE status='running'`` and report
whether it fired. This is the single owner of the invariant "a finishing worker child must never
resurrect a job a route just stopped": the atomic WHERE closes the read-then-write race an
``if job.status == 'running'`` guard leaves open, so a concurrent cancel/pause can no longer be
overwritten with done/error (which would re-open the double-release → file-delete-under-live-holder
class, since `done`/`error` hold the asset but `canceled` does not — see release_asset).
Runs in the CALLER's session and does NOT commit — the caller commits it together with its asset
writes in one transaction. It flushes first so any pending asset write takes its row lock BEFORE
this takes the job's, matching the cancel route's asset-then-job order (no deadlock). `fields` keys
are internal column names (never user input). Returns True when the row was still running and got
updated, False when a route had already moved it out of 'running'."""
if not fields:
return False
db.flush() # asset-then-job lock order: push pending asset writes before we lock the job row
# StrEnum members carry through psycopg as strings, but coerce explicitly so a bound status value
# is always the plain literal the column stores.
params = {k: (v.value if isinstance(v, JobStatus) else v) for k, v in fields.items()}
assignments = ", ".join(f"{col} = :{col}" for col in fields)
result = db.execute(
text(f"UPDATE download_jobs SET {assignments} WHERE id = :_job_id AND status = :_running"),
{**params, "_job_id": job_id, "_running": JobStatus.RUNNING.value},
)
return result.rowcount > 0
def release_asset(db: Session, job: DownloadJob) -> None:
"""Drop this job's hold on its asset when it leaves a holding state. Once NO job holds the
asset anymore, delete the file + row immediately — a deleted download should free its disk,