peter/siftlode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(downloads): revive the worker cancel branch and guard terminal writes (B-sub-1)

Browse Source 
The cancel route writes status 'canceled' but the supervisor compared against
'cancelled', so its "a route stopped it -> kill the child, free the asset"
branch was dead code. A canceled job's child ran to completion and wrote 'done'
unconditionally over the route's 'canceled'; since 'done' holds the asset but
'canceled' does not, a later delete released the asset a second time, dropping
ref_count below the live holder count and deleting a file still in use.

Add app/downloads/status.py as the single source of truth for the lifecycle
literals (JobStatus StrEnum + the HOLDING / STOPPED_BY_ROUTE sets) and use it at
the safety-critical sites: the supervisor stop check, the route cancel, and
release_asset's holding set. Make the worker's terminal done/error writes
conditional on the row still being 'running' so a child finishing just after a
route cancel/pause can no longer resurrect it. Add the missing 'canceled'
supervisor regression test.

(The full literal sweep across the downloads surface is left for the E3
downloads owner-unification; this is the data-safety subset.)
This commit is contained in:
peter
2026-08-18 00:21:43 +02:00
parent 429fbde866
commit 367bcebc9f
5 changed files with 86 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tests/test_worker_supervisor.py
+14 -1
View File
@@ -88,7 +88,7 @@ def test_no_progress_stall_kills_and_fails_the_job(monkeypatch, rec):
def test_route_pause_kills_but_keeps_the_route_status(monkeypatch, rec):
# A route flipped the job to 'paused'/'cancelled'. Kill + free the asset, but do NOT overwrite the
# A route flipped the job to 'paused'. Kill + free the asset, but do NOT overwrite the
# status with an error and do NOT requeue.
_spawn(monkeypatch, _FakeProc(["timeout"]))
_liveness(monkeypatch, ("paused", 1.0))
@@ -98,6 +98,19 @@ def test_route_pause_kills_but_keeps_the_route_status(monkeypatch, rec):
assert "set_job" not in names and "requeue" not in names
def test_route_cancel_kills_but_keeps_the_route_status(monkeypatch, rec):
# B-sub-1 regression: the cancel ROUTE writes the literal 'canceled' (models.py), but the supervisor
# used to compare against 'cancelled' — so this branch was DEAD and a canceled job's child ran on to
# completion, overwriting 'canceled' with 'done' and double-releasing its asset. The supervisor MUST
# treat the route's actual literal as a stop: kill + free the asset, no error write, no requeue.
_spawn(monkeypatch, _FakeProc(["timeout"]))
_liveness(monkeypatch, ("canceled", 1.0))
worker._run_supervised(1, 99)
names = _names(rec)
assert "stop_child" in names and "reset_asset" in names
assert "set_job" not in names and "requeue" not in names
def test_wall_clock_cap_kills_even_while_progressing(monkeypatch, rec):
monkeypatch.setattr(worker.settings, "download_wall_clock_max_seconds", 0)
_spawn(monkeypatch, _FakeProc(["timeout"]))