fix(downloads): don't let the supervisor kill a finishing child (review round 2)

- worker.py: the watchdog loop now treats only 'paused'/'cancelled' (or a vanished row) as a
  stop; a terminal status (done/error, or a re-queued contended asset) means the child already
  wrote its outcome and is exiting, so keep polling instead of killing it. Previously a
  proc.wait timeout landing between the child's terminal commit and its exit was misread as a
  route-cancel and SIGKILLed a successful download (dropping its completion notification). The
  stall/wall-clock/shutdown checks are now gated on status=='running' (#1, #5 test).
- worker.py: guard the crash-path _job_liveness probe so a DB blip there can't propagate (the
  child is already dead; boot recovery is the backstop) (#2).
- worker.py: drop _stop_child's redundant poll guard — escalate_kill no-ops on an exited proc
  and getpgid yields None for a reaped pid (#3).
- worker.py: note the extraction heartbeat is per-attempt (a single over-timeout attempt is
  still killed, bounded by socket_timeout) (#4).
This commit is contained in:
2026-08-12 23:15:41 +02:00
parent 657224c1ad
commit b42b340594
2 changed files with 44 additions and 24 deletions
+10
View File
@@ -157,6 +157,16 @@ def test_a_long_download_survives_while_progress_stays_fresh(monkeypatch, rec):
assert _names(rec) == []
def test_a_finishing_child_is_not_mistaken_for_a_cancel(monkeypatch, rec):
# The child committed its terminal state (done/error) but hasn't exited yet (e.g. slow staging
# cleanup outlasting a poll). A non-'running' status must NOT be treated as a route-cancel and
# kill the finisher (which could drop its completion notification) — let it exit on its own.
_spawn(monkeypatch, _FakeProc(["timeout", ("exit", 0)])) # alive one poll, then a clean exit
_liveness(monkeypatch, ("done", 1.0))
worker._run_supervised(1, 99)
assert _names(rec) == [] # never killed, asset/job untouched
def test_a_watchdog_error_still_stops_the_child(monkeypatch, rec):
# If the liveness probe raises (e.g. a transient DB error), the child must NOT be left running
# unsupervised — the feature's whole point. It is killed and the job is failed.