fix(downloads): don't let the supervisor kill a finishing child (review round 2)
- worker.py: the watchdog loop now treats only 'paused'/'cancelled' (or a vanished row) as a stop; a terminal status (done/error, or a re-queued contended asset) means the child already wrote its outcome and is exiting, so keep polling instead of killing it. Previously a proc.wait timeout landing between the child's terminal commit and its exit was misread as a route-cancel and SIGKILLed a successful download (dropping its completion notification). The stall/wall-clock/shutdown checks are now gated on status=='running' (#1, #5 test). - worker.py: guard the crash-path _job_liveness probe so a DB blip there can't propagate (the child is already dead; boot recovery is the backstop) (#2). - worker.py: drop _stop_child's redundant poll guard — escalate_kill no-ops on an exited proc and getpgid yields None for a reaped pid (#3). - worker.py: note the extraction heartbeat is per-attempt (a single over-timeout attempt is still killed, bounded by socket_timeout) (#4).
This commit is contained in:
@@ -157,6 +157,16 @@ def test_a_long_download_survives_while_progress_stays_fresh(monkeypatch, rec):
|
||||
assert _names(rec) == []
|
||||
|
||||
|
||||
def test_a_finishing_child_is_not_mistaken_for_a_cancel(monkeypatch, rec):
|
||||
# The child committed its terminal state (done/error) but hasn't exited yet (e.g. slow staging
|
||||
# cleanup outlasting a poll). A non-'running' status must NOT be treated as a route-cancel and
|
||||
# kill the finisher (which could drop its completion notification) — let it exit on its own.
|
||||
_spawn(monkeypatch, _FakeProc(["timeout", ("exit", 0)])) # alive one poll, then a clean exit
|
||||
_liveness(monkeypatch, ("done", 1.0))
|
||||
worker._run_supervised(1, 99)
|
||||
assert _names(rec) == [] # never killed, asset/job untouched
|
||||
|
||||
|
||||
def test_a_watchdog_error_still_stops_the_child(monkeypatch, rec):
|
||||
# If the liveness probe raises (e.g. a transient DB error), the child must NOT be left running
|
||||
# unsupervised — the feature's whole point. It is killed and the job is failed.
|
||||
|
||||
Reference in New Issue
Block a user