fix(downloads): heartbeat a slow re-encode so the stall watchdog can't false-kill it (E3 S2, B-sub-4)

The supervisor kills a job that writes nothing for download_stall_timeout_seconds
(150s) and labels it a "network" stall. But the ffmpeg progress writer only wrote
on an integer-percent CHANGE, so a re-encode that sits on one percent for minutes
(AV1/VP9 -> H.264 or a crop on the CPU-only prod host) produced no row write and
was killed mid-encode as "stalled" — and the retry stalled the same way.

Fix: the writer now heartbeats (`_touch_job`, bumping updated_at without touching
the progress bar) once per interval even when the whole percent is unchanged, so
a live-but-quiet re-encode keeps a fresh updated_at. Reorders the rate-limit gate
first so a burst still costs one DB op — either a progress write or a heartbeat.

Pure-logic test with an injected clock: three stuck-at-10% intervals write no new
progress but heartbeat each time. Full lane green, ruff clean.
This commit is contained in:
2026-08-18 21:34:36 +02:00
parent 7d89bc8b8e
commit e2362e1ed1
2 changed files with 25 additions and 1 deletions
+17
View File
@@ -37,12 +37,29 @@ class TestProgressWriter:
writes = []
clock = {"t": 1000.0}
monkeypatch.setattr(worker, "_set_job", lambda job_id, **f: writes.append(f))
monkeypatch.setattr(worker, "_touch_job", lambda job_id: None)
write = worker._ffmpeg_progress_writer(7, "processing", now=lambda: clock["t"])
write(10.1)
clock["t"] += 5
write(10.9) # still 10% — the bar shows integers
assert [w["progress"] for w in writes] == [10]
def test_a_slow_same_percent_stretch_heartbeats_the_watchdog(self, monkeypatch):
"""B-sub-4: a re-encode that sits on one integer percent for minutes must NOT be killed as
stalled — the writer touches the row (heartbeat) each interval even though it writes no new
progress, so the stall watchdog keeps seeing a fresh updated_at."""
writes, touches = [], []
clock = {"t": 1000.0}
monkeypatch.setattr(worker, "_set_job", lambda job_id, **f: writes.append(f))
monkeypatch.setattr(worker, "_touch_job", lambda job_id: touches.append(job_id))
write = worker._ffmpeg_progress_writer(7, "processing", now=lambda: clock["t"])
write(10.1) # first real write → progress 10
for _ in range(3): # three more intervals stuck on 10%
clock["t"] += 1.5
write(10.4)
assert [w["progress"] for w in writes] == [10] # progress never rewritten
assert touches == [7, 7, 7] # …but each interval heartbeats the watchdog
class TestCancelPoll:
def test_asks_once_then_reuses_the_answer(self, monkeypatch):