feat(downloads): run each download/edit in a supervised child process with a hard-kill watchdog

Defense-in-depth after the ReDoS incident. The heavy work (yt-dlp download / ffmpeg edit)
now runs in a child process (app.downloads.runner) supervised by the worker thread, not in
the thread itself. A thread can't be killed and a C-level spin can't even be cooperatively
cancelled, so one wedged download used to pin a core, hold the GIL and freeze the whole
queue; a child process can be SIGTERM/SIGKILLed.

The supervisor kills the child (and its process group, so ffmpeg grandchildren die too)
when: a route pauses/cancels the job, it makes no progress for download_stall_timeout_seconds
(150s), it runs past download_wall_clock_max_seconds (2h), the worker is shutting down, or
the child crashes without setting state. On a kill it frees the asset and fails the job with
a retryable network error (stalls are not auto-requeued). A clean exit is left alone — the
child already wrote the terminal state, including a cooperative pause via the progress hook.

- config.py: stall / wall-clock / supervisor-poll / kill-grace settings.
- worker.py: _run_supervised + _spawn_download_child / _job_liveness / _stop_child /
  _cleanup_staging; _process_job claims the asset, then hands the heavy work to the child.
- app/downloads/runner.py: the child entrypoint.
- docker-compose.localdev.yml: init:true on the worker so tini (PID 1) reaps orphaned
  ffmpeg grandchildren left by a hard-kill (the worker reaps its own direct children).
- tests: watchdog decision logic + SIGTERM->SIGKILL escalation (fake proc, no real subprocess).
This commit is contained in:
2026-08-12 22:39:17 +02:00
parent e395fb816d
commit 0ed82fe6b6
5 changed files with 407 additions and 8 deletions
+12
View File
@@ -173,6 +173,18 @@ class Settings(BaseSettings):
download_worker_concurrency: int = 2
# How often (empty pending queue) the worker polls for a new job, in seconds.
download_poll_seconds: int = 5
# --- download-subprocess watchdog (each download/edit runs in a supervised child process so a
# wedged yt-dlp/ffmpeg can be hard-killed; a thread cannot) ---
# Kill a running download if its job row hasn't advanced (no progress written) for this long.
# Generous: a live download writes progress ~1/s, so only a genuine 0-progress stall trips it.
download_stall_timeout_seconds: int = 150
# Absolute ceiling on a single download/edit regardless of progress (a legitimately huge file
# still finishes well under this; a runaway is capped).
download_wall_clock_max_seconds: int = 7200 # 2 hours
# How often the supervisor checks the child's liveness + the job row.
download_supervisor_poll_seconds: int = 3
# Grace between SIGTERM and SIGKILL when stopping a child (matches run_ffmpeg's escalation).
download_kill_grace_seconds: int = 5
# Total cache size cap across ALL users (bytes). When >0 and exceeded, the GC evicts the
# least-recently-accessed ready assets until under it. 0 = unlimited (disk is the limit).
download_total_max_bytes: int = 0