feat(downloads): run each download/edit in a supervised child process with a hard-kill watchdog
Defense-in-depth after the ReDoS incident. The heavy work (yt-dlp download / ffmpeg edit) now runs in a child process (app.downloads.runner) supervised by the worker thread, not in the thread itself. A thread can't be killed and a C-level spin can't even be cooperatively cancelled, so one wedged download used to pin a core, hold the GIL and freeze the whole queue; a child process can be SIGTERM/SIGKILLed. The supervisor kills the child (and its process group, so ffmpeg grandchildren die too) when: a route pauses/cancels the job, it makes no progress for download_stall_timeout_seconds (150s), it runs past download_wall_clock_max_seconds (2h), the worker is shutting down, or the child crashes without setting state. On a kill it frees the asset and fails the job with a retryable network error (stalls are not auto-requeued). A clean exit is left alone — the child already wrote the terminal state, including a cooperative pause via the progress hook. - config.py: stall / wall-clock / supervisor-poll / kill-grace settings. - worker.py: _run_supervised + _spawn_download_child / _job_liveness / _stop_child / _cleanup_staging; _process_job claims the asset, then hands the heavy work to the child. - app/downloads/runner.py: the child entrypoint. - docker-compose.localdev.yml: init:true on the worker so tini (PID 1) reaps orphaned ffmpeg grandchildren left by a hard-kill (the worker reaps its own direct children). - tests: watchdog decision logic + SIGTERM->SIGKILL escalation (fake proc, no real subprocess).
This commit is contained in:
@@ -76,6 +76,9 @@ services:
|
||||
GIT_SHA: ${GIT_SHA:-unknown}
|
||||
BUILD_DATE: ${BUILD_DATE:-}
|
||||
command: ["python", "-m", "app.worker"]
|
||||
# tini as PID 1: reaps orphaned ffmpeg/yt-dlp grandchildren when the watchdog hard-kills a
|
||||
# download child's process group (the worker reaps its own direct children via subprocess.wait).
|
||||
init: true
|
||||
env_file: .env
|
||||
environment:
|
||||
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD:-siftlode}@db:5432/${POSTGRES_DB:-siftlode}
|
||||
|
||||
Reference in New Issue
Block a user