peter
0ed82fe6b6
Defense-in-depth after the ReDoS incident. The heavy work (yt-dlp download / ffmpeg edit) now runs in a child process (app.downloads.runner) supervised by the worker thread, not in the thread itself. A thread can't be killed and a C-level spin can't even be cooperatively cancelled, so one wedged download used to pin a core, hold the GIL and freeze the whole queue; a child process can be SIGTERM/SIGKILLed. The supervisor kills the child (and its process group, so ffmpeg grandchildren die too) when: a route pauses/cancels the job, it makes no progress for download_stall_timeout_seconds (150s), it runs past download_wall_clock_max_seconds (2h), the worker is shutting down, or the child crashes without setting state. On a kill it frees the asset and fails the job with a retryable network error (stalls are not auto-requeued). A clean exit is left alone — the child already wrote the terminal state, including a cooperative pause via the progress hook. - config.py: stall / wall-clock / supervisor-poll / kill-grace settings. - worker.py: _run_supervised + _spawn_download_child / _job_liveness / _stop_child / _cleanup_staging; _process_job claims the asset, then hands the heavy work to the child. - app/downloads/runner.py: the child entrypoint. - docker-compose.localdev.yml: init:true on the worker so tini (PID 1) reaps orphaned ffmpeg grandchildren left by a hard-kill (the worker reaps its own direct children). - tests: watchdog decision logic + SIGTERM->SIGKILL escalation (fake proc, no real subprocess).
52 lines
2.0 KiB
Python
52 lines
2.0 KiB
Python
"""Download/edit child process — runs ONE job's heavy work (yt-dlp download or ffmpeg edit) in its
|
|
own process, launched and supervised by the worker (`app.worker._run_supervised`).
|
|
|
|
Why a separate process: a worker THREAD can't be killed, a C-level spin (a catastrophic regex, a
|
|
tight yt-dlp retry loop) can't even be cooperatively cancelled, and one such thread starves the
|
|
others via the GIL. Running the work in a child means the supervisor can SIGTERM/SIGKILL it (and its
|
|
process group, so ffmpeg grandchildren die too) on a stall / pause / wall-clock breach.
|
|
|
|
Contract: the asset is already claimed ('downloading') by the parent. This process does the work and
|
|
writes the terminal job/asset state itself (done / error, or a cooperative pause via the in-flight
|
|
progress hook), then exits 0. An unexpected crash exits non-zero and the parent fails the job.
|
|
"""
|
|
import logging
|
|
import sys
|
|
|
|
from app.db import SessionLocal
|
|
from app.models import DownloadJob
|
|
|
|
log = logging.getLogger("siftlode.worker.runner")
|
|
|
|
|
|
def main() -> None:
|
|
logging.basicConfig(level=logging.INFO)
|
|
if len(sys.argv) != 2:
|
|
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
|
|
raise SystemExit(2)
|
|
job_id = int(sys.argv[1])
|
|
|
|
with SessionLocal() as db:
|
|
job = db.get(DownloadJob, job_id)
|
|
if job is None:
|
|
return
|
|
asset_id = job.asset_id
|
|
job_kind = job.job_kind
|
|
spec = dict(job.profile_snapshot or {})
|
|
source_kind, source_ref = job.source_kind, job.source_ref
|
|
if asset_id is None:
|
|
return
|
|
|
|
# Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
|
|
# real work, and to keep the import graph obvious: runner is a thin entry over worker's helpers.
|
|
from app.worker import _download, _process_edit
|
|
|
|
if job_kind == "edit":
|
|
_process_edit(job_id, asset_id)
|
|
else:
|
|
_download(job_id, asset_id, source_kind, source_ref, spec)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|