peter
b2013a079a
worker.py (~1090 L) mixed the PARENT supervisor (claim + supervise + the process
loop) with the CHILD pipeline (the yt-dlp/ffmpeg download+edit that runs in the
spawned subprocess). Extract them:
worker.py 1090 -> 475 parent: claim/supervise/loop
downloads/pipeline.py (new 592) the child pipeline (_download/_process_edit +
hooks/progress/ffmpeg helpers + _Aborted)
downloads/jobdb.py (new 73) the short-lived-session helpers both sides
share (_set_job/_job_status/_touch_job/
_requeue_if_running/_reset_asset_pending +
the staging-dir paths)
The graph is one-directional (runner -> pipeline -> jobdb; worker -> jobdb; no
worker<->pipeline edge) — the parent never calls the child directly, it spawns
runner.py as a subprocess. Consumers rewired: runner import, and two tests follow
the moved symbols (_media_info/_produced_file -> pipeline; the ffmpeg-callback
patches target pipeline where the names are looked up).
Pure code motion, verbatim bodies. ruff clean, full lane 363 pass (unchanged).
52 lines
2.0 KiB
Python
52 lines
2.0 KiB
Python
"""Download/edit child process — runs ONE job's heavy work (yt-dlp download or ffmpeg edit) in its
|
|
own process, launched and supervised by the worker (`app.worker._run_supervised`).
|
|
|
|
Why a separate process: a worker THREAD can't be killed, a C-level spin (a catastrophic regex, a
|
|
tight yt-dlp retry loop) can't even be cooperatively cancelled, and one such thread starves the
|
|
others via the GIL. Running the work in a child means the supervisor can SIGTERM/SIGKILL it (and its
|
|
process group, so ffmpeg grandchildren die too) on a stall / pause / wall-clock breach.
|
|
|
|
Contract: the asset is already claimed ('downloading') by the parent. This process does the work and
|
|
writes the terminal job/asset state itself (done / error, or a cooperative pause via the in-flight
|
|
progress hook), then exits 0. An unexpected crash exits non-zero and the parent fails the job.
|
|
"""
|
|
import logging
|
|
import sys
|
|
|
|
from app.db import SessionLocal
|
|
from app.models import DownloadJob
|
|
|
|
log = logging.getLogger("siftlode.worker.runner")
|
|
|
|
|
|
def main() -> None:
|
|
logging.basicConfig(level=logging.INFO)
|
|
if len(sys.argv) != 2:
|
|
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
|
|
raise SystemExit(2)
|
|
job_id = int(sys.argv[1])
|
|
|
|
with SessionLocal() as db:
|
|
job = db.get(DownloadJob, job_id)
|
|
if job is None:
|
|
return
|
|
asset_id = job.asset_id
|
|
job_kind = job.job_kind
|
|
spec = dict(job.profile_snapshot or {})
|
|
source_kind, source_ref = job.source_kind, job.source_ref
|
|
if asset_id is None:
|
|
return
|
|
|
|
# Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
|
|
# real work, and to keep the import graph obvious: runner is a thin entry over the pipeline.
|
|
from app.downloads.pipeline import _download, _process_edit
|
|
|
|
if job_kind == "edit":
|
|
_process_edit(job_id, asset_id)
|
|
else:
|
|
_download(job_id, asset_id, source_kind, source_ref, spec)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|