peter
b59fd38a9c
The log now has a stored, queryable copy for the diagnostics console: every WARNING and above plus the request line of every request, kept for log_retention_days (default 7, no "forever") and pruned hourly by the new log_gc job. Rows are built on the logging thread from the same redacted fields as the stdout line, queued without blocking and written in batches by one background thread on its own connection. A full queue or a failed write drops and reports instead of retrying. Flood control keeps 30 of any one message per minute plus a count of the rest; anonymous request lines are capped at 60 a minute per client, and the client address is never stored. Deleting an account deletes its stored events. Measured on prod first: a YouTube RSS outage wrote 11k warnings in four hours, one per channel per poll. The poll now writes one warning with per-reason counts and a few examples, and a failure on our side gets one traceback per poll instead of one per channel.
92 lines
3.8 KiB
Python
92 lines
3.8 KiB
Python
"""Download/edit child process — runs ONE job's heavy work (yt-dlp download or ffmpeg edit) in its
|
|
own process, launched and supervised by the worker (`app.worker._run_supervised`).
|
|
|
|
Why a separate process: a worker THREAD can't be killed, a C-level spin (a catastrophic regex, a
|
|
tight yt-dlp retry loop) can't even be cooperatively cancelled, and one such thread starves the
|
|
others via the GIL. Running the work in a child means the supervisor can SIGTERM/SIGKILL it (and its
|
|
process group, so ffmpeg grandchildren die too) on a stall / pause / wall-clock breach.
|
|
|
|
Contract: the asset is already claimed ('downloading') by the parent. This process does the work and
|
|
writes the terminal job/asset state itself (done / error, or a cooperative pause via the in-flight
|
|
progress hook), then exits 0. An unexpected crash exits non-zero and the parent fails the job.
|
|
"""
|
|
|
|
import logging
|
|
import os
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
|
|
from app import logs
|
|
from app import logsink
|
|
from app.config import settings
|
|
from app.db import SessionLocal
|
|
from app.downloads import edit as editmod
|
|
from app.models import DownloadJob
|
|
|
|
log = logging.getLogger("siftlode.worker.runner")
|
|
|
|
|
|
def _deprioritize() -> None:
|
|
"""Lower THIS child's CPU + IO priority so its yt-dlp/ffmpeg work — every descendant inherits the
|
|
handicap — can never starve the co-located API, Postgres, or Plex playback. Run once at startup,
|
|
before any heavy work. Every step is best-effort: a failed renice must never fail a download."""
|
|
nice_inc = settings.download_child_nice
|
|
if nice_inc > 0:
|
|
try:
|
|
os.nice(nice_inc) # relative increment; the child starts at the worker's nice (0)
|
|
except OSError as exc:
|
|
log.info("could not renice download child: %s", str(exc)[:120])
|
|
cls = settings.download_child_ionice_class
|
|
if cls > 0 and shutil.which("ionice"): # util-linux; absent on some minimal images
|
|
try:
|
|
subprocess.run(
|
|
["ionice", "-c", str(cls), "-p", str(os.getpid())],
|
|
capture_output=True,
|
|
timeout=5,
|
|
check=False,
|
|
)
|
|
except (OSError, subprocess.SubprocessError) as exc:
|
|
log.info("could not ionice download child: %s", str(exc)[:120])
|
|
# Cap OpenMP-parallel ffmpeg filters to the same budget as the encoder threads, so a filter
|
|
# graph can't fan out across every core behind ffmpeg's own `-threads` cap.
|
|
os.environ.setdefault("OMP_NUM_THREADS", str(editmod.encode_thread_cap()))
|
|
|
|
|
|
def main() -> None:
|
|
logs.configure_logging("runner")
|
|
_deprioritize()
|
|
if len(sys.argv) != 2:
|
|
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
|
|
raise SystemExit(2)
|
|
job_id = int(sys.argv[1])
|
|
# This whole process exists to run ONE job — tag every line it prints with it, so the child's
|
|
# output joins up with the worker's and the API's view of the same download.
|
|
logs.set_job(job_id)
|
|
logsink.start() # flushed at exit (atexit) — this process has no shutdown hook of its own
|
|
|
|
with SessionLocal() as db:
|
|
job = db.get(DownloadJob, job_id)
|
|
if job is None:
|
|
return
|
|
asset_id = job.asset_id
|
|
job_kind = job.job_kind
|
|
spec = dict(job.profile_snapshot or {})
|
|
source_kind, source_ref = job.source_kind, job.source_ref
|
|
user_id = job.user_id
|
|
if asset_id is None:
|
|
return
|
|
|
|
# Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
|
|
# real work, and to keep the import graph obvious: runner is a thin entry over the pipeline.
|
|
from app.downloads.pipeline import _download, _process_edit
|
|
|
|
if job_kind == "edit":
|
|
_process_edit(job_id, asset_id)
|
|
else:
|
|
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|