perf(downloads): de-prioritize worker transcodes + idle-back-off polls to stop LAN slowdown

The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API,
Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and
starved web requests + playback until it finished. Full-scope S4 mitigations:

- Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp
  descendant): CPU nice increment + best-effort idle ionice class + an
  OMP_NUM_THREADS budget, all config-gated and independently disable-able.
- Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode
  reserves a core for foreground work instead of claiming all of them.
- Throttle the retention GC's whole-tree orphan sweep (yield IO after each
  batch of files) so its rglob can't monopolize disk IO in the API process.
- Idle-back-off the Download Center (2s→15s when nothing queued/running) and
  UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the
  next fetch after an enqueue (invalidateDownloads refetches immediately).

Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence,
_deprioritize (nice/ionice/OMP branches), GC sweep throttle batching.
This commit is contained in:
2026-08-28 02:37:56 +02:00
parent b7a6451dba
commit 9ae6c47d6c
9 changed files with 235 additions and 6 deletions
+18 -2
View File
@@ -25,6 +25,7 @@ import hashlib
import json
import logging
import math
import os
import queue
import shutil
import subprocess
@@ -32,9 +33,21 @@ import threading
import time
from pathlib import Path
from app.config import settings
log = logging.getLogger("siftlode.edit")
def encode_thread_cap() -> int:
"""How many encoder threads a browser-safe re-encode may use. `download_ffmpeg_threads` when the
admin pinned one (>0), else reserve a core for foreground work (max(1, cpu_count - 1)). Shared by
the ffmpeg `-threads` flag and the child's `OMP_NUM_THREADS` so both honour the same budget."""
n = settings.download_ffmpeg_threads
if n > 0:
return n
return max(1, (os.cpu_count() or 2) - 1)
class EditAborted(Exception):
"""Raised by run_ffmpeg when the job was paused/canceled mid-encode."""
@@ -510,11 +523,12 @@ def browser_transcode_flags(
def build_browser_safe_cmd(
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool, threads: int | None = None
) -> list[str]:
"""ffmpeg command that lands `src` as a browser-safe mp4 (faststart): re-encode only the
stream(s) that need it (libx264 / aac), stream-copy the rest, always into mp4. `-progress
pipe:1` streams machine-readable progress for run_ffmpeg."""
pipe:1` streams machine-readable progress for run_ffmpeg. `threads` caps the encoder so a
re-encode can't claim every core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
@@ -534,6 +548,8 @@ def build_browser_safe_cmd(
else ["-c:v", "copy"]
)
cmd += ["-c:a", "aac", "-b:a", "192k"] if reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
cmd += ["-movflags", "+faststart", str(dest)]
return cmd