perf(downloads): de-prioritize worker transcodes + idle-back-off polls to stop LAN slowdown
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API, Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and starved web requests + playback until it finished. Full-scope S4 mitigations: - Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp descendant): CPU nice increment + best-effort idle ionice class + an OMP_NUM_THREADS budget, all config-gated and independently disable-able. - Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode reserves a core for foreground work instead of claiming all of them. - Throttle the retention GC's whole-tree orphan sweep (yield IO after each batch of files) so its rglob can't monopolize disk IO in the API process. - Idle-back-off the Download Center (2s→15s when nothing queued/running) and UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the next fetch after an enqueue (invalidateDownloads refetches immediately). Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence, _deprioritize (nice/ionice/OMP branches), GC sweep throttle batching.
This commit is contained in:
@@ -86,11 +86,25 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
|
||||
|
||||
cutoff = time.time() - _ORPHAN_MIN_AGE_S
|
||||
removed = 0
|
||||
# Throttle the whole-tree walk: this runs in the API process, and an unpaced rglob+stat over a
|
||||
# large download root monopolizes disk IO long enough to make web requests + playback stutter.
|
||||
# Yield the IO channel after every `batch` files touched so foreground work interleaves.
|
||||
pause_s = max(0, settings.download_gc_sweep_pause_ms) / 1000.0
|
||||
batch = max(1, settings.download_gc_sweep_batch)
|
||||
touched = 0
|
||||
|
||||
def _throttle() -> None:
|
||||
nonlocal touched
|
||||
touched += 1
|
||||
if pause_s and touched % batch == 0:
|
||||
time.sleep(pause_s)
|
||||
|
||||
for entry in root.iterdir():
|
||||
if entry.name.startswith("."): # .edits / .plex-* / .staging / .storyboards + any dotfile
|
||||
continue
|
||||
files = [entry] if entry.is_file() else [p for p in entry.rglob("*") if p.is_file()]
|
||||
for f in files:
|
||||
_throttle()
|
||||
try:
|
||||
if f.resolve() in keep or f.stat().st_mtime > cutoff:
|
||||
continue
|
||||
|
||||
Reference in New Issue
Block a user