Files
siftlode/docker-compose.localdev.yml
T
peter 0ed82fe6b6 feat(downloads): run each download/edit in a supervised child process with a hard-kill watchdog
Defense-in-depth after the ReDoS incident. The heavy work (yt-dlp download / ffmpeg edit)
now runs in a child process (app.downloads.runner) supervised by the worker thread, not in
the thread itself. A thread can't be killed and a C-level spin can't even be cooperatively
cancelled, so one wedged download used to pin a core, hold the GIL and freeze the whole
queue; a child process can be SIGTERM/SIGKILLed.

The supervisor kills the child (and its process group, so ffmpeg grandchildren die too)
when: a route pauses/cancels the job, it makes no progress for download_stall_timeout_seconds
(150s), it runs past download_wall_clock_max_seconds (2h), the worker is shutting down, or
the child crashes without setting state. On a kill it frees the asset and fails the job with
a retryable network error (stalls are not auto-requeued). A clean exit is left alone — the
child already wrote the terminal state, including a cooperative pause via the progress hook.

- config.py: stall / wall-clock / supervisor-poll / kill-grace settings.
- worker.py: _run_supervised + _spawn_download_child / _job_liveness / _stop_child /
  _cleanup_staging; _process_job claims the asset, then hands the heavy work to the child.
- app/downloads/runner.py: the child entrypoint.
- docker-compose.localdev.yml: init:true on the worker so tini (PID 1) reaps orphaned
  ffmpeg grandchildren left by a hard-kill (the worker reaps its own direct children).
- tests: watchdog decision logic + SIGTERM->SIGKILL escalation (fake proc, no real subprocess).
2026-08-12 22:39:17 +02:00

129 lines
5.4 KiB
YAML

# Self-contained local development stack — Postgres + API + scheduler, all on this machine.
#
# Fully decoupled: its own local database (a Docker volume) and its own scheduler, so there
# is no shared DB and no migration coupling with any other instance. Edit code, rebuild, and
# everything (including the background sync) runs right here.
#
# docker compose -f docker-compose.localdev.yml up -d --build
#
# Browse at http://localhost:8080 — Google login works because the OAuth redirect is
# http://localhost:8080/auth/callback. The API runs `alembic upgrade head` on startup, so it
# applies local migrations to THIS local DB only. Seed the catalog once with a pg_dump from
# another instance if you want real data (see RUNBOOK).
services:
db:
image: postgres:16-alpine
environment:
POSTGRES_USER: ${POSTGRES_USER:-siftlode}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-siftlode}
POSTGRES_DB: ${POSTGRES_DB:-siftlode}
volumes:
- pgdata:/var/lib/postgresql/data
security_opt:
- apparmor:unconfined
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-siftlode} -d ${POSTGRES_DB:-siftlode}"]
interval: 5s
timeout: 5s
retries: 12
restart: unless-stopped
api:
build:
context: .
dockerfile: Dockerfile
args:
APP_VERSION: ${APP_VERSION:-dev}
GIT_SHA: ${GIT_SHA:-unknown}
BUILD_DATE: ${BUILD_DATE:-}
env_file: .env
environment:
# Own local database (overrides whatever DATABASE_URL is in .env).
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD:-siftlode}@db:5432/${POSTGRES_DB:-siftlode}
# This instance owns its scheduler now (its own DB, so no double-write concern).
SCHEDULER_ENABLED: "true"
# Download center: the API serves finished files (it does NOT run the worker loop).
DOWNLOAD_ROOT: /downloads
WORKER_ENABLED: "false"
# Plex HLS remux scratch on fast container-local storage (the /downloads bind-mount is a slow
# Windows Docker Desktop mount in dev; prod uses fast local disk so it keeps the default).
PLEX_HLS_DIR: /var/tmp/plex-hls
depends_on:
db:
condition: service_healthy
# Harmless on Docker Desktop; needed on some nested/hardened Docker hosts.
security_opt:
- apparmor:unconfined
ports:
- "${APP_PORT:-8080}:8000"
volumes:
# Downloaded media lives on the host so you can inspect the generated Plex tree.
# Gitignored. The worker writes here; the API reads it to serve local downloads.
- ./downloads:/downloads
# Plex media, read-only (see the plex_media volume note below). Maps to plex_path_map.
- plex_media:/plex-media:ro
restart: unless-stopped
# Dedicated download worker: same image, runs the yt-dlp job loop instead of the API.
# It shares the DB (job queue) and the downloads mount with the API.
worker:
build:
context: .
dockerfile: Dockerfile
args:
APP_VERSION: ${APP_VERSION:-dev}
GIT_SHA: ${GIT_SHA:-unknown}
BUILD_DATE: ${BUILD_DATE:-}
command: ["python", "-m", "app.worker"]
# tini as PID 1: reaps orphaned ffmpeg/yt-dlp grandchildren when the watchdog hard-kills a
# download child's process group (the worker reaps its own direct children via subprocess.wait).
init: true
env_file: .env
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD:-siftlode}@db:5432/${POSTGRES_DB:-siftlode}
# The worker must not also run the scheduler; the API owns that.
SCHEDULER_ENABLED: "false"
DOWNLOAD_ROOT: /downloads
WORKER_ENABLED: "true"
depends_on:
db:
condition: service_healthy
bgutil-pot:
condition: service_started
security_opt:
- apparmor:unconfined
volumes:
- ./downloads:/downloads
- plex_media:/plex-media:ro
restart: unless-stopped
# PO-token provider: mints YouTube Proof-of-Origin tokens on demand so the worker bypasses
# bot-detection and gets high-quality formats. The worker reaches it at http://bgutil-pot:4416
# (DOWNLOAD_POT_BASE_URL). Pin the tag to match the plugin version in requirements.txt.
bgutil-pot:
image: brainicism/bgutil-ytdlp-pot-provider:1.3.1
init: true
security_opt:
- apparmor:unconfined
restart: unless-stopped
volumes:
pgdata:
# OPTIONAL (Plex integration dev testing): the Plex media, mounted READ-ONLY so the container can
# play the physical file locally (this dev box isn't the Plex host). Only referenced when the Plex
# module is enabled; harmless otherwise (Docker mounts it lazily on first container use). On prod
# (LXC on the Plex host) this is a plain host bind-mount.
#
# Local bind to a few test movies/shows kept on this box — the full nas.lan SMB share was too slow.
# Point PLEX_MEDIA_HOST_PATH (.env, gitignored, so no site path lives in the tracked compose) at the
# Docker-Desktop host path (e.g. /run/desktop/mnt/host/d/plex-test-media for D:\plex-test-media),
# mirroring the Plex server's /data/<lib>/… layout so plex_path_map (/data=/plex-media) resolves.
# (The prior CIFS/PLEX_SMB_* approach for the whole nas.lan share lives in git history.)
plex_media:
driver: local
driver_opts:
type: none
o: bind
device: "${PLEX_MEDIA_HOST_PATH:-/run/desktop/mnt/host/d/plex-test-media}"