feat(files): every file sits in a library folder, and keeps its own expiry

The Files library starts here. A user's every download, clip and upload now
lands in a source folder made per origin (YouTube, Plex clips, uploads, one per
other site), decided once in downloads/folders.py at every place a job is made.
How long a file stays moves from the shared asset to the user's job: stamped
from its folder when it is done, never for a file the user keeps. The GC warns
each user once, then takes the expired file out of their library; the cache cap
never evicts a kept file, and an evicted one leaves every holder's library.

The media mount is /files now (FILES_ROOT, FILES_HOST_PATH, volume
siftlode_files); docs/self-hosting.md says how to carry an old volume over.
Migration 0102 places every existing job and carries each done file's expiry
over from its asset.

safe_abs_path takes a missing path and answers None, which drops one baselined
type error.
This commit is contained in:
2026-10-11 03:30:34 +02:00
parent 07e0990ecf
commit bf99444ded
49 changed files with 984 additions and 226 deletions
+7 -7
View File
@@ -17,7 +17,7 @@
# TOKEN_ENCRYPTION_KEY (Fernet), GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET,
# OAUTH_REDIRECT_URL=https://siftlode.example.com/auth/callback, ALLOWED_EMAILS, ADMIN_EMAILS,
# YOUTUBE_API_KEY. APP_VERSION is exported by the deploy step (selects the image tag).
# Optional: DOWNLOAD_HOST_PATH (host dir for the download tree) and PLEX_MEDIA_HOST_PATH (host dir
# Optional: FILES_HOST_PATH (host dir for the download tree) and PLEX_MEDIA_HOST_PATH (host dir
# holding the Plex library, mounted read-only for the optional Plex module).
name: siftlode
@@ -73,7 +73,7 @@ services:
SCHEDULER_ENABLED: "true"
# Download center: the API serves finished files + builds filmstrips (read-only root is fine —
# it only writes to the /downloads mount + tmpfs). The yt-dlp job loop runs in `worker` below.
DOWNLOAD_ROOT: /downloads
FILES_ROOT: /files
WORKER_ENABLED: "false"
depends_on:
db:
@@ -84,9 +84,9 @@ services:
# (10.x.x.x:8080) and on the trusted home LAN. Never forwarded to the public internet.
- "8080:8000"
volumes:
# Downloaded media. Defaults to a Docker-managed named volume; set DOWNLOAD_HOST_PATH in .env
# Downloaded media. Defaults to a Docker-managed named volume; set FILES_HOST_PATH in .env
# to a host directory (e.g. one your Plex server reads) to keep the Plex-style tree there.
- ${DOWNLOAD_HOST_PATH:-siftlode_downloads}:/downloads
- ${FILES_HOST_PATH:-siftlode_files}:/files
# Plex media, read-only. Only used when the optional Plex module is enabled: set
# PLEX_MEDIA_HOST_PATH in .env to the host directory holding the Plex library (the local side
# of plex_path_map, default /plex-media). Left unset → an empty named volume (harmless; the
@@ -124,7 +124,7 @@ services:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB:-siftlode}
# The API owns the scheduler; the worker must not also run it.
SCHEDULER_ENABLED: "false"
DOWNLOAD_ROOT: /downloads
FILES_ROOT: /files
WORKER_ENABLED: "true"
depends_on:
db:
@@ -137,7 +137,7 @@ services:
condition: service_started
networks: [internal]
volumes:
- ${DOWNLOAD_HOST_PATH:-siftlode_downloads}:/downloads
- ${FILES_HOST_PATH:-siftlode_files}:/files
# Plex media (read-only) — same mount as the api; used by Plex clip jobs (a later phase).
- ${PLEX_MEDIA_HOST_PATH:-siftlode_plex_media}:/plex-media:ro
mem_limit: 2g
@@ -167,7 +167,7 @@ services:
volumes:
siftlode_pgdata:
siftlode_downloads:
siftlode_files:
# Empty fallback for the Plex media mount when PLEX_MEDIA_HOST_PATH is unset (Plex module off).
siftlode_plex_media: