siftlode/docs/self-hosting.md

Self-hosting Siftlode

Run your own private Siftlode instance with Docker. You don't need the source code — the app runs from a prebuilt image, and everything user-facing (your admin account, Google sign-in, email) is configured in a web wizard on first start. There's no editing of config files by hand.

What you need

• A machine with Docker and the Docker Compose plugin (Docker Desktop on Windows/macOS, or Docker Engine on Linux).
• A few hundred MB of disk and ~1 GB RAM free for the app itself. The Files library stores media too — budget disk for whatever you download (it's bounded by per-user quotas you set as admin).
• Optional: a domain name + reverse proxy if you want HTTPS / public access (see below).

1. Get the files

Download these into a new, empty folder:

• docker-compose.selfhost.yml
• install.sh (Linux/macOS) or install.ps1 (Windows)

The app image is published at git.b1fr0st.eu/peter/siftlode (public — no login to pull).

2. Run the installer

Linux / macOS:

chmod +x install.sh
./install.sh

Windows (PowerShell):

./install.ps1

The installer asks for the public URL where the instance will be reached (just press Enter for http://localhost:8080 to try it locally). It then:

• generates the secrets it needs (SECRET_KEY, TOKEN_ENCRYPTION_KEY, a database password) into a local .env file — keep that file private. Do not change TOKEN_ENCRYPTION_KEY once the app has stored any secret (Google credentials, API keys, SMTP password, Plex token): those values are encrypted with it, and a changed key makes them unreadable. If it is ever lost or rotated, clear the affected rows from the system_config table and re-enter the secrets in the admin Configuration page. Share links are kept with it too: after a key change every existing link still opens, but its address can no longer be shown — its owner makes a new link to share it again,
• pulls the image and starts the app + database,
• prints the setup wizard URL, which looks like …/setup?token=….

3. Finish in the web wizard

Open the printed setup URL in your browser. The one-time token in it means only you (with access to the server logs) can run setup. Then click through:

1. Admin account — your email + a password. This is how you'll sign in.
2. Google sign-in (optional) — paste a Google OAuth client ID + secret to enable "Sign in with Google" and pulling your YouTube subscriptions. Skip it to use email + password only.
3. Email / SMTP (optional) — an SMTP server so the app can send verification and notification emails. Skip it — without email, new registrations are simply approved by you (the admin) instead.
4. Finish — the wizard disappears, the instance is now configured, and you land on the sign-in page. Log in with the admin account you just created.

That's it. You can change any of the optional settings later under the admin Configuration page.

Set a password even if you sign in with Google (Settings → Account → Password). If Google sign-in stops working — the client secret rotated, the consent revoked, Google down — email and password still let you in. An account that only ever used Google is otherwise locked out until an admin creates a set-password link for it (Users).

The setup wizard only exists until you finish it. After that, the setup routes are disabled and the token is invalidated — there's no setup surface left on a configured instance.

Getting a Google OAuth client (optional)

Only needed for "Sign in with Google" / YouTube access. In the Google Cloud Console: create a project → APIs & Services → Credentials → Create credentials → OAuth client ID → Web application. Add your instance's …/auth/callback URL as an Authorized redirect URI, then copy the client ID and secret into the wizard's Google step. (Enable the YouTube Data API v3 for the project too.)

HTTPS / public access

The app is served on port 8080 over plain HTTP, which is fine for a LAN or a quick trial. For public access, put a reverse proxy (Caddy, Nginx, Traefik…) in front to terminate TLS, and set the public URL in the installer to your https://… address (this also marks the session cookie secure). If you've already run the installer, edit OAUTH_REDIRECT_URL in .env to the https callback URL and docker compose -f docker-compose.selfhost.yml up -d.

Behind a proxy, set TRUSTED_PROXY_IPS. The login / registration / password-reset rate limiters identify callers by IP. Behind a proxy every request arrives from the proxy, so without this the limits would apply to everyone together. Set TRUSTED_PROXY_IPS in .env to the address your proxy connects to the app from (as the app sees it — e.g. its Docker/host/LAN IP, not the public client address), and the app will read the real client IP from the proxy's X-Forwarded-For header:

TRUSTED_PROXY_IPS=172.18.0.1        # comma-separate multiple proxies

Only that peer is trusted, so a request sent straight to port 8080 (bypassing the proxy) can't forge X-Forwarded-For to dodge the limits — it's rate-limited by its real address. Leave it empty if the app is exposed directly with no proxy. Make sure your proxy actually sets X-Forwarded-For (Caddy's reverse_proxy and Nginx's proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for both do).

Files (media storage)

The stack includes Files: one library that saves videos to the server with yt-dlp (Plex-friendly folders + .nfo/poster), lets users trim/crop/join clips, and shares them. It runs two extra containers that come up automatically — a worker (the download/edit job loop) and a small bgutil-pot sidecar (mints YouTube tokens so downloads aren't bot-blocked). No configuration is required; per-user storage quotas are set on the admin Users → Storage tab.

By default the media lives in a Docker-managed volume (siftlode_files). To keep it somewhere you can reach from other apps — e.g. a folder your Plex server indexes — point it at a host directory by adding this to .env and re-running up -d:

FILES_HOST_PATH=/mnt/media/youtube

The directory must be writable by the container user (uid 1000): sudo chown -R 1000:1000 <dir>.

Plex library (optional)

If you run a Plex server, Siftlode can browse and play your movies and shows beside your YouTube feed. Two things have to line up:

1. Server URL + token, entered as admin under Configuration → Plex. That is how Siftlode reads your libraries.
2. The media files themselves, mounted into the container read-only. Siftlode streams the files directly rather than proxying Plex, so it has to be able to open them.

Point PLEX_MEDIA_HOST_PATH at the directory holding your library and re-run up -d:

PLEX_MEDIA_HOST_PATH=/mnt/media

It appears inside the container at /plex-media, read-only. Siftlode then rewrites the paths Plex reports into that mount using the path map on the same admin page — its default, /data=/plex-media, means "wherever Plex says /data/movies/…, look in /plex-media/movies/…". Set the left-hand side to whatever prefix your Plex server uses, and mount the matching host directory so the rest of the path still lines up.

Leave PLEX_MEDIA_HOST_PATH unset if you don't use Plex — the mount falls back to an empty volume and the module stays off.

Day-to-day

# Logs / status (api = web, worker = downloads/edits)
docker compose -f docker-compose.selfhost.yml logs -f api
docker compose -f docker-compose.selfhost.yml logs -f worker
docker compose -f docker-compose.selfhost.yml ps

# Stop
docker compose -f docker-compose.selfhost.yml down

Backups

A backup is three things, not one. Copying only the database gets you an instance that starts and then cannot read its own secrets.

1. The database — accounts, subscriptions, playlists, the video catalog, and every setting you made in the admin pages. It lives in the siftlode_pgdata Docker volume; dump it with the command below.
2. .env — in particular TOKEN_ENCRYPTION_KEY. Your Google client secret, SMTP password, Plex token and every stored refresh token are encrypted with it inside the database. Restore a dump against a different key and those values are lost — you would have to clear the affected rows from system_config and re-enter every secret by hand — and no existing share link's address can be shown again (the links still open; their owners make new ones to share). Keep .env beside the dump, and just as private.
3. Downloaded media (only if you use Files) — the siftlode_files volume, or whatever FILES_HOST_PATH points at. Not needed to bring the instance back; the jobs and their metadata are in the database, the files themselves are not.

Dump and restore the database with these two commands. They also exist as scripts (scripts/backup.sh + scripts/restore.sh, or backup.ps1 / restore.ps1 on Windows) if you have the source checkout:

If you customised POSTGRES_USER / POSTGRES_DB in .env, substitute them for siftlode below — the scripts read them from the environment, these one-liners cannot.

# Back up. Write to a .part name and rename only on success: redirecting straight into the final
# name creates the file BEFORE pg_dump runs, so a failed dump leaves a plausible-looking 0-byte
# file in backups/ that you will not discover until the day you need it.
mkdir -p backups && out="backups/siftlode-$(date +%Y%m%d-%H%M%S).dump" && \
  docker compose -f docker-compose.selfhost.yml exec -T db \
    pg_dump -U siftlode -Fc siftlode > "$out.part" \
  && [ -s "$out.part" ] && mv "$out.part" "$out" && echo "wrote $out" \
  || { rm -f "$out.part"; echo "BACKUP FAILED" >&2; false; }

# Restore. Stop the app first: pg_restore --clean DROPs each object in its own transaction, and a
# running api holds locks that make those DROPs queue — abort the wait and you are left with a
# half-dropped database. Restoring with the app down also means nothing writes into a schema that
# is only partly back.
docker compose -f docker-compose.selfhost.yml stop api worker
docker compose -f docker-compose.selfhost.yml exec -T db \
  pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump

# Start it again only once the restore reported no errors — and, if you are rolling a release back,
# only AFTER pinning the older IMAGE_TAG. Every start runs `alembic upgrade head`, so bringing the
# newer image back here would re-apply the migrations you just restored away. Use `up -d`, not
# `start`: `start` relaunches the existing container, so it ignores the tag you just pinned and
# comes back on the image you are rolling back FROM.
docker compose -f docker-compose.selfhost.yml up -d

On Windows, do not pipe the dump through PowerShell — it corrupts binary data. Write it inside the container and copy it out instead. Check the exit code between the two steps and use a timestamped name: the dump inside the container keeps its name, so a cp that runs after a failed dump would copy out the previous run's file and quietly overwrite your only backup with a stale one.

Paste it as ONE block. throw only aborts the statement it is in, so as separate pasted lines the copy would still run after a failed dump — and copy out the previous run's file:

& {
  # Inside the block, because a scriptblock inherits the caller's preference — under a profile that
  # sets SilentlyContinue, every `throw` below would be swallowed and the block would run on.
  $ErrorActionPreference = "Stop"
  $compose = "docker-compose.selfhost.yml"
  New-Item -ItemType Directory -Force -Path backups | Out-Null
  $out = "backups\siftlode-$(Get-Date -Format yyyyMMdd-HHmmss).dump"
  docker compose -f $compose exec -T db pg_dump -U siftlode -Fc siftlode -f /tmp/s.dump
  $code = $LASTEXITCODE
  # Remove the in-container dump on the failure path too: leaving it there is how a LATER failed run
  # copies out this run's file and overwrites a good backup with a stale one.
  if ($code -ne 0) { docker compose -f $compose exec -T db rm -f /tmp/s.dump; throw "BACKUP FAILED - pg_dump returned $code" }
  docker compose -f $compose cp db:/tmp/s.dump $out
  $code = $LASTEXITCODE
  docker compose -f $compose exec -T db rm -f /tmp/s.dump
  # Remove the partial file too: `cp` streams, so a mid-stream failure leaves a truncated dump
  # under the final name — the plausible-looking bad backup the .part dance above avoids.
  if ($code -ne 0) { Remove-Item $out -ErrorAction SilentlyContinue; throw "BACKUP FAILED - could not copy the dump out (exit $code)" }
  if ((Get-Item $out).Length -eq 0) { Remove-Item $out; throw "BACKUP FAILED - the dump came out empty" }
  "wrote $out"
}

Upgrading

Take a backup first. Every start runs any new database migrations automatically, and a migration is not reversible by pulling the old image back — see rollback below.

docker compose -f docker-compose.selfhost.yml pull
docker compose -f docker-compose.selfhost.yml up -d

That tracks latest, which moves whenever a release is cut. To decide for yourself when you move, pin a version by putting it in .env:

IMAGE_TAG=0.80.0

Then pull + up -d stays on that version until you change the line. Released versions and what changed in them are listed in the app itself, under Release Notes in the account menu, and in CHANGELOG.md.

The media mount was renamed (the Files library): it is mounted at /files (FILES_ROOT), set from FILES_HOST_PATH, and the default volume is siftlode_files. Until you move, nothing breaks: a compose file from before still mounts /downloads and sets DOWNLOAD_ROOT, and the app still reads that name. To move to the new layout, download the new docker-compose.selfhost.yml over your old one, then:

• If you set a host directory, nothing to do: the new compose file still reads DOWNLOAD_HOST_PATH when FILES_HOST_PATH is not set. Rename it in .env when it suits you.

• If you used the default volume, copy it over once before you start the new version (Docker cannot rename a volume). Compose names a volume after the project, which is your folder's name unless you set one, so look the old one up rather than guessing it:

  docker compose -f docker-compose.selfhost.yml down
  docker volume ls --format '{{.Name}}' | grep 'siftlode_downloads$'   # e.g. mysiftlode_siftlode_downloads
  OLD=mysiftlode_siftlode_downloads                                    # the name it printed
  NEW="${OLD%siftlode_downloads}siftlode_files"
  docker volume create "$NEW"
  docker run --rm -v "$OLD":/from -v "$NEW":/to alpine cp -a /from/. /to/
  docker compose -f docker-compose.selfhost.yml up -d
  

  Open the Files library and check your files play; only then remove the old volume (docker volume rm "$OLD").

• If you build from source, a git pull brings the new docker-compose.yml, whose default volume is files (it was downloads). Copy it over before you start the new version — with your own compose file, not the self-host one above, whose volumes are other ones:

  docker compose down
  docker volume ls --format '{{.Name}}' | grep '_downloads$'   # e.g. siftlode_downloads
  OLD=siftlode_downloads                                         # the name it printed
  NEW="${OLD%downloads}files"
  docker volume create "$NEW"
  docker run --rm -v "$OLD":/from -v "$NEW":/to alpine cp -a /from/. /to/
  docker compose up -d --build
  

If the files library is not mounted where the app looks, the API says so in its log at start (The files library is not reachable), and the download worker refuses to start rather than write files into a container that throws them away. A fresh, empty volume in the library's place (the old one not copied over) is a real mount, so the worker runs; the API's log says that too, at start (The files library is empty but the database names files). Copy the old volume over, as above.

The registry keeps only the newest few releases, not every version the changelog lists. A pin is "decide when you take the next hop", not "stay here forever": once you are several releases behind, your pinned tag can no longer be pulled — which matters the day you rebuild the host, prune your Docker images, or move to a new machine. A container that is already running is unaffected (Docker keeps the image locally), and you can always go back to latest or build the image yourself from the repository. If you want to sit on one version long term, keep your own copy: docker save git.b1fr0st.eu/peter/siftlode:0.80.0 -o siftlode-0.80.0.tar.

Rollback

If a release misbehaves, set IMAGE_TAG back to the previous version and up -d. That is enough as long as the release did not change the database schema. If it did, the older image will refuse to start against the newer schema.

Restoring your pre-upgrade dump is the recommended way out of that — it puts the database back exactly where the old image expects it, in one step, with no revision arithmetic:

# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. restore the dump you took before upgrading (see Backups above). Leave the app stopped:
#    starting it here would run the NEWER image's migrations against the database you just restored.
docker compose -f docker-compose.selfhost.yml exec -T db \
  pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
# 3. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d

If you have no usable dump, you can step the schema back instead — but not with alembic downgrade -1. One release routinely ships several migrations (0.80.0 shipped four), so -1 undoes only the newest one and leaves the schema at a revision the older image also cannot resolve: you lose that migration's data and still get a crash loop. Downgrade to the revision the target version expects, which that version can tell you itself:

# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. ask the version you are rolling back TO which schema revision it expects
IMAGE_TAG=0.79.0 docker compose -f docker-compose.selfhost.yml run --rm api alembic heads
# 3. step the schema back to exactly that revision — run this with the CURRENT (newer) image,
#    because it is the one that ships the migrations being undone
docker compose -f docker-compose.selfhost.yml run --rm api alembic downgrade <revision from step 2>
# 4. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d

A downgrade drops the columns the newer version added, so anything stored only in those columns is gone. That is the price of not having the dump.

A migration failed on start

The container will keep restarting (restart: unless-stopped), and worker will sit waiting for an api that never becomes healthy. Read the reason first:

docker compose -f docker-compose.selfhost.yml logs api | tail -50

Then either fix the cause and let it retry, or roll back: restore your pre-upgrade dump, pin the previous IMAGE_TAG, and up -d. Do not delete the siftlode_pgdata volume to "start clean" unless you have a dump you have actually tested restoring — that volume is your instance.