siftlode/docs/self-hosting.md

# Self-hosting Siftlode

Run your own private Siftlode instance with Docker. You don't need the source code — the app runs
from a prebuilt image, and everything user-facing (your admin account, Google sign-in, email) is
configured in a web wizard on first start. There's no editing of config files by hand.

## What you need

- A machine with **Docker** and the **Docker Compose plugin** (Docker Desktop on Windows/macOS,
  or Docker Engine on Linux).
- A few hundred MB of disk and ~1 GB RAM free for the app itself. The Files library
  stores media too — budget disk for whatever you download (it's bounded by per-user quotas you
  set as admin).
- Optional: a domain name + reverse proxy if you want HTTPS / public access (see below).

## 1. Get the files

Download these into a new, empty folder:

- `docker-compose.selfhost.yml`
- `install.sh` (Linux/macOS) **or** `install.ps1` (Windows)

The app image is published at `git.b1fr0st.eu/peter/siftlode` (public — no login to pull).

## 2. Run the installer

**Linux / macOS:**

```bash
chmod +x install.sh
./install.sh
```

**Windows (PowerShell):**

```powershell
./install.ps1
```

The installer asks for the **public URL** where the instance will be reached (just press Enter for
`http://localhost:8080` to try it locally). It then:

- generates the secrets it needs (`SECRET_KEY`, `TOKEN_ENCRYPTION_KEY`, a database password) into a
  local `.env` file — keep that file private. **Do not change `TOKEN_ENCRYPTION_KEY` once the app has
  stored any secret** (Google credentials, API keys, SMTP password, Plex token): those values are
  encrypted with it, and a changed key makes them unreadable. If it is ever lost or rotated, clear the
  affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page,
- pulls the image and starts the app + database,
- prints the **setup wizard URL**, which looks like `…/setup?token=…`.

## 3. Finish in the web wizard

Open the printed setup URL in your browser. The one-time token in it means only you (with access to
the server logs) can run setup. Then click through:

1. **Admin account** — your email + a password. This is how you'll sign in.
2. **Google sign-in** *(optional)* — paste a Google OAuth client ID + secret to enable "Sign in with
   Google" and pulling your YouTube subscriptions. Skip it to use email + password only.
3. **Email / SMTP** *(optional)* — an SMTP server so the app can send verification and notification
   emails. Skip it — without email, new registrations are simply approved by you (the admin) instead.
4. **Finish** — the wizard disappears, the instance is now configured, and you land on the sign-in
   page. Log in with the admin account you just created.

That's it. You can change any of the optional settings later under the admin **Configuration** page.

> The setup wizard only exists until you finish it. After that, the setup routes are disabled and
> the token is invalidated — there's no setup surface left on a configured instance.

## Getting a Google OAuth client (optional)

Only needed for "Sign in with Google" / YouTube access. In the
[Google Cloud Console](https://console.cloud.google.com/): create a project → **APIs & Services →
Credentials → Create credentials → OAuth client ID** → *Web application*. Add your instance's
`…/auth/callback` URL as an **Authorized redirect URI**, then copy the **client ID** and **secret**
into the wizard's Google step. (Enable the **YouTube Data API v3** for the project too.)

## HTTPS / public access

The app is served on port `8080` over plain HTTP, which is fine for a LAN or a quick trial. For
public access, put a reverse proxy (Caddy, Nginx, Traefik…) in front to terminate TLS, and set the
**public URL** in the installer to your `https://…` address (this also marks the session cookie
secure). If you've already run the installer, edit `OAUTH_REDIRECT_URL` in `.env` to the https
callback URL and `docker compose -f docker-compose.selfhost.yml up -d`.

**Behind a proxy, set `TRUSTED_PROXY_IPS`.** The login / registration / password-reset rate limiters
identify callers by IP. Behind a proxy every request arrives from the proxy, so without this the
limits would apply to everyone together. Set `TRUSTED_PROXY_IPS` in `.env` to the address your proxy
connects to the app *from* (as the app sees it — e.g. its Docker/host/LAN IP, **not** the public
client address), and the app will read the real client IP from the proxy's `X-Forwarded-For` header:

```bash
TRUSTED_PROXY_IPS=172.18.0.1        # comma-separate multiple proxies
```

Only that peer is trusted, so a request sent straight to port `8080` (bypassing the proxy) can't forge
`X-Forwarded-For` to dodge the limits — it's rate-limited by its real address. Leave it empty if the
app is exposed directly with no proxy. Make sure your proxy actually sets `X-Forwarded-For` (Caddy's
`reverse_proxy` and Nginx's `proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for` both do).

## Files (media storage)

The stack includes **Files**: one library that saves videos to the server
with yt-dlp (Plex-friendly folders + `.nfo`/poster), lets users trim/crop/join clips, and shares
them. It runs two extra containers that come up automatically — a `worker` (the download/edit job
loop) and a small `bgutil-pot` sidecar (mints YouTube tokens so downloads aren't bot-blocked). No
configuration is required; per-user storage quotas are set on the admin **Users → Storage** tab.

By default the media lives in a Docker-managed volume (`siftlode_files`). To keep it somewhere
you can reach from other apps — e.g. a folder your **Plex** server indexes — point it at a host
directory by adding this to `.env` and re-running `up -d`:

```bash
FILES_HOST_PATH=/mnt/media/youtube
```

The directory must be writable by the container user (uid `1000`): `sudo chown -R 1000:1000 <dir>`.

## Plex library (optional)

If you run a **Plex** server, Siftlode can browse and play your movies and shows beside your YouTube
feed. Two things have to line up:

1. **Server URL + token**, entered as admin under **Configuration → Plex**. That is how Siftlode reads
   your libraries.
2. **The media files themselves**, mounted into the container read-only. Siftlode streams the files
   directly rather than proxying Plex, so it has to be able to open them.

Point `PLEX_MEDIA_HOST_PATH` at the directory holding your library and re-run `up -d`:

```bash
PLEX_MEDIA_HOST_PATH=/mnt/media
```

It appears inside the container at `/plex-media`, read-only. Siftlode then rewrites the paths Plex
reports into that mount using the **path map** on the same admin page — its default, `/data=/plex-media`,
means "wherever Plex says `/data/movies/…`, look in `/plex-media/movies/…`". Set the left-hand side to
whatever prefix your Plex server uses, and mount the matching host directory so the rest of the path
still lines up.

Leave `PLEX_MEDIA_HOST_PATH` unset if you don't use Plex — the mount falls back to an empty volume and
the module stays off.

## Day-to-day

```bash
# Logs / status (api = web, worker = downloads/edits)
docker compose -f docker-compose.selfhost.yml logs -f api
docker compose -f docker-compose.selfhost.yml logs -f worker
docker compose -f docker-compose.selfhost.yml ps

# Stop
docker compose -f docker-compose.selfhost.yml down
```

## Backups

**A backup is three things, not one.** Copying only the database gets you an instance that starts and
then cannot read its own secrets.

1. **The database** — accounts, subscriptions, playlists, the video catalog, and every setting you
   made in the admin pages. It lives in the `siftlode_pgdata` Docker volume; dump it with the command
   below.
2. **`.env`** — in particular `TOKEN_ENCRYPTION_KEY`. Your Google client secret, SMTP password, Plex
   token and every stored refresh token are encrypted with it *inside* the database. Restore a dump
   against a different key and those values are lost — you would have to clear the affected rows from
   `system_config` and re-enter every secret by hand. Keep `.env` beside the dump, and just as private.
3. **Downloaded media** *(only if you use Files)* — the `siftlode_files` volume, or
   whatever `FILES_HOST_PATH` points at. Not needed to bring the instance back; the jobs and their
   metadata are in the database, the files themselves are not.

Dump and restore the database with these two commands. They also exist as scripts
(`scripts/backup.sh` + `scripts/restore.sh`, or `backup.ps1` / `restore.ps1` on Windows) if you have
the source checkout:

If you customised `POSTGRES_USER` / `POSTGRES_DB` in `.env`, substitute them for `siftlode` below —
the scripts read them from the environment, these one-liners cannot.

```bash
# Back up. Write to a .part name and rename only on success: redirecting straight into the final
# name creates the file BEFORE pg_dump runs, so a failed dump leaves a plausible-looking 0-byte
# file in backups/ that you will not discover until the day you need it.
mkdir -p backups && out="backups/siftlode-$(date +%Y%m%d-%H%M%S).dump" && \
  docker compose -f docker-compose.selfhost.yml exec -T db \
    pg_dump -U siftlode -Fc siftlode > "$out.part" \
  && [ -s "$out.part" ] && mv "$out.part" "$out" && echo "wrote $out" \
  || { rm -f "$out.part"; echo "BACKUP FAILED" >&2; false; }

# Restore. Stop the app first: pg_restore --clean DROPs each object in its own transaction, and a
# running api holds locks that make those DROPs queue — abort the wait and you are left with a
# half-dropped database. Restoring with the app down also means nothing writes into a schema that
# is only partly back.
docker compose -f docker-compose.selfhost.yml stop api worker
docker compose -f docker-compose.selfhost.yml exec -T db \
  pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump

# Start it again only once the restore reported no errors — and, if you are rolling a release back,
# only AFTER pinning the older IMAGE_TAG. Every start runs `alembic upgrade head`, so bringing the
# newer image back here would re-apply the migrations you just restored away. Use `up -d`, not
# `start`: `start` relaunches the existing container, so it ignores the tag you just pinned and
# comes back on the image you are rolling back FROM.
docker compose -f docker-compose.selfhost.yml up -d
```

On Windows, do **not** pipe the dump through PowerShell — it corrupts binary data. Write it inside the
container and copy it out instead. Check the exit code between the two steps and use a timestamped
name: the dump inside the container keeps its name, so a `cp` that runs after a *failed* dump would
copy out the previous run's file and quietly overwrite your only backup with a stale one.

Paste it as ONE block. `throw` only aborts the statement it is in, so as separate pasted lines the
copy would still run after a failed dump — and copy out the previous run's file:

```powershell
& {
  # Inside the block, because a scriptblock inherits the caller's preference — under a profile that
  # sets SilentlyContinue, every `throw` below would be swallowed and the block would run on.
  $ErrorActionPreference = "Stop"
  $compose = "docker-compose.selfhost.yml"
  New-Item -ItemType Directory -Force -Path backups | Out-Null
  $out = "backups\siftlode-$(Get-Date -Format yyyyMMdd-HHmmss).dump"
  docker compose -f $compose exec -T db pg_dump -U siftlode -Fc siftlode -f /tmp/s.dump
  $code = $LASTEXITCODE
  # Remove the in-container dump on the failure path too: leaving it there is how a LATER failed run
  # copies out this run's file and overwrites a good backup with a stale one.
  if ($code -ne 0) { docker compose -f $compose exec -T db rm -f /tmp/s.dump; throw "BACKUP FAILED - pg_dump returned $code" }
  docker compose -f $compose cp db:/tmp/s.dump $out
  $code = $LASTEXITCODE
  docker compose -f $compose exec -T db rm -f /tmp/s.dump
  # Remove the partial file too: `cp` streams, so a mid-stream failure leaves a truncated dump
  # under the final name — the plausible-looking bad backup the .part dance above avoids.
  if ($code -ne 0) { Remove-Item $out -ErrorAction SilentlyContinue; throw "BACKUP FAILED - could not copy the dump out (exit $code)" }
  if ((Get-Item $out).Length -eq 0) { Remove-Item $out; throw "BACKUP FAILED - the dump came out empty" }
  "wrote $out"
}
```

## Upgrading

**Take a backup first.** Every start runs any new database migrations automatically, and a migration
is not reversible by pulling the old image back — see rollback below.

```bash
docker compose -f docker-compose.selfhost.yml pull
docker compose -f docker-compose.selfhost.yml up -d
```

That tracks `latest`, which moves whenever a release is cut. **To decide for yourself when you move,
pin a version** by putting it in `.env`:

```bash
IMAGE_TAG=0.80.0
```

Then `pull` + `up -d` stays on that version until you change the line. Released versions and what
changed in them are listed in the app itself, under **Release Notes** in the account menu, and in
[`CHANGELOG.md`](../CHANGELOG.md).

**The media mount was renamed** (the Files library): it is mounted at `/files` (`FILES_ROOT`), set
from `FILES_HOST_PATH`, and the default volume is `siftlode_files`. Until you move, nothing breaks: a
compose file from before still mounts `/downloads` and sets `DOWNLOAD_ROOT`, and the app still reads
that name. To move to the new layout, download the new
[`docker-compose.selfhost.yml`](../docker-compose.selfhost.yml) over your old one, then:

- **If you set a host directory**, nothing to do: the new compose file still reads
  `DOWNLOAD_HOST_PATH` when `FILES_HOST_PATH` is not set. Rename it in `.env` when it suits you.
- **If you used the default volume**, copy it over once before you start the new version (Docker
  cannot rename a volume). Compose names a volume after the project, which is your folder's name
  unless you set one, so look the old one up rather than guessing it:

  ```bash
  docker compose -f docker-compose.selfhost.yml down
  docker volume ls --format '{{.Name}}' | grep 'siftlode_downloads$'   # e.g. mysiftlode_siftlode_downloads
  OLD=mysiftlode_siftlode_downloads                                    # the name it printed
  NEW="${OLD%siftlode_downloads}siftlode_files"
  docker volume create "$NEW"
  docker run --rm -v "$OLD":/from -v "$NEW":/to alpine cp -a /from/. /to/
  docker compose -f docker-compose.selfhost.yml up -d
  ```

  Open the Files library and check your files play; only then remove the old volume
  (`docker volume rm "$OLD"`).
- **If you build from source**, a `git pull` brings the new `docker-compose.yml`, whose default
  volume is `files` (it was `downloads`). Copy it over before you start the new version — with your
  own compose file, not the self-host one above, whose volumes are other ones:

  ```bash
  docker compose down
  docker volume ls --format '{{.Name}}' | grep '_downloads$'   # e.g. siftlode_downloads
  OLD=siftlode_downloads                                         # the name it printed
  NEW="${OLD%downloads}files"
  docker volume create "$NEW"
  docker run --rm -v "$OLD":/from -v "$NEW":/to alpine cp -a /from/. /to/
  docker compose up -d --build
  ```

If the files library is not mounted where the app looks, the API says so in its log at start
(`The files library is not reachable`), and the download worker refuses to start rather than write
files into a container that throws them away. A fresh, empty volume in the library's place (the old
one not copied over) is a real mount, so the worker runs; the API's log says that too, at start
(`The files library is empty but the database names files`). Copy the old volume over, as above.

**The registry keeps only the newest few releases, not every version the changelog lists.** A pin is
"decide when you take the next hop", not "stay here forever": once you are several releases behind,
your pinned tag can no longer be pulled — which matters the day you rebuild the host, prune your
Docker images, or move to a new machine. A container that is already running is unaffected (Docker
keeps the image locally), and you can always go back to `latest` or build the image yourself from the
repository. If you want to sit on one version long term, keep your own copy:
`docker save git.b1fr0st.eu/peter/siftlode:0.80.0 -o siftlode-0.80.0.tar`.

### Rollback

If a release misbehaves, set `IMAGE_TAG` back to the previous version and `up -d`. That is enough
**as long as the release did not change the database schema**. If it did, the older image will refuse
to start against the newer schema.

**Restoring your pre-upgrade dump is the recommended way out of that** — it puts the database back
exactly where the old image expects it, in one step, with no revision arithmetic:

```bash
# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. restore the dump you took before upgrading (see Backups above). Leave the app stopped:
#    starting it here would run the NEWER image's migrations against the database you just restored.
docker compose -f docker-compose.selfhost.yml exec -T db \
  pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
# 3. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d
```

If you have no usable dump, you can step the schema back instead — but **not with `alembic downgrade
-1`**. One release routinely ships several migrations (0.80.0 shipped four), so `-1` undoes only the
newest one and leaves the schema at a revision the older image *also* cannot resolve: you lose that
migration's data and still get a crash loop. Downgrade to the revision the target version expects,
which that version can tell you itself:

```bash
# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. ask the version you are rolling back TO which schema revision it expects
IMAGE_TAG=0.79.0 docker compose -f docker-compose.selfhost.yml run --rm api alembic heads
# 3. step the schema back to exactly that revision — run this with the CURRENT (newer) image,
#    because it is the one that ships the migrations being undone
docker compose -f docker-compose.selfhost.yml run --rm api alembic downgrade <revision from step 2>
# 4. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d
```

A downgrade drops the columns the newer version added, so anything stored only in those columns is
gone. That is the price of not having the dump.

### A migration failed on start

The container will keep restarting (`restart: unless-stopped`), and `worker` will sit waiting for an
`api` that never becomes healthy. Read the reason first:

```bash
docker compose -f docker-compose.selfhost.yml logs api | tail -50
```

Then either fix the cause and let it retry, or roll back: restore your pre-upgrade dump, pin the
previous `IMAGE_TAG`, and `up -d`. Do not delete the `siftlode_pgdata` volume to "start clean" unless
you have a dump you have actually tested restoring — that volume *is* your instance.