siftlode/docs/self-hosting.md

# Self-hosting Siftlode

Run your own private Siftlode instance with Docker. You don't need the source code — the app runs
from a prebuilt image, and everything user-facing (your admin account, Google sign-in, email) is
configured in a web wizard on first start. There's no editing of config files by hand.

## What you need

- A machine with **Docker** and the **Docker Compose plugin** (Docker Desktop on Windows/macOS,
  or Docker Engine on Linux).
- A few hundred MB of disk and ~1 GB RAM free for the app itself. The optional Download Center
  stores media too — budget disk for whatever you download (it's bounded by per-user quotas you
  set as admin).
- Optional: a domain name + reverse proxy if you want HTTPS / public access (see below).

## 1. Get the files

Download these into a new, empty folder:

- `docker-compose.selfhost.yml`
- `install.sh` (Linux/macOS) **or** `install.ps1` (Windows)

The app image is published at `git.b1fr0st.eu/peter/siftlode` (public — no login to pull).

## 2. Run the installer

**Linux / macOS:**

```bash
chmod +x install.sh
./install.sh
```

**Windows (PowerShell):**

```powershell
./install.ps1
```

The installer asks for the **public URL** where the instance will be reached (just press Enter for
`http://localhost:8080` to try it locally). It then:

- generates the secrets it needs (`SECRET_KEY`, `TOKEN_ENCRYPTION_KEY`, a database password) into a
  local `.env` file — keep that file private. **Do not change `TOKEN_ENCRYPTION_KEY` once the app has
  stored any secret** (Google credentials, API keys, SMTP password, Plex token): those values are
  encrypted with it, and a changed key makes them unreadable. If it is ever lost or rotated, clear the
  affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page,
- pulls the image and starts the app + database,
- prints the **setup wizard URL**, which looks like `…/setup?token=…`.

## 3. Finish in the web wizard

Open the printed setup URL in your browser. The one-time token in it means only you (with access to
the server logs) can run setup. Then click through:

1. **Admin account** — your email + a password. This is how you'll sign in.
2. **Google sign-in** *(optional)* — paste a Google OAuth client ID + secret to enable "Sign in with
   Google" and pulling your YouTube subscriptions. Skip it to use email + password only.
3. **Email / SMTP** *(optional)* — an SMTP server so the app can send verification and notification
   emails. Skip it — without email, new registrations are simply approved by you (the admin) instead.
4. **Finish** — the wizard disappears, the instance is now configured, and you land on the sign-in
   page. Log in with the admin account you just created.

That's it. You can change any of the optional settings later under the admin **Configuration** page.

> The setup wizard only exists until you finish it. After that, the setup routes are disabled and
> the token is invalidated — there's no setup surface left on a configured instance.

## Getting a Google OAuth client (optional)

Only needed for "Sign in with Google" / YouTube access. In the
[Google Cloud Console](https://console.cloud.google.com/): create a project → **APIs & Services →
Credentials → Create credentials → OAuth client ID** → *Web application*. Add your instance's
`…/auth/callback` URL as an **Authorized redirect URI**, then copy the **client ID** and **secret**
into the wizard's Google step. (Enable the **YouTube Data API v3** for the project too.)

## HTTPS / public access

The app is served on port `8080` over plain HTTP, which is fine for a LAN or a quick trial. For
public access, put a reverse proxy (Caddy, Nginx, Traefik…) in front to terminate TLS, and set the
**public URL** in the installer to your `https://…` address (this also marks the session cookie
secure). If you've already run the installer, edit `OAUTH_REDIRECT_URL` in `.env` to the https
callback URL and `docker compose -f docker-compose.selfhost.yml up -d`.

**Behind a proxy, set `TRUSTED_PROXY_IPS`.** The login / registration / password-reset rate limiters
identify callers by IP. Behind a proxy every request arrives from the proxy, so without this the
limits would apply to everyone together. Set `TRUSTED_PROXY_IPS` in `.env` to the address your proxy
connects to the app *from* (as the app sees it — e.g. its Docker/host/LAN IP, **not** the public
client address), and the app will read the real client IP from the proxy's `X-Forwarded-For` header:

```bash
TRUSTED_PROXY_IPS=172.18.0.1        # comma-separate multiple proxies
```

Only that peer is trusted, so a request sent straight to port `8080` (bypassing the proxy) can't forge
`X-Forwarded-For` to dodge the limits — it's rate-limited by its real address. Leave it empty if the
app is exposed directly with no proxy. Make sure your proxy actually sets `X-Forwarded-For` (Caddy's
`reverse_proxy` and Nginx's `proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for` both do).

## Download Center (media storage)

The stack includes a **Download Center**: an admin-enabled feature that saves videos to the server
with yt-dlp (Plex-friendly folders + `.nfo`/poster), lets users trim/crop/join clips, and shares
them. It runs two extra containers that come up automatically — a `worker` (the download/edit job
loop) and a small `bgutil-pot` sidecar (mints YouTube tokens so downloads aren't bot-blocked). No
configuration is required; per-user storage quotas are set on the admin **Downloads → System** page.

By default the media lives in a Docker-managed volume (`siftlode_downloads`). To keep it somewhere
you can reach from other apps — e.g. a folder your **Plex** server indexes — point it at a host
directory by adding this to `.env` and re-running `up -d`:

```bash
DOWNLOAD_HOST_PATH=/mnt/media/youtube
```

The directory must be writable by the container user (uid `1000`): `sudo chown -R 1000:1000 <dir>`.

## Day-to-day

```bash
# Logs / status (api = web, worker = downloads/edits)
docker compose -f docker-compose.selfhost.yml logs -f api
docker compose -f docker-compose.selfhost.yml logs -f worker
docker compose -f docker-compose.selfhost.yml ps

# Stop
docker compose -f docker-compose.selfhost.yml down
```

## Backups

**A backup is three things, not one.** Copying only the database gets you an instance that starts and
then cannot read its own secrets.

1. **The database** — accounts, subscriptions, playlists, the video catalog, and every setting you
   made in the admin pages. It lives in the `siftlode_pgdata` Docker volume; dump it with the command
   below.
2. **`.env`** — in particular `TOKEN_ENCRYPTION_KEY`. Your Google client secret, SMTP password, Plex
   token and every stored refresh token are encrypted with it *inside* the database. Restore a dump
   against a different key and those values are lost — you would have to clear the affected rows from
   `system_config` and re-enter every secret by hand. Keep `.env` beside the dump, and just as private.
3. **Downloaded media** *(only if you use the Download Center)* — the `siftlode_downloads` volume, or
   whatever `DOWNLOAD_HOST_PATH` points at. Not needed to bring the instance back; the jobs and their
   metadata are in the database, the files themselves are not.

Dump and restore the database with these two commands. They also exist as scripts
(`scripts/backup.sh` + `scripts/restore.sh`, or `backup.ps1` / `restore.ps1` on Windows) if you have
the source checkout:

```bash
# Back up (writes backups/siftlode-<timestamp>.dump)
mkdir -p backups && docker compose -f docker-compose.selfhost.yml exec -T db \
  pg_dump -U siftlode -Fc siftlode > "backups/siftlode-$(date +%Y%m%d-%H%M%S).dump"

# Restore into the running instance
docker compose -f docker-compose.selfhost.yml exec -T db \
  pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
```

On Windows, do **not** pipe the dump through PowerShell — it corrupts binary data. Write it inside the
container and copy it out instead:

```powershell
docker compose -f docker-compose.selfhost.yml exec -T db pg_dump -U siftlode -Fc siftlode -f /tmp/s.dump
docker compose -f docker-compose.selfhost.yml cp db:/tmp/s.dump backups\siftlode.dump
```

## Upgrading

**Take a backup first.** Every start runs any new database migrations automatically, and a migration
is not reversible by pulling the old image back — see rollback below.

```bash
docker compose -f docker-compose.selfhost.yml pull
docker compose -f docker-compose.selfhost.yml up -d
```

That tracks `latest`, which moves whenever a release is cut. **To decide for yourself when you move,
pin a version** by putting it in `.env`:

```bash
IMAGE_TAG=0.80.0
```

Then `pull` + `up -d` stays on that version until you change the line. Released versions and what
changed in them are listed in the app itself, under **Release Notes** in the account menu, and in
[`CHANGELOG.md`](../CHANGELOG.md).

### Rollback

If a release misbehaves, set `IMAGE_TAG` back to the previous version and `up -d`. That is enough
**as long as the release did not change the database schema**. If it did, the older image will refuse
to start against the newer schema, and you need to step the schema back first:

```bash
# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. step the schema back one revision
docker compose -f docker-compose.selfhost.yml run --rm api alembic downgrade -1
# 3. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d
```

If you are not sure whether the schema moved, restoring your pre-upgrade dump is the safer path — it
puts the database back exactly where the old image expects it.

### A migration failed on start

The container will keep restarting (`restart: unless-stopped`), and `worker` will sit waiting for an
`api` that never becomes healthy. Read the reason first:

```bash
docker compose -f docker-compose.selfhost.yml logs api | tail -50
```

Then either fix the cause and let it retry, or roll back: restore your pre-upgrade dump, pin the
previous `IMAGE_TAG`, and `up -d`. Do not delete the `siftlode_pgdata` volume to "start clean" unless
you have a dump you have actually tested restoring — that volume *is* your instance.