peter/siftlode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
fe38dfbccae22e42cd1f696af83936dabdb4e09e
siftlode/docs/self-hosting.md
T
peter fe38dfbcca fix(e12): close the eighteen findings the E12 review round left standing 
The deep review of the branch raised 47 findings, 18 of which survived adversarial
verification. Grouped by what they actually were:

**The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user
already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one
user over quota was the only one downloading with no ceiling at all. `check_enqueue`
could not catch it: it ran while the job was queued and the user was still under the cap.
The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision
lives, and where it can be tested), and the worker now fails the job outright rather than
proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help
until space is freed.

**Silent failures in the new account panel.** All three actions went through `quiet: true`
endpoints, which suppress the global error dialog on the promise that the caller reports
inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On
the broken-SMTP instance this panel exists for, an admin would read silence as "the mail
went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`,
shared with the pending-access queue, so the two copies of these mutations cannot drift
apart again — which is how half of this was lost in the first place.

**A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps
back one migration, but a release routinely ships several (0.80.0 shipped four): it drops
the newest one's columns and leaves the schema at a revision the older image also cannot
resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is
now the recommended path, and the alembic route asks the target version for its own head
revision instead of counting backwards.

Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they
share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as
a 500; the pull cooldown extended to the sibling route spending the same QuotaAction;
restore stops the app before dropping its tables; the PowerShell cleanup can no longer
decide the script's verdict on 5.1, nor report the wrong exit code; the documented
one-liners hardened to match the scripts; the registry's retention window stated where
pinning is advertised; the two banners made mutually exclusive so neither can shadow the
other; `hover:bg-hover` (a utility this project never generated) replaced by the
`selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to
the two callers that discarded it.

Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset
banner UAT'd on localdev.
2026-09-05 03:41:58 +02:00

293 lines
15 KiB
Markdown
Raw Blame History

# Self-hosting Siftlode
Run your own private Siftlode instance with Docker. You don't need the source code — the app runs
from a prebuilt image, and everything user-facing (your admin account, Google sign-in, email) is
configured in a web wizard on first start. There's no editing of config files by hand.
## What you need
- A machine with **Docker** and the **Docker Compose plugin** (Docker Desktop on Windows/macOS,
or Docker Engine on Linux).
- A few hundred MB of disk and ~1 GB RAM free for the app itself. The optional Download Center
stores media too — budget disk for whatever you download (it's bounded by per-user quotas you
set as admin).
- Optional: a domain name + reverse proxy if you want HTTPS / public access (see below).
## 1. Get the files
Download these into a new, empty folder:
- `docker-compose.selfhost.yml`
- `install.sh` (Linux/macOS) **or** `install.ps1` (Windows)
The app image is published at `git.b1fr0st.eu/peter/siftlode` (public — no login to pull).
## 2. Run the installer
**Linux / macOS:**
```bash
chmod +x install.sh
./install.sh
```
**Windows (PowerShell):**
```powershell
./install.ps1
```
The installer asks for the **public URL** where the instance will be reached (just press Enter for
`http://localhost:8080` to try it locally). It then:
- generates the secrets it needs (`SECRET_KEY`, `TOKEN_ENCRYPTION_KEY`, a database password) into a
local `.env` file — keep that file private. **Do not change `TOKEN_ENCRYPTION_KEY` once the app has
stored any secret** (Google credentials, API keys, SMTP password, Plex token): those values are
encrypted with it, and a changed key makes them unreadable. If it is ever lost or rotated, clear the
affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page,
- pulls the image and starts the app + database,
- prints the **setup wizard URL**, which looks like `…/setup?token=…`.
## 3. Finish in the web wizard
Open the printed setup URL in your browser. The one-time token in it means only you (with access to
the server logs) can run setup. Then click through:
1. **Admin account** — your email + a password. This is how you'll sign in.
2. **Google sign-in** *(optional)* — paste a Google OAuth client ID + secret to enable "Sign in with
Google" and pulling your YouTube subscriptions. Skip it to use email + password only.
3. **Email / SMTP** *(optional)* — an SMTP server so the app can send verification and notification
emails. Skip it — without email, new registrations are simply approved by you (the admin) instead.
4. **Finish** — the wizard disappears, the instance is now configured, and you land on the sign-in
page. Log in with the admin account you just created.
That's it. You can change any of the optional settings later under the admin **Configuration** page.
> The setup wizard only exists until you finish it. After that, the setup routes are disabled and
> the token is invalidated — there's no setup surface left on a configured instance.
## Getting a Google OAuth client (optional)
Only needed for "Sign in with Google" / YouTube access. In the
[Google Cloud Console](https://console.cloud.google.com/): create a project → **APIs & Services →
Credentials → Create credentials → OAuth client ID** → *Web application*. Add your instance's
`…/auth/callback` URL as an **Authorized redirect URI**, then copy the **client ID** and **secret**
into the wizard's Google step. (Enable the **YouTube Data API v3** for the project too.)
## HTTPS / public access
The app is served on port `8080` over plain HTTP, which is fine for a LAN or a quick trial. For
public access, put a reverse proxy (Caddy, Nginx, Traefik…) in front to terminate TLS, and set the
**public URL** in the installer to your `https://…` address (this also marks the session cookie
secure). If you've already run the installer, edit `OAUTH_REDIRECT_URL` in `.env` to the https
callback URL and `docker compose -f docker-compose.selfhost.yml up -d`.
**Behind a proxy, set `TRUSTED_PROXY_IPS`.** The login / registration / password-reset rate limiters
identify callers by IP. Behind a proxy every request arrives from the proxy, so without this the
limits would apply to everyone together. Set `TRUSTED_PROXY_IPS` in `.env` to the address your proxy
connects to the app *from* (as the app sees it — e.g. its Docker/host/LAN IP, **not** the public
client address), and the app will read the real client IP from the proxy's `X-Forwarded-For` header:
```bash
TRUSTED_PROXY_IPS=172.18.0.1 # comma-separate multiple proxies
```
Only that peer is trusted, so a request sent straight to port `8080` (bypassing the proxy) can't forge
`X-Forwarded-For` to dodge the limits — it's rate-limited by its real address. Leave it empty if the
app is exposed directly with no proxy. Make sure your proxy actually sets `X-Forwarded-For` (Caddy's
`reverse_proxy` and Nginx's `proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for` both do).
## Download Center (media storage)
The stack includes a **Download Center**: an admin-enabled feature that saves videos to the server
with yt-dlp (Plex-friendly folders + `.nfo`/poster), lets users trim/crop/join clips, and shares
them. It runs two extra containers that come up automatically — a `worker` (the download/edit job
loop) and a small `bgutil-pot` sidecar (mints YouTube tokens so downloads aren't bot-blocked). No
configuration is required; per-user storage quotas are set on the admin **Downloads → System** page.
By default the media lives in a Docker-managed volume (`siftlode_downloads`). To keep it somewhere
you can reach from other apps — e.g. a folder your **Plex** server indexes — point it at a host
directory by adding this to `.env` and re-running `up -d`:
```bash
DOWNLOAD_HOST_PATH=/mnt/media/youtube
```
The directory must be writable by the container user (uid `1000`): `sudo chown -R 1000:1000 <dir>`.
## Plex library (optional)
If you run a **Plex** server, Siftlode can browse and play your movies and shows beside your YouTube
feed. Two things have to line up:
1. **Server URL + token**, entered as admin under **Configuration → Plex**. That is how Siftlode reads
your libraries.
2. **The media files themselves**, mounted into the container read-only. Siftlode streams the files
directly rather than proxying Plex, so it has to be able to open them.
Point `PLEX_MEDIA_HOST_PATH` at the directory holding your library and re-run `up -d`:
```bash
PLEX_MEDIA_HOST_PATH=/mnt/media
```
It appears inside the container at `/plex-media`, read-only. Siftlode then rewrites the paths Plex
reports into that mount using the **path map** on the same admin page — its default, `/data=/plex-media`,
means "wherever Plex says `/data/movies/…`, look in `/plex-media/movies/…`". Set the left-hand side to
whatever prefix your Plex server uses, and mount the matching host directory so the rest of the path
still lines up.
Leave `PLEX_MEDIA_HOST_PATH` unset if you don't use Plex — the mount falls back to an empty volume and
the module stays off.
## Day-to-day
```bash
# Logs / status (api = web, worker = downloads/edits)
docker compose -f docker-compose.selfhost.yml logs -f api
docker compose -f docker-compose.selfhost.yml logs -f worker
docker compose -f docker-compose.selfhost.yml ps
# Stop
docker compose -f docker-compose.selfhost.yml down
```
## Backups
**A backup is three things, not one.** Copying only the database gets you an instance that starts and
then cannot read its own secrets.
1. **The database** — accounts, subscriptions, playlists, the video catalog, and every setting you
made in the admin pages. It lives in the `siftlode_pgdata` Docker volume; dump it with the command
below.
2. **`.env`** — in particular `TOKEN_ENCRYPTION_KEY`. Your Google client secret, SMTP password, Plex
token and every stored refresh token are encrypted with it *inside* the database. Restore a dump
against a different key and those values are lost — you would have to clear the affected rows from
`system_config` and re-enter every secret by hand. Keep `.env` beside the dump, and just as private.
3. **Downloaded media** *(only if you use the Download Center)* — the `siftlode_downloads` volume, or
whatever `DOWNLOAD_HOST_PATH` points at. Not needed to bring the instance back; the jobs and their
metadata are in the database, the files themselves are not.
Dump and restore the database with these two commands. They also exist as scripts
(`scripts/backup.sh` + `scripts/restore.sh`, or `backup.ps1` / `restore.ps1` on Windows) if you have
the source checkout:
If you customised `POSTGRES_USER` / `POSTGRES_DB` in `.env`, substitute them for `siftlode` below —
the scripts read them from the environment, these one-liners cannot.
```bash
# Back up. Write to a .part name and rename only on success: redirecting straight into the final
# name creates the file BEFORE pg_dump runs, so a failed dump leaves a plausible-looking 0-byte
# file in backups/ that you will not discover until the day you need it.
mkdir -p backups && out="backups/siftlode-$(date +%Y%m%d-%H%M%S).dump" && \
docker compose -f docker-compose.selfhost.yml exec -T db \
pg_dump -U siftlode -Fc siftlode > "$out.part" \
&& [ -s "$out.part" ] && mv "$out.part" "$out" && echo "wrote $out" \
|| { rm -f "$out.part"; echo "BACKUP FAILED"; }
# Restore. Stop the app first: pg_restore --clean DROPs each object in its own transaction, and a
# running api holds locks that make those DROPs queue — abort the wait and you are left with a
# half-dropped database. Restoring with the app down also means nothing writes into a schema that
# is only partly back.
docker compose -f docker-compose.selfhost.yml stop api worker
docker compose -f docker-compose.selfhost.yml exec -T db \
pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
docker compose -f docker-compose.selfhost.yml up -d
```
On Windows, do **not** pipe the dump through PowerShell — it corrupts binary data. Write it inside the
container and copy it out instead. Check the exit code between the two steps and use a timestamped
name: the dump inside the container keeps its name, so a `cp` that runs after a *failed* dump would
copy out the previous run's file and quietly overwrite your only backup with a stale one.
```powershell
New-Item -ItemType Directory -Force -Path backups | Out-Null
$out = "backups\siftlode-$(Get-Date -Format yyyyMMdd-HHmmss).dump"
docker compose -f docker-compose.selfhost.yml exec -T db pg_dump -U siftlode -Fc siftlode -f /tmp/s.dump
if ($LASTEXITCODE -ne 0) { throw "BACKUP FAILED - pg_dump returned $LASTEXITCODE" }
docker compose -f docker-compose.selfhost.yml cp db:/tmp/s.dump $out
docker compose -f docker-compose.selfhost.yml exec -T db rm -f /tmp/s.dump
```
## Upgrading
**Take a backup first.** Every start runs any new database migrations automatically, and a migration
is not reversible by pulling the old image back — see rollback below.
```bash
docker compose -f docker-compose.selfhost.yml pull
docker compose -f docker-compose.selfhost.yml up -d
```
That tracks `latest`, which moves whenever a release is cut. **To decide for yourself when you move,
pin a version** by putting it in `.env`:
```bash
IMAGE_TAG=0.80.0
```
Then `pull` + `up -d` stays on that version until you change the line. Released versions and what
changed in them are listed in the app itself, under **Release Notes** in the account menu, and in
[`CHANGELOG.md`](../CHANGELOG.md).
**The registry keeps only the newest few releases, not every version the changelog lists.** A pin is
"decide when you take the next hop", not "stay here forever": once you are several releases behind,
your pinned tag can no longer be pulled — which matters the day you rebuild the host, prune your
Docker images, or move to a new machine. A container that is already running is unaffected (Docker
keeps the image locally), and you can always go back to `latest` or build the image yourself from the
repository. If you want to sit on one version long term, keep your own copy:
`docker save git.b1fr0st.eu/peter/siftlode:0.80.0 -o siftlode-0.80.0.tar`.
### Rollback
If a release misbehaves, set `IMAGE_TAG` back to the previous version and `up -d`. That is enough
**as long as the release did not change the database schema**. If it did, the older image will refuse
to start against the newer schema.
**Restoring your pre-upgrade dump is the recommended way out of that** — it puts the database back
exactly where the old image expects it, in one step, with no revision arithmetic:
```bash
# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. restore the dump you took before upgrading (see Backups above)
docker compose -f docker-compose.selfhost.yml exec -T db \
pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
# 3. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d
```
If you have no usable dump, you can step the schema back instead — but **not with `alembic downgrade
-1`**. One release routinely ships several migrations (0.80.0 shipped four), so `-1` undoes only the
newest one and leaves the schema at a revision the older image *also* cannot resolve: you lose that
migration's data and still get a crash loop. Downgrade to the revision the target version expects,
which that version can tell you itself:
```bash
# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. ask the version you are rolling back TO which schema revision it expects
IMAGE_TAG=0.79.0 docker compose -f docker-compose.selfhost.yml run --rm api alembic heads
# 3. step the schema back to exactly that revision — run this with the CURRENT (newer) image,
# because it is the one that ships the migrations being undone
docker compose -f docker-compose.selfhost.yml run --rm api alembic downgrade <revision from step 2>
# 4. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d
```
A downgrade drops the columns the newer version added, so anything stored only in those columns is
gone. That is the price of not having the dump.
### A migration failed on start
The container will keep restarting (`restart: unless-stopped`), and `worker` will sit waiting for an
`api` that never becomes healthy. Read the reason first:
```bash
docker compose -f docker-compose.selfhost.yml logs api | tail -50
```
Then either fix the cause and let it retry, or roll back: restore your pre-upgrade dump, pin the
previous `IMAGE_TAG`, and `up -d`. Do not delete the `siftlode_pgdata` volume to "start clean" unless
you have a dump you have actually tested restoring — that volume *is* your instance.
Reference in New Issue View Git Blame Copy Permalink