feat: one structured log for every process, with a request id
A deployment printed four different log formats and none of them could be joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in main.py for anything not started by uvicorn, logging.basicConfig() in the worker and in the download child (no timestamp at all), and alembic's fileConfig. app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log data model, no SDK): ts, level, severity, source, logger, event code, message, attributes, plus the correlation the context carries — request id, account, download job. Redaction is applied at the HANDLER, not at the call sites: every message, every attribute and every traceback is scrubbed on the way out, so a secret cannot leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies, secret config values and email addresses never reach a line. The access log is ours now (`http.request`): the route template instead of the concrete path, the status, the duration and the account — and never the query string, which uvicorn's access log printed on every single request (`?account=`, `?source=`, the `?token=` of a share link). A well-formed client request id is accepted and echoed in X-Request-Id, so a browser line and a server line join up. The account comes from `request.state`, filled by the auth dependency: the middleware is the outermost one, so it runs outside SessionMiddleware and in a different task context. `log_level` becomes an admin knob (sysconfig), applied without a restart and re-applied at startup — best-effort on purpose, since a logging knob must never be what stops the app from booting. Every compose service gets bounded json-file logs (3×50 MB): Docker keeps every line forever by default, and the API now writes one line per request. Verified on the localdev stack: JSON everywhere including alembic and the worker, the query strings gone, a supplied X-Request-Id echoed and logged, and three /healthz hits of which only the one made while the level was DEBUG produced a line.
This commit is contained in:
@@ -10,6 +10,15 @@
|
||||
# Everything else — the admin account, Google sign-in, SMTP — is set in the web wizard on
|
||||
# first run. See docs/self-hosting.md.
|
||||
|
||||
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
|
||||
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
|
||||
# busy instance and a hard ceiling on the disk it can take.
|
||||
x-logging: &logging
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "50m"
|
||||
max-file: "3"
|
||||
|
||||
services:
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
@@ -27,6 +36,7 @@ services:
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
api:
|
||||
@@ -71,6 +81,7 @@ services:
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
|
||||
@@ -109,6 +120,7 @@ services:
|
||||
- ALL
|
||||
tmpfs:
|
||||
- /tmp
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
|
||||
@@ -119,6 +131,7 @@ services:
|
||||
networks: [internal]
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
|
||||
Reference in New Issue
Block a user