feat: one structured log for every process, with a request id
A deployment printed four different log formats and none of them could be joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in main.py for anything not started by uvicorn, logging.basicConfig() in the worker and in the download child (no timestamp at all), and alembic's fileConfig. app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log data model, no SDK): ts, level, severity, source, logger, event code, message, attributes, plus the correlation the context carries — request id, account, download job. Redaction is applied at the HANDLER, not at the call sites: every message, every attribute and every traceback is scrubbed on the way out, so a secret cannot leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies, secret config values and email addresses never reach a line. The access log is ours now (`http.request`): the route template instead of the concrete path, the status, the duration and the account — and never the query string, which uvicorn's access log printed on every single request (`?account=`, `?source=`, the `?token=` of a share link). A well-formed client request id is accepted and echoed in X-Request-Id, so a browser line and a server line join up. The account comes from `request.state`, filled by the auth dependency: the middleware is the outermost one, so it runs outside SessionMiddleware and in a different task context. `log_level` becomes an admin knob (sysconfig), applied without a restart and re-applied at startup — best-effort on purpose, since a logging knob must never be what stops the app from booting. Every compose service gets bounded json-file logs (3×50 MB): Docker keeps every line forever by default, and the API now writes one line per request. Verified on the localdev stack: JSON everywhere including alembic and the worker, the query strings gone, a supplied X-Request-Id echoed and logged, and three /healthz hits of which only the one made while the level was DEBUG produced a line.
This commit is contained in:
@@ -11,6 +11,15 @@
|
||||
# applies local migrations to THIS local DB only. Seed the catalog once with a pg_dump from
|
||||
# another instance if you want real data (see RUNBOOK).
|
||||
|
||||
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
|
||||
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
|
||||
# busy instance and a hard ceiling on the disk it can take.
|
||||
x-logging: &logging
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "50m"
|
||||
max-file: "3"
|
||||
|
||||
services:
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
@@ -27,6 +36,7 @@ services:
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
api:
|
||||
@@ -63,6 +73,7 @@ services:
|
||||
- ./downloads:/downloads
|
||||
# Plex media, read-only (see the plex_media volume note below). Maps to plex_path_map.
|
||||
- plex_media:/plex-media:ro
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# Dedicated download worker: same image, runs the yt-dlp job loop instead of the API.
|
||||
@@ -96,6 +107,7 @@ services:
|
||||
volumes:
|
||||
- ./downloads:/downloads
|
||||
- plex_media:/plex-media:ro
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# PO-token provider: mints YouTube Proof-of-Origin tokens on demand so the worker bypasses
|
||||
@@ -106,6 +118,7 @@ services:
|
||||
init: true
|
||||
security_opt:
|
||||
- apparmor:unconfined
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
|
||||
Reference in New Issue
Block a user