feat: one structured log for every process, with a request id

A deployment printed four different log formats and none of them could be
joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in
main.py for anything not started by uvicorn, logging.basicConfig() in the worker
and in the download child (no timestamp at all), and alembic's fileConfig.
app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log
data model, no SDK): ts, level, severity, source, logger, event code, message,
attributes, plus the correlation the context carries — request id, account,
download job.

Redaction is applied at the HANDLER, not at the call sites: every message, every
attribute and every traceback is scrubbed on the way out, so a secret cannot
leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies,
secret config values and email addresses never reach a line.

The access log is ours now (`http.request`): the route template instead of the
concrete path, the status, the duration and the account — and never the query
string, which uvicorn's access log printed on every single request (`?account=`,
`?source=`, the `?token=` of a share link). A well-formed client request id is
accepted and echoed in X-Request-Id, so a browser line and a server line join up.
The account comes from `request.state`, filled by the auth dependency: the
middleware is the outermost one, so it runs outside SessionMiddleware and in a
different task context.

`log_level` becomes an admin knob (sysconfig), applied without a restart and
re-applied at startup — best-effort on purpose, since a logging knob must never
be what stops the app from booting. Every compose service gets bounded json-file
logs (3×50 MB): Docker keeps every line forever by default, and the API now
writes one line per request.

Verified on the localdev stack: JSON everywhere including alembic and the
worker, the query strings gone, a supplied X-Request-Id echoed and logged, and
three /healthz hits of which only the one made while the level was DEBUG
produced a line.
This commit is contained in:
2026-09-16 01:37:02 +02:00
parent a5bcd9fca2
commit 6c63df6704
18 changed files with 828 additions and 61 deletions
+8
View File
@@ -28,6 +28,14 @@ ALLOWED_EMAILS=
# Admin emails (subset of the above) get the admin role.
ADMIN_EMAILS=
# Logging (E22). LOG_LEVEL is the BOOTSTRAP level — the level a process starts at and the only one
# the worker/download child read; the API's running level is the admin Configuration page's
# `log_level` knob, which takes effect without a restart. TRACE | DEBUG | INFO | WARNING | ERROR
# (an unknown value falls back to INFO). LOG_FORMAT is "json" (one object per line — what a log
# collector ingests) or "text" (a readable line for a local terminal).
LOG_LEVEL=INFO
LOG_FORMAT=json
# Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
FRONTEND_ORIGIN=
+4 -4
View File
@@ -1,17 +1,17 @@
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from app.config import settings
from app.db import Base
from app.logs import configure_logging
import app.models # noqa: F401 (import so models register on Base.metadata)
config = context.config
config.set_main_option("sqlalchemy.url", settings.database_url)
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# Migrations run at container start (entrypoint.sh), so their output is part of the API's log
# stream — one format with everything else, not alembic.ini's own fileConfig shape.
configure_logging("alembic")
target_metadata = Base.metadata
+7
View File
@@ -7,6 +7,7 @@ from sqlalchemy.orm import Session
from starlette.requests import HTTPConnection
from app import email as email_mod
from app import logs
from app import mailtext
from app import sysconfig
from app.config import settings
@@ -348,6 +349,12 @@ def _session_user(request: Request, db: Session, user_id: int | None, is_default
if default_id and default_id not in accounts:
accounts.append(default_id)
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
# Attribute this request's log lines to the account (E22). Here, not in the request middleware:
# this is the one place a session becomes an account, and the middleware runs outside both the
# session middleware and this task's context. `request.state` is shared across middleware layers
# (it lives on the ASGI scope), so the access-log line can read it back.
logs.set_user(user.id)
request.state.user_id = user.id
return user
+10
View File
@@ -225,6 +225,16 @@ class Settings(BaseSettings):
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
sponsorblock_default: bool = False
# --- Diagnostics / logging (E22) ---
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
# startup and on change without a restart. This is what a process logs at before it can reach
# the database — and the only knob the worker/runner children read, since they are short-lived.
# TRACE (below DEBUG) is the "Teljes" diagnostics level; see app/logs.py.
log_level: str = "INFO"
# "json" (one object per line — what Loki and the log_events sink ingest) or "text" (a readable
# line for a local terminal). Redaction applies to both.
log_format: str = "json"
# --- Plex integration (optional) ---
# The Plex module plays the LOCAL physical media file directly (our own ffmpeg/direct-serve);
# Plex is used only for metadata + optional watch-state sync. Requires the Plex server to be
+5 -1
View File
@@ -17,6 +17,7 @@ import shutil
import subprocess
import sys
from app import logs
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
@@ -52,12 +53,15 @@ def _deprioritize() -> None:
def main() -> None:
logging.basicConfig(level=logging.INFO)
logs.configure_logging("runner")
_deprioritize()
if len(sys.argv) != 2:
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
raise SystemExit(2)
job_id = int(sys.argv[1])
# This whole process exists to run ONE job — tag every line it prints with it, so the child's
# output joins up with the worker's and the API's view of the same download.
logs.set_job(job_id)
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
+434
View File
@@ -0,0 +1,434 @@
"""Structured logging for every Siftlode process — the ONE place logging is configured.
Before this module a single deployment printed FOUR different log formats: uvicorn's
`log_config.json` for the API, a hand-rolled StreamHandler in `main.py` for everything that did not
start under uvicorn (tests, scripts), `logging.basicConfig()` in the worker and in the download
child (no timestamp at all), and alembic's `fileConfig`. None of them carried a correlation id, and
uvicorn's access log printed the query string of every request (`?account=1`, `?source=feed`) — so
the logs were both unjoinable and leaky. `configure_logging()` replaces all four.
Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
ts · level · sev · source · logger · event · msg · attrs
plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
`job_id`) and the app version. That shape is what E22.6's admin console filters on and what a Gitea
issue attachment will carry, so it is deliberately flat and stable.
Call sites stay ordinary stdlib logging:
log = logging.getLogger("siftlode.downloads")
log.info("job finished", extra=logs.event("download.job.done", job_id=7, bytes=n))
`event()` is optional — a bare `log.info("…")` still comes out well-formed — but the dotted code is
what makes a line queryable later, so add one wherever the line matters.
REDACTION IS APPLIED AT THE HANDLER, not at the call site: every message, every attr value and
every traceback passes `redact()` on the way out. That is the whole point — a secret cannot be
leaked by forgetting to redact it at one of ~125 call sites, and a new call site is safe by
default. Never logged, at any level: passwords, tokens (session, reset/verify, setup, share,
OAuth), cookie blobs, secret config values, and email addresses.
"""
import contextlib
import contextvars
import json
import logging
import logging.config
import re
import sys
import time
import uuid
from datetime import datetime, timezone
from typing import Any
from app.config import settings
# One level below DEBUG for the "Teljes / TRACE" diagnostics level (search strings, redacted request
# bodies). Registered on import so `logging.getLogger(...).log(logs.TRACE, …)` and a `LOG_LEVEL=TRACE`
# both work in every process, including one that never calls configure_logging (a test).
TRACE = 5
logging.addLevelName(TRACE, "TRACE")
# OpenTelemetry severity numbers for our level names. Kept beside the level because the number is
# what a log backend sorts and filters on; the name is for humans reading a terminal.
SEVERITY: dict[str, int] = {
"TRACE": 1,
"DEBUG": 5,
"INFO": 9,
"WARNING": 13,
"ERROR": 17,
"CRITICAL": 21,
}
# Which process this is. Set by configure_logging; every line carries it so one Loki/DB query can
# separate the API's view of a job from the worker's.
_source = "api"
# --- correlation context -------------------------------------------------------------------
# contextvars, like quota's actor attribution (app/quota.py): set at an entry point, read by the
# formatter, never threaded through call signatures. A child task copies the context at creation,
# so anything set before `await call_next(...)` is visible to the whole request.
_request_id: contextvars.ContextVar[str | None] = contextvars.ContextVar(
"log_request_id", default=None
)
_user_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_user_id", default=None)
_debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
"log_debug_session", default=None
)
_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
_CONTEXT_VARS = {
"request_id": _request_id,
"user_id": _user_id,
"debug_session_id": _debug_session,
"job_id": _job_id,
}
def set_user(user_id: int | None) -> None:
"""Attribute the rest of this request's log lines to `user_id`. Called from the auth dependency
— the one place the session is actually resolved into an account — rather than from the request
middleware, which runs OUTSIDE the session middleware and in a different task context."""
_user_id.set(user_id)
def set_job(job_id: int | None) -> None:
"""Attribute this process's log lines to a download job — for the download CHILD, whose whole
life is one job (the worker uses `bind()` per job instead, since it juggles several)."""
_job_id.set(job_id)
@contextlib.contextmanager
def bind(**fields: Any):
"""Bind correlation fields for the duration of a block (a worker job, a scheduler run):
with logs.bind(job_id=job.id):
...
Unknown field names are ignored rather than raising — a caller binding a field this module does
not know about is a typo, not a reason to take down a download."""
tokens = [(_CONTEXT_VARS[k].set(v)) for k, v in fields.items() if k in _CONTEXT_VARS]
keys = [k for k in fields if k in _CONTEXT_VARS]
try:
yield
finally:
for key, token in zip(keys, tokens):
_CONTEXT_VARS[key].reset(token)
def event(code: str, **attrs: Any) -> dict[str, Any]:
"""`extra=` payload for a log call: a stable dotted event code plus structured attributes.
Returns a plain dict so it composes with stdlib logging and costs nothing when the level is off
(the dict is built either way; keep the attrs cheap — no DB reads inside the call)."""
return {"event": code, "attrs": attrs}
# --- redaction -----------------------------------------------------------------------------
# Substring matching, not exact keys: the same secret travels as `password`, `current_password`,
# `new_password`, `smtp_password`, `plex_server_token`, `access_token`, … Matching on the PART is
# what makes a key we have never seen safe by default.
_SECRET_KEY_PARTS = (
"password",
"passphrase",
"token",
"secret",
"cookie",
"authorization",
"auth_header",
"ciphertext",
"private_key",
"api_key",
"apikey",
"credential",
"session_key",
)
# Query/fragment parameters that carry a credential in a URL we log (OAuth `code`/`state`, our own
# `?token=` setup and share links, `#verify=` / `#reset=` fragments).
_SECRET_PARAM_RE = re.compile(
r"\b((?:access_token|refresh_token|id_token|token|code|state|key|secret|password|pwd|sig|"
r"signature|verify|reset|setup)=)([^&\s\"']+)",
re.IGNORECASE,
)
# Deliberately loose: over-matching redacts a non-address that looks like one (harmless), while
# under-matching puts a real address in a log we attach to an issue (not harmless).
_EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
REDACTED = "[redacted]"
REDACTED_EMAIL = "[email]"
def redact_text(text: str) -> str:
"""Scrub a free-text string: credential-bearing URL params and email addresses."""
scrubbed = _SECRET_PARAM_RE.sub(lambda m: f"{m.group(1)}{REDACTED}", text)
return _EMAIL_RE.sub(REDACTED_EMAIL, scrubbed)
def redact(value: Any, *, key: str = "") -> Any:
"""Recursively scrub a structure destined for the log.
A value whose KEY names a secret is dropped whole (we never log a prefix of a token — a prefix
is still a leak and still useless). Everything else is scrubbed as text. Depth is bounded by
the structures we actually log (attrs dicts), and non-JSON types fall through to `str`."""
lowered = key.lower()
if key and any(part in lowered for part in _SECRET_KEY_PARTS):
return REDACTED
if isinstance(value, dict):
return {k: redact(v, key=str(k)) for k, v in value.items()}
if isinstance(value, (list, tuple)):
return [redact(v, key=key) for v in value]
if isinstance(value, str):
return redact_text(value)
return value
# --- formatters ----------------------------------------------------------------------------
# Attributes every LogRecord carries; anything else a caller passed via `extra=` is a structured
# attribute and is merged into `attrs` (so `extra={"job_id": 7}` works as well as `event(...)`).
_RECORD_STANDARD = frozenset(
vars(logging.LogRecord("", 0, "", 0, "", None, None)).keys()
| {"message", "asctime", "taskName", "event", "attrs"}
)
class ContextFilter(logging.Filter):
"""Copies the correlation contextvars onto the record. A Filter (not the formatter) so the
values are captured at emit time, before a QueueHandler can move the record to another
thread — the DB sink in S3 depends on that."""
def filter(self, record: logging.LogRecord) -> bool:
for name, var in _CONTEXT_VARS.items():
value = var.get()
if value is not None and not hasattr(record, name):
setattr(record, name, value)
return True
def _record_attrs(record: logging.LogRecord) -> dict[str, Any]:
extra = {k: v for k, v in vars(record).items() if k not in _RECORD_STANDARD}
attrs = dict(getattr(record, "attrs", None) or {})
attrs.update({k: v for k, v in extra.items() if k not in _CONTEXT_VARS})
return attrs
def _safe_message(record: logging.LogRecord) -> str:
"""`record.getMessage()` raises if the args don't match the format string. A broken log call
must not take down the caller, so the failure is reported IN the line instead."""
try:
return record.getMessage()
except Exception as exc: # noqa: BLE001 — any formatting error, reported not raised
return f"{record.msg!r} (log formatting failed: {exc})"
class JsonFormatter(logging.Formatter):
"""One JSON object per line — the shape Loki/the `log_events` table both ingest."""
def format(self, record: logging.LogRecord) -> str:
payload: dict[str, Any] = {
"ts": datetime.fromtimestamp(record.created, timezone.utc).isoformat(
timespec="milliseconds"
),
"level": record.levelname,
"sev": SEVERITY.get(record.levelname, 9),
"source": _source,
"logger": record.name,
"msg": redact_text(_safe_message(record)),
}
code = getattr(record, "event", None)
if code:
payload["event"] = code
for name in _CONTEXT_VARS:
value = getattr(record, name, None)
if value is not None:
payload[name] = value
attrs = _record_attrs(record)
if attrs:
payload["attrs"] = redact(attrs)
if record.exc_info:
payload["exc"] = redact_text(self.formatException(record.exc_info))
payload["version"] = settings.app_version
return json.dumps(payload, ensure_ascii=False, default=str)
class TextFormatter(logging.Formatter):
"""Human-readable single line for a terminal (`LOG_FORMAT=text`): the same content, ordered for
reading. Redaction applies here too — a dev log is still a log someone pastes into an issue."""
def format(self, record: logging.LogRecord) -> str:
stamp = datetime.fromtimestamp(record.created, timezone.utc).strftime("%H:%M:%S.%f")[:-3]
head = f"{stamp} {record.levelname:<5} [{record.name}]"
rid = getattr(record, "request_id", None)
if rid:
head += f" {rid}"
code = getattr(record, "event", None)
line = f"{head} {code + ' — ' if code else ''}{redact_text(_safe_message(record))}"
attrs = _record_attrs(record)
if attrs:
line += f" {json.dumps(redact(attrs), ensure_ascii=False, default=str)}"
if record.exc_info:
line += "\n" + redact_text(self.formatException(record.exc_info))
return line
# --- configuration -------------------------------------------------------------------------
# Third-party loggers that are noisy at our own level. Pinned explicitly so the diagnostics knob can
# lower the ROOT level to DEBUG/TRACE without drowning the output in SQL statements and HTTP chatter
# — raising THESE is a deliberate, separate act (E22.6's console).
_THIRD_PARTY_LEVELS = {
"sqlalchemy": "WARNING",
"sqlalchemy.engine": "WARNING",
"apscheduler": "WARNING",
"urllib3": "WARNING",
"httpx": "WARNING",
"httpcore": "WARNING",
"asyncio": "WARNING",
"multipart": "WARNING",
"PIL": "WARNING",
"alembic": "INFO",
# uvicorn's own startup/shutdown/WebSocket lines are worth keeping; its ACCESS log is not —
# it is one line per request with the query string in it, replaced by `log_requests` below.
"uvicorn": "INFO",
"uvicorn.error": "INFO",
"uvicorn.access": "CRITICAL",
}
def resolve_level(raw: str | None) -> int:
"""Level name (case-insensitive) → number, falling back to INFO for an unknown value. A typo in
a config knob must not silence the app."""
if not raw:
return logging.INFO
value = logging.getLevelName(raw.strip().upper())
return value if isinstance(value, int) else logging.INFO
def configure_logging(source: str, *, level: str | None = None) -> None:
"""Configure logging for THIS process. Call once, first thing, in every entry point:
`main.py` (api), `worker.main`, `downloads/runner.main`, `alembic/env.py`.
`source` names the process in every line. `level` overrides `LOG_LEVEL` (the bootstrap default);
the runtime admin knob goes through `set_level()` instead, so changing it needs no restart.
Our loggers deliberately keep `propagate=True` and own no handler: everything reaches the single
root StreamHandler. That is what keeps `caplog` working in tests, and it means a new logger name
is configured the moment it is created."""
global _source
_source = source
formatter = "json" if (settings.log_format or "json").lower() != "text" else "text"
logging.config.dictConfig(
{
"version": 1,
# Never disable loggers created at import time (module-level `getLogger` calls run
# before this function does).
"disable_existing_loggers": False,
"formatters": {
"json": {"()": f"{__name__}.JsonFormatter"},
"text": {"()": f"{__name__}.TextFormatter"},
},
"filters": {"context": {"()": f"{__name__}.ContextFilter"}},
"handlers": {
"stdout": {
"class": "logging.StreamHandler",
"stream": sys.stdout,
"formatter": formatter,
"filters": ["context"],
}
},
"root": {"level": resolve_level(level or settings.log_level), "handlers": ["stdout"]},
"loggers": {name: {"level": lvl} for name, lvl in _THIRD_PARTY_LEVELS.items()},
}
)
def set_level(raw: str | None) -> None:
"""Change the running process's level (the `log_level` admin knob). Third-party loggers keep
their pinned levels, so turning on DEBUG shows OUR detail, not SQLAlchemy's."""
logging.getLogger().setLevel(resolve_level(raw))
def current_level() -> str:
return logging.getLevelName(logging.getLogger().getEffectiveLevel())
# --- request logging -----------------------------------------------------------------------
# A client may supply its own request id so a browser log line and a server line join up (E22 S4).
# Accepted only in this shape: it ends up in a log field, and an unbounded header would be a way to
# write arbitrary text into the log.
_REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9._-]{8,64}$")
# Paths whose success is logged at DEBUG rather than INFO: the container healthcheck runs every 15s
# (5.7k lines/day saying nothing) and hashed static assets are served by the hundred per page load.
# A FAILURE on either still logs at WARNING/ERROR — this lowers the floor, it does not hide.
_QUIET_PREFIXES = ("/healthz", "/assets/", "/static/")
_request_log = logging.getLogger("siftlode.http")
async def log_requests(request, call_next):
"""The single access-log line, replacing uvicorn's.
What changes versus `uvicorn.access`: the ROUTE TEMPLATE instead of the concrete path (so
`/api/videos/{video_id}` groups), never the query string (it carried `?account=`, and at TRACE
the redacted query belongs in attrs, not in the URL), a request id echoed back in
`X-Request-Id`, the duration, and the account — read from `request.state`, which the auth
dependency fills, because this middleware sits outside both the session middleware and the
downstream task's context."""
incoming = request.headers.get("x-request-id", "")
rid = incoming if _REQUEST_ID_RE.match(incoming) else uuid.uuid4().hex[:16]
token = _request_id.set(rid)
request.state.request_id = rid
started = time.perf_counter()
try:
response = await call_next(request)
except Exception:
# The unhandled-exception path: FastAPI's ServerErrorMiddleware re-raises after its own
# 500, so without this the ONE request that most needs a log line is the one with no
# access-log entry at all.
_request_log.exception(
"request failed",
extra=event(
"http.request",
method=request.method,
path=_route_template(request),
duration_ms=round((time.perf_counter() - started) * 1000, 1),
user_id=getattr(request.state, "user_id", None),
),
)
_request_id.reset(token)
raise
duration_ms = round((time.perf_counter() - started) * 1000, 1)
status = response.status_code
quiet = request.url.path.startswith(_QUIET_PREFIXES)
if status >= 500:
level = logging.ERROR
elif status >= 400:
level = logging.WARNING
else:
level = logging.DEBUG if quiet else logging.INFO
_request_log.log(
level,
"%s %s %s",
request.method,
_route_template(request),
status,
extra=event(
"http.request",
method=request.method,
path=_route_template(request),
status=status,
duration_ms=duration_ms,
user_id=getattr(request.state, "user_id", None),
),
)
response.headers["X-Request-Id"] = rid
_request_id.reset(token)
return response
def _route_template(request) -> str:
"""The matched route's template (`/api/videos/{video_id}`), else the raw path for a 404 or a
static file. Never the query string."""
route = request.scope.get("route")
return getattr(route, "path", None) or request.url.path
+26 -11
View File
@@ -1,7 +1,6 @@
import asyncio
import logging
import math
import sys
from contextlib import asynccontextmanager, suppress
from pathlib import Path
@@ -9,16 +8,13 @@ from fastapi import FastAPI, HTTPException, Request
from fastapi.encoders import jsonable_encoder
from fastapi.exceptions import RequestValidationError
# When started via uvicorn --log-config the "siftlode" logger is already configured
# (see log_config.json). This block is a timestamped fallback for other entrypoints
# (tests, scripts) so our logs are never silently dropped.
_siftlode_logger = logging.getLogger("siftlode")
if not _siftlode_logger.handlers:
_handler = logging.StreamHandler(sys.stdout)
_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)-5s [%(name)s] %(message)s"))
_siftlode_logger.addHandler(_handler)
_siftlode_logger.setLevel(logging.INFO)
_siftlode_logger.propagate = False
# Logging is configured HERE, before the route modules are imported, so every logger they create at
# import time is already covered. ONE format for the api, the worker, the download child and alembic
# alike (app/logs.py) — this replaces the old uvicorn --log-config + the hand-rolled fallback that
# used to live here, and uvicorn's access log is replaced by the `http.request` middleware below.
from app.logs import configure_logging, current_level, log_requests, set_level
configure_logging("api")
log = logging.getLogger("siftlode.app")
from fastapi.middleware.cors import CORSMiddleware
@@ -28,6 +24,7 @@ from sqlalchemy.exc import DataError
from starlette.middleware.sessions import SessionMiddleware
from app import state
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.plex import stream as plex_stream
@@ -66,6 +63,16 @@ async def lifespan(app: FastAPI):
# First-run: if the instance isn't configured yet, mint a fresh one-time setup token and print
# the wizard URL so the operator can open it (the app runs in setup mode until they finish).
with SessionLocal() as db:
# The admin's runtime level (sysconfig `log_level`) wins over the LOG_LEVEL bootstrap
# default, so a level chosen in the UI survives a restart without touching the environment.
# Best-effort on purpose: a LOGGING knob must never be the thing that stops the app from
# booting — an unreachable or not-yet-migrated database has to start into the env level.
try:
set_level(sysconfig.get_str(db, "log_level"))
except Exception as exc: # noqa: BLE001 — any DB/config failure keeps the bootstrap level
log.warning(
"could not read the stored log level (%s); staying at %s", exc, current_level()
)
if not state.is_configured(db):
token = state.rotate_setup_token(db)
url = f"{settings.app_base}/setup?token={token}"
@@ -243,6 +250,14 @@ async def security_headers(request, call_next):
return response
# The access log (app/logs.py): request id, route template, status, duration — one line, never a
# query string. Registered LAST so it is the OUTERMOST http middleware and therefore times the whole
# request, including what the three above do. Being outermost also puts it OUTSIDE SessionMiddleware,
# which is why the account comes from `request.state.user_id` (set by the auth dependency, the one
# place a session is actually resolved into an account) rather than from `request.session`.
app.middleware("http")(log_requests)
app.include_router(health.router)
app.include_router(auth_routes.router)
app.include_router(sync.router)
+11
View File
@@ -10,6 +10,7 @@ from sqlalchemy.orm import Session
from app import audit
from app import email as email_mod
from app import logs
from app import mailtext
from app import sysconfig
from app.audit import AuditAction
@@ -29,6 +30,14 @@ def _redact(v):
return _URL_USERINFO.sub(r"\1***@", v) if isinstance(v, str) else v
def _apply_runtime(db: Session, key: str) -> None:
"""Some knobs change how THIS process behaves, not just what a later read returns — they have to
take effect now, without a restart. One dispatch for both call sites (set and reset), so the next
such key can't be wired into one of them and forgotten in the other."""
if key == "log_level":
logs.set_level(sysconfig.get_str(db, "log_level"))
@router.get("")
def get_config(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
return sysconfig.describe(db)
@@ -80,6 +89,7 @@ def set_config(
after=None if s.secret else {"value": _redact(new)},
)
db.commit()
_apply_runtime(db, key)
return sysconfig.describe(db)
@@ -104,6 +114,7 @@ def reset_config(
before=None if s.secret else {"value": _redact(old)},
)
db.commit()
_apply_runtime(db, key)
return sysconfig.describe(db)
+5
View File
@@ -150,6 +150,11 @@ SPECS: tuple[ConfigSpec, ...] = (
# --- Audit log ---
# Days to keep admin audit-log rows before the audit_gc job prunes them (0 = keep forever).
ConfigSpec("audit_retention_days", "int", "audit", "audit_retention_days", min=0, max=3650),
# --- Diagnostics (E22) ---
# Runtime log level for the API process: TRACE | DEBUG | INFO | WARNING | ERROR. Free-text like
# search_source (an unknown value falls back to INFO in logs.resolve_level, so a typo can't
# silence the app). Applied without a restart — see routes/config.py and main.py's lifespan.
ConfigSpec("log_level", "str", "diagnostics", "log_level"),
# --- Plex integration (optional; local-file playback + Plex metadata) ---
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled"),
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url"),
+6 -2
View File
@@ -28,6 +28,7 @@ from datetime import datetime, timezone
from sqlalchemy import text
from app import logs
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
@@ -492,14 +493,17 @@ def _worker_loop(idx: int) -> None:
_stop.wait(settings.download_poll_seconds)
continue
try:
_process_job(job_id)
# Tag every line this job produces (including the crash below) with the job id: a worker
# runs N jobs at once on N threads, so without it the interleaved output is unreadable.
with logs.bind(job_id=job_id):
_process_job(job_id)
except Exception: # noqa: BLE001
log.exception("processing job %s crashed", job_id)
_fail_running(job_id, error="Worker error", error_code=None)
def main() -> None:
logging.basicConfig(level=logging.INFO)
logs.configure_logging("worker")
signal.signal(signal.SIGTERM, _handle_signal)
signal.signal(signal.SIGINT, _handle_signal)
+5 -1
View File
@@ -15,4 +15,8 @@ echo "Starting Siftlode API..."
# X-Forwarded-For only when the TRUE socket peer is a configured proxy (TRUSTED_PROXY_IPS); those
# flags would make uvicorn overwrite request.client from the forgeable XFF, defeating that check and
# reopening the rate-limit bypass (SA3).
exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --log-config log_config.json --timeout-keep-alive 75
#
# No --log-config: app/logs.py configures logging for every Siftlode process (api, worker, download
# child, alembic) in one place, and it silences uvicorn's access log in favour of our own
# `http.request` line (route template, no query string, request id).
exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --timeout-keep-alive 75
-40
View File
@@ -1,40 +0,0 @@
{
"version": 1,
"disable_existing_loggers": false,
"formatters": {
"default": {
"()": "uvicorn.logging.DefaultFormatter",
"fmt": "%(asctime)s %(levelprefix)s [%(name)s] %(message)s"
},
"access": {
"()": "uvicorn.logging.AccessFormatter",
"fmt": "%(asctime)s %(levelprefix)s %(client_addr)s - \"%(request_line)s\" %(status_code)s"
},
"plain": {
"format": "%(asctime)s %(levelname)-5s [%(name)s] %(message)s"
}
},
"handlers": {
"default": {
"class": "logging.StreamHandler",
"formatter": "default",
"stream": "ext://sys.stdout"
},
"access": {
"class": "logging.StreamHandler",
"formatter": "access",
"stream": "ext://sys.stdout"
},
"plain": {
"class": "logging.StreamHandler",
"formatter": "plain",
"stream": "ext://sys.stdout"
}
},
"loggers": {
"uvicorn": { "handlers": ["default"], "level": "INFO", "propagate": false },
"uvicorn.error": { "handlers": ["default"], "level": "INFO", "propagate": false },
"uvicorn.access": { "handlers": ["access"], "level": "INFO", "propagate": false },
"siftlode": { "handlers": ["plain"], "level": "INFO", "propagate": false }
}
}
+256
View File
@@ -0,0 +1,256 @@
"""E22 S2 — the one logging configuration, its redaction, and the request line that replaced
uvicorn's access log.
Redaction is the part worth guarding hardest: it is applied at the HANDLER, so these cases are the
only thing standing between a future call site and a token in a log we attach to a Gitea issue.
The request-line cases pin what uvicorn's access log got wrong — the query string (`?account=1`,
`?token=…`) was printed on every request, and nothing correlated two lines of the same request.
Pure logic + an in-process ASGI app: no database.
"""
import json
import logging
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from app import logs
class _Sink(logging.Handler):
"""Captures records AND their formatted JSON, with the same filter/formatter the real stdout
handler uses — so a test sees exactly the bytes a deployment would write."""
def __init__(self):
super().__init__(level=logging.NOTSET)
self.lines: list[str] = []
self.setFormatter(logs.JsonFormatter())
self.addFilter(logs.ContextFilter())
def emit(self, record):
self.lines.append(self.format(record))
def objects(self) -> list[dict]:
return [json.loads(line) for line in self.lines]
@pytest.fixture
def sink():
"""Attached to `siftlode` (not root) so a `configure_logging` call inside a test — which
replaces the ROOT handlers — can't silently detach it mid-test."""
logger = logging.getLogger("siftlode")
handler = _Sink()
previous = logger.level
logger.addHandler(handler)
logger.setLevel(logs.TRACE)
try:
yield handler
finally:
logger.removeHandler(handler)
logger.setLevel(previous)
# --- redaction ------------------------------------------------------------------------------
@pytest.mark.parametrize(
"text, must_not_contain",
[
("user signed in: someone@example.com", "someone@example.com"),
("callback https://app/auth/callback?code=4/0AY&state=xyz", "4/0AY"),
("mail link https://app/#verify=abc123def", "abc123def"),
("share https://app/s?token=SHARETOKEN", "SHARETOKEN"),
("reset https://app/#reset=RESETTOKEN", "RESETTOKEN"),
],
)
def test_free_text_secrets_never_reach_the_line(text, must_not_contain):
assert must_not_contain not in logs.redact_text(text)
def test_a_key_that_names_a_secret_is_dropped_whole():
# Not truncated to a prefix: a prefix is still a leak and is useless for triage anyway.
out = logs.redact(
{
"password": "hunter2",
"current_password": "hunter2",
"plex_server_token": "abc",
"Cookie": "session=xyz",
"authorization": "Bearer abc",
"nested": {"refresh_token": "rt", "keep": "visible"},
"list": [{"api_key": "k"}],
"count": 7,
}
)
assert out["password"] == logs.REDACTED
assert out["current_password"] == logs.REDACTED
assert out["plex_server_token"] == logs.REDACTED
assert out["Cookie"] == logs.REDACTED
assert out["authorization"] == logs.REDACTED
assert out["nested"]["refresh_token"] == logs.REDACTED
assert out["nested"]["keep"] == "visible" # only the named keys go
assert out["list"][0]["api_key"] == logs.REDACTED
assert out["count"] == 7
def test_an_email_in_an_attribute_value_is_scrubbed_too():
assert logs.redact({"to": "a.b+tag@mail.example.org"})["to"] == logs.REDACTED_EMAIL
def test_redaction_runs_on_the_message_the_attrs_and_the_traceback(sink):
log = logging.getLogger("siftlode.test")
try:
raise RuntimeError("boom for someone@example.com")
except RuntimeError:
log.exception(
"failed for %s",
"someone@example.com",
extra=logs.event("test.failed", password="hunter2", url="https://x/?token=T0K3N"),
)
line = sink.lines[-1]
assert "someone@example.com" not in line
assert "hunter2" not in line
assert "T0K3N" not in line
payload = json.loads(line)
assert payload["attrs"]["password"] == logs.REDACTED
assert "RuntimeError" in payload["exc"] # the diagnosis survives the scrubbing
# --- event shape ----------------------------------------------------------------------------
def test_every_line_carries_the_agreed_fields(sink):
logging.getLogger("siftlode.test").info("hello", extra=logs.event("test.hello", n=1))
payload = sink.objects()[-1]
for field in ("ts", "level", "sev", "source", "logger", "msg", "event", "version"):
assert field in payload, field
assert payload["level"] == "INFO"
assert payload["sev"] == 9
assert payload["event"] == "test.hello"
assert payload["attrs"] == {"n": 1}
def test_a_plain_log_call_still_produces_a_well_formed_line(sink):
"""An event code is optional — 125 existing call sites have none yet (E22.5 adds them), and they
must not be worse off than before this module."""
logging.getLogger("siftlode.test").warning("just a message")
payload = sink.objects()[-1]
assert payload["msg"] == "just a message"
assert payload["level"] == "WARNING"
assert "event" not in payload
def test_a_broken_format_call_reports_itself_instead_of_raising():
"""A record is formatted directly rather than logged: a genuinely broken call ALSO breaks
pytest's own capture handler (stdlib formatting raises there too), which would fail the test for
the wrong reason. What is under test is our formatter's refusal to raise."""
record = logging.LogRecord(
"siftlode.test", logging.INFO, __file__, 0, "two args %s %s", ("only-one",), None
)
payload = json.loads(logs.JsonFormatter().format(record))
assert "log formatting failed" in payload["msg"]
def test_bound_correlation_appears_and_is_released(sink):
log = logging.getLogger("siftlode.test")
with logs.bind(job_id=42):
log.info("inside")
log.info("outside")
assert sink.objects()[-2]["job_id"] == 42
assert "job_id" not in sink.objects()[-1]
def test_an_unknown_level_name_falls_back_to_info_instead_of_silencing():
assert logs.resolve_level("nonsense") == logging.INFO
assert logs.resolve_level(None) == logging.INFO
assert logs.resolve_level("trace") == logs.TRACE
assert logs.resolve_level(" debug ") == logging.DEBUG
def test_configure_logging_silences_the_uvicorn_access_log_and_keeps_ours_propagating():
logs.configure_logging("api")
assert logging.getLogger("uvicorn.access").getEffectiveLevel() >= logging.CRITICAL
# propagate stays on so every line reaches the ONE root handler — and so pytest's caplog works.
assert logging.getLogger("siftlode.anything").propagate is True
assert len(logging.getLogger().handlers) == 1
# --- the request line -----------------------------------------------------------------------
def _client() -> TestClient:
app = FastAPI()
app.middleware("http")(logs.log_requests)
@app.get("/api/videos/{video_id}")
def _video(video_id: str, q: str | None = None):
return {"ok": video_id}
@app.get("/healthz")
def _health():
return {"ok": True}
@app.get("/boom")
def _boom():
raise RuntimeError("kaboom")
return TestClient(app, raise_server_exceptions=False)
def test_the_request_line_has_the_route_template_and_no_query_string(sink):
logging.getLogger("siftlode.http").setLevel(logs.TRACE)
r = _client().get("/api/videos/abc123?q=cats&token=SECRET")
assert r.status_code == 200
payload = sink.objects()[-1]
assert payload["event"] == "http.request"
# The TEMPLATE, so every video request groups into one queryable path...
assert payload["attrs"]["path"] == "/api/videos/{video_id}"
# ...and the query string — uvicorn's access log printed it, `?account=1` and all — is gone.
assert "SECRET" not in sink.lines[-1]
assert "cats" not in sink.lines[-1]
assert payload["attrs"]["status"] == 200
assert isinstance(payload["attrs"]["duration_ms"], float)
def test_the_request_id_is_echoed_and_shared_by_every_line_of_that_request(sink):
client = _client()
r = client.get("/api/videos/abc123")
rid = r.headers["X-Request-Id"]
assert rid and sink.objects()[-1]["request_id"] == rid
def test_a_client_supplied_id_is_reused_only_when_it_is_well_formed(sink):
client = _client()
good = client.get("/api/videos/x", headers={"X-Request-Id": "web-0123456789ab"})
assert good.headers["X-Request-Id"] == "web-0123456789ab"
# An unbounded header would be a way to write arbitrary text into the log; short/odd ones are
# replaced with our own id rather than trusted.
for bogus in ("short", "has spaces and \n newline", "x" * 200):
r = client.get("/api/videos/x", headers={"X-Request-Id": bogus})
assert r.headers["X-Request-Id"] != bogus
def test_the_healthcheck_is_logged_at_debug_so_it_cannot_drown_the_log(sink):
http = logging.getLogger("siftlode.http")
http.setLevel(logs.TRACE)
_client().get("/healthz")
assert sink.objects()[-1]["level"] == "DEBUG"
# ...but it is still logged when something is wrong with it: the floor drops, nothing is hidden.
http.setLevel(logging.INFO)
before = len(sink.lines)
_client().get("/healthz")
assert len(sink.lines) == before
def test_a_failing_request_is_logged_by_status_and_an_unhandled_error_is_logged_at_all(sink):
client = _client()
client.get("/api/videos/missing/deeper") # 404 from the router
assert sink.objects()[-1]["level"] == "WARNING"
client.get("/boom")
payload = sink.objects()[-1]
# The request that most needs a line is the one that raised: uvicorn's access log wrote it, but
# our middleware sees the exception instead of a response, so it has to log it explicitly.
assert payload["level"] == "ERROR"
assert payload["event"] == "http.request"
assert "RuntimeError" in payload["exc"]
+13
View File
@@ -11,6 +11,15 @@
# applies local migrations to THIS local DB only. Seed the catalog once with a pg_dump from
# another instance if you want real data (see RUNBOOK).
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
# busy instance and a hard ceiling on the disk it can take.
x-logging: &logging
driver: json-file
options:
max-size: "50m"
max-file: "3"
services:
db:
image: postgres:16-alpine
@@ -27,6 +36,7 @@ services:
interval: 5s
timeout: 5s
retries: 12
logging: *logging
restart: unless-stopped
api:
@@ -63,6 +73,7 @@ services:
- ./downloads:/downloads
# Plex media, read-only (see the plex_media volume note below). Maps to plex_path_map.
- plex_media:/plex-media:ro
logging: *logging
restart: unless-stopped
# Dedicated download worker: same image, runs the yt-dlp job loop instead of the API.
@@ -96,6 +107,7 @@ services:
volumes:
- ./downloads:/downloads
- plex_media:/plex-media:ro
logging: *logging
restart: unless-stopped
# PO-token provider: mints YouTube Proof-of-Origin tokens on demand so the worker bypasses
@@ -106,6 +118,7 @@ services:
init: true
security_opt:
- apparmor:unconfined
logging: *logging
restart: unless-stopped
volumes:
+13
View File
@@ -10,6 +10,15 @@
# Everything else — the admin account, Google sign-in, SMTP — is set in the web wizard on
# first run. See docs/self-hosting.md.
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
# busy instance and a hard ceiling on the disk it can take.
x-logging: &logging
driver: json-file
options:
max-size: "50m"
max-file: "3"
services:
db:
image: postgres:16-alpine
@@ -27,6 +36,7 @@ services:
interval: 10s
timeout: 5s
retries: 12
logging: *logging
restart: unless-stopped
api:
@@ -71,6 +81,7 @@ services:
timeout: 5s
retries: 5
start_period: 30s
logging: *logging
restart: unless-stopped
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
@@ -109,6 +120,7 @@ services:
- ALL
tmpfs:
- /tmp
logging: *logging
restart: unless-stopped
# PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
@@ -119,6 +131,7 @@ services:
networks: [internal]
security_opt:
- no-new-privileges:true
logging: *logging
restart: unless-stopped
volumes:
+13
View File
@@ -1,3 +1,12 @@
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
# busy instance and a hard ceiling on the disk it can take.
x-logging: &logging
driver: json-file
options:
max-size: "50m"
max-file: "3"
services:
db:
image: postgres:16-alpine
@@ -12,6 +21,7 @@ services:
interval: 5s
timeout: 5s
retries: 12
logging: *logging
restart: unless-stopped
api:
@@ -49,6 +59,7 @@ services:
timeout: 5s
retries: 5
start_period: 25s
logging: *logging
restart: unless-stopped
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
@@ -83,6 +94,7 @@ services:
- ${DOWNLOAD_HOST_PATH:-downloads}:/downloads
# Same read-only Plex library as the API — the worker resolves library paths too.
- ${PLEX_MEDIA_HOST_PATH:-plex_media}:/plex-media:ro
logging: *logging
restart: unless-stopped
# PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
@@ -90,6 +102,7 @@ services:
bgutil-pot:
image: brainicism/bgutil-ytdlp-pot-provider:1.3.1
init: true
logging: *logging
restart: unless-stopped
volumes:
+6 -1
View File
@@ -15,7 +15,8 @@
"downloads": "Downloads",
"plex": "Plex",
"tmdb": "TMDB",
"audit": "Audit log"
"audit": "Audit log",
"diagnostics": "Diagnostics"
},
"fields": {
"smtp_host": {
@@ -154,6 +155,10 @@
"label": "Retention (days)",
"hint": "How long admin audit-log entries are kept before the audit-log cleanup job prunes them. 0 = keep forever. Default 180."
},
"log_level": {
"label": "Log level",
"hint": "How much the server writes to its log. INFO is the normal setting; DEBUG and TRACE are for reproducing a problem and are far more detailed. Takes effect immediately, no restart."
},
"plex_enabled": {
"label": "Enable Plex",
"hint": "Expose the Plex library as a Feed source with a rich player. Requires the Plex server to be reachable from this host and its media on a local mount (see the path map)."
+6 -1
View File
@@ -15,7 +15,8 @@
"downloads": "Letöltések",
"plex": "Plex",
"tmdb": "TMDB",
"audit": "Naplózás"
"audit": "Naplózás",
"diagnostics": "Diagnosztika"
},
"fields": {
"smtp_host": {
@@ -154,6 +155,10 @@
"label": "Megőrzés (nap)",
"hint": "Meddig maradnak meg az admin-naplóbejegyzések, mielőtt a napló-karbantartó feladat törli őket. 0 = örökre. Alapérték 180."
},
"log_level": {
"label": "Naplózási szint",
"hint": "Mennyit naplózzon a szerver. Az INFO a szokásos; a DEBUG és a TRACE hibakereséshez való, sokkal részletesebb. Azonnal érvénybe lép, újraindítás nélkül."
},
"plex_enabled": {
"label": "Plex engedélyezése",
"hint": "A Plex-könyvtár megjelenítése forrásként, gazdag lejátszóval. Ehhez a Plex-szervernek elérhetőnek kell lennie erről a gépről, a média pedig helyi mounton (lásd az útvonal-leképezést)."