release: 0.89.0 (improvement/e22_5-backend-logging)

- The download and edit failure lines go back after their DB write: a job the
  write could not fail is not a failed job, and the worker's child_exit line
  already says it failed.
- The download child's runner is the last stop before Python's hook: an
  escaping exception is one download.child.failed line with its chain scrubbed,
  then exit 1, instead of a raw stderr traceback carrying the URL.
- A crash the database will not record carries its stack once, and the
  fail-write line has an event code.
- The job's own generic-extractor name is taken out as a literal (formats.hiding),
  so brackets and colons in a direct link's file name no longer leak; the
  pattern stays as the fallback. `Downloading just the video <id>` is covered.
- Tests: scrubbed_stack's cause, suppress, unprintable and text-format paths,
  the image steps' failure kinds, the watchdog error through the real
  _fail_running, exact poster/storyboard messages, ffmpeg skip marker.

.env.example

@@ -28,6 +28,14 @@ ALLOWED_EMAILS=
 # Admin emails (subset of the above) get the admin role.
 ADMIN_EMAILS=
 
+# Logging (E22). LOG_LEVEL is the BOOTSTRAP level — the level a process starts at and the only one
+# the worker/download child read; the API's running level is the admin Configuration page's
+# `log_level` knob, which takes effect without a restart. TRACE | DEBUG | INFO | WARNING | ERROR, in
+# any letter case; any other value stops the boot with an error. LOG_FORMAT is "json" (one object per
+# line — what a log collector ingests) or "text" (a readable line for a local terminal).
+LOG_LEVEL=INFO
+LOG_FORMAT=json
+
 # Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
 FRONTEND_ORIGIN=
 

CHANGELOG.md

@@ -7,6 +7,49 @@ edit the release notes and re-run the generator.
 
 Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
 
+## 0.89.0 — 2026-09-19
+
+A release for the log rather than for the screen: what the server writes down when something fails is now complete, said once, and free of what you downloaded or searched for.
+
+### Fixes
+
+- A download that failed because the server lost its database for a moment no longer shows the video's title and file path inside the database's own error message.
+
+### Internal
+
+- No log line carries what a person typed, watched or downloaded: search terms, titles, channel names, file paths, links and video ids are taken out — also from yt-dlp's and ffmpeg's own text, from error stacks, and from database errors. The one exception is Full diagnostics, and only under the user's own consent.
+- Every one of the backend's 114 swallowed exceptions was ruled on: 44 now leave a line with an event code, the other 70 say in the code why they need none — and a test keeps it that way. A failure is logged once, with its stack when it is our own bug, and a stalled or killed download no longer fails without a line.
+- yt-dlp's own explanations (“Requested format is not available”, a sign-in wall) reach the log instead of a stdout nobody read, and a burst of identical lines is summarised instead of flooding the stored log.
+- Three adversarial review rounds: 49 findings raised, 42 survived three independent attempts to refute them, all fixed, and every regression test was proven to fail with its own fix removed.
+
+## 0.88.0 — 2026-09-18
+
+When something goes wrong, you can now turn on detailed logging for yourself for 24 hours — and read your own log on the spot, before any of it is sent anywhere. Siftlode also has a privacy policy and terms at last, in both languages, saying exactly what is recorded and for how long.
+
+### Features
+
+- Settings → Diagnostics: choosing Detailed or Full changes nothing until you confirm a box that tells you what will be recorded, that the server keeps it, for how long, and that it switches itself off after 24 hours. Going back to Basic needs no confirmation and takes effect at once.
+- “My log”, on the same tab, shows what your browser has recorded — newest first, any line openable. Nothing leaves the browser before you have consented.
+- What you search for and filter by is recorded only at Full, and only under your own consent — never an administrator's. Passwords, tokens, cookies and email addresses are recorded at no level, by anyone.
+- Your last sign-in is shown in your account panel, and in the administrator's user list.
+- A privacy policy and terms of service, in Hungarian and English, covering what the server logs and what each diagnostics level adds.
+- Administrators get a Diagnostics section in Configuration: the log level takes effect without a restart, and stored logs are cleaned up on a schedule (7 days by default).
+- Every setting with a fixed set of values is now picked from those values instead of typed in, so a wrong one can no longer be saved.
+
+### Fixes
+
+- Every message arrived twice if you left a conversation and came back quickly: two chat connections stayed open where there should have been one.
+- Settings was unusable on a phone — the tab rail took the width and left almost nothing for the panel beside it.
+- Dates and numbers follow the language you picked in Siftlode, not the one your computer happens to be set to. An English Siftlode on a Hungarian machine no longer shows Hungarian dates.
+- An administrator setting shown as a dropdown opened on its first option instead of the one actually in force.
+- In Hungarian, the audit log and the new log settings had names you could not tell apart.
+- The scheduled feed poll now covers only channels somebody actually follows — on this instance 582 of 979 polled channels had no follower at all. The trade-off is deliberate: new uploads from a channel nobody follows no longer appear in the shared Library.
+
+### Internal
+
+- One logging mechanism replaced four. The API, the worker, the downloader and the migration tool now write the same structured line; secrets are scrubbed where the line is written rather than at each of ~125 places that write one; and every request carries an id that joins a line from your browser to the server line of the same request. The browser's log uses the same shape and the same redaction rules, checked from both sides against one shared set of cases.
+- Two adversarial review rounds over the epic: 56 findings raised, 32 survived three independent attempts to refute each, all fixed. Every regression test was proven to fail with its own fix removed — which is how a test that passed only on leftover state from an earlier test was caught, and how a focus bug that jsdom could not reproduce at all was settled in a real browser.
+
 ## 0.87.1 — 2026-09-16
 
 A YouTube video that YouTube will only show to a signed-in viewer can now be downloaded: paste your YouTube cookies in Settings the way the failure message already told you to — until now that list had no YouTube in it.

CLAUDE.md

@@ -58,7 +58,7 @@ how it gets built. Concretely:
 `routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
 `downloads/`, `youtube/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
 `sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`,
-`security`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
+`security`, `logs`, `logsink`, `diagnostics`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
 Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → module
 components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
 its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
@@ -95,6 +95,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
 | FE control              | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>`                                    |
 | FE loading/empty/error  | `QueryState` / `StateMessage`                                | ad-hoc conditionals                                    |
 | FE failed write         | `meta.failTitle`, else `notifyActionError(err, title)`       | a bare `catch {}` or `notify()` with no error attached |
+| FE network call         | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
+| FE diagnostics          | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
 
 ## House rules that bite (learned from real bugs)
 

VERSION

@@ -1 +1 @@
-0.87.1
+0.89.0

backend/alembic.ini

@@ -1,6 +1,8 @@
 [alembic]
 script_location = alembic
 prepend_sys_path = .
+# Split the path list the way the OS does (alembic's legacy space/comma/colon split is deprecated).
+path_separator = os
 
 [loggers]
 keys = root,sqlalchemy,alembic

backend/alembic/env.py

@@ -1,17 +1,17 @@
-from logging.config import fileConfig
-
 from alembic import context
 from sqlalchemy import engine_from_config, pool
 
 from app.config import settings
 from app.db import Base
+from app.logs import configure_logging
 import app.models  # noqa: F401  (import so models register on Base.metadata)
 
 config = context.config
 config.set_main_option("sqlalchemy.url", settings.database_url)
 
-if config.config_file_name is not None:
-    fileConfig(config.config_file_name)
+# Migrations run at container start (entrypoint.sh), so their output is part of the API's log
+# stream — one format with everything else, not alembic.ini's own fileConfig shape.
+configure_logging("alembic")
 
 target_metadata = Base.metadata
 

backend/alembic/versions/0085_users_last_login_at.py

@@ -0,0 +1,26 @@
+"""add users.last_login_at — when the account last signed in (E22 S1)
+
+Revision ID: 0085_users_last_login_at
+Revises: 0084_dl_job_error_detail
+Create Date: 2026-09-16
+
+Stamped by the one sign-in path (`routes/auth._establish_session`), which the Google callback and
+password login share. No backfill is possible: the signal exists only at sign-in, so existing rows
+read NULL until each account signs in again, and the admin UI says so rather than inventing a date.
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0085_users_last_login_at"
+down_revision: str | None = "0084_dl_job_error_detail"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+def upgrade() -> None:
+    op.add_column("users", sa.Column("last_login_at", sa.DateTime(timezone=True), nullable=True))
+
+
+def downgrade() -> None:
+    op.drop_column("users", "last_login_at")

backend/alembic/versions/0086_log_events.py

@@ -0,0 +1,48 @@
+"""add log_events — stored log events for diagnostics (E22 S3)
+
+Revision ID: 0086_log_events
+Revises: 0085_users_last_login_at
+Create Date: 2026-09-17
+
+Written by app/logsink.py's background writer (WARNING and above, request lines), pruned by the log_gc
+job after `log_retention_days`. Sized against prod on 2026-09-16: ~2k rows on a normal day, ~14k on a
+YouTube-RSS-outage day before flood control — a few MB over the 7-day window. `user_id` carries no
+foreign key on purpose (see models.LogEvent).
+"""
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects import postgresql
+
+revision: str = "0086_log_events"
+down_revision: str | None = "0085_users_last_login_at"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "log_events",
+        sa.Column("id", sa.BigInteger(), primary_key=True),
+        sa.Column("ts", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("level", sa.String(10), nullable=False),
+        sa.Column("sev", sa.SmallInteger(), nullable=False),
+        sa.Column("source", sa.String(16), nullable=False),
+        sa.Column("logger", sa.String(128), nullable=False),
+        sa.Column("event", sa.String(64), nullable=True),
+        sa.Column("msg", sa.Text(), nullable=False),
+        sa.Column("attrs", postgresql.JSONB(), nullable=True),
+        sa.Column("exc", sa.Text(), nullable=True),
+        sa.Column("request_id", sa.String(64), nullable=True),
+        sa.Column("user_id", sa.Integer(), nullable=True),
+        sa.Column("debug_session_id", sa.String(64), nullable=True),
+        sa.Column("job_id", sa.Integer(), nullable=True),
+        sa.Column("version", sa.String(64), nullable=True),
+    )
+    op.create_index("ix_log_events_ts", "log_events", ["ts"])
+    op.create_index("ix_log_events_request_id", "log_events", ["request_id"])
+    op.create_index("ix_log_events_user_ts", "log_events", ["user_id", "ts"])
+
+
+def downgrade() -> None:
+    op.drop_table("log_events")

backend/alembic/versions/0087_users_diagnostics_level.py

@@ -0,0 +1,31 @@
+"""add users.diagnostics_level + diagnostics_until — a user's consented diagnostics level (E22 S5)
+
+Revision ID: 0087_users_diagnostics_level
+Revises: 0086_log_events
+Create Date: 2026-09-17
+
+Both NULL = the base level (Alap). A raised level is ALWAYS paired with its expiry, computed by the
+server at consent time (`app/diagnostics.py`), so no client can ask for a level that never ends.
+Columns rather than `users.preferences`: that JSON is written wholesale by the client, and a consent
+with a lifetime must not be something a client can write.
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0087_users_diagnostics_level"
+down_revision: str | None = "0086_log_events"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+def upgrade() -> None:
+    op.add_column("users", sa.Column("diagnostics_level", sa.String(8), nullable=True))
+    op.add_column(
+        "users", sa.Column("diagnostics_until", sa.DateTime(timezone=True), nullable=True)
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("users", "diagnostics_until")
+    op.drop_column("users", "diagnostics_level")

backend/app/README.md

@@ -20,9 +20,10 @@ Three layers, by *who* runs the code — not by feature:
    run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline.
 
 3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
-   `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, … are single-file primitives many
-   layers import. They depend on nothing above them (a core module must never import a `routes/…`
-   module).
+   `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, `logs` (the one logging
+   configuration), `logsink` (its `log_events` table) and `diagnostics` (a user's consented log
+   level), … are single-file primitives many layers import. They depend on nothing above them (a core
+   module must never import a `routes/…` module).
 
 **Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
 *second* caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down

backend/app/auth.py

@@ -1,17 +1,17 @@
-import logging
-
 import httpx
 from fastapi import BackgroundTasks, Depends, HTTPException, Request
 from sqlalchemy import delete, func, select
 from sqlalchemy.orm import Session
 from starlette.requests import HTTPConnection
 
+from app import diagnostics
 from app import email as email_mod
+from app import logs
 from app import mailtext
 from app import sysconfig
 from app.config import settings
 from app.db import get_db
-from app.models import Invite, User
+from app.models import Invite, LogEvent, User
 from app.security import decrypt_optional
 from app.utils import canonical_email
 
@@ -37,7 +37,7 @@ READ_SCOPES = f"{BASE_SCOPES} {READ_SCOPE}"
 WRITE_SCOPES = f"{BASE_SCOPES} {READ_SCOPE} {WRITE_SCOPE}"
 _YOUTUBE_SCOPES = frozenset({READ_SCOPE, WRITE_SCOPE})
 
-log = logging.getLogger("siftlode.auth")
+log = logs.get_logger(__name__)
 
 
 def google_enabled(db: Session) -> bool:
@@ -220,7 +220,7 @@ def revoke_google_token(token: str | None) -> None:
             timeout=10,
         )
         log.info("Google token revoke: status=%s", resp.status_code)
-    except Exception:
+    except Exception:  # noqa: BLE001 — a revoke we cannot make must not fail the sign-out
         log.warning("Google token revoke failed", exc_info=True)
 
 
@@ -245,6 +245,12 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
     )
     # Raw delete so Postgres applies ON DELETE CASCADE / SET NULL on every dependent table.
     db.execute(delete(User).where(User.id == user_id))
+    # Stored log events name the account by id with no foreign key to cascade from (models.LogEvent),
+    # so they go explicitly — they are request history, i.e. personal data. Forgotten FIRST: this very
+    # request's access line, a queued row, a later background task would otherwise put rows back
+    # under the id after this DELETE (logs.forget_user).
+    logs.forget_user(user_id)
+    db.execute(delete(LogEvent).where(LogEvent.user_id == user_id))
     # Every canonically-equal invite goes, not just the literal spelling: is_allowed matches on the
     # canonical form, so leaving a sibling row behind would keep granting access to a mailbox whose
     # account was just erased — and prod really does carry approved pairs for one mailbox.
@@ -297,6 +303,13 @@ def resolved_user_id(conn: HTTPConnection, *, strict: bool = True) -> tuple[int
         try:
             hid = int(hdr)
         except ValueError:
+            # Under `strict` this becomes a 401 and the tab authenticates as nobody, which reads
+            # in the browser exactly like a session that expired. The value itself stays out —
+            # it came from a request, and `wallet` says which accounts were actually on offer.
+            log.debug(
+                "the active-account header could not be read as an id",
+                extra=logs.event("auth.account_header.invalid", accounts=len(wallet)),
+            )
             hid = None
         if hid is not None and hid in wallet:
             return hid, hid == default_id
@@ -348,6 +361,13 @@ def _session_user(request: Request, db: Session, user_id: int | None, is_default
     if default_id and default_id not in accounts:
         accounts.append(default_id)
         request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
+    # Attribute this request's log lines to the account (E22). Here, not in the request middleware:
+    # this is the one place a session becomes an account, and the middleware runs outside both the
+    # session middleware and this task's context. `request.state` is shared across middleware layers
+    # (it lives on the ASGI scope), so the access-log line can read it back. The user's consented
+    # diagnostics level rides along: their requests are logged — and stored — in that detail (S5).
+    logs.set_user(user.id, diagnostics.effective_level(user), user.diagnostics_until)
+    request.state.user_id = user.id
     return user
 
 
@@ -361,12 +381,20 @@ def optional_current_user(request: Request, db: Session = Depends(get_db)) -> Us
     user_id, is_default = resolved_user_id(request, strict=False)
     try:
         return _session_user(request, db, user_id, is_default)
+    # silent-ok: this is the anonymous case as much as the refused one — every landing page hit
+    # comes through here. The second attempt below is the handling, and it reports.
     except HTTPException:
         if is_default:
             return None
     try:
         return _session_user(request, db, *resolved_user_id(request, strict=False))
     except HTTPException:
+        # Both the pinned account AND the default one were refused, so the tab is signed out
+        # without being told why — and it got here from a session that named a real account.
+        log.info(
+            "a session was resolved to nobody: neither the named nor the default account held",
+            extra=logs.event("auth.session.probe_rejected", user_id=user_id),
+        )
         return None
 
 

backend/app/config.py

@@ -1,6 +1,7 @@
 from pathlib import Path
+from typing import Literal, get_args, get_origin
 
-from pydantic import model_validator
+from pydantic import ValidationInfo, field_validator, model_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 # Shipped placeholder; production must override it. Used by the startup guard below.
@@ -15,11 +16,40 @@ def _version_from_file() -> str:
             v = p.read_text(encoding="utf-8").strip()
             if v:
                 return v
+        # silent-ok: this runs at import, and `app.logs` imports THIS module — a log call here
+        # would be a circular import. The loop tries the other location, then falls back to "dev".
         except OSError:
             continue
     return "dev"
 
 
+# Closed value sets. A setting typed with one of these is a CHOICE: pydantic rejects anything else at
+# boot, and sysconfig derives the admin picker's options from the same Literal — so the list lives in
+# exactly one place and a consumer never has to guess what a typo should fall back to.
+SearchSource = Literal["scrape", "api"]
+DownloadLayout = Literal["plex", "flat"]
+LogLevel = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"]
+LogFormat = Literal["json", "text"]
+TranscodeHw = Literal["auto", "on", "off"]
+# libx264's presets, fastest first (`placebo` left out on purpose: no measurable gain for its cost).
+TranscodePreset = Literal[
+    "ultrafast", "superfast", "veryfast", "faster", "fast", "medium", "slow", "slower", "veryslow"
+]
+
+
+def choices_of(annotation: object) -> tuple[str, ...]:
+    """The allowed values of a choice-typed setting, or () for any other type."""
+    return tuple(get_args(annotation)) if get_origin(annotation) is Literal else ()
+
+
+def canonical_choice(options: tuple[str, ...], raw: object) -> str | None:
+    """The option `raw` names, in its canonical spelling (any letter case, surrounding space
+    ignored), or None. The ONE matching rule for a choice value from the env and from the admin
+    API, so a spelling one accepts the other cannot refuse."""
+    folded = str(raw).strip().casefold()
+    return next((option for option in options if option.casefold() == folded), None)
+
+
 class Settings(BaseSettings):
     model_config = SettingsConfigDict(env_file=".env", extra="ignore")
 
@@ -85,7 +115,7 @@ class Settings(BaseSettings):
     search_daily_limit_per_user: int = 70
     # Source for live YouTube search: "scrape" (InnerTube, no API quota) or "api" (search.list,
     # 100 units/page). Scrape is the default; flip to "api" if scraping is blocked/breaks.
-    search_source: str = "scrape"
+    search_source: SearchSource = "scrape"
     # Recent-first backfill: how far back to fetch on the first pass per channel.
     backfill_recent_max_videos: int = 100
     backfill_recent_max_days: int = 365
@@ -224,10 +254,27 @@ class Settings(BaseSettings):
     audit_retention_days: int = 180
     audit_gc_minutes: int = 1440
     # On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
-    download_layout: str = "plex"
+    download_layout: DownloadLayout = "plex"
     # Default SponsorBlock (skip sponsor segments) for new custom profiles.
     sponsorblock_default: bool = False
 
+    # --- Diagnostics / logging (E22) ---
+    # BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
+    # startup and on change without a restart. This is what a process logs at before it can reach
+    # the database — and the only knob the worker/runner children read, since they are short-lived.
+    # TRACE (below DEBUG) is the "Teljes" diagnostics level; see app/logs.py.
+    log_level: LogLevel = "INFO"
+    # "json" (one object per line — what Loki and the log_events sink ingest) or "text" (a readable
+    # line for a local terminal). Redaction applies to both.
+    log_format: LogFormat = "json"
+    # Also write events to the `log_events` table (app/logsink.py): WARNING and above, the request
+    # lines, and — once they exist — client batches. Off only where no table is wanted (the tests).
+    log_db_sink: bool = True
+    # Days a `log_events` row is kept (the privacy policy promises this), and how often the log_gc job
+    # enforces it — hourly, so a row never outlives the promise by more than an hour.
+    log_retention_days: int = 7
+    log_gc_minutes: int = 60
+
     # --- Plex integration (optional) ---
     # The Plex module plays the LOCAL physical media file directly (our own ffmpeg/direct-serve);
     # Plex is used only for metadata + optional watch-state sync. Requires the Plex server to be
@@ -269,7 +316,7 @@ class Settings(BaseSettings):
     # Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
     # source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
     # node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
-    plex_transcode_hw: str = "auto"
+    plex_transcode_hw: TranscodeHw = "auto"
     # Software-fallback libx264 tuning (IGNORED on the VAAPI path). `preset` trades CPU for size; `crf`
     # trades quality for size (lower = better + bigger). `max_height` caps the SOFTWARE transcode's output
     # height (0 = keep source) so a CPU re-encode of a tall source stays affordable — the GPU path always
@@ -279,7 +326,7 @@ class Settings(BaseSettings):
     # call, not a fix. Only affects files that go through remux/transcode — a direct-played file is
     # served untouched, but those are browser-native stereo anyway, not the 5.1 mixes this is for.
     plex_audio_loudnorm: bool = False
-    plex_transcode_preset: str = "veryfast"
+    plex_transcode_preset: TranscodePreset = "veryfast"
     plex_transcode_crf: int = 20
     plex_transcode_max_height: int = 1080
     # Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
@@ -328,6 +375,17 @@ class Settings(BaseSettings):
         https OAUTH_REDIRECT_URL as the signal for 'real deployment' vs local dev."""
         return self.oauth_redirect_url.lower().startswith("https://")
 
+    @field_validator("*", mode="before")
+    @classmethod
+    def _canonical_choice(cls, value: object, info: ValidationInfo) -> object:
+        """A choice setting accepts its value in any letter case (`LOG_LEVEL=debug`) and stores the
+        canonical spelling; an unknown value still fails the boot with pydantic's own message."""
+        field = cls.model_fields.get(info.field_name or "")
+        options = choices_of(field.annotation) if field else ()
+        if options and isinstance(value, str):
+            return canonical_choice(options, value) or value
+        return value
+
     @model_validator(mode="after")
     def _enforce_production_secrets(self) -> "Settings":
         """Fail fast rather than boot a public instance with the shipped placeholder

backend/app/db.py

@@ -20,6 +20,10 @@ engine = create_engine(
     max_overflow=30,
     pool_timeout=30,
     future=True,
+    # A database error renders its bound parameters into its message — a title, an uploader, a
+    # file path, a URL — and that message is logged and stored as a job's error text. The SQL
+    # stays; the values are data, and the ids an operator needs are on the line already.
+    hide_parameters=True,
 )
 SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False)
 

backend/app/diagnostics.py

@@ -0,0 +1,61 @@
+"""A user's diagnostics level (E22 S5): what they consented to, and for how long.
+
+Three levels, named in the UI Alap / Részletes / Teljes:
+
+  * INFO  — the base, for everyone, no consent. The browser keeps its log in memory; nothing extra
+            is sent or stored.
+  * DEBUG — consented. The server logs this user's requests in detail and stores every line of
+            them, and the browser streams its own log to `POST /api/diagnostics/events`.
+  * TRACE — consented. As DEBUG, plus what the user typed into a search.
+
+A raised level always lapses after `ELEVATION_HOURS`: the expiry is computed HERE at consent time,
+never taken from the client, and an expired level reads as INFO without anything having to clear it.
+The shared demo account cannot raise its level — its log would mix every visitor's session.
+
+Core layer: imports `models` and `sysconfig`.
+"""
+
+from datetime import datetime, timedelta, timezone
+from typing import Literal, get_args
+
+from sqlalchemy.orm import Session
+
+from app import sysconfig
+from app.models import User
+
+DiagnosticsLevel = Literal["INFO", "DEBUG", "TRACE"]
+LEVELS: tuple[str, ...] = get_args(DiagnosticsLevel)
+BASE: DiagnosticsLevel = "INFO"
+ELEVATION_HOURS = 24
+
+
+def effective_level(user: User, now: datetime | None = None) -> DiagnosticsLevel:
+    """The level in force for `user` right now: the consented one until it expires, else BASE."""
+    level, until = user.diagnostics_level, user.diagnostics_until
+    if level not in LEVELS or level == BASE or until is None:
+        return BASE
+    if until <= (now or datetime.now(timezone.utc)):
+        return BASE
+    return level  # type: ignore[return-value] — narrowed by the LEVELS check above
+
+
+def state(db: Session, user: User) -> dict:
+    """The `/api/me` view of it: the level in force, when a raised one lapses, and how long stored
+    log lines are kept — the consent text states that number, so it must be the instance's own."""
+    level = effective_level(user)
+    return {
+        "level": level,
+        "until": user.diagnostics_until.isoformat() if level != BASE else None,
+        "retention_days": sysconfig.get_int(db, "log_retention_days"),
+    }
+
+
+def set_level(user: User, level: DiagnosticsLevel) -> None:
+    """Record a consent (or its withdrawal). The caller commits. Raising the level again restarts
+    the clock — each consent is for the next ELEVATION_HOURS, not an extension of an older one."""
+    if level == BASE:
+        user.diagnostics_level = None
+        user.diagnostics_until = None
+        return
+    user.diagnostics_level = level
+    user.diagnostics_until = datetime.now(timezone.utc) + timedelta(hours=ELEVATION_HOURS)

backend/app/discover/pins.py

@@ -62,6 +62,8 @@ def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, obje
         return (col, sh.id) if sh is not None else None
     try:
         pid = int(ref)
+    # silent-ok: validating a reference the client supplied. "Not an integer" is answered as "no
+    # such pin target", which is the same answer this function gives for one that does not exist.
     except ValueError:
         return None
     pl = db.get(Playlist, pid)

backend/app/downloads/adapters.py

@@ -18,6 +18,7 @@ import yt_dlp
 from yt_dlp.networking.impersonate import ImpersonateTarget
 
 from app.config import settings
+from app.downloads.stated import StatedFailure
 
 # Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
 # below and the failure catalog (errors.py) decide on it, so the wording exists once.
@@ -178,11 +179,11 @@ class FacebookAdapter(SourceAdapter):
         # Measured (E11.5 review): a refused or expired cookie gets the login page (~450 KB, a login
         # form, no post id at all), which is not a post without a video — the user's cookie is the cause.
         if _quoted(post_id) not in page:
-            raise RuntimeError(POST_NOT_SHOWN_MESSAGE)
+            raise StatedFailure(POST_NOT_SHOWN_MESSAGE)
         video_id = facebook_post_video_id(page, post_id)
         if video_id is None:
             # Never fall back to the post link: that is exactly where the wrong video comes from.
-            raise RuntimeError(POST_WITHOUT_VIDEO_MESSAGE)
+            raise StatedFailure(POST_WITHOUT_VIDEO_MESSAGE)
         return f"https://www.facebook.com/watch/?v={video_id}"
 
     def client_sets(self, authed: bool = False) -> list:

backend/app/downloads/backfill.py

@@ -11,11 +11,12 @@ Run once per environment:
     docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
 """
 
-import logging
+from urllib.parse import urlsplit
 
 import yt_dlp
 from sqlalchemy import select
 
+from app import logs
 from app.config import settings
 from app.db import SessionLocal
 from app.downloads import adapters
@@ -23,12 +24,17 @@ from app.downloads import edit as editmod
 from app.downloads import formats, service
 from app.models import MediaAsset
 
-log = logging.getLogger("siftlode.backfill")
+log = logs.get_logger(__name__)
 
 
-def _extract_meta(url: str, adapter: adapters.SourceAdapter) -> dict | None:
+def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
     """Metadata-only yt-dlp extraction, using the source adapter's client sets + options (YouTube
     needs the POT/player-client strategy even just to extract)."""
+    with formats.hiding(url):
+        return _extract_meta_scoped(url, adapter, asset_id)
+
+
+def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
     for clients in adapter.client_sets():
         opts = formats.build_ydl_opts(
             {}, "%(id)s.%(ext)s", lambda d: None, extra=adapter.ydl_opts_extra(clients)
@@ -40,7 +46,19 @@ def _extract_meta(url: str, adapter: adapters.SourceAdapter) -> dict | None:
             if info:
                 return info
         except Exception as exc:  # noqa: BLE001 — best-effort backfill, keep going
-            log.info("meta extract failed for %s (%s): %s", url, clients, str(exc)[:100])
+            # The asset id, not the URL. A source URL is a row of this user's download
+            # history, and `redact_text` only takes credential params out of it — the rest
+            # of it goes in whole. The id names the same asset to anyone holding the
+            # database and nothing at all to anyone who is not; the host is kept because it
+            # is what decides which client set was tried. yt-dlp's message goes through
+            # `log_safe` for the same reason: it opens with the URL or the video id.
+            log.info(
+                "meta extract failed for asset %s on %s (%s): %s",
+                asset_id,
+                urlsplit(url).hostname or "?",  # log-pii-ok: the host, never the path or query
+                clients,
+                formats.log_safe(exc)[:100],
+            )
     return None
 
 
@@ -60,12 +78,14 @@ def run() -> dict:
             adapter = adapters.resolve_adapter(a.source_kind, a.source_ref)
             if a.uploader_url is None and adapter.wants_metadata_backfill():
                 url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
-                info = _extract_meta(url, adapter)
+                info = _extract_meta(url, adapter, a.id)
                 uu = adapter.uploader_url(info) if info else None
                 if uu:
                     a.uploader_url = uu
                     filled_url += 1
-                    log.info("asset %s uploader_url <- %s", a.id, uu)
+                    log.info(
+                        "asset %s uploader_url filled", a.id
+                    )  # the URL names what they downloaded
             if a.poster_path is None:
                 # Record a self-hosted poster for every download (the reliable og:image source):
                 # ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
@@ -73,7 +93,7 @@ def run() -> dict:
                 if rel:
                     a.poster_path = rel
                     filled_poster += 1
-                    log.info("asset %s poster <- %s", a.id, rel)
+                    log.info("asset %s poster filled", a.id)  # the path carries the title
             db.commit()
     result = {
         "assets": len(assets),

backend/app/downloads/credentials.py

@@ -23,19 +23,19 @@ Security controls enforced here:
     logged, or written to a job's error/snapshot.
 """
 
-import logging
 import os
 from pathlib import Path
 
 from sqlalchemy import select
 from sqlalchemy.orm import Session
 
+from app import logs
 from app.downloads import formats
 from app.downloads.adapters import _host_is
 from app.models import DownloadCredential, MediaAsset
 from app.security import decrypt_optional, encrypt
 
-log = logging.getLogger("siftlode.downloads.credentials")
+log = logs.get_logger(__name__)
 
 
 class HostSpec:

backend/app/downloads/edit.py

@@ -23,7 +23,6 @@ for a player hover-preview later.
 
 import hashlib
 import json
-import logging
 import math
 import os
 import queue
@@ -33,9 +32,11 @@ import threading
 import time
 from pathlib import Path
 
+from app import logs
 from app.config import settings
+from app.downloads.stated import StatedFailure
 
-log = logging.getLogger("siftlode.edit")
+log = logs.get_logger(__name__)
 
 
 def encode_thread_cap() -> int:
@@ -71,7 +72,13 @@ def normalize_edit_spec(spec: dict | None) -> dict:
     if crop:
         try:
             c = {k: int(round(float(crop[k]))) for k in ("x", "y", "w", "h")}
-        except (KeyError, TypeError, ValueError):
+        except (KeyError, TypeError, ValueError) as exc:
+            # The clip still renders — uncropped. The user asked for a crop, got a whole frame,
+            # and nothing anywhere called it a failure; this line is the only trace there is.
+            log.warning(
+                "the crop in an edit spec could not be read and was dropped",
+                extra=logs.event("edit.spec.crop_dropped", reason=type(exc).__name__),
+            )
             c = None
         if c and c["w"] > 0 and c["h"] > 0 and c["x"] >= 0 and c["y"] >= 0:
             out["crop"] = c
@@ -80,6 +87,7 @@ def normalize_edit_spec(spec: dict | None) -> dict:
     segs = spec.get("segments")
     if isinstance(segs, list) and segs:
         norm: list[dict] = []
+        unreadable = 0
         for s in segs:
             try:
                 st = max(0.0, float((s or {}).get("start_s") or 0.0))
@@ -88,9 +96,18 @@ def normalize_edit_spec(spec: dict | None) -> dict:
                     continue
                 en = float(end_raw)
             except (TypeError, ValueError):
+                # silent-ok: counted, not logged here — one cut-list can drop several, and the
+                # join that comes out is simply shorter than the one the user built. Reported
+                # once after the loop.
+                unreadable += 1
                 continue
             if en > st:
                 norm.append({"start_s": round(st, 3), "end_s": round(en, 3)})
+        if unreadable:
+            log.warning(
+                "some segments of a cut list could not be read and were dropped",
+                extra=logs.event("edit.spec.segments_dropped", dropped=unreadable, kept=len(norm)),
+            )
         norm.sort(key=lambda x: x["start_s"])
         # SEC-13: an accurate join builds one ffmpeg `filter_complex` with two filters per segment,
         # so an unbounded list becomes an unbounded command line and an ffmpeg that never finishes.
@@ -114,7 +131,13 @@ def normalize_edit_spec(spec: dict | None) -> dict:
             start = max(0.0, float(trim.get("start_s") or 0.0))
             end_raw = trim.get("end_s")
             end = float(end_raw) if end_raw is not None else None
-        except (TypeError, ValueError):
+        except (TypeError, ValueError) as exc:
+            # Either a 400 the user cannot act on, or — beside a crop — a full-length clip where
+            # they asked for a cut. Both are worth one line saying the spec was the problem.
+            log.warning(
+                "the trim in an edit spec could not be read and was dropped",
+                extra=logs.event("edit.spec.trim_dropped", reason=type(exc).__name__),
+            )
             start, end = 0.0, None
         t: dict = {"start_s": round(start, 3)}
         if end is not None and end > start:
@@ -306,11 +329,14 @@ def _parse_out_time(line: str) -> float | None:
     if line.startswith("out_time_us="):
         try:
             return int(line.split("=", 1)[1]) / 1_000_000
+        # silent-ok: one line of ffmpeg's `-progress` stream, several per second. None means "this
+        # line carried no usable time", which the caller already treats as "no progress yet".
         except ValueError:
             return None
     if line.startswith("out_time_ms="):  # (some builds mislabel this as microseconds)
         try:
             return int(line.split("=", 1)[1]) / 1_000_000
+        # silent-ok: as above, the other spelling of the same line.
         except ValueError:
             return None
     return None
@@ -329,7 +355,9 @@ def _pump(stream, sink: "queue.Queue[str | None]") -> None:
     try:
         for line in stream:
             sink.put(line)
-    except (ValueError, OSError):  # pipe closed under us by the kill path
+    # silent-ok: the pipe was closed under us by our OWN kill path — the expected end of a
+    # cancelled or timed-out run, and the code that did the killing is the code that reports it.
+    except (ValueError, OSError):
         pass
     finally:
         sink.put(None)
@@ -347,6 +375,8 @@ def _drain_tail(stream, sink: list[str]) -> None:
             sink.append(line)
             if len(sink) > 200:
                 del sink[:100]
+    # silent-ok: same closed pipe as `_pump` above, and the tail collected so far is still handed
+    # to whoever reports the failure.
     except (ValueError, OSError):
         pass
 
@@ -362,13 +392,17 @@ def escalate_kill(proc: subprocess.Popen, terminate, kill, grace_s: float, label
     try:
         proc.wait(timeout=grace_s)
         return
+    # silent-ok: the escalation below IS the handling of this timeout, and surviving THAT is what
+    # the line under the second wait reports.
     except subprocess.TimeoutExpired:
         pass
     kill()
     try:
         proc.wait(timeout=grace_s)
     except subprocess.TimeoutExpired:
-        log.error("%s pid %s survived SIGKILL", label, proc.pid)
+        # noqa TRY400: the exception IS the message here. A TimeoutExpired's traceback is this
+        # `wait` call and nothing else — it would add three frames and no fact the line lacks.
+        log.error("%s pid %s survived SIGKILL", label, proc.pid)  # noqa: TRY400
 
 
 def _stop(proc: subprocess.Popen) -> None:
@@ -404,6 +438,8 @@ def run_ffmpeg(
         while True:
             try:
                 line = lines.get(timeout=1.0)
+            # silent-ok: not a failure — the one-second timeout is how this loop gets a turn to
+            # poll for cancellation while ffmpeg is quiet.
             except queue.Empty:
                 line = ""
             if line is None:  # stdout hit EOF — ffmpeg is done (or dying)
@@ -416,11 +452,11 @@ def run_ffmpeg(
             if should_cancel():
                 raise EditAborted
             if time.monotonic() - last_output > stall_timeout_s:
-                raise RuntimeError(f"ffmpeg stalled: no progress for {int(stall_timeout_s)}s")
+                raise StatedFailure(f"ffmpeg stalled: no progress for {int(stall_timeout_s)}s")
         try:
             ret = proc.wait(timeout=_FFMPEG_EXIT_GRACE_S)
         except subprocess.TimeoutExpired:  # pipes closed but the process lingers
-            raise RuntimeError("ffmpeg did not exit after closing its output")
+            raise StatedFailure("ffmpeg did not exit after closing its output")
     except BaseException:
         # ONE exit path stops the process, so the invariant is unconditional: nothing leaves this
         # function with ffmpeg still running. Not just our own EditAborted/stall — `on_progress`
@@ -435,7 +471,7 @@ def run_ffmpeg(
             if pipe:
                 pipe.close()
     if ret != 0:
-        raise RuntimeError(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
+        raise StatedFailure(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
 
 
 # --- ffmpeg: universal browser playability --------------------------------------------------
@@ -471,7 +507,13 @@ def run_ffprobe_json(
     try:
         out = subprocess.run(args, capture_output=True, text=True, timeout=30, check=True).stdout
         return json.loads(out)
-    except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
+    except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError) as exc:
+        # None makes every field it would have filled read as absent, and a file with no streams
+        # and no duration is indistinguishable here from one that is not media at all.
+        log.warning(
+            "ffprobe could not read a file",
+            extra=logs.event("media.ffprobe_failed", reason=type(exc).__name__),
+        )
         return None
 
 
@@ -525,7 +567,13 @@ def probe_duration(path: Path) -> int:
             check=True,
         ).stdout.strip()
         return int(float(out)) if out else 0
-    except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
+    except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError) as exc:
+        # Zero is the denominator of the re-encode progress bar, so a failed probe leaves the user
+        # watching a bar that never moves rather than a job that says it went wrong.
+        log.info(
+            "the duration probe failed; progress for this job has no total",
+            extra=logs.event("media.duration_probe_failed", reason=type(exc).__name__),
+        )
         return 0
 
 
@@ -670,15 +718,28 @@ def _run_ffmpeg_best_effort(cmd: list[str], timeout_s: int) -> Exception | None:
     try:
         subprocess.run(cmd, capture_output=True, timeout=timeout_s, check=True)
         return None
+    # silent-ok: the exception is the RETURN VALUE. This helper exists so the three callers share
+    # one run/timeout policy, and each logs the failure with the context only it has.
     except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError) as exc:
         return exc
 
 
+def _step_failure(err: BaseException) -> str:
+    """What a best-effort image step's failure line says: the kind and, for ffmpeg, its exit
+    status. Never `str(err)`: a CalledProcessError or TimeoutExpired quotes ffmpeg's argv, and an
+    OSError its file name — the source path either way, which is built from the channel and the
+    title."""
+    code = getattr(err, "returncode", None)
+    return f"{type(err).__name__} {code}" if code is not None else type(err).__name__
+
+
 def _is_jpeg(path: Path) -> bool:
     """True if the file starts with the JPEG SOI marker (FF D8 FF)."""
     try:
         with path.open("rb") as f:
             return f.read(3) == b"\xff\xd8\xff"
+    # silent-ok: False only means "do not copy this straight through" — the caller then goes the
+    # ffmpeg route, which opens the same file and reports it properly if it really is unreadable.
     except OSError:
         return False
 
@@ -699,13 +760,15 @@ def thumbnail_to_jpg(src: Path | None, dest: Path, timeout_s: int = 30) -> Path
             shutil.copyfile(src, dest)
             return dest
         except OSError as exc:
-            log.info("thumbnail copy failed (%s); poster falls back to a cut frame", str(exc)[:120])
+            log.info(
+                "thumbnail copy failed (%s); poster falls back to a cut frame", _step_failure(exc)
+            )
             return None
     err = _run_ffmpeg_best_effort(build_thumbnail_jpg_cmd(src, dest), timeout_s)
     if err is not None:
         log.info(
             "thumbnail->jpg conversion failed (%s); poster falls back to a cut frame",
-            str(err)[:120],
+            _step_failure(err),
         )
         return None
     return dest if dest.exists() else None
@@ -730,7 +793,7 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
     at = min(3.0, (asset.duration_s or 10) * 0.1)
     err = _run_ffmpeg_best_effort(build_poster_cmd(src, dest, at), timeout_s)
     if err is not None:
-        log.info("poster gen skipped for asset %s: %s", asset.id, str(err)[:120])
+        log.info("poster gen skipped for asset %s: %s", asset.id, _step_failure(err))
         return None
     return rel if dest.exists() else None
 
@@ -761,7 +824,7 @@ def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict |
     cmd = build_storyboard_cmd(src, dest, asset.duration_s, geom)
     err = _run_ffmpeg_best_effort(cmd, timeout_s)
     if err is not None:
-        log.info("storyboard gen skipped for asset %s: %s", asset.id, str(err)[:120])
+        log.info("storyboard gen skipped for asset %s: %s", asset.id, _step_failure(err))
         return None
     if not dest.exists():
         return None

backend/app/downloads/formats.py

@@ -17,8 +17,134 @@ spec shape (all keys optional; defaults + browser-playability clamp applied by `
 `container`/`vcodec` keys survive only so an old stored profile deserializes without error.
 """
 
+import contextlib
+import contextvars
 import hashlib
 import json
+import logging
+import os
+import re
+import urllib.parse
+from collections.abc import Iterator
+
+from app import logs
+
+log = logs.get_logger(__name__)
+
+# yt-dlp names what it is working on in these shapes, and each one is a row of someone's download
+# history: the source URL, the `[extractor] <id>:` prefix on nearly every warning and error, the
+# file path it writes to (`storage.rel_path` builds it from the channel and the title), a
+# playlist's title, and an id quoted mid-sentence. The job id is already on every line
+# (`logs.set_job` in the download child), so none of them adds anything an operator needs. What
+# stays: the host (it says which extractor is failing) and the extractor tag. Every pattern below
+# is a shape measured in yt-dlp's own source (2026.07.04), pinned in test_ytdlp_logging.py.
+_URL_RE = re.compile(r"https?://([^/\s]+)\S*")
+# `Destination:` is yt-dlp's own word in `[download] Destination: <path>`, not an id — the path
+# after it is what `_PATH_RE` takes out.
+_ID_PREFIX_RE = re.compile(r"(\[[\w:.-]+\] )(?!Destination:)[^\s:\[\]]+:")
+# The generic extractor's "id" is the URL's basename, percent-DECODED: `My%20Holiday.mp4` is
+# `My Holiday` — spaces, brackets, colons and all, which no pattern can bound (`Our Trip: Day 1`
+# ends at which `: `?). The job knows its own URL, so the exact name is taken out as a literal
+# (`hiding`). This pattern is the fallback for a line logged outside that scope: up to the first
+# `: `. Only this extractor: every other one names a site id, and letting the prefix run to `: `
+# everywhere would take the words out of lines like `[VideoConvertor] Converting …; Destination:`.
+_GENERIC_ID_RE = re.compile(r"(\[generic\] )(?!Destination:)[^\[\]\n]+?:(?= |$)", re.M)
+# `[youtube:tab] Playlist <title>: Downloading 5 items` — the title is not an id, and has spaces.
+_PLAYLIST_TITLE_RE = re.compile(r"(\] Playlist ).+?(?=: Downloading )")
+# `[download] Finished downloading playlist: <title>`, `[download] Downloading playlist: <title>`,
+# `… Skipping already downloaded playlist: <title>` — the title runs to the end of the line.
+_TITLE_TAIL_RE = re.compile(r"(playlist: |\[download\] Downloading \w+: ).*", re.I)
+# An id quoted mid-sentence, not as a prefix: `… not available for <id>`, and `--no-playlist`'s
+# `Downloading just the video <id>` (`_yes_playlist`) / `Downloading just video <id>` (multifeed).
+_BARE_ID_RE = re.compile(
+    r"((?:subtitles not available for|Downloading just (?:the )?\w+) )[^\s,;]+"
+)
+# Two segments or more, and not glued to a word: `www.youtube.com/…` is what the URL pattern
+# leaves behind, and it must survive this one.
+_PATH_RE = re.compile(r"(?<![\w.…])/(?:[^/\s\"']+/)+[^/\s\"']*")
+
+
+_HIDDEN: contextvars.ContextVar[tuple[str, ...]] = contextvars.ContextVar(
+    "formats_hidden", default=()
+)
+
+
+def generic_id(url: str) -> str:
+    """The name yt-dlp's generic extractor gives a URL: its basename, extension off, percent-
+    decoded — `InfoExtractor._generic_id` (2026.07.04), pinned against it in the tests."""
+    return urllib.parse.unquote(os.path.splitext(url.rstrip("/").split("/")[-1])[0])
+
+
+@contextlib.contextmanager
+def hiding(url: str | None) -> Iterator[None]:
+    """For the work on one URL: `log_safe` also takes out the name yt-dlp's generic extractor
+    gives it, exactly, where yt-dlp puts it (`[generic] <name>:`). A context variable, so it
+    holds for yt-dlp's logger calls too — they run on the same thread as the extraction."""
+    name = generic_id(url) if url else ""
+    token = _HIDDEN.set((*_HIDDEN.get(), name) if name else _HIDDEN.get())
+    try:
+        yield
+    finally:
+        _HIDDEN.reset(token)
+
+
+def log_safe(text: object) -> str:
+    """yt-dlp's text (its own lines, or a DownloadError's message) with the parts that name a
+    person's download taken out. Every log line that carries yt-dlp's words goes through here,
+    not just the adapter's: yt-dlp raises the same text it has just logged, so a line that skipped
+    it would put back one line later exactly what the adapter took out."""
+    text = str(text)
+    for name in _HIDDEN.get():
+        text = text.replace(f"[generic] {name}:", "[generic] …:")
+    text = _URL_RE.sub(lambda m: f"{m.group(1)}/…", text)
+    text = _ID_PREFIX_RE.sub(r"\1…:", text)
+    text = _GENERIC_ID_RE.sub(r"\1…:", text)
+    text = _PLAYLIST_TITLE_RE.sub(r"\1…", text)
+    text = _TITLE_TAIL_RE.sub(r"\1…", text)
+    text = _BARE_ID_RE.sub(r"\1…", text)
+    return _PATH_RE.sub("…", text)
+
+
+class _YdlLog:
+    """yt-dlp's logger protocol, pointed at ours.
+
+    Without it yt-dlp explains itself to nobody: `quiet` sends its output to a stdout we do not
+    read, so "Requested format is not available", a signature-extraction failure or a POT refusal
+    reached us only as whatever text ended up in the raised exception. Measured against yt-dlp
+    2026.07.04: with `logger` set it calls `debug` for extraction and progress lines and
+    `warning`/`error` for the rest, and `quiet`/`no_warnings` do NOT gate those calls.
+
+    `debug` goes to TRACE: it is several lines per second during a download, so it shows only
+    when the whole process runs at TRACE. (Not under a user's own Teljes consent — that lives in
+    the API process's request scope, and the download child that runs yt-dlp has none.)
+
+    `error` goes to WARNING, not ERROR. yt-dlp calls it from `trouble()` just before it raises,
+    on EVERY failed attempt — including the ones the pipeline then recovers from with its fallback
+    client set. Whether the job failed is the pipeline's verdict (`download.job.failed`), and that
+    is the one line whose level should say so."""
+
+    def _emit(self, level: int, msg: str) -> None:
+        # The line goes as an ARGUMENT, not as the format string. yt-dlp writes progress as
+        # `[download]  53.2% of 12.00MiB`, and while stdlib logging only applies `%` formatting to
+        # a record that has args, this way the question never arises — and it is the shape ruff's
+        # `G` rules ask for. (The sink's flood limiter groups a bare `%s` by the text itself, so
+        # distinct yt-dlp warnings do not share one bucket — `logsink._template`.)
+        log.log(level, "%s", log_safe(msg), extra=logs.event("download.ytdlp"))
+
+    def debug(self, msg):
+        self._emit(logs.TRACE, msg)
+
+    def info(self, msg):
+        self._emit(logging.DEBUG, msg)
+
+    def warning(self, msg):
+        self._emit(logging.WARNING, msg)
+
+    def error(self, msg):
+        self._emit(logging.WARNING, msg)
+
+
+_YDL_LOG = _YdlLog()
 
 # Fields that affect the produced file — and therefore the cache identity.
 _SIG_KEYS = (
@@ -78,7 +204,13 @@ def normalize(spec: dict) -> dict:
         # rather than raising an unhandled 500 at enqueue.
         try:
             s["max_height"] = int(s["max_height"])
-        except (TypeError, ValueError):
+        except (TypeError, ValueError) as exc:
+            # "best" is the opposite of what a height cap asks for: the user set a ceiling to save
+            # bandwidth or disk, and the download quietly takes the largest rendition there is.
+            log.warning(
+                "the height cap on a download could not be read and was dropped",
+                extra=logs.event("download.format.max_height_dropped", reason=type(exc).__name__),
+            )
             s["max_height"] = None
     for b in ("embed_subs", "embed_chapters", "embed_thumbnail", "sponsorblock"):
         s[b] = bool(s[b])
@@ -181,6 +313,12 @@ def build_ydl_opts(
     if extra:
         opts.update(extra)
 
+    # With a logger attached, `quiet` and `no_warnings` stop deciding whether yt-dlp explains itself
+    # and only decide where — so both stay on above and the explanation comes here instead of to a
+    # stdout nothing reads. AFTER the adapter merge, like `cookiefile` and the size cap: an adapter
+    # must not be able to silence the one account of what went wrong that a failed job leaves.
+    opts["logger"] = _YDL_LOG
+
     # SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
     # `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set.
     #

backend/app/downloads/gc.py

@@ -16,7 +16,6 @@ Five passes each run:
 Runs in the API process (which mounts DOWNLOAD_ROOT). Pure disk/DB work — no YouTube quota.
 """
 
-import logging
 import time
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
@@ -24,14 +23,13 @@ from pathlib import Path
 from sqlalchemy import func, select
 from sqlalchemy.orm import Session
 
-from app import sysconfig
+from app import logs, sysconfig
 from app.config import settings
 from app.downloads import storage
 from app.models import DownloadJob, MediaAsset
 from app.notifications import create_notification
 
-log = logging.getLogger("siftlode.downloads.gc")
-
+log = logs.get_logger(__name__)
 # Dot-prefixed top-level trees that are NOT part of the Plex/flat asset layout — the orphan sweep
 # never touches them (user clips, Plex HLS/img caches, in-progress staging, editor storyboards).
 _SYSTEM_TREE = {".edits", ".plex-hls", ".plex-img-cache", ".staging", ".storyboards"}
@@ -99,6 +97,7 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
         if pause_s and touched % batch == 0:
             time.sleep(pause_s)
 
+    unreclaimed = 0
     for entry in root.iterdir():
         if entry.name.startswith("."):  # .edits / .plex-* / .staging / .storyboards + any dotfile
             continue
@@ -111,7 +110,9 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
                 f.unlink()
                 removed += 1
             except OSError:
-                pass
+                # silent-ok: counted, not logged per file — a broken mount would put one line
+                # under every orphan. The total is reported once at the end of the sweep.
+                unreclaimed += 1
         if entry.is_dir():  # prune emptied dirs (deepest first), then the channel dir itself
             for d in sorted(
                 (p for p in entry.rglob("*") if p.is_dir()),
@@ -120,12 +121,20 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
             ):
                 try:
                     d.rmdir()
+                # silent-ok: `rmdir` refusing means the directory still holds files, which is
+                # the ordinary answer here — this prune only takes the ones that emptied.
                 except OSError:
                     pass
             try:
                 entry.rmdir()
+            # silent-ok: the same prune one level up, at the channel directory.
             except OSError:
                 pass
+    if unreclaimed:
+        log.warning(
+            "some orphaned download files could not be reclaimed",
+            extra=logs.event("download.gc.unreclaimed", files=unreclaimed, removed=removed),
+        )
     return removed
 
 

backend/app/downloads/links.py

@@ -49,6 +49,8 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
     exp_s, sig = grant.split(".", 1)
     try:
         exp = int(exp_s)
+    # silent-ok: an untrusted grant that does not parse is simply not a valid grant, and the
+    # honest answer to it — a password prompt — is the same as for one that expired.
     except ValueError:
         return False
     if exp < datetime.now(timezone.utc).timestamp():

backend/app/downloads/og.py

@@ -66,6 +66,8 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
                     width = (data[3] << 8) + data[4]
                     return width, height
                 f.seek(length - 2, 1)  # skip this segment's payload
+    # silent-ok: None omits width/height from the unfurl card, which is what the card does for
+    # every poster whose dimensions were never asked for. Nothing is claimed that is not true.
     except OSError:
         return None
 

backend/app/downloads/pipeline.py

@@ -1,6 +1,5 @@
 """Child-side download/edit pipeline — the yt-dlp download + ffmpeg edit that runs in the spawned subprocess."""
 
-import logging
 import re
 import shutil
 import time
@@ -9,7 +8,7 @@ from pathlib import Path
 
 import yt_dlp
 
-from app import sysconfig
+from app import logs, sysconfig
 from app.config import settings
 from app.db import SessionLocal
 from app.downloads import edit as editmod
@@ -24,12 +23,13 @@ from app.downloads.jobdb import (
     _staging_dir,
     _touch_job,
 )
+from app.downloads.stated import StatedFailure
 from app.downloads.status import JobStatus
 from app.models import DownloadJob, MediaAsset
 from app.notifications import create_notification
 from app.titles import normalize_title
 
-log = logging.getLogger("siftlode.worker")
+log = logs.get_logger(__name__)
 
 
 class _Aborted(Exception):
@@ -98,6 +98,8 @@ def _parse_upload_date(raw) -> date | None:
         return None
     try:
         return datetime.strptime(str(raw), "%Y%m%d").date()
+    # silent-ok: a date we cannot read becomes a date the card does not show, which is what the
+    # card shows for every source that never supplies one. Absent is the honest rendering.
     except ValueError:
         return None
 
@@ -138,7 +140,14 @@ def _make_progress_hook(job_id: int, asset_id: int):
             try:
                 _fill_asset_meta_early(asset_id, d["info_dict"])
             except Exception:  # noqa: BLE001 — metadata is best-effort, never fail a download
-                pass
+                # Best-effort is about the DOWNLOAD, not about knowing. What this swallows is
+                # anything from a shape change in yt-dlp's info dict to the database being down,
+                # and all the user sees is a card with an id on it until the job finishes.
+                log.info(
+                    "early metadata could not be filled in for this download",
+                    extra=logs.event("download.meta.early_fill_failed", asset_id=asset_id),
+                    exc_info=True,
+                )
         if d.get("status") != "downloading":
             return
         if not allow_write():
@@ -299,7 +308,7 @@ def _produced_file(info: dict, staging: Path) -> Path:
         and not _YTDLP_TEMP_RE.search(p.name)
     ]
     if not cands:
-        raise RuntimeError("yt-dlp produced no output file")
+        raise StatedFailure("yt-dlp produced no output file")
     return max(cands, key=lambda p: p.stat().st_size)
 
 
@@ -375,13 +384,22 @@ def _extract_with_fallback(
                     "job %s: clients %s failed (%s); retrying with fallback",
                     job_id,
                     clients,
-                    str(exc)[:70],
+                    formats.log_safe(exc)[:70],
                 )
                 for p in list(staging.iterdir()):  # drop partials before the retry
                     try:
                         p.unlink()
-                    except OSError:
-                        pass
+                    except OSError as exc:
+                        # A partial left behind is not inert: the next client set resumes into it,
+                        # so the retry this loop exists to make clean starts dirty instead.
+                        log.info(
+                            "a partial file could not be removed before the retry",
+                            extra=logs.event(
+                                "download.retry.partial_unlink_failed",
+                                job_id=job_id,
+                                reason=type(exc).__name__,
+                            ),
+                        )
                 _set_job(job_id, progress=0, phase="downloading")
                 continue
             raise
@@ -464,7 +482,7 @@ def _download(
             try:
                 urlguard.check_url(url)
             except urlguard.BlockedUrl as e:
-                raise RuntimeError(f"Refusing to download from this address ({e.reason}).") from e
+                raise StatedFailure(f"Refusing to download from this address ({e.reason}).") from e
         # S6: attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
         # this asset is that user's private, salted identity (the double gate lives in credentials).
         remaining_bytes: int | None = None
@@ -586,13 +604,23 @@ def _download(
             db.commit()
             if finalized:
                 _notify_download_done(db, owner_id, meta.title, job_id, kind="download")
-                log.info("job %s done: %s", job_id, rel)
+                # The asset id, not `rel`: the path is built from the channel and the title.
+                log.info(
+                    "job %s done: asset %s",
+                    job_id,
+                    asset_id,
+                    extra=logs.event("download.job.done", job_id=job_id, asset_id=asset_id),
+                )
 
     except (_Aborted, editmod.EditAborted):
         # Paused/canceled mid-download or mid-transcode: free the asset for a later retry; keep the
         # job's status (the route set it to paused/canceled). ref_count is adjusted by the route.
         _reset_asset_pending(asset_id)
-        log.info("job %s aborted (pause/cancel)", job_id)
+        log.info(
+            "job %s aborted (pause/cancel)",
+            job_id,
+            extra=logs.event("download.job.aborted", job_id=job_id),
+        )
     except _OwnerRefused as exc:
         # Fail this job only. The asset is never marked `error`; it goes back to `pending` (below),
         # so another user's job for the same video is claimed and downloaded normally instead of
@@ -613,7 +641,12 @@ def _download(
         # it, and this one must already be out of the way. (The staging tree is still torn down in
         # `finally` below — the same narrow window the pause/cancel path above has always had.)
         _reset_asset_pending(asset_id)
-        log.info("job %s refused for its owner: %s", job_id, msg)
+        log.info(
+            "job %s refused for its owner: %s",
+            job_id,
+            formats.log_safe(msg),
+            extra=logs.event("download.job.refused", job_id=job_id),
+        )
     except Exception as exc:  # noqa: BLE001 — surface any failure to the user, keep worker alive
         msg = str(exc)[:500]
         # The catalog needs the link and whether the user's cookie went with it: the same yt-dlp
@@ -626,8 +659,9 @@ def _download(
             if asset:
                 asset.status = "error"
                 asset.error = msg
-            # Atomic fail: only while still running, so a route-set cancel/pause is never overwritten
-            # with `error` (error also holds the asset → would re-open the double-release class).
+            # Atomic fail: only while still running, so a route-set cancel/pause is never
+            # overwritten with `error` (error also holds the asset → would re-open the
+            # double-release class).
             service.finalize_if_running(
                 db,
                 job_id,
@@ -638,7 +672,10 @@ def _download(
                 phase=None,
             )
             db.commit()
-        log.warning("job %s failed: %s", job_id, msg)
+        # After the write: a job the write could not fail is not a failed job. When it raises,
+        # the runner reports this failure with it (`runner._run`) and the worker fails the job.
+        # silent-ok: not silent — `_log_download_failed` is the report.
+        _log_download_failed(job_id, msg, diagnosis.code, exc)
     finally:
         # Unlink the decrypted cookie file explicitly (the rmtree below removes the whole staging
         # tree too, but never leave a plaintext session cookie on disk a moment longer than needed).
@@ -647,7 +684,50 @@ def _download(
         shutil.rmtree(staging, ignore_errors=True)
 
 
-def _fail_edit(job_id: int, asset_id: int, msg: str) -> None:
+def _failure_stack(exc: BaseException) -> dict[str, str]:
+    """The `extra=` fragment that puts a failure's traceback on its line — or nothing.
+
+    Nothing for yt-dlp's DownloadError and for our own `StatedFailure` (`ffmpeg failed (1): …`):
+    their message is the whole story, and their stack is only the `except` that caught them.
+    Everything else keeps it — a TypeError on an odd info dict, a plain RuntimeError, an OSError,
+    a database error: that is our code or our platform failing, and the stack is the lead.
+
+    Scrubbed, link by link (`logs.scrubbed_stack`): a stack's messages are data — an OSError names
+    its file, and a database error raised inside `except DownloadError` carries yt-dlp's
+    unscrubbed text as its context."""
+    if isinstance(exc, (yt_dlp.utils.DownloadError, StatedFailure)):
+        return {}
+    return logs.scrubbed_stack(exc, formats.log_safe)
+
+
+def _log_download_failed(job_id: int, msg: str, code: str | None, exc: BaseException) -> None:
+    """The child's `download.job.failed` line. `msg` is the user's copy and keeps their URL; the
+    log's copy goes through `log_safe`, or it would put back one line later the URL and id the
+    yt-dlp adapter took out of the line before it. The stack only when the classifier could not
+    place the failure: a recognised one is fully described by its code."""
+    unplaced = code in (None, "", "unknown")
+    log.warning(
+        "job %s failed: %s",
+        job_id,
+        formats.log_safe(msg),
+        extra={
+            **logs.event(
+                "download.job.failed",
+                job_id=job_id,
+                error_code=code,
+                error_type=type(exc).__name__,
+            ),
+            **(_failure_stack(exc) if unplaced else {}),
+        },
+    )
+
+
+def _fail_edit(job_id: int, asset_id: int, msg: str, exc: BaseException | None = None) -> None:
+    """Fail an edit job, and say so ONCE — every way an edit fails comes through here, so this is
+    where its line and its event code live. `exc` is the exception when there was one.
+
+    Its traceback goes with the line unless the message already tells it all (`_failure_stack`).
+    Logged after the write, as on the download error path."""
     with SessionLocal() as db:
         asset = db.get(MediaAsset, asset_id)
         if asset:
@@ -663,7 +743,22 @@ def _fail_edit(job_id: int, asset_id: int, msg: str) -> None:
             phase=None,
         )
         db.commit()
-    log.warning("edit job %s failed: %s", job_id, msg)
+    # `msg` is the user's copy; ffmpeg's stderr tail in it can name the source file, whose path is
+    # built from the channel and the title.
+    log.warning(
+        "edit job %s failed: %s",
+        job_id,
+        formats.log_safe(msg),
+        extra={
+            **logs.event(
+                "edit.job.failed",
+                job_id=job_id,
+                asset_id=asset_id,
+                **({"error_type": type(exc).__name__} if exc is not None else {}),
+            ),
+            **(_failure_stack(exc) if exc is not None else {}),
+        },
+    )
 
 
 def _process_edit(job_id: int, asset_id: int) -> None:
@@ -715,7 +810,7 @@ def _process_edit(job_id: int, asset_id: int) -> None:
             should_cancel=_cancel_poll(job_id),
         )
         if not dest_staging.exists():
-            raise RuntimeError("ffmpeg produced no output file")
+            raise StatedFailure("ffmpeg produced no output file")
 
         rel = storage.edit_rel_path(user_id, asset_id, out_ext)
         storage.place_file(dest_staging, settings.download_root, rel)
@@ -751,13 +846,24 @@ def _process_edit(job_id: int, asset_id: int) -> None:
             db.commit()
             if finalized:
                 _notify_download_done(db, owner_id, dl_title, job_id, kind="edit")
-                log.info("edit job %s done: %s", job_id, rel)
+                # The asset id, not `rel`: the path is built from the channel and the title.
+                log.info(
+                    "edit job %s done: asset %s",
+                    job_id,
+                    asset_id,
+                    extra=logs.event("edit.job.done", job_id=job_id, asset_id=asset_id),
+                )
 
     except editmod.EditAborted:
         # Paused/canceled mid-encode: free the asset for a later retry (ref_count is the route's job).
         _reset_asset_pending(asset_id)
-        log.info("edit job %s aborted (pause/cancel)", job_id)
+        log.info(
+            "edit job %s aborted (pause/cancel)",
+            job_id,
+            extra=logs.event("edit.job.aborted", job_id=job_id),
+        )
     except Exception as exc:  # noqa: BLE001 — surface the failure, keep the worker alive
-        _fail_edit(job_id, asset_id, str(exc)[:500])
+        # silent-ok: not silent — `_fail_edit` is the report, and it takes `exc` for its stack.
+        _fail_edit(job_id, asset_id, str(exc)[:500], exc)
     finally:
         shutil.rmtree(staging, ignore_errors=True)

backend/app/downloads/probe.py

@@ -64,6 +64,8 @@ def _facts_from_probe(
         raw = fmt.get("duration")
         if raw is not None:
             duration = int(float(raw))
+    # silent-ok: an unusable duration is stored as no duration, which the UI renders as unknown
+    # rather than as zero — the honest state, and the one a re-probe can still fill in.
     except (TypeError, ValueError):
         duration = None
     return vcodec, acodec, width, height, duration

backend/app/downloads/runner.py

@@ -11,18 +11,20 @@ writes the terminal job/asset state itself (done / error, or a cooperative pause
 progress hook), then exits 0. An unexpected crash exits non-zero and the parent fails the job.
 """
 
-import logging
 import os
 import shutil
 import subprocess
 import sys
 
+from app import logs
+from app import logsink
 from app.config import settings
 from app.db import SessionLocal
 from app.downloads import edit as editmod
+from app.downloads import formats
 from app.models import DownloadJob
 
-log = logging.getLogger("siftlode.worker.runner")
+log = logs.get_logger(__name__)
 
 
 def _deprioritize() -> None:
@@ -52,12 +54,21 @@ def _deprioritize() -> None:
 
 
 def main() -> None:
-    logging.basicConfig(level=logging.INFO)
+    logs.configure_logging("runner")
     _deprioritize()
     if len(sys.argv) != 2:
-        print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
+        # noqa T201: a usage line is for the person who typed the command. Logging is already
+        # configured here, and in a deployment it writes JSON records — the one reader of this
+        # line wants the bare sentence, not a record with a request id they do not have.
+        print(  # noqa: T201
+            "usage: python -m app.downloads.runner <job_id>", file=sys.stderr
+        )
         raise SystemExit(2)
     job_id = int(sys.argv[1])
+    # This whole process exists to run ONE job — tag every line it prints with it, so the child's
+    # output joins up with the worker's and the API's view of the same download.
+    logs.set_job(job_id)
+    logsink.start()  # flushed at exit (atexit) — this process has no shutdown hook of its own
 
     with SessionLocal() as db:
         job = db.get(DownloadJob, job_id)
@@ -71,14 +82,40 @@ def main() -> None:
     if asset_id is None:
         return
 
+    _run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id)
+
+
+def _run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id) -> None:
+    """Run the job, and be the last stop before Python's own exception hook.
+
+    What gets past the pipeline is a failure it could not even record (the database refused the
+    write that fails the job). Python's hook would print it to stderr raw — and its context is the
+    download's own failure, whose text is the URL or ffmpeg's stderr naming the file. Here it is
+    one line with the whole chain scrubbed; the non-zero exit then has the worker fail the job
+    (`cause=child_exit`)."""
     # Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
     # real work, and to keep the import graph obvious: runner is a thin entry over the pipeline.
     from app.downloads.pipeline import _download, _process_edit
 
-    if job_kind == "edit":
-        _process_edit(job_id, asset_id)
-    else:
-        _download(job_id, asset_id, source_kind, source_ref, spec, user_id)
+    with formats.hiding(source_ref):
+        try:
+            if job_kind == "edit":
+                _process_edit(job_id, asset_id)
+            else:
+                _download(job_id, asset_id, source_kind, source_ref, spec, user_id)
+        except Exception as exc:  # noqa: BLE001 — whatever it is, it is reported here or raw
+            # noqa TRY400: the stack goes with it scrubbed (`scrubbed_stack`); `exc_info` is raw.
+            log.error(  # noqa: TRY400
+                "the download child for job %s failed",
+                job_id,
+                extra={
+                    **logs.event(
+                        "download.child.failed", job_id=job_id, error_type=type(exc).__name__
+                    ),
+                    **logs.scrubbed_stack(exc, formats.log_safe),
+                },
+            )
+            raise SystemExit(1) from None
 
 
 if __name__ == "__main__":

backend/app/downloads/service.py

@@ -87,6 +87,8 @@ def get_or_create_asset(
         with db.begin_nested():
             db.add(asset)
             db.flush()
+    # silent-ok: the expected outcome of the race this savepoint exists for — two jobs wanting the
+    # same asset. Losing it means the other insert won, and re-selecting is the whole handling.
     except IntegrityError:
         asset = db.execute(stmt).scalar_one()
     return asset

backend/app/downloads/stated.py

@@ -0,0 +1,13 @@
+"""The failures the download pipeline raises itself, with the whole story in the message.
+
+`ffmpeg failed (1): …`, `yt-dlp produced no output file`, a post that shows no video: the message
+says what happened, and the stack is only the `except` that caught it. A failure line leaves that
+stack off (`pipeline._failure_stack`) — but ONLY for these. A plain `RuntimeError` is also what
+Python raises for a bug of ours (a dict changed size during iteration, a generator's
+StopIteration, `RecursionError`, `NotImplementedError`), and that one's stack is the only lead.
+"""
+
+
+class StatedFailure(RuntimeError):
+    """A failure we raised on purpose, whose message is its full explanation. A `RuntimeError` so
+    every `except RuntimeError` that caught these before still does."""

backend/app/downloads/storage.py

@@ -19,8 +19,11 @@ from datetime import date
 from pathlib import Path
 from xml.sax.saxutils import escape
 
+from app import logs
 from app.downloads import mediakind
 
+log = logs.get_logger(__name__)
+
 # Reserve room for the "_id.ext" suffix + parent dirs within the 255-char component limit.
 _MAX_TITLE = 120
 # User-facing (Content-Disposition) names can be longer than an on-disk component.
@@ -294,7 +297,13 @@ def write_sidecars(download_root: str, rel: str, meta: MediaMeta, thumb_src: Pat
         if thumb_src and thumb_src.exists():
             shutil.copyfile(str(thumb_src), str(_sidecar(media, ".jpg")))
         return True
-    except OSError:
+    except OSError as exc:
+        # The download itself still completes, so the job reads as done while the library it was
+        # downloaded FOR has no .nfo to read and no poster to show.
+        log.warning(
+            "the sidecars beside a download could not be written",
+            extra=logs.event("download.sidecars.write_failed", reason=type(exc).__name__),
+        )
         return False
 
 
@@ -304,14 +313,23 @@ def delete_asset_files(download_root: str, rel: str) -> None:
     for p in (media, _sidecar(media, ".nfo"), _sidecar(media, ".jpg")):
         try:
             p.unlink()
-        except OSError:
-            pass
+        except FileNotFoundError:
+            pass  # silent-ok: already gone is the outcome this function wanted
+        except OSError as exc:
+            # The row goes and the bytes stay. Nothing else ever revisits this path, so the disk
+            # quietly keeps a file no account has any way to see, let alone delete.
+            log.info(
+                "a deleted download left a file behind",
+                extra=logs.event("download.asset.file_unlink_failed", reason=type(exc).__name__),
+            )
     # Prune empty Season/Channel dirs up to (but not including) the root.
     root = Path(download_root).resolve()
     parent = media.parent
     while parent != root and root in parent.parents:
         try:
             parent.rmdir()  # only succeeds if empty
+        # silent-ok: this `rmdir` failing is the loop's stop condition — the parent still
+        # holds another download's files, so the prune has gone as far up as it may.
         except OSError:
             break
         parent = parent.parent

backend/app/email.py

@@ -11,7 +11,6 @@ to be a person telling them nothing arrived.
 """
 
 import html
-import logging
 import re
 import smtplib
 import ssl
@@ -21,15 +20,14 @@ from email.utils import formatdate
 from sqlalchemy import select
 from sqlalchemy.orm import Session
 
-from app import mailtext, sysconfig
+from app import logs, mailtext, sysconfig
 from app.config import settings
 from app.db import SessionLocal
 from app.models import User
 from app.notifications import create_notification
 from app.ratelimit import RateLimiter
 
-log = logging.getLogger("siftlode.email")
-
+log = logs.get_logger(__name__)
 # One admin notice an hour per reason. A relay that is down fails every send, and the inbox must
 # stay readable; the log keeps the full record either way.
 _alert_throttle = RateLimiter(max_events=1, window_seconds=3600)
@@ -155,7 +153,15 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
         return False
     c = _smtp()
     if not (c["host"] and c["user"] and c["password"]):
-        log.info("SMTP not configured; skipping email %r to %s", subject, recipients)
+        # The count, not the addresses: `redact_text` rewrites every one of them to `[email]`
+        # before either sink sees the line, so the list only ever printed as noise. The subject
+        # stays — the templates are static, and the one that interpolates (`admin_new_request`)
+        # interpolates an address, which the same redactor covers.
+        log.info(
+            "SMTP not configured; skipping email %r to %d recipient(s)",
+            subject,
+            len(recipients),  # log-pii-ok: how many, not who
+        )
         _alert_admins("not_configured", subject, recipients)
         return False
     msg = EmailMessage()
@@ -176,10 +182,12 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
             s.starttls(context=ssl.create_default_context())
             s.login(c["user"], c["password"])
             s.send_message(msg)
-        log.info("Sent email %r to %s", subject, recipients)
+        # log-pii-ok: how many, not who
+        log.info("Sent email %r to %d recipient(s)", subject, len(recipients))
         return True
     except Exception:
-        log.exception("SMTP send failed (%r to %s)", subject, recipients)
+        # log-pii-ok: how many, not who
+        log.exception("SMTP send failed (%r to %d recipient(s))", subject, len(recipients))
         _alert_admins("send_failed", subject, recipients)
         return False
 

backend/app/logs.py

@@ -0,0 +1,774 @@
+"""Structured logging for every Siftlode process — the ONE place logging is configured.
+
+Before this module a single deployment printed FOUR different log formats: uvicorn's
+`log_config.json` for the API, a hand-rolled StreamHandler in `main.py` for everything that did not
+start under uvicorn (tests, scripts), `logging.basicConfig()` in the worker and in the download
+child (no timestamp at all), and alembic's `fileConfig`. None of them carried a correlation id, and
+uvicorn's access log printed the query string of every request (`?account=1`, `?source=feed`) — so
+the logs were both unjoinable and leaky. `configure_logging()` replaces all four.
+
+Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
+    ts · level · sev · source · logger · event · msg · attrs
+plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
+`job_id`) and the app version. That shape is what E22.6's admin console filters on and what a Gitea
+issue attachment will carry, so it is deliberately flat and stable.
+
+Call sites stay ordinary stdlib logging:
+
+    log = logs.get_logger(__name__)  # app/downloads/pipeline.py -> siftlode.downloads.pipeline
+    log.info("job finished", extra=logs.event("download.job.done", job_id=7, bytes=n))
+
+`event()` is optional — a bare `log.info("…")` still comes out well-formed — but the dotted code is
+what makes a line queryable later, so add one wherever the line matters.
+
+REDACTION IS APPLIED AT THE HANDLER, not at the call site: every message, every attr value and
+every traceback passes `redact()` on the way out. That is the whole point — a secret cannot be
+leaked by forgetting to redact it at one of ~125 call sites, and a new call site is safe by
+default. Never logged, at any level: passwords, tokens (session, reset/verify, setup, share,
+OAuth), cookie blobs, secret config values, and email addresses.
+"""
+
+import contextlib
+import contextvars
+import json
+import logging
+import logging.config
+import re
+import sys
+import threading
+import time
+import traceback
+import uuid
+from collections.abc import Callable
+from datetime import datetime, timezone
+from typing import Any
+from urllib.parse import parse_qsl
+
+from app.config import settings
+from app.ratelimit import RateLimiter, client_key
+
+# One level below DEBUG for the "Teljes / TRACE" diagnostics level (search strings, redacted request
+# bodies). Registered on import so `logging.getLogger(...).log(logs.TRACE, …)` and a `LOG_LEVEL=TRACE`
+# both work in every process, including one that never calls configure_logging (a test).
+TRACE = 5
+logging.addLevelName(TRACE, "TRACE")
+
+# OpenTelemetry severity numbers for our level names. Kept beside the level because the number is
+# what a log backend sorts and filters on; the name is for humans reading a terminal.
+SEVERITY: dict[str, int] = {
+    "TRACE": 1,
+    "DEBUG": 5,
+    "INFO": 9,
+    "WARNING": 13,
+    "ERROR": 17,
+    "CRITICAL": 21,
+}
+
+# Which process this is. Set by configure_logging; every line carries it so one Loki/DB query can
+# separate the API's view of a job from the worker's.
+_source = "api"
+
+
+# --- correlation context -------------------------------------------------------------------
+# contextvars, like quota's actor attribution (app/quota.py): set at an entry point, read by the
+# formatter, never threaded through call signatures. A child task copies the context at creation,
+# so anything set before `await call_next(...)` is visible to the whole request.
+_request_id: contextvars.ContextVar[str | None] = contextvars.ContextVar(
+    "log_request_id", default=None
+)
+_user_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_user_id", default=None)
+_debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
+    "log_debug_session", default=None
+)
+_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
+
+_CONTEXT_VARS = {
+    "request_id": _request_id,
+    "user_id": _user_id,
+    "debug_session_id": _debug_session,
+    "job_id": _job_id,
+}
+
+
+class _RequestScope:
+    """What the auth dependency learns about a request, for every line logged after it.
+
+    A MUTABLE object behind a contextvar, not plain contextvars, because of how a request actually
+    runs: FastAPI resolves a sync dependency (`auth.current_user`) in a threadpool on a COPY of the
+    context, so a `ContextVar.set()` there is gone by the time the handler runs — measured: that is
+    how the account never reached a handler's own lines. `log_requests` installs one scope per
+    request; every copied context shares the same object, so what the dependency writes is seen by
+    the handler and by the request line itself."""
+
+    __slots__ = ("user_id", "level")
+
+    def __init__(self) -> None:
+        self.user_id: int | None = None
+        # The user's consented diagnostics level as a logging number; None at the base level.
+        self.level: int | None = None
+
+
+_scope: contextvars.ContextVar[_RequestScope | None] = contextvars.ContextVar(
+    "log_request_scope", default=None
+)
+
+
+def set_user(user_id: int | None, level: str = "INFO", until: datetime | None = None) -> None:
+    """Attribute the rest of this request's log lines to `user_id`, and log them at the user's
+    consented diagnostics `level` until `until` (E22 S5; "INFO" = no elevation). Called from the
+    auth dependency — the one place the session is actually resolved into an account — rather than
+    from the request middleware, which runs OUTSIDE the session middleware."""
+    _user_id.set(user_id)
+    scope = _scope.get()
+    if scope is None:  # outside log_requests (a script, a unit test): this context only
+        scope = _RequestScope()
+        _scope.set(scope)
+    elevated = level != "INFO" and until is not None and until > datetime.now(timezone.utc)
+    scope.user_id = user_id
+    scope.level = resolve_level(level) if elevated else None
+    if user_id is not None:
+        _track_elevation(user_id, scope.level, until)
+
+
+def set_job(job_id: int | None) -> None:
+    """Attribute this process's log lines to a download job — for the download CHILD, whose whole
+    life is one job (the worker uses `bind()` per job instead, since it juggles several)."""
+    _job_id.set(job_id)
+
+
+@contextlib.contextmanager
+def bind(**fields: Any):
+    """Bind correlation fields for the duration of a block (a worker job, a scheduler run):
+
+        with logs.bind(job_id=job.id):
+            ...
+
+    Unknown field names are ignored rather than raising — a caller binding a field this module does
+    not know about is a typo, not a reason to take down a download."""
+    tokens = [(_CONTEXT_VARS[k].set(v)) for k, v in fields.items() if k in _CONTEXT_VARS]
+    keys = [k for k in fields if k in _CONTEXT_VARS]
+    try:
+        yield
+    finally:
+        for key, token in zip(keys, tokens):
+            _CONTEXT_VARS[key].reset(token)
+
+
+def event(code: str, /, **attrs: Any) -> dict[str, Any]:
+    """`extra=` payload for a log call: a stable dotted event code plus structured attributes.
+
+    Returns a plain dict so it composes with stdlib logging and costs nothing when the level is off
+    (the dict is built either way; keep the attrs cheap — no DB reads inside the call).
+
+    `code` is positional-only so that no attribute name is reserved: `event("x", code=...)` means
+    the attribute, not this parameter. Without the `/` it was a TypeError, and `code` is exactly
+    the word a caller reaches for when the thing that failed has one of its own."""
+    return {"event": code, "attrs": attrs}
+
+
+# --- redaction -----------------------------------------------------------------------------
+# Substring matching, not exact keys: the same secret travels as `password`, `current_password`,
+# `new_password`, `smtp_password`, `plex_server_token`, `access_token`, … Matching on the PART is
+# what makes a key we have never seen safe by default.
+_SECRET_KEY_PARTS = (
+    "password",
+    "passphrase",
+    "token",
+    "secret",
+    "cookie",
+    "authorization",
+    "auth_header",
+    "ciphertext",
+    "private_key",
+    "api_key",
+    "apikey",
+    "credential",
+    "session_key",
+)
+
+# Query/fragment parameters that carry a credential in a URL we log (OAuth `code`/`state`, our own
+# `?token=` setup and share links, `#verify=` / `#reset=` fragments).
+_SECRET_PARAM_NAMES = (
+    "access_token",
+    "refresh_token",
+    "id_token",
+    "token",
+    "code",
+    "state",
+    "key",
+    "secret",
+    "password",
+    "pwd",
+    "sig",
+    "signature",
+    "verify",
+    "reset",
+    "setup",
+)
+_SECRET_PARAM_RE = re.compile(
+    r"\b((?:" + "|".join(_SECRET_PARAM_NAMES) + r")=)([^&\s\"']+)",
+    re.IGNORECASE,
+)
+
+# Deliberately loose: over-matching redacts a non-address that looks like one (harmless), while
+# under-matching puts a real address in a log we attach to an issue (not harmless).
+_EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
+
+REDACTED = "[redacted]"
+# `extra={PERSIST: False}` keeps a line out of the `log_events` table while stdout still gets it —
+# how `log_requests` caps anonymous traffic per client (app/logsink.py reads it).
+PERSIST = "persist"
+REDACTED_EMAIL = "[email]"
+
+
+def redact_text(text: str) -> str:
+    """Scrub a free-text string: credential-bearing URL params and email addresses."""
+    scrubbed = _SECRET_PARAM_RE.sub(lambda m: f"{m.group(1)}{REDACTED}", text)
+    return _EMAIL_RE.sub(REDACTED_EMAIL, scrubbed)
+
+
+def redact_query(query: str) -> str:
+    """Scrub a raw query string. DECODED first: the text rules match what was typed, and a browser
+    sends it percent-encoded — `someone%40example.com` is an address `_EMAIL_RE` cannot see. Each
+    parameter also goes through the key rules, so `?new_password=` goes as `redact` would drop it."""
+    pairs = parse_qsl(query, keep_blank_values=True)
+    return "&".join(f"{key}={_redact_param(key, value)}" for key, value in pairs)
+
+
+def _redact_param(key: str, value: str) -> str:
+    """One decoded `key=value` pair. The parameter-NAME rule is applied to the key here rather than
+    left to `_SECRET_PARAM_RE` over the rebuilt pair: that pattern was written for the raw,
+    percent-encoded query, where a value holds no literal space, `&` or quote, and its value class
+    stops at the first one. On decoded text `?state=a%20b` would keep its tail and `?code=%20abc`
+    would not match at all — the whole credential straight into the line."""
+    if key.lower() in _SECRET_PARAM_NAMES:
+        return REDACTED
+    return str(redact(value, key=key))
+
+
+def redact(value: Any, *, key: str = "") -> Any:
+    """Recursively scrub a structure destined for the log.
+
+    A value whose KEY names a secret is dropped whole (we never log a prefix of a token — a prefix
+    is still a leak and still useless). Everything else is scrubbed as text. Depth is bounded by
+    the structures we actually log (attrs dicts), and non-JSON types fall through to `str`."""
+    lowered = key.lower()
+    if key and any(part in lowered for part in _SECRET_KEY_PARTS):
+        return REDACTED
+    if isinstance(value, dict):
+        return {k: redact(v, key=str(k)) for k, v in value.items()}
+    if isinstance(value, (list, tuple)):
+        return [redact(v, key=key) for v in value]
+    if isinstance(value, str):
+        return redact_text(value)
+    return value
+
+
+# --- formatters ----------------------------------------------------------------------------
+_SCRUBBED_STACK = "scrubbed_stack"
+
+
+def scrubbed_stack(exc: BaseException, scrub: Callable[[BaseException], str]) -> dict[str, str]:
+    """`extra=` fragment: `exc`'s traceback, rendered where an `exc_info` stack goes, with the
+    message of every exception in it put through `scrub`.
+
+    For a failure whose message can name someone's data (a download's URL, a file path, a video
+    id). The frames name code and are kept whole; the text at the foot of each link is the data.
+    A plain `exc_info` renders the chain too — the `__cause__`/`__context__` a DB error raised
+    inside `except DownloadError` carries — so scrubbing only the line's own message is not
+    enough. Merge it into the call's `extra`: `{**event(...), **scrubbed_stack(...)}`."""
+    links: list[BaseException] = []
+    link: BaseException | None = exc
+    while link is not None and all(link is not seen for seen in links):
+        links.append(link)
+        link = link.__cause__ or (None if link.__suppress_context__ else link.__context__)
+    parts: list[str] = []
+    for i in range(len(links) - 1, -1, -1):  # the oldest first, as Python prints a chain
+        link = links[i]
+        kind = type(link)
+        name = kind.__qualname__
+        if kind.__module__ not in ("builtins", "__main__"):
+            name = f"{kind.__module__}.{name}"
+        try:
+            message = scrub(link)
+        # silent-ok: the message is what could not be rendered, and the placeholder says so — the
+        # caller is in the middle of reporting one failure and must not get a second from here.
+        except Exception:  # noqa: BLE001 — a `__str__` that raises, of any shape
+            message = "<unprintable>"
+        frames = "".join(traceback.format_tb(link.__traceback__))
+        head = f"Traceback (most recent call last):\n{frames}" if frames else ""
+        parts.append(f"{head}{name}: {message}" if message else f"{head}{name}")
+        if i:
+            parts.append(
+                "\nThe above exception was the direct cause of the following exception:\n"
+                if links[i - 1].__cause__ is link
+                else "\nDuring handling of the above exception, another exception occurred:\n"
+            )
+    return {_SCRUBBED_STACK: "\n".join(parts)}
+
+
+# Attributes every LogRecord carries; anything else a caller passed via `extra=` is a structured
+# attribute and is merged into `attrs` (so `extra={"job_id": 7}` works as well as `event(...)`).
+_RECORD_STANDARD = frozenset(
+    vars(logging.LogRecord("", 0, "", 0, "", None, None)).keys()
+    | {"message", "asctime", "taskName", "event", "attrs", PERSIST, "elevated", _SCRUBBED_STACK}
+)
+
+
+class ContextFilter(logging.Filter):
+    """Copies the correlation contextvars onto the record. A Filter (not the formatter) so the
+    values are captured at emit time, before a QueueHandler can move the record to another
+    thread — the DB sink in S3 depends on that."""
+
+    def filter(self, record: logging.LogRecord) -> bool:
+        for name, var in _CONTEXT_VARS.items():
+            value = var.get()
+            if value is not None and not hasattr(record, name):
+                setattr(record, name, value)
+        scope = _scope.get()
+        if scope is not None and scope.user_id is not None and not hasattr(record, "user_id"):
+            record.user_id = scope.user_id
+        return True
+
+
+class LevelGate(logging.Filter):
+    """The configured level, enforced at the HANDLER, with one exception: the lines of a request
+    whose user raised their diagnostics level (E22 S5).
+
+    The root logger's level cannot do this alone: it is one number for the whole process, and the
+    point is to log ONE user's requests in detail while everyone else's stay at the base level. So
+    the root is lowered to the lowest level any active elevation needs (`_track_elevation`), and
+    this gate drops what only an elevation let through unless the line belongs to that elevated
+    request. It also marks such a line `elevated`, which is what makes the `log_events` sink keep
+    every line of that request, not just its warnings."""
+
+    def filter(self, record: logging.LogRecord) -> bool:
+        scope = _scope.get()
+        if scope is not None and scope.level is not None:
+            record.elevated = True
+            return record.levelno >= min(scope.level, _base_level)
+        return record.levelno >= _base_level
+
+
+def _record_attrs(record: logging.LogRecord) -> dict[str, Any]:
+    extra = {k: v for k, v in vars(record).items() if k not in _RECORD_STANDARD}
+    attrs = dict(getattr(record, "attrs", None) or {})
+    attrs.update({k: v for k, v in extra.items() if k not in _CONTEXT_VARS})
+    return attrs
+
+
+def _safe_message(record: logging.LogRecord) -> str:
+    """`record.getMessage()` raises if the args don't match the format string. A broken log call
+    must not take down the caller, so the failure is reported IN the line instead."""
+    try:
+        return record.getMessage()
+    # silent-ok: the failure is reported IN the returned line. Logging it would re-enter this same
+    # formatter with a record built out of the one that just failed.
+    except Exception as exc:  # noqa: BLE001 — any formatting error, reported not raised
+        return f"{record.msg!r} (log formatting failed: {exc})"
+
+
+def record_fields(record: logging.LogRecord) -> dict[str, Any]:
+    """The event as structured fields, REDACTED — the one definition both the stdout line and the
+    `log_events` row are built from, so the two sinks can never disagree about what is safe to keep.
+    Absent optional fields are left out; `ts` is an aware datetime."""
+    fields: dict[str, Any] = {
+        "ts": datetime.fromtimestamp(record.created, timezone.utc),
+        "level": record.levelname,
+        "sev": SEVERITY.get(record.levelname, 9),
+        "source": _source,
+        "logger": record.name,
+        "msg": redact_text(_safe_message(record)),
+    }
+    code = getattr(record, "event", None)
+    if code:
+        fields["event"] = code
+    for name in _CONTEXT_VARS:
+        value = getattr(record, name, None)
+        if value is not None:
+            fields[name] = value
+    attrs = _record_attrs(record)
+    if attrs:
+        fields["attrs"] = redact(attrs)
+    if record.exc_info:
+        fields["exc"] = redact_text(logging.Formatter().formatException(record.exc_info))
+    elif getattr(record, _SCRUBBED_STACK, None):
+        fields["exc"] = redact_text(getattr(record, _SCRUBBED_STACK))
+    elif record.exc_text:
+        # A QueueHandler-style hand-off formats the traceback into exc_text and drops exc_info.
+        fields["exc"] = redact_text(record.exc_text)
+    fields["version"] = settings.app_version
+    return fields
+
+
+class JsonFormatter(logging.Formatter):
+    """One JSON object per line — the shape Loki and the `log_events` table both ingest."""
+
+    def format(self, record: logging.LogRecord) -> str:
+        payload = record_fields(record)
+        payload["ts"] = payload["ts"].isoformat(timespec="milliseconds")
+        return json.dumps(payload, ensure_ascii=False, default=str)
+
+
+class TextFormatter(logging.Formatter):
+    """Human-readable single line for a terminal (`LOG_FORMAT=text`): the same content, ordered for
+    reading. Redaction applies here too — a dev log is still a log someone pastes into an issue."""
+
+    def format(self, record: logging.LogRecord) -> str:
+        stamp = datetime.fromtimestamp(record.created, timezone.utc).strftime("%H:%M:%S.%f")[:-3]
+        head = f"{stamp} {record.levelname:<5} [{record.name}]"
+        rid = getattr(record, "request_id", None)
+        if rid:
+            head += f" {rid}"
+        code = getattr(record, "event", None)
+        line = f"{head} {code + ' — ' if code else ''}{redact_text(_safe_message(record))}"
+        attrs = _record_attrs(record)
+        if attrs:
+            line += f" {json.dumps(redact(attrs), ensure_ascii=False, default=str)}"
+        if record.exc_info:
+            line += "\n" + redact_text(self.formatException(record.exc_info))
+        elif getattr(record, _SCRUBBED_STACK, None):
+            line += "\n" + redact_text(getattr(record, _SCRUBBED_STACK))
+        return line
+
+
+# --- configuration -------------------------------------------------------------------------
+# Third-party loggers that are noisy at our own level. Pinned explicitly so the diagnostics knob can
+# lower the ROOT level to DEBUG/TRACE without drowning the output in SQL statements and HTTP chatter
+# — raising THESE is a deliberate, separate act (E22.6's console).
+_THIRD_PARTY_LEVELS = {
+    "sqlalchemy": "WARNING",
+    "sqlalchemy.engine": "WARNING",
+    "apscheduler": "WARNING",
+    "urllib3": "WARNING",
+    "httpx": "WARNING",
+    "httpcore": "WARNING",
+    "asyncio": "WARNING",
+    "multipart": "WARNING",
+    "PIL": "WARNING",
+    "alembic": "INFO",
+    # uvicorn's own startup/shutdown/WebSocket lines are worth keeping; its ACCESS log is not —
+    # it is one line per request with the query string in it, replaced by `log_requests` below.
+    "uvicorn": "INFO",
+    "uvicorn.error": "INFO",
+    "uvicorn.access": "CRITICAL",
+}
+
+
+def get_logger(name: str) -> logging.Logger:
+    """The logger for one module: `logs.get_logger(__name__)` in `app/plex/sync.py` is
+    `siftlode.plex.sync`.
+
+    The names were literals before, and 25 of them covered 34 modules — six different Plex modules
+    all called themselves `siftlode.plex`, so neither a level pinned per logger nor E22.6's console
+    filter could tell a sync line from a stream line. Derived from `__name__` there is exactly one
+    name per module, it cannot drift from the file it sits in, and `test_logger_names.py` fails if a
+    module hand-rolls one again.
+
+    `__main__` is resolved to the module it really is. The worker runs as `python -m app.worker`
+    and spawns `python -m app.downloads.runner`, so in the two long-running processes `__name__`
+    is `__main__` — taken at its word, both would log as the bare root. `-m` records the real name
+    on the module's spec; a plain script has none and stays the root."""
+    if name == "__main__":
+        spec = getattr(sys.modules.get("__main__"), "__spec__", None)
+        name = getattr(spec, "name", None) or name
+    stem = name.removeprefix("app.") if name.startswith("app.") else ""
+    # logger-name-ok: this call IS the derivation
+    return logging.getLogger(f"siftlode.{stem}" if stem else "siftlode")
+
+
+def resolve_level(name: str) -> int:
+    """Level name → number. Both ways in (the `LOG_LEVEL` env and the `log_level` admin knob) are
+    already held to the `LogLevel` choices, so an unknown name here is a programming error."""
+    value = logging.getLevelName(name)
+    if not isinstance(value, int):
+        raise ValueError(f"unknown log level {name!r}")
+    return value
+
+
+def configure_logging(source: str) -> None:
+    """Configure logging for THIS process. Call once, first thing, in every entry point:
+    `main.py` (api), `worker.main`, `downloads/runner.main`, `alembic/env.py`.
+
+    `source` names the process in every line. The level starts at `LOG_LEVEL` (the bootstrap default);
+    the runtime admin knob goes through `set_level()` instead, so changing it needs no restart.
+
+    Our loggers deliberately keep `propagate=True` and own no handler: everything reaches the single
+    root StreamHandler. That is what keeps `caplog` working in tests, and it means a new logger name
+    is configured the moment it is created."""
+    global _source
+    _source = source
+    formatter = settings.log_format
+    logging.config.dictConfig(
+        {
+            "version": 1,
+            # Never disable loggers created at import time (module-level `getLogger` calls run
+            # before this function does).
+            "disable_existing_loggers": False,
+            "formatters": {
+                "json": {"()": f"{__name__}.JsonFormatter"},
+                "text": {"()": f"{__name__}.TextFormatter"},
+            },
+            "filters": {
+                "context": {"()": f"{__name__}.ContextFilter"},
+                "gate": {"()": f"{__name__}.LevelGate"},
+            },
+            "handlers": {
+                "stdout": {
+                    "class": "logging.StreamHandler",
+                    "stream": sys.stdout,
+                    "formatter": formatter,
+                    "filters": ["context", "gate"],
+                }
+            },
+            "root": {"level": resolve_level(settings.log_level), "handlers": ["stdout"]},
+            "loggers": {name: {"level": lvl} for name, lvl in _THIRD_PARTY_LEVELS.items()},
+        }
+    )
+    set_level(settings.log_level)
+
+
+def set_level(name: str) -> None:
+    """Change the running process's level (the `log_level` admin knob). Third-party loggers keep
+    their pinned levels, so turning on DEBUG shows OUR detail, not SQLAlchemy's."""
+    global _base_level
+    _base_level = resolve_level(name)
+    _apply_root_level(force=True)
+
+
+def current_level() -> str:
+    """The configured level — not the root logger's, which an active elevation may have lowered."""
+    return logging.getLevelName(_base_level or logging.getLogger().getEffectiveLevel())
+
+
+# --- per-user elevation (E22 S5) -----------------------------------------------------------
+# The base level (`LOG_LEVEL`, then the admin knob). NOTSET until configure_logging runs — a unit
+# test, a script — so the gate lets everything through there, as it did before the gate existed.
+_base_level = logging.NOTSET
+# user id → (level number, expiry) for every elevation this process has seen and not yet expired.
+_elevations: dict[int, tuple[int, datetime]] = {}
+_elevations_lock = threading.Lock()
+_root_applied: int | None = None
+
+
+def _track_elevation(user_id: int, level: int | None, until: datetime | None) -> None:
+    with _elevations_lock:
+        if level is None or until is None:
+            _elevations.pop(user_id, None)
+        else:
+            _elevations[user_id] = (level, until)
+        active = bool(_elevations)
+    if active or _root_applied != _base_level:
+        _apply_root_level()
+
+
+def _apply_root_level(force: bool = False) -> None:
+    """Root level = the lowest of the base and every unexpired elevation (expired ones are pruned
+    here). `setLevel` clears every logger's level cache, so it runs only when the number changes."""
+    global _root_applied
+    if _base_level == logging.NOTSET:
+        return  # not configured: the root keeps whatever the process set up
+    now = datetime.now(timezone.utc)
+    # The level is SET under the lock too, not just computed: two threads that each computed a
+    # different answer would otherwise apply them in either order, and the older one could win —
+    # the root back at the base while an elevation is still active.
+    with _elevations_lock:
+        for user_id in [u for u, (_, until) in _elevations.items() if until <= now]:
+            del _elevations[user_id]
+        lowest = min([_base_level, *(lvl for lvl, _ in _elevations.values())])
+        if force or lowest != _root_applied:
+            logging.getLogger().setLevel(lowest)
+            _root_applied = lowest
+
+
+# --- erased accounts ------------------------------------------------------------------------
+# `auth.purge_user` deletes an account's `log_events` rows, but a line about that account can still
+# be on its way: the erasing request's own access line is logged AFTER the handler committed, a queued
+# row waits for the sink's writer, a background task logs later. Those rows are stored without the
+# account (`logsink` asks here). Remembered for an hour in this process — a line later than that has
+# no request left to come from, and the retention job is the backstop for any other process.
+ERASED_MEMORY_SECONDS = 3600.0
+_erased: dict[int, float] = {}
+_erased_lock = threading.Lock()
+# Held by whoever checks rows against `is_erased` AND writes them (`erasure_guard`). `forget_user`
+# takes it too, so an erasure waits for a write that already passed the check to commit — and the
+# purge's DELETE, which runs after `forget_user`, then removes those rows as well. Without it a row
+# checked a moment before the erasure could be inserted a moment after the DELETE.
+_erasure_guard = threading.Lock()
+
+
+def erasure_guard() -> threading.Lock:
+    """Hold this around "drop erased accounts from these rows, insert them, commit"."""
+    return _erasure_guard
+
+
+def forget_user(user_id: int) -> None:
+    """Mark `user_id` erased: no line logged about it from now on is stored under its id. Call BEFORE
+    deleting its rows (see `_erasure_guard`)."""
+    now = time.monotonic()
+    with _erasure_guard, _erased_lock:
+        for stale in [u for u, at in _erased.items() if now - at > ERASED_MEMORY_SECONDS]:
+            del _erased[stale]
+        _erased[user_id] = now
+    with _elevations_lock:
+        _elevations.pop(user_id, None)
+    # That pop can have removed the lowest elevation, and the root level is the lowest of the base
+    # and every unexpired one. Recomputed here rather than left to whichever later request happens
+    # to call `_track_elevation` — the invariant `_apply_root_level` documents holds at every pop.
+    _apply_root_level()
+
+
+def is_erased(user_id: int | None) -> bool:
+    if user_id is None:
+        return False
+    with _erased_lock:
+        at = _erased.get(user_id)
+    return at is not None and time.monotonic() - at <= ERASED_MEMORY_SECONDS
+
+
+def current_source() -> str:
+    """The process name `configure_logging` was given (api, worker, runner, alembic)."""
+    return _source
+
+
+# --- request logging -----------------------------------------------------------------------
+# A client may supply its own request id so a browser log line and a server line join up (E22 S4).
+# Accepted only in this shape: it ends up in a log field, and an unbounded header would be a way to
+# write arbitrary text into the log.
+_REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9._-]{8,64}$")
+
+# Paths whose success is logged at DEBUG rather than INFO: the container healthcheck runs every 15s
+# (5.7k lines/day saying nothing) and hashed static assets are served by the hundred per page load.
+# A FAILURE on either still logs at WARNING/ERROR — this lowers the floor, it does not hide.
+_QUIET_PREFIXES = ("/healthz", "/assets/", "/static/")
+
+# Not `siftlode.logs`: this is the replacement for uvicorn's ACCESS log, and `siftlode.http` is
+# what an operator pins a level on when they want request lines and nothing else. The module it
+# happens to live in is not what the name is about.
+_request_log = logging.getLogger("siftlode.http")  # logger-name-ok: named for the traffic
+
+
+async def log_requests(request, call_next):
+    """The single access-log line, replacing uvicorn's.
+
+    What changes versus `uvicorn.access`: the ROUTE TEMPLATE instead of the concrete path (so
+    `/api/videos/{video_id}` groups), never the query string (it carried `?account=`, and at TRACE
+    the redacted query belongs in attrs, not in the URL), a request id echoed back in
+    `X-Request-Id` (with the template in `X-Route`), the duration, and the account — read from `request.state`, which the auth
+    dependency fills, because this middleware sits outside both the session middleware and the
+    downstream task's context."""
+    incoming = request.headers.get("x-request-id", "")
+    rid = incoming if _REQUEST_ID_RE.match(incoming) else uuid.uuid4().hex[:16]
+    token = _request_id.set(rid)
+    scope = _RequestScope()
+    scope_token = _scope.set(scope)
+    request.state.request_id = rid
+    started = time.perf_counter()
+    try:
+        response = await call_next(request)
+    except Exception:
+        # The unhandled-exception path: FastAPI's ServerErrorMiddleware re-raises after its own
+        # 500, so without this the ONE request that most needs a log line is the one with no
+        # access-log entry at all.
+        _request_log.exception(
+            "request failed",
+            extra={
+                **event(
+                    "http.request",
+                    method=request.method,
+                    path=_route_template(request),
+                    duration_ms=round((time.perf_counter() - started) * 1000, 1),
+                ),
+                **_request_owner(request, logging.ERROR),
+            },
+        )
+        _scope.reset(scope_token)
+        _request_id.reset(token)
+        raise
+    duration_ms = round((time.perf_counter() - started) * 1000, 1)
+    status = response.status_code
+    quiet = request.url.path.startswith(_QUIET_PREFIXES)
+    if status >= 500:
+        level = logging.ERROR
+    elif status >= 400:
+        level = logging.WARNING
+    else:
+        level = logging.DEBUG if quiet else logging.INFO
+    attrs: dict[str, Any] = {
+        "method": request.method,
+        "path": _route_template(request),
+        "status": status,
+        "duration_ms": duration_ms,
+    }
+    # Teljes (TRACE) the user consented to: the query string too — redacted, and in attrs only, never
+    # in the message.
+    if request.url.query and _traced(scope):
+        attrs["query"] = redact_query(request.url.query)
+    _request_log.log(
+        level,
+        "%s %s %s",
+        request.method,
+        _route_template(request),
+        status,
+        extra={**event("http.request", **attrs), **_request_owner(request, level)},
+    )
+    response.headers["X-Request-Id"] = rid
+    # The template, so the browser's copy of this line groups the same way (frontend lib/http.ts).
+    # Only a MATCHED route's: a template never carries a path parameter's value (a share token stays
+    # `{token}`), while the raw-path fallback of an unmatched request could — the browser then
+    # templates the path itself.
+    matched = _matched_route(request)
+    if matched:
+        response.headers["X-Route"] = matched
+    _scope.reset(scope_token)
+    _request_id.reset(token)
+    return response
+
+
+def consented_trace() -> bool:
+    """Whether the current request's OWN user turned Teljes (TRACE) on — the gate for a line that
+    carries what they typed or what was quoted back to them. The level alone is not enough: an
+    admin running the whole process at TRACE would pass it for every user, and a scheduler thread,
+    which has no request, for none of them in particular. Guard such a line with this, then log it
+    at TRACE."""
+    scope = _scope.get()
+    return scope is not None and _traced(scope)
+
+
+def _traced(scope: _RequestScope) -> bool:
+    """Whether this request's query string is logged: only under its OWN user's Teljes consent. An
+    admin setting the whole process to TRACE gets more detail, not every user's search terms — the
+    privacy policy promises those are kept only for a user who agreed to it."""
+    return scope.level is not None and scope.level <= TRACE
+
+
+# Anonymous request lines kept per client per minute. Signed-in traffic is not capped: it is a known
+# account and exactly what an admin needs to see. Anonymous traffic is where the volume can come from
+# nobody in particular — a scanner walking `/wp-login.php`, a bot on a share link — and each of its
+# 404s is a WARNING that would otherwise land in `log_events` without limit. The client key is an
+# in-memory counter key only; no address is ever written to a log line or a row.
+ANONYMOUS_LINES_PER_MINUTE = 60
+_anonymous_lines = RateLimiter(ANONYMOUS_LINES_PER_MINUTE, 60.0)
+
+
+def _request_owner(request, level: int) -> dict[str, Any]:
+    """`extra=` fields naming who the request belongs to, and whether its line may be stored. The
+    account comes from `request.state` (filled by the auth dependency), so it lands in the line's
+    top-level `user_id` — the correlation field — rather than in `attrs`."""
+    user_id = getattr(request.state, "user_id", None)
+    if user_id is not None:
+        return {"user_id": user_id}
+    if level < logging.INFO:
+        return {}  # a quiet line is never stored anyway — don't spend the client's allowance on it
+    return {} if _anonymous_lines.allow(client_key(request)) else {PERSIST: False}
+
+
+def _matched_route(request) -> str | None:
+    """The matched route's template (`/api/videos/{video_id}`), or None when nothing matched."""
+    return getattr(request.scope.get("route"), "path", None)
+
+
+def _route_template(request) -> str:
+    """The matched route's template, else the raw path for a 404 or a static file. Never the query
+    string."""
+    return _matched_route(request) or request.url.path

backend/app/logsink.py

@@ -0,0 +1,345 @@
+"""The `log_events` sink (E22 S3): the stored copy of the log the diagnostics console reads.
+
+stdout stays the complete log (docker keeps 3×50 MB of it); this table keeps the part worth querying
+for `log_retention_days`:
+
+  * every WARNING and above,
+  * the `http.request` line of every request at INFO or above (not the DEBUG-level health checks and
+    static files),
+  * every line of a request whose user raised their diagnostics level (E22 S5 — `logs.LevelGate`
+    marks them), which is the detail they consented to have kept,
+  * nothing a caller marked `extra={logs.PERSIST: False}` — how the request log caps anonymous
+    traffic per client.
+
+What the table must never do is slow down or break the code that logs. So a line is redacted and
+turned into a row on the CALLER's thread (the same `logs.record_fields` the stdout line is built
+from), put on a bounded queue without blocking, and written in batches by one background thread
+on its own connection — never inside a request's transaction. A full queue drops the row and
+counts it; a failed write drops the batch and says so on stderr. Both are reported rather than
+retried: a log that retries into a sick database is how a small outage becomes a big one.
+
+Flood control is the second half. On 2026-09-16 a YouTube RSS outage made prod log 11k near-identical
+warnings in four hours. stdout keeps them all; the table keeps `REPEATS_PER_MINUTE` of each message
+template per minute and one `log.repeats_suppressed` row saying how many more there were.
+
+Core layer: imports `logs`, `db`, `models`, `config` — nothing above them.
+"""
+
+import atexit
+import json
+import logging
+import queue
+import re
+import sys
+import threading
+import time
+from datetime import datetime, timedelta, timezone
+from typing import Any
+
+from sqlalchemy import delete, insert
+from sqlalchemy.engine import Engine
+from sqlalchemy.orm import Session
+
+from app import logs
+from app.config import settings
+from app.db import engine
+from app.models import LogEvent
+
+QUEUE_MAX = 10_000
+BATCH_MAX = 500
+FLUSH_SECONDS = 1.0
+REPEATS_PER_MINUTE = 30
+REPEAT_WINDOW_SECONDS = 60.0
+# Distinct templates tracked per window. Past this many, a new template is stored without counting
+# — the limiter must not become the unbounded thing it exists to prevent.
+_REPEAT_KEYS_MAX = 2_000
+
+# Column widths from models.LogEvent. A value past its column would fail the WHOLE batch it rides in.
+_LIMITS = {
+    "level": 10,
+    "source": 16,
+    "logger": 128,
+    "event": 64,
+    "request_id": 64,
+    "debug_session_id": 64,
+    "version": 64,
+    "msg": 8_000,
+    "exc": 32_000,
+}
+
+_log = logs.get_logger(__name__)
+
+
+def should_store(record: logging.LogRecord) -> bool:
+    """Whether this record belongs in `log_events` (see the module docstring)."""
+    if getattr(record, logs.PERSIST, True) is False:
+        return False
+    if record.levelno >= logging.WARNING or getattr(record, "elevated", False):
+        return True
+    return getattr(record, "event", None) == "http.request" and record.levelno >= logging.INFO
+
+
+def _without_nul(value: Any) -> Any:
+    """`value` with every NUL character replaced, in keys and nested values alike. Postgres refuses
+    NUL in text and `\\u0000` in JSONB, and a refused row fails the WHOLE batch it rides in — every
+    user's rows with it. Anything logged can carry one (`GET /assets/%00` logs the decoded path)."""
+    if isinstance(value, str):
+        return value.replace("\x00", "�")
+    if isinstance(value, dict):
+        return {_without_nul(k): _without_nul(v) for k, v in value.items()}
+    if isinstance(value, list):
+        return [_without_nul(v) for v in value]
+    return value
+
+
+def to_row(fields: dict[str, Any]) -> dict[str, Any]:
+    """`logs.record_fields` output → a `log_events` row: known columns only, text clipped to its
+    column and free of NUL, attrs made JSON-safe (anything exotic becomes its `str`, as on stdout)."""
+    row = {
+        column: fields.get(column) for column in LogEvent.__table__.columns.keys() if column != "id"
+    }
+    for column, limit in _LIMITS.items():
+        value = row.get(column)
+        if isinstance(value, str):
+            value = _without_nul(value)
+            row[column] = value[: limit - 1] + "…" if len(value) > limit else value
+    if row["attrs"] is not None:
+        row["attrs"] = _without_nul(
+            json.loads(json.dumps(row["attrs"], ensure_ascii=False, default=str))
+        )
+    return row
+
+
+def without_erased(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    """The rows, with the account taken off any whose account was erased meanwhile
+    (`logs.forget_user`). Checked right before the INSERT, under `logs.erasure_guard()` held until
+    the commit: a row built before the erasure can still be waiting for its batch."""
+    for row in rows:
+        if logs.is_erased(row.get("user_id")):
+            row["user_id"] = None
+    return rows
+
+
+def _notice(code: str, msg: str, **attrs: Any) -> dict[str, Any]:
+    """A row the sink writes about ITSELF. Built directly, not logged: logging it would route it back
+    through this sink (and the writer thread's own records are deliberately ignored)."""
+    return to_row(
+        {
+            "ts": datetime.now(timezone.utc),
+            "level": "WARNING",
+            "sev": logs.SEVERITY["WARNING"],
+            "source": logs.current_source(),
+            "logger": _log.name,
+            "event": code,
+            "msg": msg,
+            "attrs": attrs,
+            "version": settings.app_version,
+        }
+    )
+
+
+_DIGITS = re.compile(r"\d+")
+
+
+def _template(record: logging.LogRecord) -> str:
+    """The flood limiter's grouping key: the message template. Except when the template is a bare
+    `%s` — a pass-through for someone else's text (yt-dlp's lines, `formats._YdlLog`), where the
+    template says nothing and every distinct line would share one bucket. Then the text itself
+    groups, with its numbers folded, so `Retrying fragment 5 (1/10)` and `(2/10)` are still one
+    repeat and a different warning is not crowded out by them."""
+    msg = str(record.msg)
+    if msg != "%s" or not record.args:
+        return msg
+    return _DIGITS.sub("#", record.getMessage())[:200]
+
+
+class _Repeats:
+    """Per-template admission for one minute at a time. Used by the writer thread only (unlocked)."""
+
+    def __init__(self, per_window: int, window_seconds: float):
+        self.per_window = per_window
+        self.window = window_seconds
+        self.started = time.monotonic()
+        self.seen: dict[tuple[str, str, int], int] = {}
+
+    def admit(self, key: tuple[str, str, int] | None) -> bool:
+        if key is None:
+            return True
+        count = self.seen.get(key)
+        if count is None and len(self.seen) >= _REPEAT_KEYS_MAX:
+            return True
+        self.seen[key] = (count or 0) + 1
+        return self.seen[key] <= self.per_window
+
+    def roll(self, now: float, *, final: bool = False) -> list[dict[str, Any]]:
+        """Close the window if it is over — or, `final`, because the process is: one notice per
+        template that went past the limit. Without `final` a process that lives under a minute
+        (every download child) would drop its repeats and never say so."""
+        if not final and now - self.started < self.window:
+            return []
+        notices = [
+            _notice(
+                "log.repeats_suppressed",
+                f"{count - self.per_window} more like this were not stored",
+                logger=logger,
+                template=logs.redact_text(template),
+                suppressed=count - self.per_window,
+                window_seconds=int(self.window),
+            )
+            for (logger, template, _level), count in self.seen.items()
+            if count > self.per_window
+        ]
+        self.seen.clear()
+        self.started = now
+        return notices
+
+
+class LogSink(logging.Handler):
+    """The handler plus its writer. `start()` runs the writer thread; a test drives `write_pending()`
+    directly instead, on its own thread, with no timing involved."""
+
+    def __init__(self, bind: Engine = engine):
+        super().__init__(level=logging.NOTSET)
+        # Correlation fields are contextvars: capture them now, on the thread that logged.
+        self.addFilter(logs.ContextFilter())
+        self._bind = bind
+        self._queue: queue.Queue[tuple[tuple[str, str, int] | None, dict[str, Any]]] = queue.Queue(
+            QUEUE_MAX
+        )
+        self._dropped = 0
+        self._dropped_lock = threading.Lock()
+        self._repeats = _Repeats(REPEATS_PER_MINUTE, REPEAT_WINDOW_SECONDS)
+        self._stopping = threading.Event()
+        self._writer: threading.Thread | None = None
+
+    # --- caller side ---------------------------------------------------------------------------
+    def emit(self, record: logging.LogRecord) -> None:
+        if threading.current_thread() is self._writer:
+            return  # the writer's own trouble (a driver warning mid-insert) must not feed itself
+        if not should_store(record):
+            return
+        try:
+            row = to_row(logs.record_fields(record))
+        # silent-ok: `handleError` is the report — logging's own out-of-band channel to stderr.
+        # A log call here would be handed to this handler, which is the one that just failed.
+        except Exception:  # noqa: BLE001 — a record we cannot turn into a row is reported, not raised
+            self.handleError(record)
+            return
+        # Request lines are one per request by nature; only the repeated SAME message is a flood.
+        key = (
+            None
+            if row["event"] == "http.request"
+            else (record.name, _template(record), record.levelno)
+        )
+        try:
+            self._queue.put_nowait((key, row))
+        # silent-ok: counted, not logged — the count goes out as the `log.sink.dropped` notice row
+        # the writer emits, which is the one report that does not feed this queue again.
+        except queue.Full:
+            with self._dropped_lock:
+                self._dropped += 1
+
+    # --- writer side ---------------------------------------------------------------------------
+    def write_pending(self, wait: float = 0.0, *, final: bool = False) -> int:
+        """Write one batch: whatever is queued (up to BATCH_MAX), waiting up to `wait` seconds for the
+        first item. Returns the number of rows written, notices included. `final` closes the
+        repeat window early — the last batch before the process exits."""
+        rows: list[dict[str, Any]] = []
+        try:
+            item = self._queue.get(timeout=wait) if wait > 0 else self._queue.get_nowait()
+        # silent-ok: an idle sink. Nothing failed; there was simply nothing waiting.
+        except queue.Empty:
+            item = None
+        while item is not None:
+            key, row = item
+            if self._repeats.admit(key):
+                rows.append(row)
+            if len(rows) >= BATCH_MAX:
+                break
+            try:
+                item = self._queue.get_nowait()
+            # silent-ok: the batch ends here because the queue ran dry, which is the normal end.
+            except queue.Empty:
+                item = None
+        rows.extend(self._repeats.roll(time.monotonic(), final=final))
+        with self._dropped_lock:
+            dropped, self._dropped = self._dropped, 0
+        if dropped:
+            rows.append(
+                _notice("log.sink.dropped", f"{dropped} log events were dropped", dropped=dropped)
+            )
+        if not rows:
+            return 0
+        try:
+            # The connection is checked out BEFORE the guard, and every request path takes them in
+            # that order too (its session already holds a connection when it reaches the guard). The
+            # other way round, this thread would hold a process-global lock while waiting up to
+            # `pool_timeout` for a connection, and the requests that could return one are the ones
+            # blocked on the lock. The guard still spans the check, the insert and the commit.
+            with self._bind.connect() as conn, logs.erasure_guard(), conn.begin():
+                conn.execute(insert(LogEvent), without_erased(rows))
+        except Exception as exc:  # noqa: BLE001 — the database is down or refused the batch
+            # silent-ok / noqa T201: the sink's OWN failure. A log call here would be handed to
+            # this sink, which is the thing that just failed — straight into a loop. stderr is
+            # the one way out that does not re-enter.
+            print(  # noqa: T201
+                f"log sink: could not store {len(rows)} log events: "
+                f"{logs.redact_text(str(exc))[:300]}",
+                file=sys.stderr,
+            )
+            return 0
+        return len(rows)
+
+    def _run(self) -> None:
+        while not self._stopping.is_set():
+            self.write_pending(wait=FLUSH_SECONDS)
+        while self.write_pending():  # drain what arrived before the stop
+            pass
+        self.write_pending(final=True)  # and the repeat notices of a window cut short
+
+    def start(self) -> None:
+        self._writer = threading.Thread(target=self._run, name="log-sink", daemon=True)
+        self._writer.start()
+
+    def stop(self, timeout: float = 5.0) -> None:
+        self._stopping.set()
+        if self._writer is not None:
+            self._writer.join(timeout)
+
+
+_sink: LogSink | None = None
+
+
+def start() -> None:
+    """Attach the sink to the root logger and start its writer. Call after `logs.configure_logging`
+    (which replaces the root handlers) and once the schema exists. A no-op when `LOG_DB_SINK` is off
+    or it is already running."""
+    global _sink
+    if _sink is not None or not settings.log_db_sink:
+        return
+    _sink = LogSink()
+    # The same level gate as stdout: the root may be lowered for one user's elevated requests, and
+    # everyone else's detail must not reach the table through that gap either.
+    _sink.addFilter(logs.LevelGate())
+    logging.getLogger().addHandler(_sink)
+    _sink.start()
+    # A short-lived process (the download child) exits without a lifespan: flush on the way out.
+    atexit.register(stop)
+
+
+def stop() -> None:
+    """Detach the sink and write what is still queued."""
+    global _sink
+    if _sink is None:
+        return
+    logging.getLogger().removeHandler(_sink)
+    _sink.stop()
+    _sink = None
+
+
+def gc(db: Session, retention_days: int) -> int:
+    """Stage the deletion of rows older than `retention_days` (at least one day — stored logs always
+    expire). The caller commits. Returns the number of rows removed."""
+    cutoff = datetime.now(timezone.utc) - timedelta(days=max(1, retention_days))
+    result = db.execute(delete(LogEvent).where(LogEvent.ts < cutoff))
+    return result.rowcount or 0

backend/app/main.py

@@ -1,7 +1,5 @@
 import asyncio
-import logging
 import math
-import sys
 from contextlib import asynccontextmanager, suppress
 from pathlib import Path
 
@@ -9,25 +7,25 @@ from fastapi import FastAPI, HTTPException, Request
 from fastapi.encoders import jsonable_encoder
 from fastapi.exceptions import RequestValidationError
 
-# When started via uvicorn --log-config the "siftlode" logger is already configured
-# (see log_config.json). This block is a timestamped fallback for other entrypoints
-# (tests, scripts) so our logs are never silently dropped.
-_siftlode_logger = logging.getLogger("siftlode")
-if not _siftlode_logger.handlers:
-    _handler = logging.StreamHandler(sys.stdout)
-    _handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)-5s [%(name)s] %(message)s"))
-    _siftlode_logger.addHandler(_handler)
-    _siftlode_logger.setLevel(logging.INFO)
-    _siftlode_logger.propagate = False
+# Logging is configured HERE, before the route modules are imported, so every logger they create at
+# import time is already covered. ONE format for the api, the worker, the download child and alembic
+# alike (app/logs.py) — this replaces the old uvicorn --log-config + the hand-rolled fallback that
+# used to live here, and uvicorn's access log is replaced by the `http.request` middleware below.
+from app import logs
+from app.logs import configure_logging, current_level, log_requests, set_level
 
-log = logging.getLogger("siftlode.app")
+configure_logging("api")
+
+log = logs.get_logger(__name__)
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
 from fastapi.staticfiles import StaticFiles
 from sqlalchemy.exc import DataError
 from starlette.middleware.sessions import SessionMiddleware
 
+from app import logsink
 from app import state
+from app import sysconfig
 from app.config import settings
 from app.db import SessionLocal
 from app.plex import stream as plex_stream
@@ -38,6 +36,7 @@ from app.routes import (
     auth as auth_routes,
     channels,
     config as config_routes,
+    diagnostics as diagnostics_routes,
     discover as discover_routes,
     downloads,
     feed,
@@ -63,16 +62,28 @@ from app.scheduler import shutdown_scheduler, start_scheduler
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     log.info("Siftlode starting up")
+    # The stored copy of the log (log_events). The entrypoint has already migrated the schema.
+    logsink.start()
     # First-run: if the instance isn't configured yet, mint a fresh one-time setup token and print
     # the wizard URL so the operator can open it (the app runs in setup mode until they finish).
     with SessionLocal() as db:
+        # The admin's runtime level (sysconfig `log_level`) wins over the LOG_LEVEL bootstrap
+        # default, so a level chosen in the UI survives a restart without touching the environment.
+        # Best-effort on purpose: a LOGGING knob must never be the thing that stops the app from
+        # booting — an unreachable or not-yet-migrated database has to start into the env level.
+        try:
+            set_level(sysconfig.get_str(db, "log_level"))
+        except Exception:  # noqa: BLE001 — any DB/config failure keeps the bootstrap level
+            # With the traceback: this catch spans a connection, a query and a coercion at boot,
+            # and `%s` of the exception does not say which of them raised.
+            log.exception("could not read the stored log level; staying at %s", current_level())
         if not state.is_configured(db):
             token = state.rotate_setup_token(db)
             url = f"{settings.app_base}/setup?token={token}"
             log.warning(
                 "FIRST-RUN SETUP REQUIRED — this instance isn't configured yet.\n"
                 "    Open the install wizard (internal access only):\n    %s",
-                url,
+                url,  # log-pii-ok: this instance's OWN setup URL, and `?token=` is redacted
             )
     # No HLS session survives a restart (they live in this process), so anything still on disk is
     # a crashed run's segments — sweep them or they accumulate forever. FIRE AND FORGET, not
@@ -100,6 +111,7 @@ async def lifespan(app: FastAPI):
         # Kills a running subtitle extraction: its worker would otherwise hold interpreter exit for
         # the rest of the read (see plex_subtitles.shutdown).
         plex_subtitles.shutdown()
+        logsink.stop()  # last: write what the shutdown itself just logged
 
 
 # How often idle HLS sessions are reaped (see plex/stream.py `reap_idle`).
@@ -243,6 +255,14 @@ async def security_headers(request, call_next):
     return response
 
 
+# The access log (app/logs.py): request id, route template, status, duration — one line, never a
+# query string. Registered LAST so it is the OUTERMOST http middleware and therefore times the whole
+# request, including what the three above do. Being outermost also puts it OUTSIDE SessionMiddleware,
+# which is why the account comes from `request.state.user_id` (set by the auth dependency, the one
+# place a session is actually resolved into an account) rather than from `request.session`.
+app.middleware("http")(log_requests)
+
+
 app.include_router(health.router)
 app.include_router(auth_routes.router)
 app.include_router(sync.router)
@@ -251,6 +271,7 @@ app.include_router(feed.router)
 app.include_router(discover_routes.router)
 app.include_router(search_routes.router)
 app.include_router(me.router)
+app.include_router(diagnostics_routes.router)
 app.include_router(notifications.router)
 app.include_router(messages.router)
 app.include_router(channels.router)

backend/app/models.py

@@ -12,6 +12,7 @@ from sqlalchemy import (
     Index,
     Integer,
     JSON,
+    SmallInteger,
     String,
     Text,
     UniqueConstraint,
@@ -102,6 +103,16 @@ class User(Base, TimestampMixin, CanonicalEmailMixin):
     # current_user rejects any cookie whose epoch is stale, so a stolen/copied cookie dies on those
     # events (client-side signed cookies otherwise stay valid until they expire).
     session_epoch: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
+    # When this account last signed in (Google or password), stamped by the one sign-in path,
+    # routes/auth._establish_session. NULL = not since the column existed (0085; no backfill is
+    # possible). The shared demo account is never stamped — its "last login" would mean nobody.
+    last_login_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    # The diagnostics level this user consented to (E22 S5) and when it lapses back to the base
+    # level. Both NULL = base. Written only by `PUT /api/me/diagnostics`, which computes the expiry
+    # itself; read through `app.diagnostics.effective_level`, never directly (an expired level is
+    # not cleared eagerly — it simply stops counting).
+    diagnostics_level: Mapped[str | None] = mapped_column(String(8))
+    diagnostics_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
     display_name: Mapped[str | None] = mapped_column(String(255))
     avatar_url: Mapped[str | None] = mapped_column(String(1024))
     role: Mapped[str] = mapped_column(String(16), default="user", server_default="user")
@@ -539,6 +550,37 @@ class AuditLog(Base):
     )
 
 
+class LogEvent(Base):
+    """One stored log event (E22 S3) — the `log_events` table the diagnostics console reads. Rows are
+    written by app/logsink.py's background writer, never inside a request's transaction, already
+    redacted (the same `logs.record_fields` the stdout line is built from).
+
+    `user_id` has NO foreign key on purpose: the writer runs behind the event, so a line about an
+    account can arrive after that account was purged, and a constraint would make it fail the whole
+    batch it travels in. `auth.purge_user` deletes an account's rows explicitly instead, after
+    `logs.forget_user` makes every later write in this process store such a line without the account;
+    the retention job (`log_retention_days`) bounds whatever another process writes late."""
+
+    __tablename__ = "log_events"
+    __table_args__ = (Index("ix_log_events_user_ts", "user_id", "ts"),)
+
+    id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
+    ts: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
+    level: Mapped[str] = mapped_column(String(10))
+    sev: Mapped[int] = mapped_column(SmallInteger)
+    source: Mapped[str] = mapped_column(String(16))
+    logger: Mapped[str] = mapped_column(String(128))
+    event: Mapped[str | None] = mapped_column(String(64))
+    msg: Mapped[str] = mapped_column(Text)
+    attrs: Mapped[dict | None] = mapped_column(JSONB)
+    exc: Mapped[str | None] = mapped_column(Text)
+    request_id: Mapped[str | None] = mapped_column(String(64), index=True)
+    user_id: Mapped[int | None] = mapped_column(Integer)
+    debug_session_id: Mapped[str | None] = mapped_column(String(64))
+    job_id: Mapped[int | None] = mapped_column(Integer)
+    version: Mapped[str | None] = mapped_column(String(64))
+
+
 class AppState(Base):
     """Single-row global app state (admin-controlled)."""
 

backend/app/plex/client.py

@@ -6,16 +6,15 @@ metadata, image proxying, and (P5) optional watch-state write-back. Requests JSO
 ``Accept: application/json`` header (Plex serves XML by default).
 """
 
-import logging
 from urllib.parse import quote
 
 import httpx
 from sqlalchemy.orm import Session
 
-from app import sysconfig
+from app import logs, sysconfig
 from app.config import settings
 
-log = logging.getLogger("siftlode.plex")
+log = logs.get_logger(__name__)
 
 
 class PlexError(Exception):
@@ -106,7 +105,18 @@ class PlexClient:
         Best-effort (needs the library's related data) — returns [] if Plex has none."""
         try:
             mc = self._get(f"/library/metadata/{rating_key}/related")
-        except PlexError:
+        except PlexError as exc:
+            # An empty list is also the answer for an item that genuinely has no related titles,
+            # so the caller cannot tell the two apart — and it deliberately declines to CACHE an
+            # empty one for exactly that reason. This line is what says which it was.
+            log.debug(
+                "the related items for a Plex item could not be read",
+                extra=logs.event(
+                    "plex.client.related_unavailable",
+                    rating_key=rating_key,
+                    reason=type(exc).__name__,
+                ),
+            )
             return []
         out: list[dict] = []
         for hub in mc.get("Hub", []) or []:
@@ -139,6 +149,8 @@ class PlexClient:
             raise PlexError(str(e)) from e
         try:
             return (r.json() or {}).get("MediaContainer", {}) or {} if r.content else {}
+        # silent-ok: the HTTP call already succeeded — this is a write endpoint, and Plex answers
+        # those with an empty or non-JSON body as a matter of course.
         except ValueError:
             return {}
 

backend/app/plex/detail.py

@@ -22,6 +22,8 @@ def rich_detail(meta: dict) -> dict:
         if str(r.get("image") or "").startswith("imdb://"):
             try:
                 imdb_rating = round(float(r.get("value")), 1)
+            # silent-ok: no IMDb score, which is the common and legitimate case — the UI has a
+            # separate audience score to fall back on and shows nothing if neither exists.
             except (TypeError, ValueError):
                 pass
             break

backend/app/plex/imgcache.py

@@ -11,10 +11,13 @@ The cache directory (``cache.CACHE_DIR``) is shared with the subtitle cache (see
 from fastapi import Response
 from sqlalchemy.orm import Session
 
+from app import logs
 from app.models import PlexCollection, PlexItem, PlexSeason, PlexShow
 from app.plex.cache import CACHE_DIR
 from app.plex.lookups import by_rating_key
 
+log = logs.get_logger(__name__)
+
 _EXT_CT = [("jpg", "image/jpeg"), ("png", "image/png"), ("webp", "image/webp")]
 _CT_EXT = {ct: ext for ext, ct in _EXT_CT}
 # Target box per variant — Plex resizes to these server-side. Posters fill ≤176px cells (grid/info/
@@ -46,6 +49,8 @@ def read(key: str) -> tuple[bytes, str] | None:
         try:
             if p.exists() and p.stat().st_size > 0:
                 return p.read_bytes(), ct
+        # silent-ok: a miss. The caller fetches from Plex and writes the file again, and the write
+        # path below is the one that reports a cache that cannot be used.
         except OSError:
             pass
     return None
@@ -58,8 +63,14 @@ def write(key: str, data: bytes, ct: str) -> None:
         tmp = CACHE_DIR / f"{key}.{ext}.tmp"
         tmp.write_bytes(data)
         tmp.replace(CACHE_DIR / f"{key}.{ext}")  # atomic publish
-    except OSError:
-        pass  # cache is best-effort
+    except OSError as exc:
+        # Best-effort, and silently so: with an unwritable cache every poster on every grid goes
+        # to Plex live, for as long as nobody notices. Debug, because a grid is ~100 images and
+        # the point is the condition, not the count.
+        log.debug(
+            "the Plex image cache could not be written",
+            extra=logs.event("plex.imgcache.write_failed", reason=type(exc).__name__),
+        )
 
 
 def response(data: bytes, ct: str) -> Response:

backend/app/plex/paths.py

@@ -12,7 +12,9 @@ from pathlib import Path
 
 from sqlalchemy.orm import Session
 
-from app import sysconfig
+from app import logs, sysconfig
+
+log = logs.get_logger(__name__)
 
 
 def parse_map(raw: str) -> list[tuple[str, str]]:
@@ -47,6 +49,13 @@ def local_media_path(db: Session, plex_path: str | None) -> Path | None:
         return None
     try:
         p = Path(map_path(db, plex_path)).resolve()
-    except OSError:
+    except OSError as exc:
+        # None is answered as a 404 "no local file", which is also the answer for a title the
+        # server does not hold — so a dead mount or a wrong path map reads as missing media for
+        # the whole library at once. The path itself stays out: it carries the title.
+        log.warning(
+            "a Plex media path could not be resolved locally",
+            extra=logs.event("plex.paths.resolve_failed", reason=type(exc).__name__),
+        )
         return None
     return p if p.is_file() else None

backend/app/plex/stream.py

@@ -18,7 +18,6 @@ segments. The remux is CPU-light (video copy + a tiny audio transcode), so it ru
 CPU-only prod host; only P3's full transcode is CPU-heavy.
 """
 
-import logging
 import os
 import re
 import secrets
@@ -31,15 +30,14 @@ from pathlib import Path
 
 from sqlalchemy.orm import Session as DbSession
 
-from app import sysconfig
+from app import logs, sysconfig
 from app.config import settings
 from app.downloads.edit import encode_thread_cap
 from app.models import PlexItem
 from app.plex import paths
 from app.plex.sync import HW_DECODABLE_CODECS
 
-log = logging.getLogger("siftlode.plex")
-
+log = logs.get_logger(__name__)
 _HLS_ROOT = (
     Path(settings.plex_hls_dir)
     if settings.plex_hls_dir
@@ -135,6 +133,8 @@ class HlsSession:
         # single index. Derived, not passed, so it cannot disagree with the argv.
         self.entry = _MASTER_PL if self.variants else _INDEX_PL
         self.last_access = time.time()
+        # `artifact_may_arrive` reports a refusing session directory once, not per segment.
+        self.probe_failure_reported = False
 
 
 def _seek_opts(start_s: float) -> list[str]:
@@ -232,7 +232,15 @@ def _probe_video_kind(src: Path) -> str:
             text=True,
             timeout=10,
         )
-    except (subprocess.SubprocessError, OSError):
+    except (subprocess.SubprocessError, OSError) as exc:
+        # The safe kind is a real answer, but it is the answer for an 8-bit SDR source — a blind
+        # ffprobe sends every HDR file down it, so playback quietly degrades to a software
+        # transcode with nothing anywhere saying why. The path stays out of the line: it is the
+        # title of whatever is playing.
+        log.info(
+            "video-kind probe failed; assuming the HW-safe pipeline",
+            extra=logs.event("plex.stream.probe_kind_failed", reason=type(exc).__name__),
+        )
         return _VK_HWSAFE
     pix_fmt = bits = transfer = ""
     for line in out.stdout.splitlines():
@@ -261,7 +269,14 @@ def _opencl_present() -> bool:
     software libx264. Its own function so it's monkeypatchable in tests, mirroring _render_node_present."""
     try:
         return any(_OPENCL_ICD_DIR.glob("*.icd"))
-    except OSError:
+    except OSError as exc:
+        # A MISSING directory does not come through here — `glob` simply yields nothing for one.
+        # This is a directory we cannot read, and the answer it forces (no OpenCL) costs every HDR
+        # source the hardware tonemap path.
+        log.info(
+            "the OpenCL ICD directory could not be read; treating OpenCL as absent",
+            extra=logs.event("plex.stream.opencl_probe_failed", reason=type(exc).__name__),
+        )
         return False
 
 
@@ -404,7 +419,7 @@ def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) ->
     source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
     if not transcode:
         return (False, False)
-    mode = (sysconfig.get_str(db, "plex_transcode_hw") or "auto").lower()
+    mode = sysconfig.get_str(db, "plex_transcode_hw")
     if mode == "off" or not _render_node_present():
         return (False, False)
     if mode == "on":
@@ -447,6 +462,8 @@ def _sw_child_preexec() -> None:
     playback."""
     try:
         os.nice(_SW_TRANSCODE_NICE)
+    # silent-ok: this runs in the forked child, between fork and exec. Logging there can deadlock
+    # on a lock some other thread held at fork time, and the child is about to become ffmpeg.
     except OSError:
         pass
 
@@ -473,7 +490,14 @@ def _probe_audio_langs(src: Path) -> list[str]:
             timeout=10,
         )
         return [ln.strip() for ln in out.stdout.splitlines()]
-    except (subprocess.SubprocessError, OSError):
+    except (subprocess.SubprocessError, OSError) as exc:
+        # An empty list IS the single-audio path, so a failed probe silently collapses a file with
+        # several audio tracks down to one — and the viewer gets a player with no language picker
+        # rather than an error.
+        log.warning(
+            "audio-track probe failed; falling back to a single-audio session",
+            extra=logs.event("plex.stream.probe_audio_failed", reason=type(exc).__name__),
+        )
         return []
 
 
@@ -582,8 +606,14 @@ def _probe_first_pts(seg: Path) -> float | None:
             tok = line.strip().strip(",")
             if tok:
                 return float(tok)
-    except (subprocess.SubprocessError, ValueError, OSError):
-        pass
+    except (subprocess.SubprocessError, ValueError, OSError) as exc:
+        # None means "keep the offset the client asked for", and the whole point of this probe is
+        # that the two differ: without it the subtitle track is shifted by whatever the container
+        # started its timestamps at. Seconds of drift, and nothing on screen calls it a failure.
+        log.warning(
+            "first-PTS probe failed; the requested offset is used as-is",
+            extra=logs.event("plex.stream.first_pts_probe_failed", reason=type(exc).__name__),
+        )
     return None
 
 
@@ -592,10 +622,18 @@ def _kill(s: HlsSession) -> None:
         s.proc.terminate()
         try:
             s.proc.wait(timeout=3)
+        # silent-ok: the escalation IS the handling. A transcode that ignores TERM for three
+        # seconds gets KILL, which is exactly what this branch does.
         except subprocess.TimeoutExpired:
             s.proc.kill()
-    except Exception:
-        pass
+    except Exception as exc:  # noqa: BLE001 — teardown finishes whatever the process does
+        # Not nothing: what survives this is an ffmpeg nobody owns any more, holding a VAAPI
+        # handle and a core, and the transcode cap that is supposed to bound them counts rows in
+        # `_sessions` rather than processes. One line is the only way that ever surfaces.
+        log.info(
+            "could not stop the transcode process",
+            extra=logs.event("plex.stream.kill_failed", key=s.key, reason=type(exc).__name__),
+        )
     shutil.rmtree(s.dir, ignore_errors=True)
 
 
@@ -608,6 +646,13 @@ def _enforce_cap(cap: int) -> None:
     if len(tx) <= cap:
         return
     for key, s in sorted(tx, key=lambda kv: kv[1].last_access)[: len(tx) - cap]:
+        # Somebody's playback stops here. The route turns the missing session into a retryable
+        # 404 and the player restarts at the playhead, so from the server nothing failed — and
+        # "my film keeps rebuffering" has no other explanation anywhere.
+        log.info(
+            "a transcode session was dropped to stay under the cap",
+            extra=logs.event("plex.stream.session_culled", key=key, cap=cap, sessions=len(tx)),
+        )
         _kill(s)
         _sessions.pop(key, None)
 
@@ -664,7 +709,7 @@ def start_session(
         # remux ignore preset/crf/max_height, and remux must not touch the DB for them).
         preset, crf, max_height = "veryfast", 20, 0
         if transcode and not hw:
-            preset = sysconfig.get_str(db, "plex_transcode_preset") or "veryfast"
+            preset = sysconfig.get_str(db, "plex_transcode_preset")
             crf = sysconfig.get_int(db, "plex_transcode_crf")
             max_height = sysconfig.get_int(db, "plex_transcode_max_height")
         # Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
@@ -710,6 +755,21 @@ def start_session(
                 tonemap=tonemap,
             )
             _sessions[key] = s
+            # The one line that says which pipeline a session actually got. Everything downstream
+            # — the cap, the HW→SW fallback, the subtitle offset — is about this choice, and until
+            # now it was only inferable from the ffmpeg command nobody keeps.
+            log.info(
+                "an HLS session started",
+                extra=logs.event(
+                    "plex.stream.session_started",
+                    key=key,
+                    transcode=transcode,
+                    hw=hw,
+                    tonemap=tonemap,
+                    variants=len(variants or ()),
+                    start_s=round(start_s, 1),
+                ),
+            )
             # Admin-configurable concurrency cap (Configuration → Plex → Max concurrent transcodes);
             # at least 1 so playback can't be capped to zero.
             _enforce_cap(max(1, sysconfig.get_int(db, "plex_max_transcodes")))
@@ -728,7 +788,8 @@ def start_session(
         # ONE lock in _spawn, so we can never tear down the newer session. s.hw is now False, so the route
         # honestly reports "CPU" and the player's Preparing-stream state stays correct.
         log.warning(
-            "plex transcode: HW session produced no segments — falling back to software key=%s", key
+            "the hardware transcode produced no segments; falling back to software",
+            extra=logs.event("plex.stream.hw_fallback", key=key),
         )
         respawned = _spawn(False, False, replace_only=s)
         if respawned is None:
@@ -867,7 +928,23 @@ def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
         return True
     try:
         return (s.dir / segment_name(seq - 1, variant)).exists()
-    except OSError:
+    except OSError as exc:
+        # `exists()` answers False for a missing file without raising, so this is the session
+        # directory itself refusing — a full or unmounted disk. Every segment then 404s, which the
+        # player shows as a stall. Once per session: this runs for every segment request and
+        # every hls.js retry of one, and the sink's limiter still stores 30 a minute of a
+        # template — stdout would get them all.
+        if not s.probe_failure_reported:
+            s.probe_failure_reported = True
+            log.warning(
+                "a session segment could not be checked",
+                extra=logs.event(
+                    "plex.stream.segment_probe_failed",
+                    key=s.key,
+                    seq=seq,
+                    reason=type(exc).__name__,
+                ),
+            )
         return False
 
 
@@ -884,12 +961,16 @@ def wait_for(path: Path, timeout: float = 20.0, proc: subprocess.Popen | None =
         try:
             if path.exists() and path.stat().st_size > 0:
                 return True
+        # silent-ok: this is a 150 ms poll — a line here is a line every 150 ms for the whole
+        # timeout. The final check below decides, and the 404 the caller then returns is honest.
         except OSError:
             pass
         if proc is not None and proc.poll() is not None:
             # the producer exited — no more segments are coming; one last check then give up.
             try:
                 return path.exists() and path.stat().st_size > 0
+            # silent-ok: one last look after the producer exited. False here becomes a 404 the
+            # player recovers from, which is the honest answer either way.
             except OSError:
                 return False
         time.sleep(0.15)
@@ -922,7 +1003,16 @@ def sweep_orphans(older_than: float | None = None) -> int:
     dropped = 0
     try:
         entries = list(_HLS_ROOT.iterdir())
-    except OSError:  # the root doesn't exist yet — nothing to sweep
+    # silent-ok: nothing has been transcoded yet, so there is nothing to sweep.
+    except FileNotFoundError:
+        return 0
+    except OSError as exc:
+        # Any OTHER reason the root will not list is not "nothing to sweep": the sweep reports 0
+        # like a clean run while every crashed session's segments stay on disk forever.
+        log.warning(
+            "the HLS root could not be listed; no orphans were swept",
+            extra=logs.event("plex.stream.sweep_root_unreadable", reason=type(exc).__name__),
+        )
         return 0
     for p in entries:
         if not _SESSION_DIR_RE.match(p.name):
@@ -932,7 +1022,9 @@ def sweep_orphans(older_than: float | None = None) -> int:
         try:
             if p.stat().st_mtime >= cutoff:
                 continue  # this run made (or reused) it — never ours to delete
-        except OSError:  # vanished under us (a concurrent restart of that session)
+        # silent-ok: vanished under us — a concurrent restart of that session removed it, which
+        # is the outcome this sweep wanted anyway.
+        except OSError:
             continue
         with _lock:
             if any(s.dir == p for s in _sessions.values()):

backend/app/plex/subtitles.py

@@ -33,17 +33,18 @@ zero-based clock and shifted back to absolute time here with `shift_vtt`.
 """
 
 import json
-import logging
 import re
 import subprocess
 import tempfile
 import threading
 import time
+import logging
 from concurrent.futures import ThreadPoolExecutor
 from pathlib import Path
 
 from sqlalchemy.orm import Session
 
+from app import logs
 from app.db import SessionLocal
 from app.models import PlexItem
 from app.plex import paths as plex_paths
@@ -51,8 +52,7 @@ from app.plex.cache import CACHE_DIR
 from app.plex.client import PlexClient, PlexError
 from app.plex.lookups import by_rating_key
 
-log = logging.getLogger("siftlode.plex")
-
+log = logs.get_logger(__name__)
 # Image-based subtitle codecs (Blu-ray/DVD) — can't be turned into text WebVTT (would need burn-in).
 BITMAP_SUBS = {
     "pgs",
@@ -104,6 +104,7 @@ def shift_vtt(vtt: str, offset: float) -> str:
     straddling 0 is clamped to start at 0."""
     if not offset:
         return vtt
+    unshifted = 0
     out_blocks: list[str] = []
     for block in re.split(r"\n[ \t]*\n", vtt):
         if "-->" not in block:
@@ -121,7 +122,12 @@ def shift_vtt(vtt: str, offset: float) -> str:
                 st = _vtt_ts_to_s(left.strip()) - offset
                 en = _vtt_ts_to_s(rparts[0]) - offset
             except (ValueError, IndexError):
-                break  # unparseable timing → leave the block as-is
+                # silent-ok: left as-is means left UNSHIFTED — the viewer sees that one cue at
+                # the wrong moment. Counted rather than logged here: this is per cue, and the file
+                # being parsed is one ffmpeg wrote for us a moment ago, so a broken timestamp is
+                # a fact about the extraction, reported once below.
+                unshifted += 1
+                break
             if en <= 0:
                 keep = False  # whole cue is before the session start → drop the block
             else:
@@ -131,6 +137,11 @@ def shift_vtt(vtt: str, offset: float) -> str:
             break
         if keep:
             out_blocks.append("\n".join(lines))
+    if unshifted:
+        log.info(
+            "some cues could not be shifted and stay at their original time",
+            extra=logs.event("plex.subtitles.cues_unshifted", cues=unshifted, offset_s=offset),
+        )
     return "\n\n".join(out_blocks)
 
 
@@ -162,7 +173,15 @@ def cache_read(key: str) -> str | None:
     try:
         p = CACHE_DIR / f"{key}.vtt"
         return p.read_text(encoding="utf-8") if p.exists() else None
-    except OSError:
+    except OSError as exc:
+        # None is the same answer as "never cached", so a cache we cannot read looks exactly like
+        # a cold one — and every request re-extracts instead of saying the disk is the problem.
+        log.warning(
+            "the cached subtitle could not be read",
+            extra=logs.event(
+                "plex.subtitles.cache_read_failed", key=key, reason=type(exc).__name__
+            ),
+        )
         return None
 
 
@@ -175,7 +194,16 @@ def cache_write(key: str, vtt: str) -> bool:
         tmp.write_text(vtt, encoding="utf-8")
         tmp.replace(CACHE_DIR / f"{key}.vtt")  # atomic publish
         return True
-    except OSError:
+    except OSError as exc:
+        # `publish` turns False into "the region was not claimed" and moves on, so the track just
+        # never appears. Which errno it was — full disk, read-only mount, permissions — is the
+        # whole diagnosis, and it exists only here.
+        log.warning(
+            "the subtitle cache could not be written",
+            extra=logs.event(
+                "plex.subtitles.cache_write_failed", key=key, reason=type(exc).__name__
+            ),
+        )
         return False
 
 
@@ -206,7 +234,17 @@ def read_meta(key: str) -> tuple[list[list[float]], bool]:
     try:
         raw = json.loads(_meta_path(key).read_text(encoding="utf-8"))
         return [[float(a), float(b)] for a, b in raw.get("ranges", [])], bool(raw.get("complete"))
-    except (OSError, ValueError, TypeError):
+    # silent-ok: no sidecar at all is the documented older-build case the fallback exists for.
+    except FileNotFoundError:
+        return ([[0.0, _FULL_END_S]], True) if _exists(_vtt_path(key)) else ([], False)
+    except (OSError, ValueError, TypeError) as exc:
+        # A sidecar that EXISTS and will not parse is different, and the fallback is the dangerous
+        # answer to it: beside a cached VTT it reports the track as complete, so a partial track is
+        # never extended again and the viewer loses every cue past the window that was cached.
+        log.warning(
+            "the subtitle sidecar could not be read; the cached track is taken as complete",
+            extra=logs.event("plex.subtitles.meta_unreadable", key=key, reason=type(exc).__name__),
+        )
         return ([[0.0, _FULL_END_S]], True) if _exists(_vtt_path(key)) else ([], False)
 
 
@@ -216,6 +254,8 @@ def _exists(p: Path) -> bool:
     escaped as a 500 from the status route."""
     try:
         return p.exists()
+    # silent-ok: this function's whole job is to answer False for a path the OS refuses. The
+    # refusal IS the return value; a caller that needs to report one reports it itself.
     except (OSError, ValueError):
         return False
 
@@ -226,8 +266,18 @@ def _write_meta(key: str, ranges: list[list[float]], complete: bool) -> None:
         tmp = CACHE_DIR / f"{key}.json.tmp"
         tmp.write_text(json.dumps({"ranges": ranges, "complete": complete}), encoding="utf-8")
         tmp.replace(_meta_path(key))
-    except OSError:
-        pass
+    except OSError as exc:
+        # Without the sidecar beside it, a cached window reads back as a COMPLETE track (see
+        # `read_meta`) — so one failed write here permanently costs the rest of that subtitle.
+        log.warning(
+            "the subtitle sidecar could not be written",
+            extra=logs.event(
+                "plex.subtitles.meta_write_failed",
+                key=key,
+                complete=complete,
+                reason=type(exc).__name__,
+            ),
+        )
 
 
 def generation(key: str) -> int:
@@ -235,7 +285,9 @@ def generation(key: str) -> int:
     window is superseded by a wider one the browser refetches instead of keeping the short VTT."""
     try:
         return _vtt_path(key).stat().st_mtime_ns
-    except (OSError, ValueError):  # ValueError: a NUL in the key — `stat` refuses the path outright
+    # silent-ok: 0 for a track that is not cached yet, which is the ordinary case on every first
+    # request. (ValueError: a NUL in the key — `stat` refuses the path outright.)
+    except (OSError, ValueError):
         return 0
 
 
@@ -267,16 +319,27 @@ def merge_vtt(*sources: str) -> str:
     happen to share a time (two speakers, a sign over dialogue) are both kept."""
     seen: set[str] = set()
     cues: list[tuple[float, str]] = []
+    dropped = 0
     for src in sources:
         for block in re.split(r"\n[ \t]*\n", src or ""):
-            canon = _canonical_cue(block.strip())
+            stripped = block.strip()
+            canon = _canonical_cue(stripped)
             if canon is None:
+                # A block WITH a timing line that still would not parse is a cue we are throwing
+                # away, not a header — and the merged track would simply be missing those lines.
+                if "-->" in stripped:
+                    dropped += 1
                 continue  # header / NOTE / STYLE — the merged file gets its own single header
             start, key, text = canon
             if key in seen:
                 continue
             seen.add(key)
             cues.append((start, text))
+    if dropped:
+        log.info(
+            "some cues were dropped from the merged subtitle: their timing would not parse",
+            extra=logs.event("plex.subtitles.cues_dropped", cues=dropped, kept=len(cues)),
+        )
     cues.sort(key=lambda c: c[0])
     return "WEBVTT\n\n" + "\n\n".join(b for _, b in cues) + "\n"
 
@@ -295,6 +358,8 @@ def _canonical_cue(block: str) -> tuple[float, str, str] | None:
         try:
             st = _vtt_ts_to_s(left.strip())
             en = _vtt_ts_to_s(rparts[0])
+        # silent-ok: None here means "not a cue", and the caller tells a header apart from a
+        # broken timing line by whether the block had a `-->` at all — see `merge_vtt`.
         except (ValueError, IndexError):
             return None
         lines[i] = f"{_vtt_s_to_ts(st)} --> {_vtt_s_to_ts(en)}{settings}"
@@ -310,6 +375,8 @@ def _last_cue_end(vtt: str) -> float:
             continue
         try:
             end = max(end, _vtt_ts_to_s(ln.partition("-->")[2].strip().split(None, 1)[0]))
+        # silent-ok: this is a floor — "how far the pass provably got". Skipping a line it cannot
+        # read makes the answer smaller, never larger, so the error is always on the safe side.
         except (ValueError, IndexError):
             continue
     return end
@@ -652,12 +719,35 @@ def _run(
             return
         _extract_and_publish(rating_key, sub_ord, window_start)
     except SubtitleUnavailable as e:
+        # The player's own state was the only place this ever showed. `_pass_failed` backs the item
+        # off for an hour and `_fail` records a code, but neither leaves a line, so "my subtitles
+        # never appear" had nothing on the server to read.
+        # A final code is a fact about the file (no such track, a bitmap subtitle); anything else
+        # is this instance failing at something it could do.
+        log.log(
+            logging.INFO if e.code in FINAL_CODES else logging.WARNING,
+            "subtitle extraction gave up",
+            extra=logs.event(
+                "plex.subtitles.pass_failed",
+                rating_key=rating_key,
+                sub_ord=sub_ord,
+                code=e.code,
+                windowed=window_start is not None,
+            ),
+        )
         if e.code == "extract_failed":
             _pass_failed(rating_key, [key], window_start)
         else:
             _fail(key, e.code)
     except PlexError as e:
-        log.warning("subtitle extraction failed for %s/%s: %s", rating_key, sub_ord, e)
+        # No traceback on purpose: a PlexError carries httpx's own message, which names the method,
+        # the URL and the status — the stack under it is this `except` and the client call, and
+        # says nothing the message does not. (The URL holds no token; it travels as a header.)
+        log.warning(
+            "subtitle extraction failed talking to Plex: %s",
+            e,
+            extra=logs.event("plex.subtitles.plex_error", rating_key=rating_key, sub_ord=sub_ord),
+        )
         _fail(key, "plex_error")
     except (
         Exception
@@ -829,6 +919,16 @@ def _ffmpeg_extract(
                 try:
                     _, err = proc.communicate(timeout=_TERM_GRACE_S)
                 except subprocess.TimeoutExpired:
+                    # TERM is what makes a timed-out pass worth keeping: ffmpeg closes its outputs
+                    # on it. An ffmpeg that ignores TERM leaves only whole blocks already flushed,
+                    # so the partial track is short for a reason the later line cannot tell apart.
+                    log.info(
+                        "the subtitle extraction ignored TERM and was killed",
+                        extra=logs.event(
+                            "plex.subtitles.extract_killed_after_term",
+                            windowed=start is not None,
+                        ),
+                    )
                     proc.kill()
                     _, err = proc.communicate()
         finally:

backend/app/plex/sync.py

@@ -15,20 +15,18 @@ dead `enabled` column) is a separate follow-up. Cast + intro/credit markers are
 item-detail/player path (P2), not here.
 """
 
-import logging
 from datetime import datetime, timezone
 
 from sqlalchemy import func, select
 from sqlalchemy.orm import Session
 
-from app import sysconfig
+from app import logs, sysconfig
 from app.models import PlexCollection, PlexItem, PlexLibrary, PlexSeason, PlexShow
 from app.plex import detail, people
 from app.plex.client import PlexClient, PlexError, PlexNotConfigured
 from app.plex.lookups import by_rating_key
 
-log = logging.getLogger("siftlode.plex")
-
+log = logs.get_logger(__name__)
 # Codecs/containers a browser can play in a native <video> without any transcoding.
 _BROWSER_VIDEO = {"h264"}
 _BROWSER_AUDIO = {"aac", "mp3", "opus", "vorbis"}
@@ -102,6 +100,9 @@ def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
                 bd = s.get("bitDepth")
                 try:
                     depth = int(bd) if bd else None
+                # silent-ok: unknown bit depth, which is also what most listings report. The
+                # playback class then falls back to the codec alone, and per-stream noise over a
+                # whole library would drown the sync's own lines.
                 except (TypeError, ValueError):
                     depth = None
                 hdr = str(s.get("colorTrc") or "").lower() in _HDR_TRANSFERS
@@ -112,6 +113,8 @@ def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
 def _epoch(v) -> datetime | None:
     try:
         return datetime.fromtimestamp(int(v), tz=timezone.utc) if v else None
+    # silent-ok: a field Plex leaves out of most listing rows. None is a legitimate stored value,
+    # and every caller treats it as "not known" rather than as an error.
     except (TypeError, ValueError, OSError):
         return None
 
@@ -120,6 +123,7 @@ def _date(v):
     """Parse a Plex 'YYYY-MM-DD' string (originallyAvailableAt) → date, or None."""
     try:
         return datetime.strptime(v, "%Y-%m-%d").date() if v else None
+    # silent-ok: as `_epoch` above — an absent or unparsable air date is stored as no air date.
     except (TypeError, ValueError):
         return None
 
@@ -135,6 +139,7 @@ def _tags(meta: dict, key: str, limit: int | None = None) -> list | None:
 def _rating(meta: dict) -> float | None:
     try:
         return float(meta.get("audienceRating") or meta.get("rating"))
+    # silent-ok: TypeError on `float(None)` is the ordinary path — most items carry no rating.
     except (TypeError, ValueError):
         return None
 
@@ -266,15 +271,19 @@ def sync(db: Session) -> dict:
             # becomes filterable over successive runs (the cheap listing above only stored the top ~3).
             try:
                 stats["people_enriched"] = _enrich_people_batch(db, plex)
-            except Exception as e:  # best-effort — never fail the sync over the backfill
-                log.warning("Plex people enrich batch failed: %s", e)
+            except Exception:  # noqa: BLE001 — best-effort; never fail the sync over a backfill
+                # With the traceback: a blind catch over a batch that fetches AND writes can be
+                # a Plex shape change, a DB error or our own bug, and `%s` told them apart badly.
+                log.exception("the Plex people-enrich batch failed")
             # Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
             # playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
             # successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
             try:
                 stats["media_probed"] = _probe_media_batch(db, plex)
-            except Exception as e:  # best-effort — never fail the sync over the backfill
-                log.warning("Plex media probe batch failed: %s", e)
+            except Exception:  # noqa: BLE001 — best-effort; never fail the sync over a backfill
+                log.exception("the Plex media-probe batch failed")
+    # silent-ok: the return value IS the report. `{"skipped": …}` goes to the scheduler, which
+    # logs the job result, and to the admin route, which answers it to the caller.
     except PlexNotConfigured as e:
         return {"skipped": str(e)}
     except PlexError as e:
@@ -285,8 +294,8 @@ def sync(db: Session) -> dict:
     # appear in ANY item's cast/directors — drop those rows, mirroring how Plex drops orphaned metadata.
     try:
         stats["people_pruned"] = people.prune_orphan_people(db)
-    except Exception as e:  # best-effort housekeeping — never fail the sync over it
-        log.warning("Plex people prune failed: %s", e)
+    except Exception:  # noqa: BLE001 — best-effort housekeeping; never fail the sync over it
+        log.exception("the Plex people prune failed")
     log.info("Plex sync done: %s", stats)
     return stats
 
@@ -325,7 +334,9 @@ def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH)
     for row in rows:
         try:
             meta = plex.metadata(row.rating_key) or {}
-        except Exception as e:  # transient — leave media_probed_at NULL so it retries next run
+        # No traceback and only debug: this is per item over a whole library, and the stack is
+        # the same `plex.metadata` call every time. The retry next run is the handling.
+        except Exception as e:  # noqa: BLE001 — transient; leave media_probed_at NULL to retry
             log.debug("Plex media probe fetch failed for %s: %s", row.rating_key, e)
             continue
         bit_depth, hdr = _video_stream_props(meta)
@@ -372,7 +383,8 @@ def _enrich_people_batch(db: Session, plex: PlexClient, limit: int = _ENRICH_BAT
     for row in [*movies, *shows]:
         try:
             meta = plex.metadata(row.rating_key) or {}
-        except Exception as e:
+        # As in `_probe_media_batch` above: per item, so the message without the stack.
+        except Exception as e:  # noqa: BLE001 — transient; leave people_enriched_at NULL
             log.debug("Plex people enrich fetch failed for %s: %s", row.rating_key, e)
             continue  # leave people_enriched_at NULL so it's retried on a later run
         people.enrich_row_people(db, row, people.cast_from_meta(meta))

backend/app/plex/tmdb.py

@@ -10,7 +10,6 @@ One-parser rule (mirrors app/plex/detail.py): `parse_tmdb` is the pure parser, `
 the row. The enrich batch + scheduler entry live here too so the whole TMDB integration is one module.
 """
 
-import logging
 import time
 from datetime import datetime, timezone
 
@@ -19,12 +18,11 @@ from sqlalchemy import func, select
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.orm import Session
 
-from app import sysconfig
+from app import logs, sysconfig
 from app.config import settings
 from app.models import PlexItem, PlexPerson, PlexShow
 
-log = logging.getLogger("siftlode.tmdb")
-
+log = logs.get_logger(__name__)
 # Politeness throttle: TMDB tolerates ~50 req/s; ~33/s leaves comfortable headroom for a shared host.
 _MIN_INTERVAL = 0.03
 
@@ -142,6 +140,12 @@ def guid_to_tmdb_id(guid: str | None, media_type: str, tmdb: TmdbClient) -> int
         try:
             return int(ext)
         except ValueError:
+            # The caller stamps the watermark on a None, so this title is never looked at again:
+            # one unreadable guid costs it its artwork and its cast permanently.
+            log.info(
+                "a Plex guid named TMDB but did not carry a numeric id",
+                extra=logs.event("plex.tmdb.guid_unparsable", provider=provider),
+            )
             return None
     source = {"imdb": "imdb_id", "tvdb": "tvdb_id"}.get(provider)
     if not source:
@@ -499,7 +503,17 @@ def enrich_person(db: Session, person: PlexPerson) -> bool:
         # doesn't tie up a threadpool worker for the default 30s.
         with TmdbClient(db, timeout=8.0) as tmdb:
             payload = tmdb.fetch_person(person.tmdb_person_id)
-    except TmdbError:
+    except TmdbError as exc:
+        # False leaves the person un-enriched and unstamped, so it is retried — but the page in
+        # front of the user right now shows an empty biography with nothing to say why.
+        log.warning(
+            "the TMDB person lookup failed",
+            extra=logs.event(
+                "plex.tmdb.person_enrich_failed",
+                person_id=person.id,
+                reason=type(exc).__name__,
+            ),
+        )
         return False
     if payload is None:
         person.tmdb_enriched_at = datetime.now(timezone.utc)

backend/app/plex/watch_sync.py

@@ -17,14 +17,17 @@ and the incremental pull resolves a genuine divergence to Plex's state (see `_pu
 running a last-write-wins comparison.
 """
 
-import logging
+# On the PlexError lines in this module: none of them carries a traceback, deliberately. A
+# PlexError is httpx's own message — the method, the URL and the status of one call — and the stack
+# beneath it is always this `except` plus `PlexClient._get`. The attrs beside the line say which
+# user and which phase, which is the part the message cannot know.
 from datetime import datetime, timezone
 
 from sqlalchemy import func, select, update
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.orm import Session
 
-from app import sysconfig
+from app import logs, sysconfig
 from app.db import SessionLocal
 from app.models import PlexItem, PlexLink, PlexState
 from app.plex.client import PlexClient, PlexError, PlexNotConfigured
@@ -36,8 +39,7 @@ from app.plex.sync import (
     _paginate,
 )
 
-log = logging.getLogger("siftlode.plex.watch")
-
+log = logs.get_logger(__name__)
 # A resume position below this many seconds isn't worth mirroring — matches the player's own
 # _PROGRESS_MIN_S (a stray few-second seek is not "in progress").
 _PROGRESS_MIN_S = 5
@@ -148,6 +150,8 @@ def import_owner_watch_state(db: Session, link: PlexLink) -> dict:
             link.initial_import_done = True
             link.last_watch_sync_at = datetime.now(timezone.utc)
             db.commit()
+    # silent-ok: the `{"skipped": …}` return is the report — the admin route answers it and the
+    # scheduler logs it on the job line.
     except PlexNotConfigured as e:
         return {"skipped": str(e)}
     except PlexError as e:
@@ -326,8 +330,18 @@ def _resolve_account_id(db: Session, plex: PlexClient, link: PlexLink) -> int:
             if int(a.get("id") or -1) == 1:
                 username = a.get("name")
                 break
-    except PlexError:
-        pass
+    except PlexError as exc:
+        # The fallback account id 1 is right for the admin-token setup, so sync keeps working —
+        # but the username is never stored, and the UI then has no name to show for the link,
+        # exactly as if the server had not offered one.
+        log.info(
+            "the Plex account list could not be read; the link keeps the default account",
+            extra=logs.event(
+                "plex.watch.accounts_probe_failed",
+                user_id=link.user_id,
+                reason=type(exc).__name__,
+            ),
+        )
     link.plex_account_id = acct_id
     if username:
         link.plex_username = username
@@ -566,6 +580,7 @@ def run_plex_watch_sync(db: Session) -> dict:
                 total["repushed"] += _repush_dirty(db, plex, link)
                 link.last_watch_sync_at = now
                 db.commit()
+        # silent-ok: the `{"skipped": …}` return is the report, as above.
         except PlexNotConfigured as e:
             return {"skipped": str(e)}
         except PlexError as e:
@@ -617,6 +632,7 @@ def run_plex_watch_reconcile(db: Session) -> dict:
                 total["repushed"] += _repush_dirty(db, plex, link)
                 link.last_watch_sync_at = datetime.now(timezone.utc)
                 db.commit()
+        # silent-ok: the `{"skipped": …}` return is the report, as above.
         except PlexNotConfigured as e:
             return {"skipped": str(e)}
         except PlexError as e:

backend/app/ratelimit.py

@@ -17,10 +17,14 @@ from app.config import settings
 
 
 class RateLimiter:
-    def __init__(self, max_events: int, window_seconds: float):
+    """At most `max_events` attempts per `window_seconds` per key — or, when each attempt passes a
+    `cost`, at most that much total cost (a byte budget counts bytes, not calls)."""
+
+    def __init__(self, max_events: float, window_seconds: float):
         self.max_events = max_events
         self.window = window_seconds
-        self._hits: dict[str, list[float]] = {}
+        # key → (monotonic time, cost) of every counted attempt still inside the window
+        self._hits: dict[str, list[tuple[float, float]]] = {}
         self._lock = threading.Lock()
 
     def reset(self) -> None:
@@ -36,27 +40,27 @@ class RateLimiter:
         count."""
         cutoff = time.monotonic() - self.window
         with self._lock:
-            return sum(1 for t in self._hits.get(key, []) if t > cutoff) >= self.max_events
+            return sum(c for t, c in self._hits.get(key, []) if t > cutoff) >= self.max_events
 
-    def allow(self, key: str) -> bool:
-        """Record an attempt for ``key`` and return whether it is within the limit.
+    def allow(self, key: str, cost: float = 1) -> bool:
+        """Record an attempt costing ``cost`` for ``key`` and return whether it is within the limit.
 
-        True  -> under the cap (the attempt is counted).
-        False -> the cap for the current window is already reached (attempt NOT counted, so a
+        True  -> within the cap (the attempt is counted).
+        False -> the attempt would pass the cap for the current window (attempt NOT counted, so a
                  blocked caller can't keep pushing the window forward)."""
         now = time.monotonic()
         cutoff = now - self.window
         with self._lock:
-            hits = [t for t in self._hits.get(key, []) if t > cutoff]
-            if len(hits) >= self.max_events:
+            hits = [(t, c) for t, c in self._hits.get(key, []) if t > cutoff]
+            if sum(c for _, c in hits) + cost > self.max_events:
                 self._hits[key] = hits
                 return False
-            hits.append(now)
+            hits.append((now, cost))
             self._hits[key] = hits
             # Opportunistic cleanup so abandoned keys don't accumulate unboundedly.
             if len(self._hits) > 4096:
                 for k in list(self._hits):
-                    fresh = [t for t in self._hits[k] if t > cutoff]
+                    fresh = [(t, c) for t, c in self._hits[k] if t > cutoff]
                     if fresh:
                         self._hits[k] = fresh
                     else:
@@ -69,6 +73,8 @@ def _norm_ip(s: str) -> ipaddress.IPv4Address | ipaddress.IPv6Address | None:
     plain-IPv4 allowlist still matches a peer that surfaces in mapped form. None for a non-IP string."""
     try:
         ip = ipaddress.ip_address(s.strip())
+    # silent-ok: parsing an untrusted header value, where "not an IP" is a documented answer this
+    # function returns — and a line per request is what an attacker would aim for.
     except ValueError:
         return None
     return ip.ipv4_mapped if getattr(ip, "ipv4_mapped", None) else ip

backend/app/realtime.py

@@ -13,12 +13,12 @@ mutated from a single thread.
 """
 
 import asyncio
-import logging
 from collections import defaultdict
 
 from starlette.websockets import WebSocket
+from app import logs
 
-log = logging.getLogger("siftlode.realtime")
+log = logs.get_logger(__name__)
 
 
 class ConnectionManager:
@@ -42,9 +42,16 @@ class ConnectionManager:
         for ws in list(self._conns.get(user_id, ())):
             try:
                 await ws.send_json(payload)
-            except Exception:
-                # Stale socket; it'll be cleaned up by its own disconnect handler.
-                pass
+            except Exception as exc:  # noqa: BLE001 — one dead socket must not stop the others
+                # Stale socket; it'll be cleaned up by its own disconnect handler. But this is
+                # also where a live update goes missing: the UI simply never changes, and there
+                # is no request behind it whose failure anyone could look up.
+                log.debug(
+                    "a live update could not be pushed to a socket",
+                    extra=logs.event(
+                        "realtime.push_failed", user_id=user_id, reason=type(exc).__name__
+                    ),
+                )
 
     def push(self, user_id: int, payload: dict) -> None:
         """Schedule a push onto the WebSocket event loop. Safe to call from a sync route
@@ -54,7 +61,7 @@ class ConnectionManager:
             return
         try:
             asyncio.run_coroutine_threadsafe(self._push(user_id, payload), loop)
-        except Exception:
+        except Exception:  # noqa: BLE001 — a closed loop or socket must not fail the caller
             log.debug("ws push failed", exc_info=True)
 
 

backend/app/routes/admin.py

@@ -408,6 +408,8 @@ def _serialize_user(u: User) -> dict:
         # confusing sign-in.
         "google_email": u.google_email if u.google_email != u.email else None,
         "created_at": iso(u.created_at),
+        # NULL = no sign-in since the column existed (0085), or the shared demo account.
+        "last_login_at": iso(u.last_login_at),
     }
 
 

backend/app/routes/auth.py

@@ -17,6 +17,7 @@ from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import Session
 
 from app import email as email_mod
+from app import logs
 from app import mailtext
 from app import sysconfig
 from app.auth import (
@@ -386,7 +387,15 @@ async def callback(
         return _complete_link(request, db, link_uid, link_explicit, sub, userinfo, token)
 
     if not is_allowed(db, email):
-        log.warning("Login denied (not approved): %s", email or "<no email>")
+        # No uid to name — having no account here is the branch. And the address is left out on
+        # purpose: `redact_text` turns it into `[email]` before either sink sees it, so a line
+        # whose only variable is always `[email]` says nothing. WHO asked is in the pending
+        # invite recorded just below, which is where an admin looks anyway.
+        log.warning(
+            "Google sign-in denied: address not approved",
+            # log-pii-ok: whether an address came back, not what it was
+            extra=logs.event("auth.login.denied", reason="not_approved", has_email=bool(email)),
+        )
         # A denied Google login doubles as an access request: record it for the admin and
         # land the user on a friendly "request received" screen instead of a raw 403.
         if email:
@@ -419,15 +428,20 @@ async def callback(
             if existing.google_sub is not None and existing.google_sub != sub:
                 # A different Google identity is already bound to this email — refuse rather than
                 # hijack it. (Near-impossible with real Google accounts: email↔sub is stable.)
-                log.warning("Google login email collision (different sub): %s", email)
+                log.warning(
+                    "Google sign-in refused: a different Google identity holds this address",
+                    extra=logs.event("auth.login.denied", reason="sub_collision", uid=existing.id),
+                )
                 return RedirectResponse(url=_access_notice_url("denied"))
             if not userinfo.get("email_verified", False):
                 # Adopting (and activating) a pre-existing password account requires Google to
                 # actually attest the address — otherwise an unverified Google email that merely
                 # collides with a pending registration could seize that account. Defense-in-depth.
                 log.warning(
-                    "Google login: unverified email, refusing to adopt account: %s",
-                    email,
+                    "Google sign-in refused: unverified address, not adopting the account",
+                    extra=logs.event(
+                        "auth.login.denied", reason="email_unverified", uid=existing.id
+                    ),
                 )
                 return RedirectResponse(url=_access_notice_url("denied"))
             user = existing
@@ -445,7 +459,10 @@ async def callback(
     # A suspended account can't sign in by any method. Block before establishing the session and
     # tell the (verified) owner why — a successful Google auth proves they control this account.
     if user.is_suspended:
-        log.info("Suspended account blocked at Google login: %s", email)
+        log.info(
+            "Google sign-in blocked: the account is suspended",
+            extra=logs.event("auth.login.denied", reason="suspended", uid=user.id),
+        )
         _notify_suspended(background, user)
         return RedirectResponse(url="/?login=suspended")
     # Record what Google actually answers with, beside the address the user registered with. The two
@@ -479,17 +496,12 @@ async def callback(
     db.flush()
 
     _store_token(db, user, token)
+    _establish_session(request, user)
     db.commit()
-
-    # Multi-session: remember every account that has authenticated in this browser so the
-    # user can switch between them without going through Google again. Only accounts that
-    # actually completed OAuth here land in this list (it's the proof they may be switched to).
-    accounts = [a for a in (request.session.get("account_ids") or []) if a != user.id]
-    accounts.append(user.id)
-    request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
-    request.session["user_id"] = user.id
-    _remember_epoch(request, user)
-    log.info("Login: %s (id=%s, role=%s)", email, user.id, user.role)
+    log.info(
+        "signed in",
+        extra=logs.event("auth.login.ok", method="google", uid=user.id, role=user.role),
+    )
     return RedirectResponse(url="/")
 
 
@@ -669,19 +681,25 @@ def demo_login(body: EmailIn, request: Request, db: Session = Depends(get_db)) -
         request.session.clear()
         request.session["user_id"] = demo.id
         _remember_epoch(request, demo)
-        log.info("Demo login (key=%s, demo_id=%s)", email, demo.id)
+        # The demo key IS an address, so it is not in the line — see the denied branch above.
+        log.info("signed in", extra=logs.event("auth.login.ok", method="demo", uid=demo.id))
         return {"authenticated": True}
     return {"authenticated": False}
 
 
 def _establish_session(request: Request, user: User) -> None:
-    """Sign `user` in for this browser session, mirroring the Google callback's multi-account
-    handling so password sign-in joins the same account switcher."""
+    """Sign `user` in for this browser session — the ONE sign-in path, shared by the Google callback
+    and password login, so both join the same account switcher and both stamp `last_login_at` (the
+    caller commits). Multi-session: every account that authenticated in this browser is remembered so
+    the user can switch between them; only accounts that actually proved themselves HERE land in that
+    wallet (it is the proof they may be switched to). The demo entry deliberately does not come
+    through here: it clears the wallet instead, and its shared account has no meaningful last login."""
     accounts = [a for a in (request.session.get("account_ids") or []) if a != user.id]
     accounts.append(user.id)
     request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
     request.session["user_id"] = user.id
     _remember_epoch(request, user)
+    user.last_login_at = datetime.now(timezone.utc)
 
 
 def _app_base() -> str:
@@ -883,7 +901,12 @@ def _register_account(email: str, password: str, lang: str | None = None) -> Non
         except IntegrityError:
             # A concurrent registration for the same new email won the race between the pre-check
             # above and this flush (the unique-email constraint fires). Same already-registered
-            # no-op as the pre-check hit — roll back and stop.
+            # no-op as the pre-check hit — roll back and stop. The line carries nothing: the
+            # answer to the caller is deliberately identical either way, and so is this.
+            log.info(
+                "a registration lost the race for its address",
+                extra=logs.event("auth.register.race_lost"),
+            )
             db.rollback()
             return
         upsert_pending_invite(db, email)  # admin-approval gate
@@ -1051,7 +1074,11 @@ def password_login(
     if not user.is_active:
         raise _login_refused(403, "pending_approval", user)
     _establish_session(request, user)
-    log.info("Password login: %s (id=%s, role=%s)", email, user.id, user.role)
+    db.commit()
+    log.info(
+        "signed in",
+        extra=logs.event("auth.login.ok", method="password", uid=user.id, role=user.role),
+    )
     return {"ok": True}
 
 

backend/app/routes/channels.py

@@ -2,7 +2,6 @@
 hidden, tags) and per-channel sync state. Local reads/writes plus YouTube-side writes
 (subscribe / unsubscribe) that require the optional write scope."""
 
-import logging
 from datetime import datetime, timezone
 
 from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
@@ -10,7 +9,7 @@ from pydantic import BaseModel
 from sqlalchemy import func, select, update
 from sqlalchemy.orm import Session
 
-from app import quota
+from app import logs, quota
 from app.auth import admin_user, current_user, require_human, require_write_scope
 from app.db import get_db
 from app.models import (
@@ -35,7 +34,7 @@ from app.youtube.client import YouTubeClient, YouTubeError
 
 router = APIRouter(prefix="/api/channels", tags=["channels"])
 
-log = logging.getLogger("siftlode.api")
+log = logs.get_logger(__name__)
 
 
 @router.get("")
@@ -166,7 +165,15 @@ def discover_channels(
                 user.yt_channel_id = own
                 db.commit()
         except YouTubeError as exc:
-            log.warning("discovery: own-channel resolve failed (user %s): %s", user.id, exc)
+            # No traceback anywhere in this file's YouTube handlers: for an upstream answer
+            # `_upstream_failure` already logged the structured `youtube.api.error`, and the
+            # message names the call, its status and its reason (or, with no answer at all, the
+            # transport failure). The stack under it is this `except` and the client call.
+            log.warning(
+                "discovery: own-channel resolve failed: %s",
+                exc,
+                extra=logs.event("channels.discovery.own_channel_failed", reason=exc.reason),
+            )
 
     rows = _discovery_rows(db, user)
 
@@ -190,7 +197,11 @@ def discover_channels(
                 rows, key=lambda r: (-int(r[1]), r[0].title is None, (r[0].title or "").lower())
             )
         except YouTubeError as exc:
-            log.warning("discovery: channel enrich failed (user %s): %s", user.id, exc)
+            log.warning(
+                "discovery: channel enrich failed: %s",
+                exc,
+                extra=logs.event("channels.discovery.enrich_failed", reason=exc.reason),
+            )
             db.rollback()
 
     return [
@@ -298,7 +309,13 @@ def channel_detail(
             db.commit()
             channel = db.get(Channel, channel_id)
         except YouTubeError as exc:
-            log.warning("channel detail enrich failed (%s): %s", channel_id, exc)
+            log.warning(
+                "channel detail enrich failed: %s",
+                exc,
+                extra=logs.event(
+                    "channels.detail.enrich_failed", channel_id=channel_id, reason=exc.reason
+                ),
+            )
             db.rollback()
     sub = db.execute(
         select(Subscription).where(
@@ -559,8 +576,11 @@ def subscribe(
             # subscription resync fills in the resource id. Any other YouTube error is a real
             # problem — surface it with a clear message (422, not a 502 that the client would
             # mistake for a transient "connection lost").
-            msg = str(exc).lower()
-            if "already exists" in msg or "duplicate" in msg:
+            # On the reason code, never on the prose: the error no longer carries the body (it is
+            # the request quoted back), and a message YouTube rewords would silently turn "already
+            # following" back into a 422 the user cannot act on. `reason` is comma-joined when the
+            # body listed several.
+            if "subscriptionDuplicate" in exc.reason.split(","):
                 yt_sub_id = ""
             else:
                 raise HTTPException(
@@ -574,7 +594,15 @@ def subscribe(
             try:
                 apply_channel_details(db, yt.get_channels([channel_id]))
             except YouTubeError as exc:
-                log.warning("subscribe: stub enrich failed (%s): %s", channel_id, exc)
+                log.warning(
+                    "subscribe: stub enrich failed: %s",
+                    exc,
+                    extra=logs.event(
+                        "channels.subscribe.enrich_failed",
+                        channel_id=channel_id,
+                        reason=exc.reason,
+                    ),
+                )
     db.add(
         Subscription(
             user_id=user.id,

backend/app/routes/config.py

@@ -10,6 +10,7 @@ from sqlalchemy.orm import Session
 
 from app import audit
 from app import email as email_mod
+from app import logs
 from app import mailtext
 from app import sysconfig
 from app.audit import AuditAction
@@ -29,6 +30,14 @@ def _redact(v):
     return _URL_USERINFO.sub(r"\1***@", v) if isinstance(v, str) else v
 
 
+def _apply_runtime(db: Session, key: str) -> None:
+    """Some knobs change how THIS process behaves, not just what a later read returns — they have to
+    take effect now, without a restart. One dispatch for both call sites (set and reset), so the next
+    such key can't be wired into one of them and forgotten in the other."""
+    if key == "log_level":
+        logs.set_level(sysconfig.get_str(db, "log_level"))
+
+
 @router.get("")
 def get_config(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
     return sysconfig.describe(db)
@@ -80,6 +89,7 @@ def set_config(
             after=None if s.secret else {"value": _redact(new)},
         )
         db.commit()
+    _apply_runtime(db, key)
     return sysconfig.describe(db)
 
 
@@ -104,6 +114,7 @@ def reset_config(
         before=None if s.secret else {"value": _redact(old)},
     )
     db.commit()
+    _apply_runtime(db, key)
     return sysconfig.describe(db)
 
 

backend/app/routes/diagnostics.py

@@ -0,0 +1,179 @@
+"""The browser's log, sent while its user has a raised diagnostics level (E22 S5).
+
+The client keeps its log in memory (frontend `lib/log.ts`). Only after the user consented to Részletes
+or Teljes in Settings → Diagnosztika does it stream that log here, in batches, and each batch lands in
+`log_events` beside the server's own rows: `source="web"`, the account from the SESSION (never from
+the body), and the request id each event carries, which is what joins a browser line to the server
+line of the same request.
+
+Nothing from the body is trusted as-is:
+  * only a JSON Content-Type is read, and the whole body is capped before it is parsed, and so are
+    the number of events and each event's attributes;
+  * the pace is capped per account, and so is what an account may store per hour;
+  * everything is redacted AGAIN with the server's own rules — the client redacts at capture, but a
+    modified client would not;
+  * an event below the consented level is dropped (a Részletes consent does not store what only
+    Teljes may carry), and so is the whole batch once the level has lapsed;
+  * the timestamp is clamped to the consent window, so a row cannot dodge the retention job by
+    claiming to come from the future.
+"""
+
+import json
+from datetime import datetime, timedelta, timezone
+from typing import Any, Literal
+
+from fastapi import APIRouter, Depends, HTTPException, Request
+from pydantic import BaseModel, Field, ValidationError
+from sqlalchemy import insert
+from sqlalchemy.orm import Session
+
+from app import diagnostics, logs, logsink
+from app.auth import current_user
+from app.db import get_db
+from app.models import LogEvent, User
+from app.ratelimit import RateLimiter
+
+router = APIRouter(prefix="/api/diagnostics", tags=["diagnostics"])
+
+BODY_MAX_BYTES = 512 * 1024
+EVENTS_MAX = 200
+ATTRS_MAX_BYTES = 16 * 1024
+# The client flushes every few seconds and when its buffer fills; this is room for a burst, not a
+# pace anything legitimate keeps up for a minute.
+_batches = RateLimiter(max_events=30, window_seconds=60)
+# What one account may STORE per hour. The pace cap alone still lets a self-consented account write
+# ~15 MiB a minute into the one shared database, kept for the whole retention window; a real session
+# at Teljes stores a few hundred kB an hour. Past the budget a batch is accepted and not stored (a
+# refusal would only make the client resend it), and one notice per hour says so.
+STORED_BYTES_PER_HOUR = 5 * 1024 * 1024
+_stored_bytes = RateLimiter(max_events=STORED_BYTES_PER_HOUR, window_seconds=3600)
+_budget_notices = RateLimiter(max_events=1, window_seconds=3600)
+
+ClientLevel = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
+
+
+class ClientEvent(BaseModel):
+    ts: datetime
+    level: ClientLevel
+    event: str = Field(max_length=64, pattern=r"^[a-z][a-z0-9_]*(\.[a-z0-9_]+)*$")
+    msg: str = Field(max_length=8_000)
+    attrs: dict[str, Any] | None = None
+    request_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$")
+    tab_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9_-]{1,32}$")
+    version: str | None = Field(default=None, max_length=64)
+
+
+class ClientEventsIn(BaseModel):
+    events: list[ClientEvent] = Field(max_length=EVENTS_MAX)
+
+
+async def _capped_body(request: Request) -> bytes:
+    """The request body, refused the moment it passes BODY_MAX_BYTES while still arriving. Counted,
+    not taken from Content-Length: a declared length can be absent (chunked) or wrong, and counting
+    stops reading at the same point a declared-length check would have."""
+    body = bytearray()
+    async for chunk in request.stream():
+        body += chunk
+        if len(body) > BODY_MAX_BYTES:
+            raise HTTPException(status_code=413, detail={"code": "too_large"})
+    return bytes(body)
+
+
+async def _parsed_events(request: Request) -> ClientEventsIn:
+    """The body as the model — parsed only after the size cap, which is why this is a dependency and
+    not a plain body parameter (FastAPI reads a declared body in full before anything else runs).
+
+    JSON only, by its Content-Type, as a declared body would be. Parsing any body as JSON would let a
+    CORS-simple `text/plain` form post from a same-site page write events into a signed-in user's
+    log; the app's own client always sends `application/json`, the unload beacon included."""
+    media_type = request.headers.get("content-type", "").split(";")[0].strip().lower()
+    if media_type != "application/json" and not media_type.endswith("+json"):
+        raise HTTPException(status_code=415, detail={"code": "json_required"})
+    raw = await _capped_body(request)
+    try:
+        return ClientEventsIn.model_validate_json(raw)
+    except ValidationError as exc:
+        raise HTTPException(status_code=422, detail=json.loads(exc.json())) from exc
+
+
+def _row(item: ClientEvent, user: User, now: datetime, window_start: datetime) -> dict[str, Any]:
+    attrs = logs.redact(item.attrs) if item.attrs else None
+    if attrs is not None:
+        size = len(json.dumps(attrs, ensure_ascii=False, default=str).encode())
+        if size > ATTRS_MAX_BYTES:
+            attrs = {"attrs_dropped_bytes": size}
+    if item.tab_id:
+        attrs = {**(attrs or {}), "tab_id": item.tab_id}
+    ts = item.ts if item.ts.tzinfo else item.ts.replace(tzinfo=timezone.utc)
+    return logsink.to_row(
+        {
+            "ts": min(max(ts, window_start), now),
+            "level": item.level,
+            "sev": logs.SEVERITY[item.level],
+            "source": "web",
+            "logger": "web",
+            "event": item.event,
+            "msg": logs.redact_text(item.msg),
+            "attrs": attrs,
+            "request_id": item.request_id,
+            "user_id": user.id,
+            "version": item.version,
+        }
+    )
+
+
+@router.post("/events")
+def ingest_events(
+    # The account first: dependencies resolve in order, so an anonymous caller is refused before a
+    # byte of its body is read.
+    user: User = Depends(current_user),
+    body: ClientEventsIn = Depends(_parsed_events),
+    db: Session = Depends(get_db),
+) -> dict:
+    level = diagnostics.effective_level(user)
+    if level == diagnostics.BASE:
+        # Lapsed or withdrawn: the client stops streaming when it sees this.
+        raise HTTPException(status_code=409, detail={"code": "diagnostics_off"})
+    if not _batches.allow(str(user.id)):
+        raise HTTPException(status_code=429, detail={"code": "rate_limited"})
+    now = datetime.now(timezone.utc)
+    window_start = now - timedelta(hours=diagnostics.ELEVATION_HOURS)
+    floor = logs.SEVERITY[level]
+    rows = [
+        _row(item, user, now, window_start)
+        for item in body.events
+        if logs.SEVERITY[item.level] >= floor
+    ]
+    if not rows:
+        return {"ok": True, "stored": 0}
+    size = sum(len(json.dumps(row, ensure_ascii=False, default=str).encode()) for row in rows)
+    stored = len(rows)
+    if not _stored_bytes.allow(str(user.id), size):
+        stored = 0
+        if not _budget_notices.allow(str(user.id)):
+            return {"ok": True, "stored": 0}
+        rows = [_budget_notice(user, now)]
+    # Under the erasure guard until the commit: an account erased meanwhile gets no rows back.
+    with logs.erasure_guard():
+        db.execute(insert(LogEvent), logsink.without_erased(rows))
+        db.commit()
+    return {"ok": True, "stored": stored}
+
+
+def _budget_notice(user: User, now: datetime) -> dict[str, Any]:
+    return logsink.to_row(
+        {
+            "ts": now,
+            "level": "WARNING",
+            "sev": logs.SEVERITY["WARNING"],
+            "source": "web",
+            "logger": "web",
+            "event": "diagnostics.budget_spent",
+            # What the budget does, exactly: it is a SLIDING hour, and a refused batch is not
+            # charged to it, so smaller batches keep landing while there is room for them. Saying
+            # "the rest of the hour is not kept" would be a row contradicting the rows beneath it.
+            "msg": "The browser log passed its hourly storage budget; a batch bigger than what is left of it is not kept",
+            "attrs": {"budget_bytes": STORED_BYTES_PER_HOUR},
+            "user_id": user.id,
+        }
+    )

backend/app/routes/downloads.py

@@ -587,6 +587,8 @@ def _clean_url(raw) -> str | None:
     # authority. This one already has a "not a plausible URL" answer, so the typo belongs in it.
     try:
         parsed = urlparse(url)
+    # silent-ok: a typo in a link someone pasted. None is this function's "not a plausible URL"
+    # answer, the caller already has one, and any line here would carry the URL itself.
     except ValueError:
         return None
     if not parsed.netloc:

backend/app/routes/feed.py

@@ -256,6 +256,8 @@ def set_video_state(
                 db.delete(row)
             try:
                 db.commit()
+            # silent-ok: the documented lost race — a concurrent player checkpoint already removed
+            # or moved this row. The state the caller asked for is the state that holds either way.
             except StaleDataError:
                 db.rollback()
         return {"video_id": video_id, "status": "new"}
@@ -298,6 +300,7 @@ def clear_video_state(
         db.delete(row)
         try:
             db.commit()
+        # silent-ok: the row went under the delete, which is the outcome the delete wanted.
         except StaleDataError:
             db.rollback()
     return {"video_id": video_id, "status": "new", "position_seconds": 0}
@@ -355,6 +358,7 @@ def set_video_progress(
             # have removed/updated this row already — tolerate the lost race.
             try:
                 db.commit()
+            # silent-ok: exactly the race the comment above describes; the response stays true.
             except StaleDataError:
                 db.rollback()
         return {"video_id": video_id, "position_seconds": 0}

backend/app/routes/me.py

@@ -1,5 +1,7 @@
+from typing import Any
+
 from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
-from pydantic import BaseModel
+from pydantic import BaseModel, RootModel
 from sqlalchemy import func, select
 from sqlalchemy.orm import Session
 
@@ -15,7 +17,9 @@ from app.auth import (
     optional_current_user,
     purge_user,
 )
+from app import diagnostics
 from app import email as email_mod
+from app import logs
 from app import sysconfig
 from app.db import get_db
 from app.models import Invite, User
@@ -147,6 +151,9 @@ def get_me(
         "pending_activations": pending_activations,
         "email_configured": email_configured,
         "preferences": user.preferences or {},
+        # The consented diagnostics level in force and when it lapses (E22 S5). The client applies
+        # it to its own log and streams that log while it is raised.
+        "diagnostics": diagnostics.state(db, user),
     }
 
 
@@ -188,16 +195,45 @@ def delete_my_account(
     return {"deleted": True}
 
 
+class PreferencesIn(RootModel[dict[str, Any]]):
+    """Free-form UI preferences — a JSON object of any keys (theme, filters, …), merged into the
+    stored ones. A model rather than a bare `dict` parameter so a non-object body is a 422 at the
+    boundary like every other request body."""
+
+
 @router.put("/preferences")
 def update_preferences(
-    preferences: dict,
+    preferences: PreferencesIn,
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
 ) -> dict:
     # Merge so partial updates don't wipe other keys.
     merged = dict(user.preferences or {})
-    merged.update(preferences)
+    merged.update(preferences.root)
     user.preferences = merged
     db.add(user)
     db.commit()
     return {"preferences": merged}
+
+
+class DiagnosticsIn(BaseModel):
+    level: diagnostics.DiagnosticsLevel
+
+
+@router.put("/diagnostics")
+def update_diagnostics(
+    body: DiagnosticsIn,
+    user: User = Depends(current_user),
+    db: Session = Depends(get_db),
+) -> dict:
+    """Raise the diagnostics level (the user consented in Settings → Diagnosztika) or return to the
+    base. The expiry is the server's: a raised level always lapses after
+    `diagnostics.ELEVATION_HOURS`, whatever the client asks. The shared demo account cannot raise
+    it — its log would interleave every visitor's session under one account."""
+    if user.is_demo and body.level != diagnostics.BASE:
+        raise HTTPException(status_code=403, detail={"code": "demo", "reason": "diagnostics"})
+    diagnostics.set_level(user, body.level)
+    db.commit()
+    # The rest of THIS request, and the process's elevation registry, follow the new level now.
+    logs.set_user(user.id, diagnostics.effective_level(user), user.diagnostics_until)
+    return {"ok": True, "diagnostics": diagnostics.state(db, user)}

backend/app/routes/messages.py

@@ -449,6 +449,8 @@ async def messages_ws(ws: WebSocket) -> None:
     try:
         while True:
             await ws.receive_text()
+    # silent-ok: the ordinary way a chat socket ends — a closed tab. The `finally` below is the
+    # whole handling, and nothing failed.
     except WebSocketDisconnect:
         pass
     finally:

backend/app/routes/plex/_shared.py

@@ -1,14 +1,13 @@
 """Shared helpers, serializers and constants for the plex route package."""
 
-import logging
-
 from fastapi import HTTPException
 from sqlalchemy.orm import Session
 
+from app import logs
 from app.models import PlexItem
 from app.plex.lookups import by_rating_key
 
-log = logging.getLogger("siftlode.plex")
+log = logs.get_logger(__name__)
 
 
 def item_or_404(db: Session, rating_key: str) -> PlexItem:

backend/app/routes/plex/detail.py

@@ -9,6 +9,7 @@ from fastapi import APIRouter, Depends, HTTPException
 from sqlalchemy import func, select
 from sqlalchemy.orm import Session
 
+from app import logs
 from app.auth import current_user
 from app.db import get_db
 from app.models import (
@@ -30,6 +31,8 @@ from app.plex.lookups import by_rating_key
 from app.routes.plex._shared import _norm_lang, item_or_404
 from app.routes.plex.collections import _collection_source
 
+log = logs.get_logger(__name__)
+
 router = APIRouter(prefix="/api/plex", tags=["plex"])
 
 
@@ -65,7 +68,21 @@ def _show_enrichment(db: Session, rating_key: str) -> tuple[dict, list]:
                 related_f = ex.submit(plex.related, rating_key)  # swallows its own errors → []
                 meta = meta_f.result() or {}
                 related_raw = related_f.result()
-    except (PlexError, PlexNotConfigured):
+    except PlexNotConfigured:
+        # silent-ok: not a failure but a state — this instance has no Plex server, and the page
+        # is meant to render from the local rows alone.
+        return {}, []
+    except PlexError as exc:
+        # The detail page still renders, from the rows the sync left behind: no cast, no related
+        # strip, no rich detail. Indistinguishable on screen from a show that simply has none.
+        log.warning(
+            "live Plex enrichment failed for a show",
+            extra=logs.event(
+                "plex.detail.show_enrich_failed",
+                rating_key=rating_key,
+                reason=type(exc).__name__,
+            ),
+        )
         return {}, []
     # Don't cache an EMPTY related list: plex.related() swallows a transient failure as [] (same as a
     # genuinely no-related show), so caching it would serve an empty "related" section for the whole TTL
@@ -145,12 +162,26 @@ def show_detail(
             sh.people_enriched_at = datetime.now(timezone.utc)
             try:
                 db.commit()
-            except Exception:
+            except Exception:  # noqa: BLE001 — a failed enrich must not fail the page
+                # The rollback un-does the enrichment AND the `people_enriched_at` stamp, so the
+                # next view tries again — but if the commit keeps failing nobody ever hears, and
+                # the cast section stays empty for good.
+                log.warning(
+                    "the show enrichment could not be committed",
+                    extra=logs.event(
+                        "plex.detail.enrich_commit_failed", rating_key=sh.rating_key, kind="show"
+                    ),
+                )
                 db.rollback()
-    except Exception:
+    except Exception:  # noqa: BLE001 — the local episode list must render whatever Plex does
         # Best-effort only — the local episode list is already assembled, so any live-Plex failure
-        # (not configured, HTTP/JSON shape) must degrade, not 500.
-        pass
+        # (not configured, HTTP/JSON shape) must degrade, not 500. With the traceback, because what
+        # is left in here after the Plex errors above are handled is our own shape bugs.
+        log.warning(
+            "the live part of a show page could not be built",
+            extra=logs.event("plex.detail.show_extras_failed", rating_key=sh.rating_key),
+            exc_info=True,
+        )
 
     imdb_id = sh.imdb_id
     return {
@@ -295,6 +326,14 @@ def _episode_nav(db: Session, it: PlexItem) -> tuple[str | None, str | None]:
     try:
         i = keys.index(it.rating_key)
     except ValueError:
+        # An episode missing from its own show's ordered list — the two came out of the same sync,
+        # so this is a gap in it. The page loses its previous/next arrows and says nothing.
+        log.info(
+            "an episode is not in its show's episode list; no previous/next",
+            extra=logs.event(
+                "plex.detail.episode_nav_missing", rating_key=it.rating_key, show_id=it.show_id
+            ),
+        )
         return None, None
     return (keys[i - 1] if i > 0 else None), (keys[i + 1] if i < len(keys) - 1 else None)
 
@@ -366,8 +405,32 @@ def item_detail(
                         "text": codec not in subtitles.BITMAP_SUBS,
                     }
                 )
-    except Exception:
-        pass  # live extras (markers/streams) are best-effort; any live-Plex failure must degrade, not 500 — playback works without them
+    # Best-effort, but not free: what either branch drops is the subtitle stream list and the intro
+    # markers, so the player offers no subtitles and no skip button — and looks like a file that has
+    # neither.
+    except PlexNotConfigured:
+        # silent-ok: not a failure but a state — no Plex server, and nothing to play from anyway.
+        pass
+    except PlexError as exc:
+        log.warning(
+            "the live extras for an item could not be fetched",
+            extra=logs.event(
+                "plex.detail.extras_failed",
+                rating_key=it.rating_key,
+                kind=it.kind,
+                reason=type(exc).__name__,
+            ),
+        )
+    except Exception:  # noqa: BLE001 — playback works without the extras; never 500 here
+        # Plex answered and WE could not read it — a shape our parsing does not expect. The same
+        # split `show_detail` makes: the stack is the only thing that says where.
+        log.warning(
+            "the live extras for an item could not be read",
+            extra=logs.event(
+                "plex.detail.extras_unreadable", rating_key=it.rating_key, kind=it.kind
+            ),
+            exc_info=True,
+        )
 
     # Lazy enrichment (movies only): the live fetch above carries the FULL cast + rich detail (the sync
     # only stored the listing's top ~3), so fill a not-yet-enriched movie — its cast becomes filterable
@@ -380,7 +443,15 @@ def item_detail(
         it.people_enriched_at = datetime.now(timezone.utc)
         try:
             db.commit()
-        except Exception:
+        except Exception:  # noqa: BLE001 — a failed enrich must not fail the page
+            # Same as the show path above: the info page then serves the three names the listing
+            # sync stored, and the full cast it just fetched is thrown away unreported.
+            log.warning(
+                "the movie enrichment could not be committed",
+                extra=logs.event(
+                    "plex.detail.enrich_commit_failed", rating_key=it.rating_key, kind="movie"
+                ),
+            )
             db.rollback()
 
     hu = lang == "hu"

backend/app/routes/plex/people.py

@@ -1,6 +1,5 @@
 """Plex people: cast/crew filter-chip cards + a single person's filmography page."""
 
-import logging
 import threading
 import time
 
@@ -8,6 +7,7 @@ from fastapi import APIRouter, BackgroundTasks, Depends
 from sqlalchemy import func, or_, select
 from sqlalchemy.orm import Session
 
+from app import logs
 from app.auth import current_user
 from app.db import SessionLocal, get_db
 from app.models import (
@@ -22,7 +22,7 @@ from app.plex.cards import movie_card, show_card
 from app.plex.query import content_libs, split_csv
 
 router = APIRouter(prefix="/api/plex", tags=["plex"])
-log = logging.getLogger("siftlode.plex")
+log = logs.get_logger(__name__)
 
 
 def _resolve_known_for(

backend/app/routes/scheduler.py

@@ -22,6 +22,7 @@ from app.scheduler import (
     trigger_all,
     trigger_job,
 )
+from app.sync.explore import followed_channel
 from app.sync.runner import estimate_deep_backfill
 from app.routes._common import not_found
 
@@ -137,9 +138,17 @@ def get_scheduler(_: User = Depends(admin_user), db: Session = Depends(get_db))
 
     queue = {
         "channels_recent_pending": db.scalar(
-            select(func.count()).select_from(Channel).where(Channel.recent_synced_at.is_(None))
+            # The same selection run_recent_backfill works through, or the card counts channels
+            # the job will never touch and never reaches zero.
+            select(func.count())
+            .select_from(Channel)
+            .where(Channel.recent_synced_at.is_(None), followed_channel())
         )
         or 0,
+        # The whole opted-in BACKLOG, deliberately not `run_deep_backfill`'s next-pass selection
+        # (which also wants `recent_synced_at`): the same figure /api/sync/my-status shows the user,
+        # and a channel missing its recent pull is one the same job tick fetches first, not one the
+        # job will never touch. Narrowing it here would under-report work that is genuinely queued.
         "channels_deep_pending": deep_eta["channels_pending"],
         "deep_eta_seconds": deep_eta["eta_seconds"],
         "videos_pending_enrich": db.scalar(

backend/app/routes/search.py

@@ -13,7 +13,6 @@ this search-discovered "noise" by default. Shorts and currently-live/upcoming vi
 ingested this way (Shorts are confirmed by the youtube.com/shorts probe; live/upcoming are
 dropped from the search response up front)."""
 
-import logging
 from concurrent.futures import ThreadPoolExecutor
 from datetime import datetime, timezone
 
@@ -23,7 +22,7 @@ from sqlalchemy import and_, delete, func, or_, select, update
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.orm import Session, aliased
 
-from app import quota, sysconfig
+from app import logs, quota, sysconfig
 from app.auth import require_human
 from app.db import get_db
 from app.models import (
@@ -45,8 +44,7 @@ from app.youtube import search_scrape
 from app.youtube.client import YouTubeClient, YouTubeError
 from app.youtube.shorts import make_client, probe_is_short
 
-log = logging.getLogger("siftlode.search")
-
+log = logs.get_logger(__name__)
 router = APIRouter(prefix="/api/search", tags=["search"])
 
 # search.list costs this many quota units per page.
@@ -132,6 +130,13 @@ def _ingest_candidates(
             apply_channel_details(db, details)
             db.commit()
         except YouTubeError:
+            # The stubs remain and the details can be backfilled later — but until they are, those
+            # channels sit in the results with no thumbnail and no subscriber count, looking like
+            # channels that have neither.
+            log.warning(
+                "the channels found by a search could not be enriched",
+                extra=logs.event("search.channel_enrich_failed", channels=len(new_channel_ids)),
+            )
             db.rollback()  # details can be backfilled later; the stubs remain
 
     # 2) Video stubs, flagged via_search. Existing rows keep their flags (ON CONFLICT DO
@@ -161,6 +166,12 @@ def _ingest_candidates(
         with quota.attribute(user.id, quota.QuotaAction.VIDEOS_ENRICH):
             detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
     except YouTubeError:
+        # Every candidate then stays unenriched, and an unenriched video cannot be judged live or
+        # Short — so it is dropped. A working search returns nothing and reads as "no results".
+        log.warning(
+            "the videos found by a search could not be enriched",
+            extra=logs.event("search.video_enrich_failed", candidates=len(ids)),
+        )
         detail_by_id = {}
     enriched_at = datetime.now(
         timezone.utc
@@ -231,21 +242,29 @@ def search_youtube(
     limit = max(1, min(limit, 100))
 
     # Source: "scrape" (InnerTube, no API quota) or "api" (search.list, 100 units/page).
-    source = (sysconfig.get_str(db, "search_source") or "scrape").lower()
-    if source != "scrape":
-        source = "api"
+    source = sysconfig.get_str(db, "search_source")
 
     # Per-user daily cap applies to both sources (anti-abuse); with scrape it's the only guard
     # since the search itself spends no quota.
     daily_limit = sysconfig.get_int(db, "search_daily_limit_per_user")
     used = quota.actions_today(db, user.id, quota.QuotaAction.VIDEOS_SEARCH)
     if used >= daily_limit:
+        # The 429 is in the request line either way; which of the two caps produced it is not, and
+        # they are answered by different things — this one by tomorrow, the other by the admin.
+        log.info(
+            "a live search was refused: the per-user daily cap is spent",
+            extra=logs.event("search.refused", reason="daily_cap", limit=daily_limit),
+        )
         raise HTTPException(
             status_code=429,
             detail=f"You've reached today's YouTube search limit ({daily_limit}). Try again tomorrow.",
         )
     # The 100-unit budget pre-check only applies to the API source.
     if source == "api" and not quota.can_spend(db, SEARCH_COST):
+        log.info(
+            "a live search was refused: the shared YouTube budget is spent",
+            extra=logs.event("search.refused", reason="shared_quota", cost=SEARCH_COST),
+        )
         raise HTTPException(
             status_code=429,
             detail="The shared YouTube quota for today is used up. Try again tomorrow.",
@@ -293,7 +312,29 @@ def search_youtube(
                 collected.extend(_ingest_candidates(db, yt, user, candidates))
             except Exception as exc:  # noqa: BLE001 — any ingest failure, not just YouTubeError
                 db.rollback()
-                log.exception("live search ingest failed for %r", term)
+                # The term is NOT in this line. A search string is exactly what the privacy
+                # policy keeps out of the log unless the user turned Teljes/TRACE on for
+                # themselves, and `redact_text` does not cover it — it scrubs addresses and
+                # credential params, not what someone typed into the search box. What the
+                # ERROR line carries instead is the shape of the failure, which is what tells
+                # an ingest bug from an upstream one; the term follows at TRACE, and only
+                # under this user's own consent — not whenever the process runs at TRACE.
+                log.exception(
+                    "live search ingest failed",
+                    extra=logs.event(
+                        "search.live.ingest_failed",
+                        source=source,
+                        candidates=len(candidates),
+                        collected=len(collected),
+                    ),
+                )
+                if logs.consented_trace():
+                    log.log(
+                        logs.TRACE,
+                        "the term the ingest failed on",
+                        # log-pii-ok: only under this user's own Teljes consent
+                        extra=logs.event("search.live.term", term=term),
+                    )
                 if collected:
                     break  # keep what earlier pages already gathered
                 raise HTTPException(

backend/app/scheduler.py

@@ -2,7 +2,6 @@
 recent-first / deep backfill. Runs inside the API process (single worker)."""
 
 import contextvars
-import logging
 import threading
 from datetime import datetime, timezone
 from typing import Callable
@@ -10,7 +9,7 @@ from typing import Callable
 from apscheduler.schedulers.background import BackgroundScheduler
 from sqlalchemy import select
 
-from app import audit, progress, quota
+from app import audit, logs, logsink, progress, quota
 from app.audit import AuditAction
 from app.config import settings
 from app.db import SessionLocal
@@ -35,8 +34,7 @@ from app.sync.runner import (
     run_subscription_resync,
 )
 
-logger = logging.getLogger("siftlode.scheduler")
-
+log = logs.get_logger(__name__)
 _scheduler: BackgroundScheduler | None = None
 
 # Per-job run activity, kept in-memory for the admin Scheduler dashboard to read (the
@@ -72,6 +70,7 @@ JOB_INTERVALS: dict[str, int] = {
     "plex_watch_reconcile": settings.plex_watch_reconcile_interval_min,
     "tmdb_enrich": settings.tmdb_enrich_interval_min,
     "audit_gc": settings.audit_gc_minutes,
+    "log_gc": settings.log_gc_minutes,
 }
 
 # Sane bounds for an admin-set interval (minutes).
@@ -103,7 +102,7 @@ def apply_interval(db, job_id: str, minutes: int) -> int:
     db.commit()
     if _scheduler is not None and _scheduler.get_job(job_id) is not None:
         _scheduler.reschedule_job(job_id, trigger="interval", minutes=minutes)
-        logger.info("rescheduled job %s to every %s min", job_id, minutes)
+        log.info("rescheduled job %s to every %s min", job_id, minutes)
     return minutes
 
 
@@ -130,7 +129,7 @@ def _notify_done(db, actor_id: int, job_id: str, status: str, summary: str | Non
         )
     except Exception:
         db.rollback()
-        logger.exception("failed to write completion notification for job %s", job_id)
+        log.exception("failed to write completion notification for job %s", job_id)
 
 
 def _run_changed(result) -> bool:
@@ -166,7 +165,7 @@ def _audit_run(db, actor_id, job_id, status, summary, result) -> None:
         db.commit()
     except Exception:
         db.rollback()
-        logger.exception("failed to write audit row for job %s", job_id)
+        log.exception("failed to write audit row for job %s", job_id)
 
 
 def _job(name: str, fn) -> None:
@@ -174,7 +173,7 @@ def _job(name: str, fn) -> None:
     # HERE, atomically: whichever reaches this first wins, the other skips. This is the real
     # single-run guard — trigger_job's pre-check is only a fast best-effort answer for the UI.
     if not _claim_running(name):
-        logger.info("job %s already running — skipping this trigger", name)
+        log.info("job %s already running — skipping this trigger", name)
         return
     db = SessionLocal()
     actor_id = _manual_actor.get()  # set only for a manual "run now" trigger
@@ -187,7 +186,7 @@ def _job(name: str, fn) -> None:
 
     try:
         if is_sync_paused(db):
-            logger.info("job %s skipped (sync paused)", name)
+            log.info("job %s skipped (sync paused)", name)
             _record(
                 name,
                 status="skipped",
@@ -202,14 +201,14 @@ def _job(name: str, fn) -> None:
         with progress.bind(sink):
             result = fn(db)
         summary = _summarize(result)
-        logger.info("job %s -> %s", name, result)
+        log.info("job %s -> %s", name, result)
         _record(name, status="ok", last_finished=_now_iso(), last_result=summary, progress=None)
         if actor_id is not None:
             _notify_done(db, actor_id, name, "ok", summary)
         _audit_run(db, actor_id, name, "ok", summary, result)
     except Exception as exc:
         db.rollback()
-        logger.exception("job %s failed", name)
+        log.exception("job %s failed", name)
         err = str(exc) or exc.__class__.__name__
         _record(name, status="error", last_finished=_now_iso(), last_error=err, progress=None)
         if actor_id is not None:
@@ -381,6 +380,19 @@ def _audit_gc_job() -> None:
     _job("audit_gc", work)
 
 
+def _log_gc_job() -> None:
+    # Prune log_events rows older than log_retention_days — the limit the privacy policy states, so
+    # this runs hourly rather than daily. Pure DB, no quota.
+    def work(db):
+        from app import sysconfig
+
+        reaped = logsink.gc(db, sysconfig.get_int(db, "log_retention_days"))
+        db.commit()
+        return {"reaped": reaped}
+
+    _job("log_gc", work)
+
+
 # job_id -> wrapper. The single source of truth for which jobs exist and how to run one,
 # shared by start_scheduler (recurring registration) and trigger_job (manual "run now").
 JOB_FUNCS: dict[str, Callable[[], None]] = {
@@ -400,6 +412,7 @@ JOB_FUNCS: dict[str, Callable[[], None]] = {
     "plex_watch_reconcile": _plex_watch_reconcile_job,
     "tmdb_enrich": _tmdb_enrich_job,
     "audit_gc": _audit_gc_job,
+    "log_gc": _log_gc_job,
 }
 
 
@@ -478,7 +491,7 @@ def start_scheduler() -> None:
         scheduler.add_job(fn, "interval", minutes=iv[job_id], id=job_id)
     scheduler.start()
     _scheduler = scheduler
-    logger.info("scheduler started")
+    log.info("scheduler started")
 
 
 def shutdown_scheduler() -> None:

backend/app/security.py

@@ -1,10 +1,14 @@
 import hashlib
 
 from argon2 import PasswordHasher
+from argon2.exceptions import VerifyMismatchError
 from cryptography.fernet import Fernet
 
+from app import logs
 from app.config import settings
 
+log = logs.get_logger(__name__)
+
 # argon2id with the library defaults (sensible memory/time cost). One shared hasher instance.
 _ph = PasswordHasher()
 
@@ -24,8 +28,16 @@ def verify_password(password: str, hashed: str | None) -> bool:
         return False
     try:
         return _ph.verify(hashed, password)
-    except Exception:
-        # Any failure (mismatch, malformed hash) is a non-match — never raise to the caller.
+    except VerifyMismatchError:
+        return False  # silent-ok: a wrong password. The commonest path through this function.
+    except Exception as exc:  # noqa: BLE001 — never raise to the caller, whatever went wrong
+        # Everything else is a hash this process cannot USE — truncated in the column, written by
+        # a different argon2 build. It is answered as "wrong password", so the owner of that
+        # account is locked out by a bug and told they mistyped.
+        log.warning(
+            "a stored password hash could not be verified",
+            extra=logs.event("auth.password.hash_unusable", reason=type(exc).__name__),
+        )
         return False
 
 
@@ -60,5 +72,13 @@ def decrypt_optional(value: str | None) -> str | None:
         return None
     try:
         return _require_fernet().decrypt(value.encode()).decode()
-    except Exception:
+    except Exception as exc:  # noqa: BLE001 — callers fall back to a refresh, never to an error
+        # Info rather than warning because the docstring's legacy-plaintext case comes through
+        # here too and is harmless. What is NOT harmless looks identical: a rotated
+        # TOKEN_ENCRYPTION_KEY turns every stored token unreadable at once, and every account
+        # then reads as "not connected" with nothing saying the key is the reason.
+        log.info(
+            "a stored ciphertext could not be decrypted",
+            extra=logs.event("security.decrypt.failed", reason=type(exc).__name__),
+        )
         return None

backend/app/state.py

@@ -1,15 +1,14 @@
 """Global admin-controlled app state (e.g. pausing background sync, first-run setup)."""
 
-import logging
 import secrets
 
 from sqlalchemy.orm import Session
 
+from app import logs
 from app.models import AppState
 from app.security import hash_token
 
-log = logging.getLogger("siftlode.state")
-
+log = logs.get_logger(__name__)
 # Once configuration completes in this process it never reverts, so cache it to skip the per-request
 # DB hit in the setup-mode gate (see main.setup_gate). Starts False; refreshed from the DB.
 _configured_cache = False

backend/app/sync/autotag.py

@@ -6,17 +6,16 @@ topicDetails categories and the channel's dominant video category. System tags a
 regenerated freely; user tags are never touched here.
 """
 
-import logging
 import re
 
 from sqlalchemy import and_, exists, func, select
 from sqlalchemy.orm import Session
 
-log = logging.getLogger("siftlode.autotag")
-
-from app import progress, sysconfig
+from app import logs, progress, sysconfig
 from app.models import Channel, ChannelTag, Tag, Video
 
+log = logs.get_logger(__name__)
+
 # --- language id (offline, deterministic, small) ---
 try:
     from py3langid.langid import MODEL_FILE, LanguageIdentifier
@@ -29,7 +28,13 @@ try:
     def _restrict(codes: list[str]) -> None:
         _identifier.set_languages(codes)
 
-except Exception:  # pragma: no cover - fallback to module-level API
+except Exception:  # noqa: BLE001 — pragma: no cover, fallback to the module-level API
+    # The fallback classifies without the normalisation the identifier applies, so language tags
+    # come out subtly different — and nothing else in the app would ever hint at which path ran.
+    log.info(
+        "the langid model could not be loaded; using the module-level classifier",
+        extra=logs.event("autotag.langid.fallback"),
+    )
     import py3langid as _langid
 
     def _classify(text: str):

backend/app/sync/explore.py

@@ -8,13 +8,12 @@ stay private to users who explored (or subscribe to) the channel — the feed qu
 otherwise the cleanup job hard-deletes the channel and its untouched ephemeral videos after a
 grace period (mirrors the maintenance job's grace-then-delete safety)."""
 
-import logging
 from datetime import timedelta
 
-from sqlalchemy import delete, select
+from sqlalchemy import delete, or_, select
 from sqlalchemy.orm import Session
 
-from app import sysconfig
+from app import logs, sysconfig
 from app.models import (
     Channel,
     ExploredChannel,
@@ -28,7 +27,24 @@ from app.sync.videos import insert_stubs, stub_from_playlist_item, apply_video_d
 from app.utils import now_utc as _now
 from app.youtube.client import YouTubeClient
 
-log = logging.getLogger("siftlode.explore")
+log = logs.get_logger(__name__)
+
+
+def followed_channel():
+    """WHERE clause: a channel somebody follows — a subscription, or an exploration still inside its
+    grace window (`purge_unkept_explored` deletes the expired ExploredChannel rows first). ONE
+    definition for both halves of the question: what the scheduled RSS poll and recent backfill cover
+    (sync/runner.py) and what the explore purge may delete.
+
+    The poll and backfill cover only these (owner decision, 2026-09-17). They used to cover EVERY
+    channel row, but rows also come from search results, playlist sync and dropped subscriptions, and
+    nothing deletes those: on prod 582 of 979 polled channels had no follower, three of them dead
+    feeds answering 404 on every poll. The cost of narrowing is deliberate: the shared Library stops
+    receiving new uploads from channels nobody follows."""
+    return or_(
+        select(Subscription.id).where(Subscription.channel_id == Channel.id).exists(),
+        select(ExploredChannel.id).where(ExploredChannel.channel_id == Channel.id).exists(),
+    )
 
 
 def _enrich_new(db: Session, yt: YouTubeClient, video_ids: list[str]) -> int:
@@ -98,12 +114,8 @@ def purge_unkept_explored(db: Session) -> dict:
     ).rowcount
     db.commit()
 
-    sub_exists = select(Subscription.id).where(Subscription.channel_id == Channel.id).exists()
-    exp_exists = select(ExploredChannel.id).where(ExploredChannel.channel_id == Channel.id).exists()
     orphan_ids = (
-        db.execute(
-            select(Channel.id).where(Channel.from_explore.is_(True), ~sub_exists, ~exp_exists)
-        )
+        db.execute(select(Channel.id).where(Channel.from_explore.is_(True), ~followed_channel()))
         .scalars()
         .all()
     )

backend/app/sync/maintenance.py

@@ -20,13 +20,12 @@ Quota: phase 1 costs ~(flagged / 50) units, phase 2 ~(batch / 50). Both respect
 backfill reserve as the other jobs so interactive syncs never starve.
 """
 
-import logging
 from datetime import timedelta
 
 from sqlalchemy import select
 from sqlalchemy.orm import Session
 
-from app import progress, quota, sysconfig
+from app import logs, progress, quota, sysconfig
 from app.config import settings
 from app.models import Playlist, PlaylistItem, Video
 from app.notifications import create_notification
@@ -35,8 +34,7 @@ from app.sync.videos import apply_video_details, parse_dt
 from app.utils import now_utc as _now
 from app.youtube.client import YouTubeClient
 
-log = logging.getLogger("siftlode.maintenance")
-
+log = logs.get_logger(__name__)
 # Reasons recorded on Video.unavailable_reason. "removed" = absent from videos.list
 # (deleted or made private); "abandoned" = an upcoming premiere that never went live.
 GRACE_DAYS_BY_REASON = {

backend/app/sync/playlists.py

@@ -8,7 +8,6 @@ ingested (with a stub channel) so the mirror is faithful."""
 
 import hashlib
 import json
-import logging
 from collections.abc import Callable
 from dataclasses import dataclass
 from datetime import datetime, timezone
@@ -17,15 +16,14 @@ from typing import TypeVar
 from sqlalchemy import delete, select
 from sqlalchemy.orm import Session
 
-from app import progress, quota
+from app import logs, progress, quota
 from app.auth import has_read_scope
 from app.models import Channel, Playlist, PlaylistItem, User, Video
 from app.sync.runner import token_users
 from app.sync.videos import apply_video_details, parse_dt
 from app.youtube.client import YouTubeClient, YouTubeError
 
-log = logging.getLogger("siftlode.sync")
-
+log = logs.get_logger(__name__)
 # Cost of a single YouTube write op (playlists/playlistItems insert/update/delete).
 WRITE_UNIT_COST = 50
 
@@ -59,7 +57,17 @@ def _load_baseline(pl: Playlist) -> tuple[list[str] | None, set[str]]:
         return None, set()
     try:
         val = json.loads(pl.synced_video_ids)
-    except (ValueError, TypeError):
+    except (ValueError, TypeError) as exc:
+        # No baseline means the three-way merge has no "what we last agreed on" to subtract, so
+        # it degrades to a union: nothing is lost, but a delete made on either side comes back.
+        log.warning(
+            "the sync baseline for a playlist could not be read; merging as a union",
+            extra=logs.event(
+                "sync.playlist.baseline_unreadable",
+                playlist_id=pl.id,
+                reason=type(exc).__name__,
+            ),
+        )
         return None, set()
     if isinstance(val, list):
         return [str(v) for v in val], set()
@@ -410,8 +418,11 @@ class _Breaker:
     write site can't accidentally implement two thirds of the protocol (forgetting to record a
     failure would silently disable the breaker for that operation)."""
 
-    def __init__(self, limit: int = _MAX_CONSECUTIVE_FAILURES) -> None:
+    def __init__(
+        self, limit: int = _MAX_CONSECUTIVE_FAILURES, playlist_id: int | None = None
+    ) -> None:
         self.limit = limit
+        self.playlist_id = playlist_id  # for the failure line only
         self.streak = 0
         self.tripped = False
 
@@ -432,7 +443,9 @@ class _Breaker:
                 self.limit,
             )
 
-    def run(self, op: Callable[[], T], on_fail: Callable[[], None]) -> T | _Failed:
+    def run(
+        self, op: Callable[[], T], on_fail: Callable[[], None], action: str = "write"
+    ) -> T | _Failed:
         """Attempt one YouTube write. Returns the op's result, or `FAILED` when the breaker is
         already tripped (op not called) or the op raised YouTubeError. `on_fail` records the item
         in the caller's failure list either way."""
@@ -441,7 +454,19 @@ class _Breaker:
             return FAILED
         try:
             out = op()
-        except YouTubeError:
+        except YouTubeError as exc:
+            # The client logged the call, but not which playlist it was for or what kind of write
+            # — and the caller's failure list only ever reaches the user as a count. Not the video:
+            # what is in someone's playlist is a row of their history, the same as a download.
+            log.debug(
+                "a playlist write to YouTube failed",
+                extra=logs.event(
+                    "sync.playlist.item_push_failed",
+                    playlist_id=self.playlist_id,
+                    action=action,
+                    reason=exc.reason,
+                ),
+            )
             on_fail()
             self.hit(False)
             return FAILED
@@ -460,7 +485,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
     local = _local_video_ids(db, pl)
     inserted = deleted = reordered = created = pulled = removed_local = 0
     failures: list[str] = []
-    breaker = _Breaker()
+    breaker = _Breaker(playlist_id=pl.id)
     with YouTubeClient(db, user) as yt:
         if not pl.yt_playlist_id:
             pl.yt_playlist_id = yt.create_playlist(pl.name)
@@ -471,6 +496,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
                 added = breaker.run(
                     lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
                     lambda v=vid: failures.append(v),
+                    "add",
                 )
                 if not isinstance(added, _Failed):
                     inserted += 1
@@ -533,7 +559,17 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
         if snippet is not None and (snippet.get("title") or "") != pl.name:
             try:
                 yt.update_playlist_title(pl.yt_playlist_id, pl.name)
-            except YouTubeError:
+            except YouTubeError as exc:
+                # All the caller passes on is the word "title", which says the rename did not
+                # happen but not that YouTube refused it, or why.
+                log.debug(
+                    "a playlist rename did not reach YouTube",
+                    extra=logs.event(
+                        "sync.playlist.title_push_failed",
+                        playlist_id=pl.id,
+                        reason=exc.reason,
+                    ),
+                )
                 failures.append("title")
         # Propagate genuine local deletes.
         for vid in mp.remove_from_yt:
@@ -543,6 +579,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
             dropped = breaker.run(
                 lambda i=item_id: yt.delete_playlist_item(i),
                 lambda v=vid: failures.append(v),
+                "delete",
             )
             if not isinstance(dropped, _Failed):
                 deleted += 1
@@ -552,6 +589,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
             item_id = breaker.run(
                 lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
                 lambda v=vid: failures.append(v),
+                "add",
             )
             if not isinstance(item_id, _Failed):
                 item_id_by_vid[vid] = item_id
@@ -571,6 +609,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
                     item_id_by_vid[w], pl.yt_playlist_id, w, pos
                 ),
                 lambda w=want: failures.append(w),
+                "move",
             )
             if isinstance(moved, _Failed):
                 continue

backend/app/sync/runner.py

@@ -7,16 +7,18 @@ public reads.
 
 import logging
 import threading
+from collections import Counter
 from concurrent.futures import ThreadPoolExecutor, as_completed
 
 from sqlalchemy import select
 from sqlalchemy.orm import Session
 
-from app import progress, quota, sysconfig
+from app import logs, progress, quota, sysconfig
 
-log = logging.getLogger("siftlode.sync")
+log = logs.get_logger(__name__)
 from app.db import SessionLocal
 from app.models import Channel, OAuthToken, Subscription, User
+from app.sync.explore import followed_channel
 from app.sync.subscriptions import import_subscriptions
 from app.sync.videos import (
     apply_rss_feed,
@@ -32,6 +34,8 @@ from app.youtube.rss import RssError, fetch_channel_feed
 
 # RSS feeds are free and network-bound, so fetch many at once; DB writes stay single-threaded.
 RSS_POLL_WORKERS = 16
+# Failed channels named in a poll's summary warning; the rest are counted by reason.
+RSS_FAILURE_EXAMPLES = 5
 
 
 def token_users(db: Session) -> list[User]:
@@ -52,7 +56,7 @@ def get_service_user(db: Session) -> User | None:
 
 
 def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str, int]:
-    """Poll every channel's free RSS feed. Returns `{"new": inserted, "failed": channels_that_did
+    """Poll the free RSS feed of every followed channel (`followed_channel`). Returns `{"new": inserted, "failed": channels_that_did
     _not_complete}` — the failure count is part of the RESULT, not just a log line, so the scheduler
     card and the audit summary can tell "no new uploads" apart from "nothing worked" (an egress
     outage, or a database that can't accept writes, otherwise renders identically to a quiet hour).
@@ -67,7 +71,7 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
     carries the FIRST underlying error, because on a small instance "all channels failed" is also
     what a single dead feed id looks like."""
     if channels is None:
-        channels = db.execute(select(Channel)).scalars().all()
+        channels = db.execute(select(Channel).where(followed_channel())).scalars().all()
     total = len(channels)
     new = 0
     done = 0
@@ -76,6 +80,27 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
     # outage from one permanently dead feed id — which is the whole difference on an instance with
     # a single subscription, where BOTH render as 100% failed.
     first_error: str | None = None
+    # What failed, as counts per reason (`http_404`, `transport`, `apply`, …) plus a few examples —
+    # the ONE warning a poll writes. One line per failed channel was what turned YouTube's RSS outage
+    # of 2026-09-16 into 11k warnings in four hours; each channel's own line is DEBUG now.
+    reasons: Counter[str] = Counter()
+    examples: list[str] = []
+
+    def fail(reason: str, detail: str, *, unexpected: str | None = None) -> None:
+        """Count a channel that did not get polled. `unexpected` is the message for a failure that
+        is a bug on our side: its FIRST occurrence per reason gets a traceback at ERROR, the rest
+        DEBUG — a read-only database fails every channel the same way, and 979 identical tracebacks
+        tell nobody anything the first one didn't. Call it from inside the `except` block."""
+        nonlocal failed, first_error
+        if unexpected is not None:
+            level = logging.ERROR if reasons[reason] == 0 else logging.DEBUG
+            log.log(level, unexpected, exc_info=True)
+        failed += 1
+        reasons[reason] += 1
+        first_error = first_error or detail
+        if len(examples) < RSS_FAILURE_EXAMPLES:
+            examples.append(detail)
+
     # Fetch feeds concurrently (network-bound, no DB), then apply inserts on this thread as
     # each fetch returns — a SQLAlchemy session isn't thread-safe. Pass the channel id (a
     # plain str) to the workers; the ORM instance is touched only here on the main thread.
@@ -90,31 +115,47 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
             # A failed poll is NOT an empty feed: skip the apply so `last_rss_at` stays where it
             # was (stamping it made a permanently-broken feed look freshly polled). The shared
             # counter/progress tail lives after the statement, so the two outcomes can stay two
-            # clauses: an RssError is EXPECTED (404, timeout) and gets one warning line; anything
-            # else is a bug on our side and earns a traceback.
+            # clauses: an RssError is EXPECTED (404, timeout) and is only counted; anything else is a
+            # bug on our side and earns a traceback (the first of its kind per poll).
             except RssError as exc:
-                log.warning("RSS poll skipped for channel %s: %s", channel.id, exc)
-                failed += 1
-                first_error = first_error or f"{channel.id}: {exc}"
-            except Exception as exc:
-                log.exception("RSS fetch failed for channel %s", channel.id)
-                failed += 1
-                first_error = first_error or f"{channel.id}: {exc}"
+                log.debug("RSS poll skipped for channel %s: %s", channel.id, exc)
+                fail(exc.reason, f"{channel.id}: {exc}")
+            # silent-ok: `fail(unexpected=…)` is the report — it logs at ERROR with the
+            # traceback, once per reason per poll, which is the flood control this loop needs.
+            except Exception as exc:  # noqa: BLE001 — one channel must not stop the poll
+                fail(
+                    "error", f"{channel.id}: {exc}", unexpected=f"RSS fetch failed for {channel.id}"
+                )
             else:
                 try:
                     new += apply_rss_feed(db, channel, entries)
-                except Exception as exc:
+                # silent-ok: `fail(unexpected=…)` below is the report — ERROR with the traceback,
+                # once per reason per poll.
+                except Exception as exc:  # noqa: BLE001 — one channel must not stop the poll
                     # A write failure (read-only DB, failover) leaves the channel just as un-polled
                     # as an unreachable feed — count it, or a wholly broken run reports "failed=0".
                     db.rollback()
-                    failed += 1
-                    first_error = first_error or f"{channel.id} (apply): {exc}"
-                    log.exception("RSS apply failed for channel %s", channel.id)
+                    fail(
+                        "apply",
+                        f"{channel.id} (apply): {exc}",
+                        unexpected=f"RSS apply failed for {channel.id}",
+                    )
             # One place that advances the counter, whatever happened above.
             done += 1
             progress.report(done, total, "rss_poll")
     if failed:
-        log.warning("RSS poll: %d/%d channels could not be polled", failed, total)
+        log.warning(
+            "RSS poll: %d/%d channels could not be polled",
+            failed,
+            total,
+            extra=logs.event(
+                "sync.rss.failed",
+                failed=failed,
+                total=total,
+                reasons=dict(reasons),
+                examples=examples,
+            ),
+        )
     if total and failed == total:
         # Nothing was polled at all — an egress outage (the tinyproxy-after-reboot incident), a
         # dead DNS, a read-only database. Returning normally would record the run as `ok` with the
@@ -174,7 +215,11 @@ def run_recent_backfill(
         return {"channels_processed": 0, "videos_added": 0, "reason": "no service user"}
     if channels is None:
         channels = (
-            db.execute(select(Channel).where(Channel.recent_synced_at.is_(None))).scalars().all()
+            db.execute(
+                select(Channel).where(Channel.recent_synced_at.is_(None), followed_channel())
+            )
+            .scalars()
+            .all()
         )
     if max_channels:
         channels = channels[:max_channels]
@@ -204,7 +249,8 @@ def run_deep_backfill(db: Session, max_channels: int = 10, max_pages: int = 5) -
     if user is None:
         return {"channels_processed": 0, "videos_added": 0, "reason": "no service user"}
     # Demand-driven: only deep-backfill channels at least one user has opted into
-    # (Subscription.deep_requested). Recent backfill stays unconditional (cheap).
+    # (Subscription.deep_requested). Recent backfill needs no opt-in: it covers every followed channel
+    # (`followed_channel`).
     deep_requested = (
         select(Subscription.id)
         .where(

backend/app/sync/subscriptions.py

@@ -1,21 +1,23 @@
 """Import the authenticated user's YouTube subscriptions and channel metadata."""
 
-import logging
 from datetime import datetime, timezone
 
 from sqlalchemy import select
 from sqlalchemy.orm import Session
 
+from app import logs
 from app.models import Channel, Subscription, User
 from app.utils import normalize_description
 from app.youtube.client import YouTubeClient, best_thumbnail
 
-log = logging.getLogger("siftlode.sync")
+log = logs.get_logger(__name__)
 
 
 def _to_int(value) -> int | None:
     try:
         return int(value) if value is not None else None
+    # silent-ok: an upstream count we cannot read is stored as no count, which the card renders as
+    # absent rather than as zero. Per-channel over a whole subscription list, so it must stay quiet.
     except (TypeError, ValueError):
         return None
 
@@ -25,6 +27,8 @@ def _parse_dt(value: str | None) -> datetime | None:
         return None
     try:
         return datetime.fromisoformat(value.replace("Z", "+00:00"))
+    # silent-ok: as `_to_int` above — an unreadable upstream date becomes NULL, and every caller
+    # already treats a missing date as unknown.
     except ValueError:
         return None
 

backend/app/sync/videos.py

@@ -39,6 +39,8 @@ def parse_dt(value: str | None) -> datetime | None:
         return None
     try:
         return datetime.fromisoformat(value.replace("Z", "+00:00"))
+    # silent-ok: a pure parser with no video to name — the only thing it could put in a line is
+    # the upstream string itself. None is stored, and absent is how the UI renders it.
     except ValueError:
         return None
 

backend/app/sysconfig.py

@@ -8,26 +8,28 @@ stored Fernet-encrypted at rest (requires TOKEN_ENCRYPTION_KEY; without it, secr
 stored in the DB and stay env-only).
 """
 
-import logging
 from dataclasses import dataclass
 
 from sqlalchemy.orm import Session
 
-from app import security
-from app.config import settings
+from app import logs, security
+from app.config import Settings, canonical_choice, choices_of, settings
 from app.models import SystemConfig
 
-log = logging.getLogger("siftlode.config")
-
+log = logs.get_logger(__name__)
 # Config keys whose stored ciphertext failed to decrypt — tracked so the warning is logged once per
 # key, not once per request (bounded by the number of secret config keys).
 _decrypt_warned: set[str] = set()
+# The same for a stored value that no longer coerces (a choice list that shrank in a release): it
+# fails on EVERY read of a hot knob until someone saves it again. Keyed with the value, so saving a
+# different bad value is reported afresh (bounded by the keys times the values ever saved).
+_coerce_warned: set[tuple[str, str]] = set()
 
 
 @dataclass(frozen=True)
 class ConfigSpec:
     key: str
-    type: str  # "int" | "str" | "bool"
+    type: str  # "int" | "str" | "bool" | "choice"
     group: str  # admin-UI grouping (logically related keys stay together)
     default_attr: str  # attribute on `settings` holding the env/config default
     secret: bool = False
@@ -35,16 +37,33 @@ class ConfigSpec:
     max: int | None = None
     allow_empty: bool = False  # str keys: a stored "" is an explicit empty value, not "unset"
 
+    @property
+    def choices(self) -> tuple[str, ...]:
+        """A choice key's allowed values, read from its setting's Literal type in config.py — the one
+        place the list is written. The admin page renders them as a picker."""
+        return choices_of(Settings.model_fields[self.default_attr].annotation)
+
 
 # Registry of DB-overridable keys. Add a key here + wire its read through get_*() to move it
-# off env into the admin UI; the admin page renders it automatically from this list.
+# off env into the admin UI; the admin page renders it automatically from this list. The ORDER is the
+# admin page's tab order (first appearance of each group), so keep a group's keys together.
 SPECS: tuple[ConfigSpec, ...] = (
+    # --- Access / registration ---
+    ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
     # --- Email / SMTP (one logical group; the password is the only secret) ---
     ConfigSpec("smtp_host", "str", "email", "smtp_host"),
     ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535),
     ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True),
     ConfigSpec("smtp_from", "str", "email", "smtp_from"),
     ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True),
+    # --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
+    ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
+    # Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
+    ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
+    # --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
+    #     install wizard / Configuration page so no restart is needed when the creds change. ---
+    ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
+    ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
     # --- Quota (shared YouTube Data API daily budget) ---
     ConfigSpec("quota_daily_budget", "int", "quota", "quota_daily_budget", min=1, max=1_000_000),
     ConfigSpec(
@@ -59,7 +78,7 @@ SPECS: tuple[ConfigSpec, ...] = (
         max=100_000,
     ),
     # Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
-    ConfigSpec("search_source", "str", "quota", "search_source"),
+    ConfigSpec("search_source", "choice", "quota", "search_source"),
     # --- Backfill (recent-first first pass per channel) ---
     ConfigSpec(
         "backfill_recent_max_videos",
@@ -96,16 +115,6 @@ SPECS: tuple[ConfigSpec, ...] = (
     ConfigSpec("explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650),
     # Days an un-kept live-search result video is kept before the cleanup job removes it.
     ConfigSpec("search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650),
-    # --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
-    ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
-    # Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
-    ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
-    # --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
-    #     install wizard / Configuration page so no restart is needed when the creds change. ---
-    ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
-    ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
-    # --- Access / registration ---
-    ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
     # --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
     ConfigSpec("download_total_max_bytes", "int", "downloads", "download_total_max_bytes", min=0),
     ConfigSpec(
@@ -137,7 +146,7 @@ SPECS: tuple[ConfigSpec, ...] = (
     ConfigSpec(
         "download_gc_grace_days", "int", "downloads", "download_gc_grace_days", min=0, max=3_650
     ),
-    ConfigSpec("download_layout", "str", "downloads", "download_layout"),
+    ConfigSpec("download_layout", "choice", "downloads", "download_layout"),
     ConfigSpec(
         "download_worker_concurrency",
         "int",
@@ -150,6 +159,13 @@ SPECS: tuple[ConfigSpec, ...] = (
     # --- Audit log ---
     # Days to keep admin audit-log rows before the audit_gc job prunes them (0 = keep forever).
     ConfigSpec("audit_retention_days", "int", "audit", "audit_retention_days", min=0, max=3650),
+    # --- Diagnostics (E22) ---
+    # Runtime log level for the API process. Applied without a restart — see routes/config.py and
+    # main.py's lifespan.
+    ConfigSpec("log_level", "choice", "diagnostics", "log_level"),
+    # Days the log_events table keeps a row (the log_gc job prunes). No "0 = forever": stored logs
+    # carry account-linked request history, and the privacy policy names a limit.
+    ConfigSpec("log_retention_days", "int", "diagnostics", "log_retention_days", min=1, max=90),
     # --- Plex integration (optional; local-file playback + Plex metadata) ---
     ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled"),
     ConfigSpec("plex_server_url", "str", "plex", "plex_server_url"),
@@ -159,10 +175,10 @@ SPECS: tuple[ConfigSpec, ...] = (
     ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
     # (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
     ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
-    # Transcode re-encode tuning (E6 S1). hw = "auto"|"on"|"off" (free-text, like search_source); the
-    # libx264 knobs apply only to the software fallback. crf 0–51 (x264 range); max_height 0 = source.
-    ConfigSpec("plex_transcode_hw", "str", "plex", "plex_transcode_hw"),
-    ConfigSpec("plex_transcode_preset", "str", "plex", "plex_transcode_preset"),
+    # Transcode re-encode tuning (E6 S1). The libx264 knobs apply only to the software fallback.
+    # crf 0–51 (x264 range); max_height 0 = source.
+    ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw"),
+    ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset"),
     ConfigSpec("plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51),
     ConfigSpec(
         "plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
@@ -194,11 +210,23 @@ def _default(s: ConfigSpec):
     return getattr(settings, s.default_attr)
 
 
+def _choice(s: ConfigSpec, raw: object) -> str:
+    """The canonical spelling of `raw` among the key's choices (any letter case), or ValueError."""
+    option = canonical_choice(s.choices, raw)
+    if option is None:
+        raise ValueError(f"{s.key} must be one of: {', '.join(s.choices)}")
+    return option
+
+
 def _coerce(s: ConfigSpec, raw: str):
     if s.type == "int":
         return int(raw)
     if s.type == "bool":
         return raw == "true"
+    if s.type == "choice":
+        # A stored value that is no longer a choice (the list shrank) reads as the default, like a
+        # non-numeric int override does — see get().
+        return _choice(s, raw)
     return raw
 
 
@@ -243,7 +271,17 @@ def get(db: Session, key: str):
         return _default(s)
     try:
         return _coerce(s, raw)
-    except (TypeError, ValueError):
+    except (TypeError, ValueError) as exc:
+        # The admin set this in the UI and it is quietly not in force: the app runs on the env
+        # default while the settings page goes on showing the value that was saved.
+        if (s.key, raw) not in _coerce_warned:
+            _coerce_warned.add((s.key, raw))
+            log.warning(
+                "a stored setting could not be coerced; the default is used instead",
+                extra=logs.event(
+                    "sysconfig.override_uncoercible", key=s.key, reason=type(exc).__name__
+                ),
+            )
         return _default(s)
 
 
@@ -274,6 +312,8 @@ def set_value(db: Session, key: str, value) -> None:
         raw = str(v)
     elif s.type == "bool":
         raw = "true" if value else "false"
+    elif s.type == "choice":
+        raw = _choice(s, value)
     else:
         raw = "" if value is None else str(value)
 
@@ -316,6 +356,7 @@ def describe(db: Session) -> dict:
             "max": s.max,
             "is_set": is_set,  # an override row exists
             "allow_empty": s.allow_empty,  # if true, the UI stores "" rather than deleting the row
+            "choices": list(s.choices),  # the picker's options; [] for any other type
         }
         if s.secret:
             item["default_is_set"] = bool(_default(s))

backend/app/worker.py

@@ -16,7 +16,6 @@ Design:
   * Crash-safe: on startup, orphaned `running` jobs are requeued and `downloading` assets reset.
 """
 
-import logging
 import os
 import shutil
 import signal
@@ -24,10 +23,13 @@ import subprocess
 import sys
 import threading
 import time
+from collections.abc import Callable
 from datetime import datetime, timezone
 
 from sqlalchemy import text
 
+from app import logs
+from app import logsink
 from app import sysconfig
 from app.config import settings
 from app.db import SessionLocal
@@ -43,8 +45,7 @@ from app.downloads.jobdb import (
 from app.downloads.status import STOPPED_BY_ROUTE, JobStatus
 from app.models import DownloadJob, MediaAsset
 
-log = logging.getLogger("siftlode.worker")
-
+log = logs.get_logger(__name__)
 _stop = threading.Event()
 
 
@@ -71,13 +72,14 @@ def _wait_for_schema(timeout: float = 300.0) -> bool:
             with SessionLocal() as db:
                 db.execute(text("SELECT 1 FROM download_jobs LIMIT 1"))
             return True
-        except Exception as exc:  # noqa: BLE001 — table not migrated yet / DB not up yet
+        except Exception:  # noqa: BLE001 — table not migrated yet / DB not up yet
             if time.monotonic() > deadline:
-                log.error(
-                    "download schema still absent after %ss — the API's migrations have not landed; "
-                    "exiting so the container restarts: %s",
+                # The traceback, not `str(exc)[:120]`: this is the last thing the worker says
+                # before it exits, and which driver call failed is the whole question.
+                log.exception(
+                    "download schema still absent after %ss — the API's migrations have not "
+                    "landed; exiting so the container restarts",
                     timeout,
-                    str(exc)[:120],
                 )
                 return False
             log.info("waiting for the database schema (API migrations)…")
@@ -217,8 +219,17 @@ def _signal_group(proc: subprocess.Popen, pgid: int | None, sig: int) -> None:
             os.killpg(pgid, sig)
         else:
             proc.send_signal(sig)
-    except (ProcessLookupError, OSError):
-        pass
+    except ProcessLookupError:
+        pass  # silent-ok: the child already exited — the race this function is written around
+    except OSError as exc:
+        # Any OTHER refusal means the signal did not land and the child is still running, which
+        # is how a "cancelled" download keeps writing to a staging tree nobody is watching.
+        log.info(
+            "a download child could not be signalled",
+            extra=logs.event(
+                "download.child.signal_failed", signal=int(sig), reason=type(exc).__name__
+            ),
+        )
 
 
 def _stop_child(proc: subprocess.Popen, grace_s: float) -> None:
@@ -228,6 +239,8 @@ def _stop_child(proc: subprocess.Popen, grace_s: float) -> None:
     child already exited, and `getpgid` yields None for a reaped pid, so no separate poll guard."""
     try:
         pgid: int | None = os.getpgid(proc.pid)
+    # silent-ok: `getpgid` refuses for a reaped pid, which the docstring above already treats as
+    # the ordinary case — the escalation then signals the child alone and no-ops if it is gone.
     except OSError:
         pgid = None
     editmod.escalate_kill(
@@ -248,23 +261,42 @@ def _stop_and_free(job_id: int, asset_id: int, proc: subprocess.Popen, grace_s:
 
 
 def _fail_after_kill(
-    job_id: int, asset_id: int, proc: subprocess.Popen, grace_s: float, error: str, error_code: str
-) -> None:
+    job_id: int,
+    asset_id: int,
+    proc: subprocess.Popen,
+    grace_s: float,
+    error: str,
+    error_code: str,
+    cause: str | None = None,
+) -> bool:
     """`_stop_and_free` + mark the job errored — for the watchdog's stall/timeout kills and the
-    supervisor-error fallback."""
+    supervisor-error fallback. Returns whether the job failed here (`_fail_running`)."""
     _stop_and_free(job_id, asset_id, proc, grace_s)
-    _fail_running(job_id, error=error, error_code=error_code)
+    return _fail_running(job_id, error=error, error_code=error_code, cause=cause)
 
 
 def _fail_running(
-    job_id: int, *, error: str, error_code: str | None, error_detail: str | None = None
-) -> None:
+    job_id: int,
+    *,
+    error: str,
+    error_code: str | None,
+    error_detail: str | None = None,
+    cause: str | None = None,
+    exit_code: int | None = None,
+) -> bool:
     """Fail a claimed job — only while it is still `running` (`service.finalize_if_running`), so a
     cancel/pause a route landed first is never turned back into `error`, which holds the asset the
     route already released. `error_detail` is written every time: a failure never shows an earlier
-    failure's explanation."""
+    failure's explanation.
+
+    `cause` is the worker's half of `download.job.failed`: the failures the child never got to
+    report because it was killed, never ran or died — a stall, the wall-clock cap, a non-zero exit
+    (with its `exit_code`). Without it those left no line at all. A caller that logs its own line
+    (a worker error, with its stack — `_report_worker_error`) passes none and uses the returned
+    verdict. Logged only when the job actually failed here: a route's cancel that landed first is
+    not a failure. Returns whether it failed here."""
     with SessionLocal() as db:
-        service.finalize_if_running(
+        failed = service.finalize_if_running(
             db,
             job_id,
             status=JobStatus.ERROR,
@@ -274,6 +306,49 @@ def _fail_running(
             phase=None,
         )
         db.commit()
+    if cause is not None and failed:
+        log.warning(
+            "job %s failed in the worker: %s",
+            job_id,
+            cause,
+            extra=logs.event(
+                "download.job.failed",
+                job_id=job_id,
+                error_code=error_code,
+                cause=cause,
+                **({"exit_code": exit_code} if exit_code is not None else {}),
+            ),
+        )
+    return failed
+
+
+def _report_worker_error(job_id: int, exc: Exception, cause: str, fail: Callable[[], bool]) -> None:
+    """A worker error on a job (the supervisor loop, or processing the job, raised): fail the job,
+    then ONE line with the stack. Its event says what the error did to the job —
+    `download.job.failed` when the job failed with it, `download.worker.error` when a route had
+    already stopped it (a cancel that landed first is not a failure, `_fail_running`).
+
+    The line goes out even when failing the job raises (the database is down): as
+    `download.worker.error`, since the job was not failed, and the raise goes on to the caller —
+    without `exc` as its context, which this line has already carried with its stack."""
+    failed = False
+    try:
+        failed = fail()
+    except BaseException as write_error:
+        write_error.__suppress_context__ = True
+        raise
+    finally:
+        log.error(
+            "job %s: worker error (%s)",
+            job_id,
+            cause,
+            exc_info=exc,
+            extra=logs.event(
+                "download.job.failed" if failed else "download.worker.error",
+                job_id=job_id,
+                cause=cause,
+            ),
+        )
 
 
 def _run_supervised(job_id: int, asset_id: int) -> None:
@@ -304,6 +379,8 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
             try:
                 proc.wait(timeout=poll_s)
                 break  # child finished on its own
+            # silent-ok: the timeout is the loop's clock, not a failure — it is what gives this
+            # thread a turn to check whether a route cancelled the job.
             except subprocess.TimeoutExpired:
                 pass
             status, stale = _job_liveness(job_id)
@@ -326,15 +403,20 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
             # else: a terminal status under the cap (done/error, or a re-queued contended asset) means
             # the child is exiting on its own — keep polling; NEVER kill a finishing child (that could
             # drop its completion notice).
-    except Exception:  # noqa: BLE001 — a watchdog error must NEVER leave the child running unsupervised
-        log.exception("supervisor loop failed for job %s; stopping the child", job_id)
-        _fail_after_kill(
+    except Exception as exc:  # noqa: BLE001 — a watchdog error must NEVER leave the child running unsupervised
+        # silent-ok: not silent — `_report_worker_error` is the report.
+        _report_worker_error(
             job_id,
-            asset_id,
-            proc,
-            grace_s,
-            "The download was stopped after a worker error.",
-            "unknown",
+            exc,
+            "supervisor_error",
+            lambda: _fail_after_kill(
+                job_id,
+                asset_id,
+                proc,
+                grace_s,
+                "The download was stopped after a worker error.",
+                "unknown",
+            ),
         )
         return
 
@@ -353,9 +435,8 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
                         job_id,
                         error=f"The download process exited unexpectedly (code {proc.returncode}).",
                         error_code="unknown",
-                    )
-                    log.warning(
-                        "job %s: child exited %s without finishing", job_id, proc.returncode
+                        cause="child_exit",
+                        exit_code=proc.returncode,
                     )
         except Exception:  # noqa: BLE001 — boot recovery is the backstop for a stuck 'running' job
             log.exception("post-exit reconciliation failed for job %s", job_id)
@@ -369,6 +450,7 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
             grace_s,
             f"The download stalled (no progress for {int(stall_s)}s) and was stopped.",
             "network",
+            cause="stalled",
         )
         return
     if reason == "timeout":
@@ -379,6 +461,7 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
             grace_s,
             f"The download exceeded the {int(wall_s)}s time limit and was stopped.",
             "network",
+            cause="timeout",
         )
         return
     # cancelled / abandon / shutdown: stop + free the asset, but leave the job's own DB status.
@@ -387,10 +470,19 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
         _requeue_if_running(job_id, None)  # resume on the next boot
     elif reason == "abandon":
         log.warning(
-            "job %s: killed a child wedged past the wall-clock; DB status left intact", job_id
+            "job %s: killed a child wedged past the wall-clock; DB status left intact",
+            job_id,
+            extra=logs.event("download.child.abandoned", job_id=job_id),
         )
     else:  # cancelled
-        log.info("job %s: child stopped (route pause/cancel)", job_id)
+        # Not `download.job.aborted`: that is the child's line, logged when it notices the pause
+        # itself — and it often has, and is only tearing down when this poll comes round. This one
+        # says the worker had to stop it.
+        log.info(
+            "job %s: child stopped (route pause/cancel)",
+            job_id,
+            extra=logs.event("download.child.stopped", job_id=job_id),
+        )
 
 
 def _cleanup_staging(asset_id: int) -> None:
@@ -430,7 +522,7 @@ def _process_job(job_id: int) -> None:
             del cookie  # only whether there is one matters here
 
     if asset is None:
-        _fail_running(job_id, error="No media asset", error_code=None)
+        _fail_running(job_id, error="No media asset", error_code=None, cause="no_asset")
         return
 
     if asset_status == "ready":
@@ -442,6 +534,7 @@ def _process_job(job_id: int) -> None:
             error=asset_error or "Download failed",
             error_code=diagnosis.code,
             error_detail=diagnosis.detail,
+            cause="asset_failed",
         )
         return
     if asset_status == "downloading":
@@ -491,15 +584,34 @@ def _worker_loop(idx: int) -> None:
         if job_id is None:
             _stop.wait(settings.download_poll_seconds)
             continue
-        try:
-            _process_job(job_id)
-        except Exception:  # noqa: BLE001
-            log.exception("processing job %s crashed", job_id)
-            _fail_running(job_id, error="Worker error", error_code=None)
+        # Tag every line this job produces (including the crash below) with the job id: a worker runs
+        # N jobs at once on N threads, so without it the interleaved output is unreadable. The bind
+        # wraps the handler too — bound inside the `try`, it was already reset by the time the crash
+        # line was logged.
+        with logs.bind(job_id=job_id):
+            try:
+                _process_job(job_id)
+            except Exception as exc:  # noqa: BLE001
+                try:
+                    _report_worker_error(
+                        job_id,
+                        exc,
+                        "crash",
+                        lambda: _fail_running(job_id, error="Worker error", error_code=None),
+                    )
+                # silent-ok: not silent — `_report_worker_error` logged the crash before this
+                # re-raise. Failing the job hit the database, and letting that out would end this
+                # worker thread for good; boot recovery settles a job left `running`.
+                except Exception:  # noqa: BLE001
+                    log.exception(
+                        "could not fail job %s after its crash",
+                        job_id,
+                        extra=logs.event("download.worker.fail_write_failed", job_id=job_id),
+                    )
 
 
 def main() -> None:
-    logging.basicConfig(level=logging.INFO)
+    logs.configure_logging("worker")
     signal.signal(signal.SIGTERM, _handle_signal)
     signal.signal(signal.SIGINT, _handle_signal)
 
@@ -515,6 +627,10 @@ def main() -> None:
         if _stop.is_set():
             return  # asked to shut down while waiting — that IS a clean exit
         raise SystemExit(1)
+    # After the wait, not before: the DB is reachable now. The wait checks `download_jobs`, not
+    # `log_events`, so on the deploy that first adds the table the worker may start before the API
+    # has migrated — those few seconds of batches fail and say so on stderr; nothing else breaks.
+    logsink.start()
     _recover_orphans()
     n = max(1, settings.download_worker_concurrency)
     log.info("Download worker started (concurrency=%d, root=%s).", n, settings.download_root)

backend/app/youtube/client.py

@@ -6,24 +6,81 @@ so they don't depend on a specific user's token; subscriptions.list?mine=true al
 uses OAuth.
 """
 
-import logging
 from collections.abc import Iterator
 from datetime import datetime, timedelta, timezone
 
 import httpx
 
-from app import quota, sysconfig
+from app import logs, quota, sysconfig
 from app.models import User
 from app.security import decrypt, decrypt_optional, encrypt
 
-log = logging.getLogger("siftlode.youtube")
-
+log = logs.get_logger(__name__)
 GOOGLE_TOKEN_URL = "https://oauth2.googleapis.com/token"
 API_BASE = "https://www.googleapis.com/youtube/v3"
 
 
 class YouTubeError(Exception):
-    pass
+    """An upstream YouTube Data API failure.
+
+    `reason` is the API's own error reason code (`quotaExceeded`, `subscriptionDuplicate`,
+    `videoNotFound`, …) when the body carried one. Branch on THAT, never on the message: the
+    message is prose, it changes under us, and it is deliberately the poorer of the two here."""
+
+    def __init__(self, message: str, *, reason: str = ""):
+        super().__init__(message)
+        self.reason = reason
+
+
+def _error_reason(resp: httpx.Response) -> str:
+    """The `reason` YouTube puts on an API error. That is the whole diagnostic value of an error
+    body; the `message` beside it is prose, and upstream prose can quote the request back — a
+    search term, a playlist title."""
+    try:
+        error = resp.json().get("error")
+        # The OAuth token endpoint's shape: `{"error": "invalid_grant", "error_description": …}`.
+        if isinstance(error, str):
+            return error or "<none>"
+        errors = (error or {}).get("errors") or []
+        reasons = ",".join(e.get("reason", "") for e in errors if e.get("reason"))
+    # silent-ok: the sentinel it returns goes straight into the caller's warning line, which
+    # is the report. (Any non-JSON body — an HTML error page from a proxy.)
+    except Exception:  # noqa: BLE001
+        return "<unparseable>"
+    return reasons or "<none>"
+
+
+def _upstream_failure(method: str, path: str, resp: httpx.Response) -> YouTubeError:
+    """Report one upstream failure and build the error to raise. Every YouTube response that is
+    not a success goes through here — the Data API's reads, writes and deletes and the OAuth
+    token refresh — so what may appear in a log line cannot drift apart between them, and the
+    error a caller raises carries the reason code rather than the body, because callers put it in
+    a log line (`routes/channels.py`) and in a 422 answered to the user. (A request that got no
+    response at all is `_send`'s, with the reason `transport`.) The body is logged only under
+    the requesting user's own Teljes consent: it is the request quoted back."""
+    reason = _error_reason(resp)
+    log.warning(
+        "YouTube %s %s -> %s (%s)",
+        method,
+        path,
+        resp.status_code,
+        reason,
+        extra=logs.event(
+            "youtube.api.error",
+            method=method,
+            path=path,
+            status=resp.status_code,
+            reason=reason,
+        ),
+    )
+    if logs.consented_trace():
+        log.log(
+            logs.TRACE,
+            "the upstream body behind it",
+            # log-pii-ok: only under this user's own Teljes consent, which is what it is for
+            extra=logs.event("youtube.api.body", body=resp.text[:500]),
+        )
+    return YouTubeError(f"{method} {path} -> {resp.status_code}: {reason}", reason=reason)
 
 
 def _chunks(items: list, size: int) -> Iterator[list]:
@@ -62,13 +119,13 @@ class YouTubeClient:
         YouTubeError — degrade gracefully, log, and move on — so wrapping here keeps a
         transient network fault (e.g. the fixed-IP egress proxy being unreachable) from
         escaping as a raw httpx error and turning into an uncaught 500. HTTP status errors
-        are NOT touched here (we don't use raise_for_status): callers inspect resp.status_code
-        themselves and raise their own YouTubeError with the response body."""
+        are NOT touched here (we don't use raise_for_status): each call site inspects
+        resp.status_code itself and hands the response to `_upstream_failure`."""
         try:
             return self._http.request(method, url, **kwargs)
         except httpx.HTTPError as exc:
             raise YouTubeError(
-                f"YouTube API unreachable ({exc.__class__.__name__}: {exc})"
+                f"YouTube API unreachable ({exc.__class__.__name__}: {exc})", reason="transport"
             ) from exc
 
     # --- auth ---
@@ -99,7 +156,7 @@ class YouTubeClient:
             },
         )
         if resp.status_code != 200:
-            raise YouTubeError(f"Token refresh failed: {resp.status_code} {resp.text[:200]}")
+            raise _upstream_failure("POST", "oauth2/token", resp)
         data = resp.json()
         access = data["access_token"]
         tok.access_token = encrypt(access)
@@ -121,8 +178,7 @@ class YouTubeClient:
         resp = self._send("GET", f"{API_BASE}/{path}", params=p, headers=headers)
         quota.record_usage(cost)
         if resp.status_code != 200:
-            log.warning("YouTube API %s -> %s: %s", path, resp.status_code, resp.text[:200])
-            raise YouTubeError(f"GET {path} -> {resp.status_code}: {resp.text[:300]}")
+            raise _upstream_failure("GET", path, resp)
         return resp.json()
 
     # --- endpoints ---
@@ -247,12 +303,7 @@ class YouTubeClient:
         )
         quota.record_usage(50)
         if resp.status_code not in (200, 204):
-            log.warning(
-                "YouTube subscriptions.delete -> %s: %s",
-                resp.status_code,
-                resp.text[:200],
-            )
-            raise YouTubeError(f"DELETE subscriptions -> {resp.status_code}: {resp.text[:300]}")
+            raise _upstream_failure("DELETE", "subscriptions", resp)
 
     def insert_subscription(self, channel_id: str) -> str:
         """Subscribe to a channel on YouTube. Requires the write scope and the user's
@@ -282,8 +333,7 @@ class YouTubeClient:
         resp = self._send(method, f"{API_BASE}/{path}", params=params, json=json, headers=headers)
         quota.record_usage(50)
         if resp.status_code not in (200, 204):
-            log.warning("YouTube %s %s -> %s: %s", method, path, resp.status_code, resp.text[:200])
-            raise YouTubeError(f"{method} {path} -> {resp.status_code}: {resp.text[:300]}")
+            raise _upstream_failure(method, path, resp)
         return resp.json() if resp.content else {}
 
     def get_playlist_snippet(self, playlist_id: str) -> dict | None:

backend/app/youtube/rss.py

@@ -14,7 +14,14 @@ class RssError(Exception):
 
     Deliberately NOT the same as an empty feed: a failed poll used to return `[]`, which the
     caller stamped as a successful `last_rss_at` — so a channel whose feed 404s permanently
-    (deleted, renamed id) looked like it was being polled fine forever, and nothing surfaced it."""
+    (deleted, renamed id) looked like it was being polled fine forever, and nothing surfaced it.
+
+    `reason` is a short stable code (`http_404`, `transport`) so a poll over many channels can report
+    WHAT failed as counts instead of one line per channel."""
+
+    def __init__(self, message: str, reason: str):
+        super().__init__(message)
+        self.reason = reason
 
 
 def fetch_channel_feed(channel_id: str) -> list[dict]:
@@ -24,9 +31,12 @@ def fetch_channel_feed(channel_id: str) -> list[dict]:
     try:
         resp = httpx.get(url, timeout=20.0, headers={"User-Agent": "Siftlode/1.0"})
     except httpx.HTTPError as exc:
-        raise RssError(f"RSS fetch failed for {channel_id}: {exc}") from exc
+        raise RssError(f"RSS fetch failed for {channel_id}: {exc}", "transport") from exc
     if resp.status_code != 200:
-        raise RssError(f"RSS fetch for {channel_id} returned HTTP {resp.status_code}")
+        raise RssError(
+            f"RSS fetch for {channel_id} returned HTTP {resp.status_code}",
+            f"http_{resp.status_code}",
+        )
 
     parsed = feedparser.parse(resp.content)
     out: list[dict] = []

backend/app/youtube/search_scrape.py

@@ -21,6 +21,10 @@ import re
 
 import httpx
 
+from app import logs
+
+log = logs.get_logger(__name__)
+
 _SEARCH_URL = "https://www.youtube.com/youtubei/v1/search"
 _RESULTS_URL = "https://www.youtube.com/results"
 # Public WEB InnerTube key + a recent client version. These are also embedded in every results
@@ -74,7 +78,14 @@ def _get_config(client: httpx.Client) -> tuple[str, str]:
             key.group(1) if key else _FALLBACK_KEY,
             ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
         )
-    except httpx.HTTPError:
+    except httpx.HTTPError as exc:
+        # The fallback pair is CACHED in `_config` for the life of the process, so one failed
+        # fetch at the wrong moment pins every later search to a hard-coded key and version —
+        # and when those go stale, search returns nothing at all, with no error anywhere.
+        log.warning(
+            "the InnerTube config could not be fetched; falling back to the pinned key",
+            extra=logs.event("youtube.scrape.config_fallback", reason=type(exc).__name__),
+        )
         _config = (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
     return _config
 

backend/app/youtube/shorts.py

@@ -18,6 +18,8 @@ def probe_is_short(client: httpx.Client, video_id: str) -> bool | None:
     """True if a Short, False if a regular video, None if undetermined (retry later)."""
     try:
         resp = client.get(SHORTS_URL.format(video_id=video_id), follow_redirects=False)
+    # silent-ok: None is the documented third state — "undetermined, ask again later" — and the
+    # scheduler retries it. It is not the same answer as False, so nothing is being hidden.
     except httpx.HTTPError:
         return None
     if resp.status_code == 200:

backend/entrypoint.sh

@@ -15,4 +15,8 @@ echo "Starting Siftlode API..."
 # X-Forwarded-For only when the TRUE socket peer is a configured proxy (TRUSTED_PROXY_IPS); those
 # flags would make uvicorn overwrite request.client from the forgeable XFF, defeating that check and
 # reopening the rate-limit bypass (SA3).
-exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --log-config log_config.json --timeout-keep-alive 75
+#
+# No --log-config: app/logs.py configures logging for every Siftlode process (api, worker, download
+# child, alembic) in one place, and it silences uvicorn's access log in favour of our own
+# `http.request` line (route template, no query string, request id).
+exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --timeout-keep-alive 75

backend/log_config.json

@@ -1,40 +0,0 @@
-{
-  "version": 1,
-  "disable_existing_loggers": false,
-  "formatters": {
-    "default": {
-      "()": "uvicorn.logging.DefaultFormatter",
-      "fmt": "%(asctime)s %(levelprefix)s [%(name)s] %(message)s"
-    },
-    "access": {
-      "()": "uvicorn.logging.AccessFormatter",
-      "fmt": "%(asctime)s %(levelprefix)s %(client_addr)s - \"%(request_line)s\" %(status_code)s"
-    },
-    "plain": {
-      "format": "%(asctime)s %(levelname)-5s [%(name)s] %(message)s"
-    }
-  },
-  "handlers": {
-    "default": {
-      "class": "logging.StreamHandler",
-      "formatter": "default",
-      "stream": "ext://sys.stdout"
-    },
-    "access": {
-      "class": "logging.StreamHandler",
-      "formatter": "access",
-      "stream": "ext://sys.stdout"
-    },
-    "plain": {
-      "class": "logging.StreamHandler",
-      "formatter": "plain",
-      "stream": "ext://sys.stdout"
-    }
-  },
-  "loggers": {
-    "uvicorn": { "handlers": ["default"], "level": "INFO", "propagate": false },
-    "uvicorn.error": { "handlers": ["default"], "level": "INFO", "propagate": false },
-    "uvicorn.access": { "handlers": ["access"], "level": "INFO", "propagate": false },
-    "siftlode": { "handlers": ["plain"], "level": "INFO", "propagate": false }
-  }
-}

backend/requirements-dev.txt

@@ -1,7 +1,9 @@
 # Test + lint tooling — NEVER in the prod image. Installed only by backend/Dockerfile.test (the
 # throwaway test image), which the prod build (root Dockerfile) never references. Pinned so the
 # suite is reproducible: pytest for the pure-logic tests, and ruff at the SAME version the gate's
-# backend lint lane currently runs on the host (see the run_lint note in siftlode.sh). httpx is
-# already an app dependency (requirements.txt), so its TestClient needs nothing extra here.
+# backend lint lane runs on the host (the gate fails when the host's ruff is not this one). httpx2:
+# Starlette 1.x's TestClient is built on it (`import httpx2 as httpx`) and only falls back to the
+# app's own httpx with a deprecation warning on every run.
 pytest==8.4.2
 ruff==0.15.21
+httpx2==2.13.0

backend/ruff.toml

@@ -9,4 +9,21 @@
 line-length = 100
 
 [lint]
+# Beyond the default set, the rules that hold a LOG LINE to what E22.5 makes of it:
+#   LOG     — stdlib logging misuse (`.warn()`, a logger built by hand, `exc_info` on `.exception`)
+#   G       — the call's shape: %-style args, never an f-string the level filter cannot skip, and
+#             never `str(exc)` where `exc_info` would carry the traceback
+#   T20     — `print` instead of a log line. Two deliberate ones remain, each with its `# noqa`
+#             and a why: `downloads/runner.py`'s usage line is for whoever typed it, and
+#             `logsink.py` reports the SINK's own failure, which cannot be logged into the sink.
+#   TRY400  — `.error()` in an `except` where `.exception()` would keep the traceback
+#   TRY401  — the exception spelled into the message AND passed as `exc_info` (it is in both)
+#   S112    — `except: continue`, the silent swallow inside a loop
+#   S110    — `except: pass`, the same thing outside one
+#   BLE001  — a blind `except Exception`. Every one that remains carries `# noqa: BLE001 — <why>`,
+#             which is the point: catching everything is sometimes right (a worker thread with
+#             nobody to raise to, a best-effort backfill inside a sync) and always worth a sentence.
+# The wider rule these two enforce lives in tests/test_swallowed_exceptions.py: a handler that
+# neither logs nor re-raises says `# silent-ok: <why>`. Ruff sees only the two narrow shapes.
+extend-select = ["LOG", "G", "T20", "TRY400", "TRY401", "S110", "S112", "BLE001"]
 ignore = ["E402"]

backend/tests/conftest.py

@@ -49,7 +49,9 @@ def _db_reachable() -> bool:
         with probe.connect() as conn:
             conn.execute(text("SELECT 1"))
         return True
-    except Exception:
+    # silent-ok / noqa BLE001: the question is literally "can anything reach it", so every
+    # failure is the same answer — and this runs at import, before logging is configured.
+    except Exception:  # noqa: BLE001
         return False
     finally:
         probe.dispose()
@@ -57,6 +59,10 @@ def _db_reachable() -> bool:
 
 _DB_REACHABLE = _db_reachable()
 
+# No background log writer in tests: every TestClient would otherwise start one on app startup and
+# write into tables the fixtures truncate between tests. The sink's own tests drive it explicitly.
+settings.log_db_sink = False
+
 
 @pytest.fixture(scope="session")
 def _schema():
@@ -210,3 +216,15 @@ def _reset_rate_limiters():
         for obj in vars(mod).values():
             if isinstance(obj, RateLimiter):
                 obj.reset()
+
+
+@pytest.fixture(autouse=True)
+def _reset_log_erasures():
+    """`logs._erased` is module-level and remembers an erased account for an hour of monotonic time
+    — longer than any suite run — while the `db` fixture hands serial ids back out from 1 via
+    TRUNCATE … RESTART IDENTITY. So a test that purges user 1 would otherwise make a LATER test's
+    rows for user 1 store with `user_id = None`, failing somewhere nowhere near the cause and only
+    in some file orders. Same reasoning as the limiter sweep above, different global."""
+    from app import logs
+
+    logs._erased.clear()

backend/tests/fixtures/log_redaction_cases.json

@@ -0,0 +1,92 @@
+{
+  "_comment": "One redaction contract for both logs: backend/app/logs.py redact() and frontend/src/lib/log.ts redact() must turn every `value` (under `key`) into `expected`. Both test suites run this file (backend tests/test_logs.py, frontend src/lib/log.contract.test.ts), so the two implementations cannot drift apart. Some cases prove a secret goes; others prove ordinary content (a search term, a Plex title, a chat line) is not mangled — what a LEVEL may carry is decided at the call site, not here.",
+  "cases": [
+    {
+      "name": "a password field is dropped whole",
+      "key": "",
+      "value": {
+        "email": "a@b.co",
+        "password": "hunter2",
+        "current_password": "x"
+      },
+      "expected": {
+        "email": "[email]",
+        "password": "[redacted]",
+        "current_password": "[redacted]"
+      }
+    },
+    {
+      "name": "a token under any spelling of the key, nested and in lists",
+      "key": "",
+      "value": {
+        "plex": { "plex_server_token": "abc", "Authorization": "Bearer z" },
+        "items": [{ "access_token": "t1" }, { "id": 3 }]
+      },
+      "expected": {
+        "plex": {
+          "plex_server_token": "[redacted]",
+          "Authorization": "[redacted]"
+        },
+        "items": [{ "access_token": "[redacted]" }, { "id": 3 }]
+      }
+    },
+    {
+      "name": "E2EE material and cookies",
+      "key": "",
+      "value": {
+        "ciphertext": "AAAA",
+        "passphrase": "p",
+        "private_key": "k",
+        "cookies": "SID=1"
+      },
+      "expected": {
+        "ciphertext": "[redacted]",
+        "passphrase": "[redacted]",
+        "private_key": "[redacted]",
+        "cookies": "[redacted]"
+      }
+    },
+    {
+      "name": "a reset link and a verify fragment",
+      "key": "msg",
+      "value": "open https://app.example/#reset=QWErty123 or /?verify=abc&lang=hu",
+      "expected": "open https://app.example/#reset=[redacted] or /?verify=[redacted]&lang=hu"
+    },
+    {
+      "name": "an OAuth callback URL",
+      "key": "url",
+      "value": "/auth/callback?code=4/0Ab-xyz&state=s3cr3t&scope=email",
+      "expected": "/auth/callback?code=[redacted]&state=[redacted]&scope=email"
+    },
+    {
+      "name": "an email address inside a sentence",
+      "key": "message",
+      "value": "Invitation sent to Some.One+tag@mail.example.org.",
+      "expected": "Invitation sent to [email]."
+    },
+    {
+      "name": "a search term is kept (TRACE decides whether it is logged at all)",
+      "key": "q",
+      "value": "lo-fi beats to relax",
+      "expected": "lo-fi beats to relax"
+    },
+    {
+      "name": "a Plex title with an at-sign is not an address",
+      "key": "title",
+      "value": "Live @ Wembley 1986",
+      "expected": "Live @ Wembley 1986"
+    },
+    {
+      "name": "a chat line is text like any other; its only secret part is scrubbed",
+      "key": "body",
+      "value": "see you at 8, my mail is me@home.example",
+      "expected": "see you at 8, my mail is [email]"
+    },
+    {
+      "name": "non-text values pass through",
+      "key": "",
+      "value": { "status": 404, "ok": false, "ratio": 0.5, "none": null },
+      "expected": { "status": 404, "ok": false, "ratio": 0.5, "none": null }
+    }
+  ]
+}

backend/tests/test_channel_subscribe_upstream.py

@@ -0,0 +1,69 @@
+"""Subscribing when YouTube says no — and when "no" means "you already are".
+
+The error no longer carries YouTube's body (it is the request quoted back, and it went into a log
+line and a 422), so "already following" is recognised by the reason code and nothing else.
+DB lane.
+"""
+
+import pytest
+from fastapi import HTTPException
+from sqlalchemy import select
+
+from app.models import Channel, Subscription
+from app.routes import channels
+from app.youtube.client import YouTubeError
+
+
+def _youtube_refusing(reason: str):
+    class _Client:
+        def __init__(self, db, user):
+            pass
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *exc):
+            return False
+
+        def insert_subscription(self, channel_id):
+            raise YouTubeError(f"POST subscriptions -> 400: {reason}", reason=reason)
+
+        def get_channels(self, ids):
+            return []
+
+    return _Client
+
+
+@pytest.fixture
+def stub_channel(db):
+    db.add(Channel(id="UCdup", title="Already followed"))
+    db.commit()
+    return "UCdup"
+
+
+@pytest.mark.parametrize("reason", ["subscriptionDuplicate", "quotaExceeded,subscriptionDuplicate"])
+def test_already_following_on_youtube_is_recorded_not_refused(
+    db, make_user, monkeypatch, stub_channel, reason
+):
+    user = make_user()
+    monkeypatch.setattr(channels, "YouTubeClient", _youtube_refusing(reason))
+
+    channels.subscribe(stub_channel, user=user, db=db)
+
+    sub = db.execute(
+        select(Subscription).where(
+            Subscription.user_id == user.id, Subscription.channel_id == stub_channel
+        )
+    ).scalar_one()
+    assert sub is not None
+
+
+@pytest.mark.parametrize("reason", ["quotaExceeded", "<none>", "<unparseable>"])
+def test_any_other_refusal_is_answered_with_a_422(db, make_user, monkeypatch, stub_channel, reason):
+    user = make_user()
+    monkeypatch.setattr(channels, "YouTubeClient", _youtube_refusing(reason))
+
+    with pytest.raises(HTTPException) as caught:
+        channels.subscribe(stub_channel, user=user, db=db)
+
+    assert caught.value.status_code == 422

backend/tests/test_demo_refusal_detail.py

@@ -7,14 +7,14 @@ of the caller's already-translated fallback, so from that commit on the demo use
 this gate is UI-gated on `is_demo` either — the comfort shelf is not, so the pin button is live.
 
 A structured `{code: "demo"}` is what `localizeDetail` can turn into `errors.demo` in both locales.
-Pinned at all FOUR sites: the shared dependency, the two auth routes that hand-roll the same check,
-and the account-delete route in `routes/me.py` — which round 6 missed while its docstring claimed
+Pinned at every site: the shared dependency, the two auth routes that hand-roll the same check,
+the account-delete route in `routes/me.py` — which round 6 missed while its docstring claimed
 all the sites were covered, so the next reader had no reason to look (round 7). An asymmetric fix is
 the smell this repo bans, and a docstring asserting symmetry it does not have is worse than none.
 
 The list is derived from the source, not typed out: `test_every_is_demo_refusal_is_on_the_list`
 below fails when a fifth `is_demo` 403 appears anywhere in `app/`, so this file cannot fall behind
-the code the way round 6's did. Needs the DB.
+the code the way round 6's did. (E22 S5 added the fifth: raising the diagnostics level.) Needs the DB.
 """
 
 import re
@@ -33,6 +33,8 @@ def test_every_demo_refusal_carries_the_same_structured_reason(db, make_user, as
         ("post", "/auth/logout-others", None),
         ("post", "/auth/set-password", {"password": "a long enough password"}),
         ("delete", "/api/me/account", None),
+        # E22 S5: the shared account's log would interleave every visitor's session.
+        ("put", "/api/me/diagnostics", {"level": "DEBUG"}),
     ]:
         r = getattr(client, method)(url, **({"json": body} if body is not None else {}))
         assert r.status_code == 403, (url, r.text)
@@ -55,9 +57,9 @@ def test_every_is_demo_refusal_is_on_the_list():
     for path in sorted(app_dir.rglob("*.py")):
         lines = path.read_text(encoding="utf-8").splitlines()
         for i, line in enumerate(lines):
-            # `if user.is_demo:` / `if u.is_demo:` — the hand-rolled gate, not the ORM filters
-            # (`User.is_demo.is_(False)`) and not the column definition.
-            if not re.search(r"^\s*if\s+\w+\.is_demo\s*:", line):
+            # `if user.is_demo:` / `if u.is_demo and …:` — the hand-rolled gate, not the ORM
+            # filters (`User.is_demo.is_(False)`) and not the column definition.
+            if not re.search(r"^\s*if\s+\w+\.is_demo\b[^:]*:", line):
                 continue
             block = "\n".join(lines[i : i + 6])
             if "403" not in block:
@@ -67,9 +69,9 @@ def test_every_is_demo_refusal_is_on_the_list():
                 f"{path.relative_to(app_dir)}:{i + 1} refuses the demo account with an "
                 f"unstructured detail — the frontend cannot localize it:\n{block}"
             )
-    # auth.py's shared gate + routes/auth.py's two + routes/me.py's delete. A new one is welcome,
-    # but it has to be added to the loop above in the same commit.
-    assert len(sites) == 4, sites
+    # auth.py's shared gate + routes/auth.py's two + routes/me.py's delete and diagnostics level. A
+    # new one is welcome, but it has to be added to the loop above in the same commit.
+    assert len(sites) == 5, sites
 
 
 def test_a_real_account_is_not_refused_by_the_gate(db, make_user, as_user):

backend/tests/test_diagnostics.py

@@ -0,0 +1,492 @@
+"""E22 S5 — a user's consented diagnostics level, and the browser log it lets in.
+
+Three parts:
+  * the level itself: the server computes its expiry, a lapsed one reads as the base, the demo
+    account cannot raise it (DB lane, through the real routes);
+  * `POST /api/diagnostics/events`: refused at the base level, and nothing in a batch is trusted —
+    the account comes from the session, everything is redacted again, events under the consented
+    level are dropped, sizes and pace are capped (DB lane);
+  * the server side of the elevation: ONE user's requests are logged in detail while everyone
+    else's stay at the base level, and the account reaches a handler's own lines at all (pure, an
+    in-process app).
+"""
+
+import json
+import logging
+import threading
+from datetime import datetime, timedelta, timezone
+
+import pytest
+from fastapi import Depends, FastAPI
+from fastapi.testclient import TestClient
+from sqlalchemy import select
+
+from app import diagnostics, logs, logsink, sysconfig
+from app.config import settings
+from app.models import LogEvent, User
+from app.routes import diagnostics as diagnostics_routes
+
+_PW = "correct horse battery"
+
+
+# --- the level ------------------------------------------------------------------------------
+
+
+def test_a_raised_level_counts_until_it_lapses_and_not_a_moment_longer():
+    now = datetime.now(timezone.utc)
+    assert diagnostics.effective_level(User()) == "INFO"
+    raised = User(diagnostics_level="TRACE", diagnostics_until=now + timedelta(minutes=1))
+    assert diagnostics.effective_level(raised, now) == "TRACE"
+    assert diagnostics.effective_level(raised, now + timedelta(minutes=2)) == "INFO"
+    # A level without an expiry is not a level that never ends — it is no level at all.
+    assert diagnostics.effective_level(User(diagnostics_level="DEBUG")) == "INFO"
+
+
+def _signed_in(client, make_user, **kw):
+    """A real sign-in: `/api/me` resolves through `optional_current_user`, which the `as_user`
+    override does not cover (see test_me_email_flag)."""
+    user = make_user(email="diag@example.com", password=_PW, **kw)
+    r = client.post("/auth/password-login", json={"email": "diag@example.com", "password": _PW})
+    assert r.status_code == 200, r.text
+    return user
+
+
+def test_raising_the_level_sets_the_servers_expiry_whatever_the_client_sends(db, client, make_user):
+    user = _signed_in(client, make_user)
+    r = client.put("/api/me/diagnostics", json={"level": "DEBUG", "until": "2099-01-01T00:00:00Z"})
+    assert r.status_code == 200, r.text
+    db.refresh(user)
+    expected = datetime.now(timezone.utc) + timedelta(hours=diagnostics.ELEVATION_HOURS)
+    assert abs((user.diagnostics_until - expected).total_seconds()) < 60
+    assert r.json()["diagnostics"]["level"] == "DEBUG"
+    assert client.get("/api/me").json()["diagnostics"]["level"] == "DEBUG"
+
+    assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
+    db.refresh(user)
+    assert (user.diagnostics_level, user.diagnostics_until) == (None, None)
+    assert client.get("/api/me").json()["diagnostics"] == {
+        "level": "INFO",
+        "until": None,
+        "retention_days": 7,
+    }
+
+
+def test_a_lapsed_level_reads_as_the_base_in_me(db, client, make_user):
+    _signed_in(
+        client,
+        make_user,
+        diagnostics_level="TRACE",
+        diagnostics_until=datetime.now(timezone.utc) - timedelta(minutes=1),
+    )
+    assert client.get("/api/me").json()["diagnostics"] == {
+        "level": "INFO",
+        "until": None,
+        "retention_days": 7,
+    }
+
+
+def test_only_the_three_levels_can_be_chosen(db, make_user, as_user):
+    client = as_user(make_user())
+    for bogus in ("WARNING", "debug", "EVERYTHING"):
+        assert client.put("/api/me/diagnostics", json={"level": bogus}).status_code == 422
+
+
+def test_the_demo_account_can_return_to_the_base_but_never_raise_it(db, make_user, as_user):
+    client = as_user(make_user(is_demo=True))
+    r = client.put("/api/me/diagnostics", json={"level": "TRACE"})
+    assert r.status_code == 403 and r.json()["detail"]["code"] == "demo"
+    assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
+
+
+def test_preferences_must_be_a_json_object(db, make_user, as_user):
+    client = as_user(make_user())
+    assert client.put("/api/me/preferences", json=["not", "an", "object"]).status_code == 422
+    r = client.put("/api/me/preferences", json={"theme": "dark"})
+    assert r.status_code == 200 and r.json()["preferences"]["theme"] == "dark"
+
+
+# --- the browser's batches ------------------------------------------------------------------
+
+
+def _event(**over) -> dict:
+    return {
+        "ts": datetime.now(timezone.utc).isoformat(),
+        "level": "INFO",
+        "event": "nav.view",
+        "msg": "view",
+        "tab_id": "a1b2c3",
+        "version": "0.88.0",
+        **over,
+    }
+
+
+@pytest.fixture
+def elevated(db, make_user, as_user):
+    """A signed-in client whose user consented to `level` a minute ago, and a reader for its rows."""
+    diagnostics_routes._batches.reset()
+
+    def _make(level: str = "DEBUG"):
+        user = make_user(
+            diagnostics_level=level,
+            diagnostics_until=datetime.now(timezone.utc) + timedelta(hours=23),
+        )
+
+        def rows() -> list[LogEvent]:
+            db.expire_all()
+            return list(db.execute(select(LogEvent).order_by(LogEvent.id)).scalars())
+
+        return user, as_user(user), rows
+
+    yield _make
+    diagnostics_routes._batches.reset()
+
+
+def test_nothing_is_accepted_at_the_base_level(db, make_user, as_user):
+    client = as_user(make_user())
+    r = client.post("/api/diagnostics/events", json={"events": [_event()]})
+    assert r.status_code == 409 and r.json()["detail"]["code"] == "diagnostics_off"
+    assert not db.execute(select(LogEvent)).first()
+
+
+def test_an_anonymous_caller_is_refused(client):
+    assert client.post("/api/diagnostics/events", json={"events": [_event()]}).status_code == 401
+
+
+def test_a_batch_is_stored_as_the_sessions_web_events_and_redacted_again(elevated):
+    user, client, rows = elevated("DEBUG")
+    r = client.post(
+        "/api/diagnostics/events",
+        json={
+            "events": [
+                _event(
+                    level="WARNING",
+                    event="ui.notify",
+                    msg="sent to someone@example.com",
+                    attrs={"password": "hunter2", "url": "/?token=abc123", "user_id": 999},
+                    request_id="0123456789abcdef",
+                ),
+                _event(level="DEBUG", event="ui.click", attrs={"control": "feed.refresh"}),
+            ],
+            "user_id": 999,  # ignored: the account is the session's
+        },
+    )
+    assert r.json() == {"ok": True, "stored": 2}
+    warning, click = rows()
+    assert (warning.source, warning.logger, warning.user_id) == ("web", "web", user.id)
+    assert warning.msg == "sent to [email]"
+    assert warning.attrs["password"] == logs.REDACTED
+    assert "abc123" not in json.dumps(warning.attrs)
+    assert warning.attrs["tab_id"] == "a1b2c3"
+    assert warning.request_id == "0123456789abcdef"
+    assert (warning.sev, click.level, click.version) == (13, "DEBUG", "0.88.0")
+
+
+def test_an_erased_accounts_batch_is_stored_without_the_account(elevated):
+    """The sink's writer is not the only path into `log_events` — this route inserts too, and an
+    account erased while its browser was mid-flush must not get its rows back through it. Faked at
+    the `forget_user` seam the purge itself calls: once the User row is actually gone the session
+    no longer resolves, so the reachable window is exactly a request already past `current_user`.
+    (`logs._erased` is cleared before every test by conftest's `_reset_log_erasures`.)"""
+    user, client, rows = elevated("DEBUG")
+    logs.forget_user(user.id)
+    r = client.post("/api/diagnostics/events", json={"events": [_event(level="WARNING")]})
+    assert r.json() == {"ok": True, "stored": 1}
+    (stored,) = rows()
+    assert (stored.source, stored.user_id) == ("web", None)
+
+
+def test_an_event_under_the_consented_level_is_dropped(elevated):
+    _user, client, rows = elevated("DEBUG")
+    r = client.post(
+        "/api/diagnostics/events",
+        json={"events": [_event(level="TRACE", msg="typed: lofi"), _event(level="DEBUG")]},
+    )
+    assert r.json()["stored"] == 1
+    assert [row.level for row in rows()] == ["DEBUG"]
+
+
+def test_a_timestamp_outside_the_consent_window_is_clamped(elevated):
+    _user, client, rows = elevated("DEBUG")
+    now = datetime.now(timezone.utc)
+    future = (now + timedelta(days=365)).isoformat()
+    past = (now - timedelta(days=30)).isoformat()
+    client.post("/api/diagnostics/events", json={"events": [_event(ts=future), _event(ts=past)]})
+    late, early = rows()
+    assert late.ts <= datetime.now(timezone.utc)
+    assert early.ts >= now - timedelta(hours=diagnostics.ELEVATION_HOURS, minutes=1)
+
+
+def test_sizes_are_capped(elevated):
+    _user, client, rows = elevated("DEBUG")
+    too_many = [_event() for _ in range(diagnostics_routes.EVENTS_MAX + 1)]
+    assert client.post("/api/diagnostics/events", json={"events": too_many}).status_code == 422
+
+    huge = "x" * (diagnostics_routes.BODY_MAX_BYTES + 1)
+    r = client.post(
+        "/api/diagnostics/events",
+        content=json.dumps({"events": [_event(attrs={"blob": huge})]}),
+        headers={"content-type": "application/json"},
+    )
+    assert r.status_code == 413
+
+    big_attrs = {"blob": "y" * (diagnostics_routes.ATTRS_MAX_BYTES + 10)}
+    assert (
+        client.post("/api/diagnostics/events", json={"events": [_event(attrs=big_attrs)]}).json()[
+            "stored"
+        ]
+        == 1
+    )
+    (row,) = rows()
+    assert "blob" not in row.attrs and row.attrs["attrs_dropped_bytes"] > 0
+
+    bad_code = client.post("/api/diagnostics/events", json={"events": [_event(event="Nav View")]})
+    assert bad_code.status_code == 422
+
+
+def test_the_pace_is_capped_per_user(elevated):
+    _user, client, _rows = elevated("DEBUG")
+    codes = [
+        client.post("/api/diagnostics/events", json={"events": []}).status_code for _ in range(31)
+    ]
+    assert codes[:30] == [200] * 30 and codes[30] == 429
+
+
+def test_what_an_account_stores_per_hour_is_capped_and_said_once(elevated, monkeypatch):
+    user, client, rows = elevated("DEBUG")
+    monkeypatch.setattr(
+        diagnostics_routes, "_stored_bytes", diagnostics_routes.RateLimiter(3_000, 3600)
+    )
+    batch = {"events": [_event(msg="m" * 1_000)]}
+    stored = [client.post("/api/diagnostics/events", json=batch).json()["stored"] for _ in range(4)]
+    assert stored == [1, 1, 0, 0]
+    events = [row.event for row in rows()]
+    # Two batches, then ONE notice for the hour, not one per refused batch.
+    assert events == ["nav.view", "nav.view", "diagnostics.budget_spent"]
+    assert rows()[-1].user_id == user.id
+
+
+def test_only_a_json_body_is_read(elevated):
+    _user, client, rows = elevated("DEBUG")
+    # What a same-site `<form enctype="text/plain">` can send without a preflight.
+    r = client.post(
+        "/api/diagnostics/events",
+        content=json.dumps({"events": [_event()]}),
+        headers={"content-type": "text/plain"},
+    )
+    assert r.status_code == 415 and r.json()["detail"]["code"] == "json_required"
+    assert rows() == []
+    ok = client.post(
+        "/api/diagnostics/events",
+        content=json.dumps({"events": [_event()]}),
+        headers={"content-type": "application/json; charset=utf-8"},
+    )
+    assert ok.json()["stored"] == 1
+
+
+# --- the server side of an elevation ---------------------------------------------------------
+
+
+class _Capture(logging.Handler):
+    """The stdout handler's filters (context + gate), capturing records."""
+
+    def __init__(self):
+        super().__init__(level=logging.NOTSET)
+        self.records: list[logging.LogRecord] = []
+        self.addFilter(logs.ContextFilter())
+        self.addFilter(logs.LevelGate())
+
+    def emit(self, record):
+        self.records.append(record)
+
+
+@pytest.fixture
+def gated():
+    """A process configured at INFO with a gated handler on the ROOT (where the real one sits), and
+    every piece of module state the elevation touches restored afterwards."""
+    root = logging.getLogger()
+    saved = (logs._base_level, dict(logs._elevations), logs._root_applied, root.level)
+    handler = _Capture()
+    root.addHandler(handler)
+    logs._elevations.clear()
+    logs.set_level("INFO")
+    try:
+        yield handler
+    finally:
+        root.removeHandler(handler)
+        logs._base_level, logs._root_applied = saved[0], saved[2]
+        logs._elevations.clear()
+        logs._elevations.update(saved[1])
+        root.setLevel(saved[3])
+
+
+def _app() -> TestClient:
+    """What a real request does: the request middleware, then a SYNC auth dependency that resolves
+    the account (run by FastAPI in a threadpool, on a copy of the context), then a handler."""
+    app = FastAPI()
+    app.middleware("http")(logs.log_requests)
+    handler_log = logging.getLogger("siftlode.difftest")
+
+    def account(user_id: int, level: str = "INFO"):
+        until = datetime.now(timezone.utc) + timedelta(hours=1)
+        logs.set_user(user_id, level, until)
+
+    @app.get("/api/thing/{user_id}")
+    def _thing(user_id: int, _=Depends(account)):
+        handler_log.debug("detail only an elevated request keeps")
+        handler_log.info("an ordinary line")
+        return {}
+
+    return TestClient(app)
+
+
+def test_the_account_reaches_the_handlers_own_lines(gated):
+    _app().get("/api/thing/41")
+    line = next(r for r in gated.records if r.getMessage() == "an ordinary line")
+    assert getattr(line, "user_id", None) == 41
+
+
+def test_one_users_elevation_logs_their_requests_in_detail_and_nobody_elses(gated):
+    client = _app()
+    client.get("/api/thing/7", params={"level": "DEBUG"})
+    client.get("/api/thing/8")
+    detail = [r for r in gated.records if r.getMessage().startswith("detail only")]
+    assert [r.user_id for r in detail] == [7]
+    assert getattr(detail[0], "elevated", False) is True
+    # Every line of the elevated request is marked for storage — the sink keeps it at any level.
+    ordinary = [r for r in gated.records if r.getMessage() == "an ordinary line"]
+    assert [logsink.should_store(r) for r in ordinary] == [True, False]
+    # The process's configured level is still what it reports, however low the root went.
+    assert logs.current_level() == "INFO" and logging.getLogger().level == logging.DEBUG
+
+
+def test_the_root_level_returns_to_the_base_when_the_elevation_ends(gated):
+    client = _app()
+    client.get("/api/thing/7", params={"level": "TRACE"})
+    assert logging.getLogger().level == logs.TRACE
+    client.get("/api/thing/7")  # the same user, back at the base
+    assert logging.getLogger().level == logging.INFO
+    logs._track_elevation(9, logging.DEBUG, datetime.now(timezone.utc) - timedelta(seconds=1))
+    assert logging.getLogger().level == logging.INFO  # an already-expired one is pruned at once
+
+
+def test_the_query_string_is_logged_only_for_a_traced_request_and_redacted(gated):
+    client = _app()
+    client.get("/api/thing/7", params={"level": "DEBUG", "q": "lofi"})
+    client.get("/api/thing/7", params={"level": "TRACE", "q": "lofi", "token": "SECRET"})
+    lines = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
+    assert "query" not in lines[0].attrs
+    assert "q=lofi" in lines[1].attrs["query"] and "SECRET" not in lines[1].attrs["query"]
+
+
+def test_an_address_typed_into_a_search_is_redacted_though_the_browser_encodes_it(gated):
+    _app().get("/api/thing/7?level=TRACE&q=someone%40example.com&new_password=hunter2")
+    (line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
+    assert line.attrs["query"] == "level=TRACE&q=[email]&new_password=[redacted]"
+
+
+@pytest.mark.parametrize(
+    "raw",
+    ["state=a%20b", "code=%20SECRET", "sig=a%22b", "key=a%26b", "reset=a+b"],
+    ids=["space", "leading-space", "quote", "ampersand", "plus"],
+)
+def test_a_credential_param_goes_whole_however_its_value_decodes(raw):
+    """Decoding is what lets `_EMAIL_RE` see `%40`, but it also puts spaces and separators into a
+    value that `_SECRET_PARAM_RE`'s value class stops at — so the name rule decides, not the text."""
+    name = raw.split("=", 1)[0]
+    assert logs.redact_query(f"{raw}&q=lofi") == f"{name}={logs.REDACTED}&q=lofi"
+
+
+def test_an_admin_level_of_trace_does_not_log_other_users_query_strings(gated):
+    logs.set_level("TRACE")
+    _app().get("/api/thing/7", params={"q": "something private"})
+    (line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
+    assert "query" not in line.attrs
+
+
+def test_the_admin_level_applies_to_the_running_process_when_set_and_when_reset(
+    gated, db, make_user, as_user
+):
+    admin = as_user(make_user(role="admin"))
+    r = admin.patch("/api/admin/config/log_level", json={"value": "DEBUG"})
+    assert r.status_code == 200, r.text
+    assert logs.current_level() == "DEBUG"
+    assert admin.delete("/api/admin/config/log_level").status_code == 200
+    assert logs.current_level() == settings.log_level
+
+
+def test_erasing_the_only_elevated_account_puts_the_process_level_back(gated):
+    """The root level is the lowest of the base and every unexpired elevation. `forget_user` drops
+    an elevation like any other path that drops one, so it owes the same recompute — or the process
+    keeps building every TRACE record for an account that no longer exists, to throw them away."""
+    until = datetime.now(timezone.utc) + timedelta(hours=1)
+    try:
+        logs.set_user(7, "TRACE", until)
+        assert logging.getLogger().level == logs.TRACE
+        logs.forget_user(7)
+        assert logging.getLogger().level == logging.INFO
+    finally:
+        # `set_user` writes the request scope, and outside a request that scope is this process's
+        # for good — a later test would store its INFO lines as an elevated session's.
+        logs.set_user(None)
+
+
+def test_the_stored_admin_level_is_applied_at_boot(gated, db, monkeypatch):
+    import asyncio
+
+    import app.main as app_main
+
+    sysconfig.set_value(db, "log_level", "DEBUG")
+    db.commit()
+    monkeypatch.setattr(app_main.state, "is_configured", lambda db: True)
+    monkeypatch.setattr(app_main, "start_scheduler", lambda: None)
+    monkeypatch.setattr(app_main, "shutdown_scheduler", lambda: None)
+    monkeypatch.setattr(app_main.plex_subtitles, "shutdown", lambda: None)
+
+    async def _no_sweep():
+        return None
+
+    monkeypatch.setattr(app_main, "_sweep_hls_orphans", _no_sweep)
+    levels: list[str] = []
+
+    async def _cycle():
+        async with app_main.lifespan(app_main.app):
+            levels.append(logs.current_level())
+
+    asyncio.run(_cycle())
+    assert levels == ["DEBUG"]
+
+
+def test_the_root_level_is_applied_in_the_order_the_elevations_changed(gated, monkeypatch):
+    """Two requests change the elevations at once: A's user leaves (the root may go back to INFO),
+    B's user arrives (it must go to DEBUG). B's change is the later one, so DEBUG must be what stays
+    — even when A is paused between working out its level and applying it."""
+    root = logging.getLogger()
+    logs._track_elevation(1, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1))
+    real_set = root.setLevel
+    other: list[threading.Thread] = []
+
+    def paused_set(level):
+        if level == logging.INFO and not other:
+            b = threading.Thread(
+                target=logs._track_elevation,
+                args=(2, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1)),
+            )
+            other.append(b)
+            b.start()
+            b.join(0.3)  # B runs to completion here unless A's level change holds it off
+        real_set(level)
+
+    monkeypatch.setattr(root, "setLevel", paused_set)
+    logs._track_elevation(1, None, None)  # A
+    other[0].join(5)
+    assert root.level == logging.DEBUG and 2 in logs._elevations
+
+
+def test_a_real_sign_in_carries_the_consented_level_into_the_process(gated, client, db, make_user):
+    user = _signed_in(client, make_user)
+    assert client.put("/api/me/diagnostics", json={"level": "DEBUG"}).status_code == 200
+    logs._elevations.clear()  # only a request resolving the account through auth may put it back
+    client.get("/api/me")
+    assert logs._elevations[user.id][0] == logging.DEBUG
+    assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
+    assert user.id not in logs._elevations

backend/tests/test_download_failure_catalog.py

@@ -125,6 +125,25 @@ def test_the_download_child_knows_the_cookie_went_with_it(db, monkeypatch, tmp_p
     assert (job.error_code, job.error_detail) == ("login_required", "facebook.cookie_rejected")
 
 
+def test_a_failure_the_database_would_not_record_is_not_claimed(db, monkeypatch, tmp_path):
+    """The job is still `running`: the runner reports this with the write's error, and the worker
+    fails the job. A `download.job.failed` here as well would be a second line, and a false one."""
+    u = _user(db, "fb-dbdown@example.com")
+    a, j = _asset_and_job(db, u, sig="dbdown", asset_status="downloading", job_status="running")
+    _fail_the_child(monkeypatch, tmp_path, cookie=False)
+    said = []
+    monkeypatch.setattr(pipeline, "_log_download_failed", lambda job_id, *rest: said.append(job_id))
+
+    def db_down(*a, **k):
+        raise ConnectionError("the database went away")
+
+    monkeypatch.setattr(pipeline.service, "finalize_if_running", db_down)
+
+    with pytest.raises(ConnectionError):
+        pipeline._download(j.id, a.id, "url", _REEL, {}, u.id)
+    assert said == []
+
+
 def test_a_job_meeting_a_failed_shared_asset_gets_the_same_explanation(db):
     # Before the catalog this path wrote the text and NO code: "Download failed" for the second
     # job, "Facebook sign-in required" for the first.

backend/tests/test_failure_lines.py

@@ -0,0 +1,622 @@
+"""E22.5 review round 1: a failure is reported once, with its code, and a repeating one is not a
+flood.
+
+Each case here is a place where the first sweep got the line wrong in one of two directions: a
+failure that left no line at all (a download the watchdog killed), or one that left too many (the
+same edit failure twice, a warning per segment request, per settings read).
+"""
+
+import json
+import logging
+import re
+import shutil
+import subprocess
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+import yt_dlp
+
+from app import logs, sysconfig, worker
+from app.downloads import pipeline
+from app.downloads.stated import StatedFailure
+from app.models import DownloadJob, MediaAsset
+from app.plex import stream as plex_stream
+
+
+class _Sink(logging.Handler):
+    def __init__(self):
+        super().__init__(level=logging.NOTSET)
+        self.lines: list[str] = []
+        self.setFormatter(logs.JsonFormatter())
+        self.addFilter(logs.ContextFilter())
+
+    def emit(self, record):
+        self.lines.append(self.format(record))
+
+    def objects(self) -> list[dict]:
+        return [json.loads(line) for line in self.lines]
+
+    def events(self, code: str) -> list[dict]:
+        return [o for o in self.objects() if o.get("event") == code]
+
+
+@pytest.fixture
+def sink():
+    logger = logging.getLogger("siftlode")
+    handler = _Sink()
+    previous = logger.level
+    logger.addHandler(handler)
+    logger.setLevel(logs.TRACE)
+    try:
+        yield handler
+    finally:
+        logger.removeHandler(handler)
+        logger.setLevel(previous)
+
+
+class _NoDb:
+    """A session that finds nothing and commits nothing — these cases are about the line."""
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *exc):
+        return False
+
+    def get(self, *a, **k):
+        return None
+
+    def commit(self):
+        pass
+
+
+@pytest.fixture
+def finalized(monkeypatch):
+    """Whether `finalize_if_running` fires: True unless a case says a route got there first."""
+    state = {"fires": True}
+    monkeypatch.setattr(worker, "SessionLocal", _NoDb)
+    monkeypatch.setattr(pipeline, "SessionLocal", _NoDb)
+    fire = lambda db, job_id, **fields: state["fires"]  # noqa: E731
+    monkeypatch.setattr(worker.service, "finalize_if_running", fire)
+    return state
+
+
+# --- the worker's own failures ---------------------------------------------------------------
+@pytest.mark.parametrize("cause", ["stalled", "timeout"])
+def test_a_download_the_watchdog_kills_leaves_a_coded_line(sink, finalized, cause):
+    """The child is killed, so it never reports; before this the worker did not either, and a
+    stalled download failed with no line anywhere."""
+    worker._fail_running(42, error="stopped", error_code="network", cause=cause)
+
+    (line,) = sink.events("download.job.failed")
+    assert line["level"] == "WARNING"
+    assert line["attrs"] == {"job_id": 42, "error_code": "network", "cause": cause}
+
+
+def test_a_job_a_route_already_stopped_is_not_reported_as_failed(sink, finalized):
+    finalized["fires"] = False
+    worker._fail_running(42, error="stopped", error_code="network", cause="stalled")
+
+    assert sink.events("download.job.failed") == []
+
+
+def test_a_caller_with_its_own_line_is_not_reported_twice(sink, finalized):
+    worker._fail_running(42, error="Worker error", error_code=None)
+
+    assert sink.events("download.job.failed") == []
+
+
+def test_a_child_that_died_is_reported_with_its_exit_code(sink, finalized):
+    worker._fail_running(
+        42, error="exited", error_code="unknown", cause="child_exit", exit_code=137
+    )
+
+    (line,) = sink.events("download.job.failed")
+    assert line["attrs"]["exit_code"] == 137
+
+
+@pytest.mark.parametrize(
+    ("fires", "code"), [(True, "download.job.failed"), (False, "download.worker.error")]
+)
+def test_a_worker_error_is_one_line_that_says_what_it_did_to_the_job(sink, fires, code):
+    """A route's cancel that landed first is not a failure — the error is still reported, with
+    its stack, but not as the job failing."""
+    try:
+        raise KeyError("a worker bug")
+    except KeyError as exc:
+        worker._report_worker_error(42, exc, "crash", lambda: fires)
+
+    (line,) = [o for o in sink.objects() if o["level"] in ("WARNING", "ERROR")]
+    assert line["event"] == code
+    assert "KeyError" in line["exc"]
+
+
+def test_a_worker_error_is_reported_even_when_failing_the_job_raises(sink):
+    def db_down():
+        raise ConnectionError("the database went away")
+
+    with pytest.raises(ConnectionError):
+        worker._report_worker_error(42, KeyError("a worker bug"), "crash", db_down)
+
+    (line,) = sink.events("download.worker.error")
+    assert line["attrs"]["cause"] == "crash"
+
+
+def test_a_crash_the_database_will_not_record_does_not_end_the_worker_thread(monkeypatch, sink):
+    """The crash handler writes to the database; if that raises too, the thread must go on to
+    the next job — a dead thread is a download slot gone until the worker restarts."""
+    claims = iter([42, 43])
+    seen = []
+
+    def claim():
+        job = next(claims, None)
+        if job is None:
+            worker._stop.set()
+        return job
+
+    def crash(job_id):
+        seen.append(job_id)
+        raise RuntimeError("boom")
+
+    def db_down(*a, **k):
+        raise ConnectionError("the database went away")
+
+    monkeypatch.setattr(worker, "_claim_job", claim)
+    monkeypatch.setattr(worker, "_process_job", crash)
+    monkeypatch.setattr(worker, "_fail_running", db_down)
+    monkeypatch.setattr(worker.settings, "download_poll_seconds", 0)
+    worker._stop.clear()
+    try:
+        worker._worker_loop(0)
+    finally:
+        worker._stop.clear()
+    assert seen == [42, 43]
+
+
+class _StoppedProc:
+    """A child that is still tearing down when the supervisor polls, after a route paused it."""
+
+    pid = 4242
+    returncode = None
+
+    def wait(self, timeout=None):
+        import subprocess
+
+        raise subprocess.TimeoutExpired("child", timeout or 0)
+
+
+def test_the_worker_stopping_a_paused_child_is_not_a_second_abort_line(monkeypatch, sink):
+    """The child logs `download.job.aborted` when it notices the pause itself; the worker's line
+    is its own fact — it had to stop the child — so one pause is never two aborts."""
+    monkeypatch.setattr(worker, "_spawn_download_child", lambda job_id: _StoppedProc())
+    monkeypatch.setattr(worker, "_job_liveness", lambda job_id: ("paused", 1.0))
+    monkeypatch.setattr(worker, "_stop_and_free", lambda *a: None)
+
+    worker._run_supervised(42, 7)
+
+    assert sink.events("download.job.aborted") == []
+    assert len(sink.events("download.child.stopped")) == 1
+
+
+# --- an edit failure, once ---------------------------------------------------------------------
+def test_an_edit_failure_is_one_line_with_its_code_and_our_own_stack(sink, finalized):
+    try:
+        raise TypeError("a spec shape nobody expected")
+    except TypeError as exc:
+        pipeline._fail_edit(9, 3, str(exc), exc)
+
+    (line,) = sink.events("edit.job.failed")
+    assert line["attrs"] == {"job_id": 9, "asset_id": 3, "error_type": "TypeError"}
+    assert "TypeError" in line["exc"]
+    assert len([o for o in sink.objects() if o["level"] in ("WARNING", "ERROR")]) == 1
+
+
+def test_an_edit_that_ffmpeg_refused_names_no_file_and_carries_no_stack(sink, finalized):
+    """ffmpeg's stderr tail names its input, whose path is the channel and the title. The message
+    is scrubbed; the stack is left off, because the message is the whole story."""
+    msg = "ffmpeg failed (1): /downloads/Some_Channel/Season_2024/A_Private_Title_a3.mp4: Invalid data"
+    pipeline._fail_edit(9, 3, msg, StatedFailure(msg))
+
+    (line,) = sink.events("edit.job.failed")
+    assert "A_Private_Title" not in json.dumps(line)
+    assert "exc" not in line
+
+
+def test_an_edit_the_database_would_not_fail_is_not_reported_as_failed(sink, monkeypatch):
+    """The job is still `running`: the runner reports this failure with the write's, and the
+    worker fails the job (`child_exit`). A line here as well would be a second, and a false one."""
+
+    class _Down:
+        def __enter__(self):
+            raise ConnectionError("the database went away")
+
+        def __exit__(self, *exc):
+            return False
+
+    monkeypatch.setattr(pipeline, "SessionLocal", _Down)
+    with pytest.raises(ConnectionError):
+        pipeline._fail_edit(9, 3, "boom", KeyError("x"))
+
+    assert sink.events("edit.job.failed") == []
+
+
+class _EditDb(_NoDb):
+    """An edit job whose source is ready, for `_process_edit` to get as far as its encode."""
+
+    def get(self, model, _id):
+        if model is DownloadJob:
+            return SimpleNamespace(source_asset_id=7, edit_spec={}, user_id=1)
+        if model is MediaAsset:
+            return SimpleNamespace(
+                status="ready", rel_path="src.mp4", width=640, height=360, duration_s=10
+            )
+        return None
+
+
+def test_an_edit_that_crashes_is_said_once(sink, finalized, monkeypatch, tmp_path):
+    """The line `_process_edit`'s handler used to log before `_fail_edit` logged it again."""
+    (tmp_path / "src.mp4").write_bytes(b"x")
+    monkeypatch.setattr(pipeline, "SessionLocal", _EditDb)
+    monkeypatch.setattr(pipeline.settings, "download_root", str(tmp_path))
+    monkeypatch.setattr(pipeline, "_edit_staging_dir", lambda asset_id: tmp_path / "staging")
+
+    def odd_spec(*a, **k):
+        raise TypeError("a spec shape nobody expected")
+
+    monkeypatch.setattr(pipeline, "_set_job", odd_spec)
+
+    pipeline._process_edit(9, 3)
+
+    (line,) = [o for o in sink.objects() if o["level"] in ("WARNING", "ERROR")]
+    assert line["event"] == "edit.job.failed"
+    assert "TypeError" in line["exc"]
+
+
+def test_an_edit_whose_source_is_gone_is_found_by_the_same_code(sink, finalized):
+    pipeline._fail_edit(9, 3, "The source file is missing.")
+
+    (line,) = sink.events("edit.job.failed")
+    assert line["attrs"] == {"job_id": 9, "asset_id": 3}
+
+
+# --- a download failure, scrubbed ---------------------------------------------------------------
+def test_a_failed_download_line_does_not_put_back_what_the_adapter_took_out(sink):
+    """yt-dlp logs its error through the adapter (scrubbed) and then raises the same text, which
+    the pipeline logs again. Unscrubbed, that second line is the URL the first one hid."""
+    msg = "ERROR: Unsupported URL: https://example.org/private/path?id=123"
+    pipeline._log_download_failed(5, msg, "unknown", yt_dlp.utils.DownloadError(msg))
+
+    (line,) = sink.events("download.job.failed")
+    assert "private/path" not in json.dumps(line)
+    assert "example.org/…" in line["msg"]
+    # Unclassified, but yt-dlp's own error: its stack is the `except`, its foot the raw text.
+    assert "exc" not in line
+
+
+@pytest.mark.parametrize(
+    "exc",
+    [
+        RuntimeError("dictionary changed size during iteration"),
+        RecursionError("maximum recursion depth exceeded"),
+        NotImplementedError(),
+        FileNotFoundError(2, "No such file", "/downloads/Some_Channel/Season_2024/A_Private_Title"),
+    ],
+    ids=lambda e: type(e).__name__,
+)
+def test_a_bug_that_python_raises_keeps_its_stack_and_names_no_file(sink, exc):
+    """Only the failures WE raise with the whole story in the message go without a stack. The
+    same types are what Python raises for a bug of ours, and then the stack is the only lead."""
+    try:
+        raise exc
+    except (RuntimeError, OSError) as caught:
+        pipeline._log_download_failed(5, str(caught), "unknown", caught)
+
+    (line,) = sink.events("download.job.failed")
+    assert type(exc).__name__ in line["exc"]
+    assert "A_Private_Title" not in json.dumps(line)
+
+
+def test_a_stack_does_not_carry_the_download_error_it_was_raised_over(sink):
+    """A database error raised inside `except DownloadError` has yt-dlp's text as its context,
+    and a plain `exc_info` prints that link unscrubbed."""
+    # A variable, as in the pipeline: a frame prints its SOURCE line, and that is code, not data.
+    text = "ERROR: [youtube] dQw4w9WgXcQ: Sign in to confirm"
+    try:
+        try:
+            raise yt_dlp.utils.DownloadError(text)
+        except yt_dlp.utils.DownloadError:
+            raise ConnectionError("server closed the connection unexpectedly")  # noqa: B904
+    except ConnectionError as exc:
+        pipeline._log_download_failed(5, str(exc), "unknown", exc)
+
+    (line,) = sink.events("download.job.failed")
+    assert "During handling of the above exception" in line["exc"]
+    assert "Sign in to confirm" in line["exc"]
+    assert "dQw4w9WgXcQ" not in json.dumps(line)
+
+
+def test_an_unclassified_failure_of_our_own_keeps_its_stack(sink):
+    try:
+        raise KeyError("requested_formats")
+    except KeyError as exc:
+        pipeline._log_download_failed(5, str(exc), "unknown", exc)
+
+    (line,) = sink.events("download.job.failed")
+    assert "KeyError" in line["exc"]
+    assert line["attrs"]["error_type"] == "KeyError"
+
+
+def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkeypatch):
+    from app.downloads import adapters, backfill
+
+    url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
+
+    class _Refusing:
+        def __init__(self, opts):
+            pass
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *exc):
+            return False
+
+        def extract_info(self, target, download):
+            raise yt_dlp.utils.DownloadError(
+                f"ERROR: [youtube] dQw4w9WgXcQ: Video unavailable ({target})"
+            )
+
+    monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
+    assert backfill._extract_meta(url, adapters.YouTubeAdapter(), 11) is None
+
+    lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
+    assert lines and all("dQw4w9WgXcQ" not in json.dumps(o) for o in lines)
+
+
+def test_the_metadata_backfill_takes_out_a_generic_name_whatever_it_holds(sink, monkeypatch):
+    from app.downloads import adapters, backfill
+
+    url = "https://example.org/v/Our%20Trip%3A%20Day%201.mp4"
+
+    class _Refusing:
+        def __init__(self, opts):
+            pass
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *exc):
+            return False
+
+        def extract_info(self, target, download):
+            raise yt_dlp.utils.DownloadError("ERROR: [generic] Our Trip: Day 1: Unable to download")
+
+    monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
+    assert backfill._extract_meta(url, adapters.SourceAdapter(), 11) is None
+
+    lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
+    assert lines and all("Day 1" not in json.dumps(o) for o in lines)
+
+
+# --- a repeating failure, once ------------------------------------------------------------------
+def test_a_stale_setting_is_reported_once_not_on_every_read(sink, monkeypatch):
+    spec = next(s for s in sysconfig.SPECS if s.type == "int")
+    monkeypatch.setattr(sysconfig, "_raw_override", lambda db, key: "not-a-number")
+    monkeypatch.setattr(sysconfig, "_coerce_warned", set())
+
+    for _ in range(3):
+        sysconfig.get(None, spec.key)
+
+    assert len(sink.events("sysconfig.override_uncoercible")) == 1
+
+
+class _RefusingDir:
+    """A session directory on a disk that has gone away: every probe raises."""
+
+    def __truediv__(self, name):
+        return SimpleNamespace(exists=self._refuse)
+
+    @staticmethod
+    def _refuse():
+        raise PermissionError("the disk went read-only")
+
+
+def test_a_refusing_session_directory_is_reported_once_per_session(sink):
+    """Every segment request probes, and hls.js retries each 404 — a line per probe is a flood."""
+    session = plex_stream.HlsSession("k", Path("."), None, 0.0, ())
+    session.dir = _RefusingDir()
+
+    for seq in (3, 4, 5):
+        assert not plex_stream.artifact_may_arrive(session, plex_stream.segment_name(seq, None))
+
+    (line,) = sink.events("plex.stream.segment_probe_failed")
+    assert line["attrs"]["reason"] == "PermissionError"
+
+
+def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, tmp_path):
+    """The first attempt's DownloadError is logged when the fallback set takes over — one line
+    before the adapter's scrubbed copy of the same text would have hidden it."""
+    url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
+    attempts = iter(
+        [yt_dlp.utils.DownloadError(f"ERROR: [youtube] dQw4w9WgXcQ: Sign in ({url})"), {"id": "x"}]
+    )
+
+    class _YDL:
+        def __init__(self, opts):
+            pass
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *exc):
+            return False
+
+        def extract_info(self, target, download):
+            outcome = next(attempts)
+            if isinstance(outcome, Exception):
+                raise outcome
+            return outcome
+
+    adapter = SimpleNamespace(
+        client_sets=lambda authed: [["web"], ["tv"]],
+        resolve_url=lambda target, opts: target,
+        ydl_opts_extra=lambda clients: {},
+        retriable=lambda text: True,
+    )
+    monkeypatch.setattr(pipeline.yt_dlp, "YoutubeDL", _YDL)
+    monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: None)
+    monkeypatch.setattr(pipeline, "_set_job", lambda job_id, **fields: None)
+
+    assert pipeline._extract_with_fallback(1, 1, url, {}, tmp_path, adapter) == {"id": "x"}
+
+    (line,) = [o for o in sink.objects() if "retrying with fallback" in o["msg"]]
+    assert "dQw4w9WgXcQ" not in json.dumps(line)
+
+
+@pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
+def test_a_failed_poster_or_storyboard_names_no_file(sink, tmp_path):
+    """ffmpeg's CalledProcessError quotes its argv, and the source path in it is built from the
+    channel and the title."""
+    from app.downloads import edit as editmod
+
+    rel = "Some_Channel/Season_2024/Some_Channel_-_20240101_-_A_Private_Title_a3.mp4"
+    (tmp_path / rel).parent.mkdir(parents=True)
+    (tmp_path / rel).write_bytes(b"not a video")
+    asset = SimpleNamespace(id=3, rel_path=rel, duration_s=10.0, storyboard_path=None)
+
+    assert editmod.ensure_poster(asset, str(tmp_path)) is None
+    assert editmod.ensure_storyboard(asset, str(tmp_path)) is None
+
+    lines = [o["msg"] for o in sink.objects() if "gen skipped" in o["msg"]]
+    # The whole message: a long tmp root would push a leaked argv's path past any length cut.
+    assert len(lines) == 2
+    assert all(
+        re.fullmatch(r"\w+ gen skipped for asset 3: CalledProcessError \d+", m) for m in lines
+    )
+
+
+@pytest.mark.parametrize(
+    "err",
+    [
+        subprocess.TimeoutExpired(
+            ["ffmpeg", "-i", "/downloads/Some_Channel/A_Private_Title.mp4"], 30
+        ),
+        FileNotFoundError(2, "No such file", "/downloads/Some_Channel/A_Private_Title.jpg"),
+        subprocess.CalledProcessError(1, ["ffmpeg", "-i", "/downloads/Some_Channel/x.mp4"]),
+    ],
+    ids=lambda e: type(e).__name__,
+)
+def test_an_image_step_failure_is_its_kind_and_nothing_it_quotes(err):
+    from app.downloads import edit as editmod
+
+    code = getattr(err, "returncode", None)
+    assert editmod._step_failure(err) == " ".join(
+        [type(err).__name__, *([str(code)] if code is not None else [])]
+    )
+
+
+def test_a_thumbnail_that_cannot_be_copied_names_no_file(sink, monkeypatch, tmp_path):
+    from app.downloads import edit as editmod
+
+    src = tmp_path / "thumb.jpg"
+    src.write_bytes(b"\xff\xd8\xff rest")
+
+    def refuse(a, b):
+        raise PermissionError(
+            13, "Permission denied", "/downloads/Some_Channel/A_Private_Title.jpg"
+        )
+
+    monkeypatch.setattr(editmod.shutil, "copyfile", refuse)
+    assert editmod.thumbnail_to_jpg(src, tmp_path / "out.jpg") is None
+
+    (line,) = [o for o in sink.objects() if "thumbnail copy failed" in o["msg"]]
+    assert "Some_Channel" not in json.dumps(line)
+
+
+# --- the child's last stop ------------------------------------------------------------------------
+def test_a_failure_the_child_cannot_record_is_one_scrubbed_line_and_a_non_zero_exit(
+    sink, monkeypatch
+):
+    """The pipeline's failure write was refused. Python's own hook would print the chain raw —
+    and its context is the download's failure, the URL in it. The worker fails the job on the
+    exit code (`child_exit`), so this line does not claim the job failed."""
+    from app.downloads import runner
+
+    url = "https://example.org/v/Our%20Trip%3A%20Day%201.mp4"
+    text = f"ERROR: [generic] Our Trip: Day 1: Unable to download webpage ({url})"
+
+    def download(*a):
+        try:
+            raise yt_dlp.utils.DownloadError(text)
+        except yt_dlp.utils.DownloadError:
+            raise ConnectionError("the database went away")  # noqa: B904
+
+    monkeypatch.setattr(pipeline, "_download", download)
+    with pytest.raises(SystemExit) as exited:
+        runner._run(5, 7, "download", "url", url, {}, 1)
+
+    assert exited.value.code == 1
+    (line,) = [o for o in sink.objects() if o["level"] in ("WARNING", "ERROR")]
+    assert line["event"] == "download.child.failed"
+    assert "Unable to download webpage" in line["exc"]
+    # "Day 1" too: the fallback pattern alone stops at the first `: ` and leaves it.
+    assert "Trip" not in json.dumps(line) and "Day 1" not in json.dumps(line)
+
+
+def test_a_crash_the_database_will_not_record_carries_its_stack_once(monkeypatch, sink):
+    claims = iter([42])
+
+    def claim():
+        job = next(claims, None)
+        if job is None:
+            worker._stop.set()
+        return job
+
+    def crash(job_id):
+        raise RuntimeError("the one crash")
+
+    def db_down(*a, **k):
+        raise ConnectionError("the database went away")
+
+    monkeypatch.setattr(worker, "_claim_job", claim)
+    monkeypatch.setattr(worker, "_process_job", crash)
+    monkeypatch.setattr(worker, "_fail_running", db_down)
+    monkeypatch.setattr(worker.settings, "download_poll_seconds", 0)
+    worker._stop.clear()
+    try:
+        worker._worker_loop(0)
+    finally:
+        worker._stop.clear()
+
+    stacks = [o["exc"] for o in sink.objects() if "exc" in o]
+    assert sum("the one crash" in s for s in stacks) == 1
+    (write,) = sink.events("download.worker.fail_write_failed")
+    assert "the database went away" in write["exc"]
+
+
+def test_a_watchdog_error_on_a_job_it_failed_is_that_jobs_failure_line(
+    sink, finalized, monkeypatch
+):
+    """Through the real `_fail_after_kill` → `_fail_running`: the verdict it returns picks the
+    event, so a lost `return` would log every such failure as a mere worker error."""
+
+    class _Alive:
+        pid = 4242
+        returncode = None
+
+        def wait(self, timeout=None):
+            raise subprocess.TimeoutExpired("child", timeout or 0)
+
+    def broken(job_id):
+        raise KeyError("a watchdog bug")
+
+    monkeypatch.setattr(worker, "_spawn_download_child", lambda job_id: _Alive())
+    monkeypatch.setattr(worker, "_job_liveness", broken)
+    monkeypatch.setattr(worker, "_stop_and_free", lambda *a: None)
+
+    worker._run_supervised(42, 7)
+
+    (line,) = [o for o in sink.objects() if o["level"] in ("WARNING", "ERROR")]
+    assert (line["event"], line["attrs"]["cause"]) == ("download.job.failed", "supervisor_error")
+    assert "a watchdog bug" in line["exc"]

backend/tests/test_followed_channels.py

@@ -0,0 +1,63 @@
+"""The scheduled RSS poll and recent backfill cover FOLLOWED channels only (owner decision, 2026-09-17).
+
+A channel row also comes from a search result, a playlist item or a subscription somebody dropped,
+and nothing deletes those rows. Polling all of them was 582 of 979 channels on prod with no follower,
+three of them dead feeds answering 404 every 20 minutes. Followed = subscribed by someone, or explored
+by someone. The admin card counting "channels to sync" must use the same selection, or it counts
+channels the backfill will never touch. DB lane.
+"""
+
+from app.models import Channel, ExploredChannel, Subscription
+from app.sync import runner
+
+
+def _channels(db, make_user):
+    user = make_user()
+    for cid in ("UC_sub", "UC_explored", "UC_search", "UC_playlist"):
+        db.add(Channel(id=cid, title=cid, from_search=cid == "UC_search"))
+    db.flush()
+    db.add(Subscription(user_id=user.id, channel_id="UC_sub"))
+    db.add(ExploredChannel(user_id=user.id, channel_id="UC_explored"))
+    db.commit()
+    return user
+
+
+def test_the_rss_poll_skips_channels_nobody_follows(db, make_user, monkeypatch):
+    _channels(db, make_user)
+    polled: list[str] = []
+    monkeypatch.setattr(runner, "fetch_channel_feed", lambda cid: polled.append(cid) or [])
+    monkeypatch.setattr(runner, "apply_rss_feed", lambda db, ch, entries: 0)
+    assert runner.run_rss_poll(db) == {"new": 0, "failed": 0}
+    assert sorted(polled) == ["UC_explored", "UC_sub"]
+
+
+def test_the_recent_backfill_skips_channels_nobody_follows(db, make_user, monkeypatch):
+    user = _channels(db, make_user)
+    done: list[str] = []
+
+    class _Client:
+        def __init__(self, *a):
+            pass
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *a):
+            return False
+
+    monkeypatch.setattr(runner, "get_service_user", lambda db: user)
+    monkeypatch.setattr(runner, "YouTubeClient", _Client)
+    monkeypatch.setattr(runner.quota, "backfill_reserve_reached", lambda: False)
+    monkeypatch.setattr(
+        runner, "backfill_channel_recent", lambda db, yt, ch: done.append(ch.id) or 0
+    )
+    monkeypatch.setattr(runner, "reconcile_full_history", lambda db: 0)
+    runner.run_recent_backfill(db)
+    assert sorted(done) == ["UC_explored", "UC_sub"]
+
+
+def test_the_dashboard_counts_the_channels_the_backfill_will_process(db, make_user, as_user):
+    _channels(db, make_user)
+    admin = make_user(role="admin")
+    queue = as_user(admin).get("/api/admin/scheduler").json()["queue"]
+    assert queue["channels_recent_pending"] == 2

backend/tests/test_last_login.py

@@ -0,0 +1,97 @@
+"""`users.last_login_at` (E22 S1), DB-lane.
+
+The roadmap assumed stamping it in `_establish_session` would cover Google and password alike. It
+would not have: only password login called it, and the Google callback carried its own copy of the
+same four lines. So these pin BOTH doors through the real routes — plus the two that must not stamp
+(a refused password, the shared demo account) and the admin list that shows it.
+"""
+
+from datetime import datetime, timedelta, timezone
+
+from app.config import settings
+from app.models import DemoWhitelist
+from app.routes import auth as auth_routes
+
+_PW = "correct horse battery"
+
+
+def _recent(ts: datetime | None) -> bool:
+    return ts is not None and datetime.now(timezone.utc) - ts < timedelta(minutes=1)
+
+
+def test_a_password_sign_in_stamps_last_login(client, db, make_user):
+    user = make_user(email="pw@example.com", password=_PW)
+    assert user.last_login_at is None
+
+    r = client.post("/auth/password-login", json={"email": "pw@example.com", "password": _PW})
+
+    assert r.status_code == 200
+    db.refresh(user)
+    assert _recent(user.last_login_at)
+
+
+def test_a_refused_password_does_not_stamp(client, db, make_user):
+    user = make_user(email="wrong@example.com", password=_PW)
+
+    r = client.post(
+        "/auth/password-login", json={"email": "wrong@example.com", "password": "not it"}
+    )
+
+    assert r.status_code == 401
+    db.refresh(user)
+    assert user.last_login_at is None
+
+
+class _FakeGoogle:
+    """authlib's client, reduced to the one call the callback makes after Google redirects back."""
+
+    def __init__(self, userinfo: dict):
+        self._userinfo = userinfo
+
+    async def authorize_access_token(self, _request):
+        return {"userinfo": self._userinfo, "access_token": "at", "scope": "openid email profile"}
+
+
+def test_a_google_sign_in_stamps_last_login(client, db, make_user, monkeypatch):
+    from cryptography.fernet import Fernet
+
+    from app import security
+
+    # The callback stores the OAuth grant encrypted; a throwaway key stands in for TOKEN_ENCRYPTION_KEY.
+    monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
+    user = make_user(email="g@example.com", password=None, google_sub="sub-g")
+    monkeypatch.setattr(settings, "admin_emails", "g@example.com")  # the bootstrap allow-list
+    fake = _FakeGoogle({"sub": "sub-g", "email": "g@example.com", "email_verified": True})
+    monkeypatch.setattr(auth_routes, "google_oauth", lambda _db: fake)
+
+    r = client.get("/auth/callback", follow_redirects=False)
+
+    assert r.status_code in (302, 307) and r.headers["location"] == "/"
+    db.refresh(user)
+    assert _recent(user.last_login_at)
+    # …and it is a real sign-in, not only a stamp: the session now answers as this account.
+    assert client.get("/api/me").json()["id"] == user.id
+
+
+def test_the_demo_entry_does_not_stamp_the_shared_account(client, db):
+    db.add(DemoWhitelist(email="visitor@example.com"))
+    db.commit()
+
+    r = client.post("/auth/demo", json={"email": "visitor@example.com"})
+
+    assert r.json() == {"authenticated": True}
+    demo = auth_routes.get_or_create_demo_user(db)
+    db.refresh(demo)
+    assert demo.last_login_at is None
+
+
+def test_the_admin_user_list_carries_last_login(as_user, db, make_user):
+    admin = make_user(email="boss@example.com", role="admin")
+    never = make_user(email="never@example.com")
+    admin.last_login_at = datetime(2026, 9, 16, 8, 30, tzinfo=timezone.utc)
+    db.commit()
+
+    rows = {u["email"]: u for u in as_user(admin).get("/api/admin/users").json()}
+
+    assert rows["boss@example.com"]["last_login_at"].startswith("2026-09-16T08:30")
+    assert rows[never.email]["last_login_at"] is None

backend/tests/test_log_pii.py

@@ -0,0 +1,407 @@
+"""E22.5 S1 — what a log line may NOT carry.
+
+`logs.redact_text` is a safety net over addresses and credential params, and it was doing the whole
+job: every one of these call sites was putting user content into a line and relying on the net to
+catch it. Two things were wrong with that. What the net does NOT cover went in whole — a search
+term, a source URL, an upstream body that quotes the request back — and what it DOES cover came out
+as `[email]`, so the line's one variable was a constant and the line said nothing.
+
+The cases below pin the call sites. The last one pins the family: no future log call may pass the
+same kind of value without saying, on the line, why it is safe.
+"""
+
+import ast
+import json
+import logging
+import pathlib
+
+import httpx
+import pytest
+
+from app import logs
+from app.youtube.client import YouTubeError, _upstream_failure
+
+TERM = "the exact thing they typed"
+
+
+class _Sink(logging.Handler):
+    """The stdout handler's own filter and formatter, so a case sees the bytes a deployment writes
+    — not the record before redaction ran."""
+
+    def __init__(self):
+        super().__init__(level=logging.NOTSET)
+        self.lines: list[str] = []
+        self.setFormatter(logs.JsonFormatter())
+        self.addFilter(logs.ContextFilter())
+
+    def emit(self, record):
+        self.lines.append(self.format(record))
+
+    def objects(self) -> list[dict]:
+        return [json.loads(line) for line in self.lines]
+
+    def at(self, level: str) -> list[dict]:
+        return [o for o in self.objects() if o["level"] == level]
+
+
+@pytest.fixture
+def sink():
+    logger = logging.getLogger("siftlode")
+    handler = _Sink()
+    previous = logger.level
+    logger.addHandler(handler)
+    logger.setLevel(logs.TRACE)
+    try:
+        yield handler
+    finally:
+        logger.removeHandler(handler)
+        logger.setLevel(previous)
+
+
+def _youtube_403(term: str) -> httpx.Response:
+    """A real-shaped Data API error. `message` is prose, and prose from an upstream quotes the
+    request back — which is how a search term ends up in our log through someone else's words."""
+    return httpx.Response(
+        403,
+        json={
+            "error": {
+                "code": 403,
+                "message": f"The request for '{term}' cannot be completed: quota exceeded.",
+                "errors": [{"reason": "quotaExceeded", "domain": "youtube.quota"}],
+            }
+        },
+        request=httpx.Request("GET", "https://www.googleapis.com/youtube/v3/search"),
+    )
+
+
+def test_an_upstream_failure_logs_the_reason_code_and_not_the_body(sink):
+    err = _upstream_failure("GET", "search", _youtube_403(TERM))
+
+    (line,) = sink.at("WARNING")
+    assert line["attrs"]["reason"] == "quotaExceeded"
+    assert line["attrs"]["status"] == 403
+    assert TERM not in json.dumps(line)
+    # The error is raised to callers that put it in a log line of their own (routes/channels.py)
+    # and in a 422 answered to the user, so the body must not travel in it either.
+    assert TERM not in str(err)
+    assert err.reason == "quotaExceeded"
+
+
+@pytest.fixture
+def consented():
+    """The request scope of a user who turned Teljes (TRACE) on for themselves."""
+    scope = logs._RequestScope()
+    scope.user_id, scope.level = 7, logs.TRACE
+    token = logs._scope.set(scope)
+    try:
+        yield
+    finally:
+        logs._scope.reset(token)
+
+
+def test_the_body_is_kept_for_the_one_user_who_consented_to_it(sink, consented):
+    _upstream_failure("GET", "search", _youtube_403(TERM))
+
+    # The diagnostic is not lost, it is gated — on the consent the privacy policy describes.
+    (trace,) = sink.at("TRACE")
+    assert TERM in trace["attrs"]["body"]
+
+
+def test_a_process_running_at_trace_is_not_anyones_consent(sink):
+    """`sink` runs the whole tree at TRACE — what an admin setting `log_level` to TRACE does. That
+    buys more detail, not every user's search terms quoted back (`logs._traced`)."""
+    _upstream_failure("GET", "search", _youtube_403(TERM))
+
+    assert sink.at("TRACE") == []
+    assert TERM not in json.dumps(sink.objects())
+
+
+def test_a_scope_with_no_elevation_is_not_consent():
+    scope = logs._RequestScope()
+    scope.user_id = 7
+    token = logs._scope.set(scope)
+    try:
+        assert not logs.consented_trace()
+    finally:
+        logs._scope.reset(token)
+    assert not logs.consented_trace()  # a scheduler thread: no request at all
+
+
+def test_a_refused_token_refresh_names_the_oauth_reason_and_not_its_prose(sink, monkeypatch):
+    """The token endpoint's error is `{"error": "<code>", "error_description": <prose>}` — not the
+    Data API's shape. The refresh goes through the same door, so it gets the same treatment."""
+    from types import SimpleNamespace
+
+    from app.youtube import client as yt
+
+    resp = httpx.Response(
+        400,
+        json={"error": "invalid_grant", "error_description": "Token has been expired or revoked."},
+        request=httpx.Request("POST", "https://oauth2.googleapis.com/token"),
+    )
+    client = yt.YouTubeClient.__new__(yt.YouTubeClient)  # no HTTP client: `_send` is the stub
+    client.db = None
+    client.user = SimpleNamespace(
+        token=SimpleNamespace(access_token=None, expiry=None, refresh_token_enc=b"x")
+    )
+    monkeypatch.setattr(yt, "decrypt_optional", lambda value: None)
+    monkeypatch.setattr(yt, "decrypt", lambda value: "a-refresh-token")
+    monkeypatch.setattr(yt.sysconfig, "get_str", lambda db, key: "")
+    monkeypatch.setattr(client, "_send", lambda *a, **k: resp)
+
+    with pytest.raises(YouTubeError) as caught:
+        client._access_token()
+    err = caught.value
+
+    (line,) = sink.at("WARNING")
+    assert line["attrs"]["reason"] == "invalid_grant"
+    assert err.reason == "invalid_grant"
+    assert "revoked" not in str(err) and "revoked" not in json.dumps(line)
+
+
+def test_an_unparseable_body_still_names_the_status(sink):
+    """A proxy answering HTML has no `reason` to take — the line must still be worth reading, and
+    must still not fall back to printing the body."""
+    resp = httpx.Response(
+        502,
+        text="<html><body>Bad Gateway</body></html>",
+        request=httpx.Request("GET", "https://www.googleapis.com/youtube/v3/search"),
+    )
+    err = _upstream_failure("GET", "search", resp)
+
+    (line,) = sink.at("WARNING")
+    assert line["attrs"] == {
+        "method": "GET",
+        "path": "search",
+        "status": 502,
+        "reason": "<unparseable>",
+    }
+    assert "Bad Gateway" not in str(err)
+    assert isinstance(err, YouTubeError)
+
+
+def test_a_sign_in_names_the_account_by_id_and_not_by_address(client, make_user, sink):
+    make_user(email="signing.in@example.com", password="correct horse battery")
+
+    r = client.post(
+        "/auth/password-login",
+        json={"email": "signing.in@example.com", "password": "correct horse battery"},
+    )
+    assert r.status_code == 200
+
+    (line,) = [o for o in sink.at("INFO") if o.get("event") == "auth.login.ok"]
+    assert line["attrs"]["method"] == "password"
+    assert isinstance(line["attrs"]["uid"], int)
+    # Not `[email]` either: the redacted form is what made the old line useless.
+    assert "signing.in" not in json.dumps(line)
+    assert logs.REDACTED_EMAIL not in json.dumps(line)
+
+
+# --- the family ------------------------------------------------------------------------------
+# Names that hold something a person typed or a place they went. A log call may still pass one —
+# `main.py` logs the instance's OWN setup URL — but it has to say so on the line, which is the
+# difference between a decision and an oversight.
+_BANNED_NAMES = frozenset(
+    {
+        "term",
+        "q",
+        "query",
+        "search_term",
+        "email",
+        "emails",
+        "recipients",
+        "recipient",
+        "url",
+        "webpage_url",
+        "source_url",
+        "source_webpage_url",
+        "password",
+        # A download's path is built from its channel and its title (`storage.rel_path`), so it
+        # names what someone downloaded exactly as the URL does. Log the asset id.
+        "rel",
+        "rel_path",
+        "title",
+    }
+)
+_BANNED_ATTRS = frozenset(
+    {"text", "content", "email", "url", "query", "rel_path", "title", "display_name"}
+)
+_ESCAPE = "log-pii-ok"
+_LOG_METHODS = frozenset(
+    {"debug", "info", "warning", "warn", "error", "exception", "critical", "log"}
+)
+
+
+def _is_log_call(node: ast.AST) -> bool:
+    if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):
+        return False
+    if node.func.attr not in _LOG_METHODS:
+        return False
+    base, name = node.func.value, ""
+    while isinstance(base, ast.Attribute):
+        name = f"{base.attr}.{name}"
+        base = base.value
+    if isinstance(base, ast.Name):
+        name = f"{base.id}.{name}"
+    return "log" in name.lower()
+
+
+def _offending(node: ast.Call) -> set[str]:
+    found: set[str] = set()
+    for sub in ast.walk(node):
+        if isinstance(sub, ast.Name) and sub.id in _BANNED_NAMES:
+            found.add(sub.id)
+        elif isinstance(sub, ast.Attribute) and sub.attr in _BANNED_ATTRS:
+            found.add(f".{sub.attr}")
+    return found
+
+
+def test_no_log_call_passes_what_a_person_typed():
+    root = pathlib.Path(__file__).resolve().parents[1] / "app"
+    offenders = []
+    for path in sorted(root.rglob("*.py")):
+        src = path.read_text(encoding="utf-8")
+        lines = src.splitlines()
+        for node in ast.walk(ast.parse(src)):
+            if not _is_log_call(node):
+                continue
+            names = _offending(node)
+            if not names:
+                continue
+            # The call's own lines plus the one directly above it — a marker belongs either
+            # on the argument or on the comment line introducing the call, and nowhere
+            # further, or one call's justification would excuse its neighbour.
+            start = max(0, node.lineno - 2)
+            span = " ".join(lines[start : (node.end_lineno or node.lineno)])
+            if _ESCAPE in span:
+                continue
+            offenders.append(f"{path.relative_to(root)}:{node.lineno} {sorted(names)}")
+    assert offenders == [], (
+        "a log call carries user-typed content; log an id or a reason code instead, or mark "
+        f"the argument (or the line directly above the call) `# {_ESCAPE}: <why it is "
+        f"safe>` — {offenders}"
+    )
+
+
+_AT_LEAST_DEBUG = {"DEBUG", "INFO", "WARNING", "WARN", "ERROR", "CRITICAL", "FATAL"}
+
+
+def _may_run_at_trace(call: ast.Call) -> bool:
+    """A `log.log(level, …)` whose level is not provably DEBUG or above: `logs.TRACE`, a bare
+    `TRACE`, a literal 5 — or a variable, which could be any of them."""
+    if call.func.attr != "log" or not call.args:
+        return False
+    level = call.args[0]
+    if isinstance(level, ast.Attribute | ast.Name):
+        return (level.attr if isinstance(level, ast.Attribute) else level.id) not in _AT_LEAST_DEBUG
+    if isinstance(level, ast.Constant) and isinstance(level.value, int):
+        return level.value < logging.DEBUG
+    return True
+
+
+def _is_consent_check(test: ast.expr) -> bool:
+    """Exactly `…consented_trace()`: not `not …`, not `x or …` — each of those runs the body when
+    there is no consent."""
+    return (
+        isinstance(test, ast.Call)
+        and isinstance(test.func, ast.Attribute | ast.Name)
+        and (test.func.attr if isinstance(test.func, ast.Attribute) else test.func.id)
+        == "consented_trace"
+    )
+
+
+def _guarded_by_consent(node: ast.AST, parents: dict[ast.AST, ast.AST]) -> bool:
+    """The node runs only inside the BODY of an `if <consented_trace()>:` — not its `else`."""
+    while node in parents:
+        parent = parents[node]
+        if isinstance(parent, ast.If) and _is_consent_check(parent.test) and node in parent.body:
+            return True
+        node = parent
+    return False
+
+
+def _parents(tree: ast.AST) -> dict[ast.AST, ast.AST]:
+    return {child: node for node in ast.walk(tree) for child in ast.iter_child_nodes(node)}
+
+
+_GATE_SHAPES = {
+    "the body of the check": ("if logs.consented_trace():\n    log.log(logs.TRACE, 'x')", True),
+    "its else": (
+        "if logs.consented_trace():\n    pass\nelse:\n    log.log(logs.TRACE, 'x')",
+        False,
+    ),
+    "a negated check": ("if not logs.consented_trace():\n    log.log(logs.TRACE, 'x')", False),
+    "an or": ("if debug or logs.consented_trace():\n    log.log(logs.TRACE, 'x')", False),
+    "a nested block in the body": (
+        "if logs.consented_trace():\n    for t in ts:\n        log.log(logs.TRACE, 'x')",
+        True,
+    ),
+}
+
+
+@pytest.mark.parametrize("shape", list(_GATE_SHAPES))
+def test_the_consent_gate_is_the_body_of_the_check_and_nothing_else(shape):
+    src, gated = _GATE_SHAPES[shape]
+    tree = ast.parse(src)
+    (call,) = [n for n in ast.walk(tree) if isinstance(n, ast.Call) and _is_log_call(n)]
+    assert _guarded_by_consent(call, _parents(tree)) is gated
+
+
+@pytest.mark.parametrize(
+    ("level", "trace"),
+    [
+        ("logs.TRACE", True),
+        ("TRACE", True),
+        ("5", True),
+        ("level", True),
+        ("levels[scope]", True),
+        ("logging.DEBUG", False),
+        ("logging.WARNING", False),
+        ("20", False),
+    ],
+)
+def test_a_level_that_may_be_trace_is_treated_as_trace(level, trace):
+    (call,) = [n for n in ast.walk(ast.parse(f"log.log({level}, 'x')")) if isinstance(n, ast.Call)]
+    assert _may_run_at_trace(call) is trace
+
+
+def test_every_escaped_trace_line_is_gated_on_the_users_own_consent():
+    """`log-pii-ok` on a TRACE line means "this is what the user's Teljes consent is for". The TRACE
+    level alone does not hold that promise: an admin can run the whole process at TRACE, and a
+    scheduler thread has no user at all. So the line must sit under `logs.consented_trace()`."""
+    root = pathlib.Path(__file__).resolve().parents[1] / "app"
+    offenders = []
+    for path in sorted(root.rglob("*.py")):
+        src = path.read_text(encoding="utf-8")
+        lines = src.splitlines()
+        tree = ast.parse(src)
+        parents = _parents(tree)
+        for node in ast.walk(tree):
+            if not _is_log_call(node) or not _offending(node):
+                continue
+            span = " ".join(lines[max(0, node.lineno - 2) : (node.end_lineno or node.lineno)])
+            if (
+                _may_run_at_trace(node)
+                and _ESCAPE in span
+                and not _guarded_by_consent(node, parents)
+            ):
+                offenders.append(f"{path.relative_to(root)}:{node.lineno}")
+    assert offenders == [], (
+        f"a TRACE line with user content is not under consented_trace(): {offenders}"
+    )
+
+
+def test_a_database_error_does_not_carry_the_values_it_was_given(db):
+    """DB lane. A failed write's message is logged and stored as the job's error text; with its
+    bound parameters in it, that is the title or the file path the row was about."""
+    from sqlalchemy import exc as sa_exc
+    from sqlalchemy import text
+
+    # Postgres's own text for this one names no value ("value too long for type character
+    # varying(3)"), so what is left to leak is SQLAlchemy's `[parameters: …]` suffix.
+    db.execute(text("CREATE TEMP TABLE pii_probe (x varchar(3))"))
+    with pytest.raises(sa_exc.DBAPIError) as caught:
+        db.execute(text("INSERT INTO pii_probe VALUES (:v)"), {"v": "A_Private_Title"})
+    db.rollback()
+    assert "value too long" in str(caught.value)
+    assert "A_Private_Title" not in str(caught.value)

backend/tests/test_logger_names.py

@@ -0,0 +1,121 @@
+"""E22.5 S2 — every module's logger is named after the module.
+
+The names used to be literals, and 25 of them covered 34 modules: `plex/client.py`, `plex/stream.py`,
+`plex/subtitles.py`, `plex/sync.py` and two route modules all called themselves `siftlode.plex`, and
+`downloads/pipeline.py` called itself `siftlode.worker`. A level pinned per logger could not separate
+them, and neither can E22.6's console filter — the `logger` field is in every stored row precisely so
+it can be filtered on.
+
+So the name is derived, not written: `logs.get_logger(__name__)`. These cases pin the derivation and
+the rule, because the rule is only worth having if a hand-rolled name fails the gate.
+"""
+
+import ast
+import logging
+import pathlib
+import sys
+import types
+
+import pytest
+
+from app import logs
+
+APP = pathlib.Path(logs.__file__).resolve().parent
+_ESCAPE = "logger-name-ok"
+
+
+@pytest.mark.parametrize(
+    ("module", "expected"),
+    [
+        ("app.plex.sync", "siftlode.plex.sync"),
+        ("app.plex.stream", "siftlode.plex.stream"),
+        ("app.routes.channels", "siftlode.routes.channels"),
+        ("app.worker", "siftlode.worker"),
+        # A script that is not a module of ours (pytest itself, here): it lands on the root of our
+        # tree rather than inventing `siftlode.__main__`, which nothing would think to filter on.
+        ("__main__", "siftlode"),
+    ],
+)
+def test_the_name_follows_the_module(module, expected):
+    assert logs.get_logger(module).name == expected
+
+
+@pytest.mark.parametrize(
+    ("spec_name", "expected"),
+    [
+        # How the worker and the download child actually run: `python -m app.worker`, and the
+        # worker's own `[sys.executable, "-m", "app.downloads.runner", ...]`. `__name__` is
+        # `__main__` in both, and taken at its word both processes logged as the bare root.
+        ("app.worker", "siftlode.worker"),
+        ("app.downloads.runner", "siftlode.downloads.runner"),
+    ],
+)
+def test_a_module_run_with_dash_m_keeps_its_own_name(monkeypatch, spec_name, expected):
+    main = types.ModuleType("__main__")
+    main.__spec__ = types.SimpleNamespace(name=spec_name)
+    monkeypatch.setitem(sys.modules, "__main__", main)
+
+    assert logs.get_logger("__main__").name == expected
+
+
+def test_a_plain_script_has_no_spec_and_stays_the_root(monkeypatch):
+    main = types.ModuleType("__main__")
+    main.__spec__ = None
+    monkeypatch.setitem(sys.modules, "__main__", main)
+
+    assert logs.get_logger("__main__").name == "siftlode"
+
+
+def test_every_derived_logger_stays_under_the_one_tree():
+    """`configure_logging` hangs the level and the handler off the root, and the tests attach their
+    sink to `siftlode` — a name outside that tree would be invisible to both."""
+    for module in ("app.main", "app.plex.sync", "app.routes.plex.people", "app"):
+        assert logs.get_logger(module).name.startswith("siftlode")
+
+
+def test_the_module_logger_is_reachable_by_its_derived_name():
+    """Not a new logger object per call: the stdlib registry is keyed by name, and an operator
+    pinning `siftlode.plex.sync` has to reach the same instance the module logs through."""
+    from app.plex import sync
+
+    assert sync.log is logging.getLogger("siftlode.plex.sync")
+
+
+def test_no_module_hand_rolls_a_logger_name():
+    offenders = []
+    for path in sorted(APP.rglob("*.py")):
+        src = path.read_text(encoding="utf-8")
+        lines = src.splitlines()
+        for node in ast.walk(ast.parse(src)):
+            if not isinstance(node, ast.Call):
+                continue
+            func = node.func
+            called = (
+                func.attr
+                if isinstance(func, ast.Attribute)
+                else func.id
+                if isinstance(func, ast.Name)
+                else ""
+            )
+            if called == "get_logger":
+                # The derivation itself is only as good as its argument: a literal here does not
+                # name a module, it collapses to the root and pools modules together again.
+                arg = node.args[0] if len(node.args) == 1 else None
+                if isinstance(arg, ast.Name) and arg.id == "__name__" and not node.keywords:
+                    continue
+            elif called == "getLogger":
+                # A no-argument `logging.getLogger()` is the ROOT logger — level plumbing, not a
+                # module's own logger, and nothing to name. A `name=` keyword is still a name.
+                if not node.args and not node.keywords:
+                    continue
+            else:
+                continue
+            start = max(0, node.lineno - 2)  # the call's lines, plus the one above it
+            span = " ".join(lines[start : (node.end_lineno or node.lineno)])
+            if _ESCAPE in span:
+                continue
+            offenders.append(f"{path.relative_to(APP)}:{node.lineno}")
+    assert offenders == [], (
+        "a module names its own logger; use `logs.get_logger(__name__)`, or say why the name is "
+        f"not the module's with `# {_ESCAPE}: <why>` — {offenders}"
+    )

backend/tests/test_logs.py

@@ -0,0 +1,385 @@
+"""E22 S2 — the one logging configuration, its redaction, and the request line that replaced
+uvicorn's access log.
+
+Redaction is the part worth guarding hardest: it is applied at the HANDLER, so these cases are the
+only thing standing between a future call site and a token in a log we attach to a Gitea issue.
+The request-line cases pin what uvicorn's access log got wrong — the query string (`?account=1`,
+`?token=…`) was printed on every request, and nothing correlated two lines of the same request.
+
+Pure logic + an in-process ASGI app: no database.
+"""
+
+import json
+import logging
+from pathlib import Path
+
+import pytest
+from fastapi import FastAPI, Request
+from fastapi.testclient import TestClient
+
+from app import logs
+from app.config import LogLevel, choices_of
+
+
+class _Sink(logging.Handler):
+    """Captures records AND their formatted JSON, with the same filter/formatter the real stdout
+    handler uses — so a test sees exactly the bytes a deployment would write."""
+
+    def __init__(self):
+        super().__init__(level=logging.NOTSET)
+        self.lines: list[str] = []
+        self.records: list[logging.LogRecord] = []
+        self.setFormatter(logs.JsonFormatter())
+        self.addFilter(logs.ContextFilter())
+
+    def emit(self, record):
+        self.lines.append(self.format(record))
+        self.records.append(record)
+
+    def objects(self) -> list[dict]:
+        return [json.loads(line) for line in self.lines]
+
+
+@pytest.fixture
+def sink():
+    """Attached to `siftlode` (not root) so a `configure_logging` call inside a test — which
+    replaces the ROOT handlers — can't silently detach it mid-test."""
+    logger = logging.getLogger("siftlode")
+    handler = _Sink()
+    previous = logger.level
+    logger.addHandler(handler)
+    logger.setLevel(logs.TRACE)
+    try:
+        yield handler
+    finally:
+        logger.removeHandler(handler)
+        logger.setLevel(previous)
+
+
+# --- redaction ------------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "text, must_not_contain",
+    [
+        ("user signed in: someone@example.com", "someone@example.com"),
+        ("callback https://app/auth/callback?code=4/0AY&state=xyz", "4/0AY"),
+        ("mail link https://app/#verify=abc123def", "abc123def"),
+        ("share https://app/s?token=SHARETOKEN", "SHARETOKEN"),
+        ("reset https://app/#reset=RESETTOKEN", "RESETTOKEN"),
+    ],
+)
+def test_free_text_secrets_never_reach_the_line(text, must_not_contain):
+    assert must_not_contain not in logs.redact_text(text)
+
+
+def test_a_key_that_names_a_secret_is_dropped_whole():
+    # Not truncated to a prefix: a prefix is still a leak and is useless for triage anyway.
+    out = logs.redact(
+        {
+            "password": "hunter2",
+            "current_password": "hunter2",
+            "plex_server_token": "abc",
+            "Cookie": "session=xyz",
+            "authorization": "Bearer abc",
+            "nested": {"refresh_token": "rt", "keep": "visible"},
+            "list": [{"api_key": "k"}],
+            "count": 7,
+        }
+    )
+    assert out["password"] == logs.REDACTED
+    assert out["current_password"] == logs.REDACTED
+    assert out["plex_server_token"] == logs.REDACTED
+    assert out["Cookie"] == logs.REDACTED
+    assert out["authorization"] == logs.REDACTED
+    assert out["nested"]["refresh_token"] == logs.REDACTED
+    assert out["nested"]["keep"] == "visible"  # only the named keys go
+    assert out["list"][0]["api_key"] == logs.REDACTED
+    assert out["count"] == 7
+
+
+def test_an_email_in_an_attribute_value_is_scrubbed_too():
+    assert logs.redact({"to": "a.b+tag@mail.example.org"})["to"] == logs.REDACTED_EMAIL
+
+
+# The SAME cases frontend/src/lib/log.contract.test.ts runs against the browser's copy of redact():
+# the one thing that keeps the two logs' idea of "safe to keep" from drifting apart.
+_REDACTION_CASES = json.loads(
+    (Path(__file__).parent / "fixtures" / "log_redaction_cases.json").read_text(encoding="utf-8")
+)["cases"]
+
+
+@pytest.mark.parametrize("case", _REDACTION_CASES, ids=[c["name"] for c in _REDACTION_CASES])
+def test_the_shared_redaction_cases(case):
+    assert logs.redact(case["value"], key=case["key"]) == case["expected"]
+
+
+def test_redaction_runs_on_the_message_the_attrs_and_the_traceback(sink):
+    log = logging.getLogger("siftlode.test")
+    try:
+        raise RuntimeError("boom for someone@example.com")
+    except RuntimeError:
+        log.exception(
+            "failed for %s",
+            "someone@example.com",
+            extra=logs.event("test.failed", password="hunter2", url="https://x/?token=T0K3N"),
+        )
+    line = sink.lines[-1]
+    assert "someone@example.com" not in line
+    assert "hunter2" not in line
+    assert "T0K3N" not in line
+    payload = json.loads(line)
+    assert payload["attrs"]["password"] == logs.REDACTED
+    assert "RuntimeError" in payload["exc"]  # the diagnosis survives the scrubbing
+
+
+# --- event shape ----------------------------------------------------------------------------
+
+
+def test_every_line_carries_the_agreed_fields(sink):
+    logging.getLogger("siftlode.test").info("hello", extra=logs.event("test.hello", n=1))
+    payload = sink.objects()[-1]
+    for field in ("ts", "level", "sev", "source", "logger", "msg", "event", "version"):
+        assert field in payload, field
+    assert payload["level"] == "INFO"
+    assert payload["sev"] == 9
+    assert payload["event"] == "test.hello"
+    assert payload["attrs"] == {"n": 1}
+
+
+def test_a_plain_log_call_still_produces_a_well_formed_line(sink):
+    """An event code is optional — 125 existing call sites have none yet (E22.5 adds them), and they
+    must not be worse off than before this module."""
+    logging.getLogger("siftlode.test").warning("just a message")
+    payload = sink.objects()[-1]
+    assert payload["msg"] == "just a message"
+    assert payload["level"] == "WARNING"
+    assert "event" not in payload
+
+
+def test_a_broken_format_call_reports_itself_instead_of_raising():
+    """A record is formatted directly rather than logged: a genuinely broken call ALSO breaks
+    pytest's own capture handler (stdlib formatting raises there too), which would fail the test for
+    the wrong reason. What is under test is our formatter's refusal to raise."""
+    record = logging.LogRecord(
+        "siftlode.test", logging.INFO, __file__, 0, "two args %s %s", ("only-one",), None
+    )
+    payload = json.loads(logs.JsonFormatter().format(record))
+    assert "log formatting failed" in payload["msg"]
+
+
+def test_bound_correlation_appears_and_is_released(sink):
+    log = logging.getLogger("siftlode.test")
+    with logs.bind(job_id=42):
+        log.info("inside")
+    log.info("outside")
+    assert sink.objects()[-2]["job_id"] == 42
+    assert "job_id" not in sink.objects()[-1]
+
+
+def test_every_selectable_level_resolves_to_a_real_logging_level():
+    # The picker's options come from `LogLevel`; each must be a level `logging` knows, TRACE included.
+    assert [logs.resolve_level(n) for n in choices_of(LogLevel)] == [
+        logs.TRACE,
+        logging.DEBUG,
+        logging.INFO,
+        logging.WARNING,
+        logging.ERROR,
+    ]
+    with pytest.raises(ValueError):
+        logs.resolve_level("nonsense")
+
+
+def test_configure_logging_silences_the_uvicorn_access_log_and_keeps_ours_propagating():
+    logs.configure_logging("api")
+    assert logging.getLogger("uvicorn.access").getEffectiveLevel() >= logging.CRITICAL
+    # propagate stays on so every line reaches the ONE root handler — and so pytest's caplog works.
+    assert logging.getLogger("siftlode.anything").propagate is True
+    assert len(logging.getLogger().handlers) == 1
+
+
+# --- the request line -----------------------------------------------------------------------
+
+
+def _client() -> TestClient:
+    app = FastAPI()
+    app.middleware("http")(logs.log_requests)
+
+    @app.get("/api/videos/{video_id}")
+    def _video(video_id: str, q: str | None = None):
+        return {"ok": video_id}
+
+    @app.get("/healthz")
+    def _health():
+        return {"ok": True}
+
+    @app.get("/boom")
+    def _boom():
+        raise RuntimeError("kaboom")
+
+    return TestClient(app, raise_server_exceptions=False)
+
+
+def test_the_request_line_has_the_route_template_and_no_query_string(sink):
+    logging.getLogger("siftlode.http").setLevel(logs.TRACE)
+    r = _client().get("/api/videos/abc123?q=cats&token=SECRET")
+    assert r.status_code == 200
+    payload = sink.objects()[-1]
+    assert payload["event"] == "http.request"
+    # The TEMPLATE, so every video request groups into one queryable path...
+    assert payload["attrs"]["path"] == "/api/videos/{video_id}"
+    # ...and the query string — uvicorn's access log printed it, `?account=1` and all — is gone.
+    assert "SECRET" not in sink.lines[-1]
+    assert "cats" not in sink.lines[-1]
+    assert payload["attrs"]["status"] == 200
+    assert isinstance(payload["attrs"]["duration_ms"], float)
+
+
+def test_anonymous_request_lines_are_capped_per_client_for_storage_but_still_logged(sink):
+    # A scanner's 404s are WARNINGs, and WARNINGs are stored. Past the cap they stay on stdout but are
+    # marked not-for-storage; a signed-in request is never capped. No address appears in any line.
+    logs._anonymous_lines.reset()
+    client = _client()
+    for _ in range(logs.ANONYMOUS_LINES_PER_MINUTE + 3):
+        client.get("/wp-login.php")
+    lines = sink.objects()[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
+    assert len(lines) == logs.ANONYMOUS_LINES_PER_MINUTE + 3  # stdout keeps every one
+    records = sink.records[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
+    stored = [getattr(r, logs.PERSIST, True) for r in records]
+    assert stored.count(False) == 3 and all(stored[: logs.ANONYMOUS_LINES_PER_MINUTE])
+    assert "testclient" not in "".join(sink.lines)  # the client key never reaches the line
+    logs._anonymous_lines.reset()
+
+
+def test_a_signed_in_request_is_never_capped_and_names_its_account(sink):
+    logs._anonymous_lines.reset()
+    app = FastAPI()
+    app.middleware("http")(logs.log_requests)
+
+    @app.get("/api/me")
+    def _me(request: Request):
+        request.state.user_id = 42  # what the auth dependency does
+        return {}
+
+    client = TestClient(app)
+    for _ in range(logs.ANONYMOUS_LINES_PER_MINUTE + 3):
+        client.get("/api/me")
+    records = sink.records[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
+    assert all(getattr(r, logs.PERSIST, True) for r in records)
+    assert sink.objects()[-1]["user_id"] == 42 and "user_id" not in sink.objects()[-1]["attrs"]
+
+
+def test_the_request_id_is_echoed_and_shared_by_every_line_of_that_request(sink):
+    client = _client()
+    r = client.get("/api/videos/abc123")
+    rid = r.headers["X-Request-Id"]
+    assert rid and sink.objects()[-1]["request_id"] == rid
+
+
+def test_the_route_template_goes_back_to_the_browser_but_never_a_raw_path(sink):
+    # The browser logs its own copy of each request, grouped by this template (frontend lib/http.ts).
+    client = _client()
+    assert client.get("/api/videos/SeCrEt-VaLuE").headers["X-Route"] == "/api/videos/{video_id}"
+    # Unmatched: the only template there is the raw path, which may carry a value — so none is sent.
+    assert "X-Route" not in client.get("/api/share/SeCrEt-VaLuE").headers
+
+
+def test_a_client_supplied_id_is_reused_only_when_it_is_well_formed(sink):
+    client = _client()
+    good = client.get("/api/videos/x", headers={"X-Request-Id": "web-0123456789ab"})
+    assert good.headers["X-Request-Id"] == "web-0123456789ab"
+    # An unbounded header would be a way to write arbitrary text into the log; short/odd ones are
+    # replaced with our own id rather than trusted.
+    for bogus in ("short", "has spaces and \n newline", "x" * 200):
+        r = client.get("/api/videos/x", headers={"X-Request-Id": bogus})
+        assert r.headers["X-Request-Id"] != bogus
+
+
+def test_the_healthcheck_is_logged_at_debug_so_it_cannot_drown_the_log(sink):
+    http = logging.getLogger("siftlode.http")
+    http.setLevel(logs.TRACE)
+    _client().get("/healthz")
+    assert sink.objects()[-1]["level"] == "DEBUG"
+    # ...but it is still logged when something is wrong with it: the floor drops, nothing is hidden.
+    http.setLevel(logging.INFO)
+    before = len(sink.lines)
+    _client().get("/healthz")
+    assert len(sink.lines) == before
+
+
+def test_a_failing_request_is_logged_by_status_and_an_unhandled_error_is_logged_at_all(sink):
+    client = _client()
+    client.get("/api/videos/missing/deeper")  # 404 from the router
+    assert sink.objects()[-1]["level"] == "WARNING"
+    client.get("/boom")
+    payload = sink.objects()[-1]
+    # The request that most needs a line is the one that raised: uvicorn's access log wrote it, but
+    # our middleware sees the exception instead of a response, so it has to log it explicitly.
+    assert payload["level"] == "ERROR"
+    assert payload["event"] == "http.request"
+    assert "RuntimeError" in payload["exc"]
+
+
+# --- a stack whose messages are data (`logs.scrubbed_stack`) -----------------------------------
+def _upper(exc: BaseException) -> str:
+    return str(exc).upper()
+
+
+def _raised(make):
+    try:
+        make()
+    except Exception as exc:  # noqa: BLE001 — whatever the case raised is the case
+        return exc
+    raise AssertionError("the case did not raise")
+
+
+def test_a_raise_from_is_printed_as_its_direct_cause():
+    def make():
+        try:
+            raise KeyError("inner")
+        except KeyError as inner:
+            raise ValueError("outer") from inner
+
+    stack = logs.scrubbed_stack(_raised(make), _upper)["scrubbed_stack"]
+    assert "The above exception was the direct cause" in stack
+    assert "During handling" not in stack
+    assert "'INNER'" in stack and "OUTER" in stack
+
+
+def test_a_raise_from_none_hides_the_link_its_author_hid():
+    def make():
+        try:
+            raise KeyError("hidden")
+        except KeyError:
+            raise ValueError("shown") from None
+
+    stack = logs.scrubbed_stack(_raised(make), _upper)["scrubbed_stack"]
+    assert "HIDDEN" not in stack and "SHOWN" in stack
+
+
+def test_a_chain_that_loops_back_ends():
+    a, b = ValueError("a"), ValueError("b")
+    a.__context__, b.__context__ = b, a
+    assert "A" in logs.scrubbed_stack(a, _upper)["scrubbed_stack"]
+
+
+def test_a_message_that_cannot_be_rendered_does_not_fail_the_report():
+    def refuse(exc):
+        raise RuntimeError("no")
+
+    assert (
+        logs.scrubbed_stack(ValueError("x"), refuse)["scrubbed_stack"]
+        == "ValueError: <unprintable>"
+    )
+
+
+def test_the_text_format_carries_the_scrubbed_stack_too():
+    title = "a private title"  # a variable: a frame prints its source line, which is code
+    exc = ValueError(title)
+    record = logging.LogRecord("siftlode.x", logging.WARNING, __file__, 1, "failed", None, None)
+    for key, value in logs.scrubbed_stack(exc, lambda e: "…").items():
+        setattr(record, key, value)
+
+    line = logs.TextFormatter().format(record)
+    assert "ValueError: …" in line and "private" not in line
+    assert "scrubbed_stack" not in line  # the stack, not an attribute

backend/tests/test_logsink.py

@@ -0,0 +1,313 @@
+"""E22 S3 — the `log_events` sink: what reaches the table, in what shape, and what it refuses to do.
+
+The sink is driven synchronously (`write_pending()` on the test's own thread) so nothing here waits on
+a timer. It is attached to a private logger rather than the root, for the same reason test_logs.py's
+capture handler is: a `configure_logging` call elsewhere replaces the root handlers.
+DB lane (`db` fixture): the rows are read back from a real Postgres.
+"""
+
+import logging
+import threading
+from datetime import datetime, timedelta, timezone
+
+import pytest
+from fastapi import BackgroundTasks
+from sqlalchemy import create_engine, func, select
+
+from app import logs, logsink, scheduler
+from app.auth import purge_user
+from app.db import engine
+from app.models import LogEvent
+
+
+@pytest.fixture
+def stored(db):
+    """A LogSink on the logger `siftlode.sinktest`, and a reader for the rows it wrote."""
+    sink = logsink.LogSink(engine)
+    logger = logging.getLogger("siftlode.sinktest")
+    logger.addHandler(sink)
+    logger.setLevel(logs.TRACE)
+
+    def rows() -> list[LogEvent]:
+        sink.write_pending()
+        db.expire_all()
+        return list(db.execute(select(LogEvent).order_by(LogEvent.id)).scalars())
+
+    try:
+        yield sink, logger, rows
+    finally:
+        logger.removeHandler(sink)
+
+
+def test_warnings_and_request_lines_are_stored_and_everything_else_is_not(stored):
+    _sink, log, rows = stored
+    log.info("an ordinary info line")
+    log.debug("GET /healthz 200", extra=logs.event("http.request", path="/healthz", status=200))
+    log.info("GET /api/feed 200", extra=logs.event("http.request", path="/api/feed", status=200))
+    log.warning("scanner", extra={**logs.event("http.request", status=404), logs.PERSIST: False})
+    log.warning("disk nearly full")
+    assert [r.msg for r in rows()] == ["GET /api/feed 200", "disk nearly full"]
+
+
+def test_a_row_is_redacted_exactly_like_the_stdout_line(stored):
+    _sink, log, rows = stored
+    try:
+        raise ValueError("reset link https://x.test/#reset=abc123")
+    except ValueError:
+        log.exception(
+            "sign-in failed for someone@example.com",
+            extra=logs.event("auth.failed", password="hunter2", reason="bad_password"),
+        )
+    (row,) = rows()
+    assert row.level == "ERROR" and row.sev == logs.SEVERITY["ERROR"]
+    assert row.event == "auth.failed"
+    assert "someone@example.com" not in row.msg and logs.REDACTED_EMAIL in row.msg
+    assert row.attrs == {"password": logs.REDACTED, "reason": "bad_password"}
+    assert "abc123" not in row.exc and "ValueError" in row.exc
+    assert row.source == logs.current_source() and row.version
+
+
+def test_correlation_is_captured_when_the_line_is_logged_not_when_it_is_written(stored):
+    _sink, log, rows = stored
+    with logs.bind(job_id=7, request_id="req-0123456789"):
+        log.warning("inside the job")
+    # The writer runs later, outside the bind — the row still belongs to job 7.
+    (row,) = rows()
+    assert (row.job_id, row.request_id) == (7, "req-0123456789")
+
+
+def test_a_flood_of_one_message_is_stored_up_to_the_limit_and_then_summarised(stored):
+    sink, log, rows = stored
+    limit = logsink.REPEATS_PER_MINUTE
+    for i in range(limit + 70):
+        log.warning("RSS poll skipped for channel %s: HTTP 404", f"UC{i}")
+    log.warning("a different message")
+    assert len(rows()) == limit + 1  # the flood capped; the other message untouched
+    sink._repeats.started -= logsink.REPEAT_WINDOW_SECONDS  # the minute is over
+    notice = rows()[-1]
+    assert notice.event == "log.repeats_suppressed"
+    assert notice.attrs["suppressed"] == 70
+    assert notice.attrs["template"] == "RSS poll skipped for channel %s: HTTP 404"
+
+
+def test_a_pass_through_template_groups_by_its_text_not_by_the_bare_placeholder(stored):
+    """`"%s"` is how the yt-dlp adapter hands over yt-dlp's own line, and as a key it says nothing:
+    every distinct warning would share one bucket and the thirty-first — whatever it said — would be
+    dropped as a "repeat" of the first. The text groups instead, numbers folded, so a retry storm is
+    still one repeat and the warning that explains the failure still gets through."""
+    sink, log, rows = stored
+    limit = logsink.REPEATS_PER_MINUTE
+    for i in range(limit + 10):
+        log.warning("%s", f"[download] Got error: timed out. Retrying fragment {i} (1/10)...")
+    log.warning("%s", "[youtube] …: Requested format is not available")
+    stored_rows = rows()
+    assert len(stored_rows) == limit + 1
+    assert stored_rows[-1].msg == "[youtube] …: Requested format is not available"
+    sink._repeats.started -= logsink.REPEAT_WINDOW_SECONDS
+    notice = rows()[-1]
+    assert notice.attrs["suppressed"] == 10
+    assert notice.attrs["template"].startswith(
+        "[download] Got error: timed out. Retrying fragment #"
+    )
+
+
+def test_a_process_that_ends_inside_the_minute_still_says_what_it_dropped(stored):
+    """Every download child lives well under the limiter's minute. Its last batch closes the window
+    early, or the repeats it dropped would go unreported."""
+    sink, log, rows = stored
+    for i in range(logsink.REPEATS_PER_MINUTE + 3):
+        log.warning("RSS poll skipped for channel %s: HTTP 404", f"UC{i}")
+    rows()
+    sink.write_pending(final=True)
+    notice = rows()[-1]
+    assert notice.event == "log.repeats_suppressed"
+    assert notice.attrs["suppressed"] == 3
+
+
+def test_request_lines_are_never_treated_as_a_flood(stored):
+    _sink, log, rows = stored
+    for _ in range(logsink.REPEATS_PER_MINUTE + 5):
+        log.info("%s %s %s", "GET", "/api/feed", 200, extra=logs.event("http.request"))
+    assert len(rows()) == logsink.REPEATS_PER_MINUTE + 5
+
+
+def test_a_full_queue_drops_and_says_how_many(db, monkeypatch):
+    monkeypatch.setattr(logsink, "QUEUE_MAX", 3)
+    sink = logsink.LogSink(engine)
+    log = logging.getLogger("siftlode.sinktest.small")
+    log.addHandler(sink)
+    try:
+        for i in range(5):
+            log.warning("message number %d", i)
+    finally:
+        log.removeHandler(sink)
+    assert sink.write_pending() == 4
+    notice = db.execute(select(LogEvent).where(LogEvent.event == "log.sink.dropped")).scalar_one()
+    assert notice.attrs == {"dropped": 2}
+
+
+def test_an_unreachable_database_costs_the_batch_not_the_caller(capsys):
+    dead = create_engine(
+        "postgresql+psycopg://x:y@127.0.0.1:1/none", connect_args={"connect_timeout": 1}
+    )
+    sink = logsink.LogSink(dead)
+    log = logging.getLogger("siftlode.sinktest.dead")
+    log.addHandler(sink)
+    try:
+        log.error("this has nowhere to go")  # must not raise
+    finally:
+        log.removeHandler(sink)
+    assert sink.write_pending() == 0
+    assert "could not store 1 log events" in capsys.readouterr().err
+
+
+def test_the_writer_thread_does_not_store_its_own_records(stored):
+    sink, log, rows = stored
+    sink._writer = threading.current_thread()  # as if this line came from inside the writer
+    log.error("a driver warning raised while inserting")
+    sink._writer = None
+    assert rows() == []
+
+
+def test_gc_keeps_the_retention_window_and_never_keeps_forever(db):
+    now = datetime.now(timezone.utc)
+
+    def add(age: timedelta) -> None:
+        db.add(
+            LogEvent(ts=now - age, level="WARNING", sev=13, source="api", logger="t", msg=str(age))
+        )
+
+    for age in (timedelta(hours=1), timedelta(days=2), timedelta(days=8)):
+        add(age)
+    db.commit()
+    assert logsink.gc(db, 7) == 1
+    assert logsink.gc(db, 0) == 1  # 0 is not "forever": stored logs always expire (floor: 1 day)
+    db.commit()
+    assert [r.msg for r in db.execute(select(LogEvent)).scalars()] == [str(timedelta(hours=1))]
+
+
+def test_log_gc_is_a_scheduled_job():
+    assert "log_gc" in scheduler.JOB_FUNCS and scheduler.JOB_INTERVALS["log_gc"] >= 1
+
+
+def test_deleting_an_account_deletes_its_stored_log_events(db, make_user):
+    gone, kept = make_user(), make_user()
+    for uid in (gone.id, kept.id, gone.id):
+        db.add(
+            LogEvent(
+                ts=datetime.now(timezone.utc),
+                level="INFO",
+                sev=9,
+                source="api",
+                logger="t",
+                msg="x",
+                user_id=uid,
+            )
+        )
+    db.commit()
+    purge_user(db, gone, BackgroundTasks())
+    assert [r.user_id for r in db.execute(select(LogEvent)).scalars()] == [kept.id]
+
+
+def test_an_erased_accounts_late_lines_are_stored_without_it(stored, db, make_user):
+    """The DELETE in purge_user cannot reach a row still on its way: one queued before the erasure,
+    or the erasing request's own access line, logged after the handler committed."""
+    _sink, log, rows = stored
+    user = make_user()
+    # `logs._erased` outlives the test; conftest's `_reset_log_erasures` clears it before each one.
+    log.warning("queued before the erasure", extra={"user_id": user.id})
+    purge_user(db, user, BackgroundTasks())
+    log.info("DELETE /api/me/account 200", extra={**logs.event("http.request"), "user_id": user.id})
+    assert [(r.msg, r.user_id) for r in rows()] == [
+        ("queued before the erasure", None),
+        ("DELETE /api/me/account 200", None),
+    ]
+
+
+def test_every_path_into_the_table_drops_an_erased_account():
+    """The two behavioural tests above cover the two writers there are today (this sink and
+    `POST /api/diagnostics/events`). This one covers the THIRD one nobody has written yet: a new
+    `insert(LogEvent)` that forgets the filter would leave an erased account's rows named, and no
+    test of an existing path can notice. Source-level on purpose — it is a rule about call sites."""
+    import pathlib
+
+    root = pathlib.Path(__file__).resolve().parents[1] / "app"
+    unfiltered = [
+        f"{path.relative_to(root)}:{n}"
+        for path in root.rglob("*.py")
+        for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1)
+        if "insert(LogEvent)" in line and "without_erased" not in line
+    ]
+    assert unfiltered == [], f"insert(LogEvent) without logsink.without_erased at {unfiltered}"
+
+
+def test_a_nul_character_costs_nothing_but_itself(stored):
+    """Postgres refuses NUL in text and JSONB, and a refused row fails its whole batch — every other
+    user's rows with it. `GET /assets/%00` logs exactly that."""
+    _sink, log, rows = stored
+    log.warning("before")
+    log.warning("GET /assets/\x00 404", extra=logs.event("http.request", path="/assets/\x00"))
+    log.warning("after")
+    before, nul, after = rows()
+    assert (before.msg, after.msg) == ("before", "after")
+    assert nul.msg == "GET /assets/� 404" and nul.attrs["path"] == "/assets/�"
+
+
+def test_the_writer_drains_what_was_queued_when_it_is_stopped(db):
+    sink = logsink.LogSink(engine)
+    log = logging.getLogger("siftlode.sinktest.drain")
+    log.addHandler(sink)
+    try:
+        for i in range(logsink.BATCH_MAX + 5):
+            # noqa G004: the f-string is the POINT — a distinct rendered message per line is
+            # what keeps the flood limiter (which groups by template) out of this case.
+            log.warning(f"line {i}")  # noqa: G004
+    finally:
+        log.removeHandler(sink)
+    sink._stopping.set()
+    sink._run()  # stopped before its first wait: everything goes through the drain
+    assert (
+        db.execute(select(func.count()).select_from(LogEvent)).scalar_one() == logsink.BATCH_MAX + 5
+    )
+
+
+def test_a_stopped_writer_says_what_it_dropped_before_it_goes(db):
+    """The download child's exit is `stop()` → `_run`'s drain: the notice for the repeats it
+    dropped is written there, or a child that lived under a minute never reports them."""
+    sink = logsink.LogSink(engine)
+    log = logging.getLogger("siftlode.sinktest.dropped")
+    log.addHandler(sink)
+    try:
+        for i in range(logsink.REPEATS_PER_MINUTE + 4):
+            log.warning("RSS poll skipped for channel %s: HTTP 404", f"UC{i}")
+    finally:
+        log.removeHandler(sink)
+    sink._stopping.set()
+    sink._run()
+    last = db.execute(select(LogEvent).order_by(LogEvent.id.desc()).limit(1)).scalar_one()
+    assert (last.event, last.attrs["suppressed"]) == ("log.repeats_suppressed", 4)
+
+
+def test_the_running_sink_keeps_the_base_level_for_everyone_the_elevation_is_not_for(
+    db, monkeypatch
+):
+    """`logsink.start()` as production runs it. One user's elevation lowers the ROOT logger, so an
+    ordinary request line below the base level reaches every handler — the sink's own level gate is
+    what keeps it out of the table, as stdout's keeps it off the terminal."""
+    root = logging.getLogger()
+    saved = (logs._base_level, dict(logs._elevations), logs._root_applied, root.level)
+    monkeypatch.setattr(logsink.settings, "log_db_sink", True)
+    logs.set_level("WARNING")
+    logs._track_elevation(1, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1))
+    log = logging.getLogger("siftlode.sinktest.live")
+    try:
+        logsink.start()
+        log.info("GET /api/feed 200", extra={**logs.event("http.request"), "user_id": 2})
+        log.warning("a warning")
+    finally:
+        logsink.stop()
+        logs._base_level, logs._root_applied = saved[0], saved[2]
+        logs._elevations.clear()
+        logs._elevations.update(saved[1])
+        root.setLevel(saved[3])
+    assert [r.msg for r in db.execute(select(LogEvent)).scalars()] == ["a warning"]

backend/tests/test_swallowed_exceptions.py

@@ -0,0 +1,71 @@
+"""E22.5 S3 — every swallowed exception has been ruled on.
+
+A handler that neither logs nor re-raises is a decision: someone looked at it and decided the
+failure is not worth a line. 114 of them were in `app/` at the start of this epic, and almost none
+carried that decision anywhere — which means nobody could tell the deliberate ones from the ones
+that were never thought about. That is the whole defect: not that they are silent, but that their
+silence says nothing.
+
+So the rule is not "log everything". It is: a silent handler carries `# silent-ok: <why>` on the
+`except` line or the line above it. Ruff's S110 and BLE001 sit under this and catch the two narrow
+shapes (`except: pass`, blind `except Exception`); this case covers the rest — the handler that
+quietly returns a fallback, or `continue`s past one item of a loop.
+"""
+
+import ast
+import pathlib
+
+from app import logs
+
+APP = pathlib.Path(logs.__file__).resolve().parent
+_MARKER = "silent-ok"
+_LOG_METHODS = frozenset(
+    {"debug", "info", "warning", "warn", "error", "exception", "critical", "log"}
+)
+
+
+def _is_log_call(node: ast.AST) -> bool:
+    if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):
+        return False
+    if node.func.attr not in _LOG_METHODS:
+        return False
+    base, name = node.func.value, ""
+    while isinstance(base, ast.Attribute):
+        name = f"{base.attr}.{name}"
+        base = base.value
+    if isinstance(base, ast.Name):
+        name = f"{base.id}.{name}"
+    return "log" in name.lower()
+
+
+def _swallows(handler: ast.ExceptHandler) -> bool:
+    """Neither reports nor re-raises. A nested `raise` counts: the caller still finds out."""
+    for node in ast.walk(handler):
+        if isinstance(node, ast.Raise) or _is_log_call(node):
+            return False
+    return True
+
+
+def test_every_silent_handler_says_why_it_is_silent():
+    offenders = []
+    for path in sorted(APP.rglob("*.py")):
+        src = path.read_text(encoding="utf-8")
+        lines = src.splitlines()
+        for node in ast.walk(ast.parse(src)):
+            if not isinstance(node, ast.ExceptHandler) or not _swallows(node):
+                continue
+            # The marker may sit in the comment block attached above the `except`, on the
+            # `except` line itself, or anywhere inside the handler — all three are this handler
+            # and nothing else, so no justification can excuse a neighbour.
+            start = node.lineno - 1
+            while start > 0 and lines[start - 1].lstrip().startswith("#"):
+                start -= 1
+            end = node.end_lineno or node.lineno
+            if _MARKER in " ".join(lines[start:end]):
+                continue
+            offenders.append(f"{path.relative_to(APP)}:{node.lineno}")
+    assert offenders == [], (
+        f"{len(offenders)} exception handlers neither log nor re-raise and do not say why. Add a "
+        f"log line with an event code, or `# {_MARKER}: <why the failure is not worth a line>` — "
+        f"{offenders}"
+    )

backend/tests/test_sync_guards.py

@@ -1,6 +1,8 @@
 """Guards that stop one bad YouTube answer from cascading: the subscription prune guard, the
 playlist-push circuit breaker, and RSS telling a failure apart from an empty feed."""
 
+import logging
+
 import httpx
 import pytest
 
@@ -120,7 +122,7 @@ class TestRssPollResult:
         # One dead channel among healthy ones: counted, but the run itself is still a run.
         def fetch(cid):
             if cid == "UC_dead":
-                raise rss.RssError("404")
+                raise rss.RssError("404", "http_404")
             return [{"id": "v"}]
 
         out = self._run(
@@ -143,6 +145,54 @@ class TestRssPollResult:
         )
         assert out == {"new": 1, "failed": 1}, "an unstored channel must not read as ok"
 
+    def test_many_failures_write_one_warning_with_counts_not_one_per_channel(
+        self, monkeypatch, caplog
+    ):
+        # YouTube's RSS outage of 2026-09-16: ~900 of 979 feeds failed on every 20-minute poll and
+        # each wrote its own WARNING — 11k lines in four hours, which the log_events table would
+        # have stored too. One summary per poll, per-channel detail at DEBUG.
+        channels = [f"UC{i}" for i in range(40)]
+
+        def fetch(cid):
+            n = int(cid[2:])
+            if n < 30:
+                raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404", "http_404")
+            if n < 35:
+                raise rss.RssError(f"RSS fetch failed for {cid}: timed out", "transport")
+            return [{"id": "v"}]
+
+        caplog.set_level(logging.DEBUG, logger="siftlode.sync")
+        out = self._run(monkeypatch, fetch=fetch, apply_=lambda db, ch, e: 1, channels=channels)
+        assert out == {"new": 5, "failed": 35}
+        warnings = [r for r in caplog.records if r.levelno >= logging.WARNING]
+        assert len(warnings) == 1
+        assert warnings[0].event == "sync.rss.failed"
+        assert warnings[0].attrs["reasons"] == {"http_404": 30, "transport": 5}
+        assert len(warnings[0].attrs["examples"]) == runner.RSS_FAILURE_EXAMPLES
+        assert sum(r.levelno == logging.DEBUG for r in caplog.records) == 35
+
+    def test_a_failure_on_our_side_gets_one_traceback_per_poll_not_one_per_channel(
+        self, monkeypatch, caplog
+    ):
+        # A read-only database fails every channel's apply identically. The first gets a traceback
+        # at ERROR; repeating it for each of 979 channels would be the RSS flood all over again.
+        def apply_(db, ch, entries):
+            raise RuntimeError("read-only transaction")
+
+        caplog.set_level(logging.DEBUG, logger="siftlode.sync")
+        with pytest.raises(RuntimeError):
+            self._run(
+                monkeypatch,
+                fetch=lambda cid: [{"id": "v"}],
+                apply_=apply_,
+                channels=("A", "B", "C"),
+            )
+        errors = [r for r in caplog.records if r.levelno == logging.ERROR]
+        assert len(errors) == 1 and errors[0].exc_info is not None
+        assert (
+            sum(r.levelno == logging.DEBUG and r.exc_info is not None for r in caplog.records) == 2
+        )
+
     def test_a_quiet_hour_is_distinguishable_from_an_outage(self, monkeypatch):
         quiet = self._run(monkeypatch, fetch=lambda cid: [], apply_=lambda db, ch, e: 0)
         assert quiet == {"new": 0, "failed": 0}  # same "new", different "failed" — that's the point
@@ -152,7 +202,7 @@ class TestRssPollResult:
         # here recorded the scheduler run as "ok", with the outage only visible inside the summary
         # text — a green pill for a run that did nothing at all.
         def boom(cid):
-            raise rss.RssError("Connection refused")
+            raise rss.RssError("Connection refused", "transport")
 
         with pytest.raises(RuntimeError, match="all 2 channels"):
             self._run(monkeypatch, fetch=boom, apply_=lambda db, ch, e: 1)
@@ -162,7 +212,7 @@ class TestRssPollResult:
         # The card has to say which: `HTTP 404` = fix the subscription, `Connection refused` =
         # the egress is down.
         def dead(cid):
-            raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404")
+            raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404", "http_404")
 
         with pytest.raises(RuntimeError, match="HTTP 404"):
             self._run(monkeypatch, fetch=dead, apply_=lambda db, ch, e: 1, channels=("UC_dead",))

backend/tests/test_sysconfig_registry.py

@@ -0,0 +1,93 @@
+"""The sysconfig registry: its order is the admin page's tab order, and a closed-set knob is a CHOICE.
+
+The admin Configuration page used to keep its own hand-written group order, which had already drifted
+(`explore` was missing, so its tab sorted last). The page now follows SPECS, so SPECS has to read in
+tab order: one run of keys per group.
+
+A closed-set admin knob is a CHOICE, not free text (E22 S2 follow-up).
+
+`log_level`, `search_source`, `download_layout` and the transcode `hw`/`preset` used to be free-text
+fields: the admin could type anything and every consumer carried its own `or "auto").lower()` guess
+at what a typo meant. Now the value set is a `Literal` in config.py, pydantic holds the env to it at
+boot, sysconfig holds a stored override to it, and the admin page renders a picker from the same list.
+The DB lane (`db` fixture) covers the store; the rest is pure logic.
+"""
+
+import pytest
+from pydantic import ValidationError
+
+from app import sysconfig
+from app.config import Settings, choices_of
+
+CHOICE_KEYS = {s.key for s in sysconfig.SPECS if s.type == "choice"}
+
+
+def test_each_group_is_one_run_of_keys_so_the_registry_reads_in_tab_order():
+    runs = [
+        s.group
+        for i, s in enumerate(sysconfig.SPECS)
+        if i == 0 or sysconfig.SPECS[i - 1].group != s.group
+    ]
+    assert len(runs) == len(set(runs)), runs
+    assert runs[:2] == ["access", "email"]
+
+
+def test_every_closed_set_knob_is_a_choice_whose_options_come_from_config():
+    assert CHOICE_KEYS == {
+        "search_source",
+        "download_layout",
+        "log_level",
+        "plex_transcode_hw",
+        "plex_transcode_preset",
+    }
+    for s in sysconfig.SPECS:
+        field = Settings.model_fields[s.default_attr]
+        if s.type == "choice":
+            assert s.choices, s.key
+            assert field.default in s.choices, s.key
+        else:
+            # A Literal-typed setting exposed as "str" would be a free-text field again.
+            assert s.choices == () and choices_of(field.annotation) == (), s.key
+
+
+def test_the_env_accepts_any_letter_case_and_rejects_an_unknown_value():
+    assert Settings(log_level="debug").log_level == "DEBUG"
+    assert Settings(search_source=" API ").search_source == "api"
+    with pytest.raises(ValidationError):
+        Settings(log_level="verbose")
+
+
+class _Row:
+    def __init__(self, value):
+        self.value, self.encrypted = value, False
+
+
+class _FakeDb:
+    """Just enough of a Session for sysconfig's reads: one stored override row per key."""
+
+    def __init__(self, **rows):
+        self.rows = {k: _Row(v) for k, v in rows.items()}
+
+    def get(self, _model, key):
+        return self.rows.get(key)
+
+
+def test_a_stored_value_reads_back_canonical_and_a_stale_one_falls_back_to_the_default():
+    assert sysconfig.get_str(_FakeDb(plex_transcode_hw="OFF"), "plex_transcode_hw") == "off"
+    # A value that is no longer (or never was) a choice is not passed through to the consumer.
+    assert sysconfig.get_str(_FakeDb(plex_transcode_hw="maybe"), "plex_transcode_hw") == "auto"
+
+
+def test_describe_hands_the_picker_its_options():
+    items = {i["key"]: i for g in sysconfig.describe(_FakeDb())["groups"].values() for i in g}
+    assert items["log_level"]["choices"] == ["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"]
+    assert items["log_level"]["value"] == "INFO"
+    assert items["quota_daily_budget"]["choices"] == []
+
+
+def test_set_value_stores_the_canonical_spelling_and_refuses_anything_else(db):
+    sysconfig.set_value(db, "search_source", "API")
+    assert sysconfig.get_str(db, "search_source") == "api"
+    with pytest.raises(ValueError, match="scrape, api"):
+        sysconfig.set_value(db, "search_source", "bing")
+    db.rollback()

backend/tests/test_worker_supervisor.py

@@ -87,6 +87,7 @@ def test_no_progress_stall_kills_and_fails_the_job(monkeypatch, rec):
     fail = next(c for c in rec if c[0] == "fail")
     assert fail[2]["error_code"] == "network"
     assert "stalled" in fail[2]["error"].lower()
+    assert fail[2]["cause"] == "stalled"  # the line the killed child never gets to write
 
 
 def test_route_pause_kills_but_keeps_the_route_status(monkeypatch, rec):
@@ -120,6 +121,7 @@ def test_wall_clock_cap_kills_even_while_progressing(monkeypatch, rec):
     worker._run_supervised(1, 99)
     fail = next(c for c in rec if c[0] == "fail")
     assert "time limit" in fail[2]["error"].lower()
+    assert fail[2]["cause"] == "timeout"
 
 
 def test_child_crash_without_finishing_is_failed(monkeypatch, rec):
@@ -131,6 +133,8 @@ def test_child_crash_without_finishing_is_failed(monkeypatch, rec):
     assert "reset_asset" in names
     fail = next(c for c in rec if c[0] == "fail")
     assert "unexpectedly" in fail[2]["error"].lower()
+    # Its line is `_fail_running`'s, so it is logged only if the job really failed here.
+    assert (fail[2]["cause"], fail[2]["exit_code"]) == ("child_exit", 1)
 
 
 def test_child_nonzero_exit_that_already_set_state_is_left_alone(monkeypatch, rec):
@@ -236,3 +240,37 @@ def test_stop_child_noop_if_already_exited(killpg_rec):
     proc.returncode = 0  # already gone
     worker._stop_child(proc, grace_s=0.01)
     assert killpg_rec == []
+
+
+def test_a_crashed_jobs_crash_line_and_cleanup_carry_the_job_id(monkeypatch):
+    """The job id is bound for the WHOLE job, the crash handler included: bound inside the `try`, it
+    was already reset when the one line that says the job crashed was logged."""
+    from app import logs
+
+    seen: dict[str, object] = {}
+    claims = iter([42])
+
+    def claim():
+        job = next(claims, None)
+        if job is None:
+            worker._stop.set()
+        return job
+
+    def crash(_job_id):
+        raise RuntimeError("boom")
+
+    monkeypatch.setattr(worker, "_claim_job", claim)
+    monkeypatch.setattr(worker, "_process_job", crash)
+    monkeypatch.setattr(
+        worker.log, "error", lambda *a, **k: seen.setdefault("crash", logs._job_id.get())
+    )
+    monkeypatch.setattr(
+        worker, "_fail_running", lambda *a, **k: seen.setdefault("cleanup", logs._job_id.get())
+    )
+    monkeypatch.setattr(worker.settings, "download_poll_seconds", 0)
+    worker._stop.clear()
+    try:
+        worker._worker_loop(0)
+    finally:
+        worker._stop.clear()
+    assert seen == {"crash": 42, "cleanup": 42}

backend/tests/test_ytdlp_logging.py

@@ -0,0 +1,233 @@
+"""E22.5 S4 — yt-dlp explains itself to our log, not to a stdout nobody reads.
+
+`quiet: True` was doing more than quieting: with no `logger`, yt-dlp's whole account of a download
+went to the child process's stdout, so "Requested format is not available", a signature-extraction
+failure or a POT refusal reached us only as whatever text happened to end up in the raised
+exception. A failed download's real reason was routinely one line the app never kept.
+
+Measured against yt-dlp 2026.07.04 (the pinned version): with `logger` set, yt-dlp calls `debug`
+for its extraction and progress lines and `warning`/`error` for the rest, and neither `quiet` nor
+`no_warnings` gates those calls. The cases below pin what we do with them.
+"""
+
+import json
+import logging
+
+import pytest
+
+from app import logs
+from app.downloads import formats
+
+URL = "https://www.youtube.com/watch?v=dQw4w9WgXcQ&list=PLsomething"
+
+
+class _Sink(logging.Handler):
+    """The stdout handler's own filter and formatter, so a case sees the bytes a deployment writes."""
+
+    def __init__(self):
+        super().__init__(level=logging.NOTSET)
+        self.lines: list[str] = []
+        self.setFormatter(logs.JsonFormatter())
+        self.addFilter(logs.ContextFilter())
+
+    def emit(self, record):
+        self.lines.append(self.format(record))
+
+    def objects(self) -> list[dict]:
+        return [json.loads(line) for line in self.lines]
+
+
+@pytest.fixture
+def sink():
+    logger = logging.getLogger("siftlode")
+    handler = _Sink()
+    previous = logger.level
+    logger.addHandler(handler)
+    logger.setLevel(logs.TRACE)
+    try:
+        yield handler
+    finally:
+        logger.removeHandler(handler)
+        logger.setLevel(previous)
+
+
+@pytest.mark.parametrize(
+    ("method", "level"),
+    [
+        # Several lines per second during a download, and TRACE is the level a user turns on for
+        # their own session — so the noisy half is exactly the half the consent gates.
+        ("debug", "TRACE"),
+        ("info", "DEBUG"),
+        ("warning", "WARNING"),
+        # yt-dlp calls `error` from `trouble()` just before it raises, on every failed ATTEMPT —
+        # the pipeline's fallback client set then often succeeds. The job's verdict is the
+        # pipeline's `download.job.failed`, not this.
+        ("error", "WARNING"),
+    ],
+)
+def test_each_ytdlp_level_lands_where_it_belongs(sink, method, level):
+    getattr(formats._YDL_LOG, method)("something happened")
+
+    (line,) = sink.objects()
+    assert line["level"] == level
+    assert line["event"] == "download.ytdlp"
+    assert "something happened" in line["msg"]
+
+
+def test_the_source_url_does_not_travel_with_the_explanation(sink):
+    formats._YDL_LOG.error(f"ERROR: [youtube] {URL}: Requested format is not available")
+
+    (line,) = sink.objects()
+    # The host survives — it is what says which extractor is failing. The rest is a row of this
+    # person's download history, and the job id already names the job on every one of these lines.
+    assert "www.youtube.com/…" in line["msg"]
+    assert "dQw4w9WgXcQ" not in json.dumps(line)
+    assert "list=PLsomething" not in json.dumps(line)
+    assert "Requested format is not available" in line["msg"]
+
+
+@pytest.mark.parametrize(
+    ("line", "kept", "gone"),
+    [
+        # The shape nearly every yt-dlp warning and error has: no URL, the id as a prefix.
+        (
+            "ERROR: [youtube] dQw4w9WgXcQ: Video unavailable",
+            "ERROR: [youtube] …: Video unavailable",
+            "dQw4w9WgXcQ",
+        ),
+        # The generic extractor's "id" is the URL's own basename.
+        (
+            "WARNING: [generic] my-private-holiday-clip: Falling back on generic information",
+            "WARNING: [generic] …: Falling back on generic information",
+            "my-private-holiday-clip",
+        ),
+        # ...percent-decoded, so it can have spaces: `https://x/My%20Holiday%20in%20Crete.mp4`.
+        (
+            "ERROR: [generic] My Holiday in Crete: Unable to download webpage: HTTP Error 404",
+            "ERROR: [generic] …: Unable to download webpage: HTTP Error 404",
+            "Holiday",
+        ),
+        (
+            "[generic] My Holiday in Crete: Downloading webpage",
+            "[generic] …: Downloading webpage",
+            "Holiday",
+        ),
+        # A playlist URL is extracted too (`playlist_items: "1"`), and its title is printed.
+        (
+            "[youtube:tab] Playlist Our Wedding Videos: Downloading 1 items of 12",
+            "[youtube:tab] Playlist …: Downloading 1 items of 12",
+            "Wedding",
+        ),
+        (
+            "[download] Finished downloading playlist: Our Wedding Videos",
+            "[download] Finished downloading playlist: …",
+            "Wedding",
+        ),
+        (
+            "[download] Downloading playlist: Our Wedding Videos",
+            "[download] Downloading playlist: …",
+            "Wedding",
+        ),
+        # An id quoted mid-sentence.
+        (
+            "hu subtitles not available for dQw4w9WgXcQ",
+            "hu subtitles not available for …",
+            "dQw4w9WgXcQ",
+        ),
+        # `watch?v=…&list=…` under `noplaylist` (`_yes_playlist`), and the multifeed shape.
+        (
+            "[youtube:tab] Downloading just the video dQw4w9WgXcQ because of --no-playlist",
+            "[youtube:tab] Downloading just the video … because of --no-playlist",
+            "dQw4w9WgXcQ",
+        ),
+        (
+            "[youtube] Downloading just video dQw4w9WgXcQ because of --no-playlist",
+            "[youtube] Downloading just video … because of --no-playlist",
+            "dQw4w9WgXcQ",
+        ),
+        # `Destination` is yt-dlp's word, not an id; the path after it is built from the title.
+        (
+            "[download] Destination: /downloads/Some_Channel/Season_2024/A_Private_Title_a7.mp4",
+            "[download] Destination: …",
+            "A_Private_Title",
+        ),
+    ],
+)
+def test_what_names_the_download_comes_out_of_real_ytdlp_shapes(sink, line, kept, gone):
+    formats._YDL_LOG.warning(line)
+
+    (obj,) = sink.objects()
+    assert obj["msg"] == kept
+    assert gone not in json.dumps(obj)
+
+
+@pytest.mark.parametrize(
+    ("url", "name"),
+    [
+        ("https://example.org/v/Wedding%20%5B2024%5D.mp4", "Wedding [2024]"),
+        ("https://example.org/v/Our%20Trip%3A%20Day%201.mp4", "Our Trip: Day 1"),
+    ],
+)
+def test_the_jobs_own_generic_name_is_taken_out_whatever_it_holds(url, name):
+    """Brackets and `: ` bound no pattern; the job knows its URL, so the exact name goes."""
+    line = f"ERROR: [generic] {name}: Unable to download webpage: HTTP Error 404: Not Found"
+
+    assert formats.generic_id(url) == name
+    with formats.hiding(url):
+        assert formats.log_safe(line) == (
+            "ERROR: [generic] …: Unable to download webpage: HTTP Error 404: Not Found"
+        )
+
+
+def test_the_generic_name_is_yt_dlps_own():
+    from yt_dlp.extractor.common import InfoExtractor
+
+    for url in ("https://x.org/a/My%20Holiday.mp4", "https://x.org/a/clip/", "https://x.org/b"):
+        assert formats.generic_id(url) == InfoExtractor._generic_id(url)
+
+
+def test_a_download_error_is_scrubbed_the_same_way_wherever_it_is_logged():
+    """yt-dlp raises the text it has just logged, and the pipeline logs the exception again one
+    line later (`download.job.failed`). The same scrub, or that line puts the URL back."""
+    text = "ERROR: Unsupported URL: https://example.org/private/path?id=123"
+
+    assert formats.log_safe(text) == "ERROR: Unsupported URL: example.org/…"
+
+
+@pytest.mark.parametrize(
+    "line",
+    [
+        "nsig extraction failed: Some formats may be missing",
+        "[VideoConvertor] Converting video from webm to mp4; Destination: x",
+        "[download]  53.2% of ~  12.00MiB at    1.20MiB/s ETA 00:05",
+        "[download] 100% of   12.00MiB in 00:00:03 at 3.90MiB/s",
+    ],
+)
+def test_a_line_that_names_nothing_keeps_its_words(line):
+    """The scrub takes out what names a download, and nothing an operator reads the line for."""
+    assert formats.log_safe(line) == line
+
+
+def test_a_line_with_no_url_is_left_alone(sink):
+    formats._YDL_LOG.warning("nsig extraction failed: Some formats may be missing")
+
+    (line,) = sink.objects()
+    assert "nsig extraction failed: Some formats may be missing" in line["msg"]
+
+
+def test_the_download_options_carry_the_logger():
+    opts = formats.build_ydl_opts({}, "%(id)s.%(ext)s", lambda d: None)
+
+    assert opts["logger"] is formats._YDL_LOG
+
+
+def test_a_source_adapter_cannot_silence_it():
+    """`extra` is the adapter's own options (YouTube's player-client/POT strategy), merged wholesale.
+    The logger is set after that merge for the same reason `cookiefile` and the size cap are: an
+    adapter must not be able to take away the one account of what went wrong that a failed job
+    leaves behind."""
+    opts = formats.build_ydl_opts(
+        {}, "%(id)s.%(ext)s", lambda d: None, extra={"logger": None, "quiet": False}
+    )
+
+    assert opts["logger"] is formats._YDL_LOG

docker-compose.localdev.yml

@@ -11,6 +11,15 @@
 # applies local migrations to THIS local DB only. Seed the catalog once with a pg_dump from
 # another instance if you want real data (see RUNBOOK).
 
+# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
+# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
+# busy instance and a hard ceiling on the disk it can take.
+x-logging: &logging
+  driver: json-file
+  options:
+    max-size: "50m"
+    max-file: "3"
+
 services:
   db:
     image: postgres:16-alpine
@@ -27,6 +36,7 @@ services:
       interval: 5s
       timeout: 5s
       retries: 12
+    logging: *logging
     restart: unless-stopped
 
   api:
@@ -63,6 +73,7 @@ services:
       - ./downloads:/downloads
       # Plex media, read-only (see the plex_media volume note below). Maps to plex_path_map.
       - plex_media:/plex-media:ro
+    logging: *logging
     restart: unless-stopped
 
   # Dedicated download worker: same image, runs the yt-dlp job loop instead of the API.
@@ -96,6 +107,7 @@ services:
     volumes:
       - ./downloads:/downloads
       - plex_media:/plex-media:ro
+    logging: *logging
     restart: unless-stopped
 
   # PO-token provider: mints YouTube Proof-of-Origin tokens on demand so the worker bypasses
@@ -106,6 +118,7 @@ services:
     init: true
     security_opt:
       - apparmor:unconfined
+    logging: *logging
     restart: unless-stopped
 
 volumes:

docker-compose.selfhost.yml

@@ -10,6 +10,15 @@
 # Everything else — the admin account, Google sign-in, SMTP — is set in the web wizard on
 # first run. See docs/self-hosting.md.
 
+# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
+# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
+# busy instance and a hard ceiling on the disk it can take.
+x-logging: &logging
+  driver: json-file
+  options:
+    max-size: "50m"
+    max-file: "3"
+
 services:
   db:
     image: postgres:16-alpine
@@ -27,6 +36,7 @@ services:
       interval: 10s
       timeout: 5s
       retries: 12
+    logging: *logging
     restart: unless-stopped
 
   api:
@@ -71,6 +81,7 @@ services:
       timeout: 5s
       retries: 5
       start_period: 30s
+    logging: *logging
     restart: unless-stopped
 
   # Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
@@ -109,6 +120,7 @@ services:
       - ALL
     tmpfs:
       - /tmp
+    logging: *logging
     restart: unless-stopped
 
   # PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
@@ -119,6 +131,7 @@ services:
     networks: [internal]
     security_opt:
       - no-new-privileges:true
+    logging: *logging
     restart: unless-stopped
 
 volumes:

docker-compose.yml

@@ -1,3 +1,12 @@
+# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
+# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
+# busy instance and a hard ceiling on the disk it can take.
+x-logging: &logging
+  driver: json-file
+  options:
+    max-size: "50m"
+    max-file: "3"
+
 services:
   db:
     image: postgres:16-alpine
@@ -12,6 +21,7 @@ services:
       interval: 5s
       timeout: 5s
       retries: 12
+    logging: *logging
     restart: unless-stopped
 
   api:
@@ -49,6 +59,7 @@ services:
       timeout: 5s
       retries: 5
       start_period: 25s
+    logging: *logging
     restart: unless-stopped
 
   # Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
@@ -83,6 +94,7 @@ services:
       - ${DOWNLOAD_HOST_PATH:-downloads}:/downloads
       # Same read-only Plex library as the API — the worker resolves library paths too.
       - ${PLEX_MEDIA_HOST_PATH:-plex_media}:/plex-media:ro
+    logging: *logging
     restart: unless-stopped
 
   # PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
@@ -90,6 +102,7 @@ services:
   bgutil-pot:
     image: brainicism/bgutil-ytdlp-pot-provider:1.3.1
     init: true
+    logging: *logging
     restart: unless-stopped
 
 volumes: