fix(logs): keep what a person typed out of the backend log lines

The E22 redactor is a safety net over addresses and credential params, and every
one of these call sites was leaning on it to do the whole job. What it does not
cover went in whole, and what it does cover came out as a constant.

Not covered, so a real leak:
- routes/search.py logged the raw search term at ERROR. The privacy policy shipped
  in 0.88.0 says a search string is kept only at Teljes/TRACE, with the user's own
  consent. The ERROR line now carries the shape of the failure (source, candidate
  and collected counts) and the term follows on a TRACE line, which nothing emits
  unless that consent is in force.
- downloads/backfill.py logged the full source URL at INFO — a row of the user's
  download history. Now the asset id and the host.
- youtube/client.py logged the upstream error body, which is prose that quotes the
  request back. Every failing YouTube call now goes through one _upstream_failure()
  that logs the API's `reason` code, keeps the body for the same consented TRACE
  line, and raises a YouTubeError carrying `reason` as a field — callers put that
  error in log lines of their own (routes/channels.py) and in a 422 answered to the
  user, so the body must not travel in it either.

Covered, so merely useless: routes/auth.py named the account by an address that
reached both sinks as `[email]`, and app/email.py printed recipient lists that read
as `['[email]']`. Both now carry the uid (or a count) and a reason code, which is
also the first half of E22.5 S4's failed-sign-in coverage.

routes/channels.py branched on the text of that error message to recognise an
already-existing subscription. It now branches on `exc.reason`, with the substring
check kept beside it for a body that carried no reason at all.

tests/test_log_pii.py pins the call sites and the family: a source-level contract
test walks every log call in app/ and fails on an argument naming user-typed
content unless the line says why it is safe (`# log-pii-ok: <why>`). Eight sites
carry that marker today, each with its reason. Gate 8/8 green; all five cases
mutation-checked.
This commit is contained in:
2026-09-18 00:49:43 +02:00
parent 8c10e57208
commit dc221338be
8 changed files with 364 additions and 32 deletions
+23 -2
View File
@@ -23,7 +23,7 @@ from sqlalchemy import and_, delete, func, or_, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session, aliased
from app import quota, sysconfig
from app import logs, quota, sysconfig
from app.auth import require_human
from app.db import get_db
from app.models import (
@@ -291,7 +291,28 @@ def search_youtube(
collected.extend(_ingest_candidates(db, yt, user, candidates))
except Exception as exc: # noqa: BLE001 — any ingest failure, not just YouTubeError
db.rollback()
log.exception("live search ingest failed for %r", term)
# The term is NOT in this line. A search string is exactly what the privacy
# policy keeps out of the log unless the user turned Teljes/TRACE on for
# themselves, and `redact_text` does not cover it — it scrubs addresses and
# credential params, not what someone typed into the search box. What the
# ERROR line carries instead is the shape of the failure, which is what tells
# an ingest bug from an upstream one; the term follows at TRACE, where the
# consent that allows it is the only thing that emits the record at all.
log.exception(
"live search ingest failed",
extra=logs.event(
"search.live.ingest_failed",
source=source,
candidates=len(candidates),
collected=len(collected),
),
)
log.log(
logs.TRACE,
"the term the ingest failed on",
# log-pii-ok: TRACE is the consented level, and this is what it is for
extra=logs.event("search.live.term", term=term),
)
if collected:
break # keep what earlier pages already gathered
raise HTTPException(