peter/siftlode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: Settings → Diagnostics, a consented log level and the tab's own log

Browse Source 
A user can raise their diagnostics level to Részletes (DEBUG) or Teljes (TRACE) for 24 hours from a
new Settings tab, after a consent box that says what is recorded, that the server keeps it and for
how long (the instance's own retention), and that it lapses. Alap is the base and needs no consent.

- users.diagnostics_level/_until (migration 0087), written only by PUT /api/me/diagnostics, which
  computes the expiry itself; the demo account cannot raise it. /api/me reports the level in force.
- The server logs an elevated user's requests at that level and stores every line of them
  (logs.LevelGate on the stdout handler and the sink; the root is lowered only while an elevation
  is active). Teljes adds the redacted query string to the request line.
- POST /api/diagnostics/events stores the browser's log as source=web rows: account from the
  session, body capped while streaming, events/attrs capped, re-redacted, events under the
  consented level dropped, timestamps clamped, 30 batches a minute, 409 once the level lapsed.
- lib/diagnostics.ts applies the level to the tab and streams only what happens after consent, in
  batches, never its own request lines, with a keepalive flush on pagehide; stops on lapse (its
  own timer), on withdrawal and on 409.
- Detail hooks at DEBUG: which control was clicked (test id, icon, app copy in English — never user
  text, input values or a link's query), visibility/bfcache/resize, filter changes
  (lib/filterLog.ts; typed search only at TRACE), player events (lib/mediaLog.ts; media errors are
  warnings at every level). Offline/online at the base level.
- "Saját napló" viewer: live, newest first, level groups with counts, search, row details, copy
  the filtered events as JSON lines, clear.
- Privacy policy and Terms exist in Hungarian and English (a contract test holds both languages to
  the same sections); the policy now describes the server logs and diagnostic logging, self-service
  deletion and password sign-in.

Fixed on the way:
- A sync auth dependency ran logs.set_user on a copied context, so the account never reached a
  handler's own log lines (measured); a per-request mutable scope carries it now.
- The Settings tab rail left a phone no room (clipped labels, buttons over text): below `sm` it is
  a scrolling row above the content.
- PUT /api/me/preferences took a bare dict body; it is a Pydantic model now.
This commit is contained in:
peter
2026-09-17 02:42:12 +02:00
parent a210992bfc
commit 5883ea509d
49 changed files with 2790 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/logs.py
+131 -19
View File
@@ -35,6 +35,7 @@ import logging
import logging.config
import re
import sys
import threading
import time
import uuid
from datetime import datetime, timezone
@@ -86,11 +87,44 @@ _CONTEXT_VARS = {
}
def set_user(user_id: int | None) -> None:
"""Attribute the rest of this request's log lines to `user_id`. Called from the auth dependency
— the one place the session is actually resolved into an account — rather than from the request
middleware, which runs OUTSIDE the session middleware and in a different task context."""
class _RequestScope:
"""What the auth dependency learns about a request, for every line logged after it.
A MUTABLE object behind a contextvar, not plain contextvars, because of how a request actually
runs: FastAPI resolves a sync dependency (`auth.current_user`) in a threadpool on a COPY of the
context, so a `ContextVar.set()` there is gone by the time the handler runs — measured: that is
how the account never reached a handler's own lines. `log_requests` installs one scope per
request; every copied context shares the same object, so what the dependency writes is seen by
the handler and by the request line itself."""
__slots__ = ("user_id", "level")
def __init__(self) -> None:
self.user_id: int | None = None
# The user's consented diagnostics level as a logging number; None at the base level.
self.level: int | None = None
_scope: contextvars.ContextVar[_RequestScope | None] = contextvars.ContextVar(
"log_request_scope", default=None
)
def set_user(user_id: int | None, level: str = "INFO", until: datetime | None = None) -> None:
"""Attribute the rest of this request's log lines to `user_id`, and log them at the user's
consented diagnostics `level` until `until` (E22 S5; "INFO" = no elevation). Called from the
auth dependency — the one place the session is actually resolved into an account — rather than
from the request middleware, which runs OUTSIDE the session middleware."""
_user_id.set(user_id)
scope = _scope.get()
if scope is None: # outside log_requests (a script, a unit test): this context only
scope = _RequestScope()
_scope.set(scope)
elevated = level != "INFO" and until is not None and until > datetime.now(timezone.utc)
scope.user_id = user_id
scope.level = resolve_level(level) if elevated else None
if user_id is not None:
_track_elevation(user_id, scope.level, until)
def set_job(job_id: int | None) -> None:
@@ -193,7 +227,7 @@ def redact(value: Any, *, key: str = "") -> Any:
# attribute and is merged into `attrs` (so `extra={"job_id": 7}` works as well as `event(...)`).
_RECORD_STANDARD = frozenset(
vars(logging.LogRecord("", 0, "", 0, "", None, None)).keys()
| {"message", "asctime", "taskName", "event", "attrs", PERSIST}
| {"message", "asctime", "taskName", "event", "attrs", PERSIST, "elevated"}
)
@@ -207,9 +241,31 @@ class ContextFilter(logging.Filter):
value = var.get()
if value is not None and not hasattr(record, name):
setattr(record, name, value)
scope = _scope.get()
if scope is not None and scope.user_id is not None and not hasattr(record, "user_id"):
record.user_id = scope.user_id
return True
class LevelGate(logging.Filter):
"""The configured level, enforced at the HANDLER, with one exception: the lines of a request
whose user raised their diagnostics level (E22 S5).
The root logger's level cannot do this alone: it is one number for the whole process, and the
point is to log ONE user's requests in detail while everyone else's stay at the base level. So
the root is lowered to the lowest level any active elevation needs (`_track_elevation`), and
this gate drops what only an elevation let through unless the line belongs to that elevated
request. It also marks such a line `elevated`, which is what makes the `log_events` sink keep
every line of that request, not just its warnings."""
def filter(self, record: logging.LogRecord) -> bool:
scope = _scope.get()
if scope is not None and scope.level is not None:
record.elevated = True
return record.levelno >= min(scope.level, _base_level)
return record.levelno >= _base_level
def _record_attrs(record: logging.LogRecord) -> dict[str, Any]:
extra = {k: v for k, v in vars(record).items() if k not in _RECORD_STANDARD}
attrs = dict(getattr(record, "attrs", None) or {})
@@ -341,29 +397,73 @@ def configure_logging(source: str) -> None:
"json": {"()": f"{__name__}.JsonFormatter"},
"text": {"()": f"{__name__}.TextFormatter"},
},
"filters": {"context": {"()": f"{__name__}.ContextFilter"}},
"filters": {
"context": {"()": f"{__name__}.ContextFilter"},
"gate": {"()": f"{__name__}.LevelGate"},
},
"handlers": {
"stdout": {
"class": "logging.StreamHandler",
"stream": sys.stdout,
"formatter": formatter,
"filters": ["context"],
"filters": ["context", "gate"],
}
},
"root": {"level": resolve_level(settings.log_level), "handlers": ["stdout"]},
"loggers": {name: {"level": lvl} for name, lvl in _THIRD_PARTY_LEVELS.items()},
}
)
set_level(settings.log_level)
def set_level(name: str) -> None:
"""Change the running process's level (the `log_level` admin knob). Third-party loggers keep
their pinned levels, so turning on DEBUG shows OUR detail, not SQLAlchemy's."""
logging.getLogger().setLevel(resolve_level(name))
global _base_level
_base_level = resolve_level(name)
_apply_root_level(force=True)
def current_level() -> str:
return logging.getLevelName(logging.getLogger().getEffectiveLevel())
"""The configured level — not the root logger's, which an active elevation may have lowered."""
return logging.getLevelName(_base_level or logging.getLogger().getEffectiveLevel())
# --- per-user elevation (E22 S5) -----------------------------------------------------------
# The base level (`LOG_LEVEL`, then the admin knob). NOTSET until configure_logging runs — a unit
# test, a script — so the gate lets everything through there, as it did before the gate existed.
_base_level = logging.NOTSET
# user id → (level number, expiry) for every elevation this process has seen and not yet expired.
_elevations: dict[int, tuple[int, datetime]] = {}
_elevations_lock = threading.Lock()
_root_applied: int | None = None
def _track_elevation(user_id: int, level: int | None, until: datetime | None) -> None:
with _elevations_lock:
if level is None or until is None:
_elevations.pop(user_id, None)
else:
_elevations[user_id] = (level, until)
active = bool(_elevations)
if active or _root_applied != _base_level:
_apply_root_level()
def _apply_root_level(force: bool = False) -> None:
"""Root level = the lowest of the base and every unexpired elevation (expired ones are pruned
here). `setLevel` clears every logger's level cache, so it runs only when the number changes."""
global _root_applied
if _base_level == logging.NOTSET:
return # not configured: the root keeps whatever the process set up
now = datetime.now(timezone.utc)
with _elevations_lock:
for user_id in [u for u, (_, until) in _elevations.items() if until <= now]:
del _elevations[user_id]
lowest = min([_base_level, *(lvl for lvl, _ in _elevations.values())])
if force or lowest != _root_applied:
logging.getLogger().setLevel(lowest)
_root_applied = lowest
def current_source() -> str:
@@ -397,6 +497,8 @@ async def log_requests(request, call_next):
incoming = request.headers.get("x-request-id", "")
rid = incoming if _REQUEST_ID_RE.match(incoming) else uuid.uuid4().hex[:16]
token = _request_id.set(rid)
scope = _RequestScope()
scope_token = _scope.set(scope)
request.state.request_id = rid
started = time.perf_counter()
try:
@@ -417,6 +519,7 @@ async def log_requests(request, call_next):
**_request_owner(request, logging.ERROR),
},
)
_scope.reset(scope_token)
_request_id.reset(token)
raise
duration_ms = round((time.perf_counter() - started) * 1000, 1)
@@ -428,22 +531,23 @@ async def log_requests(request, call_next):
level = logging.WARNING
else:
level = logging.DEBUG if quiet else logging.INFO
attrs: dict[str, Any] = {
"method": request.method,
"path": _route_template(request),
"status": status,
"duration_ms": duration_ms,
}
# Teljes (TRACE, the user's or the whole process's): the query string too — redacted, and in
# attrs only, never in the message.
if request.url.query and _traced(scope):
attrs["query"] = redact_text(request.url.query)
_request_log.log(
level,
"%s %s %s",
request.method,
_route_template(request),
status,
extra={
**event(
"http.request",
method=request.method,
path=_route_template(request),
status=status,
duration_ms=duration_ms,
),
**_request_owner(request, level),
},
extra={**event("http.request", **attrs), **_request_owner(request, level)},
)
response.headers["X-Request-Id"] = rid
# The template, so the browser's copy of this line groups the same way (frontend lib/http.ts).
@@ -453,10 +557,18 @@ async def log_requests(request, call_next):
matched = _matched_route(request)
if matched:
response.headers["X-Route"] = matched
_scope.reset(scope_token)
_request_id.reset(token)
return response
def _traced(scope: _RequestScope) -> bool:
"""Whether this request is logged at TRACE — its user's consent, or the process's base level."""
if scope.level is not None and scope.level <= TRACE:
return True
return _base_level != logging.NOTSET and _base_level <= TRACE
# Anonymous request lines kept per client per minute. Signed-in traffic is not capped: it is a known
# account and exactly what an admin needs to see. Anonymous traffic is where the volume can come from
# nobody in particular — a scanner walking `/wp-login.php`, a bot on a share link — and each of its