fix: stored logs keep bad rows, erased accounts and unconsented queries out

Deep-review round 1 of E22.

- A NUL character (`GET /assets/%00` logs the decoded path) made Postgres
  refuse the row and with it the sink's whole batch, every user's rows
  included. `to_row` replaces NUL in every text column and in attrs.
- An erased account's lines kept arriving after purge_user's DELETE: the
  erasing request's own access line, a queued row, a background task.
  `logs.forget_user` runs first; later rows are stored without the account,
  and the check and the insert share a lock with the erasure.
- An admin setting the whole process to TRACE logged every user's query
  string. The query is now logged only under the user's own Teljes consent,
  and decoded before redaction, so `%40` no longer hides an address.
- The browser log ingest has an hourly storage budget per account (5 MiB,
  one notice when spent) and reads only a JSON Content-Type, so a
  same-site text/plain form post cannot write into a user's log.
  RateLimiter takes a per-attempt cost for the byte budget.
- The root level is applied under the elevation lock: two concurrent
  changes could leave it at the base while an elevation was active.
- Tests for the running sink (level gate, drain on stop) and for the
  admin log level applied at runtime and at boot.
This commit is contained in:
2026-09-17 03:53:29 +02:00
parent 9c9b1cf6d5
commit fac5c4465a
9 changed files with 349 additions and 39 deletions
+15 -11
View File
@@ -17,10 +17,14 @@ from app.config import settings
class RateLimiter:
def __init__(self, max_events: int, window_seconds: float):
"""At most `max_events` attempts per `window_seconds` per key — or, when each attempt passes a
`cost`, at most that much total cost (a byte budget counts bytes, not calls)."""
def __init__(self, max_events: float, window_seconds: float):
self.max_events = max_events
self.window = window_seconds
self._hits: dict[str, list[float]] = {}
# key → (monotonic time, cost) of every counted attempt still inside the window
self._hits: dict[str, list[tuple[float, float]]] = {}
self._lock = threading.Lock()
def reset(self) -> None:
@@ -36,27 +40,27 @@ class RateLimiter:
count."""
cutoff = time.monotonic() - self.window
with self._lock:
return sum(1 for t in self._hits.get(key, []) if t > cutoff) >= self.max_events
return sum(c for t, c in self._hits.get(key, []) if t > cutoff) >= self.max_events
def allow(self, key: str) -> bool:
"""Record an attempt for ``key`` and return whether it is within the limit.
def allow(self, key: str, cost: float = 1) -> bool:
"""Record an attempt costing ``cost`` for ``key`` and return whether it is within the limit.
True -> under the cap (the attempt is counted).
False -> the cap for the current window is already reached (attempt NOT counted, so a
True -> within the cap (the attempt is counted).
False -> the attempt would pass the cap for the current window (attempt NOT counted, so a
blocked caller can't keep pushing the window forward)."""
now = time.monotonic()
cutoff = now - self.window
with self._lock:
hits = [t for t in self._hits.get(key, []) if t > cutoff]
if len(hits) >= self.max_events:
hits = [(t, c) for t, c in self._hits.get(key, []) if t > cutoff]
if sum(c for _, c in hits) + cost > self.max_events:
self._hits[key] = hits
return False
hits.append(now)
hits.append((now, cost))
self._hits[key] = hits
# Opportunistic cleanup so abandoned keys don't accumulate unboundedly.
if len(self._hits) > 4096:
for k in list(self._hits):
fresh = [t for t in self._hits[k] if t > cutoff]
fresh = [(t, c) for t, c in self._hits[k] if t > cutoff]
if fresh:
self._hits[k] = fresh
else: