96509fe356b6a593dcbf5ee3ddb31aa483ededef
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 peter
|
4aaaaa7860 |
chore(comments): drop epic, sprint and review-round ids from the code
The code cited its plan instead of saying why: `E28 review J4`, `(R6 S3b)`, `S1 justified…`,
`deep-review round 5`. That plan lives in the private ops repo, so for every reader of this one the
citation was noise, and a sentence whose subject was the plan step ("round 2 showed…") hid its
lesson behind an id. Each one is gone and the "why" kept: a bare citation is deleted, a sentence
carrying a lesson is reworded around it ("a later attempt showed…").
Measured (`scripts/audit_metrics.py`, product code): 652 lines in 270 files carried an epic code by
the definition the next commit holds (507 / 241 by the old, two-digit-only one) -> 0. Across all
code, tests, scripts and configs: ~1100 epic-code lines plus ~320 sprint/round ids.
Behaviour unchanged, checked mechanically: every changed .py file has the same AST once docstrings
are blanked; css/sh/toml/yml/json files are identical once comments are stripped; the only
non-comment changes in TS/JS are test titles, one description string in a contract table, and five
ESLint messages, whose text is all the gates snapshot diff holds. Not touched: Markdown (docs and
the changelog keep their history), Alembic migrations, TV episode notation (`S2 · E5`), DESIGN.md
ids (`C1`, `B4`, `S3h`), tracker ids (`SEC-16`, `DX-17`).
One stale claim fixed on the way: test_hls_reaper's docstring said the player checkpoints "every
10 s whether or not it is playing", which plex/stream.py calls wrong; it now says what the code does.
|
||
|
peter
|
a7d7456279 |
feat(diagnostics): debug sessions — start, stop, the tag on every line, screenshots (E16 S1)
A session is one recording, capped at 60 minutes by the server. While it is open, a request naming the caller's own session in X-Debug-Session is logged at the session's level with every line tagged; a header naming anything else is ignored. The browser's batches count a session as consent to its level and keep the tag only on its own events. Screenshots are judged by their bytes, capped in size and count, and served back to their owner only. - debug_sessions / debug_screenshots tables, one open session per user (partial unique index), log_events.debug_session_id indexed - /api/debug-sessions: start (joins a running one), stop, list, get, events, screenshots; /api/me reports the live session - the hourly log job closes expired sessions and drops them with the logs - logsink.serialize and routes._common.capped_body shared by their readers |
||
|
peter
|
54d19cc386 |
fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
`formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
file paths, and the job-failed, refused, edit-failed and backfill lines use
it. Done lines log the asset id, not the title-built path. The PII contract
now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
a process's last batch closes the repeat window, so a short-lived download
child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
`download.job.failed` with a cause; they left no line at all before. The
crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
carry reason `transport`. Subscribe recognises a duplicate by reason code
only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
|
||
|
peter
|
6b0401b922 |
fix(logs): rule on every swallowed exception in the backend
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.
What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
degraded to a software transcode with nothing saying why; a failed audio probe
collapsed a multi-audio file to one track and the player showed no language
picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
own root, leaving every crashed session's segments on disk forever.
Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.
ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.
logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.
tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
|
||
|
peter
|
0cff964ff2 |
refactor(logs): name every logger after its module, and hold it there
25 literal names covered 34 modules. plex/client.py, plex/stream.py, plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves `siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`; routes/channels.py called itself `siftlode.api`. The `logger` column is in every stored row so E22.6's console can filter on it, and so an operator can pin a level to one part of the system — neither works when six modules share a name. `logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync. One name per module, and it cannot drift from the file it sits in. tests/test_logger_names.py pins the derivation, that everything stays under the one `siftlode` tree the handler and the tests attach to, and that no module hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`). Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic it carries rather than the module it lives in — it replaced uvicorn's access log and is what an operator pins when they want request lines and nothing else. app/scheduler.py called its logger `logger`; every other module calls it `log`. ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them: worker.py logged `str(exc)[:120]` where the traceback was the whole question, and three prints/f-strings that are deliberate now say so on the line. S110 and BLE001 stay off until S3 — 23 handlers are outstanding and each is a judgement about one of them; enabling the rules first would just mean 23 blanket noqas. |
||
|
peter
|
b2eaa9e00b |
fix: a credential, an erased account and a stale level stay out of the log
Round 2 of the E22 review, on what round 1 changed. `redact_query` decodes a parameter before the text rules see it, which is what lets `%40` be recognised as an address — but `_SECRET_PARAM_RE` was written for the raw, encoded form, where a value holds no space, `&` or quote. On decoded text its value class stops at the first one, so `?state=a%20b` kept its tail and `?code=%20abc` was not matched at all. The parameter-NAME rule now decides, on the key, before any text rule runs. `forget_user` dropped the account's level elevation without recomputing the root level, so a process whose only elevated account had just deleted itself kept building TRACE records to throw away, until some later request happened to recompute it. Every other path that drops an elevation recomputes. The sink's writer took the erasure lock and THEN waited for a pooled connection, while every request path already holds a connection when it reaches that lock — so under pool pressure the writer held a process-global lock against the very requests that could hand a connection back. Same order on both sides now; the lock still spans the check, the insert and the commit. The storage-budget notice said the rest of the hour was not kept, and then smaller batches kept landing underneath it: the window slides and a refused batch is not charged to it. Says what it does. Tests: the route's half of the erasure rule had no test — only the sink's did — and a source-level contract test now covers the third insert nobody has written yet. `logs._erased` is process-global with an hour's memory while the DB fixture hands serial ids back out from 1, so it is swept before each test like the rate limiters are. |
||
|
peter
|
fac5c4465a |
fix: stored logs keep bad rows, erased accounts and unconsented queries out
Deep-review round 1 of E22. - A NUL character (`GET /assets/%00` logs the decoded path) made Postgres refuse the row and with it the sink's whole batch, every user's rows included. `to_row` replaces NUL in every text column and in attrs. - An erased account's lines kept arriving after purge_user's DELETE: the erasing request's own access line, a queued row, a background task. `logs.forget_user` runs first; later rows are stored without the account, and the check and the insert share a lock with the erasure. - An admin setting the whole process to TRACE logged every user's query string. The query is now logged only under the user's own Teljes consent, and decoded before redaction, so `%40` no longer hides an address. - The browser log ingest has an hourly storage budget per account (5 MiB, one notice when spent) and reads only a JSON Content-Type, so a same-site text/plain form post cannot write into a user's log. RateLimiter takes a per-attempt cost for the byte budget. - The root level is applied under the elevation lock: two concurrent changes could leave it at the base while an elevation was active. - Tests for the running sink (level gate, drain on stop) and for the admin log level applied at runtime and at boot. |
||
|
peter
|
5883ea509d |
feat: Settings → Diagnostics, a consented log level and the tab's own log
A user can raise their diagnostics level to Részletes (DEBUG) or Teljes (TRACE) for 24 hours from a new Settings tab, after a consent box that says what is recorded, that the server keeps it and for how long (the instance's own retention), and that it lapses. Alap is the base and needs no consent. - users.diagnostics_level/_until (migration 0087), written only by PUT /api/me/diagnostics, which computes the expiry itself; the demo account cannot raise it. /api/me reports the level in force. - The server logs an elevated user's requests at that level and stores every line of them (logs.LevelGate on the stdout handler and the sink; the root is lowered only while an elevation is active). Teljes adds the redacted query string to the request line. - POST /api/diagnostics/events stores the browser's log as source=web rows: account from the session, body capped while streaming, events/attrs capped, re-redacted, events under the consented level dropped, timestamps clamped, 30 batches a minute, 409 once the level lapsed. - lib/diagnostics.ts applies the level to the tab and streams only what happens after consent, in batches, never its own request lines, with a keepalive flush on pagehide; stops on lapse (its own timer), on withdrawal and on 409. - Detail hooks at DEBUG: which control was clicked (test id, icon, app copy in English — never user text, input values or a link's query), visibility/bfcache/resize, filter changes (lib/filterLog.ts; typed search only at TRACE), player events (lib/mediaLog.ts; media errors are warnings at every level). Offline/online at the base level. - "Saját napló" viewer: live, newest first, level groups with counts, search, row details, copy the filtered events as JSON lines, clear. - Privacy policy and Terms exist in Hungarian and English (a contract test holds both languages to the same sections); the policy now describes the server logs and diagnostic logging, self-service deletion and password sign-in. Fixed on the way: - A sync auth dependency ran logs.set_user on a copied context, so the account never reached a handler's own log lines (measured); a per-request mutable scope carries it now. - The Settings tab rail left a phone no room (clipped labels, buttons over text): below `sm` it is a scrolling row above the content. - PUT /api/me/preferences took a bare dict body; it is a Pydantic model now. |
||
|
peter
|
b59fd38a9c |
feat: store warnings and request lines in log_events
The log now has a stored, queryable copy for the diagnostics console: every WARNING and above plus the request line of every request, kept for log_retention_days (default 7, no "forever") and pruned hourly by the new log_gc job. Rows are built on the logging thread from the same redacted fields as the stdout line, queued without blocking and written in batches by one background thread on its own connection. A full queue or a failed write drops and reports instead of retrying. Flood control keeps 30 of any one message per minute plus a count of the rest; anonymous request lines are capped at 60 a minute per client, and the client address is never stored. Deleting an account deletes its stored events. Measured on prod first: a YouTube RSS outage wrote 11k warnings in four hours, one per channel per poll. The poll now writes one warning with per-reason counts and a few examples, and a failure on our side gets one traceback per poll instead of one per channel. |