Compare commits

...
133 Commits
Author SHA1 Message Date
peter e804ff3b8e release: 0.95.0 (improvement/p1-upload-auth-first)
gate / frontend (push) Successful in 4m35s
gate / backend (push) Successful in 6m18s
2026-09-23 01:32:45 +02:00
peter cd6f119883 chore(release): 0.95.0
The P1 epic: the upload route reads its own body after `require_human`
resolves, so an anonymous caller can no longer make the server spool 500 MiB
before the auth check runs. Three deep-review fix rounds on top of it.
2026-09-23 01:32:32 +02:00
peter 321776480e fix(uploads): make the drift guards fail for the reasons they claim
Deep-review round 3, targeted at the surfaces round 2's fix commit changed.
Five of the eight findings are the same shape: the guards written to stop
half-done sweeps were themselves half-done.

test_upload.py's display-field guard walked only `ast.Assign`, so the one
writer with an unbounded REQUEST HEADER behind it — `spool_upload`'s
`display_name=display_text(...)` keyword — was invisible to the guard meant
to protect it. It also stopped at `app/routes/`, missing the four writers in
`downloads/` that actually construct the row, and asserted `not offenders`,
which enforces nothing once a refactor moves the writers out of its view.
It now walks assignments AND keywords across `app/`, and holds exact set
equality against a registry of all twelve write sites with each one's value.

test_body_limit.py's release guard used `any(... ast.walk(fn))`, which
accepts a `release_read_connection` that runs AFTER the await it exists to
precede — satisfying the letter of the rule while pinning a connection for
exactly the transfer it was supposed to free. Position is checked now, and a
late release is its own, louder failure. The registry keyed sites by file
basename while scanning recursively, and `admin.py`, `discover.py` and
`playlists.py` each exist both at the top level and under `plex/` — the same
entry-collapse the registry was widened to close. Keyed by relative path now.
And all three guards recognised a body read only by the `capped_` naming
convention, so a route spelling `await request.body()` inline declared no
ceiling, released no connection, and was invisible to every one of them.

`release_read_connection`'s precondition claimed the caller "must have
NOTHING PENDING, and that is checked rather than described" while checking
only UNFLUSHED ORM work. Measured: after `add`+`flush` all three collections
are empty and the rollback loses the row, and Core DML never enters them at
all — both of them this codebase's canonical write shapes. It now also asks
the transaction directly via `pg_current_xact_id_if_assigned()`.

`errors.tooLarge` was reworded to "try a smaller file", but that key answers
every route's body cap: the issue composer's over-size JSON draft got advice
about a file it does not have. Back to payload-neutral; the file-specific
sentence keeps its own key.

The upload dialog's new danger border outranked `dragOver`, so a user
replacing an over-cap file got no sign the drop zone was live — and with a
file picked the label shows its name rather than "drop it here", so the
border was the only feedback left.

Four further findings were refuted 3/3 and are not fixed: `aria-invalid` on
the drop zone (measured in Chromium — the AX node does carry it on a native
button), a bidi-override spoof through `display_text` (the served filename
already strips those categories, and the proposed filter would have broken
every ZWJ emoji), and `capped_body` not draining (the screenshot route's edge
ceiling equals its own cap, so that refusal is unreachable, and the log sink
is sent by beacon with nobody reading the answer).
2026-09-23 01:20:22 +02:00
peter 7c0647f87a fix(uploads): finish the sweeps round 1 left half-done
Deep-review round 2, on round 1's own fix commit. It found MORE than round 1
did (28 raw, against 13), and most of it was mine -- a new piece of UI attracts
findings a backend-only diff never does. The pattern in almost all of them is
the same: a principle applied to the site that prompted it and not to its
siblings.

* `display_text` reached three of the FIVE writers of a user-supplied name.
  `POST /api/downloads` and `POST /api/downloads/edit` still handed Postgres an
  unbounded string, so a 300-character name pasted into the Add-download dialog
  or the editor was a bare 500 (main.py re-raises every DataError but the NUL
  case). Four of the seven reviewers found it independently. The bound now lives
  in `models.py`, the three columns are DECLARED with it, and the guard for the
  next writer is parsed rather than remembered: every `display_name` /
  `display_uploader` assignment under `app/routes/` must go through the helper.
  That guard immediately found two more -- the Google sign-in callback writes
  the profile name into the same 255-wide column -- so it is six, not five.

* The connection release reached one of the THREE routes that read their own
  body. The screenshot route (10 MiB, from a phone) and the client log sink both
  still held a pooled connection idle-in-transaction for the length of the
  upload, which is the invariant db.py sizes the pool on. All three now call
  `db.release_read_connection`, whose precondition -- nothing pending on the
  shared session -- is CHECKED rather than described, because a whole-session
  rollback discarding a caller's staged write is a mistake this repo has already
  made once. `test_body_limit.py` holds the rule for the next such route.

* The AST drift registry keyed its sites by (module, limit expression), so a
  second route copied from the first collapsed onto the same entry and went
  uncovered -- the exact hole the registry exists to close. Keyed per call site
  now, and it carries the release requirement too.

* The dialog's own refusal was announced once and never again: two over-cap
  picks leave an identical node with identical text, and an untouched live
  region says nothing. It is keyed per pick now, the drop zone reads as invalid
  and points at the sentence, and the reason is reachable from the control
  rather than only from a one-shot announcement.

* Two tests rebuilt the blind spot they were written to remove: the dialog's cap
  test sat nowhere near the boundary (a `>` loosened to `>=` survived it, which
  is the mutation round 1 fixed on the server side), and the header-precedence
  test only covered an exact-case duplicate while a differently-cased one would
  have been COMBINED onto the wire rather than replaced. Both fixed, and the
  merge now dedupes case-insensitively because header names are.

Also: the upload dialog's unstubbed usage query was firing a real request from
jsdom in a neighbouring suite and leaking a delayed "connection lost" toast
across test boundaries; `GET /api/downloads/usage` had no test at all though the
whole client-side check hangs off it; and the comment claiming the edge 413 is
readable now says what was measured -- readable to 200 MiB over loopback, a bare
`onerror` through a proxy -- so the pre-send check is named as the mechanism and
that branch as the backstop it is.

Every test added or changed here was mutation-checked. One mutation SURVIVED and
is recorded rather than papered over: the `tooBig` term inside `submit()` cannot
be reached, because `ui/Button` swallows the activation itself, so it is marked
as belt and braces instead of being described as the guard.
2026-09-22 23:31:55 +02:00
peter efd7b217ca fix(uploads): make both caps answerable, and stop holding a connection
The epic-end deep review found three defects in P1 and four tests that did not
hold what they claimed. They land together because they share the upload route
and its test module; splitting by hunk would have produced commits that do not
individually run green.

* The edge ceiling and the route's own cap are the SAME number at the default
  configuration, so the edge always answers first: a declared Content-Length
  over the cap is refused before a byte is read, and an undeclared body trips
  the middleware on the same chunk the route would have tripped on. The route's
  numbered 422 -- and the drain written for it last week -- are reachable only
  once an admin LOWERS the knob, which is the one shape the browser UAT tested.
  Measured: headroom does not fix this, because the declared-length check fires
  first for any body big enough to matter and the drain pushes the count back
  over the edge for the rest. So the edge's own refusal is made readable
  instead: `errors.tooLarge` localizes `too_large` the way `upload_too_large`
  was already localized, and the dialog refuses an over-cap file BEFORE sending
  it, off the `upload_max_bytes` that GET /api/downloads/usage now reports.
  Measured in Chrome against the API directly: that 413 arrives as `onload`
  with a readable body in 43 ms.

* Reading the body behind the auth dependency put the whole transfer inside the
  request's database session -- the auth SELECT autobegins a transaction, so one
  of the twenty pooled connections sat `idle in transaction`, with nothing to
  time it out, for as long as the client cared to keep sending. That is the
  invariant db.py sizes the pool on and plex/playback.py records an incident
  for. The read transaction now ends before the await. Measured on the dev
  server: an 1818 ms upload shows no idle-in-transaction sample while the bytes
  move, only afterwards, when the handler actually writes.

* The display name moved into a request header, where nothing bounds its length,
  while the column stayed varchar(255) -- so a long enough name was a DataError
  500 after the whole file had already transferred. One `display_text` helper
  now serves all three writers (the metadata PATCH already truncated, the upload
  ingest did not), and the dialog stops at the same number.

* The gate invariant asked `route.dependant.body_params`, which holds only the
  route's OWN callable's parameters: a body declared in a SUB-DEPENDENCY -- the
  upload route's own idiom -- leaves it empty while FastAPI still reads the body
  before auth. It asks `route.body_field` now, which is what FastAPI itself
  consults. The AST registry of body-reading call sites had also never followed
  `capped_stream_to`, so the one route the whole invariant exists for was the
  single site the drift guard could not see.

Four tests that passed over their own mutation are fixed: the cap boundary (a
`>` loosened to `>=` survived the entire suite), the percent-decoding (every
filename the suite sends is ASCII, where quote() is the identity function), the
display-name header (no test at all), and the staging cleanup (deleting the
unlink left the suite green while every refused upload leaked its partial bytes
onto the volume). Every test added or changed here was mutation-checked.
2026-09-22 22:41:22 +02:00
peter 3d7ef9e945 fix(uploads): drain an over-cap body before refusing it
Found in the browser UAT, not by the gate. Moving the cap onto the arriving
bytes meant it fires while the client is still sending, and answering + closing
mid-upload is what XHR reports as a network failure: the upload dialog said
"the server couldn't be reached" while the access log showed a correct 422
carrying the limit. A refusal the client cannot read is an outage as far as the
user is concerned.

`capped_stream_to` now reads the remainder to nowhere before raising, so the
client finishes sending and reads the real answer -- re-measured in the
browser, which now names the limit. Nothing past the cap is written, and the
request is still bounded by bodylimit's ceiling, so the cost is bandwidth an
AUTHENTICATED caller was sending anyway. The anonymous path is untouched:
auth still resolves before a byte is read.

The regression test sits on the helper, not the route, and the reason is worth
keeping: the route-level version was written first and PASSED with the drain
deleted. httpx's ASGI transport pumps the request body whether or not the app
reads it, so "drained" and "abandoned" are indistinguishable through
TestClient. Its own mutation check is what exposed it.
2026-09-22 02:11:21 +02:00
peter 7fce7a7ce5 chore(deps): drop python-multipart -- nothing parses a multipart body now
Its own entry in requirements.txt said the download center's local-file upload
was "the only multipart endpoint in the app". Since that route reads a raw
body, the claim is false and the package is dead weight: no route declares
File/Form, and nothing calls `request.form()`.

Removed from requirements.txt and from its single pinned line in
requirements.lock. The lock is generated and normally regenerated wholesale,
but a full regeneration would bump every transitive dependency, which has no
business riding inside a security fix -- the next legitimate regeneration
reproduces exactly this state, since the package is gone from the source.

The `multipart` entry in logs.py's third-party level map goes with it: a level
for a logger that can no longer exist.
2026-09-22 01:59:09 +02:00
peter f5018bdcbc fix(uploads): read the body after authenticating, not before it
`POST /api/downloads/upload` declared `file: UploadFile = File(...)`, and
FastAPI parses and spools a declared body before the first dependency runs --
`require_human` included. An unauthenticated caller could therefore make the
process write the whole per-file allowance (500 MiB by default) to temp, once
per request. SEC-20's ceiling bounded that number but could not stop the
writing, which is why E27 parked it rather than closing it.

The route now takes no body parameter. `spool_upload` names `require_human`
first and only then streams `request.stream()` to the staging temp file, so a
stranger is refused before a byte moves -- the shape the screenshot route has
used since E16. Filename and display name travel as percent-encoded headers
(latin-1 on the wire, so a Hungarian filename needs the encoding) instead of
multipart parts.

`capped_stream_to` joins `capped_body` in routes/_common.py: the same counting
rule, written to a file descriptor rather than accumulated, because holding
500 MiB would trade a disk problem for a memory one. Its refusal is the
caller's, so the upload route keeps answering 422 `upload_too_large` with the
admin knob's own number.

With no envelope to allow for, the edge is now exactly that knob, and
MULTIPART_ENVELOPE_ALLOWANCE is gone.

The invariant stops being prose: test_body_limit.py asks FastAPI itself
(`route.dependant.body_params`) and fails the gate if any route with a raised
ceiling declares a body. A comment could not have prevented the defect it
described; that test would have.

Both new tests were seen red for the right reason -- before the fix the
anonymous probe got its 401 only after the server had read 4236 bytes of a
multipart body, and re-declaring the File parameter turns the invariant test
red naming the route and the parameter.
2026-09-22 01:59:03 +02:00
peter b82c22b888 release: 0.94.0 (improvement/e27-contract-hardening)
gate / frontend (push) Successful in 4m32s
gate / backend (push) Successful in 6m8s
2026-09-22 00:34:12 +02:00
peter f839a461e6 chore(release): 0.94.0 — framework & contract hardening 2026-09-22 00:34:06 +02:00
peter 8525666913 test: round 4 — an assertion that could not fail, and a comment that did not add up
Two findings, both in the review's own work; nothing in the product code. That is
the shift that ends the loop: 22 → 13 → 6 → 2, and the last two are a test and a
comment rather than a defect.

The "an override that is fine is still honoured" leg of the bounds-fallback test
stored `spec.max` — which, for this key, IS the env default, because the spec's
max is derived from it. So the assertion was satisfied by the fallback it existed
to rule out: mutating `_coerce` to reject EVERY int override left all three legs
green. It stores `spec.max - 1` now and asserts the value differs from the
default, so the leg can actually fail.

And the shrink test's comment claimed `3 - 5 = -2` for an expression that computes
`chips.length - visible` = `2 - 3 = -1` — naming neither operand nor the result,
in a comment that was rewritten one round earlier precisely for miscounting.
2026-09-22 00:22:30 +02:00
peter 3f24248e01 fix: round 3 — a vacuous ordering test, a clamp that was a silent rewrite, a regex that missed
Round 3 over round 2's fixes. The trend is 22 → 13 → 6, and the character has
shifted: these are mostly defects in the REVIEW's own work, not in the epic.

The ordering test I added to hold the middleware's position could not fail for
either arrangement it names. All four `@app.middleware("http")` layers — the
access log, the two header decorators AND the setup gate that must be inside —
register as the same `BaseHTTPMiddleware`, so asserting on the class passes for
the pre-move stack and for a cap moved back outside CORS. It names the layers by
their dispatch function now, which is the only thing that distinguishes them.

The `sysconfig` bounds check CLAMPED, and clamping is a silent rewrite — the thing
this module refuses to do everywhere else, where an unusable override falls back
to the env default and logs once. On the max side the difference is cosmetic; on
the min side it is a live bug. `download_upload_max_bytes` had `min=0` before E7,
and a stored 0 means "no per-file cap" to the route that reads it. Clamped to 1 it
becomes a ONE-BYTE cap: uploads dead, nothing in the log, the admin page showing
"1". It raises `OutOfRange` now and takes the existing fallback, with its own
event code.

The `capped_body` sweep grepped for an identifier as the limit argument, so a site
written `capped_body(request, 4 * 1024 * 1024)` — or with the parameter not named
`request` — was MISSED, and a miss reads as "no new sites". Parsed with `ast`
instead: every call is seen whatever it looks like, and an unfamiliar argument
shape fails the equality instead of vanishing.

Frontend: the shrink test drove the mock in a way the real hook cannot produce
(it moved `visible` in the same step as the shrink, and the comment miscounted the
tags), so it reproduced the right END STATE by accident rather than the sequence.
`visible` is now left alone across the shrink, which IS the precondition.

Ops: the baseline guard answered "did this run rewrite it?" and therefore went
green on the second run over an already-tightened, uncommitted baseline — the
exact state its own comment says marks the localdev stack stale ever after. It
asks both questions now. And `.dockerignore` joins BAKED_PATHS: it decides what
the COPY lines actually take, so it is an input to the build context exactly as
the copied trees are.
2026-09-22 00:12:30 +02:00
peter 6b86673af4 chore: untrack tsconfig.tsbuildinfo, and hold the CI baseline the way the local gate does
`frontend/tsconfig.tsbuildinfo` was swept into the previous commit by a `git add
-A`. It is a machine-local incremental artifact carrying absolute Windows paths,
no lane reads or regenerates it, and it was already stale in the commit that added
it — it does not list the contract test that same commit created. Untracked, and
`*.tsbuildinfo` ignored, because the day anything does start rewriting it a green
gate would leave a tracked file dirty: the exact hazard the basedpyright baseline
just got a guard for.

And CI ran bare basedpyright while the local lane gained that guard, so a
burn-down would have been green on the server and red on the developer's machine
over the same tree. Two gates, one tree, one answer.
2026-09-21 23:59:59 +02:00
peter d924c665f8 fix(frontend): round 2 — the popover latch reset read a stale, negative count
The reset I added guards on `hidden === 0`, and `hidden` can be NEGATIVE.
`useChipOverflowFit` keeps `visible` in state and re-decides in a layout effect,
so the render in which the chip set SHRINKS still carries the previous
measurement — delete three tags under an open "+N" and `chips.length - visible`
goes to -2, the equality never matches, and the flag survives the one window the
reset exists to close. The next re-fit brings the menu back on its own, which is
the original defect with an extra step. Clamped at all three lanes, with a test
that drives the shrink through the stale value.

Also: the Scheduler characterization banner said "the four tests below" and now
sits above six, the last two of which are genuine regression guards for the
range-refusal fix. A future mutation check would have read it, seen them go red,
and concluded the harness was broken rather than that the guard worked.
2026-09-21 23:59:59 +02:00
peter aa51644f5f fix(security): round 2 — the SEC-20 fix had four defects of its own
A second review round over the first round's fixes, which is where the remaining
real bugs usually are. All four are the same failure as the round it reviewed: a
claim stated in a comment that the code does not hold.

The middleware moved inside CORS, but the setup gate is registered LATER and is
therefore OUTSIDE it — so the comment's "the setup gate never runs for a body that
is already too large" was false the moment it was written. On an unconfigured
instance that gate opens a DB session per request, which is exactly the anonymous
pre-auth work SEC-20 exists to remove. The cap now sits between them, and the
order is asserted against `app.user_middleware` rather than described: prose about
a stack that `add_middleware` builds inside-out is how this got backwards.

The edge ceiling and the route's own cap were pinned EQUAL, but they measure
different things — the edge counts raw body bytes, the route counts the decoded
file. A file of exactly the advertised maximum was refused one layer before the
route that would have taken it. The edge now allows for the multipart envelope,
and the test asserts the relation and the gap instead of equality.

`sysconfig`'s new `max` was enforced on write only. That bound did not exist
before this epic, so a value that was legal when it was written can be sitting in
the database — served unclamped it produces exactly the drift the comment says is
impossible, with the admin page showing a number outside its own range. Int
overrides are clamped on read now, which also survives an env default being
lowered later.

And `DOWNLOAD_UPLOAD_MAX_BYTES=0` meant "no per-file cap" to the route but a hard
zero to the edge: every upload refused, including the admin page's own way back,
recoverable only by editing env and restarting. The setting takes `ge=1`.

Two test gaps from the same round: the header assertion that is the ONLY check of
the middleware's position was DB-gated and skipped on a host without Postgres, and
the headroom "ratio" the docblock calls load-bearing was held as `<=`, i.e. no
ratio at all. Both fixed, plus a sweep that every `capped_body` call site in the
routers is known and fits under its ceiling — the point of that list is the third
entry, not the two that exist today.
2026-09-21 23:59:45 +02:00
peter 378b7559e4 build(frontend): finish the React 19 move, and hold the versions that were left in prose
react-i18next was still on ^14.1.2. The E27 S1 plan named it as "the one peer that
actually needs moving — React 19 support is the 15 line", and then it did not
move. The 14 line's <Trans> reads `element.ref`, removed in React 19, and its
withTranslation types are built on the GLOBAL JSX namespace @types/react 19 no
longer declares — both latent only because this app imports neither. 15.7.4 fixes
both at the source (`c.props.ref ?? c.ref`, `React.JSX.LibraryManagedAttributes`).

`reactEngine.contract.test.ts` joins its Node and Python twins and holds what was
only a comment: the compiler's `target` against the installed React major, react
and react-dom together, the types against the runtime, and react-i18next's floor.
A stale target is not a corrupt bundle — the plugin validates the option — but it
is a silent one, since tsc never sees Babel's output.

pythonEngine gains the seventh spelling nobody held: `requirements.lock`'s own
regeneration recipe names a python image, and wheels resolve against THAT
interpreter, so a recipe left behind yields a lock that does not match the image
installing it.

The type lane now covers `alembic` too, for the reason the ruff lane was widened
to it — migrations run on every container start, so one that raises crash-loops
prod. It came back at zero new errors, so the baseline is untouched. `tests/` is
deliberately still out, and the config says why.

vitest.config's docblock claimed the React transform is "a no-op for the plain .ts
node tests". It is not: the compiler rewrites src/lib's hooks in that lane too,
which is the point of sharing one factory with the build — and worth knowing
before ruling the compiler out of a divergence.

Tests: useCardPager had none, in either the hook or its two callers, though its
page clamp moved to render in S2 — the transformation where a non-convergent
condition is a crash rather than a glitch. Four cases now, including termination
on an empty list. Combobox's and Avatar's close/failed-URL cases are relabelled
CHARACTERIZATION: both pass against the pre-sweep code, and the tests beside them
are the ones that discriminate.
2026-09-21 23:33:02 +02:00
peter 05121fbc99 fix(security): SEC-20's exemptions become raised ceilings, and the 413 is dressed
The module admitted its two opt-outs under a stated rule — "both enforce their own
limit on the bytes they actually read, which is what makes opting them out safe" —
and that rule is false for one of them. `POST /api/downloads/upload` declares
`file: UploadFile = File(...)`, so FastAPI parses and spools the whole multipart
part to a temp file BEFORE the first dependency runs, `require_human` included:
its in-handler cap measures bytes that have already landed. Measured on the
installed fastapi 0.141.1 / starlette 1.6.0 — starlette's max_part_size guards
non-file parts only, so an anonymous chunked upload was bounded by nothing. In the
shipped self-host compose that lands in a RAM-backed tmpfs.

The exposure predates SEC-20 (no route was capped before it); what this branch
added was the claim. So the list becomes a mapping of per-route CEILINGS instead
of a set of exemptions: every request now gets a finite number, the upload route's
being the same one the route itself enforces. `sysconfig`'s knob is bounded by it
so an admin cannot raise the handler's cap past the edge and start collecting 413s
from a layer the admin page never mentions — held by a test, not by prose.

Two more things the review caught in the same module:

The 413 was not byte-identical to `capped_body`'s after all. `json.dumps` puts
spaces where JSONResponse does not, and the test that promised to hold them
together compared PARSED JSON, so it could never have seen it. Built through the
same JSONResponse now, and compared as raw bytes.

And the middleware sat outside CORS and the security headers, so its 413 carried
neither — on the split-origin dev setup the browser drops it and the user gets a
generic network failure instead of the translated too_large, while the identical
refusal from capped_body one layer down renders fine. It moves to just inside
CORS: still ahead of the session, the setup gate and every auth dependency.

Also corrected: the docblock's "largest non-exempt body is 64 KiB, ~16x headroom"
(it is 512 KiB and 2x — the diagnostics sink, named eleven lines further down),
and CLAUDE.md's Request SIZE row, which forbade the one correct use of
capped_body that is actually in the tree.
2026-09-21 23:32:49 +02:00
peter bf518223e5 fix(frontend): the scheduler's two number editors stop swallowing what you typed
Both refused an out-of-range value by discarding it: the entry vanished, the box
snapped back to the server's number, and nothing said whether the save had
happened, been rejected, or been ignored. That is the house rule about honest
states, applied to a write instead of a read. They now keep the draft, mark the
field aria-invalid and name the range (HU+EN), and the Save control is disabled
rather than silently inert.

The interval editor also dropped a keyboard user's focus on every exit — Escape,
Cancel and Save all unmount the focused input — so it hands focus back to the
pencil that opened it, the pattern IssueComposer already uses.

Tests: the four interval-editor cases added in S2 are relabelled CHARACTERIZATION,
because they pass against the pre-sweep code too (the old effect was already
guarded by `editing`). The editor that genuinely lost typing to a poll is the
maintenance batch box, whose old effect had no guard at all, and it had no such
test — it does now, anchored on a value that follows the server so it cannot pass
on a page that simply has not polled yet.
2026-09-21 23:32:32 +02:00
peter 332d24a646 fix(frontend): four state defects the deep-review found in the E27 S2 sweep
Every one of them is the same shape: a value derived correctly, and a WRITE to
the thing it derives from left unattended. Deriving hides a stale value; it does
not consume it.

PlayerModal — the watch toggle inherited the previous video's auto-watch. Moving
the queue-advance sync into render was right for `currentVideoId`, but the same
block also synced `status`, and that one had an ordering constraint of its own:
useYtProgress's cleanup for the OUTGOING item can still call setWatched(true)
near the end of a clip, and cleanups run after the commit. As an effect the sync
landed last and corrected it; from render it lands first and the stray write
wins. Press Next in the final ten seconds and the NEXT video showed as watched —
one click on the toggle then sent onState(next, "new") for an item already new.
Fixed by keying the mirror to the id it describes, so a write for another item is
ignored rather than displayed, and no ordering has to be argued about at all.

ChannelsProvider — a focus-channel intent was masked by the page rather than
consumed, and nothing ever cleared it. The manager is a page chunk: it unmounts
on every navigation, and its seed effect runs again on each mount, so returning
to it re-applied a channel name from earlier in the session over whatever the
user had typed since. The consumed marker now lives in the provider, keyed on the
token — which also closes the case the old clear-on-leave effect never handled,
opening a channel page (that does not change `page` at all).

TagFilterChips + ChannelTags x2 — `open = openRaw && hidden > 0` hides the
overflow popover when the lane stops overflowing, but floating-ui only reports a
close for a real event on a mounted surface, so the raw flag stayed true and the
menu came back by itself on the next re-fit — a window resize or the text-size
slider is enough, with nobody near it. The flag is now reset in render against
the same condition that hides it.

Avatar + downloadRow — remembering a failed image BY URL fixed the recycled-row
frame but never forgot it, so a transient 429 pinned a channel to its initial for
the life of the instance, never re-requesting. The memory is now dropped once
`src` moves off it, which keeps both properties.

And playerModalHooks' last exhaustive-deps suppression claimed "nothing in the
body reads a render value" while reading five. The invariant that is actually
load-bearing is that each of them is a per-item snapshot — written down instead.

All four fixes are mutation-checked: each new test was seen red, against the
fixed line only, naming the real defect.
2026-09-21 23:32:22 +02:00
peter c5e022f056 refactor(frontend): the long tail — and eslint-suppressions.json is gone (E27 S2)
Closes the burn-down. Hook suppressions 113 -> 34, `exhaustive-deps` 45 -> 2 (the target was
<10), and every one of the 34 survivors is an inline directive NAMING the invariant it
protects. The suppressions file is deleted, because a bulk file hides the next violation
instead of answering it — and it cannot hold a warning-level rule at all.

The sprint plan's three verdicts missed the answer that turned out to be the most common one:
adjust the state DURING RENDER. React re-runs the component and discards that pass rather than
committing it, so the stale frame an effect-based reset paints does not exist. Applied to the
Plex player's up-next card, the playlist view reset, the feed's optimistic overrides, the
channel banner's retry count, the download center's tab guard, the Plex search highlight, the
comfort-pin latch, Welcome's per-view outcome and the card pager's page clamp. It is legal only
for a component's own state — PlexBrowse's language commit resets PlexProvider's filters, so
that one stays an effect and says why.

Mirrors of a prop or query became derived values or `draft ?? server`:
- `downloadRow` had ui/Avatar's defect exactly — a boolean "poster failed" that an effect reset,
  in rows recycled down a virtualized list, so one job's missing poster painted over the next
  one's for a frame. Keyed by URL now.
- Scheduler's two number editors held a mirror an effect re-synced, on a page that POLLS — the
  shape a poll can overwrite mid-edit. A draft while editing, the server's value otherwise; six
  tests where there were none, the one behavioural change (the save button stops being dirty at
  commit rather than at the echo) mutation-checked.
- SetupWizard's "no token" and WatchPage's "gone" are facts about the URL, so they are seeded and
  derived rather than pushed into state by an effect that had to commit first.

Elsewhere: PlexInfo's rating chip was a component declared during render — a new component type
every render, so React remounted the subtree and dropped the focus on its filter button. Hoisted.
Feed's undo action and PlexPlayer's session reclaim both re-entered their own useCallback, which
the checker reads as a forward reference; both go through the ref idiom this file already uses.
BootGate's dep list was load-bearing for correctness — a re-run meant `location.reload()` — so
the "already adopted THIS identity" comparison now enforces it in code, with a test that fails
without it (the existing reload tests all install their spy too late to catch it).

One correction to the batch before this: keying ChatThread's and Messages' decrypt passes on
`q.data` instead of `q.dataUpdatedAt` was wrong. react-query's structural sharing makes it safe
in production, but the component must not melt down when something hands back a fresh object per
render — and Messages.keyboard.dom.test does exactly that, so the effect span the suite. Both are
back on the fetch as the trigger with the payload behind a ref, which is honest AND loop-proof;
the replaced comment had warned about precisely this.

CLAUDE.md gains four canonical-patterns rows for the above, and its "React 18" line is now true.
2026-09-21 01:44:21 +02:00
peter 4065ac7e6a refactor(frontend): the two players — one real ordering bug avoided, five suppressions gone (E27 S2)
The interesting part of this batch is a change that was WRONG and the test suite said so.

PlayerModal synced its local watch-status mirror and `currentVideoId` from an effect on
`active.id`. Moving that into render is right — as an effect there was a frame per queue advance
showing the new episode's chrome over the previous item's watch state, with `navigated` still
true so the "you have navigated away" affordance flashed. But the same effect also advanced
`currentIdRef`, and that line must NOT move: useYtProgress's per-item cleanup persists the
OUTGOING item's position and guards it on `currentIdRef.current === id`. React runs every
cleanup before any setup, so in an effect the ref still names the item being left. Advanced to
render time the flip happens first, the checkpoint is skipped, and stepping next→prev back to an
item resumes from the frozen launch queue's stale position. "resumes an item played earlier in
the SAME session" caught it immediately (748 instead of 830); the ref write stays in its own
effect with the ordering written down.

The rest:
- PlayerModal's keyboard handler is a useEffectEvent, so the "bound once, reads live state"
  contract is enforced instead of asserted. `focusModal` is a useCallback now — it reads a ref
  and nothing else, and three hooks take it as a parameter where a fresh identity would spread.
- PlexPlayer's session reclaim called `loadSession` from inside loadSession's own useCallback,
  which the checker reads as a forward reference. Routed through a ref, the file's existing
  answer for exactly this (detachSessionRef, cancelAutoSkipRef).
- PlexPlayer's item-load effect excluded `prefs` by hand so a track pick from the player's own
  menu would not double-load the session. The two language prefs go through a ref instead —
  "latest value, not a trigger", said in code rather than in a comment.
- PlexPlayer closes its up-next card during the render that changes `id`, not a commit later.
- The one PlexPlayer directive that stays is the live session counter, whose cleanup must bump
  the CURRENT value: the rule's advice to copy it into a variable is precisely wrong there, and
  the comment now says why rather than pointing at another comment.

Suppressions 75 -> 70; exhaustive-deps 20 -> 17.
2026-09-21 00:27:15 +02:00
peter ce08fc2aec refactor(frontend): a stable useHistorySubview, and PlexBrowse can stop disabling the check (E27 S2)
useHistorySubview returned a fresh object with a fresh `open` on every render. Correct, but it
made "re-run when the sub-view moves" unwriteable: PlexBrowse had three effects that wanted
exactly that and had to disable the dependency check to get it, and a fourth that listed `sub`
honestly and therefore re-ran on EVERY parent render, saved only by a ref guard.

Fixed at the source, which is where the house rule says a principle belongs: `open` and `back`
are useCallbacks and the returned object is memoised. `url` is a module constant at every call
site (PLEX_SUB_URL, MODE_URL) and the body otherwise touches only module functions, so there is
nothing left to churn on. All three PlexBrowse dep lists are now the honest ones, and the
comment that claimed `open` was "behaviourally stable" is replaced by it actually being stable.

PlexBrowse's remaining four are escape hatches, and two of them are now ONE:
- `infoScrollRef` is written from four places, and the compiler objects to a ref being modified
  outside the effect that reads it. That objection is a conservative false positive — a scroll
  memory needs exactly that arrangement — but four bare mutations were worth concentrating
  anyway. One `rememberInfoScroll` writer now holds the "stash on the way out, reset on a fresh
  page" rule, and answers the checker once instead of three times.
- The language-commit effect cannot move into render: it resets PlexProvider's filters, and a
  render-phase setState may only touch the component's own state. Said so at the site.

Suppressions 78 -> 75; exhaustive-deps 23 -> 20.
2026-09-21 00:20:05 +02:00
peter ea6d829c29 refactor(frontend): the context providers adjust state in render, not after it (E27 S2)
Eleven suppressions across the five providers. The recurring shape here was "an effect that
undoes something when you navigate away", and every one of them ran a commit LATE — the new
page painted once while the old module's state was still live.

Adjusted during render instead (React discards that pass rather than committing it, so there
is no in-between frame):
- SavedViewsProvider reset the sticky view and closed its modal from an effect on `source`, so
  the first Plex render carried the feed's sticky id — long enough for a stale view chip. It
  also followed the active view into stickyId from a second effect; that one converges by
  construction, since preferSticky returns the sticky match when there is one.
- NavigationProvider ended a live YouTube search from an effect on `page`. Keyed on the page
  CHANGE rather than derived from it, because "ended" has to be permanent — deriving would have
  quietly brought the search back on return to the feed.
- ChannelsProvider's manager focus is now simply DERIVED from the page: it means nothing off
  the Channels page, and focusChannel sets it and navigates in one handler.

Honest deps by freezing what is genuinely mount-time:
- FeedFiltersProvider persisted a share link's filters with an empty dep list; listing `filters`
  honestly would have re-persisted on every edit and spun. The mount value is frozen into its
  own state, so all three deps are stable and the effect says what it does.
- ChannelsProvider's URL-seed layout effect and NavigationProvider's initial history stamp get
  the same treatment. The three slice setters they call were already stable useCallbacks
  (useFilterState / useSortState / useAccountPersistedState all return one).
- SavedViewsProvider reads `cfg` through a ref: it re-memoizes on every filter edit, and the
  per-source seed guard already enforces once. That also made the set-state finding go away —
  with the render value gone, the checker can see the seed for what it is.

WizardProvider keeps its one escape hatch, now inline with the invariant: auto-opening is a
one-way decision taken from server data as it arrives, and deriving it would reopen a wizard
the user had just closed.

Suppressions 89 -> 78; exhaustive-deps 27 -> 23.
2026-09-21 00:12:44 +02:00
peter 6a7ff330f6 refactor(frontend): the shared hooks in lib/ stop lying about their deps (E27 S2)
Ten more suppressions, and React 19 turns out to supply the missing primitive.

useEffectEvent (stable since React 19.2, which S1 brought in) is the canonical answer to the
"bind the listener once, but read the latest values" pattern these hooks were disabling the
checker for. Three of playerModalHooks' four escape hatches become honest dep lists: the
wheel-volume handler, the native-controls handover, and the fullscreen-change sync. The
contract is no longer prose — the compiler enforces it.

The fourth stays, deliberately, and now says why at the site: the YouTube player's lifecycle
effect must run once per video (re-running destroys and rebuilds the iframe), and YT.Player
takes its callbacks as an object literal, which an effect event may not be passed into.

Fixed, not suppressed:
- useDismiss captured onClose when the popover opened, so a caller that re-rendered with a
  different handler while it stayed open kept getting the OLD one called. The disabled dep list
  asserted the opposite ("read fresh from this run's closure"). Both the handler and the ref
  list are read through refs now; three tests, the two that matter mutation-checked.
- usePref wrapped its seed in a useCallback whose dep list had to be excused. useState calls a
  lazy initializer exactly once, so the memo was saving nothing; inlined.
- useCardPager clamped an out-of-range page from an effect. Adjusted during render instead —
  React discards that pass rather than committing it, so no out-of-range page is ever painted.
- plexPlayerHooks' auto-skip countdown reads its marker from the ref that already existed, so
  the dep list names its real inputs. That also made the set-state-in-effect finding go away:
  the progress is genuinely timer-driven, and the checker can now see that.
- useUrlMirror and useChipOverflowFit only needed their stable refs listed.

Two test-harness suppressions (a render-phase probe ref, a module-level setState handle) moved
inline with the reason they exist: both exist to observe a hook from outside without changing
what is being measured.

Suppressions 99 -> 89; exhaustive-deps 35 -> 27.
2026-09-21 00:07:12 +02:00
peter 7e1aeeb9fe refactor(frontend): honest effect deps in the app shell and the ui layer (E27 S2)
Burns the first 14 of the 113 hook suppressions. Three verdicts, applied per site:

A useEffect that should not exist (6):
- Avatar remembered "the image failed" as a boolean an effect reset on src change. The
  component is recycled across items in the virtualized feed, so after one avatar 429s the
  next item's slot committed the fallback initial over a perfectly good image and only the
  post-paint effect cleared it. The failure is now keyed BY URL; derived state cannot be stale.
- Combobox cleared the typed query from an effect on `open`. Moved into setOpenAnd, the single
  funnel every close goes through. That also stops a spurious onQueryChange("") AT MOUNT, which
  told a controlled caller its own initial query had been cleared before the user touched it.
- CommandPalette reset query + picked row from an effect. Closed now means UNMOUNTED, not
  hidden, so a fresh open starts blank because mounting already means that -- and the ranking
  memo over every channel, playlist and saved view stops running while the palette is shut.
- Pager's page box held a copy of the page that an effect re-synced; it now derives it and
  keeps a draft only while being edited.
- App's ?link= / ?oauth= / ?admin= one-shots asserted "once per mount" with an empty dep list.
  The ref guard the codebase already uses in 22 places ENFORCES it, so the honest deps can
  include `t` -- which a language switch legitimately changes, and which would otherwise have
  re-fired the toast.

Honest deps that also fix something (5):
- App's module preload was keyed on the account id, so a permission change that alters the
  module set without moving the id never re-preloaded. `preload()` is idempotent, so depending
  on `me` costs nothing.
- App's access-request nudge now re-issues on a language switch; it used to stay in the old
  language until the pending count changed.
- App's diagnostics effect: applyDiagnostics already no-ops on an unchanged effective level,
  so picking four fields out by hand was guarding nothing and missed any field added later.
- DataTable's two filter effects: a ref holds the latest onFiltersChange so commitFilters can
  be stable, which is what lets the effects name it instead of switching the check off.

Genuine escape hatches (7) moved OUT of eslint-suppressions.json into inline directives that
name the invariant at the site -- the toast handover's render-phase global, the live region's
deliberate arming tick, the docked one-slot component, the popover engine's floating-ui list
state, dnd-kit's per-drag `moved` flag, BackToTop's DOM read and the palette's debounced
announcement. A reader of the code now sees why; a JSON file no one opens does not.

Tests: Avatar (3, new file) and Pager (3) pin the derived-state frames with a layout-phase
probe -- layout effects run before any passive effect, so they read the frame the browser would
paint. Combobox (2) pins the close-clears-query contract and the no-callback-at-mount rule.
All three sets mutation-checked against the effect-based originals.

Suppressions 113 -> 99; exhaustive-deps 45 -> 35.
2026-09-20 23:57:08 +02:00
peter 7d682c90d5 build(frontend): React 18 -> 19 with the React Compiler (E27 S1)
The compiler auto-memoises, which is the point: it infers what the app's
112 `useCallback` and 48 `useMemo` call sites were hand-written for and
applies the equivalent itself. It is on repo-wide with no opt-out list —
it bails on anything it cannot prove safe, and says so through the lint
rules. Verified in the built bundle rather than assumed: 369
`react.memo_cache_sentinel` markers across 35 of the 76 chunks.

ONE transform, shared by vite.config.ts and vitest.config.ts through
reactPlugin.ts. Not two `react()` calls: if the compiler ran only in the
production build, every vitest run would exercise code the user never
receives and a compiler-induced change would ship green — the same
gate-vs-image split 0.93.2 was released to fix. The plugin is imported
rather than named by the usual string, because a string is invisible to
knip and, worse, a typo in it is a silent no-op that ships an uncompiled
bundle, while a bad import is a tsc error.

eslint-plugin-react-hooks 5 -> 7 brings the whole compiler rule set: 283
findings, triaged rather than bulk-suppressed, because the two halves
needed different answers.

  react-hooks/refs (217) is OFF. 153 of those 217 are this app's popover
  architecture, not its history: 79 are @floating-ui/react's documented
  `pop.refs.setReference` / `getReferenceProps` callback-ref setters, and
  74 more are passing the popover handle down as a prop, which is what
  components/ui's popover engine is for. Suppressing them would not help
  the next popover anyone writes. The remaining ~64 are genuine
  render-phase `ref.current = x` writes and are real, but the rule cannot
  tell them apart here; they are E27 S2's work.

  The other 66 go into eslint-suppressions.json as a ratchet — new
  violations fail the gate, and `--prune-suppressions` makes a fixed one
  fail too until it is removed. 45 of them are `set-state-in-effect`,
  which IS E27 S2's subject, so that sprint now has a measured worklist.

react-i18next deliberately stays on 14. It never declared a React-19
peer — every version from 14 to 17 asks only for `react >= 16.8.0` — and
what the newer lines actually require is i18next 25/26 against our 23.
That is an i18n runtime migration wearing a React hat, and it buys this
sprint nothing.

Two React 19 type changes: `useRef<T>(null)` is now `RefObject<T | null>`
(the honest shape — the element is null until it mounts), so
PlexPlayerSettings' prop follows; and reactPlugin.ts joins
tsconfig.node.json's include.

The compiler skips memoising VirtualGrid and reports it: TanStack
Virtual's `useVirtualizer()` returns functions that cannot be memoised
without going stale. That is the right trade for a component whose cost
is its row window, and it is the one such site in the app, so it carries
a targeted disable with the reason — a second incompatible library
anywhere else must still stop the gate.

Nine lanes green, vitest 1558. Browser UAT on the dev server against the
still-running React 18 build as a control: boot, feed (virtualized grid,
popover menu, a filter change that moved the list 195,187 -> 64 items
with the URL mirrored, so no stale memoised UI), command palette, player
modal — no console errors on any of it.
2026-09-20 23:21:08 +02:00
peter cc9fc3b544 build: pin one Python version across the app, the images and the gate
0.93.2 shipped a week's worth of confusion about which Node this repo
runs on; the Python side had the identical shape and nobody had looked.
Prod (Dockerfile), the test image (Dockerfile.test) and CI (check.yml)
each spelled 3.12 as an independent literal, there was no
`.python-version`, no `requires-python`, and ruff had no target-version
so it judged our syntax against some other interpreter. The dev machine
had no 3.12 installed at all — only 3.13 and 3.14 — so every ad-hoc host
script ran two majors ahead of production.

`.python-version` is now the single source of truth, and the contract is
held by tests rather than by prose. It is split in two because no single
runner can see every link: the pytest suite runs inside the test image,
which mounts backend/ alone and cannot read the repo-root Dockerfile or
the workflow.

  test_python_engine_contract.py : running interpreter == pyrightconfig
                                   == ruff.toml
  pythonEngine.contract.test.ts  : .python-version == pyrightconfig ==
                                   ruff.toml == Dockerfile ==
                                   Dockerfile.test == check.yml

Together they close the loop, and the backend half carries the assertion
that pays — that the interpreter running the suite is the pinned one. A
green suite on another minor does not mean the shipped image is green;
it means they have drifted and nobody has noticed yet.
2026-09-20 23:02:11 +02:00
peter c9101e3450 ci: type-check the backend — the gate's ninth lane (E27 S3)
Three tsc projects held the frontend and NOTHING held the backend: ~36k
lines of `app/`, annotated in 113 of 128 files, that no lane had ever
read. Asymmetric by accident, not by decision.

basedpyright, in basic mode. Measured against mypy on this tree with the
dependencies actually installed — without them basedpyright errors on
every import and mypy's --ignore-missing-imports quietly turns them into
Any and UNDER-reports. Both land ~190-200 errors with the same
distribution, so the choice fell to what differs: basedpyright is ~2x
faster (4.4s vs 7.4s, and this runs on every `siftlode lint`), resolves
our `app.*` imports from its own config with no MYPYPATH, and has a
first-class `--writebaseline`. That last one is the ratchet itself;
mypy would have needed a third-party tool for it.

198 pre-existing errors are baselined, so the lane fails on NEW errors
only and the list burns down epic by epic. A green lane means no new
damage, not a clean codebase.

It runs in the pytest test image rather than on the host, and that is
forced rather than chosen: `requirements.lock` cannot be installed on
Windows at all — uvloop refuses to build there — so a host venv could
only ever be the lock minus a package, and a type checker without the
real third-party packages reports nonsense rather than nothing. The
image installs the pin from requirements-dev.txt, so the lane's version
IS the pin and there is no host copy to drift. The cost is that
`siftlode lint` now needs Docker, which it did not before.
2026-09-20 23:02:02 +02:00
peter 51daf3e388 feat(security): cap every request body before auth runs (SEC-20)
FastAPI reads a declared body IN FULL before the first dependency, so
`POST /auth/password-login` drained an arbitrary upload from an anonymous
caller and only then answered 401. 70 of the 72 body-declaring routes
behaved the same way: one unauthenticated client could spend the
process's memory and time at will.

A raw ASGI middleware, not `@app.middleware("http")` — BaseHTTPMiddleware
cannot wrap `receive`, so it could only trust `Content-Length`, which a
client may omit (chunked) or understate. This refuses an oversized
declaration outright and counts the bytes of everything else as they
arrive, the same rule `capped_body` already applies per-route. Registered
directly inside the access log: a refusal is still logged with its
request id, but the session, CORS, the setup gate and every auth
dependency never run for a body that is already too large.

Two routes opt out, under one stated rule: an exemption is ONLY for a
route whose own cap is larger than the global one. That is why
`/api/diagnostics/events` is absent — it caps itself at 512 KiB, below
the ceiling, so exempting it could only widen what it accepts. The list
names route templates and a test resolves each against the live route
table, so a rename fails the gate instead of silently dropping an
exemption in prod.

The 413 is byte-identical to `capped_body`'s and the test compares the
two against each other rather than against a literal — two shapes for one
fact is how the `cancelled`/`canceled` bug happened.
2026-09-20 22:51:32 +02:00
peter 09d351a0e7 docs(ci): the two gate jobs queue, they do not run in parallel
Declared side by side, so they read as parallel; the lab runs a single
act_runner with one slot, so the backend job starts at the exact second the
frontend one ends. Measured across runs #86-#91: ~4 + ~6 = ~10 minutes from the
push, not ~6.

Written down because believing the comment cost something real — the ops
release read-back budgeted the max instead of the sum and timed out on its
first live run, reporting a still-running gate as a failing one.
2026-09-20 22:05:56 +02:00
peter b5078003ce release: 0.93.2 (bug/ci-node-version-split)
gate / frontend (push) Successful in 3m51s
gate / backend (push) Successful in 6m7s
2026-09-20 21:49:56 +02:00
peter f2c32cfb8e chore(release): 0.93.2 — a Plex amount reads the same in every browser 2026-09-20 21:49:36 +02:00
peter 1453fa4a33 ci: run the gate on branch pushes only, not on release tags
A release pushes main and then the v<ver> tag, and a bare `on: [push]` treated
the tag as a second push of the identical commit — every release ran the whole
gate twice to the same verdict (runs #81-#90 are five such pairs).

It has to be `branches`, not the `tags-ignore` this obviously wants to be:
Gitea's matchPushEvent requires every named filter to match and each one bails
on the wrong ref kind, so `tags-ignore` rejects every BRANCH push and silently
disables the gate. Measured in go-gitea v1.27.3, where it differs from GitHub.

A trigger filter that stops firing is invisible, so the ops `release` now reads
the verdict back for the sha it pushed and says so when there is no run for it.
2026-09-20 19:47:19 +02:00
peter efe106dfda build: pin one Node major across the dev, CI and build machines
The server gate ran node:22 while the dev gate ran node:24, and the two disagree about
Intl.NumberFormat's compact currency padding and about which realm's Blob a Response hands
back — so the frontend job had been red for 73 of its last 90 runs while the local gate was
green, and nothing said so.

Align everything on 24, with `.nvmrc` as the single source of truth: the CI container, the
prod build stage, `engines`, and `@types/node` (which was two majors ahead of the runtime, so
tsc was green over APIs the shipped engine does not have). The backend job's node only runs
the checkout action, so it tracks the pin as hygiene — node 20 is EOL — but deliberately
stays outside the contract.

The pin is held by a contract test that reads every one of those places AND the engine
running it, so the next drift is a red lane on its first run instead of a CI log nobody reads.

Verified on node:24-bookworm (the CI lane: 228 files / 1552 tests, typecheck, lint,
format:check, knip all exit 0) and on node:24-alpine (the prod build stage, v24.21.0, SPA
built). Each assertion of the new test was mutation-checked red.
2026-09-20 18:06:24 +02:00
peter ec63056bd7 test(api): assert the export body's brand, not which realm built its Blob
`expect(x).toBeInstanceOf(Blob)` asks which realm constructed the body, not
whether there is one. Under jsdom the ambient `Blob` is jsdom's class, while
`Response.blob()` hands back node's own on Node 22 and jsdom's on Node 24 — so
both assertions passed locally and failed on the server gate, for a reason that
says nothing about the code under test. `.text()` is the mirror image: jsdom's
Blob does not implement it, so reading the bytes back is green on 22 and red on
24.

Both now assert the brand (`[object Blob]`, agreed by both realms) plus size and
type. Mutation-checked in node:22-bookworm: dropping `{ raw: true }` from
`debugExport` still turns the test red, so the repaired assertion still catches
the defect it was written for.

core.test.ts also stops building its Response from a jsdom `Blob`: on Node 22
undici does not recognise it as a body and stringifies it, so the fixture's body
was the 13 bytes of "[object Blob]" rather than the zip header. A string body is
what the sibling test already used.
2026-09-20 16:33:26 +02:00
peter 09becb3d00 fix(plex): pin the compact currency format instead of inheriting the engine's
`{ notation: "compact", maximumFractionDigits: 1 }` leaves the MINIMUM to the
engine, and engines disagree: the same options render "$160.0M" on Node 22 and
"$160M" on Node 24 (measured 2026-09-20 in node:22-bookworm, the image the
server gate runs, against the local Node 24.17). Browsers split along the same
line, so a viewer on an older engine saw the padded form. `minimumFractionDigits:
0` makes both render "$160M"; the plain `compact` helper (vote counts) was
measured too and already agrees on both.

This is why the server gate's frontend job was red on PlexInfo since 09-17 while
the local gate was green: the two run different Node versions.
2026-09-20 16:33:26 +02:00
peter 8d2ec5e867 release: 0.93.1 (bug/preview-crashes-the-tracker)
gate / frontend (push) Failing after 4m10s
gate / backend (push) Successful in 5m50s
2026-09-20 07:55:37 +02:00
peter 0e9839e73d chore(release): 0.93.1 — the report preview works again 2026-09-20 07:51:16 +02:00
peter 05ab963951 fix(issues): the preview must not ask the tracker for a repository context
The composer's preview did not fail a call — it killed the Gitea process, and
with it the git server and the container registry. `POST /api/v1/markdown` with
`{"Mode": "gfm", "Context": "<repo>"}` makes Gitea resolve every hex-shaped word
as a commit link, and on the API's render path the commit checker it asks is nil:

  panic: runtime error: invalid memory address or nil pointer dereference
  renderhelper.(*commitChecker).IsCommitIDExisting(0x0)
   <- markup.hashCurrentPatternProcessor (html_commit.go:217)

Every report carries hex — a tab id, a request id — so this was not an edge case
but the only case: the preview answered "the tracker isn't answering" every time,
because the tracker had just died.

Measured on throwaway instances, not argued. Same body, same call:
  gfm + Context + "5645b1c0ae01"  -> 1.26.4 CRASH, 1.27.1 CRASH, 1.27.3 200
  gfm + Context, no hex           -> 200
  gfm, no Context                 -> 200, and the HTML keeps its tables, task
                                     lists and chroma-highlighted code blocks
Upstream fixed it in 1.27.2 ("fix: avoid markup render panic", #38698), but a
self-hoster runs whatever version they run, so the payload is what gets fixed.

Sending was never affected: the created issue's own page renders the same body
fine (measured) — a real repo context has a real commit checker. The only loss is
that a hash or a `#123` is not yet a link in the preview; the issue page links it.

The test pins the payload dict, because the defect is in what we send.
2026-09-20 07:51:08 +02:00
peter 8142484347 release: 0.93.0 (feature/e16-debug-issue)
gate / frontend (push) Failing after 4m11s
gate / backend (push) Successful in 5m58s
2026-09-20 05:54:26 +02:00
peter 0d91b82f86 chore(release): 0.93.0 — report a fault from inside the app
E16: a recorded debug session becomes a drafted issue with its own logs attached, sent to the private Gitea tracker or downloaded as a zip. Includes the four epic-end review rounds and the Plex player's focus trap.
2026-09-20 05:54:03 +02:00
peter 90fc86d437 fix(diagnostics): deep-review round 4 — a claim only its owner may write, and a player that keeps the Tab
Round 4 was targeted at the surfaces rounds 2 and 3 changed. 29 findings raised,
26 survived deduplication, 12 survived three adversarial readings each.

`set_delivery` renewed the claim but never checked it was still ours, while its
sibling `release_send` always did. `expire_on_commit` is off, so the token it
re-stamped came from the caller's own memory, not the row: a send that outlived
its 600 s lease came back, overwrote the take-over send's committed links with a
full-column write, and put its own dead token back. It now takes the token, reads
the row's claim with a targeted scalar (not a refresh, which would discard the
`issue_url` assigned one line earlier) and raises `SendClaimLost` rather than
writing. An issue created by a send that had already lost its claim is logged with
its number — that record is the only one that will exist.

The tracker's settings are read above the pre-call checkpoint instead of inside
`_client`. Reading them there was a SELECT one line after the commit that was
meant to end the transaction, and held it — idle, snapshot pinned — across the
repository check and the create. The comment and the module docstring said
otherwise; measuring it in pg_stat_activity is what settled it. Two more comments
were saying things the code had stopped doing: the `**state` spread does not carry
the claim (`set_delivery` strips and re-stamps it), and react-query does not
debounce the preview — the editor being unmounted is what paces it.

The composer stopped stranding the focus on `<body>`: a send that creates the
issue and then fails on an attachment unmounts the button that was just pressed,
and only the success path handed the focus on. Export no longer saves into a
refusal it does not need — the zip is the way out of a stuck send, and its
pre-save was the one step that could not survive one. The delivery banner
announces through a role instead of an `aria-live` created with its own message.

`historyState` returns a union, so `chipTone` matching on the same keys is checked
by tsc rather than by hope.

PlexPlayer was the only full-screen overlay in the tree with no focus containment
at all — Tab walked out of an opaque player into a page nobody can see. It now
carries the shared trap. Closing that hole exposed the one underneath it:
`useDismiss` registered as a layer that traps the Tab while installing no trap,
so the player stood its own down for its own menus. Measured in a real browser:
30 Tabs with the settings panel open, 9 escapes into the nav rail, and 0 after.
Escape ordering is unchanged — the panel first, then the player.

Three tests that did not hold their line: the advisory lock's call site (deleting
it left the suite green), the trail assertion whose `all()` was vacuous over an
empty filter, and a `layerStack` assertion the stand-in dialog already forced.
All eleven executable fixes mutation-checked red for the right reason; the two
comment corrections have nothing to pin.
2026-09-20 04:30:42 +02:00
peter 2fe58073e4 fix(diagnostics): deep-review round 3 — a keyboard owner that is not a Tab trap
Round 2 made the step prompt a blocking layer so that `j` and the `g` chords
would not reach the page behind it. The dialog it sits in gave up its Tab trap
because of that, while the prompt's own replacement trap covered only its own
two controls — so Tab from anywhere else in the dialog (the bar's Screenshot or
Stop, the dialog's own buttons) walked out behind the scrim. `layerStack` now
separates the two questions it had folded into one flag: a layer can own the
keyboard and still leave the Tab trap to the dialog below it. The mirror case,
a dialog opening over the prompt, falls out of the same split.

A send's claim is now a lease with an owner. Every progress report re-stamps
it, so a send slower than the lease cannot lose it while it is still running,
and only the send that took a claim can give it back. The issue_create audit
row is staged inside the send and rides the same checkpoint as `issue_url`, so
the trail cannot be missing for an issue that exists. The attachment-name
prefix has one definition, shared by the producer and both readers.

A running send is reported to the UI. A reload or a second tab used to render
it exactly like a send that STOPPED, and offered a retry the server was certain
to refuse; the composer now says the report is being sent, holds the draft, and
puts both the retry and Create issue out. The past-sessions list says the same
for the same reason — it is the list a user opens to check on a send, and both
"draft" and "half sent" describe one that died. `/preview` and the retry path
got a pace of their own, and the per-user byte cap took the user-scoped lock
that makes it a cap instead of a guess.

Verified in a real browser on localdev: Tab cycles the whole dialog with the
prompt open, the prompt still swallows the page's keys, and the issue link is
focusable where the composer puts it. All 24 regression tests mutation-checked.
2026-09-20 02:43:56 +02:00
peter bc03e82687 fix(issues): a send keeps its claim through every step, not just up to the issue
Creating the issue rebuilt the delivery state from scratch and dropped the
claim with it, so for a new issue the claim covered only the creation — a
draft, a screenshot or a second send could change the report under the uploads
and the body edit that follow, which is exactly what the claim is for. The
state is now built on what was read, and a test walks the send's checkpoints
to see the claim still held at each one.

Found by mutation-checking the round-2 tests: removing the claim's release
left them green, because for a new issue there was nothing left to release.
2026-09-20 01:03:23 +02:00
peter 3f983e6d76 fix(diagnostics): deep-review round 2 — a send that claims, not one that blocks
The send no longer holds the session's row lock while it talks to the tracker:
it claims the session, gives the lock back, and checkpoints each step. A Stop
is never held behind a send, while a draft, a screenshot or a second send are
refused at once instead of queueing. The claim ages out, so a request that
died mid-send does not strand the session.

What the composer says is what happened: an attachment that went is no longer
named as one that did not (the missing body edit is its own line), and the
issue's screenshot count comes from the server when the list cannot be read,
so a failed read no longer blocks saving the draft. The zip is fetched as a
request, so its pace is a message and not a saved error page. Screenshots of a
finished report can be removed again; only one still being sent is held.

A recording's log elevation can no longer outlive it. The step prompt keeps
the keyboard (the page's keys stay quiet) and traps Tab itself. An image
pasted into a dialog opened over the composer stays that dialog's.
2026-09-20 00:51:12 +02:00
peter 134ec9a68e fix(diagnostics): deep-review round 1 — a send that cannot race, lose or repeat itself
The send, the draft and the screenshots take the session's row lock, so an
autosave or a paste during a send waits for it and then sees it, and two sends
cannot open two issues. A save keeps the delivery's progress; a failed edit
that shows the screenshots is finished by the retry; an answer that is not the
tracker's API (a redirect, a proxy page) is refused, not a crash that loses the
issue's record. Only an attempt that opened an issue counts toward the pace.

Screenshots: deleting one after the send is refused, a user's storage is
capped across sessions, new sessions and zip exports are paced, images are
stored uncompressed in the zip. One liveness rule, one session view, one image
type table.

Logging: a session's elevation is tracked apart from the user's own, so the
browser's own requests (no header) no longer lower the floor a session needs;
Stop ends it at once.

Composer: an image pasted anywhere is taken; what is still unsaved is saved on
close, and "saved" is shown only when it is; the draft is held while it is
sent; a failed screenshot read says so instead of reading as none; focus moves
to what stays; the preview follows the text size. Pill: the step prompt and a
running Stop survive the pill moving between hosts, the prompt keeps the
dialog's Tab trap, the toast region lets clicks through, other tabs hear of a
Start or Stop, the last seconds are sent before the cap, a capture with no
frame is an error.
2026-09-20 00:09:52 +02:00
peter 9cf177b198 feat(diagnostics): send a bug report to a private Gitea issue tracker (E16 S4)
- admin config group "Bug reports": Gitea address, repository, token (secret),
  and "Check the connection" — the repository must answer, take issues and
  be PRIVATE (a report carries its reporter's log); checked on every send too
- POST /api/debug-sessions/{id}/issue sends the SAVED draft: create the issue,
  upload each file on its own request (the measured API), then show the
  screenshots inline; progress is kept in the draft, so a send that stops
  half-way keeps its issue and the retry sends only what is missing; 5 new
  issues an hour per user; audited as issue_create
- the composer: Create issue (only where a tracker is set up, naming the
  repository), a partial-send banner with retry, and a Preview — the tracker's
  own rendering in a sandboxed frame, or the zip's markdown without one
- the recorder's refusals answer in the localized family (debug_session_*,
  screenshot_*, issue(s)_*), not bare codes the UI could only call generic
- a tracker refusal logs the status only: its message can quote the title back
2026-09-19 22:48:20 +02:00
peter a5fae489e0 test: a request that raises is rolled back, as production rolls it back
The shared test session kept whatever a failing handler flushed and never
committed, so a later read in the test saw a write production loses when
get_db closes the session — and a handler whose failure path must commit
(E16: the tracker's issue link) looked right without the commit.
2026-09-19 22:48:20 +02:00
peter 8d0083d4e4 feat(diagnostics): the issue composer — draft, review, screenshots, zip export (E16 S3)
After Stop (or at the cap, or from Settings → Diagnosztika's past sessions)
one dialog drafts the issue: title, description, the steps drafted from the
session's markers and page changes (a marker without a label is named by its
time), the environment (version, browser, system, window, language). Beside
it, what goes with it: the browser and server logs — each reviewable line by
line, an unticked line is not sent — and the screenshots, removable, with
paste / drop / pick to add more.

- it saves as it goes (debug_sessions.draft, migration 0090): the rendered
  markdown in the user's language, the excluded rows, and its own fields to
  reopen as it was; a sent session is read-only
- the zip export (issue.md, client.log, server.log, screenshots) is built
  from the saved draft by app/issues/bundle.py — the one definition the Gitea
  delivery (S4) will share; .log, not .jsonl, which Gitea refuses
- the log line's level tags and clock move to components/logLineParts so the
  review list draws a level the way both log viewers do
2026-09-19 22:26:49 +02:00
peter 9f177557d2 docs(ui): the toast dock in the primitives catalog 2026-09-19 22:08:21 +02:00
peter 46fce4a767 feat(diagnostics): the debug recorder — start, the pill, steps, screenshots, stop (E16 S2)
- the start dialog is the consent: level (Részletes/Teljes), what is recorded,
  retention and the cap (now served as diagnostics.session_minutes)
- the pill docks under the toasts: elapsed time, Step (an optional label,
  its own Escape layer), Screenshot (tab capture where the browser has it),
  Stop; compact on a phone
- while recording, lib/log tags every event and every request path (req,
  beacon, uploadReq via one identityHeaders) names the session; applyDiagnostics
  raises the level for the session's run and never lowers a raised one, and
  lapses at whichever ends first
- Stop sends what is waiting before the server closes the session; the cap
  says so and opens the composer
- Settings → Diagnosztika and the command palette start/step/stop it
- the evidence leaves the log upload's own request lines out
- the step shortcut is deliberately palette-only: the keymap's D1 rule bars a
  new modifier chord, and a bare letter collides with the players'
2026-09-19 22:08:10 +02:00
peter 2d9595b0e5 feat(ui): a control can dock under the toasts and ride every toast host
The toast stack already has a host in the page and in every overlay (dialogs,
both players, the fullscreen player), the topmost one shows it, and inside an
overlay it sits above the player's control band. lib/toastDock lets one control
use exactly that placement; it renders outside the live region, so a ticking
control is not read out every second.
2026-09-19 22:08:10 +02:00
peter a7d7456279 feat(diagnostics): debug sessions — start, stop, the tag on every line, screenshots (E16 S1)
A session is one recording, capped at 60 minutes by the server. While it is
open, a request naming the caller's own session in X-Debug-Session is logged
at the session's level with every line tagged; a header naming anything else
is ignored. The browser's batches count a session as consent to its level and
keep the tag only on its own events. Screenshots are judged by their bytes,
capped in size and count, and served back to their owner only.

- debug_sessions / debug_screenshots tables, one open session per user
  (partial unique index), log_events.debug_session_id indexed
- /api/debug-sessions: start (joins a running one), stop, list, get, events,
  screenshots; /api/me reports the live session
- the hourly log job closes expired sessions and drops them with the logs
- logsink.serialize and routes._common.capped_body shared by their readers
2026-09-19 21:38:53 +02:00
peter 44a792c34f test(i18n): a 404 target the scan cannot read fails the backend-id contract
what=SOME_CONSTANT reached no locale check at all; only _common.py's own
pass-through may name the target by variable.
2026-09-19 21:38:53 +02:00
peter 3987cd4260 release: 0.92.0 (improvement/debt-burn)
gate / frontend (push) Failing after 4m30s
gate / backend (push) Successful in 5m25s
2026-09-19 19:53:37 +02:00
peter 95d9ed4aa6 chore(release): 0.92.0 2026-09-19 19:53:32 +02:00
peter 8d0063d514 fix(ui): the reader's own use of a stand-in stops the restore; the pill's name carries its cap
Round 2 of the deep review:
- A click on, or a key typed into, the focus keeper's stand-in fires no focusin, so a restore still
  polling for its list could take the focus from under the reader. A pointerdown or keydown now
  makes it the reader's own focus.
- The Jump-to pill hid its "Ctrl K" from its name (the voice command "click Ctrl K" still matched
  nothing); the key cap's tooltip now equals its name, so it is not read twice as a description.
- listNav's comment still cited DataTable's two layouts as the reason for its visibility filter.
2026-09-19 19:30:27 +02:00
peter 0ca0cac534 fix(api): a waiting reorder deals from the fresh order; the commit check sees staged writes
Round 2 of the deep review, on round 1's own fixes:
- `ORDER BY position … FOR UPDATE` sorts before it locks, so the reorder that waited got the
  fresh rows in its STALE order and dealt the pins it left out (a tab with Plex off) back to their
  old slots. It locks in id order — one global order, so two reorders cannot deadlock — and sorts
  the fresh rows itself. The race test now sends that partial list and fails unless tab B really
  blocked.
- The request check saw only writes the handler flushed: a `db.delete` left staged, or a flush
  followed by a released savepoint, passed. Staged changes count at the end of the request, and
  only the ROOT transaction's end clears it (a savepoint test drives the dependency directly).
- The `ok` test exempted every call as a serializer, hiding four domain REPORTS (YouTube sync,
  revert, push; Plex sync) without `ok`. They are spread into `{"ok": True, …}`; the exemption is a
  named list of real serializers.
2026-09-19 19:30:27 +02:00
peter cf0c55ba80 fix(a11y): names that carry what they show; the unused-key check stops covering whole files
- The palette launcher's name replaced its visible "Jump to…" (and the key cap's its "Ctrl K"):
  a voice-control "click Jump to" matched nothing (WCAG 2.5.3). Each name contains its text now.
- SegmentedToggle's `title` was only a tooltip on a non-focusable box — the live search's "how
  many per batch" hint reached no keyboard or screen-reader user. It is the group's accessible
  description, standalone and captioned alike.
- The unused-key contract let a template like `card.${status}` cover its whole namespace, which hid
  five dead keys (deleted). Dynamic keys are matched by shape (`editor.*.hint`); a bare namespace
  shape covers nothing, and the audit/quota namespaces are now held both ways by backendIds.
- CLAUDE.md's "FE control" row matches the lint rule it cites; each no-restricted-syntax entry has
  its reason directly above it.
2026-09-19 19:19:43 +02:00
peter fa4f823fc8 fix(ui): Back finds the item again past the focus keeper; nested drags stay in their list
Deep review of the debt burn, each measured in Chrome:
- Back from a sub-view whose focused control the swap removed: the focus keeper put the focus on a
  stand-in (the returning list's first item) and `restoreNavPosition`, finding the focus "already
  somewhere", gave up — Plex show → Back landed on tile 0, not the tile it was opened from. A
  stand-in the keeper placed now counts as untouched, like <body>. The keeper's walk up also stops
  at its container: a slot left with nothing focusable no longer sends the focus to the first
  control of the whole document.
- DataTable enabled both layouts' keyboard lists with the same flag, and `useListNav` reads its ref
  only when the flag changes: after a resize across `md`, the layout that mounted was never
  registered (no j/k) and the one that left stayed registered. Each is enabled with its layout.
- A Plex playlist nests blocks and each show's episodes in one DndContext; closestCenter and the
  keyboard coordinates weighed every droppable, so a movie landed "over" an episode — announced as
  a place in the other list, dropped somewhere else or nowhere. `sameListCollision` and
  `sameListKeyboardCoordinates` keep a drag among its siblings; onDragEnd's episode → block
  resolution is gone with the case.
- dnd-kit reports the item over itself in the pick-up's own tick; announced, "moved to position 1"
  replaced "picked up … use the arrow keys" before a screen reader read it.
2026-09-19 19:19:43 +02:00
peter 884060891e test: a handler that flushes a write and never commits it fails its test
Helpers flush and handlers commit — but every test request shared the test's session, so a
follow-up GET or an assertion read the flushed write and the suite stayed green over a handler
whose commit was missing (production's get_db closes the session: the write is rolled back behind
a 200). The db fixture now hands out `db.info["request"]`: the same session, recording each request
that ends normally with a flushed, uncommitted write, and failing the test on it at teardown. All
28 get_db overrides use it (a convention test keeps it so); the full suite passes under it, and
dropping the commit from add_item turns a playlist test red.
2026-09-19 19:19:30 +02:00
peter 955c8c3445 fix(api): a token hiccup is not "reconnect"; reorders and approvals are one unit each
Deep review of the debt burn:
- `refusal_reason` told the user to reconnect on ANY token-endpoint failure — a 5xx, a 429, a
  misconfigured client — while the grant stayed and the account still read as connected, with no
  connect button. Only `invalid_grant` (the one that drops the grant) means reconnect now.
- `pins.reorder` writes only the rows that change, so two overlapping reorders each skipped rows
  that matched in their own stale snapshot and interleaved into two pins on one slot. It locks the
  user's pins first; the second waits and deals from the first's result.
- Adding a whitelist entry committed the invite and the activation it implies separately, and its
  audit row was written before a new invite had an id (target NULL). One transaction, flushed for
  the id.
- The mutation convention test skipped async routes and dicts returned by name: request-access,
  purge-discovery and the subscription sync answered without `ok`. They do now, and the test
  covers both. The helper-commit check looked only at `_`-names; it reads every non-route
  function, which listed two deliberate committers (upsert_pending_invite, ensure_welcome) — now
  allowlisted with their reasons.
2026-09-19 19:19:29 +02:00
peter 99df44d7fb fix(api): mutations answer ok; route helpers flush and leave the commit to the handler
Two rows of the canonical-patterns table had drifted, found while closing the roadmap audit:

- 63 mutation responses carried no `ok` — `{"unsubscribed": true}`, `{"status": "ok"}`,
  `{"added": 3, …}` and so on. Each now leads with `"ok": true` (additive: every field a client
  reads is still there).
- Twelve route helpers committed on their own. Six were just the end of their handler's work and now
  flush: the playlist item add/remove and Watch later get-or-create, the Plex series/season bulk
  mark, and the invite decision + account activation — which an approval used to land in separate
  transactions. The other six commit on purpose (a token durable before its link is mailed, a
  single-use token burned as its own unit, the Google callback's tail, a background registration
  with its own session, the staged search ingest around YouTube calls), each listed with its reason.

`tests/test_route_conventions.py` holds both rules in code; CLAUDE.md's two rows point at it.
2026-09-19 18:21:06 +02:00
peter 30fbdc52b2 fix(ui): a focused, disabled FocusSafeButton no longer lights up on hover
A FocusSafeButton that holds the focus is disabled by aria-disabled, not the native attribute, so
every hover gated on `:disabled` / `enabled:` alone still fired on it — the glass lift, the primary /
danger / warn dim, and eight `enabled:hover:` chains (saved views, the player's list buttons, the
undo toolbar). Measured on localdev: the Pager's Next, focused at the last page, now keeps its resting
border and shadow under the pointer. The CSS selectors and the chains also exclude
`[aria-disabled="true"]`, and a lint rule refuses a bare `enabled:hover:` from now on.
2026-09-19 18:14:18 +02:00
peter 046ff735cf test: close the last coverage debt the roadmap still listed
- `overflow-clip` gets a safelist anchor in index.css (its only occurrence is a `.ts` constant),
  and the chip-lane contract keeps it there.
- PlexPlayer's Backspace ownership rule is pinned on all three branches: nothing above, a foreign
  layer above (it defers), and its own panel open under a layer (it still closes the panel).
- Removing a playlist item bumps the list's `updated_at`, like the three other item-only writes.
- Seen in a real browser on localdev: an exhausted comfort-shelf tile (an empty pinned playlist) is
  focusable, aria-disabled, and its Shift+F10 menu offers Play off and Remove from shelf.
2026-09-19 18:14:18 +02:00
peter 7afaba0547 feat(playlists): say which lists YouTube's API cannot give us
The module shows fewer lists than youtube.com, and without a word about it that read as a Siftlode
bug (the owner's ask, 2026-09-03). Owner's pick from the S7 mockups (2026-09-19): one quiet line at
the foot of the rail, "Missing a list? Why?", whose Why? opens three facts — YouTube's own Watch
later and History are closed to its API (the Watch later list here is Siftlode's own), only lists
you own sync, and an account with no YouTube channel has no playlists at all.
2026-09-19 17:58:40 +02:00
peter db8ed5a004 feat(ui): a visible way into the command palette
Ctrl/Cmd+K and the shortcut sheet had no entry point but a palette row, behind the other
undiscoverable chord. Owner's pick from the S7 mockups (2026-09-19): a key cap at the end of the
header's search field opens the palette, and a page with no search field shows a small "Jump to…"
pill with the same key cap in its place, so the entry is on every page. The cap reads the chord the
palette row binds from the keymap table (Ctrl K, or ⌘ K on a Mac) and stays out of a phone's
header, where there is no keyboard to press it on.
2026-09-19 17:58:39 +02:00
peter 0ef25a8037 feat(ui): the six native selects become the house controls
Owner's pick from the S7 mockups (2026-09-19): a short fixed set is a SegmentedToggle — the live
search's result count, the table's rows per page, a download profile's mode (on its own row, so the
three worded choices fit a phone) and audio format — and a longer set or the user's own list is a
FilterSelect: the profile's quality (seven heights) and the download dialog's profile.

A lint rule refuses a native <select> from now on (the canonical "FE control" row). The px-frozen
size rules also caught up with a fraction: `text-[9.5px]` had slipped past the integer-only pattern
in the Glass tuner, which is rem now.
2026-09-19 17:53:22 +02:00
peter 17a26894b0 fix(a11y): every sortable list speaks the user's language to a screen reader
dnd-kit's screen-reader text is hard-coded English and names an item by its id ("Draggable item
10357 was dropped", a role of "sortable"). Only the YouTube playlist replaced it; the Plex playlist,
the saved-views list and the side-panel groups read it out under a Hungarian UI — found while
checking the Plex playlist's keyboard reorder on localdev.

`ui/sortableA11y` is now the one place: `useSortableA11y` gives a DndContext its instructions and
announcements (HU/EN, `common.dnd.*`), `useSortableItem` announces the role. The YouTube playlist's
own announcements move onto it, which also fixes them: its lists reorder on drop, so reading the
dragged row's own index said "moved to position 1" at every step and "dropped at position 1"
wherever the row went. The place announced is now the slot the row would land in.

A `local/sortable-a11y` lint rule (with its contract test) refuses a DndContext without
`accessibility` and a direct `useSortable`.
2026-09-19 17:37:21 +02:00
peter 946c8f26db test(plex): drive the playlist's reorder and rollback through a hook
The Plex playlist's reorder is a dnd-kit drag, and neither of its sensors activates under jsdom,
so `commit()`'s failure path could not be reached from a test. The query, the optimistic copy and
the two writes (reorder, remove) move into `usePlexPlaylistEdits`, unchanged, and are driven
directly: a failed reorder is said and the rows go back; an accepted order lands in the detail
cache; a failed removal comes back.

It also settles the `putBack` question that could not be driven before: with the refreshed list
taken in BEFORE the reorder fails, a rollback bound to the render that started the reorder puts a
removed row back, and the deep-equal refetch never clears it. The live cache read the code has now
is what keeps it out; the test fails if it goes back to the closure's `q.data`.
2026-09-19 17:30:31 +02:00
peter 8759b5860d test(be): every dependency override goes through override_dep
22 test files set `app.dependency_overrides[...]` by hand and tore down with a blanket `.clear()`
in a `finally`, which wipes every override rather than the test's own and stops fixtures composing.
They now take the `override_dep` fixture (helpers get it passed in), whose teardown restores each
key's prior value; no clear() carried meaning between two phases of a test. A convention test fails
any test file that writes `dependency_overrides` directly.
2026-09-19 17:26:07 +02:00
peter dc902d2c0e test(fe): cover the palette's content, AltGr, the banner's name, share colours, playlist headers
- AppCommandPalette had no test: modules gated by the account, hidden channels left out, the
  playlist row selecting before it navigates, and the surprise riding setPage's `then`.
- keymap: an Alt/AltGr-modified key (Ctrl+Alt on Windows, a constant on the HU layout) fires no
  bare-key or Mod row.
- Banner: the dismiss button falls back to "Close" rather than going unnamed.
- ShareDialog: a protected link's lit lock and Revoke's danger colour stay on the icon.
- Both playlist headers are rendered now: the YouTube one needed the toolbar slot it portals into.
  Meta line sums known runtimes and treats an ABSENT duration as nothing (null never exercised the
  guard: `3000 + null` is 3000); a one-row list offers no view controls.
2026-09-19 17:23:55 +02:00
peter cedcbb1434 test(i18n): every locale key must be read by the app; 75 dead keys removed
A contract test scans the source (tests excluded) for each key as a whole literal, or under a
dynamic prefix like `plex.sort.${k}`, with plural forms counted through their base key. It found 75
keys that had outlived the code showing them (the old sidebar sort vocabulary, the old login page, a
pager the DataTable replaced, …); they are gone from EN and HU. A key read in a way the scan cannot
see goes in an explicit ALLOWED map with its reason.
2026-09-19 17:15:18 +02:00
peter 4f72b193aa fix(i18n): every label a screen reader or tooltip says is translated
- The Plex playlist rows' "reorder"/"remove", the Plex filter chip's "remove" and the player
  slider's "reset" were English literals; each now names its row or setting in HU/EN.
- The playlist show header's chevron had no accessible name; its decorative thumbnail button left
  the tab order.
- The opt-in Glass tuner panel was English only: every word is translated, and each slider's label
  now names its number box and range.
- New `local/literal-a11y-text` lint rule (with its contract test): a literal word in aria-label,
  title, placeholder, alt and the other spoken/shown attributes fails lint.
2026-09-19 17:12:47 +02:00
peter 7b94e9a651 fix(home): shelf honours the Plex gate; hero unpins its origin; palette asks for a surprise
- A13: with Plex off, the Home shelf and the hero's comfort tier go YouTube-only like the rails
  beside them (owner's call: hide the tiles). The pins are kept; GET /pins still lists them.
- Reorder deals the listed pins into the slots they already hold, so a partial list (hidden Plex
  pins, a stale tab) no longer puts two pins on one position; repeats are ignored.
- A21: a comfort hero carries `pinned_from`, the shelf entry it was resolved from; its menu acts on
  that pin, so a pinned show's next episode offers "Unpin" instead of creating a second pin.
- B12: the palette's "Meglepetés" asks for the surprise tier (the next press leading with it); the
  rotation carries on from there.
- B13: the surprise token names the history entry it was requested for (minted by setPage's new
  `then`, after the push) and is dropped on any other entry, so a navigation that never mounted
  Home cannot replay it later.
2026-09-19 17:07:11 +02:00
peter c6bf84a578 fix(home): a shelf move acts on the order shown, one write at a time
- During a budget switch the rail shows the previous budget's payload
  (keepPreviousData) while the new key is empty; the move read that empty
  cache, found no tile and silently did nothing. It now acts on the order
  the rail rendered.
- A refused move was undone only by a re-read; when that read failed too,
  the shelf kept showing the order the user had just been told was not
  saved. Moves are now written one at a time (the move rows wait, with a
  "saving" note), which makes restoring the shown order correct — the
  overlapping-moves problem that ruled a rollback out cannot occur.
2026-09-19 08:13:07 +02:00
peter 444f9b8166 fix(ui): a control removed by its own action leaves the focus in its slot
The second half of the pending-button sweep: where the pressed control does
not merely disable but LEAVES the page — an invite row gone after Approve,
the inbox's Clear all gone with the last notification, a section gone once
its last row is handled — the browser dropped the focus to <body> and the
next Tab started over at the top.

useFocusKeeper, on the page scroller: when the element holding the focus is
removed and the focus fell to <body>, the first focusable thing in its slot
takes it (what followed, else what preceded, else the nearest ancestor still
on the page). Portaled layers (menus, dialogs) sit outside the scroller and
return focus themselves; listNav's keyboard lists are left alone — their `j`
already goes on from the slot, and a row scrolled out of a virtualized list
must not drag the focus after it.
2026-09-19 08:10:03 +02:00
peter 7a47223a58 fix(ui): no button drops the keyboard focus when its own action disables it
A focused button that turns natively disabled loses the focus to <body> in
Chrome. The app disables the very button that was pressed on ~60 sites — a
mutation pending, a pager at its last page, undo with nothing left, a player
at the end of its queue, a form gone clean — so a keyboard user was thrown
out of the page on every such press.

One primitive instead of a fix per site: ui/FocusSafeButton renders a
focused-and-disabled (or busy) button as aria-disabled with a click guard,
and applies the native attribute again once the focus is elsewhere. It asks
the document at render time as well as listening for focus events, because
a window without focus fires none (measured). ui/Button and ui/form's Switch
are built on it; 57 hand-rolled buttons swapped their tag for it keeping
their classes, and three hand-copies of the rule (the channel page's Block
and Hide, ChannelDiscovery's Subscribe) now pass `busy`. The soft state's
dim look is one rule in index.css.

`local/focus-safe-disabled` refuses a new raw <button disabled>, pinned by
focusSafe.contract.test.ts. lib/plexDetailUi moved to components/: it is
components, and lib may not import the primitive.

Measured on localdev in Chrome: the audit log's Next button keeps the focus
on reaching the last page, reads aria-disabled, and the guarded click does
nothing.
2026-09-19 08:07:25 +02:00
peter bc138e3b37 fix(ui): a disabled menu row says why it is off, under its label
The reason for a disabled menu row lived in a `title` on a natively disabled
button: no pointer events, so no tooltip, and roving focus skips the row —
four translated sentences reached nobody (the Watch later rename/delete
lock, "no other list to move to", the reorder gate). An aria-disabled row was
tried and reverted in E10's review. ActionMenu items now take a `note`,
rendered as a muted line under a disabled row's label: seen, and readable in
a screen reader's browse mode. Only the icon and label dim — row-level
opacity would dim the explanation too. The reorder group states its reason
once, under its first row.
2026-09-19 07:57:20 +02:00
peter df7c3db085 test(ui): the px-box probe does not trip its own rule; tsc lib-safe last()
The pxBox contract test held the literal w-[150px] it probes, which the new
lint rule refuses, and stickyTop's test used Array.at, outside the tsconfig
lib. Both landed because a local lane check piped its exit code into tail.
2026-09-19 07:53:01 +02:00
peter 6fda49369f fix(ui): a sticky table header turns the scroller's top fade off itself; DataTable renders one layout
- "A page with a sticky ColumnHead must declare fadeTop: false" lived in
  prose: a registry flag on three pages plus an App special case for the
  Channels card layout, and the next table page would have re-acquired the
  fade-on-the-header defect with every lane green. ColumnHead now registers
  itself with the enclosing PageScroller (ui/stickyTop) while mounted; the
  registry flag, the App branch and the prop plumbing are gone. Measured on
  localdev: the top mask is 0px under Channels' and Audit's headers, 20px on
  Home and the Feed.
- ui/DataTable mounted its table AND its card list at every width, one
  hidden by CSS: double DOM and render cost, duplicate ids, two keyboard
  copies of every row. It now mounts the one the `md` media query calls for
  (lib/useMediaQuery); the keyboard test asserts one layout at each width.
- The Plex module's content width went px → rem with the rest of the page.
2026-09-19 07:51:33 +02:00
peter 8018cd4685 fix(ui): the platform layer speaks platform words; boxes scale with the text
- PriorityStepper labelled itself with channels.row.* and told every caller
  it orders the Feed; Combobox said plex.loading / plex.filter.moreRefine /
  sidebar.match on the feed sidebar. The generic strings moved to common.*,
  and what a priority MEANS arrives as the stepper's `hint` prop.
  uiI18n.contract.test.ts pins components/ui/ to the platform namespaces —
  the import boundary could never see an i18n key.
- 32 px-frozen box utilities (w-[150px], max-w-[1600px], left-[18px], …)
  became spacing-scale steps (w-37.5, max-w-400, left-4.5): identical at the
  default text size, and now they grow with the Settings slider instead of
  clipping or re-wrapping their own labels. An eslint rule refuses the px
  form beside the older text-[Npx] one, pinned by pxBox.contract.test.ts;
  measured in the browser that every new step is generated at its old px.
- chipLane's import-strip regex ran past a side-effect import into the next
  statement's `from`, hiding a lane between them; it now stops at the
  import's own `;`.
2026-09-19 07:46:33 +02:00
peter 29875b7367 fix(ui): stable shelf-card menu triggers, a named Button by type, true layout comments
- Home's rail handed each card a fresh `() => menu.triggerProps(c)` per
  render, defeating memo(VideoCard) on every tile whenever the menu's own
  state changed. `useItemTriggers` (ui/useContextMenu) keeps one trigger per
  item, index-aligned; tested for identity across renders.
- The Button accessible-name union had one hole: `<Button icon={…} />` with
  no children and no size typechecked into a nameless button. A textful
  button now needs children or a name; pinned with @ts-expect-error.
- comfortPin's header said the pin belongs "on every card and row", which
  the code never meant (an episode, a season, a person are not pin kinds).
  It now says what is pinnable — and the show page's Related strip, whose
  shows ARE pinnable, gets the pin menu it was missing (and a rem width).
- gridColumns / feedView / VirtualGrid claimed layouts 1024–2560 are
  "byte-identical" under the column cap; the small cards are capped at 2560
  with the filter rail collapsed. The comments say so.
2026-09-19 07:40:48 +02:00
peter 7ae5c57b6b fix(i18n): every refusal the API answers is structured and said in the reader's language
~150 HTTPExceptions carried a hand-written English sentence as their detail,
shown verbatim under the Hungarian UI — in the blocking error dialog and on
every inline surface that printed HttpError.detail (sign-in, setup wizard,
change password, the passphrase gate, live search). Four also quoted an
upstream exception's text back to the user.

- app/refusal.py: `refused(status, reason, **params)` → {code: "refused",
  reason, …params}; localizeDetail translates errors.refused.<reason> with the
  params interpolated. "Unknown X" 404s use the existing not_found(what).
- sysconfig.set_value raises InvalidSetting (reason + bounds) instead of an
  English ValueError; the three writers answer it through setting_refused.
- The Plex connection/write/image failures log what the upstream said and
  answer a reason; the search scraper's failure is logged and answered as
  youtube/unavailable (it crashed refusal_reason before).
- HttpError.detail now only ever holds a translated reason: a string detail
  (only the framework's own can remain) is left out, so reading .detail is
  safe everywhere — including destructured, the hole the raw-detail source
  scan could not see. That scan and its five-file exception list are gone.
- Enforced: test_http_details.py fails on any HTTPException with a sentence
  for a detail; the contract test requires every refused reason and every
  not_found "what" in both languages, and no key for one that is gone.
- local/jsx-comment-text: a // inside JSX children renders as text; the rule
  is ten lines in eslint.config.js rather than eslint-plugin-react's ~1800
  lockfile lines.
2026-09-19 07:34:17 +02:00
peter dc4e8b9fcd fix(plex): playback sessions per viewer, bounded in total; a fair share of subtitle passes
HLS sessions were keyed by the item, so a second account opening the same
film killed the first one's session; the displaced player reclaimed at its
playhead and evicted the other in turn. The registry is now keyed by
(viewer, rating_key), segment requests resolve by both, and the session
directory carries the viewer (the orphan sweep still recognises the older
shapes).

Only transcodes counted toward a cap, so remuxes — each a whole segmented
copy of the film on the scratch volume — were bounded by the library size.
Three bounds now apply after every start, each dropping the least recently
used: two sessions per account, the transcode cap, and a new admin knob,
plex_max_sessions (default 8), over every session. A superseded session
request returns at once, since it waits on the killed ffmpeg, so an account
cannot park more request workers than it may hold sessions.

The subtitle extractor's full-pass slots were global and minutes long each;
one account opening subtitles on film after film could hold them all. An
account may now hold three of the six, and still gets windows past that.
2026-09-19 07:16:17 +02:00
peter 6f9ba428d5 fix(playlists): the move's conflict fallback says why it is silent
The house guard requires a silent handler to carry a silent-ok note; the
savepoint fallback added in e74986cc did not, and the gate's pytest lane
caught it.
2026-09-19 07:10:43 +02:00
peter c60359bfed fix(plex): subtitles — a retried full pass resumes, a stalled track is polled slowly
A full pass that ran out of time kept what it reached, but its retry an hour
later started again at 0, so a file too big to read inside the extraction
timeout never completed. The pass now starts where every embedded track's
coverage from the head ends (less the window pre-roll, so a cue across the
join is read whole). The seek needs -avoid_negative_ts on this read as on a
window, measured on a real file: without it the cue running at the start
shifts the whole output.

A partial track whose full pass had failed never reached a settled state, so
the player polled it every 2 s for the rest of the film. The status now says
whether the track is still growing, and a stalled one is polled every 15 s —
still inside the 60 s lookahead that reads the next window before an edge.
2026-09-19 07:07:53 +02:00
peter e6b76e156a refactor(discover): one shelf query per kind, one RAIL_LIMIT, a true rollup comment
The comfort shelf resolved each pinned channel with three queries (the
row, its unwatched count, its newest video) and each playlist with two.
channels_summary gets the count and the newest row per channel as window
functions over one filtered set, playlists_summary reads every pinned list
in one query, and the rows come in one IN() each — so the shelf's cost no
longer grows with its length. A test counts the statements.

RAIL_LIMIT had two definitions held in step by a comment; the Plex Discover
route now imports the domain's. And rollup's "continue the latest-started
episode" was false: it takes the last in-progress one in broadcast order,
which is also what Home's next_episode picks.
2026-09-19 07:03:35 +02:00
peter e74986cc9a fix(playlists): a racing move is not a 500; ownership through owned_or_404 everywhere
move_item checked "already in the target?" and then re-parented without a
lock, so a second tab's add or the scheduler's YouTube merge committing the
same video in between hit uq_playlist_video and answered a raw 500. The
re-parent now runs in a savepoint and a conflict falls back to the
already-present outcome. The target is touched only when the row landed in
it (an untouched list was promoted in the rail's "recently changed" sort),
and _touch stamps the database clock like the mixin's own onupdate.

The canonical-patterns table names owned_or_404 against ad-hoc ownership
helpers; six were left (_own_playlist, _own_playlist_or_404, _own_view,
_own_tag, notifications._owned, downloads._own_job), two of them
hand-rolled with a bare English 404. All call sites now use owned_or_404,
and the download/link/notification 404s are structured like the rest. The
contract test reads every "what is gone" the routes name, and the YouTube
refusal reasons, and requires both languages to name each.
2026-09-19 07:01:18 +02:00
peter 2df45b27d3 fix(plex): a refused finish restores no resume point at the media's own end
The rollback of a refused watched mark guarded against Plex's REPORTED
duration. On a file shorter than its metadata the end sits outside the
finish margin, so playing to the end and having the mark refused stored the
very end as a resume point on an unwatched row: a ~99% bar that reopened in
the last seconds. The media's own ended flag now decides.
2026-09-19 06:58:11 +02:00
peter bacd6aec4f fix(plex): one scrobble per finish, and no resume push lands after it
Two near-end checkpoints arrive together routinely (the 10 s tick and the
debounced seeked flush); both read "not watched" before writing and both
scrobbled, bumping Plex's viewCount twice. The transition is now decided by
the write: an insert that did not conflict, or an update conditional on the
old status.

The finish also popped the push-ordering entry that _resume_push_superseded
reads, so an older resume push still in flight could land at Plex after the
scrobble and revive a resume point on a watched item. It now stamps the
entry instead; the existing eviction bounds the map.
2026-09-19 06:57:20 +02:00
peter fad301e7a4 fix(youtube): a refused YouTube action answers a 422 that says what to do
The two manual syncs (subscriptions import, playlist pull) had no YouTubeError
handler, so a quota, token or transport failure was a raw 500 and the blocking
"Server error" dialog (E9 F1). The other YouTube routes answered either the
upstream's English or a playlist-scoped "youtube_failed".

One helper, routes/_common.youtube_refusal: 422 {code: "youtube", reason} with
the reason sorted by what the user can do — quota (wait for the reset),
reconnect (the token step refused), unavailable (retry). Used by every route
that surfaces a YouTube failure; localizeDetail translates it.

"Reconnect" had no path: a grant Google revoked still read as granted, so
Settings and the wizard offered no connect button while every call failed.
An invalid_grant from the token endpoint now clears the stored grant.
2026-09-19 06:55:40 +02:00
peter 1cfb19bdc7 release: 0.91.0 (bug/prod-uat-2026-09-19)
gate / frontend (push) Failing after 4m0s
gate / backend (push) Successful in 5m23s
2026-09-19 05:56:19 +02:00
peter 90c30d596f chore(release): 0.91.0 2026-09-19 05:56:12 +02:00
peter 7ed6ccf171 fix(feed): the card menu's reset uses the card's own reset string
Two keys for one action had already drifted once; the menu now reads card.resetState like the
card's button.
2026-09-19 05:54:19 +02:00
peter ce30b99327 docs(nav): restoreNavPosition's doc back on restoreNavPosition 2026-09-19 05:54:19 +02:00
peter a7891d03c5 fix(sync): only a refusal about the channel spends its fallback window
A failed connection or token step answered nothing about the channel, yet stamped it: one proxy blip
during an RSS blackout delayed every due channel by a whole window. YouTubeError now names the
endpoint that answered; a failure no endpoint gave, or a spent quota, ends the round and stamps
nothing.
2026-09-19 05:54:19 +02:00
peter 038da5374f test: the boundary stays quiet while reloading for a new build; Back restores Home's whole question 2026-09-19 05:47:17 +02:00
peter 92979fa920 refactor(ui): one rule for resettable, role key and display name; an honest reset label
The card menu's reset read 'Clear watch progress' but also un-hides and un-watches; it now says so.
The resettable rule, the demo-first role key and the display-name fallback each lived in several
components tied by comments; they are shared helpers now, and the menu reuses the card's own
strings instead of duplicate keys.
2026-09-19 05:47:17 +02:00
peter 2d5bd275fa fix(diagnostics): a pivot or Back applies the search text at once
The query read the typing-debounced search, so for 300 ms after a jump it mixed the new view's
filters with the old view's text and fetched rows neither view shows. useDebounced takes a settle key
whose change applies the value immediately.
2026-09-19 05:47:16 +02:00
peter 49fe56b6e7 fix(nav): Back's focus keeps its place among items of the same name
Restoring by name picked the first item of that name, so Back on the log console (repeated
http.request lines) or two same-titled downloads landed on the wrong one. The saved place wins while
it still holds the name; otherwise the nearest item of that name.
2026-09-19 05:47:16 +02:00
peter 1fa6a0bea8 fix(sync): a failed feed-fallback read waits its window out, and spent quota ends the round
A channel whose uploads playlist also failed was never stamped, so a terminated channel was charged
again on every 20-minute poll instead of once per rss_fallback_hours. An upstream quotaExceeded now
stops the round instead of charging every remaining channel the same refusal.
2026-09-19 05:47:16 +02:00
peter 582f20fab6 fix(sync): the stored RSS warning says what the API fallback read
An INFO line outside a request never reaches log_events, so the fallback's
own summary was invisible in the Server log. Its counts ride on the
sync.rss.failed warning, the one line of a poll the console keeps.
Measured on localdev: 375 of 375 dark channels read, 403 new videos.
2026-09-19 05:05:13 +02:00
peter 6cb8231a9f fix(logs): the browser's copy of a polled read is quiet too
The server now logs polled reads at DEBUG, and the browser's copy of each
request line promises the server's level: at INFO the unread-count polls
filled My log within minutes and, at Detailed, the shipped log. The server
marks a line it logged at DEBUG with X-Log-Quiet, so the polled-route list
stays in one place.
2026-09-19 05:03:31 +02:00
peter 6bbf532b70 fix(diagnostics): count the 24 hours from the moment the level was raised
The countdown's clock was the moment Settings mounted, so a level raised
two minutes after opening the page read "24 h 2 m" until the first tick
(measured on prod). The clock is read at render; the interval only
re-renders.
2026-09-19 05:03:31 +02:00
peter 2fa5c8c394 fix(ui): nothing past the right edge of a phone
The Feed's chips + Sort + View row did not fit one line at 375px and pushed
the page 21px wide; the toast stack capped its width by the viewport while
it sits right of the sidebar. The toolbar rows (Feed, Plex browse) wrap,
and the toasts are capped by their host.
2026-09-19 04:54:18 +02:00
peter 373415a2eb feat(account): show the account its own last sign-in
Settings → Account names the latest sign-in on any device (only the
admin's user list did, though the 0.88 notes said otherwise), and the role
in the reader's language instead of the raw `user`.
2026-09-19 04:51:43 +02:00
peter 582c033e1c feat(feed): the card menu carries the card's own actions
Right-click, long-press or Shift+F10 on a feed card offered only the
comfort-shelf pin; watched, saved, hidden and the progress reset were
reachable only by hovering or Tabbing through every card.
2026-09-19 04:51:43 +02:00
peter ef652ee5d8 fix(ui): honest tab titles and a download field that names every site
Signed out, the tab read Home's question over the sign-in form; the legal
pages read just "Siftlode". The download field asked only for a YouTube
link although Facebook and the rest download too.
2026-09-19 04:51:43 +02:00
peter 250e3a9f90 fix(home): Back keeps the question, and the focus finds its card by name
Opening a card left Home, and Back came back to "Doesn't matter" with a
different hero, whose card then dropped out of the Continue rail and moved
the keyboard onto its neighbour. The budget, reroll seed and seen list live
on the history entry now, and a returning list focuses the item by its
name before its place.
2026-09-19 04:51:43 +02:00
peter 6096dd6927 fix(diagnostics): Back from a request's lines returns to the view it came from
Every filter change replaced the history entry, so Back after a request or
job pivot left the console altogether (the 0.90 notes promised otherwise).
A pivot follows a link and is its own entry now; the console reads the
landed entry back on popstate. The other filters still adjust in place.
2026-09-19 04:51:42 +02:00
peter 51e98bfc38 fix(app): reload onto a new build instead of crashing on a missing chunk
A tab left open across a release asked for a lazy chunk the server no
longer has (prod: DownloadDialog 404 an hour after 0.89) and fell to the
crash screen. `vite:preloadError` now reloads the tab once a minute at
most; a second failure is let through to the error boundary, which now
wraps every entry — the public /watch, /privacy and /terms pages crashed
to a blank page.
2026-09-19 04:51:31 +02:00
peter 3ee00fb051 fix(logs): keep polling reads and a channel-less account out of the warnings
The reads the frontend polls (unread counts, /api/me, the download index,
sync status, subtitle progress) are logged at DEBUG: at INFO they were ~95%
of the stored log on prod. A scheduler job failure and a playlist sync
failure carry event codes, and a Google account without a YouTube channel
has an empty playlist list instead of a warning every six hours.
2026-09-19 04:51:31 +02:00
peter 29094f5f51 fix(sync): read a dark RSS feed through the API instead
YouTube's RSS goes dark per channel for hours (prod 2026-09-18/19: 393 of
396 feeds answered 404 for four hours, sticky per channel, so a retry does
not help), and nothing else read a followed channel's new uploads. A
channel whose feed has not answered for `rss_fallback_hours` (admin knob,
default 2, 0 = off) now gets one page of its uploads playlist through the
Data API (1 unit), at most once per window, yielding to the backfill
reserve. Spend is attributed to its own quota action.
2026-09-19 04:51:31 +02:00
peter 16a44dec79 release: 0.90.0 (feature/e22_6-diagnostics-console)
gate / frontend (push) Failing after 4m16s
gate / backend (push) Successful in 5m19s
2026-09-19 03:57:44 +02:00
peter 4e3e68d6e6 chore(release): 0.90.0 2026-09-19 03:57:29 +02:00
peter eb29f22eb7 fix(diagnostics): say which read failed, and let Live restart
- "Load older" is not offered off a loading filter's placeholder rows,
  whose cursor belongs to the previous filter.
- A failed older page, stale rows and missing counts each get their own
  message (HU/EN).
- Live shows as off once a failed read stopped the polling; pressing it
  again reads now.
2026-09-19 03:48:01 +02:00
peter 8f6f89832f fix(diagnostics): honest live console after review
- Live stops polling on a failed read and marks the list as possibly
  stale (with Retry), instead of re-raising the error dialog every tick.
- While live, a refetch reads the newest page only, whatever the cache
  holds (a late "Load older", a revisited filter set).
- Copy is disabled and the list dimmed while a filter change loads.
- Dates follow the UI language; the pivot chip uses the shared .chip.
- Tests: query-string wire test, Copy, every facet's skip key.
2026-09-19 03:42:26 +02:00
peter 1c49870a7a fix(sync): don't try a timed-out feed again
A request that hung for the full timeout says the way out is hanging, not
that the endpoint is flaky; three of them per channel stretched a poll over
a dead egress to about three times its interval.
2026-09-19 03:42:26 +02:00
peter 142593083f feat(diagnostics): walk the log lines from the keyboard; name the console "Server log"
Both log viewers put their lines in a NavGrid: j/k (and the arrows) step
from line to line, and Enter opens one exactly as a click does (the line's
button is its `data-nav-open`). A focused line is named by its code and
message.

The console's nav entry is "Szervernapló" / "Server log", its page title:
"Diagnosztika" was already the Settings tab that holds the user's own log,
and two places with one name read as one place.
2026-09-19 02:02:45 +02:00
peter 8cfab498cf fix(sync): try a channel's feed again before counting it as failed
YouTube's feed endpoint answers the same channel with 404, 500 and 200 in
turn: on prod 182, then 296 of 396 channels failed a poll, and a probe of
24 channels saw 15 fail a first request while 16 had succeeded by the
fifth. A channel that failed once waited for the next poll, 20 minutes
later, so new uploads showed up late.

A feed is now tried up to three times (after 1 s and 2 s) on a transport
error, a 404, a 429 or a 5xx; a refusal such as 403 is not retried. The
last failure is what gets counted. A feed that is really gone costs two
more requests per poll.
2026-09-19 01:58:40 +02:00
peter b5996a0e56 feat(diagnostics): the admin log console
Admin → Diagnostics reads the server's stored log: one list, newest first,
where a line opens in place to its full record (time, source and logger,
account, request and job ids, version, message, attributes, stack). A
request or job id narrows the list to that request or job with every other
filter off, shown as a removable chip. Filters: level group, source,
account, event, period and a free-text search; all of them live in the URL
(`l*` keys, which setPage clears like the Plex and Channels keys), so a
filtered view is a link and Back restores it. "Live" re-reads the newest
page every few seconds while the tab is visible; paging into older rows
turns it off. Copy puts the loaded lines on the clipboard as JSON lines.

The facet counts now honour every filter but their own, as the feed's do:
a request pivot narrows them, and a picked level does not zero its
siblings. A value in use stays offered even when the counts drop it.

"Saját napló" and the console draw a line with one shared component,
`LogLine`, whose phone layout gives the message its own full-width line
(sharing the code's line left it a few characters). The level groups
moved to lib/log.ts for both.

The console's own reads are logged at DEBUG on success, like the
healthcheck: at INFO its live tail filled the log with itself.
2026-09-19 01:54:48 +02:00
peter 5e208353e3 feat(diagnostics): admin query API over the stored log
GET /api/admin/logs pages `log_events` newest first on a (ts, id) keyset:
a browser row's ts is clamped into its consent window, so ts alone is not
unique. Filters AND together: level, source, account (or `none`), event
code (or `none`), request id, job id, a free-text search over the
message, code, request id, stack and attributes (typed text is a literal),
and a time window. Each page names the accounts it mentions.

GET /api/admin/logs/facets counts what the filter chips offer over the
window alone, so a chip row does not shrink as filters are picked.

Admin-only and read-only; reading is not audited (the console's live tail
polls). There is deliberately no "ids after N" tail: several processes
write this table and a batch takes its ids at insert, not at commit, so a
tail keyed on ids could pass a row that commits late.
2026-09-19 01:40:46 +02:00
peter 66dbd0af81 fix(search): typed text is a literal in every LIKE pattern
The feed's channel-name match and the live search's term fold built their
LIKE pattern from the typed text as it was, so `_` and `%` acted as
wildcards: `build_log` also matched `buildXlog`. The term fold also tested
a substring, so a term already present inside a longer word (`cat` in
`category`) was never folded into search_terms. It now compares whole
terms, escaped.

`like_escape` moves from the Plex query module to `app.utils`, beside
`to_tsquery_str`, and every pattern built from typed text goes through it.
2026-09-19 01:33:32 +02:00
peter 59fd068711 release: 0.89.0 (improvement/e22_5-backend-logging)
gate / frontend (push) Failing after 3m28s
gate / backend (push) Successful in 5m1s
2026-09-19 00:05:53 +02:00
peter 8e12eee996 chore(release): 0.89.0 2026-09-19 00:05:47 +02:00
peter 89c77ae6b5 fix(logs): deep-review round 3 — a failure nobody could record is one scrubbed line
- The download and edit failure lines go back after their DB write: a job the
  write could not fail is not a failed job, and the worker's child_exit line
  already says it failed.
- The download child's runner is the last stop before Python's hook: an
  escaping exception is one download.child.failed line with its chain scrubbed,
  then exit 1, instead of a raw stderr traceback carrying the URL.
- A crash the database will not record carries its stack once, and the
  fail-write line has an event code.
- The job's own generic-extractor name is taken out as a literal (formats.hiding),
  so brackets and colons in a direct link's file name no longer leak; the
  pattern stays as the fallback. `Downloading just the video <id>` is covered.
- Tests: scrubbed_stack's cause, suppress, unprintable and text-format paths,
  the image steps' failure kinds, the watchdog error through the real
  _fail_running, exact poster/storyboard messages, ffmpeg skip marker.
2026-09-18 23:44:42 +02:00
peter 0b672ed6e7 fix(logs): deep-review round 2 — a stack names code, not data, and survives a failed write
- A failure keeps its traceback unless it is yt-dlp's or our own StatedFailure (a
  message that is the whole story). A plain RuntimeError, RecursionError or OSError
  is also what a bug of ours raises, and its stack is the only lead.
- The stack is scrubbed link by link (logs.scrubbed_stack): a chained DownloadError
  under a DB error no longer prints yt-dlp's text unscrubbed.
- Engine hide_parameters: a DB error's message no longer carries the title or path
  it was writing.
- The download and edit failure lines go out even when their DB write fails.
- The worker's child_exit/crash/supervisor lines say download.job.failed only when
  the job failed there; a crash the DB will not record no longer ends the thread.
  Stopping a paused child is download.child.stopped, not a second abort.
- log_safe also covers generic ids with spaces, playlist titles and bare ids.
- The ffmpeg image steps log the failure kind, not the argv with its path.
- The consent contract accepts only the body of a plain consented_trace() check,
  and any level that may be TRACE.
- Tests for each; every one mutation-checked.
2026-09-18 22:50:37 +02:00
peter 54d19cc386 fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
  `formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
  file paths, and the job-failed, refused, edit-failed and backfill lines use
  it. Done lines log the asset id, not the title-built path. The PII contract
  now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
  failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
  is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
  set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
  a process's last batch closes the repeat window, so a short-lived download
  child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
  `download.job.failed` with a cause; they left no line at all before. The
  crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
  and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
  not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
  carry reason `transport`. Subscribe recognises a duplicate by reason code
  only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
  directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
  were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
2026-09-18 03:16:15 +02:00
peter 04b58e5c6f feat(logs): give the log the events that no request line already carries
The gap this sprint was scoped against turned out to be smaller than the survey
said, and in a way worth writing down: `log_requests` already records every HTTP
call with its route template, status, duration, request id and user id, and the
audit log already records every admin act on someone else's account. Putting a
line in each of routes/downloads.py's twenty mutating endpoints would have added
a second copy of the first and nothing else. So this instruments what neither of
them can see.

yt-dlp now explains itself to our log. `quiet: True` was doing more than quieting:
with no `logger` its whole account of a download went to the child's stdout, so
"Requested format is not available", a signature-extraction failure or a POT
refusal reached us only as whatever text ended up in the raised exception — the
real reason a download failed was routinely one line the app never kept. Measured
against the pinned yt-dlp 2026.07.04 before wiring it: with `logger` set it calls
`debug` for extraction and progress lines and `warning`/`error` for the rest, and
neither `quiet` nor `no_warnings` gates those calls. `debug` lands at TRACE, being
several lines a second, so the noisy half is the half the user's own consent
gates. The full source URL is trimmed to its host on the way in — it is a row of
someone's download history, and `logs.set_job` already puts the job id on every
one of these lines. The option is set AFTER the adapter merge, like `cookiefile`
and the size cap: a source adapter must not be able to take away the one account
of what went wrong that a failed job leaves behind.

An HLS session now says which pipeline it got (transcode/remux, hw, tonemap,
variants, offset) — everything downstream is about that choice, and it was only
inferable from an ffmpeg command line nobody keeps. A session culled by the
concurrent-transcode cap says so: somebody's playback stops there, the route turns
it into a retryable 404, and "my film keeps rebuffering" had no other explanation
anywhere. The HW→SW fallback keeps its line and gains an event code.

A live search refused by a quota cap says WHICH cap: the 429 is in the request
line either way, but the per-user daily cap is answered by tomorrow and the shared
budget by the admin.

The download and edit job lifecycle lines gain event codes, which is what E22.6's
console filters on — the job id was already a structured field.

tests/test_ytdlp_logging.py; three of its four cases mutation-checked. A fourth
(a `%` in a progress line) survived its mutation — stdlib only applies `%`
formatting to a record that has args — so it is gone rather than kept as a case
nothing can break; the reasoning moved into the comment on `_emit`.
2026-09-18 01:28:52 +02:00
peter 6b0401b922 fix(logs): rule on every swallowed exception in the backend
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.

What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
  degraded to a software transcode with nothing saying why; a failed audio probe
  collapsed a multi-audio file to one track and the player showed no language
  picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
  complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
  out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
  therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
  and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
  stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
  the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
  fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
  own root, leaving every crashed session's segments on disk forever.

Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.

ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.

logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.

tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
2026-09-18 01:20:42 +02:00
peter 0cff964ff2 refactor(logs): name every logger after its module, and hold it there
25 literal names covered 34 modules. plex/client.py, plex/stream.py,
plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves
`siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`;
routes/channels.py called itself `siftlode.api`. The `logger` column is in every
stored row so E22.6's console can filter on it, and so an operator can pin a level
to one part of the system — neither works when six modules share a name.

`logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync.
One name per module, and it cannot drift from the file it sits in.
tests/test_logger_names.py pins the derivation, that everything stays under the
one `siftlode` tree the handler and the tests attach to, and that no module
hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`).
Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic
it carries rather than the module it lives in — it replaced uvicorn's access log
and is what an operator pins when they want request lines and nothing else.

app/scheduler.py called its logger `logger`; every other module calls it `log`.

ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them:
worker.py logged `str(exc)[:120]` where the traceback was the whole question, and
three prints/f-strings that are deliberate now say so on the line. S110 and BLE001
stay off until S3 — 23 handlers are outstanding and each is a judgement about one
of them; enabling the rules first would just mean 23 blanket noqas.
2026-09-18 00:57:57 +02:00
peter dc221338be fix(logs): keep what a person typed out of the backend log lines
The E22 redactor is a safety net over addresses and credential params, and every
one of these call sites was leaning on it to do the whole job. What it does not
cover went in whole, and what it does cover came out as a constant.

Not covered, so a real leak:
- routes/search.py logged the raw search term at ERROR. The privacy policy shipped
  in 0.88.0 says a search string is kept only at Teljes/TRACE, with the user's own
  consent. The ERROR line now carries the shape of the failure (source, candidate
  and collected counts) and the term follows on a TRACE line, which nothing emits
  unless that consent is in force.
- downloads/backfill.py logged the full source URL at INFO — a row of the user's
  download history. Now the asset id and the host.
- youtube/client.py logged the upstream error body, which is prose that quotes the
  request back. Every failing YouTube call now goes through one _upstream_failure()
  that logs the API's `reason` code, keeps the body for the same consented TRACE
  line, and raises a YouTubeError carrying `reason` as a field — callers put that
  error in log lines of their own (routes/channels.py) and in a 422 answered to the
  user, so the body must not travel in it either.

Covered, so merely useless: routes/auth.py named the account by an address that
reached both sinks as `[email]`, and app/email.py printed recipient lists that read
as `['[email]']`. Both now carry the uid (or a count) and a reason code, which is
also the first half of E22.5 S4's failed-sign-in coverage.

routes/channels.py branched on the text of that error message to recognise an
already-existing subscription. It now branches on `exc.reason`, with the substring
check kept beside it for a body that carried no reason at all.

tests/test_log_pii.py pins the call sites and the family: a source-level contract
test walks every log call in app/ and fails on an argument naming user-typed
content unless the line says why it is safe (`# log-pii-ok: <why>`). Eight sites
carry that marker today, each with its reason. Gate 8/8 green; all five cases
mutation-checked.
2026-09-18 00:49:43 +02:00
470 changed files with 26969 additions and 3682 deletions
+54 -6
View File
@@ -1,22 +1,50 @@
# Server-side mirror of the local `siftlode check` gate (E1 S3 / DX-1), so `main` is always green
# independent of any one machine. Two parallel jobs match the two local lanes:
# independent of any one machine. Two jobs match the two local lanes:
# frontend — tsc (3 projects) + eslint + prettier --check + knip + vitest
# backend — ruff + the full pytest suite (pure-logic + the DB-backed lane) against a real Postgres
#
# They are declared side by side but they DO NOT run in parallel: the lab has a single act_runner
# with one slot, so they queue. Measured across runs #86-#91 — the backend job starts at the exact
# second the frontend one ends, every time, and the pair costs ~4 + ~6 = **~10 minutes from the
# push**. Anything that waits on this gate has to budget for the sum, not the max; the ops
# `release` read-back timed out on its first real run for believing otherwise.
#
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
# failed to come up is a hard error, never a silent skip. node 22 matches the prod build image
# (Dockerfile `node:22-alpine`).
# failed to come up is a hard error, never a silent skip.
#
# The frontend container's node major is NOT free: it must equal the prod build image's and the dev
# machine's, because vitest/tsc read the engine's own Intl and web-API implementations and those split
# between majors. `.nvmrc` is the single source of truth and
# `frontend/src/lib/nodeEngine.contract.test.ts` fails this gate if any of them drift apart.
name: gate
on: [push, pull_request]
# Branch pushes only. A release pushes `main` and then the `v<ver>` tag, and a bare `on: [push]`
# treated the tag as a second push of the IDENTICAL commit — every release ran this whole gate
# twice, to the same verdict, for nothing (runs #81-#90 are five such pairs). `branches: ['**']`
# keeps every branch covered and drops tag refs.
#
# It has to be `branches`, NOT the `tags-ignore` this obviously wants to be. Gitea's matchPushEvent
# requires EVERY named filter to match, and each one bails on the wrong ref kind: `tags-ignore`
# starts with `if !refName.IsTag() { break }`, so a filter meant to skip tags rejects every BRANCH
# push and the gate stops running entirely. `branches` bails the same way on a tag, which is the
# direction we want. (Measured in go-gitea v1.27.3 modules/actions/workflows.go, not assumed — it
# differs from GitHub, where `tags-ignore` alone leaves branches unfiltered.)
#
# If that filter were ever wrong the gate would not run at all — which is why `siftlode release`
# now ends by reading the verdict back for the sha it pushed, and says so out loud when there is
# no run for it. A gate that silently stops running is the failure this repo already lived through.
on:
push:
branches: ["**"]
pull_request:
jobs:
frontend:
runs-on: ubuntu-latest
container: node:22-bookworm
container: node:24-bookworm
defaults:
run:
working-directory: frontend
@@ -34,7 +62,11 @@ jobs:
# A combined python+node image: python 3.12 (matches the prod Dockerfile) for the tests, and node
# so the JS actions (checkout) run. actions/setup-python is NOT usable here — the runner can't reach
# GitHub's python-versions download, so it must be baked into the image.
container: nikolaik/python-nodejs:python3.12-nodejs20
#
# This node runs the checkout action, never our code, so it is deliberately OUTSIDE the engine
# contract the frontend job is held to — it tracks `.nvmrc` as hygiene (node 20 went EOL in April
# 2026), but a missing upstream tag here must never be able to block the gate.
container: nikolaik/python-nodejs:python3.12-nodejs24
services:
postgres:
image: postgres:16-alpine
@@ -61,6 +93,22 @@ jobs:
run: python -m ruff check --config backend/ruff.toml backend scripts
- name: ruff format --check
run: python -m ruff format --check --config backend/ruff.toml backend scripts
# The ninth lane (E27 S3). Run from backend/ so it resolves exactly as the local lane does —
# pyrightconfig.json's include/extraPaths are relative to the config, but the baseline is
# found relative to the working directory. New errors fail; the ~200 pre-existing ones are in
# .basedpyright/baseline.json and burn down epic by epic. Matching is per (file, rule, column
# span) plus a count, so a new error of the SAME shape in a file where a sibling was just
# fixed is absorbed — burn the hot files down first.
#
# The second step is not redundant. basedpyright TIGHTENS the baseline in place when the count
# drops, with no --writebaseline, and CI throws its workspace away — so without this a
# burn-down is green here and red on the developer's `siftlode check`, which has the same
# guard. Two gates over one tree must not answer differently.
- name: basedpyright
run: python -m basedpyright --project pyrightconfig.json
working-directory: backend
- name: basedpyright baseline is committed
run: git diff --exit-code -- backend/.basedpyright/baseline.json
- name: pytest
run: python -m pytest
working-directory: backend
+4
View File
@@ -37,3 +37,7 @@ backups/
Thumbs.db
.idea/
.vscode/
# TypeScript incremental build artifacts — machine-local, absolute-path-bearing, and no gate lane
# reads or regenerates them, so a committed one is stale from the commit that adds it.
*.tsbuildinfo
+1
View File
@@ -0,0 +1 @@
24
+1
View File
@@ -0,0 +1 @@
3.12
+171
View File
@@ -7,6 +7,177 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.95.0 — 2026-09-23
A file that is too big to upload says so before it is sent, instead of failing as if the server were down - and the server no longer writes half a gigabyte to disk for someone who never signed in.
### Fixes
- Picking a file over the size limit used to end in "the server couldn't be reached" - after the whole file had been sent. The dialog now checks the size the moment you choose the file, tells you what the limit is, and does not start. If a refusal does come from the server, it reads as a sentence about the size rather than as a network failure.
- A long name on a download no longer breaks it. Anything typed past the length of the field used to fail the whole operation with a server error - on an upload, after the entire file had already transferred. Names are simply shortened now, on every screen where one can be set.
- While the file you picked is too large, the drop zone still lights up when you drag a replacement onto it. It used to stay red, which read as though drag-and-drop had stopped working - in exactly the moment you were trying to fix the problem.
### Internal
- An upload is now read only after the server knows who is sending it. Until now anyone at all could make the server write up to 500 MB to disk and only then be told they were not allowed to upload, because the file arrived before the password was ever checked. It is read a piece at a time, straight to disk, and stops the moment it passes the limit.
- An upload no longer holds a database connection open while the file is transferring. One slow upload used to occupy a connection for its entire duration, and enough of them at once would have starved everything else on the server.
- The parser for the old upload format is gone from the app entirely, along with the size allowance it required.
- Three adversarial review rounds closed the work: 13 findings, then 28, then 8. The last round found almost nothing in the app itself - its findings were in the tests the round before had written, which is why it was the last. Four of its claims were disproved rather than fixed, one of them a proposed change that would have broken emoji in names. Every regression test was proven to fail with its own fix taken out.
## 0.94.0 — 2026-09-22
No new features, on purpose: a release spent entirely on the machinery underneath, so that whole classes of fault stop being possible rather than being fixed one at a time.
### Fixes
- Typing an impossible number into a schedule - 0 minutes, 5000 minutes - used to close the editor and quietly keep the old value, as if it had been saved. The editor now says what the allowed range is, refuses to save until the number is inside it, and hands the keyboard back to the button you opened it from when you leave with Escape.
### Internal
- Every request is now measured against a size limit before anything else reads it - before a password is checked, before an address is even resolved. Until now a stranger could make the app swallow a 32 MB upload and only then be told the password was wrong. Each address carries its own ceiling, including the ones that legitimately receive a film.
- The server's Python is type-checked by the release gate for the first time: about 36 000 lines carried type annotations that nothing had ever verified. The mistakes already there are written down so they can be worked off release by release, and a new one now fails the build.
- The app moved to React 19 and its compiler. The compiler now does, automatically and correctly, the bookkeeping that was written by hand in about 150 places - and hand-written bookkeeping is where most of this year's stale-value and stale-click bugs came from.
- The 45 places where the old code told that same checker to stop looking at a piece of work are gone. Each was either a real bug, work that should never have been in that part of the code, or a genuine exception that now states the rule it depends on in code rather than in a comment.
- The engine versions the app is built and shipped with are now pinned and checked on both sides, the way the JavaScript side was fixed in the previous release.
- Four adversarial review rounds closed the work: 22 findings, then 13, then 6, then 2 - and the last two were in the review's own tests, not in the app. Everything found was fixed here, and every regression test was proven to fail with its own fix taken out.
## 0.93.2 — 2026-09-20
An amount on a Plex film reads the same in every browser.
### Fixes
- A film's budget and revenue could read "$160.0M" in one browser and "$160M" in another, from the same number: asked only for a limit of one decimal, a browser is free to pad up to it or not. The amount is now written the same way everywhere.
### Internal
- The machine that builds a release, the one that checks it and the one it is written on had drifted onto three different versions of the JavaScript engine — the reason those checks quietly disagreed for a month. They are held to one version now, and a check fails the moment any of them moves away from the others.
- Those checks no longer run twice over the same code on every release, and their verdict is read back as part of releasing instead of waiting on a page nobody opens.
## 0.93.1 — 2026-09-20
The report composer's preview works again.
### Fixes
- Looking at a report before sending it said "the issue tracker isn't answering" — every time, for everyone. The preview asked the tracker to resolve links against the repository, and a report always carries something that looks like a commit hash: an id of the tab, of a request. On a tracker older than Gitea 1.27.2 that request does not fail, it stops the tracker — so the preview now asks for the rendering without the repository, which shows the same tables, lists and code, and can no longer take a tracker down. Sending a report was never affected.
## 0.93.0 — 2026-09-20
A way to report a fault from inside the app: record what you did, mark the steps, attach screenshots — and the app writes the report for you, with the logs of exactly that session attached.
### Features
- Start a recording (the command palette, or Settings → Diagnostics) and a pill follows you: a timer, “Step” to mark what you just did, a screenshot button, and Stop. Everything you and the server do while it runs is collected under that one session, and nothing else is.
- Stop opens the composer: a drafted title and description, the steps you marked interleaved with the pages you visited, a table of your browser and the app's version, and the attachments — each one reviewable, each one removable.
- The log lines that would be sent can be read before they go, line by line, and any of them left out.
- Screenshots come from a paste, a drop, a file, or capturing the tab — up to eight, and any of them can be dropped again before sending.
- Send opens an issue on the project's tracker with the report and its logs attached, and the preview shows it exactly as the tracker will render it. Five reports an hour.
- Without a tracker configured — or whenever you would rather keep it — the same report downloads as a zip instead.
- An administrator sets the tracker up in Settings → System (address, repository, token) and can test the connection there. The server refuses to send to a public repository: a report carries its reporter's log.
### Fixes
- The Plex player keeps the keyboard inside itself: Tab no longer walks out of a playing film into the page behind it, not even while its own settings panel is open.
- A report cannot be sent twice at once, from any tab: while one send runs the others say so, and a send that fails half-way retries only what is missing — never a second issue.
- A recording ends by itself after an hour, and both Stop and that hour return your diagnostics level to whatever it was before you started.
### Internal
- A session's logs and screenshots are purged with the session after 7 days, like every other diagnostic line, and another account's session id is ignored rather than refused.
- Four adversarial review rounds at the epic's end: 80 findings fixed across them, every regression test proven to fail with its own fix removed, and the keyboard and focus work measured in a real browser rather than argued.
## 0.92.0 — 2026-09-19
A debt burn: every item the roadmap had deferred is done or dropped — Plex playback per viewer, refusals said in your language, controls that keep the keyboard focus, and a screen reader that hears Hungarian.
### Features
- The command palette has a visible way in: a Ctrl K key cap in the header's search field, and a “Jump to…” pill on pages without one.
- The Playlists rail says, under “Missing a list? Why?”, which lists YouTube's API does not hand out (Watch later, History, lists saved from other channels).
- The last native drop-downs are the app's own controls: the live search's result count, rows per page, a download format's content, audio format and quality, and the download dialog's format.
- Plex plays one session per viewer, so two accounts can watch the same film at once; an administrator sets the ceiling on concurrent sessions (Configuration → Plex).
### Fixes
- A YouTube action YouTube refuses says what to do — wait for tomorrow's quota, reconnect the account, or try again later — instead of a server error; a passing Google hiccup no longer asks you to reconnect.
- Every refusal the server gives is said in your language, and every label a screen reader or tooltip reads is translated.
- Dragging to reorder a list is announced in your language and with the right position, and a Plex playlist keeps a drag within the list it started in.
- A button that turns itself off, or a row that removes itself, leaves the keyboard focus where it was; Back from a Plex title returns the focus to the tile you opened, even after using a button there.
- With Plex turned off, the Home shelf shows your YouTube pins only, and reordering it leaves the hidden Plex pins in their places; two reorders at once can no longer put two pins on one slot.
- The Home hero from a pinned show offers to unpin the show, and “Surprise me” from the command palette really asks for a surprise.
- A finished Plex title scrobbles once and gets no resume point pushed after it; a subtitle pass that stalled resumes instead of starting over.
- Tables keep their j/k keys after the window is resized across the phone breakpoint, and boxes grow with the text-size setting.
- Whitelisting an address and activating its account happen in one step, and the audit log names the entry.
### Internal
- Every mutation answers `ok`, and route helpers leave the commit to the request — both held by tests; a request that leaves a write uncommitted now fails its test.
- New lint rules and contracts: no native select, no hover that ignores aria-disabled, no untranslated accessible text, no unused translation key.
- Two adversarial review rounds at the epic's end: 23 findings, all fixed, each regression test proven to fail with its own fix removed.
## 0.91.0 — 2026-09-19
What a test run on the live instance turned up, fixed: new videos keep arriving while YouTube's feeds are down, an open tab survives a release, and Back returns to what you left.
### Features
- A card's menu (right-click or long press) now offers the card's own actions — watched, save, hide, reset — beside the comfort-shelf pin.
- Settings → Account shows your role and your last sign-in. 0.88 announced the last sign-in in your account panel; it is actually there now.
### Fixes
- When YouTube's feed for a channel stays dark for hours, as it did for 393 of 396 channels on 18–19 September, the channel's newest uploads are read through the YouTube API instead, at most once every 2 hours per channel (adjustable by an administrator, 0 turns it off).
- A tab left open across a release no longer falls to the crash screen when it opens something it had not loaded yet: it reloads onto the new version once.
- Back from a request's lines in the Server log returns to the view the pivot was made from, search text included, as 0.90 promised.
- Back to Home keeps the question you asked (the time budget and the hero you rerolled to), and the keyboard lands on the card you opened, even when the rail's cards moved or several share a name.
- Browser tabs are named after the page again: signed out, and on the privacy and terms pages.
- On a phone, the feed and Plex toolbars wrap instead of running past the edge, and so do notifications.
- The countdown of a raised diagnostics level counts from the moment it was raised, not from when the page was opened.
- The download box's placeholder names every site it takes, Facebook included.
### Internal
- The screens' own polling reads (the unread badges, the download list, the scheduler) are logged at debug level in the server's log and in the browser's copy of it, where they made up about 95% of the lines.
- An account with no YouTube channel no longer writes a warning every six hours, and a scheduler job that fails carries an event code.
- Two adversarial review rounds: 15 findings raised, 11 survived, all fixed, and every regression test was proven to fail with its own fix removed.
## 0.90.0 — 2026-09-19
The server's log gets a screen of its own: an admin can now read what the API, the downloader and the browsers wrote down, filter it, and follow one request or job from start to end.
### Features
- Admin → Server log: the stored log, newest first, with a search, level chips, and source, account, event and period filters. A line opens in place to its full record; its request id or job id narrows the list to everything that request or job wrote.
- Filters live in the address bar, so a filtered view is a link, and Back brings the previous view back.
- Live follows the newest lines every few seconds; Copy puts the loaded lines on the clipboard, oldest first. A failed read says so instead of quietly freezing the list.
- Both log viewers (Server log and My log) can be walked with j/k and a line opened with Enter.
### Fixes
- New videos no longer arrive late from channels whose YouTube feed answers 404 or 500 now and then: a feed is tried again before the channel counts as failed.
- A % or _ typed into a search is searched for as itself, not as a wildcard.
### Internal
- The console's own reads are logged at debug level, so watching the log does not fill it.
- Two adversarial review rounds: 19 findings raised, 12 survived, all fixed, and every regression test was proven to fail with its own fix removed.
## 0.89.0 — 2026-09-19
A release for the log rather than for the screen: what the server writes down when something fails is now complete, said once, and free of what you downloaded or searched for.
### Fixes
- A download that failed because the server lost its database for a moment no longer shows the video's title and file path inside the database's own error message.
### Internal
- No log line carries what a person typed, watched or downloaded: search terms, titles, channel names, file paths, links and video ids are taken out — also from yt-dlp's and ffmpeg's own text, from error stacks, and from database errors. The one exception is Full diagnostics, and only under the user's own consent.
- Every one of the backend's 114 swallowed exceptions was ruled on: 44 now leave a line with an event code, the other 70 say in the code why they need none — and a test keeps it that way. A failure is logged once, with its stack when it is our own bug, and a stalled or killed download no longer fails without a line.
- yt-dlp's own explanations (“Requested format is not available”, a sign-in wall) reach the log instead of a stdout nobody read, and a burst of identical lines is summarised instead of flooding the stored log.
- Three adversarial review rounds: 49 findings raised, 42 survived three independent attempts to refute them, all fixed, and every regression test was proven to fail with its own fix removed.
## 0.88.0 — 2026-09-18
When something goes wrong, you can now turn on detailed logging for yourself for 24 hours — and read your own log on the spot, before any of it is sent anywhere. Siftlode also has a privacy policy and terms at last, in both languages, saying exactly what is recorded and for how long.
+35 -29
View File
@@ -7,17 +7,18 @@ worse than no rule — if you change how the code works, change this file in the
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a download center). It is a
**modular monolith**: one Postgres, one FastAPI process (uvicorn + APScheduler), one worker process
(yt-dlp/ffmpeg), one PO-token sidecar. React 18 + react-query frontend with a router-free navigation
provider.
(yt-dlp/ffmpeg), one PO-token sidecar. React 19 (with the React Compiler) + react-query frontend, with a
router-free navigation provider.
## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + typecheck (3 tsc projects) + eslint + prettier
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + typecheck (3 tsc projects) + eslint + prettier
--check + knip + vitest + pytest**. It runs on every `publish`, which **refuses to build on failure — there is deliberately no
`--skip` flag.** Run it and get it green **before** any merge, release, or handing work to review.
- Full gate (agents with the private ops dispatcher): `siftlode check`.
- Directly, without the dispatcher: `cd frontend && npm run typecheck && npm run lint && npm run knip && npm run test`, plus backend `ruff check backend` + `ruff format --check backend` and `pytest` (the DB-lane needs Postgres).
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` and burn down epic by epic. Fixed some? Regenerate with `--writebaseline`.
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
## Workflow
@@ -56,9 +57,9 @@ how it gets built. Concretely:
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`,
`security`, `logs`, `logsink`, `diagnostics`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `bodylimit`, `progress`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → module
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
@@ -74,29 +75,34 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
## Canonical patterns — when you have a choice, pick THIS
| Area | Use | Not |
| ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------ |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`) | ad-hoc `{"unsubscribed"}`/`{"cleared": n}`/echo |
| Transaction commit | the handler/job-wrapper commits; helpers `flush` | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE action button | `ui/Button` — pick a `SIZES` rung | a hand-rolled `<button className="px-3 py-2 …">` |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| Area | Use | Not |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE action button | `ui/Button` — pick a `SIZES` rung | a hand-rolled `<button className="px-3 py-2 …">` |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE control | `SegmentedToggle`; a long or user's list: `FilterSelect` | a native `<select>` (lint-enforced) |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+), else a latest-value ref | an excused dep list asserting the same thing in prose |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
## House rules that bite (learned from real bugs)
+3 -1
View File
@@ -10,7 +10,9 @@ ARG GIT_SHA=unknown
ARG BUILD_DATE=
# Stage 1: build the frontend SPA
FROM node:22-alpine AS frontend
# The major is pinned to `.nvmrc` and held there by frontend/src/lib/nodeEngine.contract.test.ts:
# this image, the CI gate's image and the dev machine must be ONE engine (see that test for why).
FROM node:24-alpine AS frontend
WORKDIR /fe
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
COPY frontend/package.json frontend/package-lock.json ./
+1 -1
View File
@@ -1 +1 @@
0.88.0
0.95.0
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,30 @@
"""add channels.feed_fallback_at — when a channel's uploads were last read through the API because
its RSS feed would not answer
Revision ID: 0088_channels_feed_fallback_at
Revises: 0087_users_diagnostics_level
Create Date: 2026-09-19
YouTube's RSS feeds go dark for hours at a time, per channel (2026-09-18/19: 393 of 396 answered 404
for four hours; the same channel answers 404 to every retry). The RSS poll then reads the newest
uploads through the Data API instead, at a quota cost, and at most once per `rss_fallback_hours` per
channel: this column is that clock. NULL = never needed it.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0088_channels_feed_fallback_at"
down_revision: str | None = "0087_users_diagnostics_level"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"channels", sa.Column("feed_fallback_at", sa.DateTime(timezone=True), nullable=True)
)
def downgrade() -> None:
op.drop_column("channels", "feed_fallback_at")
@@ -0,0 +1,81 @@
"""add debug_sessions + debug_screenshots, and index log_events.debug_session_id (E16)
Revision ID: 0089_debug_sessions
Revises: 0088_channels_feed_fallback_at
Create Date: 2026-09-19
A debug session is one recording: Start → reproduce → Stop (app/debugsession.py). Every log line
of that user's requests while it runs carries its id in `log_events.debug_session_id` — the column
E22 created and left NULL — and the issue it ends in is built from exactly those rows, so the column
gets its index now. Partial: nearly every row has no session, and an index of NULLs would be most
of the table for nothing.
At most one OPEN session per user, held by a partial unique index rather than by the start handler
alone: two tabs pressing Start at once would otherwise both pass a "none open" check.
Screenshots are rows, not files: bounded (size and count per session, app/debugsession.py), gone
with their session by cascade, and a self-host has no file volume to configure for them.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0089_debug_sessions"
down_revision: str | None = "0088_channels_feed_fallback_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"debug_sessions",
sa.Column("id", sa.String(16), primary_key=True),
sa.Column(
"user_id",
sa.Integer,
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("level", sa.String(8), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("until", sa.DateTime(timezone=True), nullable=False),
sa.Column("ended_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("ended_reason", sa.String(16), nullable=True),
sa.Column("issue_url", sa.String(512), nullable=True),
)
op.create_index(
"uq_debug_sessions_open_per_user",
"debug_sessions",
["user_id"],
unique=True,
postgresql_where=sa.text("ended_at IS NULL"),
)
op.create_table(
"debug_screenshots",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"session_id",
sa.String(16),
sa.ForeignKey("debug_sessions.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("mime", sa.String(16), nullable=False),
sa.Column("size", sa.Integer, nullable=False),
sa.Column("data", sa.LargeBinary, nullable=False),
)
op.create_index(
"ix_log_events_debug_session",
"log_events",
["debug_session_id"],
postgresql_where=sa.text("debug_session_id IS NOT NULL"),
)
def downgrade() -> None:
op.drop_index("ix_log_events_debug_session", table_name="log_events")
op.drop_table("debug_screenshots")
op.drop_index("uq_debug_sessions_open_per_user", table_name="debug_sessions")
op.drop_table("debug_sessions")
@@ -0,0 +1,27 @@
"""add debug_sessions.draft — the issue being written from a session (E16 S3)
Revision ID: 0090_debug_session_draft
Revises: 0089_debug_sessions
Create Date: 2026-09-19
The composer saves as it goes: the title, the rendered markdown body that an export or an issue
carries, the log rows the user took out, and the form's own fields so reopening it later (Settings →
Diagnosztika lists past sessions) shows what was written. NULL until the composer first saves.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0090_debug_session_draft"
down_revision: str | None = "0089_debug_sessions"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("debug_sessions", sa.Column("draft", JSONB, nullable=True))
def downgrade() -> None:
op.drop_column("debug_sessions", "draft")
+5 -4
View File
@@ -13,16 +13,17 @@ Three layers, by *who* runs the code — not by feature:
response models, status-code mapping, serialization. It owns logic that runs *only* on a request.
Business logic that a request happens to trigger does **not** live here if anything else runs it too.
2. **`plex/`, `downloads/`, `sync/` — domain logic shared by more than one caller.** Anything the
2. **`plex/`, `downloads/`, `sync/`, `issues/` — domain logic shared by more than one caller.** Anything the
**scheduler or the worker** runs (not just a request) lives in one of these domain packages, so the
route, the worker, and the scheduler share one definition. Examples from this codebase:
`downloads/service.py` owns enqueue + the job state transitions (routes call it); `plex/sync.py` is
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline.
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline;
`issues/bundle.py` is the one definition of a bug report's files, for the zip export and the tracker.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table) and `diagnostics` (a user's consented log
level), … are single-file primitives many layers import. They depend on nothing above them (a core
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level) and `debugsession` (a recording that tags its user's lines, E16), … are single-file primitives many layers import. They depend on nothing above them (a core
module must never import a `routes/…` module).
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
+3
View File
@@ -58,6 +58,9 @@ class AuditAction:
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
# The log itself was cleared (a single row survives the clear as tamper-evidence)
AUDIT_CLEAR = "audit_clear"
# E16: a user sent a debug session's issue to the tracker — the one act that takes their log
# off this instance, so it is recorded (target = the session, summary = the issue's URL).
ISSUE_CREATE = "issue_create"
def record(
+29 -14
View File
@@ -1,12 +1,10 @@
import logging
import httpx
from fastapi import BackgroundTasks, Depends, HTTPException, Request
from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from starlette.requests import HTTPConnection
from app import diagnostics
from app import debugsession
from app import email as email_mod
from app import logs
from app import mailtext
@@ -16,6 +14,7 @@ from app.db import get_db
from app.models import Invite, LogEvent, User
from app.security import decrypt_optional
from app.utils import canonical_email
from app.refusal import refused
PASSWORD_MIN_LEN = 10
@@ -39,7 +38,7 @@ READ_SCOPES = f"{BASE_SCOPES} {READ_SCOPE}"
WRITE_SCOPES = f"{BASE_SCOPES} {READ_SCOPE} {WRITE_SCOPE}"
_YOUTUBE_SCOPES = frozenset({READ_SCOPE, WRITE_SCOPE})
log = logging.getLogger("siftlode.auth")
log = logs.get_logger(__name__)
def google_enabled(db: Session) -> bool:
@@ -222,7 +221,7 @@ def revoke_google_token(token: str | None) -> None:
timeout=10,
)
log.info("Google token revoke: status=%s", resp.status_code)
except Exception:
except Exception: # noqa: BLE001 — a revoke we cannot make must not fail the sign-out
log.warning("Google token revoke failed", exc_info=True)
@@ -305,6 +304,13 @@ def resolved_user_id(conn: HTTPConnection, *, strict: bool = True) -> tuple[int
try:
hid = int(hdr)
except ValueError:
# Under `strict` this becomes a 401 and the tab authenticates as nobody, which reads
# in the browser exactly like a session that expired. The value itself stays out —
# it came from a request, and `wallet` says which accounts were actually on offer.
log.debug(
"the active-account header could not be read as an id",
extra=logs.event("auth.account_header.invalid", accounts=len(wallet)),
)
hid = None
if hid is not None and hid in wallet:
return hid, hid == default_id
@@ -336,19 +342,19 @@ def current_user(request: Request, db: Session = Depends(get_db)) -> User:
def _session_user(request: Request, db: Session, user_id: int | None, is_default: bool) -> User:
if not user_id:
raise HTTPException(status_code=401, detail="Not authenticated")
raise refused(401, "not_authenticated")
user = db.get(User, user_id)
if user is None or not user.is_active or user.is_suspended:
# Suspended/deactivated mid-session → end it so the block takes effect at once.
_end_account_session(request, user_id, is_default)
raise HTTPException(status_code=401, detail="Not authenticated")
raise refused(401, "not_authenticated")
# SA4: reject a cookie whose recorded epoch is behind the account's current one (revoked by a
# password reset/change or "log out everywhere"). A cookie with no recorded epoch (pre-SA4, or an
# account signed in before this shipped) is treated as epoch 0 — so the FIRST bump revokes it too.
epochs = request.session.get("epochs") or {}
if int(epochs.get(str(user_id), 0)) != (user.session_epoch or 0):
_end_account_session(request, user_id, is_default)
raise HTTPException(status_code=401, detail="Not authenticated")
raise refused(401, "not_authenticated")
# Always keep the session default in the switchable list — covers sessions created before
# multi-session existed (their account_ids was never seeded), so the switcher isn't empty.
default_id = request.session.get("user_id")
@@ -361,7 +367,11 @@ def _session_user(request: Request, db: Session, user_id: int | None, is_default
# session middleware and this task's context. `request.state` is shared across middleware layers
# (it lives on the ASGI scope), so the access-log line can read it back. The user's consented
# diagnostics level rides along: their requests are logged — and stored — in that detail (S5).
logs.set_user(user.id, diagnostics.effective_level(user), user.diagnostics_until)
# So does a debug session the request names (E16): only the caller's own open one counts, and a
# header naming anything else is ignored rather than refused — a stale tab must keep working.
session = debugsession.from_header(db, user, request.headers.get(debugsession.HEADER))
level, until = debugsession.level_for(user, session)
logs.set_user(user.id, level, until, session.id if session else None)
request.state.user_id = user.id
return user
@@ -376,12 +386,20 @@ def optional_current_user(request: Request, db: Session = Depends(get_db)) -> Us
user_id, is_default = resolved_user_id(request, strict=False)
try:
return _session_user(request, db, user_id, is_default)
# silent-ok: this is the anonymous case as much as the refused one — every landing page hit
# comes through here. The second attempt below is the handling, and it reports.
except HTTPException:
if is_default:
return None
try:
return _session_user(request, db, *resolved_user_id(request, strict=False))
except HTTPException:
# Both the pinned account AND the default one were refused, so the tab is signed out
# without being told why — and it got here from a session that named a real account.
log.info(
"a session was resolved to nobody: neither the named nor the default account held",
extra=logs.event("auth.session.probe_rejected", user_id=user_id),
)
return None
@@ -405,10 +423,7 @@ def require_write_scope(user: User = Depends(require_human)) -> User:
checks that had drifted into three different 403 messages across channels/playlists. Layers on
`require_human` because writing to YouTube needs a real, non-demo identity."""
if not has_write_scope(user):
raise HTTPException(
status_code=403,
detail="Enable YouTube editing in Settings first.",
)
raise refused(403, "youtube_editing_off")
return user
@@ -416,7 +431,7 @@ def admin_user(user: User = Depends(current_user)) -> User:
"""Dependency: require the signed-in user to be an admin. Single source of truth for the
admin gate — every admin route/action depends on this rather than re-checking the role."""
if user.role != "admin":
raise HTTPException(status_code=403, detail="Admin only")
raise refused(403, "admin_only")
return user
+231
View File
@@ -0,0 +1,231 @@
"""The global request-body cap (SEC-20).
**The hole.** FastAPI reads a declared body IN FULL before auth or any other dependency runs — the
body parameter has to be parsed to be passed in. So `POST /auth/password-login` drains a 32 MiB
upload from an anonymous caller and only then answers 401. 70 of the 72 body-declaring routes behave
the same way: one unauthenticated client can spend the process's memory and time at will.
**Why a middleware and not `capped_body` on every route.** `routes/_common.py:capped_body` is the
right tool where a route genuinely takes bulk bytes, and the two routes that use it use it
correctly. But putting it on all 70 others would (a) fight the canonical "Request body → a Pydantic
model" row, since capping means taking the body as raw bytes and parsing it by hand, and (b) leave
the NEXT route added uncovered, because nothing would remind its author. A default that has to be
opted into is not a default. This module inverts it: the cap applies to everything, and the few
routes that need more opt OUT, in one list, next to the code that reads it.
**Why raw ASGI and not `@app.middleware("http")`.** Starlette's BaseHTTPMiddleware cannot wrap
`receive`, so it can only read `Content-Length` — which a client may omit entirely (chunked) or
simply understate. The bytes have to be counted as they arrive, which is the same rule `capped_body`
already applies per-route, and counting needs the receive channel.
**Why the refusal is byte-identical to `capped_body`'s.** Two shapes for one fact is how the
`cancelled`/`canceled` bug happened. The 413 body here is built through the same `JSONResponse`
FastAPI renders `HTTPException(413, {"code": "too_large"})` with — not a hand-rolled `json.dumps`,
whose default separators differ by two bytes — and `tests/test_body_limit.py` compares the RAW BYTES
of the two against each other rather than against a literal, so they cannot drift apart. The same
reasoning decides where the middleware is registered (`main.py`): just inside CORS, so this 413
carries the same headers `capped_body`'s does.
**No admin knob, deliberately.** The ceiling is a constant, not a `sysconfig` value: an attacker-
facing limit that an admin can raise from the UI is a limit that can be raised by whoever gets the
admin session, and nobody has asked to tune it. If a self-hoster ever needs a different ceiling it
belongs in `config.py` (env, set before the process starts), not in the database.
"""
import re
from starlette.responses import JSONResponse
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from app import debugsession
from app.config import settings
# 1 MiB. The largest body any route on the default ceiling legitimately takes is 512 KiB (the client
# log sink, `routes/diagnostics.py:BODY_MAX_BYTES`), so this leaves 2x headroom — a ceiling that
# refuses a legitimate request would be a worse bug than the one this module fixes. That ratio is
# the load-bearing fact, so `test_body_limit.py` holds it rather than the figure above.
DEFAULT_MAX_BODY_BYTES = 1024 * 1024
# The routes that need MORE than the default, as (method, path TEMPLATE) → their own ceiling. The
# template, not a concrete path, so the list reads like the route decorators it mirrors and
# `test_body_limit.py` can assert that each one still resolves to a real route: a renamed route
# fails the gate instead of silently losing its allowance in prod.
#
# These are RAISED CEILINGS, NOT EXEMPTIONS, and the difference is the whole point. An exemption
# would mean "this route is counted by something else" — which is only true when the route's own
# limit applies to the bytes as they ARRIVE. That, in turn, is only true of a route with NO DECLARED
# BODY PARAMETER: FastAPI parses and spools a declared body BEFORE the first dependency runs, so a
# route that declares one has already stored the bytes by the time its own cap — or its auth — gets
# a say.
#
# Both listed routes now satisfy that. The screenshot route always did (`capped_body` streams it,
# and the owner check resolves first). The upload route did NOT until P1: it declared
# `file: UploadFile = File(...)`, so an anonymous caller could make the process write the whole
# per-file allowance to temp before `require_human` refused them — this middleware bounded that
# number but could not stop the writing. It now reads its own body from a dependency declared after
# the auth one, so its in-handler cap measures bytes as they arrive, like everything else here.
#
# That rule is no longer only written here: `test_body_limit.py` parses the routes and fails the
# gate if any route in this dict declares a body parameter. A comment could not have stopped the
# defect it describes; the test can.
#
# A route belongs here ONLY IF ITS OWN CAP IS LARGER THAN THE DEFAULT. That is why
# `POST /api/diagnostics/events` is absent: its `capped_body` limit is 512 KiB, already below the
# default, so listing it could only widen what it accepts.
ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# Ingesting a local file. The body IS the file since P1 — no multipart envelope to allow for —
# so this ceiling and the route's own cap count exactly the same bytes. `sysconfig`'s
# `download_upload_max_bytes` is bounded by this same constant (see sysconfig.SPECS), so an
# admin can only ever LOWER the route's cap below the edge, never raise it past one.
#
# WHICH LAYER ANSWERS, measured rather than assumed (deep-review round 1):
# * knob LOWERED (an override below the default) — the route's cap is the smaller number, so
# `capped_stream_to` trips first and answers 422 `upload_too_large` carrying the admin's own
# figure, after draining the rest of the body so the client can read it.
# * knob at the DEFAULT — the two numbers are EQUAL, and the edge always wins: a declared
# Content-Length over the cap is refused here before a byte is read, and an undeclared body
# trips this counter on the same chunk the route would have tripped on. The route's 422 is
# unreachable in that configuration, and no amount of headroom would change it — the
# declared-length check fires first for any body big enough to matter, and the route's drain
# pushes the count past the edge for the rest.
#
# So this 413 is a refusal a real user meets on a default install, not a theoretical edge, and
# it has to be as readable as the route's own: the client localizes `too_large` (`errors.tooLarge`)
# exactly as it does `upload_too_large`, and the upload dialog refuses an over-cap file before
# sending it at all, off the `upload_max_bytes` that `GET /api/downloads/usage` reports.
("POST", "/api/downloads/upload"): settings.download_upload_max_bytes,
("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
}
# The refusal, byte-for-byte what `capped_body`'s `HTTPException(413, {"code": "too_large"})` renders
# as — built through the same JSONResponse so the separators match too, and asserted against a real
# route's raw bytes in `test_body_limit.py` rather than against a literal here.
TOO_LARGE_BODY = JSONResponse({"detail": {"code": "too_large"}}, status_code=413).body
_PATH_PARAM = re.compile(r"\{[^/{}]+\}")
def template_to_regex(template: str) -> re.Pattern[str]:
"""`/api/debug-sessions/{sid}/screenshots` → a regex matching one concrete path.
A path parameter matches `[^/]+` — one segment, never across a `/`, so `{sid}` cannot swallow a
deeper path and hand an exemption to a route that was never granted one.
"""
literals = [re.escape(part) for part in _PATH_PARAM.split(template)]
return re.compile("^" + "[^/]+".join(literals) + "$")
def _declared_length(scope: Scope) -> int | None:
"""The `Content-Length` header as an int, or None when absent or unparseable (a chunked body has
none, and a malformed one is not a number we may trust — both fall through to counting)."""
for name, value in scope.get("headers", ()):
if name == b"content-length":
try:
return int(value)
except ValueError:
# silent-ok: a malformed Content-Length is the caller's doing, not a fault of ours,
# and it is about to be answered anyway — returning None drops the request into the
# counting path, which refuses on the real bytes. Logging it would hand an anonymous
# client a free line in our log for every request it cares to malform.
return None
return None
async def _send_too_large(send: Send) -> None:
await send(
{
"type": "http.response.start",
"status": 413,
"headers": [
(b"content-type", b"application/json"),
(b"content-length", str(len(TOO_LARGE_BODY)).encode()),
],
}
)
await send({"type": "http.response.body", "body": TOO_LARGE_BODY})
class BodyLimitMiddleware:
"""Refuses any request body over its route's ceiling, before the application sees it.
Registered just inside CORS in `main.py` — so the 413 is decorated like every other response
and a browser can actually read it — but outside the session, the setup gate and every auth
dependency, none of which run for a request that is already too large.
Every request gets a finite ceiling: `ROUTE_MAX_BODY_BYTES` raises it for the two routes that
genuinely take bulk bytes, and there is no path that skips counting altogether.
"""
def __init__(
self,
app: ASGIApp,
*,
limit: int = DEFAULT_MAX_BODY_BYTES,
routes: dict[tuple[str, str], int] | None = None,
) -> None:
self.app = app
self.limit = limit
self._routes = tuple(
(method, template_to_regex(path), cap)
for (method, path), cap in (ROUTE_MAX_BODY_BYTES if routes is None else routes).items()
)
def limit_for(self, method: str, path: str) -> int:
"""This request's ceiling: the route's own if it has one, else the global default. Every
request gets a finite number — there is no path through here that is simply not counted."""
for m, rx, cap in self._routes:
if m == method and rx.match(path):
return cap
return self.limit
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
limit = self.limit_for(scope.get("method", ""), scope.get("path", ""))
declared = _declared_length(scope)
if declared is not None and declared > limit:
# Refused on the declaration alone: nothing downstream is called, and not one byte of
# the body is read.
await _send_too_large(send)
return
seen = 0
refused = False
started = False
async def counting_receive() -> Message:
nonlocal seen, refused
message = await receive()
if message["type"] != "http.request":
return message
seen += len(message.get("body", b""))
if seen > limit:
# The app is blocked on this receive, so it has not answered yet. Telling it the
# client went away is the only in-band way to stop it reading: `request.stream()`
# turns an `http.disconnect` into ClientDisconnect, which unwinds the handler. Our
# own 413 goes out below, once the app has finished unwinding.
refused = True
return {"type": "http.disconnect"}
return message
async def guarded_send(message: Message) -> None:
nonlocal started
if refused and not started:
return # discard the app's answer to the disconnect; the 413 below is the response
started = True
await send(message)
try:
await self.app(scope, counting_receive, guarded_send)
except Exception: # noqa: BLE001 — only swallowed when WE caused it, see below
# ClientDisconnect (or whatever a handler raises on a truncated read) is the expected
# consequence of the disconnect we just faked, and the caller must get the 413 rather
# than a 500. Anything raised when we did NOT refuse is a real failure and is re-raised
# untouched.
if not refused:
raise
if refused and not started:
await _send_too_large(send)
+22 -2
View File
@@ -1,7 +1,7 @@
from pathlib import Path
from typing import Literal, get_args, get_origin
from pydantic import ValidationInfo, field_validator, model_validator
from pydantic import Field, ValidationInfo, field_validator, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
# Shipped placeholder; production must override it. Used by the startup guard below.
@@ -16,6 +16,8 @@ def _version_from_file() -> str:
v = p.read_text(encoding="utf-8").strip()
if v:
return v
# silent-ok: this runs at import, and `app.logs` imports THIS module — a log call here
# would be a circular import. The loop tries the other location, then falls back to "dev".
except OSError:
continue
return "dev"
@@ -117,6 +119,9 @@ class Settings(BaseSettings):
# Recent-first backfill: how far back to fetch on the first pass per channel.
backfill_recent_max_videos: int = 100
backfill_recent_max_days: int = 365
# A channel whose RSS feed has not answered for this many hours gets its newest uploads read
# through the Data API instead (1 unit), at most once per this many hours. 0 = never.
rss_fallback_hours: int = 2
# Shorts are confirmed by probing youtube.com/shorts/<id>. Only videos at or below
# this duration (and not livestreams) are probed; longer videos are never Shorts.
@@ -242,7 +247,11 @@ class Settings(BaseSettings):
# Per-file cap for local uploads (E7). Unlike a download (whose size is unknown until the worker
# finishes), an upload's size is known at ingest, so this bounds a single file directly; the
# user's overall byte footprint is still governed by download_default_max_bytes.
download_upload_max_bytes: int = 524_288_000 # 500 MiB
# ge=1: this number is also the EDGE ceiling (`bodylimit.ROUTE_MAX_BODY_BYTES`), where 0 is a
# real limit of zero rather than the route's "no per-file cap" sentinel — an instance set to 0
# would refuse every upload, including the admin page's own way back. To lift the cap in
# practice, set a large number.
download_upload_max_bytes: int = Field(default=524_288_000, ge=1) # 500 MiB
# Retention: an asset expires this many days after its last access; the GC job then
# deletes it (once no job references it) after an additional grace window.
download_retention_days: int = 30
@@ -310,6 +319,10 @@ class Settings(BaseSettings):
# re-encode is CPU-heavy (the VAAPI hardware path is far cheaper and could sustain more); direct-serve
# and remux (stream-copy) are not capped by this.
plex_max_transcodes: int = 4
# Max concurrent playback sessions of ANY kind — remux (stream-copy) included. Cheap on the CPU but
# each one is a whole segmented copy of the film on the scratch volume, written faster than
# realtime. One account holds at most two regardless (plex/stream.py `_MAX_SESSIONS_PER_USER`).
plex_max_sessions: int = 8
# --- Transcode (playable="transcode") re-encode tuning (E6 S1) ---
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
@@ -342,6 +355,13 @@ class Settings(BaseSettings):
tmdb_api_base: str = "https://api.themoviedb.org/3"
# Metadata language for TMDB fetches (keyword/overview locale). Stored EN by default.
tmdb_language: str = "en-US"
# --- Bug reports (E16): where a debug session's issue goes. All three empty = no tracker; the
# composer still exports a zip. The repository must be PRIVATE — an issue carries the reporter's
# log — and app/issues/gitea.py refuses a public one. Runtime knobs (sysconfig, admin UI). ---
issues_gitea_url: str = ""
issues_gitea_repo: str = ""
issues_gitea_token: str = ""
# How often the TMDB enrichment job fetches a batch of not-yet-enriched Plex titles, in minutes.
tmdb_enrich_interval_min: int = 120
# Titles fetched per TMDB enrichment run (TMDB is fast + generously rate-limited).
+52 -1
View File
@@ -1,6 +1,6 @@
from collections.abc import Iterator
from sqlalchemy import create_engine
from sqlalchemy import create_engine, text
from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
from app.config import settings
@@ -20,6 +20,10 @@ engine = create_engine(
max_overflow=30,
pool_timeout=30,
future=True,
# A database error renders its bound parameters into its message — a title, an uploader, a
# file path, a URL — and that message is logged and stored as a job's error text. The SQL
# stays; the values are data, and the ids an operator needs are on the line already.
hide_parameters=True,
)
SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False)
@@ -30,3 +34,50 @@ def get_db() -> Iterator[Session]:
yield db
finally:
db.close()
def release_read_connection(db: Session) -> None:
"""End a request's READ transaction so its pooled connection goes back BEFORE a long await.
SQLAlchemy 2.0 autobegins on the first SELECT, so a request that has merely authenticated is
already holding one of the connections above — `idle in transaction` to Postgres, and nothing
times it out. A route that then awaits something slow (a request body still arriving, an
external fetch) pins it for the whole wait, which is precisely the invariant the pool is sized
on in the comment above. Measured: the next checkout after this call reuses the very backend
that was released, and a 1.8 s upload goes from one sustained `idle in transaction` to none.
Every route that reads its own body from a dependency needs this, and `test_body_limit.py`
holds that — the three that do are the ones with a raised (or self-imposed) body ceiling, and
they got here one at a time, which is how the upload route ended up the only one fixed.
The caller must have NOTHING PENDING, and that is checked rather than described. The session is
shared with every dependency that ran before, so a rollback here would silently discard a write
one of them staged — `service.enqueue` already learned that lesson the expensive way and uses a
SAVEPOINT for it. CLAUDE.md: a rule stated in a comment must also be enforced in code.
TWO checks, because the obvious one is blind to the shapes this codebase actually writes in.
`db.new`/`deleted`/`dirty` hold only UNFLUSHED unit-of-work: the moment a helper calls
`db.flush()` — CLAUDE.md's canonical split is "the handler commits; helpers flush" — the object
is persistent with no unflushed history and all three sets are empty again, and Core DML
(`db.execute(update(...))`) never enters them at all. Measured: after `add`+`flush` the sets are
empty, the old check passed, and the rollback lost the row. So the transaction is asked
directly. Postgres assigns a transaction id on the first WRITE and never for a read, so
`pg_current_xact_id_if_assigned()` is NULL for exactly the state this function is safe in — and
it sees a raw `text("UPDATE …")` too, which no session-level bookkeeping would.
"""
if db.new or db.deleted or db.dirty:
raise RuntimeError(
"release_read_connection() with pending work: something ahead of this dependency "
"staged a write, which ending the transaction would discard. Commit it first, or do "
"it after the await."
)
if (
db.in_transaction()
and db.execute(text("SELECT pg_current_xact_id_if_assigned()")).scalar() is not None
):
raise RuntimeError(
"release_read_connection() on a transaction that has already WRITTEN: something ahead "
"of this dependency flushed a row or ran an UPDATE/INSERT/DELETE, which ending the "
"transaction would discard. Commit it first, or do it after the await."
)
db.rollback()
+485
View File
@@ -0,0 +1,485 @@
"""Debug sessions (E16): Start → reproduce → Stop, and the evidence a bug report is built from.
While a session is open its user's browser names it in `X-Debug-Session` on every request. The auth
dependency accepts the header only for the caller's OWN open session (anything else is ignored, never
an error — a stale tab must not break the app) and then logs that request at the session's level with
every line tagged by its id; the browser tags its own events the same way. The issue is later built
from exactly the rows carrying the tag, so what the review list shows is what gets sent.
Starting a session IS the consent for its level, for its duration: the start dialog carries the
consent text. It needs no change to the user's own diagnostics level, and none is made — when the
session ends the level in force is simply the user's own again.
A session is capped at `MAX_MINUTES`, set by the server at Start. An open one past its cap is over
whether or not anything has closed it yet: it reads as over at once, a new Start closes it, and the
hourly log job sweeps the rest. Sessions go with the logs they point at, after `log_retention_days`.
Core layer: imports `models`, `diagnostics`.
"""
import secrets
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from typing import Literal, get_args
from uuid import uuid4
from sqlalchemy import and_, delete, func, not_, select, update
from sqlalchemy.orm import Session, defer
from app import diagnostics
from app.models import DebugScreenshot, DebugSession, LogEvent, User
HEADER = "x-debug-session"
MAX_MINUTES = diagnostics.SESSION_MINUTES
SCREENSHOT_MAX_BYTES = 10 * 1024 * 1024
SCREENSHOTS_MAX = 8
# What one user may keep in screenshots across ALL their sessions (until retention takes them): the
# per-session cap alone would let Start → 8 images → Stop repeat without end.
SCREENSHOT_USER_MAX_BYTES = 100 * 1024 * 1024
# The screenshot types kept, and the extension each is exported and sent under — one table, so an
# accepted type always has a name.
IMAGE_TYPES = {"image/png": "png", "image/jpeg": "jpg", "image/webp": "webp"}
# What a screenshot is called in the zip and on the issue. `issues/bundle.py` builds the names from
# this and `IMAGE_TYPES`; this module and `issues/gitea.py` read them back to tell a screenshot's
# link from a log's. One definition, because three copies of a name is how the next rename breaks
# `images_pending` into silence.
SCREENSHOT_PREFIX = "screenshot-"
# Rows an issue can carry. A 60-minute session at Teljes is a few thousand rows; past this the
# bundle says how many it left out rather than growing without bound.
BUNDLE_MAX = 5_000
SessionLevel = Literal["DEBUG", "TRACE"]
SESSION_LEVELS: tuple[str, ...] = get_args(SessionLevel)
# Detail order of the diagnostics levels: a session raises the level in force, never lowers it.
_DETAIL = {"INFO": 0, "DEBUG": 1, "TRACE": 2}
_ID_LEN = 16
class EndReason(StrEnum):
STOPPED = "stopped"
EXPIRED = "expired"
def is_live(session: DebugSession, now: datetime | None = None) -> bool:
"""Open and within its cap — THE liveness rule: an open session past its cap is over."""
return session.ended_at is None and session.until > (now or datetime.now(timezone.utc))
def open_session(db: Session, user: User) -> DebugSession | None:
"""The user's live session, or None. Read-only: one past its cap is over and reads as None;
`start` and the hourly sweep are what close it."""
session = _unclosed(db, user)
if session is None or not is_live(session, datetime.now(timezone.utc)):
return None
return session
def _unclosed(db: Session, user: User) -> DebugSession | None:
return db.scalar(
select(DebugSession).where(DebugSession.user_id == user.id, DebugSession.ended_at.is_(None))
)
def start(db: Session, user: User, level: SessionLevel) -> DebugSession:
"""Open a session, or return the live one — a second tab pressing Start joins the recording
rather than ending it. One past its cap is closed first: the one-open-per-user index would
refuse the new row beside it. Flushes; the caller commits. Two concurrent Starts can still both
get here: the index refuses the second, and the caller joins the one that won."""
now = datetime.now(timezone.utc)
current = _unclosed(db, user)
if current is not None:
if is_live(current, now):
return current
end(current, EndReason.EXPIRED)
db.flush()
session = DebugSession(
id=secrets.token_hex(_ID_LEN // 2),
user_id=user.id,
level=level,
started_at=now,
until=now + timedelta(minutes=MAX_MINUTES),
)
db.add(session)
db.flush()
return session
def stop(session: DebugSession) -> None:
"""The user's Stop. Past its cap the session had already ended on its own, and a late Stop must
not rename that."""
end(session, EndReason.STOPPED if is_live(session) else EndReason.EXPIRED)
def end(session: DebugSession, reason: EndReason) -> None:
"""Close a session. An expired one is dated at its cap, not at whenever it was noticed."""
if session.ended_at is not None:
return
session.ended_at = (
session.until
if reason is EndReason.EXPIRED
else min(datetime.now(timezone.utc), session.until)
)
session.ended_reason = reason
def from_header(db: Session, user: User, value: str | None) -> DebugSession | None:
"""The session a request names, if it is the caller's own and still live; else None."""
if not value or len(value) != _ID_LEN or not all(c in "0123456789abcdef" for c in value):
return None
session = db.get(DebugSession, value)
if (
session is None
or session.user_id != user.id
or not is_live(session, datetime.now(timezone.utc))
):
return None
return session
def level_for(user: User, session: DebugSession | None) -> tuple[str, datetime | None]:
"""The diagnostics level (and its expiry) a request is logged at: the user's own, raised to the
session's while one is named — whichever records more.
A named session also caps the expiry at its own: this pair is what the process tracks under the
session (app/logs.py), and an expiry outliving the recording would hold the root logger down
after it ended."""
own = diagnostics.effective_level(user)
until = user.diagnostics_until if own != diagnostics.BASE else None
if session is None:
return own, until
if _DETAIL[session.level] <= _DETAIL[own]:
return own, min(until, session.until) if until else session.until
return session.level, session.until
def lock(db: Session, session: DebugSession) -> None:
"""Re-read the session under a row lock, held until the caller commits. Everything that changes
what an issue is made of — the draft, the screenshots, the send itself — takes it first: a send
in flight sees none of them change under it, and each of them sees a send that committed."""
db.refresh(session, with_for_update=True)
def save_draft(session: DebugSession, draft: dict) -> None:
"""Keep the composer's draft (the caller validated, locked and commits). Shape: `title`, `body`
(the rendered markdown an export or an issue carries), `excluded` (log row ids taken out of the
evidence) and `form` (the composer's own fields, to reopen it as it was). A delivery's progress
is not the composer's to replace, and is kept."""
progress = (session.draft or {}).get("delivery")
session.draft = {**draft, "delivery": progress} if progress else draft
# --- a delivery's progress -----------------------------------------------------------------
# Written by app/issues/gitea.py into the draft; read HERE, so the send's decision and what the
# composer shows are one reading of it.
def delivery(session: DebugSession) -> dict:
return dict((session.draft or {}).get("delivery") or {})
def not_uploaded(session: DebugSession) -> list[str]:
"""The files a send that stopped half-way has still to upload — by name, as the issue carries
them. What the composer names to the user."""
return list(delivery(session).get("pending") or [])
def images_pending(session: DebugSession) -> bool:
"""Whether the issue's body still has to be edited to show the screenshots it already carries
(every upload went, the edit that shows them did not)."""
state = delivery(session)
shown = state.get("images_shown")
return not shown and any(n.startswith(SCREENSHOT_PREFIX) for n in state.get("links") or {})
def unfinished(session: DebugSession) -> bool:
"""Whether a send owes the issue anything — the retry's question."""
return bool(not_uploaded(session)) or images_pending(session)
# A send claims the session while it talks to the tracker, instead of holding the row's lock across
# those calls: a Stop, an autosave or a screenshot would queue behind a lock for the whole send.
# They see the claim and are refused at once instead. A claim not renewed for this long was left by
# a request that died mid-send (the process restarted), and the next send takes it over — which is
# why a live send RENEWS it every time it reports progress (`set_delivery`). A claim is a lease, and
# a send slower than the lease must not lose it while it is still running.
SEND_CLAIM_SECONDS = 600
def _claim(session: DebugSession) -> dict:
claim = delivery(session).get("sending")
return claim if isinstance(claim, dict) else {}
def _stamp(token: str) -> dict:
return {"at": datetime.now(timezone.utc).isoformat(), "by": token}
def sending(session: DebugSession) -> bool:
"""Whether a send is under way right now (a live claim)."""
at = _claim(session).get("at")
if not at:
return False
return (datetime.now(timezone.utc) - datetime.fromisoformat(at)).total_seconds() < (
SEND_CLAIM_SECONDS
)
def claim_send(session: DebugSession) -> str:
"""Mark the session as being sent, and return the token that OWNS this claim. The caller holds
the lock and commits, which is what makes the claim visible to everyone else."""
token = uuid4().hex
_write_delivery(session, {**delivery(session), "sending": _stamp(token)})
return token
def release_send(session: DebugSession, token: str) -> None:
"""Give back our OWN claim, and only ours: one that aged out and was taken over belongs to the
send holding it now, and handing that one back would let a third send in beside it."""
if _claim(session).get("by") != token:
return
_write_delivery(session, {k: v for k, v in delivery(session).items() if k != "sending"})
class SendClaimLost(Exception):
"""This send's claim is no longer on the row: it aged out and another send took it over. The
send must stop — what it uploads from here it can no longer record, so a retry would send it
again — and it must not write, because the row now holds the other send's progress."""
def claim_owner(db: Session, session: DebugSession) -> str | None:
"""Who holds the row's claim RIGHT NOW, read from the ROW and not from the object.
`expire_on_commit` is off (app/db.py), so a checkpoint does not re-read: a send that stalled
past its lease still carries its own dead token in memory long after another send took over.
A targeted scalar, not `lock`/`db.refresh` — a refresh here would discard the `issue_url` that
`gitea.deliver` assigns one line before its first progress report, and that link is the one
thing the send must not lose."""
return db.scalar(
select(DebugSession.draft["delivery"]["sending"]["by"].as_string()).where(
DebugSession.id == session.id
)
)
def set_delivery(db: Session, session: DebugSession, state: dict, token: str) -> None:
"""Write a delivery's progress (app/issues/gitea.py keeps it; this module reads it), and renew
the claim while doing it — a send still reporting progress is still alive, so one slower than
the lease cannot expire while it is still running.
The claim is not the caller's to carry: `gitea.deliver` threads ONE snapshot of this dict
through a whole send, so a progress report that carried the claim wrote a stale one back at
every step. It is stamped from `token` here instead, and only after the ROW confirms the claim
is still ours. Without that check the write is a full-column overwrite with no WHERE: a send
that lost its lease came back and stomped the take-over send's committed links, then put its
own dead token back on an idle session (deep-review round 4). `release_send` has always made
the same check; this is the other half of it."""
if claim_owner(db, session) != token:
raise SendClaimLost(session.id)
state = {k: v for k, v in state.items() if k != "sending"}
state["sending"] = _stamp(token)
_write_delivery(session, state)
def _write_delivery(session: DebugSession, state: dict) -> None:
# A new dict, so the JSONB column sees the change.
session.draft = {**(session.draft or {}), "delivery": state}
def serialize(session: DebugSession | None, screenshots: int = 0) -> dict | None:
if session is None:
return None
return {
"id": session.id,
"level": session.level,
"started_at": session.started_at.isoformat(),
"until": session.until.isoformat(),
"ended_at": session.ended_at.isoformat() if session.ended_at else None,
"ended_reason": session.ended_reason,
"issue_url": session.issue_url,
# A send that stopped half-way: the files it has still to upload, and whether the issue's
# body has still to be edited to show the screenshots it already carries. Both are true of a
# send that is RUNNING too — every step commits its progress — so `sending` goes with them:
# without it a reload or a second tab renders a live send as a failed one, and offers a
# retry that is certain to be refused.
"pending_files": not_uploaded(session),
"images_pending": images_pending(session),
"sending": sending(session),
"screenshots": screenshots,
"draft": session.draft,
}
def view(db: Session, session: DebugSession) -> dict:
"""The session as the API serves it — one shape wherever it is served."""
return serialize(session, screenshot_count(db, session)) or {}
def screenshot_count(db: Session, session: DebugSession) -> int:
return (
db.scalar(
select(func.count())
.select_from(DebugScreenshot)
.where(DebugScreenshot.session_id == session.id)
)
or 0
)
# --- screenshots ---------------------------------------------------------------------------
# By content, not by the declared type: what is stored is served back to a browser as an image.
_SIGNATURES = (
(b"\x89PNG\r\n\x1a\n", "image/png"),
(b"\xff\xd8\xff", "image/jpeg"),
)
def sniff(data: bytes) -> str | None:
"""The image type `data` really is — one of IMAGE_TYPES — or None."""
for magic, mime in _SIGNATURES:
if data.startswith(magic):
return mime
if len(data) >= 12 and data[:4] == b"RIFF" and data[8:12] == b"WEBP":
return "image/webp"
return None
def lock_user(db: Session, user_id: int) -> None:
"""Hold this user's own lock for the rest of the transaction. A session's row lock bounds what
THAT session may keep; the byte cap below is the user's across all of them, and a lock on one
session cannot serialize a check that reads them all — two uploads to two sessions both read the
total before either commits, and both pass."""
db.execute(select(func.pg_advisory_xact_lock(user_id)))
def stored_screenshot_bytes(db: Session, user_id: int) -> int:
return (
db.scalar(
select(func.coalesce(func.sum(DebugScreenshot.size), 0))
.join(DebugSession, DebugSession.id == DebugScreenshot.session_id)
.where(DebugSession.user_id == user_id)
)
or 0
)
class ScreenshotRefused(Exception):
"""A screenshot that cannot be kept; `code` is the refusal's machine word."""
def __init__(self, code: str) -> None:
super().__init__(code)
self.code = code
def add_screenshot(db: Session, session: DebugSession, data: bytes) -> DebugScreenshot:
"""Keep one screenshot. Locks the user and the session, and flushes; the caller commits.
Refused while a send is under way or once the issue was sent (it can no longer carry it), past
the per-session count or the user's storage, or when it is not an image we can show."""
if not data:
raise ScreenshotRefused("empty")
mime = sniff(data)
if mime is None:
raise ScreenshotRefused("not_an_image")
# Under both locks: two uploads at once cannot both pass the count (the session's), nor both
# pass the user's byte cap from two different sessions (the user's). Always in this order.
lock_user(db, session.user_id)
lock(db, session)
if sending(session):
raise ScreenshotRefused("sending")
if session.issue_url is not None:
raise ScreenshotRefused("already_sent")
if screenshot_count(db, session) >= SCREENSHOTS_MAX:
raise ScreenshotRefused("too_many")
if stored_screenshot_bytes(db, session.user_id) + len(data) > SCREENSHOT_USER_MAX_BYTES:
raise ScreenshotRefused("storage_full")
shot = DebugScreenshot(
session_id=session.id,
created_at=datetime.now(timezone.utc),
mime=mime,
size=len(data),
data=data,
)
db.add(shot)
db.flush()
return shot
def remove_screenshot(db: Session, session: DebugSession, shot: DebugScreenshot) -> None:
"""Delete one screenshot, under the lock. Refused while a send is under way or unfinished: a
delivery names the screenshots by their order, and removing one would renumber what it still
has to send. Once the issue has everything, this only drops OUR copy — the issue keeps its
own. The caller commits."""
lock(db, session)
if sending(session):
raise ScreenshotRefused("sending")
if unfinished(session):
raise ScreenshotRefused("already_sent")
db.delete(shot)
db.flush()
def screenshots(db: Session, session: DebugSession) -> list[DebugScreenshot]:
"""The session's screenshots, oldest first, WITHOUT their bytes loaded."""
return list(
db.scalars(
select(DebugScreenshot)
.where(DebugScreenshot.session_id == session.id)
.options(defer(DebugScreenshot.data))
.order_by(DebugScreenshot.id)
)
)
# --- the evidence --------------------------------------------------------------------------
# The browser's own log upload (`POST` here, every few seconds while recording) is how the evidence
# arrives, not part of it: its server lines would fill the bundle with the recording's plumbing. The
# browser leaves its copies out for the same reason (frontend lib/diagnostics `isOwnRequest`).
INGEST_PATH = "/api/diagnostics/events"
def _evidence(session: DebugSession):
upload_line = and_(
LogEvent.event == "http.request", LogEvent.attrs["path"].astext == INGEST_PATH
)
# `IS TRUE`, not the bare condition: for a row with no event or no attrs it is NULL, and
# `NOT NULL` would drop exactly the rows that are evidence.
return (LogEvent.debug_session_id == session.id, not_(upload_line.is_(True)))
def bundle(
db: Session, session: DebugSession, *, drop_excluded: bool = False
) -> tuple[list[LogEvent], int]:
"""The session's log rows, browser and server interleaved in time, capped at BUNDLE_MAX — and
how many rows it has in all, so a cut bundle can say so. `drop_excluded` leaves out the rows
the user took out in the composer: what an export or an issue carries."""
where = list(_evidence(session))
excluded = (session.draft or {}).get("excluded") or [] if drop_excluded else []
if excluded:
where.append(LogEvent.id.not_in(excluded))
rows = list(
db.scalars(
select(LogEvent).where(*where).order_by(LogEvent.ts, LogEvent.id).limit(BUNDLE_MAX)
)
)
total = len(rows)
if total == BUNDLE_MAX:
total = db.scalar(select(func.count()).select_from(LogEvent).where(*where)) or total
return rows, total
# --- housekeeping (the hourly log job) -----------------------------------------------------
def sweep(db: Session) -> int:
"""Stage closing every open session past its cap; returns how many."""
now = datetime.now(timezone.utc)
result = db.execute(
update(DebugSession)
.where(DebugSession.ended_at.is_(None), DebugSession.until <= now)
.values(ended_at=DebugSession.until, ended_reason=EndReason.EXPIRED.value)
.execution_options(synchronize_session=False)
)
return result.rowcount or 0
def gc(db: Session, retention_days: int) -> int:
"""Stage deleting sessions (and by cascade their screenshots) started before the log retention
window — the rows they would bundle are gone by then too. Returns how many."""
cutoff = datetime.now(timezone.utc) - timedelta(days=max(1, retention_days))
result = db.execute(delete(DebugSession).where(DebugSession.started_at < cutoff))
return result.rowcount or 0
+4
View File
@@ -27,6 +27,9 @@ DiagnosticsLevel = Literal["INFO", "DEBUG", "TRACE"]
LEVELS: tuple[str, ...] = get_args(DiagnosticsLevel)
BASE: DiagnosticsLevel = "INFO"
ELEVATION_HOURS = 24
# How long a debug session records at most (E16, app/debugsession.py). Here, beside the other
# consent window, because the start dialog's consent text states it and reads it from `state`.
SESSION_MINUTES = 60
def effective_level(user: User, now: datetime | None = None) -> DiagnosticsLevel:
@@ -47,6 +50,7 @@ def state(db: Session, user: User) -> dict:
"level": level,
"until": user.diagnostics_until.isoformat() if level != BASE else None,
"retention_days": sysconfig.get_int(db, "log_retention_days"),
"session_minutes": SESSION_MINUTES,
}
+5 -1
View File
@@ -81,7 +81,11 @@ def pick(
return None
def _comfort() -> dict | None:
for card in pins.playables(pins.shelf(db, user) if shelf is None else shelf):
if shelf is None:
own = pins.shelf(db, user, include_plex=rails.plex_available(db))
else:
own = shelf
for card in pins.playables(own):
if cards.key(card) not in skip and fits(card["remaining_seconds"], budget):
return card
return None
+109 -15
View File
@@ -8,14 +8,16 @@ second one.
Every shelf card carries a resolved `next`: the thing that actually plays when you click the tile —
a show's in-progress-or-next episode, a channel's newest unwatched video, a playlist's first
unwatched item. It costs a query per non-leaf pin, which is affordable precisely because this shelf is
short and hand-curated, and it buys the property the whole surface exists for: one click, no
intermediate page, no second decision.
unwatched item. That buys the property the whole surface exists for: one click, no intermediate page,
no second decision. Channels and playlists are resolved for the whole shelf at once (`_Batch`), so the
cost does not grow a query or two per pin.
Domain module, so nothing here raises HTTP. `resolve_ref` returns None for "no such thing, or not
yours" and the route turns that into its 404 — core never imports routes.
"""
from dataclasses import dataclass, field
from sqlalchemy import func, select
from sqlalchemy.orm import Session
@@ -62,6 +64,8 @@ def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, obje
return (col, sh.id) if sh is not None else None
try:
pid = int(ref)
# silent-ok: validating a reference the client supplied. "Not an integer" is answered as "no
# such pin target", which is the same answer this function gives for one that does not exist.
except ValueError:
return None
pl = db.get(Playlist, pid)
@@ -71,6 +75,44 @@ def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, obje
return (col, pl.id) if pl is not None and pl.user_id == user.id else None
def reorder(db: Session, user: User, ids: list[int]) -> None:
"""Put the listed pins in the listed order, WITHIN the slots they already hold.
The client sends the order it can SEE, which is not always the whole shelf: with Plex off the Home
rail hides the Plex pins, and a stale tab may not know a pin added elsewhere. Numbering only the
listed ids 0..n-1 collided with every pin left out — two tiles on one position, and the hidden
ones jumped about when they came back. So the pins left out keep their slots, and the listed ones
are dealt into the slots the listed set occupies. Ids that are not this user's, or repeated, are
ignored rather than failing a stale replay wholesale. Flushes; the route commits."""
# Locked: the rows it leaves alone are part of the answer too. Two overlapping reorders each
# skipped the rows that already matched in their own snapshot and interleaved into two pins on
# one slot; the second now waits and deals from the first one's result. Locked in id order and
# sorted HERE: with `ORDER BY position … FOR UPDATE` Postgres sorts before it locks, so the one
# that waited got the fresh rows in its stale order and dealt the pins it left out back to their
# old slots — and one global lock order means two reorders cannot deadlock.
locked = db.execute(
select(ComfortPin)
.where(ComfortPin.user_id == user.id)
.order_by(ComfortPin.id)
.with_for_update()
.execution_options(populate_existing=True)
).scalars()
current = sorted(locked, key=lambda p: (p.position, p.id))
owned = {p.id for p in current}
listed: list[int] = []
for i in ids:
if i in owned and i not in listed:
listed.append(i)
chosen = set(listed)
queue = iter(listed)
order = [next(queue) if p.id in chosen else p.id for p in current]
by_id = {p.id: p for p in current}
for position, pin_id in enumerate(order):
if by_id[pin_id].position != position:
by_id[pin_id].position = position
db.flush()
def list_pins(db: Session, user: User) -> list[ComfortPin]:
return list(
db.execute(
@@ -101,7 +143,34 @@ def next_position(db: Session, user: User) -> int:
)
def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
@dataclass
class _Batch:
"""What the shelf resolves for every pinned channel and playlist in one go."""
channels: dict[str, Channel] = field(default_factory=dict)
channel_summary: dict[str, tuple[int, dict | None]] = field(default_factory=dict)
playlists: dict[int, Playlist] = field(default_factory=dict)
playlist_summary: dict[int, tuple[int, dict | None]] = field(default_factory=dict)
@classmethod
def load(cls, db: Session, user: User, pins: list[ComfortPin]) -> "_Batch":
ch_ids = [p.channel_id for p in pins if p.channel_id is not None]
pl_ids = [p.playlist_id for p in pins if p.playlist_id is not None]
b = cls()
if ch_ids:
b.channels = {
c.id: c for c in db.scalars(select(Channel).where(Channel.id.in_(ch_ids)))
}
b.channel_summary = queries.channels_summary(db, user, ch_ids)
if pl_ids:
b.playlists = {
p.id: p for p in db.scalars(select(Playlist).where(Playlist.id.in_(pl_ids)))
}
b.playlist_summary = queries.playlists_summary(db, user, pl_ids)
return b
def _shelf_card(db: Session, user: User, pin: ComfortPin, batch: _Batch) -> dict | None:
"""One pin as a unified card with its resolved `next`. None when the pinned row has gone (the FK
cascade removes the pin itself, so this is only the window between a delete and a refetch)."""
card: dict | None = None
@@ -115,11 +184,11 @@ def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
nxt = dict(card) # a video plays itself
elif pin.channel_id is not None:
ch = db.get(Channel, pin.channel_id)
ch = batch.channels.get(pin.channel_id)
if ch is None:
return None
card = cards.from_channel(ch, queries.channel_unwatched_count(db, user, ch.id))
row = queries.channel_next_video(db, user, ch.id)
count, row = batch.channel_summary.get(ch.id, (0, None))
card = cards.from_channel(ch, count)
nxt = cards.from_video(row) if row else None
elif pin.plex_item_id is not None:
@@ -140,10 +209,10 @@ def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
nxt = cards.from_plex(ep) if ep else None
elif pin.playlist_id is not None:
pl = db.get(Playlist, pin.playlist_id)
pl = batch.playlists.get(pin.playlist_id)
if pl is None:
return None
count, row = queries.playlist_summary(db, user, pl.id)
count, row = batch.playlist_summary.get(pl.id, (0, None))
card = cards.from_playlist(pl, count, row.get("thumbnail_url") if row else None)
nxt = cards.from_video(row) if row else None
@@ -157,11 +226,24 @@ def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
return card
def shelf(db: Session, user: User) -> list[dict]:
"""The whole shelf, in the user's own order."""
_PLEX_COLUMNS = ("plex_item_id", "plex_show_id")
def shelf(db: Session, user: User, *, include_plex: bool = True) -> list[dict]:
"""The whole shelf, in the user's own order.
`include_plex=False` is the Home surface with the deployment's Plex toggle off (the owner's call,
2026-09-19): the rails beside it go YouTube-only, and a Plex tile there would offer a Play that
lands on a gated module. The pins themselves are kept — the pin menus still read the full shelf
(`GET /pins`), and switching Plex back on brings the tiles back in their old slots
(`reorder` keeps a hidden pin's place)."""
out = []
for pin in list_pins(db, user):
card = _shelf_card(db, user, pin)
pins = list_pins(db, user)
if not include_plex:
pins = [p for p in pins if all(getattr(p, c) is None for c in _PLEX_COLUMNS)]
batch = _Batch.load(db, user, pins)
for pin in pins:
card = _shelf_card(db, user, pin, batch)
if card is not None:
out.append(card)
return out
@@ -169,5 +251,17 @@ def shelf(db: Session, user: User) -> list[dict]:
def playables(shelf_cards: list[dict]) -> list[dict]:
"""The `next` of every shelf entry that has one — what the hero may draw from. A pinned channel
with nothing unwatched left, or an empty playlist, simply is not an answer right now."""
return [c["next"] for c in shelf_cards if c.get("next")]
with nothing unwatched left, or an empty playlist, simply is not an answer right now.
Each carries `pinned_from`: the pin it was resolved from. A show's next EPISODE is not itself
pinned, so without it the hero's menu looked the episode up, missed, and offered "Pin" under a
kicker that says "From your shelf" — and taking it inserted a second pin. With it, the hero's
pin row acts on the shelf entry the card came from (the owner's call, 2026-09-19: "Unpin")."""
return [
{
**c["next"],
"pinned_from": {k: c[k] for k in ("source", "kind", "id", "title")},
}
for c in shelf_cards
if c.get("next")
]
+44 -31
View File
@@ -43,8 +43,9 @@ from app.plex import rollup
from app.plex.cards import leaf_card, movie_card, show_card
from app.plex.query import apply_show_state_filter, content_libs
# How many rows a single rail may carry. Matches `routes/plex/discover.py`'s RAIL_LIMIT on purpose —
# the two surfaces should feel like the same app, and a rail is a rail.
# How many rows a single rail may carry — Home's rails and the Plex module's Discover rails alike
# (`routes/plex/discover.py` imports it): the two surfaces should feel like the same app, and a rail
# is a rail. One definition, because a comment was all that held two copies in step.
RAIL_LIMIT = 20
# The feed parameters a Home rail runs with. Spelled out rather than borrowed from the route's
@@ -129,18 +130,6 @@ def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) ->
return [r for r in out if not r["position_seconds"]][:limit]
def channel_next_video(db: Session, user: User, channel_id: str) -> dict | None:
"""The newest unwatched video of a pinned channel — what "put on that channel" resolves to."""
# `library_source="all"`: a pinned channel means the whole channel, including videos that only
# ever arrived through a search. The feed's "organic" default is a feed-scoping choice, not a
# statement about what belongs to a channel.
params = dict(_FEED_BASE, channel_ids=[channel_id], scope="all", library_source="all")
query, rank_expr = filtered_query(db, user, **params)
keys = sort_keys("newest", 0, rank_expr)
row = db.execute(query.order_by(*order_by(keys)).limit(1)).first()
return serialize(row) if row is not None else None
def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
"""One video as a feed row, with this user's watch state.
@@ -158,31 +147,55 @@ def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
return serialize(row) if row is not None else None
def channel_unwatched_count(db: Session, user: User, channel_id: str) -> int:
"""How many unwatched videos a pinned channel is holding — the tile's only honest promise, since a
channel has no runtime of its own."""
params = dict(_FEED_BASE, channel_ids=[channel_id], scope="all", library_source="all")
query, _ = filtered_query(db, user, **params)
return int(db.execute(select(func.count()).select_from(query.subquery())).scalar() or 0)
def channels_summary(
db: Session, user: User, channel_ids: list[str]
) -> dict[str, tuple[int, dict | None]]:
"""For each pinned channel: (unwatched count, its newest unwatched video) — the tile's only honest
promise, since a channel has no runtime of its own, and what "put on that channel" resolves to.
ONE query for the whole shelf, not two per channel: the count and the first row per channel are
window functions over the same filtered set. `library_source="all"`: a pinned channel means the
whole channel, including videos that only ever arrived through a search — the feed's "organic"
default is a feed-scoping choice, not a statement about what belongs to a channel."""
if not channel_ids:
return {}
params = dict(_FEED_BASE, channel_ids=list(channel_ids), scope="all", library_source="all")
query, rank_expr = filtered_query(db, user, **params)
newest = order_by(sort_keys("newest", 0, rank_expr))
per_channel = {"partition_by": Video.channel_id}
sub = query.add_columns(
func.row_number().over(**per_channel, order_by=newest).label("_rn"),
func.count().over(**per_channel).label("_total"),
).subquery()
rows = db.execute(select(sub).where(sub.c._rn == 1)).all()
return {r.channel_id: (int(r._total), serialize(r)) for r in rows}
def playlist_summary(db: Session, user: User, playlist_id: int) -> tuple[int, dict | None]:
"""(item count, the row to play next) for a pinned playlist — its first not-yet-watched item, or
its first item once the whole list is watched. The same row doubles as the tile's artwork, so a
shelf of playlists is scannable instead of a row of blank boxes."""
def playlists_summary(
db: Session, user: User, playlist_ids: list[int]
) -> dict[int, tuple[int, dict | None]]:
"""For each pinned playlist: (item count, the row to play next) — its first not-yet-watched item,
or its first item once the whole list is watched. The same row doubles as the tile's artwork, so
a shelf of playlists is scannable instead of a row of blank boxes. One query for the whole shelf."""
if not playlist_ids:
return {}
st = aliased(VideoState)
rows = db.execute(
select(*feed_columns(user, st), PlaylistItem.position)
select(*feed_columns(user, st), PlaylistItem.playlist_id.label("_playlist"))
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
.where(PlaylistItem.playlist_id == playlist_id)
.order_by(PlaylistItem.position.asc())
.where(PlaylistItem.playlist_id.in_(playlist_ids))
.order_by(PlaylistItem.playlist_id, PlaylistItem.position.asc())
).all()
if not rows:
return 0, None
nxt = next((r for r in rows if (r.status or "new") not in ("watched", "hidden")), rows[0])
return len(rows), serialize(nxt)
grouped: dict[int, list] = {}
for r in rows:
grouped.setdefault(r._playlist, []).append(r)
out: dict[int, tuple[int, dict | None]] = {}
for pid, items in grouped.items():
nxt = next((r for r in items if (r.status or "new") not in ("watched", "hidden")), items[0])
out[pid] = (len(items), serialize(nxt))
return out
# ── Plex ──────────────────────────────────────────────────────────────────────────────────────────
+3 -2
View File
@@ -18,6 +18,7 @@ import yt_dlp
from yt_dlp.networking.impersonate import ImpersonateTarget
from app.config import settings
from app.downloads.stated import StatedFailure
# Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
# below and the failure catalog (errors.py) decide on it, so the wording exists once.
@@ -178,11 +179,11 @@ class FacebookAdapter(SourceAdapter):
# Measured (E11.5 review): a refused or expired cookie gets the login page (~450 KB, a login
# form, no post id at all), which is not a post without a video — the user's cookie is the cause.
if _quoted(post_id) not in page:
raise RuntimeError(POST_NOT_SHOWN_MESSAGE)
raise StatedFailure(POST_NOT_SHOWN_MESSAGE)
video_id = facebook_post_video_id(page, post_id)
if video_id is None:
# Never fall back to the post link: that is exactly where the wrong video comes from.
raise RuntimeError(POST_WITHOUT_VIDEO_MESSAGE)
raise StatedFailure(POST_WITHOUT_VIDEO_MESSAGE)
return f"https://www.facebook.com/watch/?v={video_id}"
def client_sets(self, authed: bool = False) -> list:
+27 -7
View File
@@ -11,11 +11,12 @@ Run once per environment:
docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
"""
import logging
from urllib.parse import urlsplit
import yt_dlp
from sqlalchemy import select
from app import logs
from app.config import settings
from app.db import SessionLocal
from app.downloads import adapters
@@ -23,12 +24,17 @@ from app.downloads import edit as editmod
from app.downloads import formats, service
from app.models import MediaAsset
log = logging.getLogger("siftlode.backfill")
log = logs.get_logger(__name__)
def _extract_meta(url: str, adapter: adapters.SourceAdapter) -> dict | None:
def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
"""Metadata-only yt-dlp extraction, using the source adapter's client sets + options (YouTube
needs the POT/player-client strategy even just to extract)."""
with formats.hiding(url):
return _extract_meta_scoped(url, adapter, asset_id)
def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
for clients in adapter.client_sets():
opts = formats.build_ydl_opts(
{}, "%(id)s.%(ext)s", lambda d: None, extra=adapter.ydl_opts_extra(clients)
@@ -40,7 +46,19 @@ def _extract_meta(url: str, adapter: adapters.SourceAdapter) -> dict | None:
if info:
return info
except Exception as exc: # noqa: BLE001 — best-effort backfill, keep going
log.info("meta extract failed for %s (%s): %s", url, clients, str(exc)[:100])
# The asset id, not the URL. A source URL is a row of this user's download
# history, and `redact_text` only takes credential params out of it — the rest
# of it goes in whole. The id names the same asset to anyone holding the
# database and nothing at all to anyone who is not; the host is kept because it
# is what decides which client set was tried. yt-dlp's message goes through
# `log_safe` for the same reason: it opens with the URL or the video id.
log.info(
"meta extract failed for asset %s on %s (%s): %s",
asset_id,
urlsplit(url).hostname or "?", # log-pii-ok: the host, never the path or query
clients,
formats.log_safe(exc)[:100],
)
return None
@@ -60,12 +78,14 @@ def run() -> dict:
adapter = adapters.resolve_adapter(a.source_kind, a.source_ref)
if a.uploader_url is None and adapter.wants_metadata_backfill():
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
info = _extract_meta(url, adapter)
info = _extract_meta(url, adapter, a.id)
uu = adapter.uploader_url(info) if info else None
if uu:
a.uploader_url = uu
filled_url += 1
log.info("asset %s uploader_url <- %s", a.id, uu)
log.info(
"asset %s uploader_url filled", a.id
) # the URL names what they downloaded
if a.poster_path is None:
# Record a self-hosted poster for every download (the reliable og:image source):
# ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
@@ -73,7 +93,7 @@ def run() -> dict:
if rel:
a.poster_path = rel
filled_poster += 1
log.info("asset %s poster <- %s", a.id, rel)
log.info("asset %s poster filled", a.id) # the path carries the title
db.commit()
result = {
"assets": len(assets),
+2 -2
View File
@@ -23,19 +23,19 @@ Security controls enforced here:
logged, or written to a job's error/snapshot.
"""
import logging
import os
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential, MediaAsset
from app.security import decrypt_optional, encrypt
log = logging.getLogger("siftlode.downloads.credentials")
log = logs.get_logger(__name__)
class HostSpec:
+78 -15
View File
@@ -23,7 +23,6 @@ for a player hover-preview later.
import hashlib
import json
import logging
import math
import os
import queue
@@ -33,9 +32,11 @@ import threading
import time
from pathlib import Path
from app import logs
from app.config import settings
from app.downloads.stated import StatedFailure
log = logging.getLogger("siftlode.edit")
log = logs.get_logger(__name__)
def encode_thread_cap() -> int:
@@ -71,7 +72,13 @@ def normalize_edit_spec(spec: dict | None) -> dict:
if crop:
try:
c = {k: int(round(float(crop[k]))) for k in ("x", "y", "w", "h")}
except (KeyError, TypeError, ValueError):
except (KeyError, TypeError, ValueError) as exc:
# The clip still renders — uncropped. The user asked for a crop, got a whole frame,
# and nothing anywhere called it a failure; this line is the only trace there is.
log.warning(
"the crop in an edit spec could not be read and was dropped",
extra=logs.event("edit.spec.crop_dropped", reason=type(exc).__name__),
)
c = None
if c and c["w"] > 0 and c["h"] > 0 and c["x"] >= 0 and c["y"] >= 0:
out["crop"] = c
@@ -80,6 +87,7 @@ def normalize_edit_spec(spec: dict | None) -> dict:
segs = spec.get("segments")
if isinstance(segs, list) and segs:
norm: list[dict] = []
unreadable = 0
for s in segs:
try:
st = max(0.0, float((s or {}).get("start_s") or 0.0))
@@ -88,9 +96,18 @@ def normalize_edit_spec(spec: dict | None) -> dict:
continue
en = float(end_raw)
except (TypeError, ValueError):
# silent-ok: counted, not logged here — one cut-list can drop several, and the
# join that comes out is simply shorter than the one the user built. Reported
# once after the loop.
unreadable += 1
continue
if en > st:
norm.append({"start_s": round(st, 3), "end_s": round(en, 3)})
if unreadable:
log.warning(
"some segments of a cut list could not be read and were dropped",
extra=logs.event("edit.spec.segments_dropped", dropped=unreadable, kept=len(norm)),
)
norm.sort(key=lambda x: x["start_s"])
# SEC-13: an accurate join builds one ffmpeg `filter_complex` with two filters per segment,
# so an unbounded list becomes an unbounded command line and an ffmpeg that never finishes.
@@ -114,7 +131,13 @@ def normalize_edit_spec(spec: dict | None) -> dict:
start = max(0.0, float(trim.get("start_s") or 0.0))
end_raw = trim.get("end_s")
end = float(end_raw) if end_raw is not None else None
except (TypeError, ValueError):
except (TypeError, ValueError) as exc:
# Either a 400 the user cannot act on, or — beside a crop — a full-length clip where
# they asked for a cut. Both are worth one line saying the spec was the problem.
log.warning(
"the trim in an edit spec could not be read and was dropped",
extra=logs.event("edit.spec.trim_dropped", reason=type(exc).__name__),
)
start, end = 0.0, None
t: dict = {"start_s": round(start, 3)}
if end is not None and end > start:
@@ -306,11 +329,14 @@ def _parse_out_time(line: str) -> float | None:
if line.startswith("out_time_us="):
try:
return int(line.split("=", 1)[1]) / 1_000_000
# silent-ok: one line of ffmpeg's `-progress` stream, several per second. None means "this
# line carried no usable time", which the caller already treats as "no progress yet".
except ValueError:
return None
if line.startswith("out_time_ms="): # (some builds mislabel this as microseconds)
try:
return int(line.split("=", 1)[1]) / 1_000_000
# silent-ok: as above, the other spelling of the same line.
except ValueError:
return None
return None
@@ -329,7 +355,9 @@ def _pump(stream, sink: "queue.Queue[str | None]") -> None:
try:
for line in stream:
sink.put(line)
except (ValueError, OSError): # pipe closed under us by the kill path
# silent-ok: the pipe was closed under us by our OWN kill path — the expected end of a
# cancelled or timed-out run, and the code that did the killing is the code that reports it.
except (ValueError, OSError):
pass
finally:
sink.put(None)
@@ -347,6 +375,8 @@ def _drain_tail(stream, sink: list[str]) -> None:
sink.append(line)
if len(sink) > 200:
del sink[:100]
# silent-ok: same closed pipe as `_pump` above, and the tail collected so far is still handed
# to whoever reports the failure.
except (ValueError, OSError):
pass
@@ -362,13 +392,17 @@ def escalate_kill(proc: subprocess.Popen, terminate, kill, grace_s: float, label
try:
proc.wait(timeout=grace_s)
return
# silent-ok: the escalation below IS the handling of this timeout, and surviving THAT is what
# the line under the second wait reports.
except subprocess.TimeoutExpired:
pass
kill()
try:
proc.wait(timeout=grace_s)
except subprocess.TimeoutExpired:
log.error("%s pid %s survived SIGKILL", label, proc.pid)
# noqa TRY400: the exception IS the message here. A TimeoutExpired's traceback is this
# `wait` call and nothing else — it would add three frames and no fact the line lacks.
log.error("%s pid %s survived SIGKILL", label, proc.pid) # noqa: TRY400
def _stop(proc: subprocess.Popen) -> None:
@@ -404,6 +438,8 @@ def run_ffmpeg(
while True:
try:
line = lines.get(timeout=1.0)
# silent-ok: not a failure — the one-second timeout is how this loop gets a turn to
# poll for cancellation while ffmpeg is quiet.
except queue.Empty:
line = ""
if line is None: # stdout hit EOF — ffmpeg is done (or dying)
@@ -416,11 +452,11 @@ def run_ffmpeg(
if should_cancel():
raise EditAborted
if time.monotonic() - last_output > stall_timeout_s:
raise RuntimeError(f"ffmpeg stalled: no progress for {int(stall_timeout_s)}s")
raise StatedFailure(f"ffmpeg stalled: no progress for {int(stall_timeout_s)}s")
try:
ret = proc.wait(timeout=_FFMPEG_EXIT_GRACE_S)
except subprocess.TimeoutExpired: # pipes closed but the process lingers
raise RuntimeError("ffmpeg did not exit after closing its output")
raise StatedFailure("ffmpeg did not exit after closing its output")
except BaseException:
# ONE exit path stops the process, so the invariant is unconditional: nothing leaves this
# function with ffmpeg still running. Not just our own EditAborted/stall — `on_progress`
@@ -435,7 +471,7 @@ def run_ffmpeg(
if pipe:
pipe.close()
if ret != 0:
raise RuntimeError(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
raise StatedFailure(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
# --- ffmpeg: universal browser playability --------------------------------------------------
@@ -471,7 +507,13 @@ def run_ffprobe_json(
try:
out = subprocess.run(args, capture_output=True, text=True, timeout=30, check=True).stdout
return json.loads(out)
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError) as exc:
# None makes every field it would have filled read as absent, and a file with no streams
# and no duration is indistinguishable here from one that is not media at all.
log.warning(
"ffprobe could not read a file",
extra=logs.event("media.ffprobe_failed", reason=type(exc).__name__),
)
return None
@@ -525,7 +567,13 @@ def probe_duration(path: Path) -> int:
check=True,
).stdout.strip()
return int(float(out)) if out else 0
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError) as exc:
# Zero is the denominator of the re-encode progress bar, so a failed probe leaves the user
# watching a bar that never moves rather than a job that says it went wrong.
log.info(
"the duration probe failed; progress for this job has no total",
extra=logs.event("media.duration_probe_failed", reason=type(exc).__name__),
)
return 0
@@ -670,15 +718,28 @@ def _run_ffmpeg_best_effort(cmd: list[str], timeout_s: int) -> Exception | None:
try:
subprocess.run(cmd, capture_output=True, timeout=timeout_s, check=True)
return None
# silent-ok: the exception is the RETURN VALUE. This helper exists so the three callers share
# one run/timeout policy, and each logs the failure with the context only it has.
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError) as exc:
return exc
def _step_failure(err: BaseException) -> str:
"""What a best-effort image step's failure line says: the kind and, for ffmpeg, its exit
status. Never `str(err)`: a CalledProcessError or TimeoutExpired quotes ffmpeg's argv, and an
OSError its file name — the source path either way, which is built from the channel and the
title."""
code = getattr(err, "returncode", None)
return f"{type(err).__name__} {code}" if code is not None else type(err).__name__
def _is_jpeg(path: Path) -> bool:
"""True if the file starts with the JPEG SOI marker (FF D8 FF)."""
try:
with path.open("rb") as f:
return f.read(3) == b"\xff\xd8\xff"
# silent-ok: False only means "do not copy this straight through" — the caller then goes the
# ffmpeg route, which opens the same file and reports it properly if it really is unreadable.
except OSError:
return False
@@ -699,13 +760,15 @@ def thumbnail_to_jpg(src: Path | None, dest: Path, timeout_s: int = 30) -> Path
shutil.copyfile(src, dest)
return dest
except OSError as exc:
log.info("thumbnail copy failed (%s); poster falls back to a cut frame", str(exc)[:120])
log.info(
"thumbnail copy failed (%s); poster falls back to a cut frame", _step_failure(exc)
)
return None
err = _run_ffmpeg_best_effort(build_thumbnail_jpg_cmd(src, dest), timeout_s)
if err is not None:
log.info(
"thumbnail->jpg conversion failed (%s); poster falls back to a cut frame",
str(err)[:120],
_step_failure(err),
)
return None
return dest if dest.exists() else None
@@ -730,7 +793,7 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
at = min(3.0, (asset.duration_s or 10) * 0.1)
err = _run_ffmpeg_best_effort(build_poster_cmd(src, dest, at), timeout_s)
if err is not None:
log.info("poster gen skipped for asset %s: %s", asset.id, str(err)[:120])
log.info("poster gen skipped for asset %s: %s", asset.id, _step_failure(err))
return None
return rel if dest.exists() else None
@@ -761,7 +824,7 @@ def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict |
cmd = build_storyboard_cmd(src, dest, asset.duration_s, geom)
err = _run_ffmpeg_best_effort(cmd, timeout_s)
if err is not None:
log.info("storyboard gen skipped for asset %s: %s", asset.id, str(err)[:120])
log.info("storyboard gen skipped for asset %s: %s", asset.id, _step_failure(err))
return None
if not dest.exists():
return None
+139 -1
View File
@@ -17,8 +17,134 @@ spec shape (all keys optional; defaults + browser-playability clamp applied by `
`container`/`vcodec` keys survive only so an old stored profile deserializes without error.
"""
import contextlib
import contextvars
import hashlib
import json
import logging
import os
import re
import urllib.parse
from collections.abc import Iterator
from app import logs
log = logs.get_logger(__name__)
# yt-dlp names what it is working on in these shapes, and each one is a row of someone's download
# history: the source URL, the `[extractor] <id>:` prefix on nearly every warning and error, the
# file path it writes to (`storage.rel_path` builds it from the channel and the title), a
# playlist's title, and an id quoted mid-sentence. The job id is already on every line
# (`logs.set_job` in the download child), so none of them adds anything an operator needs. What
# stays: the host (it says which extractor is failing) and the extractor tag. Every pattern below
# is a shape measured in yt-dlp's own source (2026.07.04), pinned in test_ytdlp_logging.py.
_URL_RE = re.compile(r"https?://([^/\s]+)\S*")
# `Destination:` is yt-dlp's own word in `[download] Destination: <path>`, not an id — the path
# after it is what `_PATH_RE` takes out.
_ID_PREFIX_RE = re.compile(r"(\[[\w:.-]+\] )(?!Destination:)[^\s:\[\]]+:")
# The generic extractor's "id" is the URL's basename, percent-DECODED: `My%20Holiday.mp4` is
# `My Holiday` — spaces, brackets, colons and all, which no pattern can bound (`Our Trip: Day 1`
# ends at which `: `?). The job knows its own URL, so the exact name is taken out as a literal
# (`hiding`). This pattern is the fallback for a line logged outside that scope: up to the first
# `: `. Only this extractor: every other one names a site id, and letting the prefix run to `: `
# everywhere would take the words out of lines like `[VideoConvertor] Converting …; Destination:`.
_GENERIC_ID_RE = re.compile(r"(\[generic\] )(?!Destination:)[^\[\]\n]+?:(?= |$)", re.M)
# `[youtube:tab] Playlist <title>: Downloading 5 items` — the title is not an id, and has spaces.
_PLAYLIST_TITLE_RE = re.compile(r"(\] Playlist ).+?(?=: Downloading )")
# `[download] Finished downloading playlist: <title>`, `[download] Downloading playlist: <title>`,
# `… Skipping already downloaded playlist: <title>` — the title runs to the end of the line.
_TITLE_TAIL_RE = re.compile(r"(playlist: |\[download\] Downloading \w+: ).*", re.I)
# An id quoted mid-sentence, not as a prefix: `… not available for <id>`, and `--no-playlist`'s
# `Downloading just the video <id>` (`_yes_playlist`) / `Downloading just video <id>` (multifeed).
_BARE_ID_RE = re.compile(
r"((?:subtitles not available for|Downloading just (?:the )?\w+) )[^\s,;]+"
)
# Two segments or more, and not glued to a word: `www.youtube.com/…` is what the URL pattern
# leaves behind, and it must survive this one.
_PATH_RE = re.compile(r"(?<![\w.…])/(?:[^/\s\"']+/)+[^/\s\"']*")
_HIDDEN: contextvars.ContextVar[tuple[str, ...]] = contextvars.ContextVar(
"formats_hidden", default=()
)
def generic_id(url: str) -> str:
"""The name yt-dlp's generic extractor gives a URL: its basename, extension off, percent-
decoded — `InfoExtractor._generic_id` (2026.07.04), pinned against it in the tests."""
return urllib.parse.unquote(os.path.splitext(url.rstrip("/").split("/")[-1])[0])
@contextlib.contextmanager
def hiding(url: str | None) -> Iterator[None]:
"""For the work on one URL: `log_safe` also takes out the name yt-dlp's generic extractor
gives it, exactly, where yt-dlp puts it (`[generic] <name>:`). A context variable, so it
holds for yt-dlp's logger calls too — they run on the same thread as the extraction."""
name = generic_id(url) if url else ""
token = _HIDDEN.set((*_HIDDEN.get(), name) if name else _HIDDEN.get())
try:
yield
finally:
_HIDDEN.reset(token)
def log_safe(text: object) -> str:
"""yt-dlp's text (its own lines, or a DownloadError's message) with the parts that name a
person's download taken out. Every log line that carries yt-dlp's words goes through here,
not just the adapter's: yt-dlp raises the same text it has just logged, so a line that skipped
it would put back one line later exactly what the adapter took out."""
text = str(text)
for name in _HIDDEN.get():
text = text.replace(f"[generic] {name}:", "[generic] …:")
text = _URL_RE.sub(lambda m: f"{m.group(1)}/…", text)
text = _ID_PREFIX_RE.sub(r"\1…:", text)
text = _GENERIC_ID_RE.sub(r"\1…:", text)
text = _PLAYLIST_TITLE_RE.sub(r"\1…", text)
text = _TITLE_TAIL_RE.sub(r"\1…", text)
text = _BARE_ID_RE.sub(r"\1…", text)
return _PATH_RE.sub("…", text)
class _YdlLog:
"""yt-dlp's logger protocol, pointed at ours.
Without it yt-dlp explains itself to nobody: `quiet` sends its output to a stdout we do not
read, so "Requested format is not available", a signature-extraction failure or a POT refusal
reached us only as whatever text ended up in the raised exception. Measured against yt-dlp
2026.07.04: with `logger` set it calls `debug` for extraction and progress lines and
`warning`/`error` for the rest, and `quiet`/`no_warnings` do NOT gate those calls.
`debug` goes to TRACE: it is several lines per second during a download, so it shows only
when the whole process runs at TRACE. (Not under a user's own Teljes consent — that lives in
the API process's request scope, and the download child that runs yt-dlp has none.)
`error` goes to WARNING, not ERROR. yt-dlp calls it from `trouble()` just before it raises,
on EVERY failed attempt — including the ones the pipeline then recovers from with its fallback
client set. Whether the job failed is the pipeline's verdict (`download.job.failed`), and that
is the one line whose level should say so."""
def _emit(self, level: int, msg: str) -> None:
# The line goes as an ARGUMENT, not as the format string. yt-dlp writes progress as
# `[download] 53.2% of 12.00MiB`, and while stdlib logging only applies `%` formatting to
# a record that has args, this way the question never arises — and it is the shape ruff's
# `G` rules ask for. (The sink's flood limiter groups a bare `%s` by the text itself, so
# distinct yt-dlp warnings do not share one bucket — `logsink._template`.)
log.log(level, "%s", log_safe(msg), extra=logs.event("download.ytdlp"))
def debug(self, msg):
self._emit(logs.TRACE, msg)
def info(self, msg):
self._emit(logging.DEBUG, msg)
def warning(self, msg):
self._emit(logging.WARNING, msg)
def error(self, msg):
self._emit(logging.WARNING, msg)
_YDL_LOG = _YdlLog()
# Fields that affect the produced file — and therefore the cache identity.
_SIG_KEYS = (
@@ -78,7 +204,13 @@ def normalize(spec: dict) -> dict:
# rather than raising an unhandled 500 at enqueue.
try:
s["max_height"] = int(s["max_height"])
except (TypeError, ValueError):
except (TypeError, ValueError) as exc:
# "best" is the opposite of what a height cap asks for: the user set a ceiling to save
# bandwidth or disk, and the download quietly takes the largest rendition there is.
log.warning(
"the height cap on a download could not be read and was dropped",
extra=logs.event("download.format.max_height_dropped", reason=type(exc).__name__),
)
s["max_height"] = None
for b in ("embed_subs", "embed_chapters", "embed_thumbnail", "sponsorblock"):
s[b] = bool(s[b])
@@ -181,6 +313,12 @@ def build_ydl_opts(
if extra:
opts.update(extra)
# With a logger attached, `quiet` and `no_warnings` stop deciding whether yt-dlp explains itself
# and only decide where — so both stay on above and the explanation comes here instead of to a
# stdout nothing reads. AFTER the adapter merge, like `cookiefile` and the size cap: an adapter
# must not be able to silence the one account of what went wrong that a failed job leaves.
opts["logger"] = _YDL_LOG
# SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
# `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set.
#
+14 -5
View File
@@ -16,7 +16,6 @@ Five passes each run:
Runs in the API process (which mounts DOWNLOAD_ROOT). Pure disk/DB work — no YouTube quota.
"""
import logging
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
@@ -24,14 +23,13 @@ from pathlib import Path
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.config import settings
from app.downloads import storage
from app.models import DownloadJob, MediaAsset
from app.notifications import create_notification
log = logging.getLogger("siftlode.downloads.gc")
log = logs.get_logger(__name__)
# Dot-prefixed top-level trees that are NOT part of the Plex/flat asset layout — the orphan sweep
# never touches them (user clips, Plex HLS/img caches, in-progress staging, editor storyboards).
_SYSTEM_TREE = {".edits", ".plex-hls", ".plex-img-cache", ".staging", ".storyboards"}
@@ -99,6 +97,7 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
if pause_s and touched % batch == 0:
time.sleep(pause_s)
unreclaimed = 0
for entry in root.iterdir():
if entry.name.startswith("."): # .edits / .plex-* / .staging / .storyboards + any dotfile
continue
@@ -111,7 +110,9 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
f.unlink()
removed += 1
except OSError:
pass
# silent-ok: counted, not logged per file — a broken mount would put one line
# under every orphan. The total is reported once at the end of the sweep.
unreclaimed += 1
if entry.is_dir(): # prune emptied dirs (deepest first), then the channel dir itself
for d in sorted(
(p for p in entry.rglob("*") if p.is_dir()),
@@ -120,12 +121,20 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
):
try:
d.rmdir()
# silent-ok: `rmdir` refusing means the directory still holds files, which is
# the ordinary answer here — this prune only takes the ones that emptied.
except OSError:
pass
try:
entry.rmdir()
# silent-ok: the same prune one level up, at the channel directory.
except OSError:
pass
if unreclaimed:
log.warning(
"some orphaned download files could not be reclaimed",
extra=logs.event("download.gc.unreclaimed", files=unreclaimed, removed=removed),
)
return removed
+2
View File
@@ -49,6 +49,8 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
exp_s, sig = grant.split(".", 1)
try:
exp = int(exp_s)
# silent-ok: an untrusted grant that does not parse is simply not a valid grant, and the
# honest answer to it — a password prompt — is the same as for one that expired.
except ValueError:
return False
if exp < datetime.now(timezone.utc).timestamp():
+2
View File
@@ -66,6 +66,8 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
width = (data[3] << 8) + data[4]
return width, height
f.seek(length - 2, 1) # skip this segment's payload
# silent-ok: None omits width/height from the unfurl card, which is what the card does for
# every poster whose dimensions were never asked for. Nothing is claimed that is not true.
except OSError:
return None
+129 -23
View File
@@ -1,6 +1,5 @@
"""Child-side download/edit pipeline — the yt-dlp download + ffmpeg edit that runs in the spawned subprocess."""
import logging
import re
import shutil
import time
@@ -9,7 +8,7 @@ from pathlib import Path
import yt_dlp
from app import sysconfig
from app import logs, sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
@@ -24,12 +23,13 @@ from app.downloads.jobdb import (
_staging_dir,
_touch_job,
)
from app.downloads.stated import StatedFailure
from app.downloads.status import JobStatus
from app.models import DownloadJob, MediaAsset
from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset
from app.notifications import create_notification
from app.titles import normalize_title
log = logging.getLogger("siftlode.worker")
log = logs.get_logger(__name__)
class _Aborted(Exception):
@@ -98,6 +98,8 @@ def _parse_upload_date(raw) -> date | None:
return None
try:
return datetime.strptime(str(raw), "%Y%m%d").date()
# silent-ok: a date we cannot read becomes a date the card does not show, which is what the
# card shows for every source that never supplies one. Absent is the honest rendering.
except ValueError:
return None
@@ -138,7 +140,14 @@ def _make_progress_hook(job_id: int, asset_id: int):
try:
_fill_asset_meta_early(asset_id, d["info_dict"])
except Exception: # noqa: BLE001 — metadata is best-effort, never fail a download
pass
# Best-effort is about the DOWNLOAD, not about knowing. What this swallows is
# anything from a shape change in yt-dlp's info dict to the database being down,
# and all the user sees is a card with an id on it until the job finishes.
log.info(
"early metadata could not be filled in for this download",
extra=logs.event("download.meta.early_fill_failed", asset_id=asset_id),
exc_info=True,
)
if d.get("status") != "downloading":
return
if not allow_write():
@@ -299,7 +308,7 @@ def _produced_file(info: dict, staging: Path) -> Path:
and not _YTDLP_TEMP_RE.search(p.name)
]
if not cands:
raise RuntimeError("yt-dlp produced no output file")
raise StatedFailure("yt-dlp produced no output file")
return max(cands, key=lambda p: p.stat().st_size)
@@ -375,13 +384,22 @@ def _extract_with_fallback(
"job %s: clients %s failed (%s); retrying with fallback",
job_id,
clients,
str(exc)[:70],
formats.log_safe(exc)[:70],
)
for p in list(staging.iterdir()): # drop partials before the retry
try:
p.unlink()
except OSError:
pass
except OSError as exc:
# A partial left behind is not inert: the next client set resumes into it,
# so the retry this loop exists to make clean starts dirty instead.
log.info(
"a partial file could not be removed before the retry",
extra=logs.event(
"download.retry.partial_unlink_failed",
job_id=job_id,
reason=type(exc).__name__,
),
)
_set_job(job_id, progress=0, phase="downloading")
continue
raise
@@ -464,7 +482,7 @@ def _download(
try:
urlguard.check_url(url)
except urlguard.BlockedUrl as e:
raise RuntimeError(f"Refusing to download from this address ({e.reason}).") from e
raise StatedFailure(f"Refusing to download from this address ({e.reason}).") from e
# S6: attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
# this asset is that user's private, salted identity (the double gate lives in credentials).
remaining_bytes: int | None = None
@@ -579,20 +597,30 @@ def _download(
}
if job and not job.display_name and meta.title:
# display_name is VARCHAR(255) — clip a long title (see _fill_asset_meta_early).
fields["display_name"] = meta.title[:255]
fields["display_name"] = meta.title[:DISPLAY_TEXT_MAX]
# Atomic terminal write: finalizes ONLY while the row is still running, so a child that
# finishes just after a route cancel/pause can't resurrect it (see finalize_if_running).
finalized = service.finalize_if_running(db, job_id, **fields)
db.commit()
if finalized:
_notify_download_done(db, owner_id, meta.title, job_id, kind="download")
log.info("job %s done: %s", job_id, rel)
# The asset id, not `rel`: the path is built from the channel and the title.
log.info(
"job %s done: asset %s",
job_id,
asset_id,
extra=logs.event("download.job.done", job_id=job_id, asset_id=asset_id),
)
except (_Aborted, editmod.EditAborted):
# Paused/canceled mid-download or mid-transcode: free the asset for a later retry; keep the
# job's status (the route set it to paused/canceled). ref_count is adjusted by the route.
_reset_asset_pending(asset_id)
log.info("job %s aborted (pause/cancel)", job_id)
log.info(
"job %s aborted (pause/cancel)",
job_id,
extra=logs.event("download.job.aborted", job_id=job_id),
)
except _OwnerRefused as exc:
# Fail this job only. The asset is never marked `error`; it goes back to `pending` (below),
# so another user's job for the same video is claimed and downloaded normally instead of
@@ -613,7 +641,12 @@ def _download(
# it, and this one must already be out of the way. (The staging tree is still torn down in
# `finally` below — the same narrow window the pause/cancel path above has always had.)
_reset_asset_pending(asset_id)
log.info("job %s refused for its owner: %s", job_id, msg)
log.info(
"job %s refused for its owner: %s",
job_id,
formats.log_safe(msg),
extra=logs.event("download.job.refused", job_id=job_id),
)
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
msg = str(exc)[:500]
# The catalog needs the link and whether the user's cookie went with it: the same yt-dlp
@@ -626,8 +659,9 @@ def _download(
if asset:
asset.status = "error"
asset.error = msg
# Atomic fail: only while still running, so a route-set cancel/pause is never overwritten
# with `error` (error also holds the asset → would re-open the double-release class).
# Atomic fail: only while still running, so a route-set cancel/pause is never
# overwritten with `error` (error also holds the asset → would re-open the
# double-release class).
service.finalize_if_running(
db,
job_id,
@@ -638,7 +672,10 @@ def _download(
phase=None,
)
db.commit()
log.warning("job %s failed: %s", job_id, msg)
# After the write: a job the write could not fail is not a failed job. When it raises,
# the runner reports this failure with it (`runner._run`) and the worker fails the job.
# silent-ok: not silent — `_log_download_failed` is the report.
_log_download_failed(job_id, msg, diagnosis.code, exc)
finally:
# Unlink the decrypted cookie file explicitly (the rmtree below removes the whole staging
# tree too, but never leave a plaintext session cookie on disk a moment longer than needed).
@@ -647,7 +684,50 @@ def _download(
shutil.rmtree(staging, ignore_errors=True)
def _fail_edit(job_id: int, asset_id: int, msg: str) -> None:
def _failure_stack(exc: BaseException) -> dict[str, str]:
"""The `extra=` fragment that puts a failure's traceback on its line — or nothing.
Nothing for yt-dlp's DownloadError and for our own `StatedFailure` (`ffmpeg failed (1): …`):
their message is the whole story, and their stack is only the `except` that caught them.
Everything else keeps it — a TypeError on an odd info dict, a plain RuntimeError, an OSError,
a database error: that is our code or our platform failing, and the stack is the lead.
Scrubbed, link by link (`logs.scrubbed_stack`): a stack's messages are data — an OSError names
its file, and a database error raised inside `except DownloadError` carries yt-dlp's
unscrubbed text as its context."""
if isinstance(exc, (yt_dlp.utils.DownloadError, StatedFailure)):
return {}
return logs.scrubbed_stack(exc, formats.log_safe)
def _log_download_failed(job_id: int, msg: str, code: str | None, exc: BaseException) -> None:
"""The child's `download.job.failed` line. `msg` is the user's copy and keeps their URL; the
log's copy goes through `log_safe`, or it would put back one line later the URL and id the
yt-dlp adapter took out of the line before it. The stack only when the classifier could not
place the failure: a recognised one is fully described by its code."""
unplaced = code in (None, "", "unknown")
log.warning(
"job %s failed: %s",
job_id,
formats.log_safe(msg),
extra={
**logs.event(
"download.job.failed",
job_id=job_id,
error_code=code,
error_type=type(exc).__name__,
),
**(_failure_stack(exc) if unplaced else {}),
},
)
def _fail_edit(job_id: int, asset_id: int, msg: str, exc: BaseException | None = None) -> None:
"""Fail an edit job, and say so ONCE — every way an edit fails comes through here, so this is
where its line and its event code live. `exc` is the exception when there was one.
Its traceback goes with the line unless the message already tells it all (`_failure_stack`).
Logged after the write, as on the download error path."""
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
@@ -663,7 +743,22 @@ def _fail_edit(job_id: int, asset_id: int, msg: str) -> None:
phase=None,
)
db.commit()
log.warning("edit job %s failed: %s", job_id, msg)
# `msg` is the user's copy; ffmpeg's stderr tail in it can name the source file, whose path is
# built from the channel and the title.
log.warning(
"edit job %s failed: %s",
job_id,
formats.log_safe(msg),
extra={
**logs.event(
"edit.job.failed",
job_id=job_id,
asset_id=asset_id,
**({"error_type": type(exc).__name__} if exc is not None else {}),
),
**(_failure_stack(exc) if exc is not None else {}),
},
)
def _process_edit(job_id: int, asset_id: int) -> None:
@@ -715,7 +810,7 @@ def _process_edit(job_id: int, asset_id: int) -> None:
should_cancel=_cancel_poll(job_id),
)
if not dest_staging.exists():
raise RuntimeError("ffmpeg produced no output file")
raise StatedFailure("ffmpeg produced no output file")
rel = storage.edit_rel_path(user_id, asset_id, out_ext)
storage.place_file(dest_staging, settings.download_root, rel)
@@ -751,13 +846,24 @@ def _process_edit(job_id: int, asset_id: int) -> None:
db.commit()
if finalized:
_notify_download_done(db, owner_id, dl_title, job_id, kind="edit")
log.info("edit job %s done: %s", job_id, rel)
# The asset id, not `rel`: the path is built from the channel and the title.
log.info(
"edit job %s done: asset %s",
job_id,
asset_id,
extra=logs.event("edit.job.done", job_id=job_id, asset_id=asset_id),
)
except editmod.EditAborted:
# Paused/canceled mid-encode: free the asset for a later retry (ref_count is the route's job).
_reset_asset_pending(asset_id)
log.info("edit job %s aborted (pause/cancel)", job_id)
log.info(
"edit job %s aborted (pause/cancel)",
job_id,
extra=logs.event("edit.job.aborted", job_id=job_id),
)
except Exception as exc: # noqa: BLE001 — surface the failure, keep the worker alive
_fail_edit(job_id, asset_id, str(exc)[:500])
# silent-ok: not silent — `_fail_edit` is the report, and it takes `exc` for its stack.
_fail_edit(job_id, asset_id, str(exc)[:500], exc)
finally:
shutil.rmtree(staging, ignore_errors=True)
+2
View File
@@ -64,6 +64,8 @@ def _facts_from_probe(
raw = fmt.get("duration")
if raw is not None:
duration = int(float(raw))
# silent-ok: an unusable duration is stored as no duration, which the UI renders as unknown
# rather than as zero — the honest state, and the one a re-probe can still fill in.
except (TypeError, ValueError):
duration = None
return vcodec, acodec, width, height, duration
+38 -7
View File
@@ -11,7 +11,6 @@ writes the terminal job/asset state itself (done / error, or a cooperative pause
progress hook), then exits 0. An unexpected crash exits non-zero and the parent fails the job.
"""
import logging
import os
import shutil
import subprocess
@@ -22,9 +21,10 @@ from app import logsink
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import formats
from app.models import DownloadJob
log = logging.getLogger("siftlode.worker.runner")
log = logs.get_logger(__name__)
def _deprioritize() -> None:
@@ -57,7 +57,12 @@ def main() -> None:
logs.configure_logging("runner")
_deprioritize()
if len(sys.argv) != 2:
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
# noqa T201: a usage line is for the person who typed the command. Logging is already
# configured here, and in a deployment it writes JSON records — the one reader of this
# line wants the bare sentence, not a record with a request id they do not have.
print( # noqa: T201
"usage: python -m app.downloads.runner <job_id>", file=sys.stderr
)
raise SystemExit(2)
job_id = int(sys.argv[1])
# This whole process exists to run ONE job — tag every line it prints with it, so the child's
@@ -77,14 +82,40 @@ def main() -> None:
if asset_id is None:
return
_run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id)
def _run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id) -> None:
"""Run the job, and be the last stop before Python's own exception hook.
What gets past the pipeline is a failure it could not even record (the database refused the
write that fails the job). Python's hook would print it to stderr raw — and its context is the
download's own failure, whose text is the URL or ffmpeg's stderr naming the file. Here it is
one line with the whole chain scrubbed; the non-zero exit then has the worker fail the job
(`cause=child_exit`)."""
# Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
# real work, and to keep the import graph obvious: runner is a thin entry over the pipeline.
from app.downloads.pipeline import _download, _process_edit
if job_kind == "edit":
_process_edit(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
with formats.hiding(source_ref):
try:
if job_kind == "edit":
_process_edit(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
except Exception as exc: # noqa: BLE001 — whatever it is, it is reported here or raw
# noqa TRY400: the stack goes with it scrubbed (`scrubbed_stack`); `exc_info` is raw.
log.error( # noqa: TRY400
"the download child for job %s failed",
job_id,
extra={
**logs.event(
"download.child.failed", job_id=job_id, error_type=type(exc).__name__
),
**logs.scrubbed_stack(exc, formats.log_safe),
},
)
raise SystemExit(1) from None
if __name__ == "__main__":
+10 -5
View File
@@ -20,7 +20,7 @@ from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobStatus
from app.models import Channel, DownloadJob, MediaAsset, Video
from app.models import DISPLAY_TEXT_MAX, Channel, DownloadJob, MediaAsset, Video
class EditError(Exception):
@@ -87,6 +87,8 @@ def get_or_create_asset(
with db.begin_nested():
db.add(asset)
db.flush()
# silent-ok: the expected outcome of the race this savepoint exists for — two jobs wanting the
# same asset. Losing it means the other insert won, and re-selecting is the whole handling.
except IntegrityError:
asset = db.execute(stmt).scalar_one()
return asset
@@ -177,7 +179,7 @@ def enqueue(
profile_id=profile_id,
profile_snapshot=spec,
display_name=display_name
or (asset.title[:255] if asset.status == "ready" and asset.title else None),
or (asset.title[:DISPLAY_TEXT_MAX] if asset.status == "ready" and asset.title else None),
status="queued",
queue_pos=next_queue_pos(db),
)
@@ -196,10 +198,13 @@ def enqueue(
def _clip_title(source_title: str | None, display_name: str | None) -> str:
# Clipped to DISPLAY_TEXT_MAX, not to a literal: this feeds `DownloadJob.display_name` through
# `enqueue_edit`, the same column `display_text` guards. A machine-made title, so it is clipped
# rather than run through `display_text` — blank here means "no title", not None.
if display_name:
return display_name[:255]
return display_name[:DISPLAY_TEXT_MAX]
base = source_title or "Clip"
return f"{base} (clip)"[:255]
return f"{base} (clip)"[:DISPLAY_TEXT_MAX]
def enqueue_edit(
@@ -253,7 +258,7 @@ def enqueue_edit(
source_asset_id=src_asset.id,
edit_spec=spec,
profile_snapshot={},
display_name=display_name or asset.title,
display_name=display_name or asset.title[:DISPLAY_TEXT_MAX],
status="queued",
queue_pos=next_queue_pos(db),
)
+13
View File
@@ -0,0 +1,13 @@
"""The failures the download pipeline raises itself, with the whole story in the message.
`ffmpeg failed (1): …`, `yt-dlp produced no output file`, a post that shows no video: the message
says what happened, and the stack is only the `except` that caught it. A failure line leaves that
stack off (`pipeline._failure_stack`) — but ONLY for these. A plain `RuntimeError` is also what
Python raises for a bug of ours (a dict changed size during iteration, a generator's
StopIteration, `RecursionError`, `NotImplementedError`), and that one's stack is the only lead.
"""
class StatedFailure(RuntimeError):
"""A failure we raised on purpose, whose message is its full explanation. A `RuntimeError` so
every `except RuntimeError` that caught these before still does."""
+21 -3
View File
@@ -19,8 +19,11 @@ from datetime import date
from pathlib import Path
from xml.sax.saxutils import escape
from app import logs
from app.downloads import mediakind
log = logs.get_logger(__name__)
# Reserve room for the "_id.ext" suffix + parent dirs within the 255-char component limit.
_MAX_TITLE = 120
# User-facing (Content-Disposition) names can be longer than an on-disk component.
@@ -294,7 +297,13 @@ def write_sidecars(download_root: str, rel: str, meta: MediaMeta, thumb_src: Pat
if thumb_src and thumb_src.exists():
shutil.copyfile(str(thumb_src), str(_sidecar(media, ".jpg")))
return True
except OSError:
except OSError as exc:
# The download itself still completes, so the job reads as done while the library it was
# downloaded FOR has no .nfo to read and no poster to show.
log.warning(
"the sidecars beside a download could not be written",
extra=logs.event("download.sidecars.write_failed", reason=type(exc).__name__),
)
return False
@@ -304,14 +313,23 @@ def delete_asset_files(download_root: str, rel: str) -> None:
for p in (media, _sidecar(media, ".nfo"), _sidecar(media, ".jpg")):
try:
p.unlink()
except OSError:
pass
except FileNotFoundError:
pass # silent-ok: already gone is the outcome this function wanted
except OSError as exc:
# The row goes and the bytes stay. Nothing else ever revisits this path, so the disk
# quietly keeps a file no account has any way to see, let alone delete.
log.info(
"a deleted download left a file behind",
extra=logs.event("download.asset.file_unlink_failed", reason=type(exc).__name__),
)
# Prune empty Season/Channel dirs up to (but not including) the root.
root = Path(download_root).resolve()
parent = media.parent
while parent != root and root in parent.parents:
try:
parent.rmdir() # only succeeds if empty
# silent-ok: this `rmdir` failing is the loop's stop condition — the parent still
# holds another download's files, so the prune has gone as far up as it may.
except OSError:
break
parent = parent.parent
+2 -2
View File
@@ -25,7 +25,7 @@ from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import edit as editmod
from app.downloads import mediakind, probe, quota, service, storage
from app.models import DownloadJob, MediaAsset
from app.models import DownloadJob, MediaAsset, display_text
def _title_from_filename(filename: str) -> str:
@@ -99,7 +99,7 @@ def ingest_upload(
source_kind="upload",
source_ref=ref,
profile_snapshot={},
display_name=(display_name or "").strip() or None,
display_name=display_text(display_name),
status="done",
progress=100,
queue_pos=service.next_queue_pos(db),
+15 -7
View File
@@ -11,7 +11,6 @@ to be a person telling them nothing arrived.
"""
import html
import logging
import re
import smtplib
import ssl
@@ -21,15 +20,14 @@ from email.utils import formatdate
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import mailtext, sysconfig
from app import logs, mailtext, sysconfig
from app.config import settings
from app.db import SessionLocal
from app.models import User
from app.notifications import create_notification
from app.ratelimit import RateLimiter
log = logging.getLogger("siftlode.email")
log = logs.get_logger(__name__)
# One admin notice an hour per reason. A relay that is down fails every send, and the inbox must
# stay readable; the log keeps the full record either way.
_alert_throttle = RateLimiter(max_events=1, window_seconds=3600)
@@ -155,7 +153,15 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
return False
c = _smtp()
if not (c["host"] and c["user"] and c["password"]):
log.info("SMTP not configured; skipping email %r to %s", subject, recipients)
# The count, not the addresses: `redact_text` rewrites every one of them to `[email]`
# before either sink sees the line, so the list only ever printed as noise. The subject
# stays — the templates are static, and the one that interpolates (`admin_new_request`)
# interpolates an address, which the same redactor covers.
log.info(
"SMTP not configured; skipping email %r to %d recipient(s)",
subject,
len(recipients), # log-pii-ok: how many, not who
)
_alert_admins("not_configured", subject, recipients)
return False
msg = EmailMessage()
@@ -176,10 +182,12 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
s.starttls(context=ssl.create_default_context())
s.login(c["user"], c["password"])
s.send_message(msg)
log.info("Sent email %r to %s", subject, recipients)
# log-pii-ok: how many, not who
log.info("Sent email %r to %d recipient(s)", subject, len(recipients))
return True
except Exception:
log.exception("SMTP send failed (%r to %s)", subject, recipients)
# log-pii-ok: how many, not who
log.exception("SMTP send failed (%r to %d recipient(s))", subject, len(recipients))
_alert_admins("send_failed", subject, recipients)
return False
+4 -3
View File
@@ -30,7 +30,7 @@ from app.models import (
Video,
VideoState,
)
from app.utils import iso, to_tsquery_str
from app.utils import iso, like_escape, to_tsquery_str
def _channel_url(channel_id: str, handle: str | None) -> str:
@@ -272,8 +272,9 @@ def filtered_query(
# `unaccent(channels.title) ILIKE '%q%'` straight into the per-video predicate defeated the
# search_vector GIN — the planner detoasted every non-matching tsvector to test the `@@`
# (~2.6s/search on the prod-sized set); channel_id.in_(...) keeps the GIN as a BitmapOr branch
# (PERF-12). Empty match -> false() so the OR collapses to the text match alone.
like = func.unaccent(f"%{q}%")
# (PERF-12). Empty match -> false() so the OR collapses to the text match alone. The query
# is a literal inside the pattern: `_` in `my_channel` is not a wildcard.
like = func.unaccent(f"%{like_escape(q)}%")
matching_channel_ids = (
db.execute(select(Channel.id).where(func.unaccent(Channel.title).ilike(like)))
.scalars()
+3
View File
@@ -0,0 +1,3 @@
"""Bug reports built from debug sessions (E16): the files an issue carries, and where they go —
downloaded as a zip, or (S4) sent to a Gitea issue tracker. The session itself — what is recorded
and tagged — is the core module `app/debugsession.py`."""
+79
View File
@@ -0,0 +1,79 @@
"""What an issue from a debug session carries, as files — one definition for the zip export and the
Gitea upload, so the two cannot drift apart.
* `issue.md` — the title and the body the user wrote and reviewed in the composer (rendered there,
in their language; the server does not rewrite it);
* `client.log` / `server.log` — the session's evidence (`debugsession.bundle`) minus the rows the
user took out, one JSON object per line, browser rows and server rows apart. `.log`, not
`.jsonl`: Gitea refuses that extension (measured, plans/e16 S0);
* the screenshots, in the order they were taken.
Every row is already redacted twice (at capture, and again when stored), and nothing here adds a
field the review list did not show.
"""
import io
import json
import zipfile
from dataclasses import dataclass
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import debugsession, logsink
from app.models import DebugScreenshot, DebugSession
@dataclass(frozen=True)
class IssueFile:
name: str
data: bytes
mime: str
def title_of(session: DebugSession) -> str:
# Not a sentence of ours: the composer always saves a title in the user's language, and a
# session exported before it ever saved is named by its date alone, in no language at all.
title = ((session.draft or {}).get("title") or "").strip()
return title or f"{session.started_at:%Y-%m-%d %H:%M} · {session.id}"
def body_of(session: DebugSession) -> str:
return ((session.draft or {}).get("body") or "").strip()
def _lines(rows) -> bytes:
return "".join(
json.dumps(logsink.serialize(r), ensure_ascii=False, default=str) + "\n" for r in rows
).encode()
def files(db: Session, session: DebugSession) -> list[IssueFile]:
"""The session's issue as files: the text, the two logs, the screenshots."""
rows, _ = debugsession.bundle(db, session, drop_excluded=True)
text = f"# {title_of(session)}\n\n{body_of(session)}\n"
out = [
IssueFile("issue.md", text.encode(), "text/markdown"),
IssueFile("client.log", _lines(r for r in rows if r.source == "web"), "text/plain"),
IssueFile("server.log", _lines(r for r in rows if r.source != "web"), "text/plain"),
]
shots = db.scalars(
select(DebugScreenshot)
.where(DebugScreenshot.session_id == session.id)
.order_by(DebugScreenshot.id)
)
for n, shot in enumerate(shots, start=1):
ext = debugsession.IMAGE_TYPES[shot.mime]
name = f"{debugsession.SCREENSHOT_PREFIX}{n}.{ext}"
out.append(IssueFile(name, shot.data, shot.mime))
return out
def zipped(items: list[IssueFile]) -> bytes:
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as z:
for item in items:
# The images are compressed already: deflating them again costs time and wins nothing.
method = zipfile.ZIP_STORED if item.mime.startswith("image/") else zipfile.ZIP_DEFLATED
z.writestr(item.name, item.data, compress_type=method)
return buf.getvalue()
+288
View File
@@ -0,0 +1,288 @@
"""The Gitea issue tracker a debug session's issue goes to (E16 S4) — measured against Gitea 1.27.1
before a line of this was written (plans/e16-debug-issue.md, S0).
Configured by an admin (sysconfig `issues_gitea_url` / `_repo` / `_token`, the token encrypted at
rest). The repository must be PRIVATE: an issue carries its reporter's log — page titles, filter
values, at Teljes (TRACE) what they typed into search — and a public repository would publish it. That is
checked on every send, not only in the admin's "test connection", because a repository can be made
public after the fact.
What an issue is made of comes from `bundle.files` (the same files the zip export carries). Sending
is three kinds of call, measured:
* `POST /repos/{repo}/issues` — the title and the body the user wrote;
* `POST /repos/{repo}/issues/{n}/assets?name=…` — ONE file per request (field `attachment`);
* `PATCH /repos/{repo}/issues/{n}` — the body again, with the screenshots shown inline once their
links exist.
A private repository's attachment answers 404 to a browser not signed in to Gitea, which is why the
composer's own preview shows our copies, never these links.
Delivery keeps its progress in the session's draft (`delivery`: the issue number, the files still to
upload, the links of those uploaded, whether the body shows the screenshots yet), so a failure
half-way keeps the issue that exists and a retry does only what is missing — never a second issue.
Every step is checkpointed (the caller's `db.commit`), so what did happen is kept whatever fails
next — and the first checkpoint comes before the first call, with the tracker's settings read ABOVE
it, so neither a row lock nor a read snapshot nor a pooled connection is held across the calls. The
caller claims the session for the send first (`debugsession.claim_send`) and passes the claim's
token in, which is what keeps a second send, an autosave or a screenshot out of the way; the claim
is renewed by every progress report, and a report whose claim is gone raises rather than writing
over the send that took it. `debugsession.not_uploaded` / `images_pending` read that progress.
"""
from collections.abc import Callable, Iterator
from contextlib import contextmanager
from typing import Any, Literal
import httpx
from sqlalchemy.orm import Session
from app import debugsession, logs, sysconfig
from app.issues import bundle
from app.models import DebugSession
log = logs.get_logger(__name__)
TIMEOUT_SECONDS = 20.0
Reason = Literal[
"not_configured", # url, repository or token missing
"unreachable", # no answer, or a 5xx
"bad_token", # 401/403: the token is wrong or lacks write:issue / read:repository
"repo_not_found", # 404 on the repository
"public_repo", # the repository is public — an issue would publish the reporter's log
"issues_disabled", # the repository has its issue tracker switched off
"rejected", # any other 4xx (a file type the instance does not allow, …), or an answer that is
# not the API's (a redirect — an http:// address behind https —, a page that is not JSON)
]
class GiteaError(Exception):
def __init__(self, reason: Reason) -> None:
super().__init__(reason)
self.reason: Reason = reason
def _settings(db: Session) -> tuple[str, str, str]:
return (
sysconfig.get_str(db, "issues_gitea_url").strip().rstrip("/"),
sysconfig.get_str(db, "issues_gitea_repo").strip().strip("/"),
sysconfig.get_str(db, "issues_gitea_token").strip(),
)
def target(db: Session) -> dict | None:
"""Where a report goes, for the composer to say so — the repository — or None when no tracker is
configured (the report can then only be downloaded)."""
url, repo, token = _settings(db)
return {"repo": repo} if url and repo and token else None
@contextmanager
def _client(settings: tuple[str, str, str]) -> Iterator[tuple[httpx.Client, str]]:
# Takes the settings already read, not the session: reading them HERE opened a transaction one
# line after `deliver` had ended one, and held it across the calls below (deep-review round 4).
url, repo, token = settings
if not (url and repo and token):
raise GiteaError("not_configured")
with httpx.Client(
base_url=f"{url}/api/v1",
headers={"Authorization": f"token {token}", "Accept": "application/json"},
timeout=TIMEOUT_SECONDS,
) as client:
yield client, repo
def _call(client: httpx.Client, method: str, path: str, **kwargs) -> httpx.Response:
try:
r = client.request(method, path, **kwargs)
except httpx.HTTPError as exc:
log.warning(
"the issue tracker did not answer",
extra=logs.event("issues.gitea.unreachable", path=path, error=type(exc).__name__),
)
raise GiteaError("unreachable") from None
if r.status_code < 300:
return r
# The status only: Gitea's message can quote back what was sent — the user's own title.
log.warning(
"the issue tracker refused a call",
extra=logs.event("issues.gitea.refused", path=path, status=r.status_code),
)
if r.status_code in (401, 403):
raise GiteaError("bad_token")
if r.status_code == 404:
raise GiteaError("repo_not_found")
if r.status_code >= 500:
raise GiteaError("unreachable")
raise GiteaError("rejected")
def _json(r: httpx.Response, *keys: str) -> dict[str, Any]:
"""The answer as the API's JSON, holding `keys` — anything else is refused, not a crash, so a
step after the issue was created cannot take down the request that holds its record.
The CREATE call is the one place this cannot keep that record: its answer IS the record. An
answer it cannot read there leaves the session unsent, and a retry opens another issue — the
price of not being able to tell a refused create from an unreadable one."""
try:
data = r.json()
# silent-ok: not JSON is one of the answers refused (and logged) just below.
except ValueError:
data = None
if not isinstance(data, dict) or any(k not in data for k in keys):
log.warning(
"the issue tracker answered something that is not its API",
extra=logs.event("issues.gitea.unexpected", status=r.status_code),
)
raise GiteaError("rejected")
return data
def _private_repo(client: httpx.Client, repo: str) -> dict:
data = _json(_call(client, "GET", f"/repos/{repo}"))
if not data.get("private"):
raise GiteaError("public_repo")
if not data.get("has_issues"):
raise GiteaError("issues_disabled")
return data
def check(db: Session) -> dict:
"""The admin's "test connection": the repository answers, is private and takes issues."""
with _client(_settings(db)) as (client, repo):
data = _private_repo(client, repo)
return {"repo": data.get("full_name") or repo, "url": data.get("html_url")}
def preview(db: Session, text: str) -> str:
"""The body as Gitea itself will render it (its markdown flavour, its sanitiser).
**No `Context`, deliberately — a repository context here does not fail the call, it kills the
tracker.** With one, Gitea's post-processor turns every hex-shaped word into a commit link, and
on the API's render path the commit checker it asks is nil: the whole Gitea process segfaults
and restarts, taking the git server (and, here, the container registry) down with it. Every
report carries hex — a tab id, a request id — so this was not an edge case but the only case.
Measured on throwaway instances (2026-09-20): `{"Mode": "gfm", "Context": "<repo>"}` with
`5645b1c0ae01` in the text crashes 1.26.4 and 1.27.1 and is fixed in 1.27.3 — but a self-hoster
runs whatever version they run, so the payload stays safe here rather than relying on theirs.
Dropping `Context` is not a downgrade of the preview: measured against the same body, the HTML
keeps its tables, its task lists and its highlighted code blocks. It only stops resolving
repo-relative links (a hash, a `#123`), which the issue page itself still resolves.
"""
with _client(_settings(db)) as (client, _repo):
r = _call(client, "POST", "/markdown", json={"Text": text, "Mode": "gfm"})
return r.text
def deliver(
db: Session,
session: DebugSession,
*,
token: str,
checkpoint: Callable[[], None],
before_create: Callable[[], None] | None = None,
on_created: Callable[[], None] | None = None,
) -> str:
"""Send the session's issue, or finish sending it; returns its URL.
`token` is the caller's claim on this session (`debugsession.claim_send`). Every progress report
renews it and checks the row still holds it, so a send that outlived its lease stops here rather
than writing over the one that took over (`debugsession.SendClaimLost`).
`checkpoint` persists what has happened so far and is called after every step (the caller's
commit — this module does not decide when a transaction ends). `before_create` runs right before
the tracker is asked to open a NEW issue, after the checks that can refuse it without asking,
and may raise to stop it; `on_created` runs right after one was opened, in time to be committed
by the SAME checkpoint that first writes `issue_url` — anything the caller must not be able to
lose separately from the link belongs there.
"""
state = debugsession.delivery(session)
files = bundle.files(db, session)
attachments = [f for f in files if f.name != "issue.md"]
settings = _settings(db)
# Reading the bundle and the tracker's settings opened a transaction; end it before the first
# call, so nothing holds a snapshot (and an idle connection) across minutes of tracker I/O.
# The settings are read HERE and handed to `_client`, not read inside it: reading them there
# was a SELECT one line after this commit, which reopened the transaction and held it — idle,
# with its snapshot pinned — across the repository check and the create (deep-review round 4
# measured it in pg_stat_activity). A comment saying otherwise sat right here.
checkpoint()
with _client(settings) as (client, repo):
_private_repo(client, repo)
if "number" not in state:
if before_create is not None:
before_create()
created = _json(
_call(
client,
"POST",
f"/repos/{repo}/issues",
json={"title": bundle.title_of(session), "body": bundle.body_of(session)},
),
"number",
"html_url",
)
# Built ON the state read at the start so nothing already in it is dropped. The claim
# is NOT among those things, whatever this comment used to say: `set_delivery` strips
# any `sending` passed to it and stamps the row's own from `token` (round 4 — removing
# the spread left the whole suite green, because the claim never travelled here).
state = {
**state,
"number": created["number"],
"pending": [f.name for f in attachments],
"links": {},
}
session.issue_url = created["html_url"]
try:
debugsession.set_delivery(db, session, state, token)
except debugsession.SendClaimLost:
# The one step where losing the claim costs something irreversible: the issue is
# already open on the tracker and this is what would have recorded its link. We
# still must not write — another send owns the row and is opening its own issue —
# so say which issue was left behind, because nothing else will.
# noqa TRY400: the traceback is one SendClaimLost carrying the session id and
# nothing else; the issue number below is the whole news.
log.error( # noqa: TRY400
"an issue was created by a send that had already lost its claim",
extra=logs.event(
"issues.gitea.orphaned",
number=created["number"],
session_id=session.id,
),
)
raise
if on_created is not None:
on_created()
checkpoint()
log.info(
"issue created",
extra=logs.event("issues.gitea.created", number=created["number"]),
)
number = state["number"]
for item in attachments:
if item.name not in state["pending"]:
continue
asset = _json(
_call(
client,
"POST",
f"/repos/{repo}/issues/{number}/assets",
params={"name": item.name},
files={"attachment": (item.name, item.data, item.mime)},
),
"browser_download_url",
)
state = {
**state,
"pending": [n for n in state["pending"] if n != item.name],
"links": {**state["links"], item.name: asset["browser_download_url"]},
}
debugsession.set_delivery(db, session, state, token)
checkpoint()
shots = [n for n in state["links"] if n.startswith(debugsession.SCREENSHOT_PREFIX)]
if shots and not state.get("images_shown"):
images = "\n".join(f"![{n}]({state['links'][n]})" for n in sorted(shots))
body = f"{bundle.body_of(session)}\n\n---\n\n{images}"
_call(client, "PATCH", f"/repos/{repo}/issues/{number}", json={"body": body})
debugsession.set_delivery(db, session, {**state, "images_shown": True}, token)
checkpoint()
return session.issue_url or ""
+175 -22
View File
@@ -15,7 +15,7 @@ issue attachment will carry, so it is deliberately flat and stable.
Call sites stay ordinary stdlib logging:
log = logging.getLogger("siftlode.downloads")
log = logs.get_logger(__name__) # app/downloads/pipeline.py -> siftlode.downloads.pipeline
log.info("job finished", extra=logs.event("download.job.done", job_id=7, bytes=n))
`event()` is optional — a bare `log.info("…")` still comes out well-formed — but the dotted code is
@@ -37,7 +37,9 @@ import re
import sys
import threading
import time
import traceback
import uuid
from collections.abc import Callable
from datetime import datetime, timezone
from typing import Any
from urllib.parse import parse_qsl
@@ -98,12 +100,14 @@ class _RequestScope:
request; every copied context shares the same object, so what the dependency writes is seen by
the handler and by the request line itself."""
__slots__ = ("user_id", "level")
__slots__ = ("user_id", "level", "debug_session_id")
def __init__(self) -> None:
self.user_id: int | None = None
# The user's consented diagnostics level as a logging number; None at the base level.
self.level: int | None = None
# The user's own open debug session this request named (E16), else None.
self.debug_session_id: str | None = None
_scope: contextvars.ContextVar[_RequestScope | None] = contextvars.ContextVar(
@@ -111,11 +115,23 @@ _scope: contextvars.ContextVar[_RequestScope | None] = contextvars.ContextVar(
)
def set_user(user_id: int | None, level: str = "INFO", until: datetime | None = None) -> None:
def set_user(
user_id: int | None,
level: str = "INFO",
until: datetime | None = None,
debug_session: str | None = None,
) -> None:
"""Attribute the rest of this request's log lines to `user_id`, and log them at the user's
consented diagnostics `level` until `until` (E22 S5; "INFO" = no elevation). Called from the
auth dependency — the one place the session is actually resolved into an account — rather than
from the request middleware, which runs OUTSIDE the session middleware."""
from the request middleware, which runs OUTSIDE the session middleware.
`debug_session` tags every line with the user's open debug session (E16). The caller has
already checked it is theirs and open; a session's level arrives here as `level`/`until`.
The root logger's floor is tracked per elevation: a request without the session's header (an
`<img>`, a `<video>` — the browser sends those itself) carries only the user's own level, and
must not lower the floor a concurrent request of the session still needs."""
_user_id.set(user_id)
scope = _scope.get()
if scope is None: # outside log_requests (a script, a unit test): this context only
@@ -123,9 +139,11 @@ def set_user(user_id: int | None, level: str = "INFO", until: datetime | None =
_scope.set(scope)
elevated = level != "INFO" and until is not None and until > datetime.now(timezone.utc)
scope.user_id = user_id
scope.debug_session_id = debug_session
scope.level = resolve_level(level) if elevated else None
if user_id is not None:
_track_elevation(user_id, scope.level, until)
key = user_id if debug_session is None else (user_id, debug_session)
_track_elevation(key, scope.level, until)
def set_job(job_id: int | None) -> None:
@@ -152,11 +170,15 @@ def bind(**fields: Any):
_CONTEXT_VARS[key].reset(token)
def event(code: str, **attrs: Any) -> dict[str, Any]:
def event(code: str, /, **attrs: Any) -> dict[str, Any]:
"""`extra=` payload for a log call: a stable dotted event code plus structured attributes.
Returns a plain dict so it composes with stdlib logging and costs nothing when the level is off
(the dict is built either way; keep the attrs cheap — no DB reads inside the call)."""
(the dict is built either way; keep the attrs cheap — no DB reads inside the call).
`code` is positional-only so that no attribute name is reserved: `event("x", code=...)` means
the attribute, not this parameter. Without the `/` it was a TypeError, and `code` is exactly
the word a caller reaches for when the thing that failed has one of its own."""
return {"event": code, "attrs": attrs}
@@ -259,11 +281,53 @@ def redact(value: Any, *, key: str = "") -> Any:
# --- formatters ----------------------------------------------------------------------------
_SCRUBBED_STACK = "scrubbed_stack"
def scrubbed_stack(exc: BaseException, scrub: Callable[[BaseException], str]) -> dict[str, str]:
"""`extra=` fragment: `exc`'s traceback, rendered where an `exc_info` stack goes, with the
message of every exception in it put through `scrub`.
For a failure whose message can name someone's data (a download's URL, a file path, a video
id). The frames name code and are kept whole; the text at the foot of each link is the data.
A plain `exc_info` renders the chain too — the `__cause__`/`__context__` a DB error raised
inside `except DownloadError` carries — so scrubbing only the line's own message is not
enough. Merge it into the call's `extra`: `{**event(...), **scrubbed_stack(...)}`."""
links: list[BaseException] = []
link: BaseException | None = exc
while link is not None and all(link is not seen for seen in links):
links.append(link)
link = link.__cause__ or (None if link.__suppress_context__ else link.__context__)
parts: list[str] = []
for i in range(len(links) - 1, -1, -1): # the oldest first, as Python prints a chain
link = links[i]
kind = type(link)
name = kind.__qualname__
if kind.__module__ not in ("builtins", "__main__"):
name = f"{kind.__module__}.{name}"
try:
message = scrub(link)
# silent-ok: the message is what could not be rendered, and the placeholder says so — the
# caller is in the middle of reporting one failure and must not get a second from here.
except Exception: # noqa: BLE001 — a `__str__` that raises, of any shape
message = "<unprintable>"
frames = "".join(traceback.format_tb(link.__traceback__))
head = f"Traceback (most recent call last):\n{frames}" if frames else ""
parts.append(f"{head}{name}: {message}" if message else f"{head}{name}")
if i:
parts.append(
"\nThe above exception was the direct cause of the following exception:\n"
if links[i - 1].__cause__ is link
else "\nDuring handling of the above exception, another exception occurred:\n"
)
return {_SCRUBBED_STACK: "\n".join(parts)}
# Attributes every LogRecord carries; anything else a caller passed via `extra=` is a structured
# attribute and is merged into `attrs` (so `extra={"job_id": 7}` works as well as `event(...)`).
_RECORD_STANDARD = frozenset(
vars(logging.LogRecord("", 0, "", 0, "", None, None)).keys()
| {"message", "asctime", "taskName", "event", "attrs", PERSIST, "elevated"}
| {"message", "asctime", "taskName", "event", "attrs", PERSIST, "elevated", _SCRUBBED_STACK}
)
@@ -280,6 +344,12 @@ class ContextFilter(logging.Filter):
scope = _scope.get()
if scope is not None and scope.user_id is not None and not hasattr(record, "user_id"):
record.user_id = scope.user_id
if (
scope is not None
and scope.debug_session_id is not None
and not hasattr(record, "debug_session_id")
):
record.debug_session_id = scope.debug_session_id
return True
@@ -314,6 +384,8 @@ def _safe_message(record: logging.LogRecord) -> str:
must not take down the caller, so the failure is reported IN the line instead."""
try:
return record.getMessage()
# silent-ok: the failure is reported IN the returned line. Logging it would re-enter this same
# formatter with a record built out of the one that just failed.
except Exception as exc: # noqa: BLE001 — any formatting error, reported not raised
return f"{record.msg!r} (log formatting failed: {exc})"
@@ -342,6 +414,8 @@ def record_fields(record: logging.LogRecord) -> dict[str, Any]:
fields["attrs"] = redact(attrs)
if record.exc_info:
fields["exc"] = redact_text(logging.Formatter().formatException(record.exc_info))
elif getattr(record, _SCRUBBED_STACK, None):
fields["exc"] = redact_text(getattr(record, _SCRUBBED_STACK))
elif record.exc_text:
# A QueueHandler-style hand-off formats the traceback into exc_text and drops exc_info.
fields["exc"] = redact_text(record.exc_text)
@@ -375,6 +449,8 @@ class TextFormatter(logging.Formatter):
line += f" {json.dumps(redact(attrs), ensure_ascii=False, default=str)}"
if record.exc_info:
line += "\n" + redact_text(self.formatException(record.exc_info))
elif getattr(record, _SCRUBBED_STACK, None):
line += "\n" + redact_text(getattr(record, _SCRUBBED_STACK))
return line
@@ -390,7 +466,6 @@ _THIRD_PARTY_LEVELS = {
"httpx": "WARNING",
"httpcore": "WARNING",
"asyncio": "WARNING",
"multipart": "WARNING",
"PIL": "WARNING",
"alembic": "INFO",
# uvicorn's own startup/shutdown/WebSocket lines are worth keeping; its ACCESS log is not —
@@ -401,6 +476,28 @@ _THIRD_PARTY_LEVELS = {
}
def get_logger(name: str) -> logging.Logger:
"""The logger for one module: `logs.get_logger(__name__)` in `app/plex/sync.py` is
`siftlode.plex.sync`.
The names were literals before, and 25 of them covered 34 modules — six different Plex modules
all called themselves `siftlode.plex`, so neither a level pinned per logger nor E22.6's console
filter could tell a sync line from a stream line. Derived from `__name__` there is exactly one
name per module, it cannot drift from the file it sits in, and `test_logger_names.py` fails if a
module hand-rolls one again.
`__main__` is resolved to the module it really is. The worker runs as `python -m app.worker`
and spawns `python -m app.downloads.runner`, so in the two long-running processes `__name__`
is `__main__` — taken at its word, both would log as the bare root. `-m` records the real name
on the module's spec; a plain script has none and stays the root."""
if name == "__main__":
spec = getattr(sys.modules.get("__main__"), "__spec__", None)
name = getattr(spec, "name", None) or name
stem = name.removeprefix("app.") if name.startswith("app.") else ""
# logger-name-ok: this call IS the derivation
return logging.getLogger(f"siftlode.{stem}" if stem else "siftlode")
def resolve_level(name: str) -> int:
"""Level name → number. Both ways in (the `LOG_LEVEL` env and the `log_level` admin knob) are
already held to the `LogLevel` choices, so an unknown name here is a programming error."""
@@ -469,18 +566,21 @@ def current_level() -> str:
# The base level (`LOG_LEVEL`, then the admin knob). NOTSET until configure_logging runs — a unit
# test, a script — so the gate lets everything through there, as it did before the gate existed.
_base_level = logging.NOTSET
# user id → (level number, expiry) for every elevation this process has seen and not yet expired.
_elevations: dict[int, tuple[int, datetime]] = {}
# → (level number, expiry) for every elevation this process has seen and not yet expired: a user's
# own, by user id, and a debug session's (E16), by (user id, session id) — its requests alone carry
# its level, so it is tracked apart from the user's own requests.
_ElevationKey = int | tuple[int, str]
_elevations: dict[_ElevationKey, tuple[int, datetime]] = {}
_elevations_lock = threading.Lock()
_root_applied: int | None = None
def _track_elevation(user_id: int, level: int | None, until: datetime | None) -> None:
def _track_elevation(key: _ElevationKey, level: int | None, until: datetime | None) -> None:
with _elevations_lock:
if level is None or until is None:
_elevations.pop(user_id, None)
_elevations.pop(key, None)
else:
_elevations[user_id] = (level, until)
_elevations[key] = (level, until)
active = bool(_elevations)
if active or _root_applied != _base_level:
_apply_root_level()
@@ -497,8 +597,8 @@ def _apply_root_level(force: bool = False) -> None:
# different answer would otherwise apply them in either order, and the older one could win —
# the root back at the base while an elevation is still active.
with _elevations_lock:
for user_id in [u for u, (_, until) in _elevations.items() if until <= now]:
del _elevations[user_id]
for key in [k for k, (_, until) in _elevations.items() if until <= now]:
del _elevations[key]
lowest = min([_base_level, *(lvl for lvl, _ in _elevations.values())])
if force or lowest != _root_applied:
logging.getLogger().setLevel(lowest)
@@ -535,13 +635,24 @@ def forget_user(user_id: int) -> None:
del _erased[stale]
_erased[user_id] = now
with _elevations_lock:
_elevations.pop(user_id, None)
for key in [
k for k in _elevations if k == user_id or (isinstance(k, tuple) and k[0] == user_id)
]:
del _elevations[key]
# That pop can have removed the lowest elevation, and the root level is the lowest of the base
# and every unexpired one. Recomputed here rather than left to whichever later request happens
# to call `_track_elevation` — the invariant `_apply_root_level` documents holds at every pop.
_apply_root_level()
def end_session_elevation(session_id: str) -> None:
"""A debug session was stopped: its elevation ends now, not at its cap."""
with _elevations_lock:
for key in [k for k in _elevations if isinstance(k, tuple) and k[1] == session_id]:
del _elevations[key]
_apply_root_level()
def is_erased(user_id: int | None) -> bool:
if user_id is None:
return False
@@ -562,11 +673,37 @@ def current_source() -> str:
_REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9._-]{8,64}$")
# Paths whose success is logged at DEBUG rather than INFO: the container healthcheck runs every 15s
# (5.7k lines/day saying nothing) and hashed static assets are served by the hundred per page load.
# A FAILURE on either still logs at WARNING/ERROR — this lowers the floor, it does not hide.
_QUIET_PREFIXES = ("/healthz", "/assets/", "/static/")
# (5.7k lines/day saying nothing), hashed static assets are served by the hundred per page load, and
# the admin log console reads the stored log — its live tail every few seconds, so at INFO the log
# would fill with the console watching itself. A FAILURE on any of them still logs at WARNING/ERROR
# — this lowers the floor, it does not hide.
_QUIET_PREFIXES = ("/healthz", "/assets/", "/static/", "/api/admin/logs")
_request_log = logging.getLogger("siftlode.http")
# The reads the frontend POLLS (a `useLiveQuery`, the `me` heartbeat, the sync status, the subtitle
# progress), by route template, GET only — logged at DEBUG like the prefixes above. One open tab asked
# for the two unread counts, `/api/me` and the download index every 30 s, and at INFO those were ~95%
# of the stored log (measured on prod 2026-09-19): 11k lines a day per tab burying every line someone
# caused. The same route answered with a failure still logs at WARNING/ERROR.
_POLLED_ROUTES = frozenset(
{
"/api/me",
"/api/me/notifications/unread_count",
"/api/messages/unread_count",
"/api/downloads",
"/api/downloads/index",
"/api/downloads/usage",
"/api/sync/my-status",
"/api/admin/scheduler",
"/api/admin/downloads",
"/api/admin/downloads/storage",
"/api/plex/subtitle/{rating_key}/{sub_ord}/status",
}
)
# Not `siftlode.logs`: this is the replacement for uvicorn's ACCESS log, and `siftlode.http` is
# what an operator pins a level on when they want request lines and nothing else. The module it
# happens to live in is not what the name is about.
_request_log = logging.getLogger("siftlode.http") # logger-name-ok: named for the traffic
async def log_requests(request, call_next):
@@ -608,7 +745,9 @@ async def log_requests(request, call_next):
raise
duration_ms = round((time.perf_counter() - started) * 1000, 1)
status = response.status_code
quiet = request.url.path.startswith(_QUIET_PREFIXES)
quiet = request.url.path.startswith(_QUIET_PREFIXES) or (
request.method == "GET" and _route_template(request) in _POLLED_ROUTES
)
if status >= 500:
level = logging.ERROR
elif status >= 400:
@@ -634,6 +773,10 @@ async def log_requests(request, call_next):
extra={**event("http.request", **attrs), **_request_owner(request, level)},
)
response.headers["X-Request-Id"] = rid
# The browser logs its own copy of this line (frontend lib/http.ts) and must pick the same level:
# it learns "this one is quiet" here instead of keeping a second copy of the polled-route list.
if level == logging.DEBUG:
response.headers["X-Log-Quiet"] = "1"
# The template, so the browser's copy of this line groups the same way (frontend lib/http.ts).
# Only a MATCHED route's: a template never carries a path parameter's value (a share token stays
# `{token}`), while the raw-path fallback of an unmatched request could — the browser then
@@ -646,6 +789,16 @@ async def log_requests(request, call_next):
return response
def consented_trace() -> bool:
"""Whether the current request's OWN user turned Teljes (TRACE) on — the gate for a line that
carries what they typed or what was quoted back to them. The level alone is not enough: an
admin running the whole process at TRACE would pass it for every user, and a scheduler thread,
which has no request, for none of them in particular. Guard such a line with this, then log it
at TRACE."""
scope = _scope.get()
return scope is not None and _traced(scope)
def _traced(scope: _RequestScope) -> bool:
"""Whether this request's query string is logged: only under its OWN user's Teljes consent. An
admin setting the whole process to TRACE gets more detail, not every user's search terms — the
+62 -10
View File
@@ -29,6 +29,7 @@ import atexit
import json
import logging
import queue
import re
import sys
import threading
import time
@@ -43,6 +44,7 @@ from app import logs
from app.config import settings
from app.db import engine
from app.models import LogEvent
from app.utils import iso
QUEUE_MAX = 10_000
BATCH_MAX = 500
@@ -66,7 +68,7 @@ _LIMITS = {
"exc": 32_000,
}
_log = logging.getLogger("siftlode.logsink")
_log = logs.get_logger(__name__)
def should_store(record: logging.LogRecord) -> bool:
@@ -137,6 +139,21 @@ def _notice(code: str, msg: str, **attrs: Any) -> dict[str, Any]:
)
_DIGITS = re.compile(r"\d+")
def _template(record: logging.LogRecord) -> str:
"""The flood limiter's grouping key: the message template. Except when the template is a bare
`%s` — a pass-through for someone else's text (yt-dlp's lines, `formats._YdlLog`), where the
template says nothing and every distinct line would share one bucket. Then the text itself
groups, with its numbers folded, so `Retrying fragment 5 (1/10)` and `(2/10)` are still one
repeat and a different warning is not crowded out by them."""
msg = str(record.msg)
if msg != "%s" or not record.args:
return msg
return _DIGITS.sub("#", record.getMessage())[:200]
class _Repeats:
"""Per-template admission for one minute at a time. Used by the writer thread only (unlocked)."""
@@ -155,9 +172,11 @@ class _Repeats:
self.seen[key] = (count or 0) + 1
return self.seen[key] <= self.per_window
def roll(self, now: float) -> list[dict[str, Any]]:
"""Close the window if it is over: one notice per template that went past the limit."""
if now - self.started < self.window:
def roll(self, now: float, *, final: bool = False) -> list[dict[str, Any]]:
"""Close the window if it is over — or, `final`, because the process is: one notice per
template that went past the limit. Without `final` a process that lives under a minute
(every download child) would drop its repeats and never say so."""
if not final and now - self.started < self.window:
return []
notices = [
_notice(
@@ -202,6 +221,8 @@ class LogSink(logging.Handler):
return
try:
row = to_row(logs.record_fields(record))
# silent-ok: `handleError` is the report — logging's own out-of-band channel to stderr.
# A log call here would be handed to this handler, which is the one that just failed.
except Exception: # noqa: BLE001 — a record we cannot turn into a row is reported, not raised
self.handleError(record)
return
@@ -209,21 +230,25 @@ class LogSink(logging.Handler):
key = (
None
if row["event"] == "http.request"
else (record.name, str(record.msg), record.levelno)
else (record.name, _template(record), record.levelno)
)
try:
self._queue.put_nowait((key, row))
# silent-ok: counted, not logged — the count goes out as the `log.sink.dropped` notice row
# the writer emits, which is the one report that does not feed this queue again.
except queue.Full:
with self._dropped_lock:
self._dropped += 1
# --- writer side ---------------------------------------------------------------------------
def write_pending(self, wait: float = 0.0) -> int:
def write_pending(self, wait: float = 0.0, *, final: bool = False) -> int:
"""Write one batch: whatever is queued (up to BATCH_MAX), waiting up to `wait` seconds for the
first item. Returns the number of rows written, notices included."""
first item. Returns the number of rows written, notices included. `final` closes the
repeat window early — the last batch before the process exits."""
rows: list[dict[str, Any]] = []
try:
item = self._queue.get(timeout=wait) if wait > 0 else self._queue.get_nowait()
# silent-ok: an idle sink. Nothing failed; there was simply nothing waiting.
except queue.Empty:
item = None
while item is not None:
@@ -234,9 +259,10 @@ class LogSink(logging.Handler):
break
try:
item = self._queue.get_nowait()
# silent-ok: the batch ends here because the queue ran dry, which is the normal end.
except queue.Empty:
item = None
rows.extend(self._repeats.roll(time.monotonic()))
rows.extend(self._repeats.roll(time.monotonic(), final=final))
with self._dropped_lock:
dropped, self._dropped = self._dropped, 0
if dropped:
@@ -254,8 +280,12 @@ class LogSink(logging.Handler):
with self._bind.connect() as conn, logs.erasure_guard(), conn.begin():
conn.execute(insert(LogEvent), without_erased(rows))
except Exception as exc: # noqa: BLE001 — the database is down or refused the batch
print(
f"log sink: could not store {len(rows)} log events: {logs.redact_text(str(exc))[:300]}",
# silent-ok / noqa T201: the sink's OWN failure. A log call here would be handed to
# this sink, which is the thing that just failed — straight into a loop. stderr is
# the one way out that does not re-enter.
print( # noqa: T201
f"log sink: could not store {len(rows)} log events: "
f"{logs.redact_text(str(exc))[:300]}",
file=sys.stderr,
)
return 0
@@ -266,6 +296,7 @@ class LogSink(logging.Handler):
self.write_pending(wait=FLUSH_SECONDS)
while self.write_pending(): # drain what arrived before the stop
pass
self.write_pending(final=True) # and the repeat notices of a window cut short
def start(self) -> None:
self._writer = threading.Thread(target=self._run, name="log-sink", daemon=True)
@@ -313,3 +344,24 @@ def gc(db: Session, retention_days: int) -> int:
cutoff = datetime.now(timezone.utc) - timedelta(days=max(1, retention_days))
result = db.execute(delete(LogEvent).where(LogEvent.ts < cutoff))
return result.rowcount or 0
def serialize(row: LogEvent) -> dict:
"""A stored row as the API sends it — one shape for every reader: the admin console (E22.6) and
a debug session's evidence list (E16)."""
return {
"id": row.id,
"ts": iso(row.ts),
"level": row.level,
"source": row.source,
"logger": row.logger,
"event": row.event,
"msg": row.msg,
"attrs": row.attrs,
"exc": row.exc,
"request_id": row.request_id,
"user_id": row.user_id,
"job_id": row.job_id,
"debug_session_id": row.debug_session_id,
"version": row.version,
}
+42 -17
View File
@@ -1,5 +1,4 @@
import asyncio
import logging
import math
from contextlib import asynccontextmanager, suppress
from pathlib import Path
@@ -12,17 +11,19 @@ from fastapi.exceptions import RequestValidationError
# import time is already covered. ONE format for the api, the worker, the download child and alembic
# alike (app/logs.py) — this replaces the old uvicorn --log-config + the hand-rolled fallback that
# used to live here, and uvicorn's access log is replaced by the `http.request` middleware below.
from app import logs
from app.logs import configure_logging, current_level, log_requests, set_level
configure_logging("api")
log = logging.getLogger("siftlode.app")
log = logs.get_logger(__name__)
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
from fastapi.staticfiles import StaticFiles
from sqlalchemy.exc import DataError
from starlette.middleware.sessions import SessionMiddleware
from app import bodylimit
from app import logsink
from app import state
from app import sysconfig
@@ -32,10 +33,12 @@ from app.plex import stream as plex_stream
from app.plex import subtitles as plex_subtitles
from app.routes import (
admin,
admin_logs as admin_logs_routes,
audit as audit_routes,
auth as auth_routes,
channels,
config as config_routes,
debug_sessions as debug_sessions_routes,
diagnostics as diagnostics_routes,
discover as discover_routes,
downloads,
@@ -73,17 +76,17 @@ async def lifespan(app: FastAPI):
# booting — an unreachable or not-yet-migrated database has to start into the env level.
try:
set_level(sysconfig.get_str(db, "log_level"))
except Exception as exc: # noqa: BLE001 — any DB/config failure keeps the bootstrap level
log.warning(
"could not read the stored log level (%s); staying at %s", exc, current_level()
)
except Exception: # noqa: BLE001 — any DB/config failure keeps the bootstrap level
# With the traceback: this catch spans a connection, a query and a coercion at boot,
# and `%s` of the exception does not say which of them raised.
log.exception("could not read the stored log level; staying at %s", current_level())
if not state.is_configured(db):
token = state.rotate_setup_token(db)
url = f"{settings.app_base}/setup?token={token}"
log.warning(
"FIRST-RUN SETUP REQUIRED — this instance isn't configured yet.\n"
" Open the install wizard (internal access only):\n %s",
url,
url, # log-pii-ok: this instance's OWN setup URL, and `?token=` is redacted
)
# No HLS session survives a restart (they live in this process), so anything still on disk is
# a crashed run's segments — sweep them or they accumulate forever. FIRE AND FORGET, not
@@ -187,15 +190,6 @@ app.add_middleware(
https_only=settings.session_https_only, # Secure flag when served over HTTPS (prod)
)
if settings.frontend_origin:
app.add_middleware(
CORSMiddleware,
allow_origins=[settings.frontend_origin],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Paths reachable while the instance is unconfigured (setup mode). Everything else under /api or
# /auth is locked until the wizard finishes; static/SPA always loads so the wizard page can render.
_SETUP_OPEN_PREFIXES = ("/api/setup", "/api/version")
@@ -215,6 +209,34 @@ async def setup_gate(request, call_next):
return await call_next(request)
# SEC-20 — the global request-body cap (app/bodylimit.py). FastAPI reads a declared body IN FULL
# before the first dependency, so without this the anonymous login route is a free memory/time sink.
#
# WHERE it sits is the whole design, and it is pinned by `test_body_limit.py` rather than by this
# comment, because `add_middleware` inverts the reading order (last registered = OUTERMOST) and a
# stack described in prose drifts. Registered here it lands directly INSIDE CORS and directly
# OUTSIDE the setup gate, which is the narrowest position that satisfies both constraints:
#
# * inside CORS and the security headers, so the 413 is decorated like every other response. A
# refusal the browser drops for a missing Access-Control-Allow-Origin is no better than a hang,
# and `capped_body`'s 413 one layer down IS decorated — two shapes for one fact is exactly what
# bodylimit.py exists to avoid.
# * outside the setup gate, the session and every auth dependency under the router — none of which
# run for a body that is already too large. The setup gate matters specifically: on an instance
# that is not yet configured it opens a DB session per request, which is precisely the anonymous
# pre-auth work SEC-20 exists to remove.
app.add_middleware(bodylimit.BodyLimitMiddleware)
if settings.frontend_origin:
app.add_middleware(
CORSMiddleware,
allow_origins=[settings.frontend_origin],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@app.middleware("http")
async def static_cache_headers(request, call_next):
"""Long-cache the content-hashed SPA bundles. Vite hashes their filename, so a given
@@ -257,7 +279,8 @@ async def security_headers(request, call_next):
# The access log (app/logs.py): request id, route template, status, duration — one line, never a
# query string. Registered LAST so it is the OUTERMOST http middleware and therefore times the whole
# request, including what the three above do. Being outermost also puts it OUTSIDE SessionMiddleware,
# request, including what every layer above does — the body cap included, so a 413 is still logged
# with its request id and duration. Being outermost also puts it OUTSIDE SessionMiddleware,
# which is why the account comes from `request.state.user_id` (set by the auth dependency, the one
# place a session is actually resolved into an account) rather than from `request.session`.
app.middleware("http")(log_requests)
@@ -272,6 +295,7 @@ app.include_router(discover_routes.router)
app.include_router(search_routes.router)
app.include_router(me.router)
app.include_router(diagnostics_routes.router)
app.include_router(debug_sessions_routes.router)
app.include_router(notifications.router)
app.include_router(messages.router)
app.include_router(channels.router)
@@ -285,6 +309,7 @@ app.include_router(admin.router)
app.include_router(config_routes.router)
app.include_router(scheduler_routes.router)
app.include_router(audit_routes.router)
app.include_router(admin_logs_routes.router)
app.include_router(quota.router)
app.include_router(version.router)
app.include_router(setup_routes.router)
+90 -4
View File
@@ -12,6 +12,7 @@ from sqlalchemy import (
Index,
Integer,
JSON,
LargeBinary,
SmallInteger,
String,
Text,
@@ -22,6 +23,34 @@ from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from app.db import Base
# Every user-supplied display string in this schema shares one width, and it lives HERE so the
# columns below are declared with it rather than with a literal: a name a person typed goes into
# `User.display_name` (their Google profile name), `DownloadJob.display_name` and
# `DownloadJob.display_uploader`. A value wider than the column is a psycopg DataError, which
# `main.py` re-raises for everything but the NUL case — i.e. a bare 500 for a name somebody pasted.
DISPLAY_TEXT_MAX = 255
def display_text(raw: str | None) -> str | None:
"""A user-supplied display string on its way into one of those columns: stripped, truncated,
blank -> None.
One definition because there are SIX writers and they had drifted: `POST /api/downloads`,
`POST /api/downloads/edit`, `POST /api/downloads/upload`, the metadata PATCH's two fields, and
the Google sign-in callback. Only the PATCH truncated. Truncating is the right answer rather
than refusing: the name is a label on work the user has already done — a file already
transferred, an account already authenticated — and losing its tail is not worth failing the
request over.
`test_upload.py` holds both halves: that every route assigning one of these fields goes through
this helper (parsed, not grepped), and that each of those routes truncates rather than 500s. A
sweep is exactly the kind of thing that gets done five-sixths of the way, which is how this
ended up being written twice.
"""
return (raw or "").strip()[:DISPLAY_TEXT_MAX] or None
from app.utils import canonical_email
@@ -113,7 +142,7 @@ class User(Base, TimestampMixin, CanonicalEmailMixin):
# not cleared eagerly — it simply stops counting).
diagnostics_level: Mapped[str | None] = mapped_column(String(8))
diagnostics_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
display_name: Mapped[str | None] = mapped_column(String(255))
display_name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
avatar_url: Mapped[str | None] = mapped_column(String(1024))
role: Mapped[str] = mapped_column(String(16), default="user", server_default="user")
# The user's own YouTube channel id, resolved lazily (channels.list?mine=true) and
@@ -241,6 +270,9 @@ class Channel(Base, TimestampMixin):
details_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
recent_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
last_rss_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# When the newest uploads were last read through the Data API because the RSS feed would not
# answer (`sync/runner.py` `run_feed_fallback`): the once-per-`rss_fallback_hours` clock.
feed_fallback_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
backfill_cursor: Mapped[str | None] = mapped_column(String(128))
backfill_done: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this channel only entered the catalog because a live YouTube search
@@ -562,7 +594,15 @@ class LogEvent(Base):
the retention job (`log_retention_days`) bounds whatever another process writes late."""
__tablename__ = "log_events"
__table_args__ = (Index("ix_log_events_user_ts", "user_id", "ts"),)
__table_args__ = (
Index("ix_log_events_user_ts", "user_id", "ts"),
# E16: an issue's bundle is `WHERE debug_session_id = …`; nearly every row has none.
Index(
"ix_log_events_debug_session",
"debug_session_id",
postgresql_where="debug_session_id IS NOT NULL",
),
)
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
ts: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
@@ -581,6 +621,52 @@ class LogEvent(Base):
version: Mapped[str | None] = mapped_column(String(64))
class DebugSession(Base):
"""One recording: Start → reproduce → Stop (E16, app/debugsession.py). While it is open, every
request of its user that names it (`X-Debug-Session`) is logged at `level` and tagged with `id`,
and the issue it ends in is built from the rows carrying that tag. `until` is the hard cap the
server set at Start; an open session past it is over whether or not the sweep has closed it yet.
One OPEN session per user (`uq_debug_sessions_open_per_user`, partial on `ended_at IS NULL`)."""
__tablename__ = "debug_sessions"
__table_args__ = (
Index(
"uq_debug_sessions_open_per_user",
"user_id",
unique=True,
postgresql_where="ended_at IS NULL",
),
)
id: Mapped[str] = mapped_column(String(16), primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
level: Mapped[str] = mapped_column(String(8))
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
until: Mapped[datetime] = mapped_column(DateTime(timezone=True))
ended_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# app/debugsession.py `EndReason`: stopped by the user, or past `until`.
ended_reason: Mapped[str | None] = mapped_column(String(16))
issue_url: Mapped[str | None] = mapped_column(String(512))
# The issue being written from it (E16 S3; app/debugsession.py `save_draft` holds its shape).
draft: Mapped[dict | None] = mapped_column(JSONB)
class DebugScreenshot(Base):
"""A screenshot attached to a debug session (E16) — pasted, dropped, or captured from the pill.
Stored in the row: bounded per session (app/debugsession.py), and gone with it by cascade."""
__tablename__ = "debug_screenshots"
id: Mapped[int] = mapped_column(primary_key=True)
session_id: Mapped[str] = mapped_column(
ForeignKey("debug_sessions.id", ondelete="CASCADE"), index=True
)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
mime: Mapped[str] = mapped_column(String(16))
size: Mapped[int] = mapped_column(Integer)
data: Mapped[bytes] = mapped_column(LargeBinary)
class AppState(Base):
"""Single-row global app state (admin-controlled)."""
@@ -895,11 +981,11 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
ForeignKey("download_profiles.id", ondelete="SET NULL")
)
profile_snapshot: Mapped[dict] = mapped_column(JSON)
display_name: Mapped[str | None] = mapped_column(String(255))
display_name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
# Per-user display overrides + extras (like display_name — metadata only, never the file).
# `display_uploader`/`display_uploader_url` override the asset's auto channel name/link;
# `extra_links` is an optional list of extra reference URLs shown on the card + watch page.
display_uploader: Mapped[str | None] = mapped_column(String(255))
display_uploader: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
display_uploader_url: Mapped[str | None] = mapped_column(Text)
extra_links: Mapped[list | None] = mapped_column(JSON)
status: Mapped[str] = mapped_column(
+16 -4
View File
@@ -6,16 +6,15 @@ metadata, image proxying, and (P5) optional watch-state write-back. Requests JSO
``Accept: application/json`` header (Plex serves XML by default).
"""
import logging
from urllib.parse import quote
import httpx
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.config import settings
log = logging.getLogger("siftlode.plex")
log = logs.get_logger(__name__)
class PlexError(Exception):
@@ -106,7 +105,18 @@ class PlexClient:
Best-effort (needs the library's related data) — returns [] if Plex has none."""
try:
mc = self._get(f"/library/metadata/{rating_key}/related")
except PlexError:
except PlexError as exc:
# An empty list is also the answer for an item that genuinely has no related titles,
# so the caller cannot tell the two apart — and it deliberately declines to CACHE an
# empty one for exactly that reason. This line is what says which it was.
log.debug(
"the related items for a Plex item could not be read",
extra=logs.event(
"plex.client.related_unavailable",
rating_key=rating_key,
reason=type(exc).__name__,
),
)
return []
out: list[dict] = []
for hub in mc.get("Hub", []) or []:
@@ -139,6 +149,8 @@ class PlexClient:
raise PlexError(str(e)) from e
try:
return (r.json() or {}).get("MediaContainer", {}) or {} if r.content else {}
# silent-ok: the HTTP call already succeeded — this is a write endpoint, and Plex answers
# those with an empty or non-JSON body as a matter of course.
except ValueError:
return {}
+2
View File
@@ -22,6 +22,8 @@ def rich_detail(meta: dict) -> dict:
if str(r.get("image") or "").startswith("imdb://"):
try:
imdb_rating = round(float(r.get("value")), 1)
# silent-ok: no IMDb score, which is the common and legitimate case — the UI has a
# separate audience score to fall back on and shows nothing if neither exists.
except (TypeError, ValueError):
pass
break
+13 -2
View File
@@ -11,10 +11,13 @@ The cache directory (``cache.CACHE_DIR``) is shared with the subtitle cache (see
from fastapi import Response
from sqlalchemy.orm import Session
from app import logs
from app.models import PlexCollection, PlexItem, PlexSeason, PlexShow
from app.plex.cache import CACHE_DIR
from app.plex.lookups import by_rating_key
log = logs.get_logger(__name__)
_EXT_CT = [("jpg", "image/jpeg"), ("png", "image/png"), ("webp", "image/webp")]
_CT_EXT = {ct: ext for ext, ct in _EXT_CT}
# Target box per variant — Plex resizes to these server-side. Posters fill ≤176px cells (grid/info/
@@ -46,6 +49,8 @@ def read(key: str) -> tuple[bytes, str] | None:
try:
if p.exists() and p.stat().st_size > 0:
return p.read_bytes(), ct
# silent-ok: a miss. The caller fetches from Plex and writes the file again, and the write
# path below is the one that reports a cache that cannot be used.
except OSError:
pass
return None
@@ -58,8 +63,14 @@ def write(key: str, data: bytes, ct: str) -> None:
tmp = CACHE_DIR / f"{key}.{ext}.tmp"
tmp.write_bytes(data)
tmp.replace(CACHE_DIR / f"{key}.{ext}") # atomic publish
except OSError:
pass # cache is best-effort
except OSError as exc:
# Best-effort, and silently so: with an unwritable cache every poster on every grid goes
# to Plex live, for as long as nobody notices. Debug, because a grid is ~100 images and
# the point is the condition, not the count.
log.debug(
"the Plex image cache could not be written",
extra=logs.event("plex.imgcache.write_failed", reason=type(exc).__name__),
)
def response(data: bytes, ct: str) -> Response:
+11 -2
View File
@@ -12,7 +12,9 @@ from pathlib import Path
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
log = logs.get_logger(__name__)
def parse_map(raw: str) -> list[tuple[str, str]]:
@@ -47,6 +49,13 @@ def local_media_path(db: Session, plex_path: str | None) -> Path | None:
return None
try:
p = Path(map_path(db, plex_path)).resolve()
except OSError:
except OSError as exc:
# None is answered as a 404 "no local file", which is also the answer for a title the
# server does not hold — so a dead mount or a wrong path map reads as missing media for
# the whole library at once. The path itself stays out: it carries the title.
log.warning(
"a Plex media path could not be resolved locally",
extra=logs.event("plex.paths.resolve_failed", reason=type(exc).__name__),
)
return None
return p if p.is_file() else None
-4
View File
@@ -298,10 +298,6 @@ class LibraryFilters:
}
def like_escape(s: str) -> str:
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
# --- Watch-state scope (shared by the /library grid AND the /facets aggregation) ---------------
#
# The `show` scope (all | watched | in_progress | unwatched) filters the browse result by per-user
+3 -1
View File
@@ -43,7 +43,9 @@ def rollup(cards: list[dict]) -> dict:
]
resume = None
if inprog:
resume = inprog[-1] # continue the latest-started episode
# The LAST in-progress episode in broadcast order — no timestamp is consulted. Home's
# `discover.queries.next_episode` picks the same one, so both Resume buttons agree.
resume = inprog[-1]
else:
resume = next((c for c in cards if c.get("status") != "watched"), None)
return {
+180 -50
View File
@@ -18,7 +18,6 @@ segments. The remux is CPU-light (video copy + a tiny audio transcode), so it ru
CPU-only prod host; only P3's full transcode is CPU-heavy.
"""
import logging
import os
import re
import secrets
@@ -31,15 +30,14 @@ from pathlib import Path
from sqlalchemy.orm import Session as DbSession
from app import sysconfig
from app import logs, sysconfig
from app.config import settings
from app.downloads.edit import encode_thread_cap
from app.models import PlexItem
from app.plex import paths
from app.plex.sync import HW_DECODABLE_CODECS
log = logging.getLogger("siftlode.plex")
log = logs.get_logger(__name__)
_HLS_ROOT = (
Path(settings.plex_hls_dir)
if settings.plex_hls_dir
@@ -62,8 +60,10 @@ _WARMUP_SEG_TIMEOUT_S = 8.0
# its index for that one. See `segment_name` for why this is a constant and not a comment.
_VIDEO_VARIANT = "0"
_SESSION_IDLE_S = 600 # reap a session with no access for this long
# A session directory is named `{rating_key}_{int(start_s)}_{tag}_{aoff:+.2f}` (see start_session);
# before multi-audio it was just `{rating_key}_{int(start_s)}`. `sweep_orphans` deletes ONLY names
# A session directory is named `{rating_key}_u{user_id}_{int(start_s)}_{tag}_{aoff:+.2f}` (see
# start_session); before per-viewer sessions it was `{rating_key}_{int(start_s)}_{tag}_{aoff:+.2f}`,
# and before multi-audio just `{rating_key}_{int(start_s)}` — the older shapes stay sweepable, since a
# restart leaves them behind like any other. `sweep_orphans` deletes ONLY names
# matching one of those two shapes: `_HLS_ROOT` is admin-configurable (PLEX_HLS_DIR), so it may
# legitimately point at a shared scratch path we do not own, and a blanket "delete every directory
# under the root" would take the neighbours with it.
@@ -74,7 +74,9 @@ _SESSION_IDLE_S = 600 # reap a session with no access for this long
# non-numeric rating_key would merely leave its directory unswept, which is the safe direction to
# fail. Covered by test_plex_stream.py — this regex is the only thing standing between the sweep
# and someone else's data.
_SESSION_DIR_RE = re.compile(r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$")
_SESSION_DIR_RE = re.compile(
r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{2}$"
)
# When this process started. `sweep_orphans` only deletes directories older than this: everything
# newer belongs to a session THIS run created, and since `start_session` reuses a directory path
# for the same (rating_key, offset, tag), "stale name" is not the same as "stale directory".
@@ -90,7 +92,16 @@ _PROCESS_START = time.time()
_MTIME_GRACE_S = 2.0
_lock = threading.Lock()
_sessions: dict[str, "HlsSession"] = {}
# Keyed by VIEWER and item, (user_id, rating_key). Keyed by the item alone, a second account opening
# the same film `_kill`ed the first one's session: the displaced viewer got a 404, reclaimed at its
# playhead, and so evicted the other in turn — two people could not watch one film at once. (One
# account on two devices still shares a session; that case ends in the player's bounded reclaim.)
_sessions: dict[tuple[int, str], "HlsSession"] = {}
# How many live sessions one account may hold at once (two devices, or a next item starting while the
# last one winds down). A new one past it replaces that account's least recently used — whose waiting
# session request then returns at once, since `wait_for` watches the killed ffmpeg — so one account
# can neither pile up film-sized scratch copies nor park more than this many request workers.
_MAX_SESSIONS_PER_USER = 2
class HlsSession:
@@ -104,16 +115,17 @@ class HlsSession:
transcode: bool = False,
hw: bool = False,
tonemap: bool = False,
user_id: int = 0,
):
# Unguessable per-session id, carried in the URLs the client is handed. Sessions are keyed
# by rating_key alone, so a seek-restart or a SECOND viewer of the same item replaces the
# registered one — and the old viewer's next request for `seg_12.ts`, resolved by rating_key,
# was answered 200 with the NEW session's twelfth segment, produced from a different offset
# Unguessable per-session id, carried in the URLs the client is handed. A seek-restart
# replaces the viewer's registered session for the item — and the old player's next request
# for `seg_12.ts`, resolved by (viewer, rating_key) alone, was answered 200 with the NEW session's twelfth segment, produced from a different offset
# under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it
# (deep-review round 5). A request naming a session that is no longer registered is a 404,
# which the player recovers from by restarting at the playhead.
self.id = secrets.token_urlsafe(9)
self.key = key
self.key = key # the item's rating_key (what the logs name)
self.user_id = user_id # whose session it is: the registry key is (user_id, key)
self.dir = directory
self.proc = proc
self.transcode = transcode # video re-encoded (non-h264 source), not stream-copied
@@ -135,6 +147,8 @@ class HlsSession:
# single index. Derived, not passed, so it cannot disagree with the argv.
self.entry = _MASTER_PL if self.variants else _INDEX_PL
self.last_access = time.time()
# `artifact_may_arrive` reports a refusing session directory once, not per segment.
self.probe_failure_reported = False
def _seek_opts(start_s: float) -> list[str]:
@@ -232,7 +246,15 @@ def _probe_video_kind(src: Path) -> str:
text=True,
timeout=10,
)
except (subprocess.SubprocessError, OSError):
except (subprocess.SubprocessError, OSError) as exc:
# The safe kind is a real answer, but it is the answer for an 8-bit SDR source — a blind
# ffprobe sends every HDR file down it, so playback quietly degrades to a software
# transcode with nothing anywhere saying why. The path stays out of the line: it is the
# title of whatever is playing.
log.info(
"video-kind probe failed; assuming the HW-safe pipeline",
extra=logs.event("plex.stream.probe_kind_failed", reason=type(exc).__name__),
)
return _VK_HWSAFE
pix_fmt = bits = transfer = ""
for line in out.stdout.splitlines():
@@ -261,7 +283,14 @@ def _opencl_present() -> bool:
software libx264. Its own function so it's monkeypatchable in tests, mirroring _render_node_present."""
try:
return any(_OPENCL_ICD_DIR.glob("*.icd"))
except OSError:
except OSError as exc:
# A MISSING directory does not come through here — `glob` simply yields nothing for one.
# This is a directory we cannot read, and the answer it forces (no OpenCL) costs every HDR
# source the hardware tonemap path.
log.info(
"the OpenCL ICD directory could not be read; treating OpenCL as absent",
extra=logs.event("plex.stream.opencl_probe_failed", reason=type(exc).__name__),
)
return False
@@ -447,6 +476,8 @@ def _sw_child_preexec() -> None:
playback."""
try:
os.nice(_SW_TRANSCODE_NICE)
# silent-ok: this runs in the forked child, between fork and exec. Logging there can deadlock
# on a lock some other thread held at fork time, and the child is about to become ffmpeg.
except OSError:
pass
@@ -473,7 +504,14 @@ def _probe_audio_langs(src: Path) -> list[str]:
timeout=10,
)
return [ln.strip() for ln in out.stdout.splitlines()]
except (subprocess.SubprocessError, OSError):
except (subprocess.SubprocessError, OSError) as exc:
# An empty list IS the single-audio path, so a failed probe silently collapses a file with
# several audio tracks down to one — and the viewer gets a player with no language picker
# rather than an error.
log.warning(
"audio-track probe failed; falling back to a single-audio session",
extra=logs.event("plex.stream.probe_audio_failed", reason=type(exc).__name__),
)
return []
@@ -582,8 +620,14 @@ def _probe_first_pts(seg: Path) -> float | None:
tok = line.strip().strip(",")
if tok:
return float(tok)
except (subprocess.SubprocessError, ValueError, OSError):
pass
except (subprocess.SubprocessError, ValueError, OSError) as exc:
# None means "keep the offset the client asked for", and the whole point of this probe is
# that the two differ: without it the subtitle track is shifted by whatever the container
# started its timestamps at. Seconds of drift, and nothing on screen calls it a failure.
log.warning(
"first-PTS probe failed; the requested offset is used as-is",
extra=logs.event("plex.stream.first_pts_probe_failed", reason=type(exc).__name__),
)
return None
@@ -592,29 +636,62 @@ def _kill(s: HlsSession) -> None:
s.proc.terminate()
try:
s.proc.wait(timeout=3)
# silent-ok: the escalation IS the handling. A transcode that ignores TERM for three
# seconds gets KILL, which is exactly what this branch does.
except subprocess.TimeoutExpired:
s.proc.kill()
except Exception:
pass
except Exception as exc: # noqa: BLE001 — teardown finishes whatever the process does
# Not nothing: what survives this is an ffmpeg nobody owns any more, holding a VAAPI
# handle and a core, and the transcode cap that is supposed to bound them counts rows in
# `_sessions` rather than processes. One line is the only way that ever surfaces.
log.info(
"could not stop the transcode process",
extra=logs.event("plex.stream.kill_failed", key=s.key, reason=type(exc).__name__),
)
shutil.rmtree(s.dir, ignore_errors=True)
def _enforce_cap(cap: int) -> None:
# Caller holds _lock. Only TRANSCODE sessions count toward plex_max_transcodes — remux
# (stream-copy) and direct-serve are cheap and deliberately uncapped (see config.py:
# plex_max_transcodes, and the user-facing config hint). Drop the least-recently-accessed
# transcode sessions over the cap, leaving any concurrent remux streams untouched.
tx = [(key, s) for key, s in _sessions.items() if s.transcode]
if len(tx) <= cap:
return
for key, s in sorted(tx, key=lambda kv: kv[1].last_access)[: len(tx) - cap]:
_kill(s)
_sessions.pop(key, None)
def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
"""Caller holds _lock. Three bounds, applied after a session is registered, each dropping the
least recently accessed sessions over it:
* this account's own sessions (`_MAX_SESSIONS_PER_USER`) — first, so a single account filling the
box evicts itself before anyone else;
* TRANSCODE sessions (plex_max_transcodes): the CPU/GPU-heavy ones;
* EVERY session (plex_max_sessions). A remux is cheap on the CPU but not on the disk: each one is a
whole segmented copy of the film under `_HLS_ROOT`, written faster than realtime. Counted only
by the transcode cap, N distinct items were N ffmpegs and N film-sized scratch directories.
Somebody's playback stops at each drop. The route turns the missing session into a retryable 404
and the player restarts at the playhead, so from the server nothing failed — and "my film keeps
rebuffering" has no other explanation anywhere, hence the line."""
def cull(pool: list[tuple[tuple[int, str], HlsSession]], cap: int, bound: str) -> None:
if len(pool) <= cap:
return
for reg, sess in sorted(pool, key=lambda kv: kv[1].last_access)[: len(pool) - cap]:
log.info(
"a session was dropped to stay under a cap",
extra=logs.event(
"plex.stream.session_culled",
key=sess.key,
bound=bound,
cap=cap,
sessions=len(pool),
),
)
_kill(sess)
_sessions.pop(reg, None)
cull([kv for kv in _sessions.items() if kv[0][0] == user_id], _MAX_SESSIONS_PER_USER, "user")
cull([kv for kv in _sessions.items() if kv[1].transcode], max_transcodes, "transcode")
cull(list(_sessions.items()), max_sessions, "total")
def start_session(
db: DbSession,
item: PlexItem,
user_id: int,
start_s: float,
audio_ord: int | None = None,
aoff: float = 0.0,
@@ -634,6 +711,7 @@ def start_session(
# session respawn (the HW->SW fallback) must not pick up a mid-playback config change.
loudnorm = sysconfig.get_bool(db, "plex_audio_loudnorm")
key = item.rating_key
reg = (user_id, key) # the registry key: one session per viewer per item
start_s = max(0.0, float(start_s))
audio_langs = _probe_audio_langs(src) if multi else []
is_multi = len(audio_langs) > 1 # only worth a master playlist when there really are ≥2 tracks
@@ -646,7 +724,7 @@ def start_session(
# the tag with a DASH — `_SESSION_DIR_RE` treats the tag as one underscore-free field, and that
# regex is the only thing keeping the orphan sweep off a shared scratch path (see its comment).
tag = ("multi" if is_multi else str(audio_ord)) + ("-ln" if loudnorm else "")
directory = _HLS_ROOT / f"{key}_{int(start_s)}_{tag}_{aoff:+.2f}"
directory = _HLS_ROOT / f"{key}_u{user_id}_{int(start_s)}_{tag}_{aoff:+.2f}"
def _seg(i: int) -> Path:
return directory / segment_name(i, _VIDEO_VARIANT if variants else None)
@@ -672,9 +750,9 @@ def start_session(
# platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
with _lock:
if replace_only is not None and _sessions.get(key) is not replace_only:
if replace_only is not None and _sessions.get(reg) is not replace_only:
return None # superseded/culled while we waited out of lock — don't clobber the newer one
old = _sessions.pop(key, None)
old = _sessions.pop(reg, None)
if old is not None:
_kill(old)
shutil.rmtree(directory, ignore_errors=True)
@@ -708,11 +786,31 @@ def start_session(
transcode=transcode,
hw=hw,
tonemap=tonemap,
user_id=user_id,
)
_sessions[reg] = s
# The one line that says which pipeline a session actually got. Everything downstream
# — the cap, the HW→SW fallback, the subtitle offset — is about this choice, and until
# now it was only inferable from the ffmpeg command nobody keeps.
log.info(
"an HLS session started",
extra=logs.event(
"plex.stream.session_started",
key=key,
transcode=transcode,
hw=hw,
tonemap=tonemap,
variants=len(variants or ()),
start_s=round(start_s, 1),
),
)
# Admin-configurable caps (Configuration → Plex); at least 1 each so playback can't be
# capped to zero.
_enforce_caps(
user_id,
max(1, sysconfig.get_int(db, "plex_max_transcodes")),
max(1, sysconfig.get_int(db, "plex_max_sessions")),
)
_sessions[key] = s
# Admin-configurable concurrency cap (Configuration → Plex → Max concurrent transcodes);
# at least 1 so playback can't be capped to zero.
_enforce_cap(max(1, sysconfig.get_int(db, "plex_max_transcodes")))
return s
s = _spawn(*_select_encode(db, item, transcode, src))
@@ -728,7 +826,8 @@ def start_session(
# ONE lock in _spawn, so we can never tear down the newer session. s.hw is now False, so the route
# honestly reports "CPU" and the player's Preparing-stream state stays correct.
log.warning(
"plex transcode: HW session produced no segments — falling back to software key=%s", key
"the hardware transcode produced no segments; falling back to software",
extra=logs.event("plex.stream.hw_fallback", key=key),
)
respawned = _spawn(False, False, replace_only=s)
if respawned is None:
@@ -737,7 +836,7 @@ def start_session(
# turns a None return into a "Media file not found on disk" 404 — dishonest for a file that
# exists. A (dead) session instead degrades to a retryable 404 ("No active playback session" /
# "Not available yet") the client re-requests.
return current_session(key) or s
return current_session(user_id, key) or s
s = respawned
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
if ready:
@@ -768,9 +867,9 @@ def start_session(
return s
def current_session(key: str) -> HlsSession | None:
def current_session(user_id: int, key: str) -> HlsSession | None:
with _lock:
s = _sessions.get(key)
s = _sessions.get((user_id, key))
if s is not None:
s.last_access = time.time()
return s
@@ -867,7 +966,23 @@ def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
return True
try:
return (s.dir / segment_name(seq - 1, variant)).exists()
except OSError:
except OSError as exc:
# `exists()` answers False for a missing file without raising, so this is the session
# directory itself refusing — a full or unmounted disk. Every segment then 404s, which the
# player shows as a stall. Once per session: this runs for every segment request and
# every hls.js retry of one, and the sink's limiter still stores 30 a minute of a
# template — stdout would get them all.
if not s.probe_failure_reported:
s.probe_failure_reported = True
log.warning(
"a session segment could not be checked",
extra=logs.event(
"plex.stream.segment_probe_failed",
key=s.key,
seq=seq,
reason=type(exc).__name__,
),
)
return False
@@ -884,12 +999,16 @@ def wait_for(path: Path, timeout: float = 20.0, proc: subprocess.Popen | None =
try:
if path.exists() and path.stat().st_size > 0:
return True
# silent-ok: this is a 150 ms poll — a line here is a line every 150 ms for the whole
# timeout. The final check below decides, and the 404 the caller then returns is honest.
except OSError:
pass
if proc is not None and proc.poll() is not None:
# the producer exited — no more segments are coming; one last check then give up.
try:
return path.exists() and path.stat().st_size > 0
# silent-ok: one last look after the producer exited. False here becomes a 404 the
# player recovers from, which is the honest answer either way.
except OSError:
return False
time.sleep(0.15)
@@ -922,7 +1041,16 @@ def sweep_orphans(older_than: float | None = None) -> int:
dropped = 0
try:
entries = list(_HLS_ROOT.iterdir())
except OSError: # the root doesn't exist yet — nothing to sweep
# silent-ok: nothing has been transcoded yet, so there is nothing to sweep.
except FileNotFoundError:
return 0
except OSError as exc:
# Any OTHER reason the root will not list is not "nothing to sweep": the sweep reports 0
# like a clean run while every crashed session's segments stay on disk forever.
log.warning(
"the HLS root could not be listed; no orphans were swept",
extra=logs.event("plex.stream.sweep_root_unreadable", reason=type(exc).__name__),
)
return 0
for p in entries:
if not _SESSION_DIR_RE.match(p.name):
@@ -932,7 +1060,9 @@ def sweep_orphans(older_than: float | None = None) -> int:
try:
if p.stat().st_mtime >= cutoff:
continue # this run made (or reused) it — never ours to delete
except OSError: # vanished under us (a concurrent restart of that session)
# silent-ok: vanished under us — a concurrent restart of that session removed it, which
# is the outcome this sweep wanted anyway.
except OSError:
continue
with _lock:
if any(s.dir == p for s in _sessions.values()):
@@ -968,21 +1098,21 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
now = time.time()
dropped = 0
with _lock:
for key, s in list(_sessions.items()):
for reg, s in list(_sessions.items()):
done = s.proc.poll() is not None
idle = now - s.last_access
if idle > _SESSION_IDLE_S or (done and idle > finished_idle_s):
_kill(s)
_sessions.pop(key, None)
_sessions.pop(reg, None)
dropped += 1
return dropped
def touch_session(key: str) -> None:
"""Say a viewer is still there for `key`, without fetching anything. `last_access` used to move
def touch_session(user_id: int, key: str) -> None:
"""Say this viewer is still there for `key`, without fetching anything. `last_access` used to move
only on an HLS file fetch, which a paused player does not do — so the only way to keep a paused
viewer's session alive was to keep EVERY abandoned one just as long."""
with _lock:
s = _sessions.get(key)
s = _sessions.get((user_id, key))
if s is not None:
s.last_access = time.time()
+179 -20
View File
@@ -33,17 +33,18 @@ zero-based clock and shifted back to absolute time here with `shift_vtt`.
"""
import json
import logging
import re
import subprocess
import tempfile
import threading
import time
import logging
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from sqlalchemy.orm import Session
from app import logs
from app.db import SessionLocal
from app.models import PlexItem
from app.plex import paths as plex_paths
@@ -51,8 +52,7 @@ from app.plex.cache import CACHE_DIR
from app.plex.client import PlexClient, PlexError
from app.plex.lookups import by_rating_key
log = logging.getLogger("siftlode.plex")
log = logs.get_logger(__name__)
# Image-based subtitle codecs (Blu-ray/DVD) — can't be turned into text WebVTT (would need burn-in).
BITMAP_SUBS = {
"pgs",
@@ -104,6 +104,7 @@ def shift_vtt(vtt: str, offset: float) -> str:
straddling 0 is clamped to start at 0."""
if not offset:
return vtt
unshifted = 0
out_blocks: list[str] = []
for block in re.split(r"\n[ \t]*\n", vtt):
if "-->" not in block:
@@ -121,7 +122,12 @@ def shift_vtt(vtt: str, offset: float) -> str:
st = _vtt_ts_to_s(left.strip()) - offset
en = _vtt_ts_to_s(rparts[0]) - offset
except (ValueError, IndexError):
break # unparseable timing → leave the block as-is
# silent-ok: left as-is means left UNSHIFTED — the viewer sees that one cue at
# the wrong moment. Counted rather than logged here: this is per cue, and the file
# being parsed is one ffmpeg wrote for us a moment ago, so a broken timestamp is
# a fact about the extraction, reported once below.
unshifted += 1
break
if en <= 0:
keep = False # whole cue is before the session start → drop the block
else:
@@ -131,6 +137,11 @@ def shift_vtt(vtt: str, offset: float) -> str:
break
if keep:
out_blocks.append("\n".join(lines))
if unshifted:
log.info(
"some cues could not be shifted and stay at their original time",
extra=logs.event("plex.subtitles.cues_unshifted", cues=unshifted, offset_s=offset),
)
return "\n\n".join(out_blocks)
@@ -162,7 +173,15 @@ def cache_read(key: str) -> str | None:
try:
p = CACHE_DIR / f"{key}.vtt"
return p.read_text(encoding="utf-8") if p.exists() else None
except OSError:
except OSError as exc:
# None is the same answer as "never cached", so a cache we cannot read looks exactly like
# a cold one — and every request re-extracts instead of saying the disk is the problem.
log.warning(
"the cached subtitle could not be read",
extra=logs.event(
"plex.subtitles.cache_read_failed", key=key, reason=type(exc).__name__
),
)
return None
@@ -175,7 +194,16 @@ def cache_write(key: str, vtt: str) -> bool:
tmp.write_text(vtt, encoding="utf-8")
tmp.replace(CACHE_DIR / f"{key}.vtt") # atomic publish
return True
except OSError:
except OSError as exc:
# `publish` turns False into "the region was not claimed" and moves on, so the track just
# never appears. Which errno it was — full disk, read-only mount, permissions — is the
# whole diagnosis, and it exists only here.
log.warning(
"the subtitle cache could not be written",
extra=logs.event(
"plex.subtitles.cache_write_failed", key=key, reason=type(exc).__name__
),
)
return False
@@ -206,7 +234,17 @@ def read_meta(key: str) -> tuple[list[list[float]], bool]:
try:
raw = json.loads(_meta_path(key).read_text(encoding="utf-8"))
return [[float(a), float(b)] for a, b in raw.get("ranges", [])], bool(raw.get("complete"))
except (OSError, ValueError, TypeError):
# silent-ok: no sidecar at all is the documented older-build case the fallback exists for.
except FileNotFoundError:
return ([[0.0, _FULL_END_S]], True) if _exists(_vtt_path(key)) else ([], False)
except (OSError, ValueError, TypeError) as exc:
# A sidecar that EXISTS and will not parse is different, and the fallback is the dangerous
# answer to it: beside a cached VTT it reports the track as complete, so a partial track is
# never extended again and the viewer loses every cue past the window that was cached.
log.warning(
"the subtitle sidecar could not be read; the cached track is taken as complete",
extra=logs.event("plex.subtitles.meta_unreadable", key=key, reason=type(exc).__name__),
)
return ([[0.0, _FULL_END_S]], True) if _exists(_vtt_path(key)) else ([], False)
@@ -216,6 +254,8 @@ def _exists(p: Path) -> bool:
escaped as a 500 from the status route."""
try:
return p.exists()
# silent-ok: this function's whole job is to answer False for a path the OS refuses. The
# refusal IS the return value; a caller that needs to report one reports it itself.
except (OSError, ValueError):
return False
@@ -226,8 +266,18 @@ def _write_meta(key: str, ranges: list[list[float]], complete: bool) -> None:
tmp = CACHE_DIR / f"{key}.json.tmp"
tmp.write_text(json.dumps({"ranges": ranges, "complete": complete}), encoding="utf-8")
tmp.replace(_meta_path(key))
except OSError:
pass
except OSError as exc:
# Without the sidecar beside it, a cached window reads back as a COMPLETE track (see
# `read_meta`) — so one failed write here permanently costs the rest of that subtitle.
log.warning(
"the subtitle sidecar could not be written",
extra=logs.event(
"plex.subtitles.meta_write_failed",
key=key,
complete=complete,
reason=type(exc).__name__,
),
)
def generation(key: str) -> int:
@@ -235,7 +285,9 @@ def generation(key: str) -> int:
window is superseded by a wider one the browser refetches instead of keeping the short VTT."""
try:
return _vtt_path(key).stat().st_mtime_ns
except (OSError, ValueError): # ValueError: a NUL in the key — `stat` refuses the path outright
# silent-ok: 0 for a track that is not cached yet, which is the ordinary case on every first
# request. (ValueError: a NUL in the key — `stat` refuses the path outright.)
except (OSError, ValueError):
return 0
@@ -267,16 +319,27 @@ def merge_vtt(*sources: str) -> str:
happen to share a time (two speakers, a sign over dialogue) are both kept."""
seen: set[str] = set()
cues: list[tuple[float, str]] = []
dropped = 0
for src in sources:
for block in re.split(r"\n[ \t]*\n", src or ""):
canon = _canonical_cue(block.strip())
stripped = block.strip()
canon = _canonical_cue(stripped)
if canon is None:
# A block WITH a timing line that still would not parse is a cue we are throwing
# away, not a header — and the merged track would simply be missing those lines.
if "-->" in stripped:
dropped += 1
continue # header / NOTE / STYLE — the merged file gets its own single header
start, key, text = canon
if key in seen:
continue
seen.add(key)
cues.append((start, text))
if dropped:
log.info(
"some cues were dropped from the merged subtitle: their timing would not parse",
extra=logs.event("plex.subtitles.cues_dropped", cues=dropped, kept=len(cues)),
)
cues.sort(key=lambda c: c[0])
return "WEBVTT\n\n" + "\n\n".join(b for _, b in cues) + "\n"
@@ -295,6 +358,8 @@ def _canonical_cue(block: str) -> tuple[float, str, str] | None:
try:
st = _vtt_ts_to_s(left.strip())
en = _vtt_ts_to_s(rparts[0])
# silent-ok: None here means "not a cue", and the caller tells a header apart from a
# broken timing line by whether the block had a `-->` at all — see `merge_vtt`.
except (ValueError, IndexError):
return None
lines[i] = f"{_vtt_s_to_ts(st)} --> {_vtt_s_to_ts(en)}{settings}"
@@ -310,6 +375,8 @@ def _last_cue_end(vtt: str) -> float:
continue
try:
end = max(end, _vtt_ts_to_s(ln.partition("-->")[2].strip().split(None, 1)[0]))
# silent-ok: this is a floor — "how far the pass provably got". Skipping a line it cannot
# read makes the answer smaller, never larger, so the error is always on the safe side.
except (ValueError, IndexError):
continue
return end
@@ -394,11 +461,18 @@ _FULL_POOL = ThreadPoolExecutor(max_workers=2, thread_name_prefix="plex-subs-ful
# which time something has finished.
_MAX_WINDOWS_INFLIGHT = 8
_MAX_FULL_INFLIGHT = 6
# …of which one account may hold this many. The full-pass slots are minutes each, and a hostile (or
# merely curious) account opening subtitles on film after film could otherwise hold all of them while
# everyone else's tracks never completed. A viewer over their share still gets windows — the part
# that makes a track watchable now — just not another whole-file read until one of theirs ends.
_MAX_FULL_PER_USER = 3
_MAX_TRACK_WINDOWS = 2
_LOCK = threading.Lock()
# Queued or running jobs → the absolute span a window job reads (None for a full pass).
_INFLIGHT: dict[tuple, tuple[float, float] | None] = {}
# rating_key → the account whose poll queued that item's full pass (see _MAX_FULL_PER_USER).
_FULL_OWNERS: dict[str, int] = {}
# Track-level failures: a track that cannot be served at all (no item, a bitmap track, Plex down…).
_FAILURES: dict[str, tuple[str, float]] = {}
# rating_key → when its full pass last failed (see FULL_RETRY_S).
@@ -448,6 +522,7 @@ def reset_state() -> None:
global _STOPPING
with _LOCK:
_INFLIGHT.clear()
_FULL_OWNERS.clear()
_FAILURES.clear()
_FULL_FAILED.clear()
_WIN_FAILED.clear()
@@ -531,7 +606,7 @@ def _window_failed(key: str, at: float) -> bool:
return any(a <= at <= b for a, b, _ in _live_window_failures(key))
def ensure(rating_key: str, sub_ord: int, at: float = 0.0) -> dict:
def ensure(rating_key: str, sub_ord: int, at: float = 0.0, viewer: int = 0) -> dict:
"""State of one subtitle track, scheduling whatever extraction is still missing. NEVER blocks.
``ready`` = fully extracted · ``partial`` = usable, covers ``at``, still growing · ``preparing``
@@ -574,8 +649,23 @@ def ensure(rating_key: str, sub_ord: int, at: float = 0.0) -> dict:
if edge - at <= LOOKAHEAD_S and edge < _FULL_END_S:
probe = edge + 1.0
want_window = not _window_failed(key, probe)
_schedule(rating_key, sub_ord, probe, want_window=want_window, want_full=not full_failed)
return {"state": "partial" if covered else "preparing", "code": None, "gen": gen}
_schedule(
rating_key,
sub_ord,
probe,
want_window=want_window,
want_full=not full_failed,
viewer=viewer,
)
# `growing`: whether the track will fill in on its own. False once the full pass has failed and
# is backed off — a partial track then stays partial until a window is asked for, so the player
# may poll slowly (only to read ahead of an edge) instead of every two seconds for the whole film.
return {
"state": "partial" if covered else "preparing",
"code": None,
"gen": gen,
"growing": not full_failed,
}
def cached_track(rating_key: str, sub_ord: int) -> tuple[str | None, str | None]:
@@ -592,7 +682,12 @@ def cached_track(rating_key: str, sub_ord: int) -> tuple[str | None, str | None]
def _schedule(
rating_key: str, sub_ord: int, at: float, want_window: bool, want_full: bool = True
rating_key: str,
sub_ord: int,
at: float,
want_window: bool,
want_full: bool = True,
viewer: int = 0,
) -> None:
"""Kick off the jobs this track still needs: the window that makes it watchable now, and the one
full pass that makes it complete. A 1 s status poll (or a second viewer) joins a job already
@@ -625,8 +720,15 @@ def _schedule(
ident = ("win", rating_key, sub_ord, w_start)
_INFLIGHT[ident] = (w_start, w_start + WINDOW_PRE_S + WINDOW_LEN_S)
jobs.append((ident, w_start))
if want_full and n_full < _MAX_FULL_INFLIGHT and ("full", rating_key) not in _INFLIGHT:
mine = sum(1 for owner in _FULL_OWNERS.values() if owner == viewer)
if (
want_full
and n_full < _MAX_FULL_INFLIGHT
and ("full", rating_key) not in _INFLIGHT
and mine < _MAX_FULL_PER_USER
):
_INFLIGHT[("full", rating_key)] = None
_FULL_OWNERS[rating_key] = viewer
jobs.append((("full", rating_key), None))
for ident, window_start in jobs:
pool = _FULL_POOL if window_start is None else _WIN_POOL
@@ -652,12 +754,35 @@ def _run(
return
_extract_and_publish(rating_key, sub_ord, window_start)
except SubtitleUnavailable as e:
# The player's own state was the only place this ever showed. `_pass_failed` backs the item
# off for an hour and `_fail` records a code, but neither leaves a line, so "my subtitles
# never appear" had nothing on the server to read.
# A final code is a fact about the file (no such track, a bitmap subtitle); anything else
# is this instance failing at something it could do.
log.log(
logging.INFO if e.code in FINAL_CODES else logging.WARNING,
"subtitle extraction gave up",
extra=logs.event(
"plex.subtitles.pass_failed",
rating_key=rating_key,
sub_ord=sub_ord,
code=e.code,
windowed=window_start is not None,
),
)
if e.code == "extract_failed":
_pass_failed(rating_key, [key], window_start)
else:
_fail(key, e.code)
except PlexError as e:
log.warning("subtitle extraction failed for %s/%s: %s", rating_key, sub_ord, e)
# No traceback on purpose: a PlexError carries httpx's own message, which names the method,
# the URL and the status — the stack under it is this `except` and the client call, and
# says nothing the message does not. (The URL holds no token; it travels as a header.)
log.warning(
"subtitle extraction failed talking to Plex: %s",
e,
extra=logs.event("plex.subtitles.plex_error", rating_key=rating_key, sub_ord=sub_ord),
)
_fail(key, "plex_error")
except (
Exception
@@ -667,6 +792,8 @@ def _run(
finally:
with _LOCK:
_INFLIGHT.pop(ident, None)
if ident[0] == "full":
_FULL_OWNERS.pop(rating_key, None)
def _pass_failed(
@@ -732,10 +859,13 @@ def _extract_and_publish(rating_key: str, sub_ord: int, window_start: float | No
for i, s in enumerate(subs)
if s.get("index") is not None and not _is_bitmap(s)
}
start = window_start or 0.0
# A full pass starts where every track's leading coverage ends: a pass that ran out of time
# (`EXTRACT_TIMEOUT_S`) kept what it reached, and restarting from 0 each `FULL_RETRY_S` meant a file
# too big to read in one go never completed. Windows handle an offset start already; so does this.
start = window_start if window_start is not None else _full_pass_start(rating_key, targets)
length = (WINDOW_LEN_S + WINDOW_PRE_S) if window_start is not None else None
end = _FULL_END_S if length is None else start + length
texts, clean = _ffmpeg_extract(str(src), sorted(targets), window_start, length)
texts, clean = _ffmpeg_extract(str(src), sorted(targets), start or None, length)
for emb_ord, vtt in texts.items():
key = cache_key(rating_key, targets[emb_ord])
# A pass that exited non-zero holds only what ffmpeg reached before it died. Claim no more
@@ -756,6 +886,24 @@ def _extract_and_publish(rating_key: str, sub_ord: int, window_start: float | No
_pass_failed(rating_key, [key], window_start, (covered_end, end))
def _full_pass_start(rating_key: str, targets: dict[int, int]) -> float:
"""Where the full pass may begin: the end of the span from 0 that EVERY target track already has
(one read feeds them all, so the least-covered track decides), backed off by `WINDOW_PRE_S` so a
cue running across the join is read whole — `merge_vtt` folds the overlap. 0 when any track has
nothing from the start."""
ends = []
for ord_ in targets.values():
key = cache_key(rating_key, ord_)
ranges, _ = read_meta(key) if _exists(_vtt_path(key)) else ([], False)
lead = [b for a, b in ranges if a <= 0.0]
if not lead:
return 0.0
ends.append(max(lead))
if not ends:
return 0.0
return max(0.0, float(f"{min(ends) - WINDOW_PRE_S:.3f}"))
def _sidecar_vtt(db: Session, stream: dict) -> str:
"""An external subtitle file, fetched through Plex and converted to WebVTT."""
with PlexClient(db) as plex:
@@ -794,9 +942,10 @@ def _ffmpeg_extract(
cmd += ["-map", f"0:s:{emb_ord}"]
if length is not None:
cmd += ["-t", f"{length:.3f}"]
if start:
# The input seek lands on the keyframe at or BEFORE `start`, so a cue already running
# at `start` comes out at a negative time — and by default the muxer then shifts the
# WHOLE output so that cue sits at 0: every cue of the window late by the distance to
# WHOLE output so that cue sits at 0: every cue of the read late by the distance to
# that keyframe (1–5 s measured on a real library file, a whole GOP on a synthetic
# one), and doubled for good once the full pass merged in beside it. Kept negative, the
# times are exact on the zero-based clock (`_vtt_ts_to_s` reads the sign).
@@ -829,6 +978,16 @@ def _ffmpeg_extract(
try:
_, err = proc.communicate(timeout=_TERM_GRACE_S)
except subprocess.TimeoutExpired:
# TERM is what makes a timed-out pass worth keeping: ffmpeg closes its outputs
# on it. An ffmpeg that ignores TERM leaves only whole blocks already flushed,
# so the partial track is short for a reason the later line cannot tell apart.
log.info(
"the subtitle extraction ignored TERM and was killed",
extra=logs.event(
"plex.subtitles.extract_killed_after_term",
windowed=start is not None,
),
)
proc.kill()
_, err = proc.communicate()
finally:
+24 -12
View File
@@ -15,20 +15,18 @@ dead `enabled` column) is a separate follow-up. Cast + intro/credit markers are
item-detail/player path (P2), not here.
"""
import logging
from datetime import datetime, timezone
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.models import PlexCollection, PlexItem, PlexLibrary, PlexSeason, PlexShow
from app.plex import detail, people
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.lookups import by_rating_key
log = logging.getLogger("siftlode.plex")
log = logs.get_logger(__name__)
# Codecs/containers a browser can play in a native <video> without any transcoding.
_BROWSER_VIDEO = {"h264"}
_BROWSER_AUDIO = {"aac", "mp3", "opus", "vorbis"}
@@ -102,6 +100,9 @@ def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
bd = s.get("bitDepth")
try:
depth = int(bd) if bd else None
# silent-ok: unknown bit depth, which is also what most listings report. The
# playback class then falls back to the codec alone, and per-stream noise over a
# whole library would drown the sync's own lines.
except (TypeError, ValueError):
depth = None
hdr = str(s.get("colorTrc") or "").lower() in _HDR_TRANSFERS
@@ -112,6 +113,8 @@ def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
def _epoch(v) -> datetime | None:
try:
return datetime.fromtimestamp(int(v), tz=timezone.utc) if v else None
# silent-ok: a field Plex leaves out of most listing rows. None is a legitimate stored value,
# and every caller treats it as "not known" rather than as an error.
except (TypeError, ValueError, OSError):
return None
@@ -120,6 +123,7 @@ def _date(v):
"""Parse a Plex 'YYYY-MM-DD' string (originallyAvailableAt) → date, or None."""
try:
return datetime.strptime(v, "%Y-%m-%d").date() if v else None
# silent-ok: as `_epoch` above — an absent or unparsable air date is stored as no air date.
except (TypeError, ValueError):
return None
@@ -135,6 +139,7 @@ def _tags(meta: dict, key: str, limit: int | None = None) -> list | None:
def _rating(meta: dict) -> float | None:
try:
return float(meta.get("audienceRating") or meta.get("rating"))
# silent-ok: TypeError on `float(None)` is the ordinary path — most items carry no rating.
except (TypeError, ValueError):
return None
@@ -266,15 +271,19 @@ def sync(db: Session) -> dict:
# becomes filterable over successive runs (the cheap listing above only stored the top ~3).
try:
stats["people_enriched"] = _enrich_people_batch(db, plex)
except Exception as e: # best-effort — never fail the sync over the backfill
log.warning("Plex people enrich batch failed: %s", e)
except Exception: # noqa: BLE001 — best-effort; never fail the sync over a backfill
# With the traceback: a blind catch over a batch that fetches AND writes can be
# a Plex shape change, a DB error or our own bug, and `%s` told them apart badly.
log.exception("the Plex people-enrich batch failed")
# Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
# playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
try:
stats["media_probed"] = _probe_media_batch(db, plex)
except Exception as e: # best-effort — never fail the sync over the backfill
log.warning("Plex media probe batch failed: %s", e)
except Exception: # noqa: BLE001 — best-effort; never fail the sync over a backfill
log.exception("the Plex media-probe batch failed")
# silent-ok: the return value IS the report. `{"skipped": …}` goes to the scheduler, which
# logs the job result, and to the admin route, which answers it to the caller.
except PlexNotConfigured as e:
return {"skipped": str(e)}
except PlexError as e:
@@ -285,8 +294,8 @@ def sync(db: Session) -> dict:
# appear in ANY item's cast/directors — drop those rows, mirroring how Plex drops orphaned metadata.
try:
stats["people_pruned"] = people.prune_orphan_people(db)
except Exception as e: # best-effort housekeeping — never fail the sync over it
log.warning("Plex people prune failed: %s", e)
except Exception: # noqa: BLE001 — best-effort housekeeping; never fail the sync over it
log.exception("the Plex people prune failed")
log.info("Plex sync done: %s", stats)
return stats
@@ -325,7 +334,9 @@ def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH)
for row in rows:
try:
meta = plex.metadata(row.rating_key) or {}
except Exception as e: # transient — leave media_probed_at NULL so it retries next run
# No traceback and only debug: this is per item over a whole library, and the stack is
# the same `plex.metadata` call every time. The retry next run is the handling.
except Exception as e: # noqa: BLE001 — transient; leave media_probed_at NULL to retry
log.debug("Plex media probe fetch failed for %s: %s", row.rating_key, e)
continue
bit_depth, hdr = _video_stream_props(meta)
@@ -372,7 +383,8 @@ def _enrich_people_batch(db: Session, plex: PlexClient, limit: int = _ENRICH_BAT
for row in [*movies, *shows]:
try:
meta = plex.metadata(row.rating_key) or {}
except Exception as e:
# As in `_probe_media_batch` above: per item, so the message without the stack.
except Exception as e: # noqa: BLE001 — transient; leave people_enriched_at NULL
log.debug("Plex people enrich fetch failed for %s: %s", row.rating_key, e)
continue # leave people_enriched_at NULL so it's retried on a later run
people.enrich_row_people(db, row, people.cast_from_meta(meta))
+19 -5
View File
@@ -10,7 +10,6 @@ One-parser rule (mirrors app/plex/detail.py): `parse_tmdb` is the pure parser, `
the row. The enrich batch + scheduler entry live here too so the whole TMDB integration is one module.
"""
import logging
import time
from datetime import datetime, timezone
@@ -19,12 +18,11 @@ from sqlalchemy import func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.config import settings
from app.models import PlexItem, PlexPerson, PlexShow
log = logging.getLogger("siftlode.tmdb")
log = logs.get_logger(__name__)
# Politeness throttle: TMDB tolerates ~50 req/s; ~33/s leaves comfortable headroom for a shared host.
_MIN_INTERVAL = 0.03
@@ -142,6 +140,12 @@ def guid_to_tmdb_id(guid: str | None, media_type: str, tmdb: TmdbClient) -> int
try:
return int(ext)
except ValueError:
# The caller stamps the watermark on a None, so this title is never looked at again:
# one unreadable guid costs it its artwork and its cast permanently.
log.info(
"a Plex guid named TMDB but did not carry a numeric id",
extra=logs.event("plex.tmdb.guid_unparsable", provider=provider),
)
return None
source = {"imdb": "imdb_id", "tvdb": "tvdb_id"}.get(provider)
if not source:
@@ -499,7 +503,17 @@ def enrich_person(db: Session, person: PlexPerson) -> bool:
# doesn't tie up a threadpool worker for the default 30s.
with TmdbClient(db, timeout=8.0) as tmdb:
payload = tmdb.fetch_person(person.tmdb_person_id)
except TmdbError:
except TmdbError as exc:
# False leaves the person un-enriched and unstamped, so it is retried — but the page in
# front of the user right now shows an empty biography with nothing to say why.
log.warning(
"the TMDB person lookup failed",
extra=logs.event(
"plex.tmdb.person_enrich_failed",
person_id=person.id,
reason=type(exc).__name__,
),
)
return False
if payload is None:
person.tmdb_enriched_at = datetime.now(timezone.utc)
+22 -6
View File
@@ -17,14 +17,17 @@ and the incremental pull resolves a genuine divergence to Plex's state (see `_pu
running a last-write-wins comparison.
"""
import logging
# On the PlexError lines in this module: none of them carries a traceback, deliberately. A
# PlexError is httpx's own message — the method, the URL and the status of one call — and the stack
# beneath it is always this `except` plus `PlexClient._get`. The attrs beside the line say which
# user and which phase, which is the part the message cannot know.
from datetime import datetime, timezone
from sqlalchemy import func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.db import SessionLocal
from app.models import PlexItem, PlexLink, PlexState
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
@@ -36,8 +39,7 @@ from app.plex.sync import (
_paginate,
)
log = logging.getLogger("siftlode.plex.watch")
log = logs.get_logger(__name__)
# A resume position below this many seconds isn't worth mirroring — matches the player's own
# _PROGRESS_MIN_S (a stray few-second seek is not "in progress").
_PROGRESS_MIN_S = 5
@@ -148,6 +150,8 @@ def import_owner_watch_state(db: Session, link: PlexLink) -> dict:
link.initial_import_done = True
link.last_watch_sync_at = datetime.now(timezone.utc)
db.commit()
# silent-ok: the `{"skipped": …}` return is the report — the admin route answers it and the
# scheduler logs it on the job line.
except PlexNotConfigured as e:
return {"skipped": str(e)}
except PlexError as e:
@@ -326,8 +330,18 @@ def _resolve_account_id(db: Session, plex: PlexClient, link: PlexLink) -> int:
if int(a.get("id") or -1) == 1:
username = a.get("name")
break
except PlexError:
pass
except PlexError as exc:
# The fallback account id 1 is right for the admin-token setup, so sync keeps working —
# but the username is never stored, and the UI then has no name to show for the link,
# exactly as if the server had not offered one.
log.info(
"the Plex account list could not be read; the link keeps the default account",
extra=logs.event(
"plex.watch.accounts_probe_failed",
user_id=link.user_id,
reason=type(exc).__name__,
),
)
link.plex_account_id = acct_id
if username:
link.plex_username = username
@@ -566,6 +580,7 @@ def run_plex_watch_sync(db: Session) -> dict:
total["repushed"] += _repush_dirty(db, plex, link)
link.last_watch_sync_at = now
db.commit()
# silent-ok: the `{"skipped": …}` return is the report, as above.
except PlexNotConfigured as e:
return {"skipped": str(e)}
except PlexError as e:
@@ -617,6 +632,7 @@ def run_plex_watch_reconcile(db: Session) -> dict:
total["repushed"] += _repush_dirty(db, plex, link)
link.last_watch_sync_at = datetime.now(timezone.utc)
db.commit()
# silent-ok: the `{"skipped": …}` return is the report, as above.
except PlexNotConfigured as e:
return {"skipped": str(e)}
except PlexError as e:
+1
View File
@@ -37,6 +37,7 @@ class QuotaAction:
PLAYLISTS_DELETE = "playlists_delete"
VIDEOS_BACKFILL_RECENT = "videos_backfill_recent"
VIDEOS_BACKFILL_FULL = "videos_backfill_full"
VIDEOS_FEED_FALLBACK = "videos_feed_fallback"
VIDEOS_ENRICH = "videos_enrich"
VIDEOS_LOOKUP = "videos_lookup"
VIDEOS_SEARCH = "videos_search"
+2
View File
@@ -73,6 +73,8 @@ def _norm_ip(s: str) -> ipaddress.IPv4Address | ipaddress.IPv6Address | None:
plain-IPv4 allowlist still matches a peer that surfaces in mapped form. None for a non-IP string."""
try:
ip = ipaddress.ip_address(s.strip())
# silent-ok: parsing an untrusted header value, where "not an IP" is a documented answer this
# function returns — and a line per request is what an attacker would aim for.
except ValueError:
return None
return ip.ipv4_mapped if getattr(ip, "ipv4_mapped", None) else ip
+13 -6
View File
@@ -13,12 +13,12 @@ mutated from a single thread.
"""
import asyncio
import logging
from collections import defaultdict
from starlette.websockets import WebSocket
from app import logs
log = logging.getLogger("siftlode.realtime")
log = logs.get_logger(__name__)
class ConnectionManager:
@@ -42,9 +42,16 @@ class ConnectionManager:
for ws in list(self._conns.get(user_id, ())):
try:
await ws.send_json(payload)
except Exception:
# Stale socket; it'll be cleaned up by its own disconnect handler.
pass
except Exception as exc: # noqa: BLE001 — one dead socket must not stop the others
# Stale socket; it'll be cleaned up by its own disconnect handler. But this is
# also where a live update goes missing: the UI simply never changes, and there
# is no request behind it whose failure anyone could look up.
log.debug(
"a live update could not be pushed to a socket",
extra=logs.event(
"realtime.push_failed", user_id=user_id, reason=type(exc).__name__
),
)
def push(self, user_id: int, payload: dict) -> None:
"""Schedule a push onto the WebSocket event loop. Safe to call from a sync route
@@ -54,7 +61,7 @@ class ConnectionManager:
return
try:
asyncio.run_coroutine_threadsafe(self._push(user_id, payload), loop)
except Exception:
except Exception: # noqa: BLE001 — a closed loop or socket must not fail the caller
log.debug("ws push failed", exc_info=True)
+37
View File
@@ -0,0 +1,37 @@
"""A request the app refuses, said in a way the UI can put in the reader's language.
Every refusal the API answers carries a structured `detail` — `{code, reason, …params}` — and the client
translates it (`lib/api/core.ts` `localizeDetail`). A bare English string reached the Hungarian UI
verbatim: the blocking error dialog, and every inline surface that printed `HttpError.detail`, showed
whatever sentence the route happened to hand-write. `refused` is the generic family
(`errors.refused.<reason>`, the params interpolated into the sentence); the older specific codes
(`not_found`, `youtube`, `playlist`, the admin refusals, …) stay, since each is already translated.
A test (`test_http_details.py`) reads every `HTTPException` in the app and fails on a string detail,
and the frontend's contract test requires every `refused(…, "<reason>")` to be named in both
languages — so the rule is enforced, not described.
"""
from fastapi import HTTPException
def refused(status: int, reason: str, **params: object) -> HTTPException:
"""The refusal to raise: `raise refused(409, "view_name_taken", name=name)`. `params` travel in
the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see."""
return HTTPException(status_code=status, detail={"code": "refused", "reason": reason, **params})
# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the frontend's
# contract test can read every reason this can produce.
_SETTING_REASONS = {
"not_a_number": "setting_not_a_number",
"too_low": "setting_too_low",
"too_high": "setting_too_high",
"not_a_choice": "setting_not_a_choice",
"no_encryption_key": "no_encryption_key",
}
def setting_refused(exc) -> HTTPException:
"""A value the admin entered that `sysconfig.set_value` refused (`exc`: its `InvalidSetting`)."""
return refused(400, _SETTING_REASONS[exc.reason], key=exc.key, **exc.params)
+92 -4
View File
@@ -6,13 +6,16 @@ and "fetch a row this user owns or 404" idioms have one definition, one status c
one existence-leak policy.
"""
from typing import Any, TypeVar
from typing import IO, Any, TypeVar
from fastapi import HTTPException
from fastapi import HTTPException, Request
from sqlalchemy.orm import Session
from app.db import Base
from app.models import User
from app.issues import gitea
from app.refusal import refused
from app.youtube.client import YouTubeError, refusal_reason
M = TypeVar("M", bound=Base)
@@ -45,9 +48,94 @@ def owned_or_404(
) -> M:
"""Fetch a row by primary key that `user` owns, or raise 404. A row owned by someone
else 404s (NOT 403) on purpose — a 403 would confirm the id exists to a stranger.
Generalizes the near-identical per-route ownership helpers (e.g. plex's
`_own_playlist_or_404`, the saved-views owner check)."""
It replaced the near-identical per-route ownership helpers, and is called directly — a thin
`_own_x` wrapper around it is the ad-hoc shape CLAUDE.md's patterns table rules out."""
obj = get_or_404(db, model, id_, what=what)
if getattr(obj, owner_attr) != user.id:
raise not_found(what)
return obj
def youtube_refusal(exc: Exception) -> HTTPException:
"""The answer to an action YouTube refused: a 422 whose reason the UI translates
(`errors.youtube.<reason>`). One definition, because the routes had grown three shapes of it —
a bare English string quoting the upstream message, a playlist-scoped reason, and nothing at
all (a raw 500 from the manual syncs). The upstream failure is already logged where it
happened (`youtube.api.error`). A failure that is not the Data API's (the search scraper's) is
YouTube being unavailable to us — the retryable answer."""
reason = refusal_reason(exc) if isinstance(exc, YouTubeError) else "unavailable"
return HTTPException(status_code=422, detail={"code": "youtube", "reason": reason})
async def capped_body(request: Request, limit: int) -> bytes:
"""The request body, refused (413 `too_large`) the moment it passes `limit` bytes while still
arriving. Counted, not taken from Content-Length: a declared length can be absent (chunked) or
wrong, and counting stops reading at the same point a declared-length check would have."""
body = bytearray()
async for chunk in request.stream():
body += chunk
if len(body) > limit:
raise HTTPException(status_code=413, detail={"code": "too_large"})
return bytes(body)
async def capped_stream_to(
request: Request, out: IO[bytes], limit: int, refusal: HTTPException
) -> int:
"""`capped_body`'s sibling for a body too large to hold: the same counting rule, written
straight to `out` instead of accumulated, returning the size.
Two deliberate differences from `capped_body`. It never builds a `bytearray` — the upload route
admits 500 MiB by default, so holding the body would trade a disk problem for a memory one. And
the refusal is the CALLER'S, passed in rather than assumed: `capped_body`'s flat 413 is right
for a screenshot, while the upload route answers 422 `upload_too_large` carrying the admin
knob's own number, which is what its dialog shows the user. One counting rule, two answers,
neither of them a second copy of the loop.
The caller owns `out` and owns cleaning it up: a body refused half-way has already written what
arrived before the limit was passed.
**The refusal DRAINS the rest of the body before it is raised, and that is not politeness.**
Answering mid-upload and closing the connection is what the client sees as a network failure,
not as a refusal: the browser's XHR fires `onerror` before it can read the response, so a file
over the limit reported itself as "the server couldn't be reached" instead of naming the limit —
measured in a real browser, not reasoned about. Reading the remainder to nowhere lets the client
finish sending, read the 422 and say the true thing.
The bytes are discarded, never written, and the whole request is already bounded by
`bodylimit`'s per-route ceiling — so the worst case is bandwidth an authenticated caller was
sending anyway, against the old behaviour which wrote exactly those bytes to disk."""
size = 0
over = False
async for chunk in request.stream():
size += len(chunk)
if over:
continue # draining — see the docstring; counted, not stored
if size > limit:
over = True
continue
out.write(chunk)
if over:
raise refusal
return size
def issue_refused(exc: gitea.GiteaError) -> HTTPException:
"""The refusal an issue-tracker failure (E16) answers as — the send, the preview and the
admin's connection test alike. Literal reasons, one per case: the frontend's contract test reads
them to require a sentence for each in both languages."""
match exc.reason:
case "not_configured":
return refused(400, "issues_not_configured")
case "unreachable":
return refused(422, "issues_unreachable")
case "bad_token":
return refused(422, "issues_bad_token")
case "repo_not_found":
return refused(422, "issues_repo_not_found")
case "public_repo":
return refused(422, "issues_public_repo")
case "issues_disabled":
return refused(422, "issues_disabled")
case _:
return refused(422, "issues_rejected")
+27 -25
View File
@@ -39,6 +39,7 @@ from app.routes.auth import (
send_verification_link,
)
from app.utils import canonical_email, iso, valid_email
from app.refusal import refused
router = APIRouter(prefix="/api/admin", tags=["admin"])
@@ -169,7 +170,7 @@ def _activate_user_by_email(db: Session, email: str) -> None:
user = next((u for u in candidates if u.email == email), None) or next(iter(candidates), None)
if user is not None and not user.is_active:
user.is_active = True
db.commit()
db.flush() # the handler commits (CLAUDE.md: helpers flush)
def _decide(db: Session, invite_id: int, admin: User, approved: bool) -> Invite:
@@ -179,7 +180,7 @@ def _decide(db: Session, invite_id: int, admin: User, approved: bool) -> Invite:
# sending a duplicate decision mail, even flipping an already-approved invite to "denied" and
# mailing a rejection to an already-active account.
if inv.status != "pending":
raise HTTPException(status_code=409, detail="Invite already decided")
raise refused(409, "invite_decided")
inv.status = "approved" if approved else "denied"
inv.decided_at = datetime.now(timezone.utc)
inv.decided_by = admin.email
@@ -191,7 +192,7 @@ def _decide(db: Session, invite_id: int, admin: User, approved: bool) -> Invite:
target_id=inv.id,
summary=inv.email,
)
db.commit()
db.flush() # the handler commits, with the activation that goes with it
return inv
@@ -210,6 +211,9 @@ def approve_invite(
link, has_account, hours = _welcome_mail_parts(db, inv.email)
lang = mailtext.lang_for_email(db, inv.email)
background.add_task(email_mod.send_access_approved, inv.email, lang, link, has_account, hours)
# Before the response, so the mail (a background task, after it) never describes a decision
# the database does not hold yet.
db.commit()
return _serialize(inv, has_account=bool(_accounts_for(db, [inv.email])))
@@ -226,6 +230,7 @@ def deny_invite(
background.add_task(
email_mod.send_access_denied, inv.email, mailtext.lang_for_email(db, inv.email)
)
db.commit()
return _serialize(inv, has_account=bool(_accounts_for(db, [inv.email])))
@@ -242,7 +247,7 @@ def add_invite(
"""Manually whitelist an email (approved straight away). Convenience for the admin."""
email = body.email.strip().lower()
if not valid_email(email):
raise HTTPException(status_code=400, detail="Enter a valid email address.")
raise refused(400, "invalid_email")
# Reuse a canonically-equal row instead of adding a second one for the same mailbox. Prod carries
# exactly that pair for one person from 2026-06-16 (hartmann.peter1986@ / hartmannpeter1986@),
# which is how an admin ends up approving the same human twice and still seeing them bounce.
@@ -259,11 +264,13 @@ def add_invite(
inv.status = "approved"
inv.decided_at = datetime.now(timezone.utc)
inv.decided_by = admin.email
db.flush() # a new invite's id, for the audit row — before it, the row named no target
audit.record(
db, admin.id, AuditAction.INVITE_ADD, target_type="invite", target_id=inv.id, summary=email
)
db.commit()
# One transaction: the whitelisting and the activation it implies land together or not at all.
_activate_user_by_email(db, email)
db.commit()
return _serialize(inv, has_account=bool(_accounts_for(db, [email])))
@@ -366,10 +373,7 @@ def delete_invite(
promoted to admin on its next Google sign-in). The admin has to hear that, not guess it."""
inv = get_or_404(db, Invite, invite_id, what="invite")
if _accounts_for(db, [inv.email]):
raise HTTPException(
status_code=409,
detail="That address already has an account — delete the account instead.",
)
raise refused(409, "address_has_account")
still_allowed_by_env = _env_allowed(inv.email)
audit.record(
db,
@@ -435,19 +439,19 @@ def set_user_role(
account, and can't remove the last remaining admin. Emails the user when it actually changes."""
role = body.role
if role not in ("user", "admin"):
raise HTTPException(status_code=400, detail="role must be 'user' or 'admin'")
raise refused(400, "invalid_role")
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(status_code=400, detail="The demo account's role can't be changed.")
raise refused(400, "demo_role")
if target.id == admin.id:
raise HTTPException(status_code=400, detail="You can't change your own role.")
raise refused(400, "own_role")
if (
target.role == "admin"
and role == "user"
and not target.is_suspended
and count_admins(db, active_only=True) <= 1
):
raise HTTPException(status_code=400, detail="Can't remove the last admin.")
raise refused(400, "last_admin")
changed = target.role != role
if changed:
audit.record(
@@ -493,16 +497,16 @@ def set_user_suspended(
suspended = body.suspended
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(status_code=400, detail="The demo account can't be suspended.")
raise refused(400, "demo_suspend")
if target.id == admin.id:
raise HTTPException(status_code=400, detail="You can't suspend your own account.")
raise refused(400, "own_suspend")
if (
suspended
and target.role == "admin"
and not target.is_suspended
and count_admins(db, active_only=True) <= 1
):
raise HTTPException(status_code=400, detail="Can't suspend the last active admin.")
raise refused(400, "last_active_admin")
was_suspended = target.is_suspended
if was_suspended != suspended:
audit.record(
@@ -537,17 +541,15 @@ def admin_delete_user(
not the demo account, not yourself (use Settings → Account), and not the last admin."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(status_code=400, detail="The demo account can't be deleted.")
raise refused(400, "demo_delete")
if target.id == admin.id:
raise HTTPException(
status_code=400, detail="Delete your own account from Settings → Account."
)
raise refused(400, "own_delete")
if (
target.role == "admin"
and not target.is_suspended
and count_admins(db, active_only=True) <= 1
):
raise HTTPException(status_code=400, detail="Can't delete the last admin.")
raise refused(400, "last_admin")
# Record BEFORE the purge — the target row (and its email) is gone afterwards.
audit.record(
db,
@@ -691,7 +693,7 @@ def activate_user(
`_activate_user_by_email`). What was missing was any way to know they had to."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(status_code=400, detail="The demo account can't be activated.")
raise refused(400, "demo_activate")
if target.is_active:
return _serialize_user(target) # idempotent: a double click is not an error
if not is_allowed(db, target.email):
@@ -788,7 +790,7 @@ def add_demo_whitelist(
sign-in). Idempotent — re-adding an existing email just returns it."""
email = body.email.strip().lower()
if not valid_email(email):
raise HTTPException(status_code=400, detail="Enter a valid email address.")
raise refused(400, "invalid_email")
row = db.execute(select(DemoWhitelist).where(DemoWhitelist.email == email)).scalar_one_or_none()
if row is None:
row = DemoWhitelist(
@@ -835,7 +837,7 @@ def reset_demo(admin: User = Depends(admin_user), db: Session = Depends(get_db))
db, admin.id, AuditAction.DEMO_RESET, summary=f"{seeded} sample playlists re-seeded"
)
db.commit()
return {"reset": True, "playlists_seeded": seeded}
return {"ok": True, "reset": True, "playlists_seeded": seeded}
@router.post("/purge-discovery")
@@ -854,4 +856,4 @@ def purge_discovery(admin: User = Depends(admin_user), db: Session = Depends(get
after=result,
)
db.commit()
return result
return {"ok": True, **result}
+216
View File
@@ -0,0 +1,216 @@
"""The admin diagnostics console's read side (E22.6): query `log_events`, the server's stored log.
Admin-only and read-only — rows leave the table through the retention job (`log_retention_days`),
never through here. Reading is not audited: it changes nothing, and the console's live tail polls,
so a row per read would bury the audit log's real entries.
What a row holds was redacted when it was written (`logs.record_fields`, and the ingest route for a
browser's rows), so this serves the columns as they are; a value the redactor took out reads as
`[redacted]` / `[email]`, which is the point — the console shows that something was there.
Paging is a keyset over `(ts desc, id desc)`: a browser row's `ts` is clamped into its consent
window, so `ts` alone is not unique, and an offset would skip or repeat rows while the writer adds
new ones at the top. The live tail re-reads the first page rather than asking for "ids after N":
several processes write this table, and a batch's ids are taken when it inserts, not when it
commits, so a slower batch can land below an id the tail already passed — and would never show.
"""
import base64
import binascii
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import Text, and_, cast, func, or_, select
from sqlalchemy.orm import Session
from app import logsink
from app.auth import admin_user
from app.db import get_db
from app.models import LogEvent, User
from app.utils import like_escape
router = APIRouter(prefix="/api/admin/logs", tags=["admin"])
PAGE_MAX = 200
# A facet list is a chip row, not a report: past this many distinct events the rest are reachable
# through the free-text search, and the response stays small whatever a self-host logs.
FACET_EVENTS_MAX = 100
Level = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
# The filters a facet can leave out of its own count (`LogFilters.conditions`).
Facet = Literal["levels", "sources", "user", "event"]
# `none` selects the rows that have no account / no event code (a scheduler line, an anonymous
# request) — a real value to filter on, not the absence of a filter.
NONE = "none"
def _utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the column is `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def _encode_cursor(row: LogEvent) -> str:
return base64.urlsafe_b64encode(json.dumps([row.ts.isoformat(), row.id]).encode()).decode()
def _decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
return _utc(datetime.fromisoformat(ts)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def _accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the
line was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
@dataclass(frozen=True)
class LogFilters:
"""The console's filters, shared by the page and its facet counts."""
levels: list[str]
sources: list[str]
user: str | None
event: str | None
request_id: str | None
job_id: int | None
q: str | None
since: datetime | None
until: datetime | None
def conditions(self, skip: Facet | None = None) -> list:
"""The WHERE conditions, ANDed by the caller. `skip` leaves one filter out: a facet counts
its own values under every OTHER filter, so picking a level does not zero the other
levels' counts (the way back), while a request pivot does narrow them (honest counts)."""
out = []
if self.levels and skip != "levels":
out.append(LogEvent.level.in_(self.levels))
if self.sources and skip != "sources":
out.append(LogEvent.source.in_(self.sources))
if self.user is not None and skip != "user":
out.append(
LogEvent.user_id.is_(None)
if self.user == NONE
else LogEvent.user_id == int(self.user)
)
if self.event is not None and skip != "event":
out.append(
LogEvent.event.is_(None) if self.event == NONE else LogEvent.event == self.event
)
if self.request_id is not None:
out.append(LogEvent.request_id == self.request_id)
if self.job_id is not None:
out.append(LogEvent.job_id == self.job_id)
if self.since is not None:
out.append(LogEvent.ts >= _utc(self.since))
if self.until is not None:
out.append(LogEvent.ts < _utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
or_(
LogEvent.msg.ilike(like),
LogEvent.event.ilike(like),
LogEvent.request_id.ilike(like),
LogEvent.exc.ilike(like),
cast(LogEvent.attrs, Text).ilike(like),
)
)
return out
def log_filters(
levels: list[Level] = Query(default=[]),
sources: list[str] = Query(default=[], max_length=8),
user: str | None = Query(default=None, pattern=r"^(none|[0-9]{1,10})$"),
event: str | None = Query(
default=None, max_length=64, pattern=r"^(none|[a-z][a-z0-9_]*(\.[a-z0-9_]+)*)$"
),
request_id: str | None = Query(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$"),
job_id: int | None = Query(default=None, ge=1),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> LogFilters:
return LogFilters(list(levels), list(sources), user, event, request_id, job_id, q, since, until)
@router.get("")
def list_logs(
filters: LogFilters = Depends(log_filters),
before: str | None = Query(default=None, max_length=200),
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""One page of rows, newest first, matching every filter given (they AND together); `before`
continues from a previous page's `next_cursor`."""
query = select(LogEvent).where(*filters.conditions())
if before is not None:
ts, row_id = _decode_cursor(before)
# The `ts <=` conjunct is implied by the OR; it is there so the planner can bound the
# `ts` index scan instead of filtering every newer row first.
query = query.where(
and_(
LogEvent.ts <= ts,
or_(LogEvent.ts < ts, and_(LogEvent.ts == ts, LogEvent.id < row_id)),
)
)
# One more than asked, to know whether there is a next page without a second count query.
rows = (
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
)
more = len(rows) > limit
rows = rows[:limit]
return {
"items": [logsink.serialize(r) for r in rows],
"accounts": _accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": _encode_cursor(rows[-1]) if more else None,
}
@router.get("/facets")
def log_facets(
filters: LogFilters = Depends(log_filters),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""What the filter chips offer, with counts: the levels, sources and accounts present, and the
most frequent events — each counted under every filter but its own (`LogFilters.conditions`).
`total` is the "all levels" count: every filter but the level one."""
def counts(col, skip: Facet, cap: int | None = None) -> list[dict]:
n = func.count().label("n")
stmt = select(col, n).where(*filters.conditions(skip)).group_by(col).order_by(n.desc(), col)
if cap is not None:
stmt = stmt.limit(cap)
return [
{"value": NONE if value is None else value, "count": count}
for value, count in db.execute(stmt).all()
]
users = counts(LogEvent.user_id, "user")
total = select(func.count()).select_from(LogEvent).where(*filters.conditions("levels"))
return {
"levels": counts(LogEvent.level, "levels"),
"sources": counts(LogEvent.source, "sources"),
"events": counts(LogEvent.event, "event", FACET_EVENTS_MAX),
"users": [{**u, "value": str(u["value"])} for u in users],
"accounts": _accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"total": db.scalar(total) or 0,
}
+1 -1
View File
@@ -68,4 +68,4 @@ def clear_audit(admin: User = Depends(admin_user), db: Session = Depends(get_db)
n = db.execute(delete(AuditLog)).rowcount or 0
audit.record(db, admin.id, AuditAction.AUDIT_CLEAR, summary=f"Cleared {n} audit entries")
db.commit()
return {"cleared": n}
return {"ok": True, "cleared": n}
+61 -37
View File
@@ -17,6 +17,7 @@ from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import email as email_mod
from app import logs
from app import mailtext
from app import sysconfig
from app.auth import (
@@ -38,10 +39,11 @@ from app.auth import (
)
from app.config import settings
from app.db import SessionLocal, get_db
from app.models import AuthToken, DemoWhitelist, Invite, OAuthToken, User
from app.models import AuthToken, DemoWhitelist, Invite, OAuthToken, User, display_text
from app.ratelimit import RateLimiter, client_key
from app.security import encrypt, hash_password, hash_token, verify_password
from app.utils import canonical_email, valid_email
from app.refusal import refused
VERIFY_TTL = timedelta(hours=24)
# A confirmation link that rides an APPROVAL or an invitation lives longer than one the person is
@@ -377,7 +379,7 @@ async def callback(
userinfo = token.get("userinfo")
if not userinfo or not userinfo.get("sub"):
raise HTTPException(status_code=400, detail="No user info returned by Google")
raise refused(400, "google_no_userinfo")
sub = userinfo["sub"]
email = (userinfo.get("email") or "").lower()
@@ -386,7 +388,15 @@ async def callback(
return _complete_link(request, db, link_uid, link_explicit, sub, userinfo, token)
if not is_allowed(db, email):
log.warning("Login denied (not approved): %s", email or "<no email>")
# No uid to name — having no account here is the branch. And the address is left out on
# purpose: `redact_text` turns it into `[email]` before either sink sees it, so a line
# whose only variable is always `[email]` says nothing. WHO asked is in the pending
# invite recorded just below, which is where an admin looks anyway.
log.warning(
"Google sign-in denied: address not approved",
# log-pii-ok: whether an address came back, not what it was
extra=logs.event("auth.login.denied", reason="not_approved", has_email=bool(email)),
)
# A denied Google login doubles as an access request: record it for the admin and
# land the user on a friendly "request received" screen instead of a raw 403.
if email:
@@ -419,15 +429,20 @@ async def callback(
if existing.google_sub is not None and existing.google_sub != sub:
# A different Google identity is already bound to this email — refuse rather than
# hijack it. (Near-impossible with real Google accounts: email↔sub is stable.)
log.warning("Google login email collision (different sub): %s", email)
log.warning(
"Google sign-in refused: a different Google identity holds this address",
extra=logs.event("auth.login.denied", reason="sub_collision", uid=existing.id),
)
return RedirectResponse(url=_access_notice_url("denied"))
if not userinfo.get("email_verified", False):
# Adopting (and activating) a pre-existing password account requires Google to
# actually attest the address — otherwise an unverified Google email that merely
# collides with a pending registration could seize that account. Defense-in-depth.
log.warning(
"Google login: unverified email, refusing to adopt account: %s",
email,
"Google sign-in refused: unverified address, not adopting the account",
extra=logs.event(
"auth.login.denied", reason="email_unverified", uid=existing.id
),
)
return RedirectResponse(url=_access_notice_url("denied"))
user = existing
@@ -445,7 +460,10 @@ async def callback(
# A suspended account can't sign in by any method. Block before establishing the session and
# tell the (verified) owner why — a successful Google auth proves they control this account.
if user.is_suspended:
log.info("Suspended account blocked at Google login: %s", email)
log.info(
"Google sign-in blocked: the account is suspended",
extra=logs.event("auth.login.denied", reason="suspended", uid=user.id),
)
_notify_suspended(background, user)
return RedirectResponse(url="/?login=suspended")
# Record what Google actually answers with, beside the address the user registered with. The two
@@ -454,7 +472,7 @@ async def callback(
if email:
user.google_email = email
_sync_google_email(db, user, email)
user.display_name = userinfo.get("name")
user.display_name = display_text(userinfo.get("name"))
user.avatar_url = userinfo.get("picture")
# ADMIN_EMAILS (env) is the bootstrap admin list and always wins, so the configured admin can't
# be locked out. Everyone else's role is managed in the admin UI and stored in the DB, so a
@@ -481,7 +499,10 @@ async def callback(
_store_token(db, user, token)
_establish_session(request, user)
db.commit()
log.info("Login: %s (id=%s, role=%s)", email, user.id, user.role)
log.info(
"signed in",
extra=logs.event("auth.login.ok", method="google", uid=user.id, role=user.role),
)
return RedirectResponse(url="/")
@@ -573,7 +594,7 @@ def _complete_link(
if linked_email:
user.google_email = linked_email # see the sign-in callback: the other half of the identity
if not user.display_name:
user.display_name = userinfo.get("name")
user.display_name = display_text(userinfo.get("name"))
if not user.avatar_url:
user.avatar_url = userinfo.get("picture")
_store_token(db, user, token)
@@ -633,15 +654,15 @@ async def request_access(
endpoint can't be probed for throttling either."""
email = (body.email or "").strip().lower()
if not valid_email(email):
raise HTTPException(status_code=400, detail="Enter a valid email address.")
raise refused(400, "invalid_email")
if not _request_access_limiter.allow(client_key(request)):
return {"status": "pending"}
return {"ok": True, "status": "pending"}
if is_allowed(db, email):
return {"status": "pending"} # already allowed — don't reveal it
return {"ok": True, "status": "pending"} # already allowed — don't reveal it
req = upsert_pending_invite(db, email, body.lang)
if req.is_new:
notify_admins_of_request(db, background, email)
return {"status": "pending"}
return {"ok": True, "status": "pending"}
@router.post("/demo")
@@ -652,7 +673,7 @@ def demo_login(body: EmailIn, request: Request, db: Session = Depends(get_db)) -
exactly like a non-match (authenticated=false) so the field can't be hammered as an
enumeration oracle. On a match it sets the session and the client reloads into the app."""
if not _demo_limiter.allow(client_key(request)):
return {"authenticated": False}
return {"ok": True, "authenticated": False}
email = (body.email or "").strip().lower()
if valid_email(email) and is_demo_allowed(db, email):
demo = get_or_create_demo_user(db)
@@ -661,9 +682,10 @@ def demo_login(body: EmailIn, request: Request, db: Session = Depends(get_db)) -
request.session.clear()
request.session["user_id"] = demo.id
_remember_epoch(request, demo)
log.info("Demo login (key=%s, demo_id=%s)", email, demo.id)
return {"authenticated": True}
return {"authenticated": False}
# The demo key IS an address, so it is not in the line — see the denied branch above.
log.info("signed in", extra=logs.event("auth.login.ok", method="demo", uid=demo.id))
return {"ok": True, "authenticated": True}
return {"ok": True, "authenticated": False}
def _establish_session(request: Request, user: User) -> None:
@@ -794,23 +816,20 @@ def register(
password = body.password or ""
# Input validation is safe to reveal (independent of whether the email exists).
if not valid_email(email):
raise HTTPException(status_code=400, detail="Enter a valid email address.")
raise refused(400, "invalid_email")
if len(password) < PASSWORD_MIN_LEN:
raise HTTPException(
status_code=400,
detail=f"Password must be at least {PASSWORD_MIN_LEN} characters.",
)
raise refused(400, "weak_password", min=PASSWORD_MIN_LEN)
if not sysconfig.get_bool(db, "allow_registration"):
raise HTTPException(status_code=403, detail="Registration is currently closed.")
raise refused(403, "registration_closed")
if not _register_limiter.allow(client_key(request)):
return {"status": "ok"} # silently throttle; uniform response
return {"ok": True, "status": "ok"} # silently throttle; uniform response
# Do the existence check + account creation OFF the response path (a background task with its own
# session). The create path hashes the password + writes several rows + schedules emails; doing it
# inline would make a NEW email respond measurably slower than an already-registered one (which
# skips all that) — a timing oracle for enumeration. Off-path, any valid email responds the same.
background.add_task(_register_account, email, password, body.lang)
return {"status": "ok"}
return {"ok": True, "status": "ok"}
def _register_account(email: str, password: str, lang: str | None = None) -> None:
@@ -880,7 +899,12 @@ def _register_account(email: str, password: str, lang: str | None = None) -> Non
except IntegrityError:
# A concurrent registration for the same new email won the race between the pre-check
# above and this flush (the unique-email constraint fires). Same already-registered
# no-op as the pre-check hit — roll back and stop.
# no-op as the pre-check hit — roll back and stop. The line carries nothing: the
# answer to the caller is deliberately identical either way, and so is this.
log.info(
"a registration lost the race for its address",
extra=logs.event("auth.register.race_lost"),
)
db.rollback()
return
upsert_pending_invite(db, email) # admin-approval gate
@@ -971,11 +995,11 @@ def verify_email_resend(
mail had no way back. Uniform response + timing regardless of whether the email has a pending
account (the lookup runs off the response path), so it can't probe for registered emails."""
if not _verify_resend_limiter.allow(client_key(request)):
return {"status": "ok"} # silently throttle; uniform response
return {"ok": True, "status": "ok"} # silently throttle; uniform response
email = (body.email or "").strip().lower()
if valid_email(email):
background.add_task(_resend_verify_if_eligible, email)
return {"status": "ok"}
return {"ok": True, "status": "ok"}
def _resend_verify_if_eligible(email: str) -> None:
@@ -1049,7 +1073,10 @@ def password_login(
raise _login_refused(403, "pending_approval", user)
_establish_session(request, user)
db.commit()
log.info("Password login: %s (id=%s, role=%s)", email, user.id, user.role)
log.info(
"signed in",
extra=logs.event("auth.login.ok", method="password", uid=user.id, role=user.role),
)
return {"ok": True}
@@ -1062,14 +1089,14 @@ def password_reset_request(
"""Request a password-reset link. Uniform response + timing regardless of whether the email has a
password account (the lookup runs off the response path), so it can't probe for registered emails."""
if not _reset_limiter.allow(client_key(request)):
return {"status": "ok"}
return {"ok": True, "status": "ok"}
email = (body.email or "").strip().lower()
# Do the account lookup + token issue + email OFF the response path (a background task with its
# own session), so the endpoint takes the same time for any valid email whether or not it has a
# password account — no timing-based enumeration. Any valid email schedules the same task.
if valid_email(email):
background.add_task(_send_reset_if_eligible, email)
return {"status": "ok"}
return {"ok": True, "status": "ok"}
def _send_reset_if_eligible(email: str) -> None:
@@ -1264,13 +1291,10 @@ def set_password(
raise HTTPException(status_code=403, detail={"code": "demo"})
new_password = body.password or ""
if len(new_password) < PASSWORD_MIN_LEN:
raise HTTPException(
status_code=400,
detail=f"Password must be at least {PASSWORD_MIN_LEN} characters.",
)
raise refused(400, "weak_password", min=PASSWORD_MIN_LEN)
if user.password_hash:
if not verify_password(body.current_password or "", user.password_hash):
raise HTTPException(status_code=403, detail="Current password is incorrect.")
raise refused(403, "wrong_password")
user.password_hash = hash_password(new_password)
user.password_revoked_at = None # same as the reset path: a password exists again
# SA4: a password change logs out every OTHER session; re-stamp this cookie at the new epoch so
+56 -33
View File
@@ -2,15 +2,14 @@
hidden, tags) and per-channel sync state. Local reads/writes plus YouTube-side writes
(subscribe / unsubscribe) that require the optional write scope."""
import logging
from datetime import datetime, timezone
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
from fastapi import APIRouter, BackgroundTasks, Depends
from pydantic import BaseModel
from sqlalchemy import func, select, update
from sqlalchemy.orm import Session
from app import quota
from app import logs, quota
from app.auth import admin_user, current_user, require_human, require_write_scope
from app.db import get_db
from app.models import (
@@ -26,16 +25,17 @@ from app.models import (
User,
Video,
)
from app.routes._common import get_or_404, not_found, owned_or_404
from app.routes._common import get_or_404, not_found, owned_or_404, youtube_refusal
from app.sync.explore import explore_ingest_page
from app.sync.runner import backfill_channel_now, run_recent_backfill
from app.sync.subscriptions import apply_channel_details
from app.utils import iso
from app.youtube.client import YouTubeClient, YouTubeError
from app.refusal import refused
router = APIRouter(prefix="/api/channels", tags=["channels"])
log = logging.getLogger("siftlode.api")
log = logs.get_logger(__name__)
@router.get("")
@@ -166,7 +166,15 @@ def discover_channels(
user.yt_channel_id = own
db.commit()
except YouTubeError as exc:
log.warning("discovery: own-channel resolve failed (user %s): %s", user.id, exc)
# No traceback anywhere in this file's YouTube handlers: for an upstream answer
# `_upstream_failure` already logged the structured `youtube.api.error`, and the
# message names the call, its status and its reason (or, with no answer at all, the
# transport failure). The stack under it is this `except` and the client call.
log.warning(
"discovery: own-channel resolve failed: %s",
exc,
extra=logs.event("channels.discovery.own_channel_failed", reason=exc.reason),
)
rows = _discovery_rows(db, user)
@@ -190,7 +198,11 @@ def discover_channels(
rows, key=lambda r: (-int(r[1]), r[0].title is None, (r[0].title or "").lower())
)
except YouTubeError as exc:
log.warning("discovery: channel enrich failed (user %s): %s", user.id, exc)
log.warning(
"discovery: channel enrich failed: %s",
exc,
extra=logs.event("channels.discovery.enrich_failed", reason=exc.reason),
)
db.rollback()
return [
@@ -298,7 +310,13 @@ def channel_detail(
db.commit()
channel = db.get(Channel, channel_id)
except YouTubeError as exc:
log.warning("channel detail enrich failed (%s): %s", channel_id, exc)
log.warning(
"channel detail enrich failed: %s",
exc,
extra=logs.event(
"channels.detail.enrich_failed", channel_id=channel_id, reason=exc.reason
),
)
db.rollback()
sub = db.execute(
select(Subscription).where(
@@ -366,12 +384,9 @@ def explore_channel(
backfill quota reserve; blocked for the demo account (it spends shared quota)."""
channel = get_or_404(db, Channel, channel_id, what="channel")
if _is_blocked(db, user, channel_id):
raise HTTPException(status_code=403, detail="You've blocked this channel.")
raise refused(403, "channel_blocked")
if quota.backfill_reserve_reached():
raise HTTPException(
status_code=429,
detail="The shared daily quota reserve is reached — try exploring later.",
)
raise refused(429, "quota_reserve")
# Record/refresh the exploration (DB only) before ingesting, so the explorer can see the
# videos as soon as they land. Flag the channel ephemeral unless it's organically followed.
@@ -407,8 +422,9 @@ def explore_channel(
channel = db.get(Channel, channel_id)
result = explore_ingest_page(db, yt, channel, page_token)
except YouTubeError as exc:
raise HTTPException(status_code=422, detail=f"YouTube couldn't load this channel: {exc}")
raise youtube_refusal(exc)
return {
"ok": True,
"channel_id": channel_id,
"ingested": result["ingested"],
"next_page_token": result["next_page_token"],
@@ -490,6 +506,7 @@ def update_channel(
background.add_task(backfill_channel_now, channel_id, user.id)
return {
"ok": True,
"id": channel_id,
"priority": sub.priority,
"hidden": sub.hidden,
@@ -524,7 +541,7 @@ def reset_backfill(
db.commit()
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_BACKFILL_RECENT):
run_recent_backfill(db, [channel], max_channels=1)
return {"id": channel_id, "reset": True}
return {"ok": True, "id": channel_id, "reset": True}
@router.post("/{channel_id}/subscribe")
@@ -545,7 +562,7 @@ def subscribe(
)
).scalar_one_or_none()
if existing is not None:
raise HTTPException(status_code=409, detail="Already subscribed to this channel.")
raise refused(409, "already_subscribed")
with (
quota.attribute(user.id, quota.QuotaAction.CHANNELS_SUBSCRIBE),
YouTubeClient(db, user) as yt,
@@ -559,13 +576,14 @@ def subscribe(
# subscription resync fills in the resource id. Any other YouTube error is a real
# problem — surface it with a clear message (422, not a 502 that the client would
# mistake for a transient "connection lost").
msg = str(exc).lower()
if "already exists" in msg or "duplicate" in msg:
# On the reason code, never on the prose: the error no longer carries the body (it is
# the request quoted back), and a message YouTube rewords would silently turn "already
# following" back into a 422 the user cannot act on. `reason` is comma-joined when the
# body listed several.
if "subscriptionDuplicate" in exc.reason.split(","):
yt_sub_id = ""
else:
raise HTTPException(
status_code=422, detail=f"YouTube couldn't subscribe you: {exc}"
)
raise youtube_refusal(exc)
# Enrich a stub channel (title/thumbnail/subscriber count) so it shows up properly right
# away, on BOTH the normal and the desync path — a resurfaced stub still needs metadata to
# render. Keep the client open (we're still inside the `with`); a metadata failure here
@@ -574,7 +592,15 @@ def subscribe(
try:
apply_channel_details(db, yt.get_channels([channel_id]))
except YouTubeError as exc:
log.warning("subscribe: stub enrich failed (%s): %s", channel_id, exc)
log.warning(
"subscribe: stub enrich failed: %s",
exc,
extra=logs.event(
"channels.subscribe.enrich_failed",
channel_id=channel_id,
reason=exc.reason,
),
)
db.add(
Subscription(
user_id=user.id,
@@ -594,7 +620,7 @@ def subscribe(
.values(via_explore=False)
)
db.commit()
return {"subscribed": channel_id}
return {"ok": True, "subscribed": channel_id}
@router.delete("/{channel_id}/subscription")
@@ -608,10 +634,7 @@ def unsubscribe(
should Hide instead."""
sub = _user_subscription(db, user, channel_id)
if not sub.yt_subscription_id:
raise HTTPException(
status_code=400,
detail="No YouTube subscription id on record — sync subscriptions first.",
)
raise refused(400, "no_subscription_id")
try:
with (
quota.attribute(user.id, quota.QuotaAction.CHANNELS_UNSUBSCRIBE),
@@ -621,10 +644,10 @@ def unsubscribe(
except YouTubeError as exc:
# 422 (a real, explainable error → speaking modal), not 502 which the client treats as
# a transient "connection lost" gateway blip.
raise HTTPException(status_code=422, detail=f"YouTube couldn't unsubscribe you: {exc}")
raise youtube_refusal(exc)
db.delete(sub)
db.commit()
return {"unsubscribed": channel_id}
return {"ok": True, "unsubscribed": channel_id}
@router.get("/blocked/list")
@@ -665,7 +688,7 @@ def block_channel(
)
)
db.commit()
return {"blocked": channel_id}
return {"ok": True, "blocked": channel_id}
@router.delete("/{channel_id}/block")
@@ -680,7 +703,7 @@ def unblock_channel(
)
)
db.commit()
return {"unblocked": channel_id}
return {"ok": True, "unblocked": channel_id}
class AttachTagIn(BaseModel):
@@ -715,7 +738,7 @@ def attach_tag(
)
)
db.commit()
return {"channel_id": channel_id, "tag_id": body.tag_id}
return {"ok": True, "channel_id": channel_id, "tag_id": body.tag_id}
@router.delete("/{channel_id}/tags/{tag_id}")
@@ -735,4 +758,4 @@ def detach_tag(
if row is not None:
db.delete(row)
db.commit()
return {"channel_id": channel_id, "tag_id": tag_id}
return {"ok": True, "channel_id": channel_id, "tag_id": tag_id}
+27 -13
View File
@@ -4,7 +4,7 @@ app.sysconfig. Secret values (e.g. SMTP password) are write-only — never retur
import re
from typing import Any
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy.orm import Session
@@ -17,6 +17,10 @@ from app.audit import AuditAction
from app.auth import admin_user
from app.db import get_db
from app.models import User
from app.refusal import refused
from app.issues import gitea
from app.routes._common import issue_refused, not_found
from app.refusal import setting_refused
router = APIRouter(prefix="/api/admin/config", tags=["admin"])
@@ -59,19 +63,16 @@ def set_config(
) -> dict:
s = sysconfig.spec(key)
if s is None:
raise HTTPException(status_code=404, detail="Unknown config key")
raise not_found("config_key")
if "value" not in body.model_fields_set:
raise HTTPException(status_code=400, detail="Missing 'value'")
raise refused(400, "value_missing")
if s.secret and not sysconfig.secrets_manageable():
raise HTTPException(
status_code=400,
detail="Set TOKEN_ENCRYPTION_KEY to store secrets in the database.",
)
raise refused(400, "no_encryption_key")
old = None if s.secret else sysconfig.get(db, key)
try:
sysconfig.set_value(db, key, body.value)
except (TypeError, ValueError) as exc:
raise HTTPException(status_code=400, detail=str(exc) or "Invalid value")
except sysconfig.InvalidSetting as exc:
raise setting_refused(exc) from None
# Never log a secret's value — only that it was changed, by whom. Non-secret values are logged
# but with URL credentials redacted (a proxy/URL setting can embed user:pass). Skip the row when
# a non-secret value didn't actually change (a no-op re-save shouldn't add trail noise); secrets
@@ -101,7 +102,7 @@ def reset_config(
) -> dict:
s = sysconfig.spec(key)
if s is None:
raise HTTPException(status_code=404, detail="Unknown config key")
raise not_found("config_key")
old = None if s.secret else sysconfig.get(db, key)
sysconfig.reset(db, key)
audit.record(
@@ -126,8 +127,21 @@ def send_test_email(
"""Send a test email to the requesting admin using the current (DB or env) SMTP config —
lets the admin verify the settings actually work."""
if not email_mod.email_enabled():
raise HTTPException(status_code=400, detail="SMTP is not configured.")
raise refused(400, "smtp_off")
sent = email_mod.send_test(user.email, mailtext.lang_for_user(user))
if not sent:
raise HTTPException(status_code=422, detail="SMTP send failed — check the settings.")
return {"sent": True, "to": user.email}
raise refused(422, "smtp_failed")
return {"ok": True, "sent": True, "to": user.email}
@router.post("/test-issues")
def test_issue_tracker(
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Check the configured issue tracker (E16): the repository answers with this token, takes
issues, and is PRIVATE — the one it must be, since an issue carries its reporter's log."""
try:
return {"ok": True, **gitea.check(db)}
except gitea.GiteaError as exc:
raise issue_refused(exc) from None
+418
View File
@@ -0,0 +1,418 @@
"""Debug sessions (E16): Start → reproduce → Stop, the screenshots, and the evidence under review.
The session itself — what tags a request, its cap, its level — is `app/debugsession.py`. These routes
are the user's handle on their OWN sessions only: another user's id 404s (`owned_or_404`), including
for an admin, who reads everyone's lines in the diagnostics console instead. The shared demo account
cannot record: its log would interleave every visitor under one account.
Screenshots arrive as a raw image body (`image/png`, `image/jpeg`, `image/webp`), not a multipart
form — one image per request, capped while it arrives, and kept only if its bytes really are one of
those types, whatever the Content-Type claimed.
"""
import json
from typing import Any
from fastapi import APIRouter, Depends, HTTPException, Request, Response
from pydantic import BaseModel, Field
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import audit, debugsession, logs, logsink
from app.audit import AuditAction
from app.auth import require_human
from app.db import get_db, release_read_connection
from app.issues import bundle, gitea
from app.ratelimit import RateLimiter
from app.refusal import refused
from app.models import DebugScreenshot, DebugSession, User
from app.routes._common import capped_body, issue_refused, not_found, owned_or_404
from app.utils import iso
router = APIRouter(prefix="/api/debug-sessions", tags=["diagnostics"])
log = logs.get_logger(__name__)
# Sessions listed in Settings → Diagnosztika, newest first: enough to reach last week's unsent draft.
LIST_MAX = 20
class StartIn(BaseModel):
level: debugsession.SessionLevel
_view = debugsession.view
# New sessions a user may open per hour (joining the running one is not counted): a session is a
# place to keep screenshots, and this — with the per-user storage cap — bounds how many there are.
_new_sessions = RateLimiter(max_events=20, window_seconds=3600)
@router.post("")
def start_session(
body: StartIn,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Start recording, or join the recording already running (another tab started it)."""
joining = debugsession.open_session(db, user) is not None
if not joining and not _new_sessions.allow(str(user.id)):
raise refused(429, "debug_session_rate_limited", limit=_new_sessions.max_events)
try:
session = debugsession.start(db, user, body.level)
db.commit()
except IntegrityError:
# A concurrent Start won the one-open-per-user index: join that one.
db.rollback()
session = debugsession.open_session(db, user)
if session is None:
raise refused(409, "debug_session_conflict") from None
# The rest of THIS request is part of the session too.
level, until = debugsession.level_for(user, session)
logs.set_user(user.id, level, until, session.id)
log.info(
"debug session started",
extra=logs.event("debug.session.started", level=session.level, until=iso(session.until)),
)
return {"ok": True, "session": _view(db, session)}
@router.post("/{sid}/stop")
def stop_session(
sid: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
session = owned_or_404(db, DebugSession, sid, user, what="debug_session")
if session.ended_at is None:
# Logged BEFORE the session closes, while this request still carries its tag: the last line
# of the evidence is the Stop itself.
log.info("debug session stopped", extra=logs.event("debug.session.stopped"))
debugsession.stop(session)
db.commit()
logs.end_session_elevation(session.id)
return {"ok": True, "session": _view(db, session)}
@router.get("")
def list_sessions(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
sessions = db.scalars(
select(DebugSession)
.where(DebugSession.user_id == user.id)
.order_by(DebugSession.started_at.desc())
.limit(LIST_MAX)
).all()
return {"items": [_view(db, s) for s in sessions]}
@router.get("/{sid}")
def get_session(
sid: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
return _view(db, owned_or_404(db, DebugSession, sid, user, what="debug_session"))
@router.get("/{sid}/events")
def session_events(
sid: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
"""The rows an issue from this session would carry — browser and server, in time order."""
session = owned_or_404(db, DebugSession, sid, user, what="debug_session")
rows, total = debugsession.bundle(db, session)
return {"items": [logsink.serialize(r) for r in rows], "total": total}
# --- the draft and its export ---------------------------------------------------------------
# The composer's own fields (steps, description…) are its business, kept whole so it reopens as it
# was — bounded here so a draft cannot become a place to store anything else.
FORM_MAX_BYTES = 64 * 1024
# Zips a user may build per minute: each holds every screenshot in memory while it is built.
_exports = RateLimiter(max_events=6, window_seconds=60)
class DraftIn(BaseModel):
title: str = Field(max_length=200)
# The rendered markdown the export and the issue carry.
body: str = Field(max_length=50_000)
# Log rows taken out of the evidence in the review.
excluded: list[int] = Field(default_factory=list, max_length=debugsession.BUNDLE_MAX)
form: dict[str, Any] = Field(default_factory=dict)
@router.put("/{sid}/draft")
def save_draft(
sid: str,
body: DraftIn,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
session = owned_or_404(db, DebugSession, sid, user, what="debug_session")
if len(json.dumps(body.form, ensure_ascii=False).encode()) > FORM_MAX_BYTES:
raise HTTPException(status_code=413, detail={"code": "too_large"})
# Under the lock, and refused while a send is under way: the issue is being built from the draft
# as it was saved, and this would change it under the send.
debugsession.lock(db, session)
if debugsession.sending(session):
raise refused(409, "issue_sending")
if session.issue_url is not None:
# Sent: the issue is on the tracker now, and a draft edited after it would say otherwise.
raise refused(409, "issue_already_sent")
debugsession.save_draft(session, body.model_dump())
db.commit()
return {"ok": True, "session": _view(db, session)}
@router.get("/{sid}/export")
def export_session(
sid: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> Response:
"""The issue as a zip — the text, the two logs and the screenshots — for an instance without an
issue tracker, or for sending it by hand. Built from the SAVED draft: what the composer shows is
what it saved."""
session = owned_or_404(db, DebugSession, sid, user, what="debug_session")
if not _exports.allow(str(user.id)):
raise refused(429, "export_rate_limited")
data = bundle.zipped(bundle.files(db, session))
name = f"siftlode-debug-{session.started_at:%Y%m%d-%H%M}-{session.id[:6]}.zip"
return Response(
content=data,
media_type="application/zip",
headers={
"Content-Disposition": f'attachment; filename="{name}"',
"Cache-Control": "no-store",
},
)
# --- the issue tracker (S4) -----------------------------------------------------------------
# Issues a user may open per hour — counted once the tracker is ASKED to open one (an answer that
# goes missing may still have created it, so an attempt cannot be given back); finishing a half-sent
# one, or an attempt refused before the tracker was asked, is not counted. The composer sends one at
# a time; this is room for a bad afternoon, not a pace anything legitimate keeps up.
_new_issues = RateLimiter(max_events=5, window_seconds=3600)
# Attempts on the tracker per hour, creating or finishing. Deliberately NOT the budget above: a
# half-sent issue must always be finishable, so the pace that guards creating one must never be what
# stands between a user and their own report. This one only bounds a loop — every attempt re-reads
# the whole bundle into memory and re-offers the pending file to the tracker, and a retry that is
# refused for good (the tracker's own limits) never leaves `pending`. No hand reaches it.
_sends = RateLimiter(max_events=60, window_seconds=3600)
# The composer's live preview, rendered BY the tracker: one authenticated call each, with our token.
# The composer's toggle fires it once per switch INTO preview — the editor is the other branch, so
# no keystroke can change the query key while the preview is up — and react-query's cache makes an
# unchanged draft free. (It does NOT debounce, whatever this comment used to say: a key that changes
# per keystroke is one fetch per keystroke, measured. Showing the preview BESIDE the fields would
# need a real debounce first.) This bounds what a loop can aim at the tracker.
_previews = RateLimiter(max_events=60, window_seconds=60)
def _issues_rate_limited() -> HTTPException:
return refused(429, "issues_rate_limited", limit=_new_issues.max_events)
@router.post("/{sid}/issue")
def send_issue(
sid: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
"""Send the session's issue to the tracker — from the SAVED draft, as the export is built — or
finish a send that stopped half-way (the issue exists; only what is missing is sent)."""
session = owned_or_404(db, DebugSession, sid, user, what="debug_session")
# The session is CLAIMED for the send rather than locked through it: the tracker is slow (a
# dozen calls, 20 s each), and a row lock held across them would queue this user's Stop, their
# autosave and their screenshots behind it. They see the claim instead and are refused at once.
debugsession.lock(db, session)
if debugsession.sending(session):
raise refused(409, "issue_sending")
new = session.issue_url is None
if not new and not debugsession.unfinished(session):
return {"ok": True, "session": _view(db, session)}
key = str(user.id)
if new:
if not ((session.draft or {}).get("title") or "").strip():
raise refused(422, "issue_title_required")
if _new_issues.exhausted(key):
raise _issues_rate_limited()
if not _sends.allow(key):
raise refused(429, "issue_rate_limited", limit=_sends.max_events)
token = debugsession.claim_send(session)
db.commit() # the claim is what everything else sees; the lock ends here
def count_new_issue() -> None:
if not _new_issues.allow(key):
raise _issues_rate_limited()
try:
gitea.deliver(
db,
session,
token=token,
checkpoint=db.commit,
before_create=count_new_issue,
on_created=lambda: _record_created(db, user, session),
)
except debugsession.SendClaimLost:
# This send outlived its lease and another one took the session over while it was still
# talking to the tracker. It stopped rather than writing over that one's progress — and a
# send IS under way, just not ours, which is what the caller needs to hear.
raise refused(409, "issue_sending") from None
except gitea.GiteaError as exc:
# Keep what did happen: an issue created before an upload failed stays linked, and the
# retry does only what is still missing.
raise issue_refused(exc) from None
except HTTPException:
raise # the pace, refused before the tracker was asked
finally:
_end_send(db, session, token)
return {"ok": True, "session": _view(db, session)}
def _end_send(db: Session, session: DebugSession, token: str) -> None:
"""Give the claim back, whatever else happened. A failure here must not replace the failure that
brought us: a dead transaction (the database went away mid-send) is rolled back, and the claim
then simply ages out."""
try:
debugsession.lock(db, session)
debugsession.release_send(session, token)
db.commit()
except Exception: # noqa: BLE001 — whatever went wrong here, the send's own failure is the news
db.rollback()
log.warning(
"a send could not give its claim back",
extra=logs.event("issues.claim.stuck", session_id=session.id),
)
def _record_created(db: Session, user: User, session: DebugSession) -> None:
"""The trail for the one act that takes a user's log off this instance. Staged inside the send,
so the checkpoint that first writes `issue_url` commits BOTH — recorded later it could be lost
on its own (the process dies mid-send), and no retry could write it afterwards: the next send
sees the link and knows the issue is not new."""
audit.record(
db,
user.id,
AuditAction.ISSUE_CREATE,
target_type="debug_session",
target_id=session.id,
summary=session.issue_url or "",
)
class PreviewIn(BaseModel):
text: str = Field(max_length=50_000)
@router.post("/{sid}/preview")
def preview_issue(
sid: str,
body: PreviewIn,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""The body as the tracker will render it. Gitea sanitises its own output; the composer still
shows it in a sandboxed frame, never in the app's own document."""
owned_or_404(db, DebugSession, sid, user, what="debug_session")
if not _previews.allow(str(user.id)):
raise refused(429, "preview_rate_limited")
try:
return {"ok": True, "html": gitea.preview(db, body.text)}
except gitea.GiteaError as exc:
raise issue_refused(exc) from None
# --- screenshots ---------------------------------------------------------------------------
def _shot_view(shot: DebugScreenshot) -> dict:
return {"id": shot.id, "mime": shot.mime, "size": shot.size, "created_at": iso(shot.created_at)}
@router.get("/{sid}/screenshots")
def list_screenshots(
sid: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
session = owned_or_404(db, DebugSession, sid, user, what="debug_session")
return {"items": [_shot_view(s) for s in debugsession.screenshots(db, session)]}
def _upload_target(
sid: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> DebugSession:
return owned_or_404(db, DebugSession, sid, user, what="debug_session")
async def _image_body(request: Request, db: Session = Depends(get_db)) -> bytes:
# The owner check above has already run, and it was a SELECT — so this session is holding a
# pooled connection `idle in transaction` and would hold it for the whole 10 MiB upload, from a
# client that may be on a phone. Hand it back first; the handler's own work re-takes one. The
# `session` object the handler holds is expired by this and reloads on its next attribute read,
# which is the correct moment to touch the database again.
release_read_connection(db)
return await capped_body(request, debugsession.SCREENSHOT_MAX_BYTES)
@router.post("/{sid}/screenshots")
def add_screenshot(
# The owner check FIRST: dependencies resolve in order, so a stranger's upload is refused
# before a byte of it is read.
session: DebugSession = Depends(_upload_target),
data: bytes = Depends(_image_body),
db: Session = Depends(get_db),
) -> dict:
try:
shot = debugsession.add_screenshot(db, session, data)
except debugsession.ScreenshotRefused as exc:
raise _screenshot_refused(exc.code) from None
db.commit()
return {"ok": True, "screenshot": _shot_view(shot)}
def _screenshot_refused(code: str) -> HTTPException:
# Literal reasons, one per case: the frontend's contract test reads them to require a sentence
# for each in both languages.
if code == "already_sent":
return refused(409, "screenshot_already_sent")
if code == "too_many":
return refused(422, "screenshot_too_many", limit=debugsession.SCREENSHOTS_MAX)
if code == "sending":
return refused(409, "issue_sending")
if code == "storage_full":
mb = debugsession.SCREENSHOT_USER_MAX_BYTES // (1024 * 1024)
return refused(422, "screenshot_storage_full", limit=mb)
if code == "empty":
return refused(422, "screenshot_empty")
return refused(422, "screenshot_not_an_image")
def _session_screenshot(db: Session, sid: str, shot_id: int, user: User) -> DebugScreenshot:
return _screenshot_of(
db, owned_or_404(db, DebugSession, sid, user, what="debug_session"), shot_id
)
def _screenshot_of(db: Session, session: DebugSession, shot_id: int) -> DebugScreenshot:
shot = db.get(DebugScreenshot, shot_id)
if shot is None or shot.session_id != session.id:
raise not_found("screenshot")
return shot
@router.get("/{sid}/screenshots/{shot_id}")
def get_screenshot(
sid: str, shot_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> Response:
shot = _session_screenshot(db, sid, shot_id, user)
# `nosniff`: the type was checked against the bytes at upload, and the browser must not guess.
return Response(
content=shot.data,
media_type=shot.mime,
headers={"Cache-Control": "private, max-age=3600", "X-Content-Type-Options": "nosniff"},
)
@router.delete("/{sid}/screenshots/{shot_id}")
def delete_screenshot(
sid: str, shot_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
session = owned_or_404(db, DebugSession, sid, user, what="debug_session")
shot = _screenshot_of(db, session, shot_id)
try:
debugsession.remove_screenshot(db, session, shot)
except debugsession.ScreenshotRefused as exc:
raise _screenshot_refused(exc.code) from None
db.commit()
return {"ok": True, "deleted": shot_id}
+32 -21
View File
@@ -13,7 +13,9 @@ Nothing from the body is trusted as-is:
* everything is redacted AGAIN with the server's own rules — the client redacts at capture, but a
modified client would not;
* an event below the consented level is dropped (a Részletes consent does not store what only
Teljes may carry), and so is the whole batch once the level has lapsed;
Teljes may carry), and so is the whole batch once the level has lapsed — where an open debug
session the request names counts as consent to ITS level (E16), and only its own events keep
its tag;
* the timestamp is clamped to the consent window, so a row cannot dodge the retention job by
claiming to come from the future.
"""
@@ -27,11 +29,12 @@ from pydantic import BaseModel, Field, ValidationError
from sqlalchemy import insert
from sqlalchemy.orm import Session
from app import diagnostics, logs, logsink
from app import debugsession, diagnostics, logs, logsink
from app.auth import current_user
from app.db import get_db
from app.models import LogEvent, User
from app.db import get_db, release_read_connection
from app.models import DebugSession, LogEvent, User
from app.ratelimit import RateLimiter
from app.routes._common import capped_body
router = APIRouter(prefix="/api/diagnostics", tags=["diagnostics"])
@@ -60,6 +63,9 @@ class ClientEvent(BaseModel):
attrs: dict[str, Any] | None = None
request_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$")
tab_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9_-]{1,32}$")
# The debug session the event was captured in (E16); kept only when it is the one this request
# names in `X-Debug-Session` and the auth dependency found it the caller's own and open.
debug_session_id: str | None = Field(default=None, pattern=r"^[0-9a-f]{16}$")
version: str | None = Field(default=None, max_length=64)
@@ -67,19 +73,7 @@ class ClientEventsIn(BaseModel):
events: list[ClientEvent] = Field(max_length=EVENTS_MAX)
async def _capped_body(request: Request) -> bytes:
"""The request body, refused the moment it passes BODY_MAX_BYTES while still arriving. Counted,
not taken from Content-Length: a declared length can be absent (chunked) or wrong, and counting
stops reading at the same point a declared-length check would have."""
body = bytearray()
async for chunk in request.stream():
body += chunk
if len(body) > BODY_MAX_BYTES:
raise HTTPException(status_code=413, detail={"code": "too_large"})
return bytes(body)
async def _parsed_events(request: Request) -> ClientEventsIn:
async def _parsed_events(request: Request, db: Session = Depends(get_db)) -> ClientEventsIn:
"""The body as the model — parsed only after the size cap, which is why this is a dependency and
not a plain body parameter (FastAPI reads a declared body in full before anything else runs).
@@ -89,14 +83,24 @@ async def _parsed_events(request: Request) -> ClientEventsIn:
media_type = request.headers.get("content-type", "").split(";")[0].strip().lower()
if media_type != "application/json" and not media_type.endswith("+json"):
raise HTTPException(status_code=415, detail={"code": "json_required"})
raw = await _capped_body(request)
# `current_user` resolved above on a SELECT, so this session is holding a pooled connection.
# Give it back before awaiting the body — the sink is a background beacon and its sender may be
# on a link far slower than the 512 KiB suggests. See `db.release_read_connection`.
release_read_connection(db)
raw = await capped_body(request, BODY_MAX_BYTES)
try:
return ClientEventsIn.model_validate_json(raw)
except ValidationError as exc:
raise HTTPException(status_code=422, detail=json.loads(exc.json())) from exc
def _row(item: ClientEvent, user: User, now: datetime, window_start: datetime) -> dict[str, Any]:
def _row(
item: ClientEvent,
user: User,
now: datetime,
window_start: datetime,
session: DebugSession | None,
) -> dict[str, Any]:
attrs = logs.redact(item.attrs) if item.attrs else None
if attrs is not None:
size = len(json.dumps(attrs, ensure_ascii=False, default=str).encode())
@@ -117,6 +121,9 @@ def _row(item: ClientEvent, user: User, now: datetime, window_start: datetime) -
"attrs": attrs,
"request_id": item.request_id,
"user_id": user.id,
"debug_session_id": (
session.id if session and item.debug_session_id == session.id else None
),
"version": item.version,
}
)
@@ -126,11 +133,15 @@ def _row(item: ClientEvent, user: User, now: datetime, window_start: datetime) -
def ingest_events(
# The account first: dependencies resolve in order, so an anonymous caller is refused before a
# byte of its body is read.
request: Request,
user: User = Depends(current_user),
body: ClientEventsIn = Depends(_parsed_events),
db: Session = Depends(get_db),
) -> dict:
level = diagnostics.effective_level(user)
# A debug session the request names raises the level for its own events (E16): starting one is
# the consent to that level for its duration, whatever the user's own level is.
session = debugsession.from_header(db, user, request.headers.get(debugsession.HEADER))
level, _ = debugsession.level_for(user, session)
if level == diagnostics.BASE:
# Lapsed or withdrawn: the client stops streaming when it sees this.
raise HTTPException(status_code=409, detail={"code": "diagnostics_off"})
@@ -140,7 +151,7 @@ def ingest_events(
window_start = now - timedelta(hours=diagnostics.ELEVATION_HOURS)
floor = logs.SEVERITY[level]
rows = [
_row(item, user, now, window_start)
_row(item, user, now, window_start, session)
for item in body.events
if logs.SEVERITY[item.level] >= floor
]
+12 -17
View File
@@ -11,9 +11,9 @@ thing it exists to remove.
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import select, update
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -23,7 +23,7 @@ from app.discover import hero as hero_mod
from app.discover import pins, rails
from app.discover.budget import Budget, DEFAULT_BUDGET
from app.models import ComfortPin, User
from app.routes._common import owned_or_404
from app.routes._common import owned_or_404, not_found
router = APIRouter(prefix="/api/discover", tags=["discover"])
@@ -53,7 +53,9 @@ def get_discover(
`resume` and `shelf` are built once and handed to the hero, so the two rails the hero draws from
are not queried twice for a single page load."""
resume = rails.continue_watching(db, user)
shelf = pins.shelf(db, user)
# Plex off → the shelf rail (and so the hero's comfort tier) goes YouTube-only like the rails
# beside it; the pins are kept and `GET /pins` still lists them (`pins.shelf`).
shelf = pins.shelf(db, user, include_plex=rails.plex_available(db))
return {
"budget": budget,
"hero": hero_mod.pick(db, user, budget, seed, resume=resume, shelf=shelf),
@@ -125,7 +127,7 @@ def add_pin(
constraint is the mechanism, not the message."""
resolved = pins.resolve_ref(db, user, body.kind, body.ref)
if resolved is None:
raise HTTPException(status_code=404, detail="Unknown item")
raise not_found("item")
column, value = resolved
existing = db.execute(
@@ -157,7 +159,7 @@ def add_pin(
# between `resolve_ref` confirming it and this INSERT (a Plex sync dropping a removed
# item), which trips the FK. Semantically that is the same "no such thing" the pre-check
# returns above — re-raising handed the user a 500 they could do nothing with.
raise HTTPException(status_code=404, detail="Unknown item")
raise not_found("item")
return {"ok": True, "id": winner.id, "created": False}
return {"ok": True, "id": pin.id, "created": True}
@@ -180,18 +182,11 @@ def reorder_pins(
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Rewrite the shelf order from a full list of pin ids.
"""Rewrite the shelf order from the list of pin ids the client shows.
Whole-list rather than a move-one delta: the shelf is short, and a full rewrite cannot leave two
tiles claiming the same slot the way an interrupted sequence of single moves can. Ids that are not
this user's are ignored rather than 404'd — a stale tab replaying an order that includes a pin
deleted meanwhile should reorder what still exists, not fail wholesale."""
owned = {p.id for p in pins.list_pins(db, user)}
for position, pin_id in enumerate(i for i in body.ids if i in owned):
db.execute(
update(ComfortPin)
.where(ComfortPin.id == pin_id, ComfortPin.user_id == user.id)
.values(position=position)
)
tiles claiming the same slot the way an interrupted sequence of single moves can. The list may be
PARTIAL (hidden Plex pins, a stale tab) — `pins.reorder` keeps what it leaves out in place."""
pins.reorder(db, user, body.ids)
db.commit()
return {"ok": True}
+165 -93
View File
@@ -9,11 +9,13 @@ serves finished files (range-aware, with the user's custom display name), and sh
import os
import re
import tempfile
from collections.abc import AsyncIterator
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from pathlib import Path
from urllib.parse import parse_qs, urlparse
from urllib.parse import parse_qs, unquote, urlparse
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import func, select
@@ -22,7 +24,7 @@ from sqlalchemy.orm import Session
from app import audit, sysconfig
from app.auth import admin_user, require_human
from app.config import settings
from app.db import get_db
from app.db import get_db, release_read_connection
from app.downloads import credentials as dlcreds
from app.downloads import edit as editmod
from app.downloads import links as linksmod
@@ -37,11 +39,13 @@ from app.models import (
DownloadShare,
MediaAsset,
User,
display_text,
)
from app.routes._common import get_or_404, not_found
from app.routes._common import capped_stream_to, get_or_404, not_found, owned_or_404
from app.security import hash_password
from app.userscope import is_messageable_user, messageable_clauses
from app.utils import iso
from app.refusal import refused
router = APIRouter(prefix="/api/downloads", tags=["downloads"])
admin_router = APIRouter(prefix="/api/admin/downloads", tags=["admin-downloads"])
@@ -63,7 +67,7 @@ def resolve_source(raw: str) -> tuple[str, str]:
as a raw URL for yt-dlp's generic extractor."""
raw = (raw or "").strip()
if not raw:
raise HTTPException(status_code=400, detail="A video link or id is required.")
raise refused(400, "link_required")
if _VIDEO_ID.match(raw):
return "youtube", raw
if raw.startswith(("http://", "https://")):
@@ -93,7 +97,7 @@ def resolve_source(raw: str) -> tuple[str, str]:
if _VIDEO_ID.match(vid):
return "youtube", vid
return "url", raw
raise HTTPException(status_code=400, detail="That doesn't look like a valid video link.")
raise refused(400, "invalid_link")
def _serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
@@ -166,15 +170,8 @@ def _serialize_job(db: Session, job: DownloadJob) -> dict:
return _serialize(job, asset)
def _own_job(db: Session, user: User, job_id: int) -> DownloadJob:
job = db.get(DownloadJob, job_id)
if job is None or job.user_id != user.id:
raise HTTPException(status_code=404, detail="Unknown download")
return job
def _own_job_for_update(db: Session, user: User, job_id: int) -> DownloadJob:
"""`_own_job` for a transition that moves its asset's ref count (resume, cancel, delete, redownload)
"""The ownership check (`owned_or_404`) for a transition that moves its asset's ref count (resume, cancel, delete, redownload)
or writes the status those decide on (pause). The job is re-read under a row lock, so two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
@@ -183,7 +180,7 @@ def _own_job_for_update(db: Session, user: User, job_id: int) -> DownloadJob:
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`service.finalize_if_running`), so the two can never deadlock."""
while True:
asset_id = _own_job(db, user, job_id).asset_id
asset_id = owned_or_404(db, DownloadJob, job_id, user, what="download").asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
@@ -193,7 +190,7 @@ def _own_job_for_update(db: Session, user: User, job_id: int) -> DownloadJob:
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
raise HTTPException(status_code=404, detail="Unknown download")
raise not_found("download")
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
@@ -240,7 +237,7 @@ def create_profile(
name = body.name.strip()
spec = body.spec
if not name:
raise HTTPException(status_code=400, detail="A profile name is required.")
raise refused(400, "profile_name_required")
maxpos = db.execute(
select(func.coalesce(func.max(DownloadProfile.sort_order), 100)).where(
DownloadProfile.user_id == user.id
@@ -268,16 +265,16 @@ def update_profile(
) -> dict:
p = db.get(DownloadProfile, profile_id)
if p is None or p.is_builtin or p.user_id != user.id:
raise HTTPException(status_code=404, detail="Unknown profile")
raise not_found("profile")
sent = body.model_fields_set
if "name" in sent:
name = (body.name or "").strip()
if not name:
raise HTTPException(status_code=400, detail="A profile name is required.")
raise refused(400, "profile_name_required")
p.name = name[:80]
if "spec" in sent:
if body.spec is None:
raise HTTPException(status_code=400, detail="spec must be an object")
raise refused(400, "spec_not_object")
p.spec = body.spec
db.commit()
return _serialize_profile(p)
@@ -289,7 +286,7 @@ def delete_profile(
) -> dict:
p = db.get(DownloadProfile, profile_id)
if p is None or p.is_builtin or p.user_id != user.id:
raise HTTPException(status_code=404, detail="Unknown profile")
raise not_found("profile")
db.delete(p)
db.commit()
return {"deleted": profile_id}
@@ -319,7 +316,7 @@ def set_credential(
"""Store (encrypted) this user's cookies.txt for one supported host. Write-only: the value is
never echoed back, logged, or audited — only the fact that it changed."""
if not dlcreds.is_supported_host(host_key):
raise HTTPException(status_code=404, detail="Unknown host")
raise not_found("host")
try:
cleaned = dlcreds.validate_cookies(body.cookies)
except dlcreds.InvalidCookies as e:
@@ -341,7 +338,7 @@ def delete_credential(
host_key: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
if not dlcreds.is_supported_host(host_key):
raise HTTPException(status_code=404, detail="Unknown host")
raise not_found("host")
if dlcreds.delete_credential(db, user.id, host_key):
audit.record(
db,
@@ -362,7 +359,7 @@ def _resolve_spec(
if profile_id is not None:
p = db.get(DownloadProfile, profile_id)
if p is None or (not p.is_builtin and p.user_id != user.id):
raise HTTPException(status_code=404, detail="Unknown profile")
raise not_found("profile")
return p.spec, p.id
if spec is not None:
return spec, None
@@ -373,7 +370,7 @@ def _resolve_spec(
.limit(1)
).scalar_one_or_none()
if fallback is None:
raise HTTPException(status_code=400, detail="No download profile available.")
raise refused(400, "no_profile")
return fallback.spec, fallback.id
@@ -404,7 +401,7 @@ def enqueue_download(
status_code=422, detail={"code": "url_blocked", "reason": e.reason}
) from e
spec, profile_id = _resolve_spec(db, user, body.profile_id, body.spec)
display_name = (body.display_name or "").strip() or None
display_name = display_text(body.display_name)
try:
job = service.enqueue(db, user.id, source_kind, source_ref, spec, profile_id, display_name)
except quota.QuotaExceeded as e:
@@ -412,61 +409,119 @@ def enqueue_download(
return _serialize_job(db, job)
# Read the upload in 1 MiB chunks so a large file never sits fully in memory, and the size cap is
# enforced on the ACTUAL bytes read (a client-set Content-Length can't be trusted to gate it).
_UPLOAD_CHUNK = 1024 * 1024
# The upload's filename and display name travel as HEADERS, not as multipart parts. That is not a
# style choice: a DECLARED body parameter (`file: UploadFile = File(...)`) is read and spooled by
# FastAPI *before the first dependency runs*, so `require_human` could not gate it and an anonymous
# caller could make the process write the whole per-file allowance to temp, per request (P1; SEC-20
# bounded the number but could not stop the writing). Reading the body ourselves, from a dependency
# declared AFTER the auth one, is the only shape that closes it — the same one
# `POST /api/debug-sessions/{sid}/screenshots` uses, and `test_body_limit.py` now enforces it for
# every route with a raised ceiling rather than leaving it to a comment like this one.
#
# Headers are latin-1 on the wire, so both are percent-encoded by the client and unquoted here; a
# Hungarian filename would otherwise be unsendable.
UPLOAD_FILENAME_HEADER = "x-upload-filename"
UPLOAD_DISPLAY_NAME_HEADER = "x-upload-display-name"
@router.post("/upload")
def upload_download(
file: UploadFile = File(...),
display_name: str | None = Form(None),
@dataclass(frozen=True)
class SpooledUpload:
"""What the body dependency hands the handler: the bytes already on disk, plus the two names
that came alongside them."""
path: Path
size: int
user_id: int
filename: str
display_name: str | None
def _header_text(request: Request, name: str) -> str:
"""One percent-encoded header, decoded. A header that is not valid UTF-8 once unquoted is the
caller's doing, not a server fault — it degrades to empty rather than 500ing, and the route's
own fallback names the file."""
raw = request.headers.get(name, "")
try:
return unquote(raw, errors="strict").strip()
except (UnicodeDecodeError, ValueError):
# silent-ok: a mis-encoded header is the caller's own malformed request, and this one is
# cosmetic — it names the file. Refusing the upload over it would fail a 500 MiB transfer on
# a label; logging it would hand any client a free log line per request.
return ""
async def spool_upload(
request: Request,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Ingest a local file into the caller's library (E7 S5). Any file type is accepted; media
(video/audio) gets the full function set, a generic file is download-only. The file is streamed
to a temp path under the download volume (enforcing the per-file cap as it goes), then handed to
the non-yt-dlp ingest which probes + places it and returns an already-`done` job.
) -> AsyncIterator[SpooledUpload]:
"""Stream the request body to a staging temp file, enforcing the per-file cap as it arrives.
A sync (threadpool) handler on purpose: the ingest does blocking work — an ffprobe of up to 30s
plus sync DB writes — so it must NOT run on the event loop. `file.file` is the sync file object
Starlette spools the multipart body into."""
`user` is taken but unused: it is the whole point. Dependencies resolve in declaration order, so
naming `require_human` here makes the auth check a precondition of reading the body at all — a
stranger is refused before a byte moves. The `yield` shape owns the temp file's life: ingest
MOVES it into place on success (the unlink is then a no-op), and on any other exit — cap hit,
empty, quota, probe or move error, a client that vanished mid-upload — it is removed."""
cap = sysconfig.get_int(db, "download_upload_max_bytes")
user_id = user.id
# Hand the pooled connection back BEFORE the transfer starts — see `db.release_read_connection`
# for why, and for the precondition it enforces. Everything this dependency still needs is
# already an int or a header in hand: touching an ORM attribute below would reload `user` and
# check a connection straight back out.
release_read_connection(db)
staging = Path(settings.download_root) / ".staging" / "uploads"
staging.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(staging))
tmp_path = Path(tmp)
try:
size = 0
with os.fdopen(fd, "wb") as out:
while chunk := file.file.read(_UPLOAD_CHUNK):
size += len(chunk)
if cap and size > cap:
raise HTTPException(
status_code=422, detail={"code": "upload_too_large", "limit": cap}
)
out.write(chunk)
if size == 0:
raise HTTPException(status_code=400, detail={"code": "upload_empty"})
try:
job = uploadmod.ingest_upload(
db,
user.id,
filename=file.filename or "upload",
src_path=tmp_path,
size=size,
display_name=(display_name or "").strip() or None,
size = await capped_stream_to(
request,
out,
cap,
HTTPException(status_code=422, detail={"code": "upload_too_large", "limit": cap}),
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
return _serialize_job(db, job)
yield SpooledUpload(
path=tmp_path,
size=size,
user_id=user_id,
filename=_header_text(request, UPLOAD_FILENAME_HEADER) or "upload",
display_name=display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER)),
)
finally:
# Ingest MOVES the temp file into place on success (unlink is then a no-op); on any early exit
# — cap hit, empty, quota, probe/move error — this removes the abandoned staging file.
tmp_path.unlink(missing_ok=True)
@router.post("/upload")
def upload_download(
upload: SpooledUpload = Depends(spool_upload),
db: Session = Depends(get_db),
) -> dict:
"""Ingest a local file into the caller's library (E7 S5). Any file type is accepted; media
(video/audio) gets the full function set, a generic file is download-only. The body IS the file
(see the header note above); `spool_upload` has already authenticated the caller and streamed it
to a temp path under the download volume, so what is left here is the non-yt-dlp ingest which
probes + places it and returns an already-`done` job.
A sync (threadpool) handler on purpose: the ingest does blocking work — an ffprobe of up to 30s
plus sync DB writes — so it must NOT run on the event loop. The streaming half is async because
reading the body is; keeping them in separate callables is what lets each run where it belongs."""
if upload.size == 0:
raise HTTPException(status_code=400, detail={"code": "upload_empty"})
try:
job = uploadmod.ingest_upload(
db,
upload.user_id,
filename=upload.filename,
src_path=upload.path,
size=upload.size,
display_name=upload.display_name,
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
return _serialize_job(db, job)
class EnqueueEditIn(BaseModel):
source_job_id: int = 0
edit_spec: dict
@@ -484,9 +539,9 @@ def enqueue_edit(
source file is only read, never modified, so editing a shared video is safe."""
source_job = _accessible_job(db, user, body.source_job_id)
if source_job.status != "done" or source_job.asset_id is None:
raise HTTPException(status_code=409, detail="You can only edit a finished download.")
raise refused(409, "edit_unfinished")
spec = body.edit_spec
display_name = (body.display_name or "").strip() or None
display_name = display_text(body.display_name)
try:
job = service.enqueue_edit(db, user.id, source_job, spec, display_name)
except quota.QuotaExceeded as e:
@@ -522,7 +577,20 @@ def list_downloads(
@router.get("/usage")
def my_usage(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
return quota.usage(db, user.id)
"""The caller's download quota, plus the per-file upload cap.
The cap rides here rather than in `quota.usage` because it is not a quota value — it is a
`sysconfig` knob about one route — and this is the route layer, where composing the two is the
job. The upload dialog needs it to refuse an over-cap file BEFORE sending it: both server-side
caps answer too late to be pleasant (the edge ceiling refuses the request outright, see
`bodylimit.ROUTE_MAX_BODY_BYTES`), and making the user push 600 MB up the wire to be told the
limit is 500 is a bad answer even when it is a correct one. It is a convenience, not the
boundary: the caps stay where they are and still refuse a client that ignores this.
"""
return {
**quota.usage(db, user.id),
"upload_max_bytes": sysconfig.get_int(db, "download_upload_max_bytes"),
}
# Priority when a source has several jobs (e.g. a done one plus a fresh queued one).
@@ -587,6 +655,8 @@ def _clean_url(raw) -> str | None:
# authority. This one already has a "not a plausible URL" answer, so the typo belongs in it.
try:
parsed = urlparse(url)
# silent-ok: a typo in a link someone pasted. None is this function's "not a plausible URL"
# answer, the caller already has one, and any line here would carry the URL itself.
except ValueError:
return None
if not parsed.netloc:
@@ -612,12 +682,12 @@ def update_download_meta(
) -> dict:
"""Update a download's display metadata: title, channel name + link, and extra reference URLs.
All are display-only overrides — the on-disk file keeps its system-decided, id-bound name."""
job = _own_job(db, user, job_id)
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
sent = body.model_fields_set
if "display_name" in sent:
job.display_name = ((body.display_name or "").strip())[:255] or None
job.display_name = display_text(body.display_name)
if "display_uploader" in sent:
job.display_uploader = ((body.display_uploader or "").strip())[:255] or None
job.display_uploader = display_text(body.display_uploader)
if "display_uploader_url" in sent:
job.display_uploader_url = _clean_url(body.display_uploader_url)
if "extra_links" in sent:
@@ -666,9 +736,9 @@ def redownload_download(
if job.job_kind in ("edit", "upload"):
# Neither has a re-fetchable source: an editor clip is derived locally, an upload IS the
# source. Re-running yt-dlp against source_kind='upload'/'edit' would just fail.
raise HTTPException(status_code=409, detail="This item can't be re-downloaded.")
raise refused(409, "not_redownloadable")
if job.status in ("queued", "running", "paused"):
raise HTTPException(status_code=409, detail="This download is still in progress.")
raise refused(409, "still_downloading")
service.redownload(db, job)
return _serialize_job(db, job)
@@ -697,7 +767,7 @@ def delete_download(
def _accessible_job(db: Session, user: User, job_id: int) -> DownloadJob:
job = db.get(DownloadJob, job_id)
if job is None:
raise HTTPException(status_code=404, detail="Unknown download")
raise not_found("download")
if job.user_id == user.id:
return job
shared = db.execute(
@@ -706,7 +776,7 @@ def _accessible_job(db: Session, user: User, job_id: int) -> DownloadJob:
)
).first()
if shared is None:
raise HTTPException(status_code=404, detail="Unknown download")
raise not_found("download")
return job
@@ -720,10 +790,10 @@ def download_file(
job = _accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or asset.status != "ready" or not asset.rel_path:
raise HTTPException(status_code=409, detail="This download isn't ready.")
raise refused(409, "download_not_ready")
path = storage.safe_abs_path(settings.download_root, asset.rel_path)
if path is None:
raise HTTPException(status_code=404, detail="File is no longer available.")
raise not_found("file")
filename = storage.download_filename(
job.display_name or asset.title or asset.source_ref, asset.container, path
@@ -762,7 +832,7 @@ def storyboard_meta(
job = _accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or asset.status != "ready" or not asset.rel_path:
raise HTTPException(status_code=409, detail="This download isn't ready.")
raise refused(409, "download_not_ready")
meta = editmod.ensure_storyboard(asset, settings.download_root)
if meta is None:
return {"available": False}
@@ -783,10 +853,10 @@ def poster_image(job_id: int, user: User = Depends(require_human), db: Session =
job = _accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or not asset.poster_path:
raise HTTPException(status_code=404, detail="No poster.")
raise not_found("poster")
path = storage.safe_abs_path(settings.download_root, asset.poster_path)
if path is None:
raise HTTPException(status_code=404, detail="No poster.")
raise not_found("poster")
return FileResponse(path, media_type="image/jpeg")
@@ -797,10 +867,10 @@ def storyboard_image(
job = _accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
if asset is None or not asset.storyboard_path:
raise HTTPException(status_code=404, detail="No filmstrip.")
raise not_found("filmstrip")
path = storage.safe_abs_path(settings.download_root, asset.storyboard_path)
if path is None:
raise HTTPException(status_code=404, detail="No filmstrip.")
raise not_found("filmstrip")
return FileResponse(path, media_type="image/jpeg")
@@ -818,12 +888,12 @@ def share_download(
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
job = _own_job(db, user, job_id)
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
if job.status != "done" or job.asset_id is None:
raise HTTPException(status_code=409, detail="Only a finished download can be shared.")
raise refused(409, "share_unfinished")
target = body.email.strip().lower()
if not target:
raise HTTPException(status_code=400, detail="A recipient email is required.")
raise refused(400, "recipient_required")
recipient = db.execute(
select(User).where(func.lower(User.email) == target)
).scalar_one_or_none()
@@ -832,7 +902,7 @@ def share_download(
if not is_messageable_user(recipient):
raise not_found("recipient")
if recipient.id == user.id:
raise HTTPException(status_code=400, detail="That's already your download.")
raise refused(400, "share_own")
exists = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == recipient.id
@@ -851,7 +921,7 @@ def share_download(
commit=False,
)
db.commit()
return {"shared_with": recipient.email}
return {"ok": True, "shared_with": recipient.email}
@router.delete("/{job_id}/share/{email}")
@@ -861,7 +931,9 @@ def unshare_download(
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
_own_job(db, user, job_id) # ownership guard (404s if not the caller's job)
owned_or_404(
db, DownloadJob, job_id, user, what="download"
) # ownership guard (404s if not the caller's job)
recipient = db.execute(
select(User).where(func.lower(User.email) == email.strip().lower())
).scalar_one_or_none()
@@ -893,7 +965,7 @@ def remove_shared_with_me(
if row is not None:
db.delete(row)
db.commit()
return {"removed": job_id}
return {"ok": True, "removed": job_id}
# --- recipients (for the internal "share with a user" picker) ------------------------------
@@ -923,10 +995,10 @@ def share_recipients(
def _own_link(db: Session, user: User, link_id: int) -> DownloadLink:
link = db.get(DownloadLink, link_id)
if link is None:
raise HTTPException(status_code=404, detail="Unknown link")
raise not_found("link")
job = db.get(DownloadJob, link.job_id)
if job is None or job.user_id != user.id:
raise HTTPException(status_code=404, detail="Unknown link")
raise not_found("link")
return link
@@ -942,7 +1014,7 @@ def _link_expiry(days: int | None) -> datetime | None:
def list_links(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> list[dict]:
_own_job(db, user, job_id)
owned_or_404(db, DownloadJob, job_id, user, what="download")
rows = (
db.execute(
select(DownloadLink)
@@ -973,9 +1045,9 @@ def create_link(
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
job = _own_job(db, user, job_id)
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
if job.status != "done" or job.asset_id is None:
raise HTTPException(status_code=409, detail="Only a finished download can be shared.")
raise refused(409, "share_unfinished")
pw = body.password.strip()
link = DownloadLink(
job_id=job_id,
+17 -12
View File
@@ -1,6 +1,6 @@
from datetime import date, datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, Query
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import and_, func, or_, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
@@ -29,10 +29,11 @@ from app.models import (
Video,
VideoState,
)
from app.routes._common import get_or_404
from app.routes._common import get_or_404, youtube_refusal, not_found
from app.sync.videos import parse_iso8601_duration
from app.utils import iso
from app.youtube.client import YouTubeClient, YouTubeError
from app.refusal import refused
router = APIRouter(prefix="/api", tags=["feed"])
@@ -120,7 +121,7 @@ def get_feed(
vals, cursor_id = decode_cursor(keys, cursor)
except ValueError:
# `from None`: the ValueError is an expected bad-input signal, not an error to chain.
raise HTTPException(status_code=400, detail="Invalid feed cursor") from None
raise refused(400, "invalid_cursor") from None
for k, v in zip(keys, vals):
k["value"] = v
query = query.where(keyset_predicate(keys, cursor_id))
@@ -235,7 +236,7 @@ def set_video_state(
) -> dict:
status = body.status
if status not in VALID_STATES:
raise HTTPException(status_code=400, detail=f"status must be one of {VALID_STATES}")
raise refused(400, "invalid_status")
get_or_404(db, Video, video_id, what="video")
now = datetime.now(timezone.utc)
@@ -256,9 +257,11 @@ def set_video_state(
db.delete(row)
try:
db.commit()
# silent-ok: the documented lost race — a concurrent player checkpoint already removed
# or moved this row. The state the caller asked for is the state that holds either way.
except StaleDataError:
db.rollback()
return {"video_id": video_id, "status": "new"}
return {"ok": True, "video_id": video_id, "status": "new"}
# watched / hidden: atomic upsert. The player marks "watched" at end-of-playback at
# the same instant a progress checkpoint may DELETE the in-progress row; a plain
@@ -279,7 +282,7 @@ def set_video_state(
)
db.execute(stmt)
db.commit()
return {"video_id": video_id, "status": status}
return {"ok": True, "video_id": video_id, "status": status}
@router.delete("/videos/{video_id}/state")
@@ -298,9 +301,10 @@ def clear_video_state(
db.delete(row)
try:
db.commit()
# silent-ok: the row went under the delete, which is the outcome the delete wanted.
except StaleDataError:
db.rollback()
return {"video_id": video_id, "status": "new", "position_seconds": 0}
return {"ok": True, "video_id": video_id, "status": "new", "position_seconds": 0}
class VideoProgressIn(BaseModel):
@@ -355,9 +359,10 @@ def set_video_progress(
# have removed/updated this row already — tolerate the lost race.
try:
db.commit()
# silent-ok: exactly the race the comment above describes; the response stays true.
except StaleDataError:
db.rollback()
return {"video_id": video_id, "position_seconds": 0}
return {"ok": True, "video_id": video_id, "position_seconds": 0}
if row is None:
row = VideoState(user_id=user.id, video_id=video_id)
@@ -365,7 +370,7 @@ def set_video_progress(
row.position_seconds = position
row.progress_updated_at = datetime.now(timezone.utc)
db.commit()
return {"video_id": video_id, "position_seconds": position}
return {"ok": True, "video_id": video_id, "position_seconds": position}
@router.get("/videos/{video_id}")
@@ -398,7 +403,7 @@ def get_video_detail(
# The DB path above is free; the off-catalog YouTube lookup below spends a quota unit, so the
# demo account can't reach it — it just sees "unknown" for a video we don't already store.
if user.is_demo:
raise HTTPException(status_code=404, detail="Unknown video")
raise not_found("video")
try:
with (
@@ -407,9 +412,9 @@ def get_video_detail(
):
items = yt.get_videos([video_id])
except YouTubeError as exc:
raise HTTPException(status_code=422, detail=f"YouTube lookup failed: {exc}")
raise youtube_refusal(exc)
if not items:
raise HTTPException(status_code=404, detail="Unknown video")
raise not_found("video")
snippet = items[0].get("snippet", {})
stats = items[0].get("statistics", {})
+25 -14
View File
@@ -17,12 +17,16 @@ from app.auth import (
optional_current_user,
purge_user,
)
from app import diagnostics
from app import debugsession, diagnostics
from app.issues import gitea
from app import email as email_mod
from app import logs
from app import sysconfig
from app.db import get_db
from app.models import Invite, User
from app.utils import iso
from app.refusal import refused
from app.routes._common import not_found
router = APIRouter(prefix="/api/me", tags=["me"])
@@ -69,7 +73,7 @@ def switch_account(
here) are allowed, and access is re-checked in case the invite was revoked since."""
target = body.user_id
if target not in (request.session.get("account_ids") or []):
raise HTTPException(status_code=403, detail="Not an account you've signed into here.")
raise refused(403, "not_your_account")
# An account `_session_user` would refuse — gone, deactivated, suspended, or its session revoked — is
# refused here too, and dropped from the switch list the way `_session_user` drops it. Made the session
# default instead, the next request's current_user would refuse it as the DEFAULT and clear the whole
@@ -77,22 +81,20 @@ def switch_account(
u = db.get(User, target)
if u is None:
drop_from_wallet(request.session, target)
raise HTTPException(status_code=404, detail="That account no longer exists.")
raise not_found("account")
if not is_allowed(db, u.email):
raise HTTPException(status_code=403, detail="That account no longer has access.")
raise refused(403, "account_no_access")
if not u.is_active:
drop_from_wallet(request.session, target)
raise HTTPException(status_code=403, detail="That account no longer has access.")
raise refused(403, "account_no_access")
if u.is_suspended:
drop_from_wallet(request.session, target)
raise HTTPException(status_code=403, detail="That account is suspended.")
raise refused(403, "account_suspended")
# Revoked: a password reset/change or "log out everywhere" bumped the account's epoch.
epochs = request.session.get("epochs") or {}
if int(epochs.get(str(target), 0)) != (u.session_epoch or 0):
drop_from_wallet(request.session, target)
raise HTTPException(
status_code=403, detail="That account was signed out elsewhere — sign in again."
)
raise refused(403, "account_signed_out")
request.session["user_id"] = target
return {"ok": True, "user_id": target}
@@ -138,6 +140,9 @@ def get_me(
user.google_email if user.google_email and user.google_email != user.email else None
),
"has_password": user.password_hash is not None,
# The account's latest sign-in on any device — shown in Settings → Account, so a sign-in the
# user does not recognise is something they can see (null: none since 0.88, or the demo).
"last_login_at": iso(user.last_login_at),
# Instance-wide: whether Google sign-in / YouTube connect is available at all (the UI hides
# YouTube-access affordances and the onboarding nudge when it isn't configured).
"google_enabled": google_enabled(db),
@@ -154,9 +159,18 @@ def get_me(
# The consented diagnostics level in force and when it lapses (E22 S5). The client applies
# it to its own log and streams that log while it is raised.
"diagnostics": diagnostics.state(db, user),
# The live debug session, if one is recording (E16) — a reload resumes it.
"debug_session": _live_session(db, user),
# Where a bug report goes (E16 S4): the tracker's repository, or None — download only.
"issues": gitea.target(db),
}
def _live_session(db: Session, user: User) -> dict | None:
session = debugsession.open_session(db, user)
return None if session is None else debugsession.view(db, session)
@router.delete("/account")
def delete_my_account(
request: Request,
@@ -177,10 +191,7 @@ def delete_my_account(
if user.is_demo:
raise HTTPException(status_code=403, detail={"code": "demo", "reason": "delete_account"})
if user.role == "admin" and count_admins(db) <= 1:
raise HTTPException(
status_code=400,
detail="You're the only admin — promote another admin before deleting your account.",
)
raise refused(400, "only_admin_delete")
user_id = user.id
# Full erasure (cascades) + access-request cleanup + Google-grant revocation; shared with the
# admin delete path. Capture the id first — `user` is detached once purge_user deletes the row.
@@ -213,7 +224,7 @@ def update_preferences(
user.preferences = merged
db.add(user)
db.commit()
return {"preferences": merged}
return {"ok": True, "preferences": merged}
class DiagnosticsIn(BaseModel):
+19 -15
View File
@@ -16,7 +16,7 @@ routing and unread counts.
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, WebSocket, WebSocketDisconnect
from fastapi import APIRouter, Depends, WebSocket, WebSocketDisconnect
from pydantic import BaseModel
from sqlalchemy import and_, case, delete, func, or_, select
from sqlalchemy.orm import Session
@@ -29,6 +29,8 @@ from app.realtime import manager
from app.security import verify_password
from app.userscope import is_messageable_user, messageable_clauses
from app.utils import iso
from app.refusal import refused
from app.routes._common import not_found
router = APIRouter(prefix="/api/messages", tags=["messages"])
@@ -152,7 +154,7 @@ def setup_keys(
"""Store this user's public key + passphrase-wrapped private key. Set-once: changing the
key would orphan every existing conversation key, so re-setup is rejected here."""
if db.get(MessageKey, user.id) is not None:
raise HTTPException(status_code=409, detail="Secure messaging is already set up.")
raise refused(409, "messaging_set_up")
db.add(
MessageKey(
user_id=user.id,
@@ -164,7 +166,7 @@ def setup_keys(
)
)
db.commit()
return {"configured": True}
return {"ok": True, "configured": True}
@router.post("/keys/reset")
@@ -185,11 +187,11 @@ def reset_keys(
less (Google-only) account has nothing to re-verify, so it relies on its live session plus the
client-side danger confirmation."""
if not _reset_key_limiter.allow(str(user.id)):
raise HTTPException(status_code=429, detail="Too many attempts. Try again shortly.")
raise refused(429, "too_many_attempts")
if user.password_hash and not verify_password(
payload.current_password or "", user.password_hash
):
raise HTTPException(status_code=403, detail="Current password is incorrect.")
raise refused(403, "wrong_password")
# Delete the now-dead ciphertext first (both directions), then the wrapped keypair itself so a
# fresh setup is allowed. Order doesn't matter (no FK between them), but do it in one commit.
db.execute(
@@ -202,7 +204,7 @@ def reset_keys(
if key is not None:
db.delete(key)
db.commit()
return {"configured": False}
return {"ok": True, "configured": False}
@router.get("/keys/{user_id}")
@@ -223,10 +225,10 @@ def public_key(
p = db.get(User, user_id)
has_history = db.scalar(select(Message.id).where(cond).limit(1))
if p is None or (not is_messageable_user(p) and not has_history):
raise HTTPException(status_code=404, detail="User not found")
raise not_found("user")
k = db.get(MessageKey, user_id)
if k is None:
raise HTTPException(status_code=404, detail="That member hasn't set up messaging yet.")
raise refused(404, "no_messaging")
return {"user_id": user_id, "public_key": k.public_key}
@@ -344,7 +346,7 @@ def get_thread(
# still opens). Otherwise the same 404 as a non-existent id → no enumeration oracle.
has_history = db.scalar(select(Message.id).where(cond).limit(1))
if p is None or (not is_messageable_user(p) and not has_history):
raise HTTPException(status_code=404, detail="User not found")
raise not_found("user")
partner = _serialize_user(p)
msgs = (
db.execute(select(Message).where(cond).order_by(Message.created_at.asc())).scalars().all()
@@ -374,20 +376,20 @@ def send_message(
"""Send an end-to-end-encrypted message. The server stores only the ciphertext the client
produced, then pushes it live to the recipient (and the sender's other devices)."""
if not _send_limiter.allow(f"msg:{user.id}"):
raise HTTPException(status_code=429, detail="Too many messages — slow down a moment.")
raise refused(429, "message_rate")
ct = (payload.ciphertext or "").strip()
iv = (payload.iv or "").strip()
if not ct or not iv:
raise HTTPException(status_code=400, detail="Message is empty.")
raise refused(400, "message_empty")
if len(ct) > MAX_CIPHERTEXT:
raise HTTPException(status_code=400, detail="Message is too long.")
raise refused(400, "message_too_long")
if payload.recipient_id == user.id:
raise HTTPException(status_code=400, detail="You can't message yourself.")
raise refused(400, "message_self")
recipient = db.get(User, payload.recipient_id)
if not is_messageable_user(recipient):
raise HTTPException(status_code=404, detail="Recipient not available.")
raise not_found("recipient")
if db.get(MessageKey, recipient.id) is None:
raise HTTPException(status_code=404, detail="That member hasn't set up messaging yet.")
raise refused(404, "no_messaging")
m = Message(kind="user", sender_id=user.id, recipient_id=recipient.id, ciphertext=ct, iv=iv)
db.add(m)
db.commit()
@@ -449,6 +451,8 @@ async def messages_ws(ws: WebSocket) -> None:
try:
while True:
await ws.receive_text()
# silent-ok: the ordinary way a chat socket ends — a closed tab. The `finally` below is the
# whole handling, and nothing failed.
except WebSocketDisconnect:
pass
finally:
+4 -10
View File
@@ -5,7 +5,7 @@ All endpoints use `current_user` (not `require_human`): notifications and their
state are personal UI state with no quota or YouTube involvement, so the shared demo account
manages its own inbox normally."""
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, Query
from sqlalchemy import func, select, update
from sqlalchemy.orm import Session
@@ -14,6 +14,7 @@ from app.db import get_db
from app.models import Notification, User
from app.notifications import trim_read
from app.utils import iso
from app.routes._common import owned_or_404
router = APIRouter(prefix="/api/me/notifications", tags=["notifications"])
@@ -72,20 +73,13 @@ def unread_count(user: User = Depends(current_user), db: Session = Depends(get_d
return {"count": count or 0}
def _owned(db: Session, notification_id: int, user: User) -> Notification:
n = db.get(Notification, notification_id)
if n is None or n.user_id != user.id:
raise HTTPException(status_code=404, detail="Notification not found")
return n
@router.post("/{notification_id}/read")
def mark_read(
notification_id: int,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
n = _owned(db, notification_id, user)
n = owned_or_404(db, Notification, notification_id, user, what="notification")
n.read = True
db.commit()
trim_read(db, user.id)
@@ -110,7 +104,7 @@ def dismiss(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
n = _owned(db, notification_id, user)
n = owned_or_404(db, Notification, notification_id, user, what="notification")
# Dismiss implies read (it no longer counts toward the unread badge).
n.dismissed = True
n.read = True
+68 -48
View File
@@ -3,11 +3,10 @@ CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-w
mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import func, select, and_
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session, aliased
from app import quota
@@ -23,7 +22,7 @@ from app.sync.playlists import (
sync_user_playlists,
)
from app.ratelimit import RateLimiter
from app.routes._common import get_or_404, owned_or_404
from app.routes._common import get_or_404, owned_or_404, youtube_refusal
from app.youtube.client import YouTubeClient, YouTubeError
router = APIRouter(prefix="/api/playlists", tags=["playlists"])
@@ -48,10 +47,6 @@ def _check_pull_cooldown(user: User) -> None:
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
def _own_playlist(db: Session, user: User, playlist_id: int) -> Playlist:
return owned_or_404(db, Playlist, playlist_id, user, what="playlist")
def _refused(status: int, reason: str) -> HTTPException:
"""A playlist action the server refuses, as a reason the UI translates
(`playlists.errors.<reason>`) — a bare English detail reached the Hungarian UI verbatim."""
@@ -64,8 +59,11 @@ def _touch(pl: Playlist) -> None:
`UpdatedAtMixin`'s `onupdate` fires on an UPDATE of the PLAYLIST row, and add / remove /
reorder / move write the item rows instead — so without this the rail's "recently changed"
sort would only ever see renames. The assignment is what makes the column mean "last time
this list changed", which is what the rail promises the reader."""
pl.updated_at = datetime.now(timezone.utc)
this list changed", which is what the rail promises the reader.
The DATABASE's clock, like the mixin's own `onupdate`: one sort key written from two clocks
(this process's and Postgres's) could order a rename and an item edit the wrong way round."""
pl.updated_at = func.now()
def _next_playlist_position(db: Session, user_id: int) -> int:
@@ -108,7 +106,7 @@ def _add_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) ->
if mark_dirty:
recompute_dirty(db, pl)
_touch(pl)
db.commit()
db.flush() # the handler commits (CLAUDE.md: helpers flush)
return True
@@ -126,7 +124,7 @@ def _remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool)
if mark_dirty:
recompute_dirty(db, pl)
_touch(pl)
db.commit()
db.flush() # the handler commits (CLAUDE.md: helpers flush)
return True
@@ -280,7 +278,7 @@ def _watch_later(db: Session, user: User) -> Playlist:
position=_next_playlist_position(db, user.id),
)
db.add(pl)
db.commit()
db.flush() # the id, for the item the caller adds next; the handler commits
return pl
@@ -295,7 +293,8 @@ def add_watch_later(
get_or_404(db, Video, video_id, what="video")
pl = _watch_later(db, user)
_add_item(db, pl, video_id, mark_dirty=False)
return {"saved": True, "playlist_id": pl.id}
db.commit()
return {"ok": True, "saved": True, "playlist_id": pl.id}
@router.delete("/watch-later/{video_id}")
@@ -309,7 +308,8 @@ def remove_watch_later(
)
if pl is not None:
_remove_item(db, pl, video_id, mark_dirty=False)
return {"saved": False}
db.commit()
return {"ok": True, "saved": False}
@router.post("/sync-youtube")
@@ -322,8 +322,12 @@ def sync_youtube(
if not has_read_scope(user):
raise _refused(403, "youtube_read_scope")
_check_pull_cooldown(user)
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return sync_user_playlists(db, user)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return {"ok": True, **sync_user_playlists(db, user)}
except YouTubeError as exc:
db.rollback()
raise youtube_refusal(exc)
@router.post("/{playlist_id}/revert-youtube")
@@ -335,7 +339,7 @@ def revert_youtube(
"""Reset one YouTube-linked playlist to its current YouTube state, discarding local
edits (and clearing the dirty flag). The user-facing 'undo all my changes' for a list
once the in-session undo history is gone."""
pl = _own_playlist(db, user, playlist_id)
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
if not pl.yt_playlist_id:
raise _refused(400, "not_on_youtube")
if not has_read_scope(user):
@@ -343,16 +347,16 @@ def revert_youtube(
_check_pull_cooldown(user)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return repull_playlist(db, user, pl)
except YouTubeError:
return {"ok": True, **repull_playlist(db, user, pl)}
except YouTubeError as exc:
db.rollback()
raise _refused(422, "youtube_failed")
raise youtube_refusal(exc)
def _pushable(db: Session, user: User, playlist_id: int) -> Playlist:
"""A local playlist the user may push to YouTube (create or update). The built-in
Watch later and read-only YouTube mirrors are never pushable."""
pl = _own_playlist(db, user, playlist_id)
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
if pl.kind == "watch_later":
raise _refused(400, "watch_later_not_pushable")
return pl
@@ -371,8 +375,8 @@ def push_plan(
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
plan = plan_merge(db, user, pl)
except YouTubeError:
raise _refused(422, "youtube_failed")
except YouTubeError as exc:
raise youtube_refusal(exc)
plan["remaining_today"] = quota.remaining_today(db)
plan["affordable"] = plan["units_estimate"] <= plan["remaining_today"]
return plan
@@ -403,10 +407,10 @@ def push(
plan = plan_merge(db, user, pl, live=live)
if plan["units_estimate"] > quota.remaining_today(db):
raise _refused(429, "quota_exceeded")
return merge_playlist(db, user, pl, live=live)
except YouTubeError:
return {"ok": True, **merge_playlist(db, user, pl, live=live)}
except YouTubeError as exc:
db.rollback()
raise _refused(422, "youtube_failed")
raise youtube_refusal(exc)
@router.get("/{playlist_id}")
@@ -415,7 +419,7 @@ def get_playlist(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist(db, user, playlist_id)
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
state = aliased(VideoState)
rows = db.execute(
select(*feed_columns(user, state))
@@ -447,7 +451,7 @@ def rename_playlist(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist(db, user, playlist_id)
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
if "name" in body.model_fields_set:
name = (body.name or "").strip()
if not name:
@@ -484,7 +488,7 @@ def delete_playlist(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist(db, user, playlist_id)
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
if pl.kind == "watch_later":
raise _refused(400, "watch_later_undeletable")
if on_youtube:
@@ -496,8 +500,8 @@ def delete_playlist(
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_DELETE):
with YouTubeClient(db, user) as yt:
yt.delete_playlist(pl.yt_playlist_id)
except YouTubeError:
raise _refused(422, "youtube_failed")
except YouTubeError as exc:
raise youtube_refusal(exc)
db.delete(pl) # items cascade
db.commit()
return {"deleted": playlist_id}
@@ -510,11 +514,12 @@ def add_item(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist(db, user, playlist_id)
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
video_id = body.video_id
get_or_404(db, Video, video_id, what="video")
_add_item(db, pl, video_id, mark_dirty=True)
return {"playlist_id": pl.id, "video_id": video_id}
db.commit()
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
@router.delete("/{playlist_id}/items/{video_id}")
@@ -524,9 +529,10 @@ def remove_item(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist(db, user, playlist_id)
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
_remove_item(db, pl, video_id, mark_dirty=True)
return {"playlist_id": pl.id, "video_id": video_id}
db.commit()
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
class MoveItemIn(BaseModel):
@@ -548,7 +554,7 @@ def move_item(
exactly the kind of half-applied state this codebase treats as a bug rather than an edge
case. Idempotent: if the video is already in the target, the source row is simply dropped
(moving into a list that already has it is a removal, not a duplicate)."""
src = _own_playlist(db, user, playlist_id)
src = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
# Its own reason: "this playlist doesn't exist any more" about a TARGET that is gone was read as
# the list on screen — which is still there.
dst = owned_or_404(db, Playlist, body.target_playlist_id, user, what="move_target")
@@ -571,19 +577,33 @@ def move_item(
PlaylistItem.playlist_id == dst.id, PlaylistItem.video_id == video_id
)
)
if already is not None:
db.delete(item) # the uq_playlist_video constraint would reject a second row anyway
else:
dst_changed = False
if already is None:
# Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
# survives the move, and there is no window where the video belongs to neither list.
item.playlist_id = dst.id
item.position = _next_item_position(db, dst.id)
# Both sides changed, so both get the dirty recompute and the timestamp — the asymmetric
# version of this (touching only the source) is the recurring smell CLAUDE.md warns about.
# survives the move, and there is no window where the video belongs to neither list. The
# check above is not a lock: a second tab's add or the scheduler's YouTube merge can insert
# the same (playlist, video) in between, and uq_playlist_video then refuses the re-parent.
# That is the already-present outcome arriving late, not a failure — answer it the same way
# instead of a raw 500.
try:
with db.begin_nested():
item.playlist_id = dst.id
item.position = _next_item_position(db, dst.id)
db.flush()
dst_changed = True
# silent-ok: the conflict IS the answer — the target already holds the video, which is the
# idempotent outcome; the savepoint put the row back in the source list, dropped below.
except IntegrityError:
pass
if not dst_changed:
db.delete(item)
# The source always changed; the target only when the row landed in it. `_touch` on a target
# that already had the video promoted an untouched list in the rail's "recently changed" sort.
recompute_dirty(db, src)
recompute_dirty(db, dst)
_touch(src)
_touch(dst)
if dst_changed:
recompute_dirty(db, dst)
_touch(dst)
db.commit()
return {"ok": True, "from_playlist_id": src.id, "to_playlist_id": dst.id, "video_id": video_id}
@@ -599,7 +619,7 @@ def reorder_items(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist(db, user, playlist_id)
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
order = body.video_ids
items = {
it.video_id: it
@@ -625,4 +645,4 @@ def reorder_items(
recompute_dirty(db, pl)
_touch(pl)
db.commit()
return {"playlist_id": pl.id, "count": pos}
return {"ok": True, "playlist_id": pl.id, "count": pos}
+4 -5
View File
@@ -1,20 +1,19 @@
"""Shared helpers, serializers and constants for the plex route package."""
import logging
from fastapi import HTTPException
from sqlalchemy.orm import Session
from app import logs
from app.models import PlexItem
from app.plex.lookups import by_rating_key
from app.routes._common import not_found
log = logging.getLogger("siftlode.plex")
log = logs.get_logger(__name__)
def item_or_404(db: Session, rating_key: str) -> PlexItem:
it = by_rating_key(db, PlexItem, rating_key)
if it is None:
raise HTTPException(status_code=404, detail="Unknown Plex item")
raise not_found("plex_item")
return it
+18 -10
View File
@@ -2,12 +2,12 @@
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.auth import admin_user, current_user
from app.db import get_db
from app.models import (
@@ -20,6 +20,9 @@ from app.plex import sync as plex_sync
from app.plex import watch_sync as plex_watch
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.utils import iso
from app.refusal import refused
log = logs.get_logger(__name__)
router = APIRouter(prefix="/api/plex", tags=["plex"])
@@ -35,10 +38,15 @@ def test_connection(_: User = Depends(admin_user), db: Session = Depends(get_db)
info = plex.server_info()
sections = plex.sections()
media_check = _media_check(db, plex, sections)
except PlexNotConfigured as e:
raise HTTPException(status_code=400, detail=str(e))
except PlexNotConfigured:
raise refused(400, "plex_not_configured") from None
except PlexError as e:
raise HTTPException(status_code=422, detail=f"Plex connection failed: {e}")
# What the upstream said is for the log, not the toast: it can quote the server URL back.
log.warning(
"the Plex connection test failed",
extra=logs.event("plex.admin.connection_failed", reason=type(e).__name__, error=str(e)),
)
raise refused(422, "plex_unreachable") from None
return {
"ok": True,
"server_name": info.get("friendlyName") or info.get("title1") or "Plex",
@@ -95,7 +103,7 @@ def _media_check(db: Session, plex: PlexClient, sections: list[dict]) -> dict:
def trigger_sync(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
"""Admin: mirror the enabled Plex sections into the local catalog. Synchronous (can take a
while on a large library); a background/scheduled sync also runs periodically."""
return plex_sync.sync(db)
return {"ok": True, **plex_sync.sync(db)}
def _link_dict(link: PlexLink | None) -> dict:
@@ -129,7 +137,7 @@ def watch_link_set(
admin token, so no separate Plex login. Enabling it for the first time runs the one-time
"Plex is master" import immediately so the owner's existing Plex history shows up in Siftlode."""
if not sysconfig.get_bool(db, "plex_enabled"):
raise HTTPException(status_code=409, detail="The Plex module is disabled.")
raise refused(409, "plex_disabled")
enabled = body.enabled
link = db.execute(select(PlexLink).where(PlexLink.user_id == user.id)).scalars().first()
if link is None:
@@ -141,7 +149,7 @@ def watch_link_set(
result: dict = {}
if enabled and not link.initial_import_done:
result = plex_watch.import_owner_watch_state(db, link)
return {**_link_dict(link), "import": result or None}
return {"ok": True, **_link_dict(link), "import": result or None}
@router.post("/watch/import")
@@ -150,6 +158,6 @@ def watch_import_now(user: User = Depends(admin_user), db: Session = Depends(get
intersection; Siftlode-only states are preserved. Idempotent."""
link = db.execute(select(PlexLink).where(PlexLink.user_id == user.id)).scalars().first()
if link is None or not link.sync_enabled:
raise HTTPException(status_code=409, detail="Enable Plex watch-sync first.")
raise refused(409, "watch_sync_off")
result = plex_watch.import_owner_watch_state(db, link)
return {**_link_dict(link), "import": result}
return {"ok": True, **_link_dict(link), "import": result}
+23 -21
View File
@@ -9,6 +9,7 @@ from pydantic import BaseModel
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import logs
from app.auth import admin_user, current_user
from app.db import get_db
from app.models import (
@@ -24,7 +25,12 @@ from app.plex import sync as plex_sync
from app.plex.cards import movie_card, show_card
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.lookups import by_rating_key
from app.plex.query import content_libs, like_escape, runtime_by_key, top_art_keys
from app.plex.query import content_libs, runtime_by_key, top_art_keys
from app.utils import like_escape
from app.refusal import refused
from app.routes._common import not_found
log = logs.get_logger(__name__)
# The most-restrictive → least ordering used to pick a collection's effective content rating from its
# members (Plex shows one cert on the collection; we don't store it, so derive it).
@@ -233,7 +239,7 @@ def collection_detail(
endpoints below."""
col = by_rating_key(db, PlexCollection, rating_key)
if col is None:
raise HTTPException(status_code=404, detail="Unknown collection")
raise not_found("collection")
movies = (
db.execute(
select(PlexItem)
@@ -324,9 +330,9 @@ def collection_detail(
def _editable_collection_or_403(db: Session, rating_key: str) -> PlexCollection:
col = by_rating_key(db, PlexCollection, rating_key)
if col is None:
raise HTTPException(status_code=404, detail="Unknown collection")
raise not_found("collection")
if not _collection_editable(col):
raise HTTPException(status_code=403, detail="This collection is read-only")
raise refused(403, "collection_read_only")
return col
@@ -361,14 +367,13 @@ def _write_or_pending_error(wrote: bool, e: Exception) -> HTTPException:
Both use 422, NOT a 5xx: the frontend treats 502/503/504 as "gateway unreachable" and raises a
sticky offline banner (see api/core.ts RETRIABLE_GATEWAY). A 422 surfaces the detail as an ordinary
action-error toast, which is what both of these are — the app answered, it didn't go unreachable."""
if wrote:
detail = (
"Collection updated in Plex, but refreshing the local copy failed; "
f"it will reconcile on the next sync ({e})"
)
else:
detail = f"Plex write failed: {e}"
return HTTPException(status_code=422, detail=detail)
log.warning(
"a Plex collection write did not settle",
extra=logs.event(
"plex.collections.write_failed", wrote=wrote, reason=type(e).__name__, error=str(e)
),
)
return refused(422, "collection_pending_sync") if wrote else refused(422, "plex_write_failed")
class CreateCollectionIn(BaseModel):
@@ -388,7 +393,7 @@ def create_collection(
title = body.title.strip()
seed = body.item_rating_key
if not title or not seed:
raise HTTPException(status_code=400, detail="title and item_rating_key are required")
raise refused(400, "collection_fields_required")
if library:
lib = (
db.execute(select(PlexLibrary).where(PlexLibrary.plex_key == library)).scalars().first()
@@ -398,7 +403,7 @@ def create_collection(
seed_item = by_rating_key(db, PlexItem, seed) or by_rating_key(db, PlexShow, seed)
lib = db.get(PlexLibrary, seed_item.library_id) if seed_item else None
if lib is None:
raise HTTPException(status_code=404, detail="Unknown library")
raise not_found("library")
_reject_duplicate_name(db, lib.id, title)
wrote = False
try:
@@ -472,7 +477,7 @@ def collection_rename(
col = _editable_collection_or_403(db, rating_key)
title = body.title.strip()
if not title:
raise HTTPException(status_code=400, detail="title is required")
raise refused(400, "title_required")
_reject_duplicate_name(db, col.library_id, title, exclude_rk=rating_key)
lib = db.get(PlexLibrary, col.library_id)
wrote = False
@@ -501,7 +506,7 @@ def collection_delete(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise HTTPException(status_code=422, detail=f"Plex write failed: {e}")
raise _write_or_pending_error(False, e) from None
return {"deleted": rating_key}
@@ -520,13 +525,10 @@ def collection_set_editable(
(IMDb/TMDb/…) can never be made editable — they're regenerated outside Plex."""
col = by_rating_key(db, PlexCollection, rating_key)
if col is None:
raise HTTPException(status_code=404, detail="Unknown collection")
raise not_found("collection")
want = body.editable
if want and (col.smart or _collection_source(col) != "collection"):
raise HTTPException(
status_code=400,
detail="Smart / auto-list collections can't be made editable",
)
raise refused(400, "collection_smart")
col.editable = want
db.commit()
return _collection_card(col)
+82 -10
View File
@@ -5,10 +5,11 @@ import time
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import logs
from app.auth import current_user
from app.db import get_db
from app.models import (
@@ -29,6 +30,9 @@ from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.lookups import by_rating_key
from app.routes.plex._shared import _norm_lang, item_or_404
from app.routes.plex.collections import _collection_source
from app.routes._common import not_found
log = logs.get_logger(__name__)
router = APIRouter(prefix="/api/plex", tags=["plex"])
@@ -65,7 +69,21 @@ def _show_enrichment(db: Session, rating_key: str) -> tuple[dict, list]:
related_f = ex.submit(plex.related, rating_key) # swallows its own errors → []
meta = meta_f.result() or {}
related_raw = related_f.result()
except (PlexError, PlexNotConfigured):
except PlexNotConfigured:
# silent-ok: not a failure but a state — this instance has no Plex server, and the page
# is meant to render from the local rows alone.
return {}, []
except PlexError as exc:
# The detail page still renders, from the rows the sync left behind: no cast, no related
# strip, no rich detail. Indistinguishable on screen from a show that simply has none.
log.warning(
"live Plex enrichment failed for a show",
extra=logs.event(
"plex.detail.show_enrich_failed",
rating_key=rating_key,
reason=type(exc).__name__,
),
)
return {}, []
# Don't cache an EMPTY related list: plex.related() swallows a transient failure as [] (same as a
# genuinely no-related show), so caching it would serve an empty "related" section for the whole TTL
@@ -92,7 +110,7 @@ def show_detail(
hu = _norm_lang(lang) == "hu"
sh = by_rating_key(db, PlexShow, rating_key)
if sh is None:
raise HTTPException(status_code=404, detail="Unknown Plex show")
raise not_found("show")
seasons = (
db.execute(
select(PlexSeason)
@@ -145,12 +163,26 @@ def show_detail(
sh.people_enriched_at = datetime.now(timezone.utc)
try:
db.commit()
except Exception:
except Exception: # noqa: BLE001 — a failed enrich must not fail the page
# The rollback un-does the enrichment AND the `people_enriched_at` stamp, so the
# next view tries again — but if the commit keeps failing nobody ever hears, and
# the cast section stays empty for good.
log.warning(
"the show enrichment could not be committed",
extra=logs.event(
"plex.detail.enrich_commit_failed", rating_key=sh.rating_key, kind="show"
),
)
db.rollback()
except Exception:
except Exception: # noqa: BLE001 — the local episode list must render whatever Plex does
# Best-effort only — the local episode list is already assembled, so any live-Plex failure
# (not configured, HTTP/JSON shape) must degrade, not 500.
pass
# (not configured, HTTP/JSON shape) must degrade, not 500. With the traceback, because what
# is left in here after the Plex errors above are handled is our own shape bugs.
log.warning(
"the live part of a show page could not be built",
extra=logs.event("plex.detail.show_extras_failed", rating_key=sh.rating_key),
exc_info=True,
)
imdb_id = sh.imdb_id
return {
@@ -295,6 +327,14 @@ def _episode_nav(db: Session, it: PlexItem) -> tuple[str | None, str | None]:
try:
i = keys.index(it.rating_key)
except ValueError:
# An episode missing from its own show's ordered list — the two came out of the same sync,
# so this is a gap in it. The page loses its previous/next arrows and says nothing.
log.info(
"an episode is not in its show's episode list; no previous/next",
extra=logs.event(
"plex.detail.episode_nav_missing", rating_key=it.rating_key, show_id=it.show_id
),
)
return None, None
return (keys[i - 1] if i > 0 else None), (keys[i + 1] if i < len(keys) - 1 else None)
@@ -366,8 +406,32 @@ def item_detail(
"text": codec not in subtitles.BITMAP_SUBS,
}
)
except Exception:
pass # live extras (markers/streams) are best-effort; any live-Plex failure must degrade, not 500 — playback works without them
# Best-effort, but not free: what either branch drops is the subtitle stream list and the intro
# markers, so the player offers no subtitles and no skip button — and looks like a file that has
# neither.
except PlexNotConfigured:
# silent-ok: not a failure but a state — no Plex server, and nothing to play from anyway.
pass
except PlexError as exc:
log.warning(
"the live extras for an item could not be fetched",
extra=logs.event(
"plex.detail.extras_failed",
rating_key=it.rating_key,
kind=it.kind,
reason=type(exc).__name__,
),
)
except Exception: # noqa: BLE001 — playback works without the extras; never 500 here
# Plex answered and WE could not read it — a shape our parsing does not expect. The same
# split `show_detail` makes: the stack is the only thing that says where.
log.warning(
"the live extras for an item could not be read",
extra=logs.event(
"plex.detail.extras_unreadable", rating_key=it.rating_key, kind=it.kind
),
exc_info=True,
)
# Lazy enrichment (movies only): the live fetch above carries the FULL cast + rich detail (the sync
# only stored the listing's top ~3), so fill a not-yet-enriched movie — its cast becomes filterable
@@ -380,7 +444,15 @@ def item_detail(
it.people_enriched_at = datetime.now(timezone.utc)
try:
db.commit()
except Exception:
except Exception: # noqa: BLE001 — a failed enrich must not fail the page
# Same as the show path above: the info page then serves the three names the listing
# sync stored, and the full cast it just fetched is thrown away unreported.
log.warning(
"the movie enrichment could not be committed",
extra=logs.event(
"plex.detail.enrich_commit_failed", rating_key=it.rating_key, kind="movie"
),
)
db.rollback()
hu = lang == "hu"
+1 -1
View File
@@ -32,12 +32,12 @@ from app.plex.query import (
meta_filter_conds,
top_art_keys,
)
from app.discover.queries import RAIL_LIMIT
from app.plex.tmdb import resolve_recs
from app.routes.plex._shared import _norm_lang
router = APIRouter(prefix="/api/plex", tags=["plex"])
RAIL_LIMIT = 20
GENRE_TILES = 12
SURPRISE_PER_KIND = 3 # Surprise-me: at most this many movies + this many shows (6 total)
+2 -2
View File
@@ -1,6 +1,5 @@
"""Plex people: cast/crew filter-chip cards + a single person's filmography page."""
import logging
import threading
import time
@@ -8,6 +7,7 @@ from fastapi import APIRouter, BackgroundTasks, Depends
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app import logs
from app.auth import current_user
from app.db import SessionLocal, get_db
from app.models import (
@@ -22,7 +22,7 @@ from app.plex.cards import movie_card, show_card
from app.plex.query import content_libs, split_csv
router = APIRouter(prefix="/api/plex", tags=["plex"])
log = logging.getLogger("siftlode.plex")
log = logs.get_logger(__name__)
def _resolve_known_for(
+92 -56
View File
@@ -17,7 +17,8 @@ from fastapi import (
)
from fastapi.responses import FileResponse
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import select
from sqlalchemy import select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.auth import current_user, resolved_user_id
@@ -38,6 +39,8 @@ from app.plex import stream as plex_stream
from app.plex import watch_sync as plex_watch
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.routes.plex._shared import item_or_404
from app.refusal import refused
from app.routes._common import not_found
router = APIRouter(prefix="/api/plex", tags=["plex"])
@@ -48,7 +51,7 @@ def _require_session(request: Request) -> int:
low-sensitivity; a valid session is enough."""
uid, _ = resolved_user_id(request)
if not uid:
raise HTTPException(status_code=401, detail="Not authenticated")
raise refused(401, "not_authenticated")
return uid
@@ -77,15 +80,17 @@ def image(rating_key: str, request: Request, variant: str = "thumb") -> Response
with SessionLocal() as db:
key = imgcache.key_for(db, str(rating_key), v)
if not key:
raise HTTPException(status_code=404, detail="No image")
raise not_found("image")
plex = PlexClient(db) # reads base URL + token now; the fetch below needs no DB
except PlexNotConfigured as e:
raise HTTPException(status_code=400, detail=str(e))
except PlexNotConfigured:
raise refused(400, "plex_not_configured") from None
try:
with plex:
data, ct = plex.image_bytes(key, w, h)
except PlexError as e:
raise HTTPException(status_code=502, detail=f"Plex image fetch failed: {e}")
# silent-ok: a byte proxy for an <img>; the browser shows the placeholder and no reader sees the
# detail — the 502 is the whole answer, as for the person-image proxy below.
except PlexError:
raise refused(502, "image_fetch_failed") from None
imgcache.write(cache_key, data, ct)
return imgcache.response(data, ct)
@@ -120,7 +125,7 @@ def subtitle_status(
`at` is the viewer's absolute position: the extractor windows around it, so what gets read first
is the part actually about to be watched."""
return subtitles.ensure(rating_key, sub_ord, at)
return subtitles.ensure(rating_key, sub_ord, at, viewer=user.id)
@router.get("/subtitle/{rating_key}/{sub_ord}")
@@ -213,7 +218,7 @@ def person_image(u: str, request: Request) -> Response:
Signed-session auth (no DB) so it, like /image, never holds a pool connection during the fetch."""
_require_session(request)
if not u.startswith(plex_people.PERSON_IMG_HOST):
raise HTTPException(status_code=400, detail="Unsupported image host")
raise refused(400, "image_host")
cache_key = "person_" + hashlib.sha1(u.encode()).hexdigest()
hit = imgcache.read(cache_key)
if hit:
@@ -223,7 +228,7 @@ def person_image(u: str, request: Request) -> Response:
r = c.get(u)
r.raise_for_status()
except httpx.HTTPError:
raise HTTPException(status_code=502, detail="Image fetch failed")
raise refused(502, "image_fetch_failed")
ct = r.headers.get("content-type", "image/jpeg")
imgcache.write(cache_key, r.content, ct)
return imgcache.response(r.content, ct)
@@ -295,6 +300,37 @@ def _push_watch(
)
def _mark_finished(db: Session, user_id: int, item_id: int, now: datetime) -> bool:
"""Mark the item watched (position cleared) and say whether THIS call made it so. Two near-end
checkpoints arrive together routinely (the 10 s tick and the debounced `seeked` flush), and a
read-then-write let both see "not watched" and both scrobble. The transition is decided by the
write itself: an insert that did not conflict, or an update conditional on the old status —
under READ COMMITTED the second writer re-reads the committed row and matches nothing."""
values = {
"status": "watched",
"watched_at": now,
"position_seconds": 0,
"progress_updated_at": now,
"synced_to_plex": False,
}
inserted = db.execute(
pg_insert(PlexState)
.values(user_id=user_id, item_id=item_id, **values)
.on_conflict_do_nothing(constraint="uq_plex_user_item")
.returning(PlexState.id)
).first()
if inserted is not None:
return True
mine = (PlexState.user_id == user_id) & (PlexState.item_id == item_id)
if db.execute(
update(PlexState).where(mine, PlexState.status != "watched").values(values)
).rowcount:
return True
# Already watched: a re-finish still restamps the row, as it always did — just no scrobble.
db.execute(update(PlexState).where(mine).values(values))
return False
class ItemProgressIn(BaseModel):
# Finite only: JSON parsing here accepts `Infinity`/`NaN`, and the handler's `int()` turned
# one into a 500.
@@ -330,11 +366,27 @@ def item_progress(
# finish latch is up, or before the first position is known. Those are the states in which
# reaping is the right answer anyway, and a viewer who scrubs back into a reaped session gets
# the same 404 → reclaim the player already handles.
plex_stream.touch_session(str(rating_key))
plex_stream.touch_session(user.id, str(rating_key))
position = max(0, int(body.position_seconds))
duration = int(body.duration_seconds)
final = body.final
now = datetime.now(timezone.utc)
near_end = duration > 0 and position > duration - _FINISH_MARGIN_S
if near_end:
flipped = _mark_finished(db, user.id, it.id, now)
# The finish supersedes every resume push still in flight for this item: stamping it as the
# latest push makes `_resume_push_superseded` drop them. Popping the entry instead (to keep
# the map small) removed the very guard, and an older resume push could then land at Plex
# AFTER the scrobble and revive a resume point on a watched item. Eviction bounds the map.
with _push_throttle_lock:
_last_push_at[(user.id, it.id)] = now
db.commit()
# Only the checkpoint that flipped the row scrobbles: a later checkpoint of the same finished
# item — or a second one racing this one — would otherwise bump Plex's viewCount again.
if flipped:
_push_watch(background, db, user.id, it.id, rating_key, "watched")
return {"ok": True, "status": "watched", "position_seconds": 0}
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user.id, PlexState.item_id == it.id)
@@ -342,28 +394,6 @@ def item_progress(
.scalars()
.first()
)
was_watched = st is not None and st.status == "watched"
near_end = duration > 0 and position > duration - _FINISH_MARGIN_S
if near_end:
if st is None:
st = PlexState(user_id=user.id, item_id=it.id)
db.add(st)
st.status = "watched"
st.watched_at = now
st.position_seconds = 0
st.progress_updated_at = now
st.synced_to_plex = False
# Terminal state for resume write-back: drop the throttle/ordering entry so the map doesn't
# retain a dead key for every finished item.
with _push_throttle_lock:
_last_push_at.pop((user.id, it.id), None)
db.commit()
# Immediate on the watched transition; skip a repeat scrobble if it was already watched
# (a later checkpoint of the same finished item would otherwise bump Plex's viewCount again).
if not was_watched:
_push_watch(background, db, user.id, it.id, rating_key, "watched")
return {"status": "watched", "position_seconds": 0}
if position < _PROGRESS_MIN_S:
if st is not None and st.status == "new":
@@ -372,7 +402,7 @@ def item_progress(
st.position_seconds = 0
st.progress_updated_at = now
db.commit()
return {"position_seconds": 0}
return {"ok": True, "position_seconds": 0}
if st is None:
st = PlexState(user_id=user.id, item_id=it.id)
@@ -397,7 +427,7 @@ def item_progress(
duration * 1000,
intent=now,
)
return {"position_seconds": position}
return {"ok": True, "position_seconds": position}
class ItemStateIn(BaseModel):
@@ -418,7 +448,7 @@ def item_state(
it = item_or_404(db, rating_key)
status = body.status
if status not in ("new", "watched", "hidden"):
raise HTTPException(status_code=400, detail="Invalid status")
raise refused(400, "invalid_status")
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user.id, PlexState.item_id == it.id)
@@ -443,7 +473,7 @@ def item_state(
db.commit()
if mirrored_on_plex:
_push_watch(background, db, user.id, it.id, rating_key, "unwatched")
return {"status": "new"}
return {"ok": True, "status": "new"}
if st is None:
st = PlexState(user_id=user.id, item_id=it.id)
db.add(st)
@@ -455,7 +485,7 @@ def item_state(
db.commit()
if status == "watched" and prev_status != "watched":
_push_watch(background, db, user.id, it.id, rating_key, "watched")
return {"status": status}
return {"ok": True, "status": status}
def _bulk_state(
@@ -497,7 +527,9 @@ def _bulk_state(
db.delete(st)
if prev != "new":
to_push.append((e.id, e.rating_key, "unwatched"))
db.commit()
# Flush, not commit: the handler commits, and it does so before the response — which is when
# the background push below runs, so the push always reads the committed marks.
db.flush()
if pushable and to_push:
# One coalesced background task for the whole batch (reuses a single session + Plex client),
# not one task per episode — a 200-ep show mark is 1 task, not 200.
@@ -520,7 +552,7 @@ def show_state(
"""Mark a whole show watched/unwatched (all its episodes) for the current user + push to Plex."""
sh = by_rating_key(db, PlexShow, rating_key)
if sh is None:
raise HTTPException(status_code=404, detail="Unknown Plex show")
raise not_found("show")
watched = body.watched
eps = (
db.execute(select(PlexItem).where(PlexItem.show_id == sh.id, PlexItem.kind == "episode"))
@@ -528,7 +560,8 @@ def show_state(
.all()
)
changed = _bulk_state(background, db, user.id, eps, watched)
return {"changed": changed, "watched": watched}
db.commit()
return {"ok": True, "changed": changed, "watched": watched}
@router.post("/season/{rating_key}/state")
@@ -542,7 +575,7 @@ def season_state(
"""Mark a whole season watched/unwatched (all its episodes) for the current user + push to Plex."""
se = by_rating_key(db, PlexSeason, rating_key)
if se is None:
raise HTTPException(status_code=404, detail="Unknown Plex season")
raise not_found("season")
watched = body.watched
eps = (
db.execute(select(PlexItem).where(PlexItem.season_id == se.id, PlexItem.kind == "episode"))
@@ -550,7 +583,8 @@ def season_state(
.all()
)
changed = _bulk_state(background, db, user.id, eps, watched)
return {"changed": changed, "watched": watched}
db.commit()
return {"ok": True, "changed": changed, "watched": watched}
@router.post("/stream/{rating_key}/session")
@@ -575,21 +609,23 @@ def stream_session(
# the client can switch audio without a restart — even for otherwise direct-playable files.
if it.playable == "direct" and audio is None and not aoff and not multi:
return {
"ok": True,
"mode": "direct",
"url": f"/api/plex/stream/{it.rating_key}/file",
"start_s": 0.0,
}
if plex_paths.local_media_path(db, it.file_path) is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
s = plex_stream.start_session(db, it, start, audio, aoff, multi, transcode=transcode)
raise refused(404, "media_missing")
s = plex_stream.start_session(db, it, user.id, start, audio, aoff, multi, transcode=transcode)
if s is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
raise refused(404, "media_missing")
# Report the REAL media start (the keyframe ffmpeg landed on, which hls.js zero-bases to), not the
# requested offset — the client's absolute clock and the subtitle cue-shift both key off this.
# `transcode`/`hw`/`tonemap` let the player show an honest cold-start ("Preparing stream…") and
# surface the active mode + encoder in the stats overlay (`tonemap` = the HW HDR→SDR path); both
# segment streams are otherwise identical.
return {
"ok": True,
"mode": "hls",
"start_s": s.media_start_s,
# The session id is part of the path (not a query param) so the playlist's RELATIVE segment
@@ -614,7 +650,7 @@ def stream_file(
it = item_or_404(db, rating_key)
src = plex_paths.local_media_path(db, it.file_path)
if src is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
raise refused(404, "media_missing")
if download:
# A clean, safe download name from the title + the real file extension.
ext = src.suffix or (f".{it.container}" if it.container else "")
@@ -651,9 +687,9 @@ def stream_hls_file(
references its segments/subtitles by relative name, all resolving back here.
The session id in the path is what makes "back here" mean the same session. Sessions are keyed by
rating_key alone and a seek-restart or a second viewer of the same item replaces the registered
one, while the DIRECTORY is per (key, offset, tag) — so resolving by rating_key handed the old
viewer the new session's `seg_12.ts`, produced from a different offset under `-copyts`: wrong
(viewer, rating_key) and a seek-restart replaces the viewer's registered one, while the DIRECTORY
is per (key, viewer, offset, tag) — so resolving by the key alone handed the old player the new
session's `seg_12.ts`, produced from a different offset under `-copyts`: wrong
bytes with a 200, which the player has no way to notice (deep-review round 5). A request for a
session that is no longer the registered one is a 404, and the player reclaims at the playhead.
@@ -668,10 +704,10 @@ def stream_hls_file(
or "\\" in filename
or ".." in filename
):
raise HTTPException(status_code=404, detail="Not found")
s = plex_stream.current_session(str(rating_key))
raise not_found("file")
s = plex_stream.current_session(user.id, str(rating_key))
if s is None or s.id != session_id:
raise HTTPException(status_code=404, detail="No active playback session")
raise refused(404, "session_gone")
path = s.dir / filename
# Only a name THIS session can actually produce is worth waiting for: its own playlists, and a
# segment of one of its own renditions whose predecessor is already on disk (ffmpeg writes them
@@ -681,14 +717,14 @@ def stream_hls_file(
# past the produced region — or a deliberate flood of hopeless names — from parking threadpool
# workers and their DB connections (see `artifact_may_arrive`).
if not plex_stream.artifact_may_arrive(s, filename):
raise HTTPException(status_code=404, detail="Not available yet")
raise refused(404, "not_ready")
timeout = 30 if ext == ".m3u8" else 20
# `proc` so a session whose ffmpeg has already exited answers at once instead of polling out the
# whole timeout: this is a SYNC route on the ~40-slot threadpool, holding a pooled DB connection
# with it, so every such wait is a worker (and a connection) parked for 20 s — an ordinary seek
# past the produced region was enough to do it (round 5).
if not plex_stream.wait_for(path, timeout, proc=s.proc):
raise HTTPException(status_code=404, detail="Not available yet")
raise refused(404, "not_ready")
try:
content = path.read_bytes()
except OSError:
@@ -699,7 +735,7 @@ def stream_hls_file(
# restarting at the playhead. Ordinary trigger: `loadSession` awaits the new session — which
# kills and rmtrees the old one — before destroying the old hls instance, so the outgoing
# instance's in-flight segment request races the rmtree on every seek-restart.
raise HTTPException(status_code=404, detail="No active playback session") from None
raise refused(404, "session_gone") from None
return Response(
content=content,
media_type=_HLS_CT[ext],
+16 -19
View File
@@ -2,7 +2,7 @@
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import delete, func, select, update
from sqlalchemy.orm import Session
@@ -19,15 +19,12 @@ from app.models import (
)
from app.plex.cards import leaf_card
from app.plex.lookups import by_rating_key
from app.routes._common import owned_or_404
from app.routes._common import owned_or_404, not_found
from app.refusal import refused
router = APIRouter(prefix="/api/plex", tags=["plex"])
def _own_playlist_or_404(db: Session, pid: int, user: User) -> PlexPlaylist:
return owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
def _playlist_card(pl: PlexPlaylist, count: int, thumb_rk: str | None) -> dict:
return {
"id": pl.id,
@@ -120,7 +117,7 @@ def create_playlist(
) -> dict:
title = body.title.strip()
if not title:
raise HTTPException(status_code=400, detail="title is required")
raise refused(400, "title_required")
pl = PlexPlaylist(user_id=user.id, title=title)
db.add(pl)
db.flush()
@@ -140,7 +137,7 @@ def create_playlist(
def playlist_detail(
pid: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
rows = db.execute(_playlist_item_query(pid)).all()
items = [it for (_pi, it) in rows]
states = {
@@ -183,10 +180,10 @@ def rename_playlist(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
title = body.title.strip()
if not title:
raise HTTPException(status_code=400, detail="title is required")
raise refused(400, "title_required")
pl.title = title
db.commit()
count = (
@@ -203,7 +200,7 @@ def rename_playlist(
def delete_playlist(
pid: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
db.delete(pl)
db.commit()
return {"deleted": pid}
@@ -220,10 +217,10 @@ def playlist_add_item(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
it = by_rating_key(db, PlexItem, body.item_rating_key)
if it is None:
raise HTTPException(status_code=404, detail="Unknown item")
raise not_found("item")
exists = (
db.execute(
select(PlexPlaylistItem).where(
@@ -252,7 +249,7 @@ def playlist_remove_item(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
it = by_rating_key(db, PlexItem, item_rating_key)
if it is not None:
db.execute(
@@ -297,7 +294,7 @@ def playlist_add_bulk(
) -> dict:
"""Append many items at once (a whole season/show). Already-present items are skipped; the input
order is preserved for the newly-added ones. Returns how many were added + the new total."""
pl = _own_playlist_or_404(db, pid, user)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
items = _items_by_rks(db, body.item_rating_keys)
existing = set(
db.execute(
@@ -328,7 +325,7 @@ def playlist_add_bulk(
)
or 0
)
return {"added": added, "item_count": total}
return {"ok": True, "added": added, "item_count": total}
@router.post("/playlists/{pid}/items/remove-bulk")
@@ -340,7 +337,7 @@ def playlist_remove_bulk(
) -> dict:
"""Remove many items at once (a whole season/show, or a multi-select). Returns how many were
removed + the new total."""
pl = _own_playlist_or_404(db, pid, user)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
ids = [it.id for it in _items_by_rks(db, body.item_rating_keys)]
removed = 0
if ids:
@@ -359,7 +356,7 @@ def playlist_remove_bulk(
)
or 0
)
return {"removed": removed, "item_count": total}
return {"ok": True, "removed": removed, "item_count": total}
@router.put("/playlists/{pid}/order")
@@ -370,7 +367,7 @@ def reorder_playlist(
db: Session = Depends(get_db),
) -> dict:
"""Set the playlist order from an explicit list of item rating_keys (0-based positions)."""
pl = _own_playlist_or_404(db, pid, user)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
order = body.item_rating_keys
id_by_rk = {
it.rating_key: it.id
+15 -11
View File
@@ -6,7 +6,7 @@ the app or any account data. Password-protected links exchange the password once
a short-lived signed grant that the media URL carries (a `<video src>` can't send a header).
"""
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi import APIRouter, Depends, Request
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import select
@@ -19,6 +19,8 @@ from app.downloads import storage
from app.models import DownloadJob, DownloadLink, MediaAsset
from app.ratelimit import RateLimiter, client_key
from app.security import verify_password
from app.refusal import refused
from app.routes._common import not_found
router = APIRouter(prefix="/api/public", tags=["public"])
@@ -37,7 +39,7 @@ _unlock_failures = RateLimiter(max_events=50, window_seconds=300)
def _link_or_404(db: Session, token: str) -> DownloadLink:
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
if link is None or linksmod.is_expired(link):
raise HTTPException(status_code=404, detail="This link is no longer available.")
raise refused(404, "link_gone")
return link
@@ -45,12 +47,12 @@ def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAs
job = db.get(DownloadJob, link.job_id)
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if job is None or asset is None or asset.status != "ready" or not asset.rel_path:
raise HTTPException(status_code=404, detail="This video is no longer available.")
raise refused(404, "video_gone")
# Verify the file is actually on disk here (not just flagged ready), so the watch page never
# loads a player for a missing file — meta and file agree.
path = storage.safe_abs_path(settings.download_root, asset.rel_path)
if path is None:
raise HTTPException(status_code=404, detail="This video is no longer available.")
raise refused(404, "video_gone")
return job, asset
@@ -100,22 +102,24 @@ def watch_unlock(
link = _link_or_404(db, token)
viewer = f"{token}|{client_key(request)}"
if _unlock_failures.exhausted(token) or not _unlock_limit.allow(viewer):
raise HTTPException(status_code=429, detail="Too many attempts. Try again later.")
raise refused(429, "too_many_attempts")
if not link.password_hash:
# Not a password link — nothing to unlock; just hand back the plain meta.
job, asset = _ready_target(db, link)
return {
"ok": True,
"needs_password": False,
**linksmod.public_meta(link, job, asset, _file_url(token), _poster_url(token, asset)),
}
if not verify_password(body.password, link.password_hash):
_unlock_failures.allow(token)
raise HTTPException(status_code=403, detail="Wrong password.")
raise refused(403, "wrong_share_password")
job, asset = _ready_target(db, link)
link.view_count += 1
db.commit()
grant = linksmod.make_grant(token, link.password_version)
return {
"ok": True,
"needs_password": False,
**linksmod.public_meta(
link, job, asset, _file_url(token, grant), _poster_url(token, asset, grant)
@@ -129,11 +133,11 @@ def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
`allow_download` link serves it as an attachment. Password links require a valid grant."""
link = _link_or_404(db, token)
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
raise HTTPException(status_code=403, detail="This link needs a password.")
raise refused(403, "share_password_required")
asset = _ready_asset(db, link)
path = storage.safe_abs_path(settings.download_root, asset.rel_path)
if path is None:
raise HTTPException(status_code=404, detail="This video is no longer available.")
raise refused(404, "video_gone")
filename = storage.download_filename(asset.title or "video", asset.container, path)
# The served Content-Type comes from the asset's ffprobe-verified media_kind + a container
@@ -163,11 +167,11 @@ def watch_poster(token: str, g: str | None = None, db: Session = Depends(get_db)
and the link-preview image). Password links require a valid grant, like the file."""
link = _link_or_404(db, token)
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
raise HTTPException(status_code=403, detail="This link needs a password.")
raise refused(403, "share_password_required")
job, asset = _ready_target(db, link)
if not asset.poster_path:
raise HTTPException(status_code=404, detail="No poster.")
raise not_found("poster")
path = storage.safe_abs_path(settings.download_root, asset.poster_path)
if path is None:
raise HTTPException(status_code=404, detail="No poster.")
raise not_found("poster")
return FileResponse(path, media_type="image/jpeg")
+9 -12
View File
@@ -1,4 +1,4 @@
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import func, select, update
from sqlalchemy.exc import IntegrityError
@@ -8,6 +8,7 @@ from app.auth import require_human
from app.db import get_db
from app.models import SavedView, User
from app.routes._common import owned_or_404
from app.refusal import refused
# Saved "smart views": per-user named snapshots of a module's filters (feed or plex, keyed by
# `source`). Gated behind require_human so the shared demo account can't pollute a communal list
@@ -66,7 +67,7 @@ def create_view(
) -> dict:
name = body.name.strip()
if not name:
raise HTTPException(status_code=400, detail="name is required")
raise refused(400, "name_required")
source = _norm_source(body.source)
# Position is per (user, source), so each module's views number 0..N independently.
maxpos = db.execute(
@@ -86,14 +87,10 @@ def create_view(
db.commit()
except IntegrityError:
db.rollback()
raise HTTPException(status_code=409, detail=f"You already have a view named “{name}”.")
raise refused(409, "view_name_taken", name=name)
return _serialize(view)
def _own_view(db: Session, user: User, view_id: int) -> SavedView:
return owned_or_404(db, SavedView, view_id, user, what="view")
class UpdateViewIn(BaseModel):
# PATCH: presence decides what's touched (read via model_fields_set). filters, when sent,
# must be a real object — an explicit null is rejected (a view can't have null filters).
@@ -109,16 +106,16 @@ def update_view(
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
view = _own_view(db, user, view_id)
view = owned_or_404(db, SavedView, view_id, user, what="view")
sent = body.model_fields_set
if "name" in sent:
name = (body.name or "").strip()
if not name:
raise HTTPException(status_code=400, detail="name cannot be empty")
raise refused(400, "name_required")
view.name = name[:80]
if "filters" in sent:
if body.filters is None:
raise HTTPException(status_code=400, detail="filters must be an object")
raise refused(400, "filters_not_object")
view.filters = body.filters
if "is_default" in sent:
make_default = bool(body.is_default)
@@ -138,7 +135,7 @@ def update_view(
db.commit()
except IntegrityError:
db.rollback()
raise HTTPException(status_code=409, detail=f"You already have a view named “{view.name}”.")
raise refused(409, "view_name_taken", name=view.name)
return _serialize(view)
@@ -148,7 +145,7 @@ def delete_view(
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
view = _own_view(db, user, view_id)
view = owned_or_404(db, SavedView, view_id, user, what="view")
db.delete(view)
db.commit()
return {"deleted": view_id}
+11 -12
View File
@@ -2,7 +2,7 @@
per-job run activity (from the in-process scheduler), the work still queued (DB-derived),
and the shared quota picture."""
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
@@ -25,6 +25,8 @@ from app.scheduler import (
from app.sync.explore import followed_channel
from app.sync.runner import estimate_deep_backfill
from app.routes._common import not_found
from app.refusal import refused
from app.refusal import setting_refused
router = APIRouter(prefix="/api/admin/scheduler", tags=["admin"])
@@ -46,10 +48,7 @@ def set_job_interval(
raise not_found("job")
minutes = body.interval_minutes
if not (MIN_INTERVAL <= minutes <= MAX_INTERVAL):
raise HTTPException(
status_code=400,
detail=f"interval must be between {MIN_INTERVAL} and {MAX_INTERVAL} minutes",
)
raise refused(400, "invalid_interval", min=MIN_INTERVAL, max=MAX_INTERVAL)
applied = apply_interval(db, job_id, minutes)
audit.record(
db,
@@ -61,7 +60,7 @@ def set_job_interval(
after={"interval_minutes": applied},
)
db.commit()
return {"id": job_id, "interval_minutes": applied}
return {"ok": True, "id": job_id, "interval_minutes": applied}
@router.post("/jobs/{job_id}/run")
@@ -77,14 +76,14 @@ def run_job_now(
except KeyError:
raise not_found("job")
if result == "already_running":
raise HTTPException(status_code=409, detail="Job is already running")
return {"id": job_id, "started": True}
raise refused(409, "job_running")
return {"ok": True, "id": job_id, "started": True}
@router.post("/run-all")
def run_all_now(user: User = Depends(admin_user)) -> dict:
"""Trigger every job that isn't already running. Returns the ids actually started."""
return {"started": trigger_all(actor_id=user.id)}
return {"ok": True, "started": trigger_all(actor_id=user.id)}
class MaintenanceSettingsIn(BaseModel):
@@ -102,8 +101,8 @@ def set_maintenance_settings(
# set_value validates against the spec's min/max (one source of bounds), mirroring the config PATCH.
try:
sysconfig.set_value(db, "maintenance_revalidate_batch", body.revalidate_batch)
except (TypeError, ValueError) as exc:
raise HTTPException(status_code=400, detail=str(exc) or "Invalid value")
except sysconfig.InvalidSetting as exc:
raise setting_refused(exc) from None
new = sysconfig.get_int(db, "maintenance_revalidate_batch")
audit.record(
db,
@@ -113,7 +112,7 @@ def set_maintenance_settings(
after={"revalidate_batch": new},
)
db.commit()
return {"revalidate_batch": new}
return {"ok": True, "revalidate_batch": new}
@router.get("")
+84 -31
View File
@@ -13,17 +13,16 @@ this search-discovered "noise" by default. Shorts and currently-live/upcoming vi
ingested this way (Shorts are confirmed by the youtube.com/shorts probe; live/upcoming are
dropped from the search response up front)."""
import logging
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import and_, delete, func, or_, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session, aliased
from app import quota, sysconfig
from app import logs, quota, sysconfig
from app.auth import require_human
from app.db import get_db
from app.models import (
@@ -39,14 +38,16 @@ from app.models import (
VideoState,
)
from app.feedquery import feed_columns, serialize
from app.utils import like_escape
from app.sync.subscriptions import apply_channel_details
from app.sync.videos import insert_stubs, apply_video_details, parse_dt
from app.youtube import search_scrape
from app.youtube.client import YouTubeClient, YouTubeError
from app.youtube.shorts import make_client, probe_is_short
from app.refusal import refused
from app.routes._common import youtube_refusal
log = logging.getLogger("siftlode.search")
log = logs.get_logger(__name__)
router = APIRouter(prefix="/api/search", tags=["search"])
# search.list costs this many quota units per page.
@@ -132,6 +133,13 @@ def _ingest_candidates(
apply_channel_details(db, details)
db.commit()
except YouTubeError:
# The stubs remain and the details can be backfilled later — but until they are, those
# channels sit in the results with no thumbnail and no subscriber count, looking like
# channels that have neither.
log.warning(
"the channels found by a search could not be enriched",
extra=logs.event("search.channel_enrich_failed", channels=len(new_channel_ids)),
)
db.rollback() # details can be backfilled later; the stubs remain
# 2) Video stubs, flagged via_search. Existing rows keep their flags (ON CONFLICT DO
@@ -161,6 +169,12 @@ def _ingest_candidates(
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_ENRICH):
detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
except YouTubeError:
# Every candidate then stays unenriched, and an unenriched video cannot be judged live or
# Short — so it is dropped. A working search returns nothing and reads as "no results".
log.warning(
"the videos found by a search could not be enriched",
extra=logs.event("search.video_enrich_failed", candidates=len(ids)),
)
detail_by_id = {}
enriched_at = datetime.now(
timezone.utc
@@ -213,6 +227,25 @@ def _new_first(db: Session, user: User, ids: list[str]) -> list[str]:
return [i for i in ids if i not in have] + [i for i in ids if i in have]
def _fold_term(db: Session, video_ids: list[str], term: str) -> None:
"""Append `term` to each video's space-joined `search_terms`, unless it is already one of them.
"Already one of them" is a whole-term match, as a literal: a substring test skipped `cat` on a
video already found for `category`, and an unescaped one read the `_` in `my_channel` as a
wildcard — either way the term was never folded in, and local search lost that relevance."""
padded = func.concat(" ", func.coalesce(Video.search_terms, ""), " ")
db.execute(
update(Video)
.where(Video.id.in_(video_ids))
.where(padded.notilike(f"% {like_escape(term)} %"))
.values(
search_terms=func.trim(
func.coalesce(Video.search_terms, "").op("||")(" ").op("||")(term)
)
)
)
@router.get("/youtube")
def search_youtube(
q: str | None = None,
@@ -227,7 +260,7 @@ def search_youtube(
page, so it returns a single page (≤50) regardless. `cursor` lets the caller fetch further."""
term = (q or "").strip()
if not term:
raise HTTPException(status_code=400, detail="A search term is required.")
raise refused(400, "search_term_required")
limit = max(1, min(limit, 100))
# Source: "scrape" (InnerTube, no API quota) or "api" (search.list, 100 units/page).
@@ -238,16 +271,20 @@ def search_youtube(
daily_limit = sysconfig.get_int(db, "search_daily_limit_per_user")
used = quota.actions_today(db, user.id, quota.QuotaAction.VIDEOS_SEARCH)
if used >= daily_limit:
raise HTTPException(
status_code=429,
detail=f"You've reached today's YouTube search limit ({daily_limit}). Try again tomorrow.",
# The 429 is in the request line either way; which of the two caps produced it is not, and
# they are answered by different things — this one by tomorrow, the other by the admin.
log.info(
"a live search was refused: the per-user daily cap is spent",
extra=logs.event("search.refused", reason="daily_cap", limit=daily_limit),
)
raise refused(429, "search_daily_limit", limit=daily_limit)
# The 100-unit budget pre-check only applies to the API source.
if source == "api" and not quota.can_spend(db, SEARCH_COST):
raise HTTPException(
status_code=429,
detail="The shared YouTube quota for today is used up. Try again tomorrow.",
log.info(
"a live search was refused: the shared YouTube budget is spent",
extra=logs.event("search.refused", reason="shared_quota", cost=SEARCH_COST),
)
raise refused(429, "quota_spent")
# Channels this user has blocked are dropped before ingestion — never shown, never stored.
blocked = set(
@@ -272,9 +309,14 @@ def search_youtube(
# dropout (auto-retry + "connection lost" banner — lib/api/core RETRIABLE_GATEWAY),
# which hides the real message and burns a second quota-costed search. Upstream failure
# on a JSON endpoint = 422 + message; reserve 5xx for the raw byte-proxies. (B-core-1)
raise HTTPException(
status_code=422, detail=f"YouTube search failed: {exc}"
) from exc
# A Data API failure is logged where it happened; the scraper's is not, and its text
# no longer rides the answer — so it is said here.
if isinstance(exc, search_scrape.ScrapeError):
log.warning(
"the YouTube search scrape failed",
extra=logs.event("search.scrape_failed", error=str(exc)),
)
raise youtube_refusal(exc) from exc
# Drop currently-live/upcoming, channel-less, and blocked-channel results.
candidates = [
@@ -291,12 +333,32 @@ def search_youtube(
collected.extend(_ingest_candidates(db, yt, user, candidates))
except Exception as exc: # noqa: BLE001 — any ingest failure, not just YouTubeError
db.rollback()
log.exception("live search ingest failed for %r", term)
# The term is NOT in this line. A search string is exactly what the privacy
# policy keeps out of the log unless the user turned Teljes/TRACE on for
# themselves, and `redact_text` does not cover it — it scrubs addresses and
# credential params, not what someone typed into the search box. What the
# ERROR line carries instead is the shape of the failure, which is what tells
# an ingest bug from an upstream one; the term follows at TRACE, and only
# under this user's own consent — not whenever the process runs at TRACE.
log.exception(
"live search ingest failed",
extra=logs.event(
"search.live.ingest_failed",
source=source,
candidates=len(candidates),
collected=len(collected),
),
)
if logs.consented_trace():
log.log(
logs.TRACE,
"the term the ingest failed on",
# log-pii-ok: only under this user's own Teljes consent
extra=logs.event("search.live.term", term=term),
)
if collected:
break # keep what earlier pages already gathered
raise HTTPException(
status_code=422, detail=f"YouTube search failed: {exc}"
) from exc
raise youtube_refusal(exc) from exc
next_cursor = page["next_page_token"]
# The API source pages cost 100 units each — never auto-page; one page per request.
if source == "api" or len(collected) >= limit or not next_cursor:
@@ -321,16 +383,7 @@ def search_youtube(
.values([{"user_id": user.id, "video_id": vid} for vid in ordered])
.on_conflict_do_nothing(index_elements=["user_id", "video_id"])
)
db.execute(
update(Video)
.where(Video.id.in_(ordered))
.where(func.coalesce(Video.search_terms, "").notilike(f"%{term}%"))
.values(
search_terms=func.trim(
func.coalesce(Video.search_terms, "").op("||")(" ").op("||")(term)
)
)
)
_fold_term(db, ordered, term)
db.commit()
ordered = _new_first(db, user, ordered)
@@ -358,7 +411,7 @@ def clear_search(
find. Kept videos and organically-owned ones are left untouched."""
ids = body.video_ids
if not ids:
return {"removed": 0}
return {"ok": True, "removed": 0}
db.execute(
delete(SearchFind).where(SearchFind.user_id == user.id, SearchFind.video_id.in_(ids))
)
@@ -378,4 +431,4 @@ def clear_search(
)
).rowcount
db.commit()
return {"removed": removed}
return {"ok": True, "removed": removed}
+14 -14
View File
@@ -8,7 +8,7 @@ X-Setup-Token header).
from typing import Any
from fastapi import APIRouter, Depends, Header, HTTPException
from fastapi import APIRouter, Depends, Header
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.orm import Session
@@ -21,6 +21,8 @@ from app.auth import PASSWORD_MIN_LEN
from app.db import get_db
from app.models import User
from app.security import hash_password
from app.refusal import refused
from app.refusal import setting_refused
router = APIRouter(prefix="/api/setup", tags=["setup"])
@@ -31,9 +33,9 @@ def require_setup(
) -> Session:
"""Gate for setup steps. Returns the db session for the handler to reuse."""
if state.is_configured(db):
raise HTTPException(status_code=409, detail="This instance is already set up.")
raise refused(409, "already_set_up")
if not state.check_setup_token(db, x_setup_token):
raise HTTPException(status_code=403, detail="Invalid or missing setup token.")
raise refused(403, "setup_token")
return db
@@ -66,11 +68,9 @@ def setup_admin(body: SetupAdminIn, db: Session = Depends(require_setup)) -> dic
email = body.email.strip().lower()
password = body.password
if "@" not in email:
raise HTTPException(status_code=400, detail="Enter a valid email address.")
raise refused(400, "invalid_email")
if len(password) < PASSWORD_MIN_LEN:
raise HTTPException(
status_code=400, detail=f"Password must be at least {PASSWORD_MIN_LEN} characters."
)
raise refused(400, "weak_password", min=PASSWORD_MIN_LEN)
# Resolve canonically, not by the literal string (DX-17). The wizard can be re-run, and E9.5's
# whole point is that two spellings of one Gmail mailbox are one person — matching literally here
# would let a second run under the other spelling create the DUPLICATE admin row the epic exists
@@ -97,16 +97,16 @@ def setup_config(settings_map: dict[str, Any], db: Session = Depends(require_set
saved: list[str] = []
for key, value in settings_map.items():
if sysconfig.spec(key) is None:
raise HTTPException(status_code=400, detail=f"Unknown setting: {key}")
raise refused(400, "unknown_setting", key=key)
if value is None or value == "":
continue
try:
sysconfig.set_value(db, key, value)
except (ValueError, RuntimeError) as exc:
raise HTTPException(status_code=400, detail=str(exc))
except sysconfig.InvalidSetting as exc:
raise setting_refused(exc) from None
saved.append(key)
db.commit() # sysconfig.set_value only flushes now — persist the wizard's config in one commit
return {"saved": saved}
return {"ok": True, "saved": saved}
class SetupTestEmailIn(BaseModel):
@@ -121,9 +121,9 @@ def setup_test_email(body: SetupTestEmailIn, db: Session = Depends(require_setup
"""Send a test email to confirm the SMTP settings the wizard just saved."""
to = body.to.strip()
if "@" not in to:
raise HTTPException(status_code=400, detail="Enter a valid email address.")
raise refused(400, "invalid_email")
if not email_mod.send_test(to, mailtext.normalize(body.lang)):
raise HTTPException(status_code=400, detail="Couldn't send — check the SMTP settings.")
raise refused(400, "smtp_failed")
return {"ok": True}
@@ -133,6 +133,6 @@ def setup_finish(db: Session = Depends(require_setup)) -> dict:
invalidates the setup token and disables the wizard)."""
has_admin = db.execute(select(User).where(User.role == "admin")).first() is not None
if not has_admin:
raise HTTPException(status_code=400, detail="Create the admin account first.")
raise refused(400, "admin_first")
state.mark_configured(db)
return {"ok": True}
+16 -10
View File
@@ -7,10 +7,13 @@ from app.audit import AuditAction
from app.auth import admin_user, current_user, has_read_scope, require_human
from app.db import get_db
from app.models import Channel, Subscription, User, Video
from app.routes._common import youtube_refusal
from app.ratelimit import RateLimiter
from app.sync.runner import estimate_deep_backfill
from app.sync.subscriptions import import_subscriptions
from app.scheduler import running_job_ids
from app.youtube.client import YouTubeError
from app.refusal import refused
router = APIRouter(prefix="/api/sync", tags=["sync"])
@@ -26,17 +29,20 @@ _sync_limiter = RateLimiter(max_events=3, window_seconds=600)
def sync_subscriptions(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
if not has_read_scope(user):
# No YouTube read grant yet — the onboarding wizard hasn't been completed.
raise HTTPException(
status_code=403,
detail="Connect your YouTube account (read access) to import subscriptions.",
)
raise refused(403, "youtube_read_scope")
if not _sync_limiter.allow(str(user.id)):
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
with quota.measured(db, user.id, quota.QuotaAction.SUBSCRIPTIONS_PULL) as m:
result = import_subscriptions(db, user)
try:
with quota.measured(db, user.id, quota.QuotaAction.SUBSCRIPTIONS_PULL) as m:
result = import_subscriptions(db, user)
except YouTubeError as exc:
# The onboarding wizard's first call: an unguarded failure here was a raw 500 and the
# blocking "Server error" dialog, with nothing the user could act on (E9 F1).
db.rollback()
raise youtube_refusal(exc)
result["quota_used_estimate"] = m.used
result["quota_remaining_today"] = m.remaining
return result
return {"ok": True, **result}
@router.get("/status")
@@ -137,7 +143,7 @@ def deep_all(
update(Subscription).where(Subscription.user_id == user.id).values(deep_requested=on)
)
db.commit()
return {"updated": result.rowcount, "deep_requested": on}
return {"ok": True, "updated": result.rowcount, "deep_requested": on}
@router.post("/pause")
@@ -145,7 +151,7 @@ def pause_sync(admin: User = Depends(admin_user), db: Session = Depends(get_db))
state.set_sync_paused(db, True)
audit.record(db, admin.id, AuditAction.SYNC_PAUSE, summary="Background sync paused")
db.commit()
return {"paused": True}
return {"ok": True, "paused": True}
@router.post("/resume")
@@ -153,4 +159,4 @@ def resume_sync(admin: User = Depends(admin_user), db: Session = Depends(get_db)
state.set_sync_paused(db, False)
audit.record(db, admin.id, AuditAction.SYNC_RESUME, summary="Background sync resumed")
db.commit()
return {"paused": False}
return {"ok": True, "paused": False}
+24 -14
View File
@@ -1,4 +1,4 @@
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import func, or_, select
from sqlalchemy.exc import IntegrityError
@@ -9,6 +9,7 @@ from app.db import get_db
from app.models import ChannelTag, Tag, User
from app.routes._common import owned_or_404
from app.sync.tagsuggest import compute_tag_suggestions
from app.refusal import refused
router = APIRouter(prefix="/api/tags", tags=["tags"])
@@ -60,7 +61,7 @@ def create_tag(
) -> dict:
name = body.name.strip()
if not name:
raise HTTPException(status_code=400, detail="name is required")
raise refused(400, "name_required")
# An unknown category silently falls back to "other" (unchanged behaviour) rather than 422.
category = body.category if body.category in ("language", "topic", "other") else "other"
tag = Tag(
@@ -74,13 +75,14 @@ def create_tag(
db.commit()
except IntegrityError:
db.rollback()
raise HTTPException(status_code=409, detail=f"You already have a tag named “{name}”.")
return {"id": tag.id, "name": tag.name, "color": tag.color, "category": tag.category}
def _own_tag(db: Session, user: User, tag_id: int) -> Tag:
# System tags (user_id NULL) are not user-editable → owned_or_404 404s them too.
return owned_or_404(db, Tag, tag_id, user, what="tag")
raise refused(409, "tag_name_taken", name=name)
return {
"ok": True,
"id": tag.id,
"name": tag.name,
"color": tag.color,
"category": tag.category,
}
class TagUpdateIn(BaseModel):
@@ -97,12 +99,13 @@ def update_tag(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
tag = _own_tag(db, user, tag_id)
# System tags (user_id NULL) are not user-editable: owned_or_404 404s them too.
tag = owned_or_404(db, Tag, tag_id, user, what="tag")
sent = body.model_fields_set
if "name" in sent:
name = (body.name or "").strip()
if not name:
raise HTTPException(status_code=400, detail="name cannot be empty")
raise refused(400, "name_required")
tag.name = name
if "color" in sent:
tag.color = body.color
@@ -110,8 +113,14 @@ def update_tag(
db.commit()
except IntegrityError:
db.rollback()
raise HTTPException(status_code=409, detail=f"You already have a tag named “{tag.name}”.")
return {"id": tag.id, "name": tag.name, "color": tag.color, "category": tag.category}
raise refused(409, "tag_name_taken", name=tag.name)
return {
"ok": True,
"id": tag.id,
"name": tag.name,
"color": tag.color,
"category": tag.category,
}
@router.delete("/{tag_id}")
@@ -120,7 +129,8 @@ def delete_tag(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
tag = _own_tag(db, user, tag_id)
# System tags (user_id NULL) are not user-editable: owned_or_404 404s them too.
tag = owned_or_404(db, Tag, tag_id, user, what="tag")
db.delete(tag) # ChannelTag rows cascade on tag delete.
db.commit()
return {"deleted": tag_id}
+16 -14
View File
@@ -2,7 +2,6 @@
recent-first / deep backfill. Runs inside the API process (single worker)."""
import contextvars
import logging
import threading
from datetime import datetime, timezone
from typing import Callable
@@ -10,7 +9,7 @@ from typing import Callable
from apscheduler.schedulers.background import BackgroundScheduler
from sqlalchemy import select
from app import audit, logsink, progress, quota
from app import audit, debugsession, logs, logsink, progress, quota
from app.audit import AuditAction
from app.config import settings
from app.db import SessionLocal
@@ -35,8 +34,7 @@ from app.sync.runner import (
run_subscription_resync,
)
logger = logging.getLogger("siftlode.scheduler")
log = logs.get_logger(__name__)
_scheduler: BackgroundScheduler | None = None
# Per-job run activity, kept in-memory for the admin Scheduler dashboard to read (the
@@ -104,7 +102,7 @@ def apply_interval(db, job_id: str, minutes: int) -> int:
db.commit()
if _scheduler is not None and _scheduler.get_job(job_id) is not None:
_scheduler.reschedule_job(job_id, trigger="interval", minutes=minutes)
logger.info("rescheduled job %s to every %s min", job_id, minutes)
log.info("rescheduled job %s to every %s min", job_id, minutes)
return minutes
@@ -131,7 +129,7 @@ def _notify_done(db, actor_id: int, job_id: str, status: str, summary: str | Non
)
except Exception:
db.rollback()
logger.exception("failed to write completion notification for job %s", job_id)
log.exception("failed to write completion notification for job %s", job_id)
def _run_changed(result) -> bool:
@@ -167,7 +165,7 @@ def _audit_run(db, actor_id, job_id, status, summary, result) -> None:
db.commit()
except Exception:
db.rollback()
logger.exception("failed to write audit row for job %s", job_id)
log.exception("failed to write audit row for job %s", job_id)
def _job(name: str, fn) -> None:
@@ -175,7 +173,7 @@ def _job(name: str, fn) -> None:
# HERE, atomically: whichever reaches this first wins, the other skips. This is the real
# single-run guard — trigger_job's pre-check is only a fast best-effort answer for the UI.
if not _claim_running(name):
logger.info("job %s already running — skipping this trigger", name)
log.info("job %s already running — skipping this trigger", name)
return
db = SessionLocal()
actor_id = _manual_actor.get() # set only for a manual "run now" trigger
@@ -188,7 +186,7 @@ def _job(name: str, fn) -> None:
try:
if is_sync_paused(db):
logger.info("job %s skipped (sync paused)", name)
log.info("job %s skipped (sync paused)", name)
_record(
name,
status="skipped",
@@ -203,14 +201,14 @@ def _job(name: str, fn) -> None:
with progress.bind(sink):
result = fn(db)
summary = _summarize(result)
logger.info("job %s -> %s", name, result)
log.info("job %s -> %s", name, result)
_record(name, status="ok", last_finished=_now_iso(), last_result=summary, progress=None)
if actor_id is not None:
_notify_done(db, actor_id, name, "ok", summary)
_audit_run(db, actor_id, name, "ok", summary, result)
except Exception as exc:
db.rollback()
logger.exception("job %s failed", name)
log.exception("job %s failed", name, extra=logs.event("scheduler.job.failed", job=name))
err = str(exc) or exc.__class__.__name__
_record(name, status="error", last_finished=_now_iso(), last_error=err, progress=None)
if actor_id is not None:
@@ -388,9 +386,13 @@ def _log_gc_job() -> None:
def work(db):
from app import sysconfig
reaped = logsink.gc(db, sysconfig.get_int(db, "log_retention_days"))
days = sysconfig.get_int(db, "log_retention_days")
reaped = logsink.gc(db, days)
# Debug sessions (E16) go with the rows they bundle; an open one past its cap is closed.
closed = debugsession.sweep(db)
sessions = debugsession.gc(db, days)
db.commit()
return {"reaped": reaped}
return {"reaped": reaped, "sessions_closed": closed, "sessions_reaped": sessions}
_job("log_gc", work)
@@ -493,7 +495,7 @@ def start_scheduler() -> None:
scheduler.add_job(fn, "interval", minutes=iv[job_id], id=job_id)
scheduler.start()
_scheduler = scheduler
logger.info("scheduler started")
log.info("scheduler started")
def shutdown_scheduler() -> None:
+23 -3
View File
@@ -1,10 +1,14 @@
import hashlib
from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError
from cryptography.fernet import Fernet
from app import logs
from app.config import settings
log = logs.get_logger(__name__)
# argon2id with the library defaults (sensible memory/time cost). One shared hasher instance.
_ph = PasswordHasher()
@@ -24,8 +28,16 @@ def verify_password(password: str, hashed: str | None) -> bool:
return False
try:
return _ph.verify(hashed, password)
except Exception:
# Any failure (mismatch, malformed hash) is a non-match — never raise to the caller.
except VerifyMismatchError:
return False # silent-ok: a wrong password. The commonest path through this function.
except Exception as exc: # noqa: BLE001 — never raise to the caller, whatever went wrong
# Everything else is a hash this process cannot USE — truncated in the column, written by
# a different argon2 build. It is answered as "wrong password", so the owner of that
# account is locked out by a bug and told they mistyped.
log.warning(
"a stored password hash could not be verified",
extra=logs.event("auth.password.hash_unusable", reason=type(exc).__name__),
)
return False
@@ -60,5 +72,13 @@ def decrypt_optional(value: str | None) -> str | None:
return None
try:
return _require_fernet().decrypt(value.encode()).decode()
except Exception:
except Exception as exc: # noqa: BLE001 — callers fall back to a refresh, never to an error
# Info rather than warning because the docstring's legacy-plaintext case comes through
# here too and is harmless. What is NOT harmless looks identical: a rotated
# TOKEN_ENCRYPTION_KEY turns every stored token unreadable at once, and every account
# then reads as "not connected" with nothing saying the key is the reason.
log.info(
"a stored ciphertext could not be decrypted",
extra=logs.event("security.decrypt.failed", reason=type(exc).__name__),
)
return None
+2 -3
View File
@@ -1,15 +1,14 @@
"""Global admin-controlled app state (e.g. pausing background sync, first-run setup)."""
import logging
import secrets
from sqlalchemy.orm import Session
from app import logs
from app.models import AppState
from app.security import hash_token
log = logging.getLogger("siftlode.state")
log = logs.get_logger(__name__)
# Once configuration completes in this process it never reverts, so cache it to skip the per-request
# DB hit in the setup-mode gate (see main.setup_gate). Starts False; refreshed from the DB.
_configured_cache = False
+10 -5
View File
@@ -6,17 +6,16 @@ topicDetails categories and the channel's dominant video category. System tags a
regenerated freely; user tags are never touched here.
"""
import logging
import re
from sqlalchemy import and_, exists, func, select
from sqlalchemy.orm import Session
log = logging.getLogger("siftlode.autotag")
from app import progress, sysconfig
from app import logs, progress, sysconfig
from app.models import Channel, ChannelTag, Tag, Video
log = logs.get_logger(__name__)
# --- language id (offline, deterministic, small) ---
try:
from py3langid.langid import MODEL_FILE, LanguageIdentifier
@@ -29,7 +28,13 @@ try:
def _restrict(codes: list[str]) -> None:
_identifier.set_languages(codes)
except Exception: # pragma: no cover - fallback to module-level API
except Exception: # noqa: BLE001 — pragma: no cover, fallback to the module-level API
# The fallback classifies without the normalisation the identifier applies, so language tags
# come out subtly different — and nothing else in the app would ever hint at which path ran.
log.info(
"the langid model could not be loaded; using the module-level classifier",
extra=logs.event("autotag.langid.fallback"),
)
import py3langid as _langid
def _classify(text: str):
+2 -3
View File
@@ -8,13 +8,12 @@ stay private to users who explored (or subscribe to) the channel — the feed qu
otherwise the cleanup job hard-deletes the channel and its untouched ephemeral videos after a
grace period (mirrors the maintenance job's grace-then-delete safety)."""
import logging
from datetime import timedelta
from sqlalchemy import delete, or_, select
from sqlalchemy.orm import Session
from app import sysconfig
from app import logs, sysconfig
from app.models import (
Channel,
ExploredChannel,
@@ -28,7 +27,7 @@ from app.sync.videos import insert_stubs, stub_from_playlist_item, apply_video_d
from app.utils import now_utc as _now
from app.youtube.client import YouTubeClient
log = logging.getLogger("siftlode.explore")
log = logs.get_logger(__name__)
def followed_channel():
+2 -4
View File
@@ -20,13 +20,12 @@ Quota: phase 1 costs ~(flagged / 50) units, phase 2 ~(batch / 50). Both respect
backfill reserve as the other jobs so interactive syncs never starve.
"""
import logging
from datetime import timedelta
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import progress, quota, sysconfig
from app import logs, progress, quota, sysconfig
from app.config import settings
from app.models import Playlist, PlaylistItem, Video
from app.notifications import create_notification
@@ -35,8 +34,7 @@ from app.sync.videos import apply_video_details, parse_dt
from app.utils import now_utc as _now
from app.youtube.client import YouTubeClient
log = logging.getLogger("siftlode.maintenance")
log = logs.get_logger(__name__)
# Reasons recorded on Video.unavailable_reason. "removed" = absent from videos.list
# (deleted or made private); "abandoned" = an upcoming premiere that never went live.
GRACE_DAYS_BY_REASON = {
+60 -13
View File
@@ -8,7 +8,6 @@ ingested (with a stub channel) so the mirror is faithful."""
import hashlib
import json
import logging
from collections.abc import Callable
from dataclasses import dataclass
from datetime import datetime, timezone
@@ -17,15 +16,14 @@ from typing import TypeVar
from sqlalchemy import delete, select
from sqlalchemy.orm import Session
from app import progress, quota
from app import logs, progress, quota
from app.auth import has_read_scope
from app.models import Channel, Playlist, PlaylistItem, User, Video
from app.sync.runner import token_users
from app.sync.videos import apply_video_details, parse_dt
from app.youtube.client import YouTubeClient, YouTubeError
log = logging.getLogger("siftlode.sync")
log = logs.get_logger(__name__)
# Cost of a single YouTube write op (playlists/playlistItems insert/update/delete).
WRITE_UNIT_COST = 50
@@ -59,7 +57,17 @@ def _load_baseline(pl: Playlist) -> tuple[list[str] | None, set[str]]:
return None, set()
try:
val = json.loads(pl.synced_video_ids)
except (ValueError, TypeError):
except (ValueError, TypeError) as exc:
# No baseline means the three-way merge has no "what we last agreed on" to subtract, so
# it degrades to a union: nothing is lost, but a delete made on either side comes back.
log.warning(
"the sync baseline for a playlist could not be read; merging as a union",
extra=logs.event(
"sync.playlist.baseline_unreadable",
playlist_id=pl.id,
reason=type(exc).__name__,
),
)
return None, set()
if isinstance(val, list):
return [str(v) for v in val], set()
@@ -410,8 +418,11 @@ class _Breaker:
write site can't accidentally implement two thirds of the protocol (forgetting to record a
failure would silently disable the breaker for that operation)."""
def __init__(self, limit: int = _MAX_CONSECUTIVE_FAILURES) -> None:
def __init__(
self, limit: int = _MAX_CONSECUTIVE_FAILURES, playlist_id: int | None = None
) -> None:
self.limit = limit
self.playlist_id = playlist_id # for the failure line only
self.streak = 0
self.tripped = False
@@ -432,7 +443,9 @@ class _Breaker:
self.limit,
)
def run(self, op: Callable[[], T], on_fail: Callable[[], None]) -> T | _Failed:
def run(
self, op: Callable[[], T], on_fail: Callable[[], None], action: str = "write"
) -> T | _Failed:
"""Attempt one YouTube write. Returns the op's result, or `FAILED` when the breaker is
already tripped (op not called) or the op raised YouTubeError. `on_fail` records the item
in the caller's failure list either way."""
@@ -441,7 +454,19 @@ class _Breaker:
return FAILED
try:
out = op()
except YouTubeError:
except YouTubeError as exc:
# The client logged the call, but not which playlist it was for or what kind of write
# — and the caller's failure list only ever reaches the user as a count. Not the video:
# what is in someone's playlist is a row of their history, the same as a download.
log.debug(
"a playlist write to YouTube failed",
extra=logs.event(
"sync.playlist.item_push_failed",
playlist_id=self.playlist_id,
action=action,
reason=exc.reason,
),
)
on_fail()
self.hit(False)
return FAILED
@@ -460,7 +485,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
local = _local_video_ids(db, pl)
inserted = deleted = reordered = created = pulled = removed_local = 0
failures: list[str] = []
breaker = _Breaker()
breaker = _Breaker(playlist_id=pl.id)
with YouTubeClient(db, user) as yt:
if not pl.yt_playlist_id:
pl.yt_playlist_id = yt.create_playlist(pl.name)
@@ -471,6 +496,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
added = breaker.run(
lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
lambda v=vid: failures.append(v),
"add",
)
if not isinstance(added, _Failed):
inserted += 1
@@ -533,7 +559,17 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
if snippet is not None and (snippet.get("title") or "") != pl.name:
try:
yt.update_playlist_title(pl.yt_playlist_id, pl.name)
except YouTubeError:
except YouTubeError as exc:
# All the caller passes on is the word "title", which says the rename did not
# happen but not that YouTube refused it, or why.
log.debug(
"a playlist rename did not reach YouTube",
extra=logs.event(
"sync.playlist.title_push_failed",
playlist_id=pl.id,
reason=exc.reason,
),
)
failures.append("title")
# Propagate genuine local deletes.
for vid in mp.remove_from_yt:
@@ -543,6 +579,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
dropped = breaker.run(
lambda i=item_id: yt.delete_playlist_item(i),
lambda v=vid: failures.append(v),
"delete",
)
if not isinstance(dropped, _Failed):
deleted += 1
@@ -552,6 +589,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
item_id = breaker.run(
lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
lambda v=vid: failures.append(v),
"add",
)
if not isinstance(item_id, _Failed):
item_id_by_vid[vid] = item_id
@@ -571,6 +609,7 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
item_id_by_vid[w], pl.yt_playlist_id, w, pos
),
lambda w=want: failures.append(w),
"move",
)
if isinstance(moved, _Failed):
continue
@@ -631,11 +670,19 @@ def sync_all_playlists(db: Session) -> dict:
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
total += sync_user_playlists(db, user).get("synced", 0)
except YouTubeError:
except YouTubeError as exc:
db.rollback()
log.warning("Playlist sync failed for user %s", user.id)
log.warning(
"Playlist sync failed for user %s",
user.id,
extra=logs.event("sync.playlists.failed", user_id=user.id, reason=exc.reason),
)
except Exception:
db.rollback()
log.exception("Playlist sync crashed for user %s", user.id)
log.exception(
"Playlist sync crashed for user %s",
user.id,
extra=logs.event("sync.playlists.crashed", user_id=user.id),
)
progress.report(i, len(users), "playlist_sync")
return {"users": len(users), "playlists_synced": total}

Some files were not shown because too many files have changed in this diff Show More