0cff964ff253dd0a659c31fa4dc1dbc106a15430
520
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 peter
|
0cff964ff2 |
refactor(logs): name every logger after its module, and hold it there
25 literal names covered 34 modules. plex/client.py, plex/stream.py, plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves `siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`; routes/channels.py called itself `siftlode.api`. The `logger` column is in every stored row so E22.6's console can filter on it, and so an operator can pin a level to one part of the system — neither works when six modules share a name. `logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync. One name per module, and it cannot drift from the file it sits in. tests/test_logger_names.py pins the derivation, that everything stays under the one `siftlode` tree the handler and the tests attach to, and that no module hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`). Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic it carries rather than the module it lives in — it replaced uvicorn's access log and is what an operator pins when they want request lines and nothing else. app/scheduler.py called its logger `logger`; every other module calls it `log`. ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them: worker.py logged `str(exc)[:120]` where the traceback was the whole question, and three prints/f-strings that are deliberate now say so on the line. S110 and BLE001 stay off until S3 — 23 handlers are outstanding and each is a judgement about one of them; enabling the rules first would just mean 23 blanket noqas. |
||
|
peter
|
dc221338be |
fix(logs): keep what a person typed out of the backend log lines
The E22 redactor is a safety net over addresses and credential params, and every one of these call sites was leaning on it to do the whole job. What it does not cover went in whole, and what it does cover came out as a constant. Not covered, so a real leak: - routes/search.py logged the raw search term at ERROR. The privacy policy shipped in 0.88.0 says a search string is kept only at Teljes/TRACE, with the user's own consent. The ERROR line now carries the shape of the failure (source, candidate and collected counts) and the term follows on a TRACE line, which nothing emits unless that consent is in force. - downloads/backfill.py logged the full source URL at INFO — a row of the user's download history. Now the asset id and the host. - youtube/client.py logged the upstream error body, which is prose that quotes the request back. Every failing YouTube call now goes through one _upstream_failure() that logs the API's `reason` code, keeps the body for the same consented TRACE line, and raises a YouTubeError carrying `reason` as a field — callers put that error in log lines of their own (routes/channels.py) and in a 422 answered to the user, so the body must not travel in it either. Covered, so merely useless: routes/auth.py named the account by an address that reached both sinks as `[email]`, and app/email.py printed recipient lists that read as `['[email]']`. Both now carry the uid (or a count) and a reason code, which is also the first half of E22.5 S4's failed-sign-in coverage. routes/channels.py branched on the text of that error message to recognise an already-existing subscription. It now branches on `exc.reason`, with the substring check kept beside it for a body that carried no reason at all. tests/test_log_pii.py pins the call sites and the family: a source-level contract test walks every log call in app/ and fails on an argument naming user-typed content unless the line says why it is safe (`# log-pii-ok: <why>`). Eight sites carry that marker today, each with its reason. Gate 8/8 green; all five cases mutation-checked. |
||
|
peter
|
4ab128ccab | Merge branch 'main' into feature/e22-activity-trail | ||
|
peter
|
07e3ea3737 |
docs: the deep-backfill card counts the backlog, not the next pass
Two reviewers in a row read the admin card's selection against `run_deep_backfill`'s, saw it was missing the job's `recent_synced_at` filter, and called it a card that counts channels the job will never touch. It is the opposite: the card is the same backlog figure /api/sync/my-status shows the user, and a channel still missing its recent pull is one the same job tick fetches first. Narrowing it would hide genuinely queued work. Written down where the next reader will look. |
||
|
peter
|
b2eaa9e00b |
fix: a credential, an erased account and a stale level stay out of the log
Round 2 of the E22 review, on what round 1 changed. `redact_query` decodes a parameter before the text rules see it, which is what lets `%40` be recognised as an address — but `_SECRET_PARAM_RE` was written for the raw, encoded form, where a value holds no space, `&` or quote. On decoded text its value class stops at the first one, so `?state=a%20b` kept its tail and `?code=%20abc` was not matched at all. The parameter-NAME rule now decides, on the key, before any text rule runs. `forget_user` dropped the account's level elevation without recomputing the root level, so a process whose only elevated account had just deleted itself kept building TRACE records to throw away, until some later request happened to recompute it. Every other path that drops an elevation recomputes. The sink's writer took the erasure lock and THEN waited for a pooled connection, while every request path already holds a connection when it reaches that lock — so under pool pressure the writer held a process-global lock against the very requests that could hand a connection back. Same order on both sides now; the lock still spans the check, the insert and the commit. The storage-budget notice said the rest of the hour was not kept, and then smaller batches kept landing underneath it: the window slides and a refused batch is not charged to it. Says what it does. Tests: the route's half of the erasure rule had no test — only the sink's did — and a source-level contract test now covers the third insert nobody has written yet. `logs._erased` is process-global with an hour's memory while the DB fixture hands serial ids back out from 1, so it is swept before each test like the rate limiters are. |
||
|
peter
|
4bfba4aa86 |
refactor: one definition of a followed channel and of a choice spelling
The explore purge rebuilt followed_channel's two EXISTS clauses inline, so what is polled and what may be deleted could drift; it now uses the predicate, which moved to sync/explore.py. The env validator and sysconfig each folded a choice value on their own; both call config.canonical_choice. The deep-backfill comment no longer calls recent backfill unconditional. |
||
|
peter
|
86bc008a81 |
fix: a crashed job's crash line carries its job id
The bind sat inside the try, so it was already reset when the crash line and the cleanup were logged — the one line most worth finding by job id had none. |
||
|
peter
|
fac5c4465a |
fix: stored logs keep bad rows, erased accounts and unconsented queries out
Deep-review round 1 of E22. - A NUL character (`GET /assets/%00` logs the decoded path) made Postgres refuse the row and with it the sink's whole batch, every user's rows included. `to_row` replaces NUL in every text column and in attrs. - An erased account's lines kept arriving after purge_user's DELETE: the erasing request's own access line, a queued row, a background task. `logs.forget_user` runs first; later rows are stored without the account, and the check and the insert share a lock with the erasure. - An admin setting the whole process to TRACE logged every user's query string. The query is now logged only under the user's own Teljes consent, and decoded before redaction, so `%40` no longer hides an address. - The browser log ingest has an hourly storage budget per account (5 MiB, one notice when spent) and reads only a JSON Content-Type, so a same-site text/plain form post cannot write into a user's log. RateLimiter takes a per-attempt cost for the byte budget. - The root level is applied under the elevation lock: two concurrent changes could leave it at the base while an elevation was active. - Tests for the running sink (level gate, drain on stop) and for the admin log level applied at runtime and at boot. |
||
|
peter
|
5883ea509d |
feat: Settings → Diagnostics, a consented log level and the tab's own log
A user can raise their diagnostics level to Részletes (DEBUG) or Teljes (TRACE) for 24 hours from a new Settings tab, after a consent box that says what is recorded, that the server keeps it and for how long (the instance's own retention), and that it lapses. Alap is the base and needs no consent. - users.diagnostics_level/_until (migration 0087), written only by PUT /api/me/diagnostics, which computes the expiry itself; the demo account cannot raise it. /api/me reports the level in force. - The server logs an elevated user's requests at that level and stores every line of them (logs.LevelGate on the stdout handler and the sink; the root is lowered only while an elevation is active). Teljes adds the redacted query string to the request line. - POST /api/diagnostics/events stores the browser's log as source=web rows: account from the session, body capped while streaming, events/attrs capped, re-redacted, events under the consented level dropped, timestamps clamped, 30 batches a minute, 409 once the level lapsed. - lib/diagnostics.ts applies the level to the tab and streams only what happens after consent, in batches, never its own request lines, with a keepalive flush on pagehide; stops on lapse (its own timer), on withdrawal and on 409. - Detail hooks at DEBUG: which control was clicked (test id, icon, app copy in English — never user text, input values or a link's query), visibility/bfcache/resize, filter changes (lib/filterLog.ts; typed search only at TRACE), player events (lib/mediaLog.ts; media errors are warnings at every level). Offline/online at the base level. - "Saját napló" viewer: live, newest first, level groups with counts, search, row details, copy the filtered events as JSON lines, clear. - Privacy policy and Terms exist in Hungarian and English (a contract test holds both languages to the same sections); the policy now describes the server logs and diagnostic logging, self-service deletion and password sign-in. Fixed on the way: - A sync auth dependency ran logs.set_user on a copied context, so the account never reached a handler's own log lines (measured); a per-request mutable scope carries it now. - The Settings tab rail left a phone no room (clipped labels, buttons over text): below `sm` it is a scrolling row above the content. - PUT /api/me/preferences took a bare dict body; it is a Pydantic model now. |
||
|
peter
|
a210992bfc |
feat: client log with request ids, joined to the server's
The browser now keeps its half of the one event shape logs.py defines: lib/log.ts is a levelled, redacted, bounded (2000 events) in-memory ring with the server's level names and severity numbers. Nothing leaves the browser yet; sending is the consented diagnostics level (S5). - lib/http.ts `send` is the one fetch: every request carries an X-Request-Id and becomes one http.request event. The server echoes the matched route template in X-Route (only for a matched route, so a raw path with a value in it is never sent back); without one the client templates the path itself. req, beacon, the upload XHR, the public watch page and the legal pages all go through it. - Hooks: app start snapshot, uncaught errors and rejections, History push/replace/popstate (a scroll-only replace is not an event; parameter values only at DEBUG, typed search only at TRACE), notifications, the error dialog, the crash boundary, failed queries and mutations (key head only), and the messages socket (lifecycle, frame types at DEBUG). - Redaction is one contract: backend/tests/fixtures/log_redaction_cases.json runs against both redact() implementations. - Lint: fetch, XMLHttpRequest and WebSocket each belong to one logging module, console to none; a contract test proves the rules fire. Fixes a socket takeover: the close of a socket released by the last subscriber cleared the newer socket a quick re-subscribe had opened and scheduled a third, leaving two live sockets that delivered every message twice (measured in Chromium: 3 created / 2 live before, 2 / 1 after). |
||
|
peter
|
0517de1f59 |
fix: poll and backfill only channels somebody follows
The scheduled RSS poll and recent backfill covered every channel row, but rows also come from search results, playlist sync and dropped subscriptions, and nothing deletes them. On prod 582 of 979 polled channels had no follower, three of them dead feeds answering 404 on every poll. Both jobs now take subscribed or explored channels only, and the scheduler card counts the same set. As a deliberate trade-off, the shared Library no longer receives new uploads from channels nobody follows. |
||
|
peter
|
b59fd38a9c |
feat: store warnings and request lines in log_events
The log now has a stored, queryable copy for the diagnostics console: every WARNING and above plus the request line of every request, kept for log_retention_days (default 7, no "forever") and pruned hourly by the new log_gc job. Rows are built on the logging thread from the same redacted fields as the stdout line, queued without blocking and written in batches by one background thread on its own connection. A full queue or a failed write drops and reports instead of retrying. Flood control keeps 30 of any one message per minute plus a count of the rest; anonymous request lines are capped at 60 a minute per client, and the client address is never stored. Deleting an account deletes its stored events. Measured on prod first: a YouTube RSS outage wrote 11k warnings in four hours, one per channel per poll. The poll now writes one warning with per-reason counts and a few examples, and a failure on our side gets one traceback per poll instead of one per channel. |
||
|
peter
|
07766c43e8 |
feat: closed-set admin knobs are pickers, not free text
log_level, search_source, download_layout and the transcode hw/preset were typed by hand, and every consumer carried its own guess at what a typo meant. The value set is now a Literal in config.py: pydantic rejects a bad env value at boot (any letter case accepted), sysconfig refuses a bad override and derives the admin picker's options from the same Literal, and the consumers read the value as-is. The Configuration page renders a choice key as a segmented bar, or a dropdown past six options, with translated option names. Its tabs now follow the registry's order instead of a hand-kept copy that had already dropped the explore group. A single-choice FilterSelect now opens on the option in use: in the browser it could land on the first row instead. |
||
|
peter
|
2b56c5c2eb |
fix: name the error yt-dlp actually raises for a cookie-emptied client set
A review round read yt-dlp's source and found the comment behind the previous commit's safety-net marker did not match it. Measured in the worker with a real account cookie and `player_client=android_vr`: yt-dlp logs `Skipping client "android_vr" since it does not support cookies`, drops it, and the request left with no client fails with "Requested format is not available" — not "No player clients have been requested". That raise sits EARLIER in yt-dlp, before the cookie-capability removal, so the cookie path can never reach it; and an unknown or renamed client is skipped with a warning and replaced by yt-dlp's defaults, so the other scenario the comment named cannot reach it either. So the marker bought nothing: the symptom that does occur was already retriable here before this branch. It is dropped, the comment now states what was measured, and the test asserts the real symptom instead of a message the code cannot produce. The cookie-capability filter itself is vindicated by the same measurement — without it every signed-in download would burn its first attempt on a client yt-dlp discards. Also from the round: an enqueue test for a `source_kind="youtube"` job, the one join nothing else made (a YouTube job stores a bare 11-char ref and the URL the cookie gate keys on is rebuilt by the adapter — both ends were pinned apart, the join was not, and a break there would stop every YouTube cookie silently while signed-out downloads kept working). And copy corrections: the group's only unhedged claim is hedged, "described the video" loses the pipeline vocabulary, and four Hungarian phrasings that read as translated English are rewritten. |
||
|
peter
|
c06a456a9a |
fix: a YouTube stream refused after extraction is the gate, not a network fault
Measured on the isolated test stack the same night, on the very same video ~90 minutes after it was answering with the bot message: yt-dlp got the player response and picked its formats, and the CDN then refused the stream itself — "unable to download video data: HTTP Error 403: Forbidden". Signed out it did that on every attempt; with the owner's cookie attached the video downloaded in full. So the wall moves between the player response and the stream, and the text it wears there was costing the download twice. retriable: the format URLs belong to the client that minted them, so the next client set is a real second chance — but this text classifies as `network` and nothing in the YouTube marker list matched it, so the ladder ended the job on attempt 1 and never tried the fallback. catalog: `youtube.media_forbidden` for the cookie-less case, so the UI stops calling it a network error when nothing was wrong with the network, and names what actually gets past it. Scoped exactly as narrowly as the measurement: another host keeps the generic `network` category, and so does a download that already carried a cookie — telling a signed-in user to sign in is the failure the catalog exists to avoid. EN + HU copy. |
||
|
peter
|
864f673ef5 |
fix: a YouTube video behind the bot wall is downloadable with cookies
A YouTube link can fail with "Sign in to confirm you're not a bot" while every other YouTube download works — YouTube gates INDIVIDUAL videos behind a signed-in session. Measured on prod job 273: all twelve of yt-dlp's player clients fail on that one video while its neighbours download fine, the bgutil PO-token sidecar is live and answering, and the newest yt-dlp fails identically. So the download is not broken by our stack or a stale yt-dlp, and no retry, client switch or upgrade gets past it. A cookie is the only way in. The failure copy already sends the user to Settings -> Account -> Download authentication, but YouTube was not in the cookie host registry, so that dropdown could not help them: the app asked for something it did not offer. credentials: YouTube joins SUPPORTED_HOSTS (youtube.com, youtu.be, youtube-nocookie.com). The Settings dropdown, the SEC-12 host scope and the private-asset cache salt all derive from the row, so that is the whole wiring. adapters: a signed-in YouTube download runs only cookie-capable player clients. Measured against yt-dlp's own INNERTUBE_CLIENTS table: with account cookies attached yt-dlp REMOVES every client whose SUPPORTS_COOKIES is False, and a set it empties that way raises "No player clients have been requested" — which is not a per-client flake it retries past. Our primary set is android_vr, one of those, so without this the cookie would have failed the download on attempt 1 and never reached the POT-backed web fallback it exists for. A test pins the client list against the installed yt-dlp so a version bump turns the gate red instead of silently un-authing every YouTube download, and the empty-set message joins the retriable markers as the second line of defence. errors: catalog entries for the wall with and without a cookie, so a user who has already added cookies is told they were rejected (YouTube rotates exported ones) rather than to add some. The marker is quote-agnostic: YouTube writes the apostrophe as U+2019, so "you're not a bot" with a straight quote never matches the real text. EN + HU copy. |
||
|
peter
|
6c63df6704 |
feat: one structured log for every process, with a request id
A deployment printed four different log formats and none of them could be joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in main.py for anything not started by uvicorn, logging.basicConfig() in the worker and in the download child (no timestamp at all), and alembic's fileConfig. app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log data model, no SDK): ts, level, severity, source, logger, event code, message, attributes, plus the correlation the context carries — request id, account, download job. Redaction is applied at the HANDLER, not at the call sites: every message, every attribute and every traceback is scrubbed on the way out, so a secret cannot leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies, secret config values and email addresses never reach a line. The access log is ours now (`http.request`): the route template instead of the concrete path, the status, the duration and the account — and never the query string, which uvicorn's access log printed on every single request (`?account=`, `?source=`, the `?token=` of a share link). A well-formed client request id is accepted and echoed in X-Request-Id, so a browser line and a server line join up. The account comes from `request.state`, filled by the auth dependency: the middleware is the outermost one, so it runs outside SessionMiddleware and in a different task context. `log_level` becomes an admin knob (sysconfig), applied without a restart and re-applied at startup — best-effort on purpose, since a logging knob must never be what stops the app from booting. Every compose service gets bounded json-file logs (3×50 MB): Docker keeps every line forever by default, and the API now writes one line per request. Verified on the localdev stack: JSON everywhere including alembic and the worker, the query strings gone, a supplied X-Request-Id echoed and logged, and three /healthz hits of which only the one made while the level was DEBUG produced a line. |
||
|
peter
|
11aef4ab7a |
feat: record when an account last signed in
users.last_login_at (migration 0085, no backfill) is stamped by _establish_session, which the Google callback now shares with password login instead of repeating its session setup inline. The demo entry does not stamp its shared account. The admin user list carries the field. |
||
|
peter
|
47d968eeea |
fix: review round 4 — an account switch unloads as the old account, a share link's lockout costs five networks
Deep-review round 4, targeted at round 3's changes (f569b19c..e7f38396). Accounts - Switching, adding or signing out an account moved the tab's pin before the page unloaded, so the players' resume beacons fired on the way out under the NEW account: one account's playback position saved into the other. The pin is now handed to the next page (pinOnNextPage) and taken over at import; clearActiveAccount is gone. - Switching to a suspended account was refused but left it in the switcher, refused on every click; a deactivated one was not refused at all, and made the default it signed every account in the browser out on the next request. Every account _session_user would refuse — gone, deactivated, suspended, revoked — is refused and dropped from the list. Share page (/watch) - An IPv6 client is limited by its /64 (client_ip becomes client_key, for every limiter): keyed on the full address, one subscriber could rotate through endless identities. The comments now state what the per-link ceiling costs: guessing from five networks can keep a link shut for its window, and guesses already running when it fills are still checked. - The link is looked up before the limiter records the attempt, so made-up tokens no longer grow it by an entry each. Forms - Failing the same way twice is announced again: the sign-in, passphrase and setup errors are keyed on the attempt (a clear and a set in one handler left the alert node untouched). - The install wizard's own validation showed the generic error — it threw plain objects its catch did not read. It sets the step's message. Tests: a re-pin leaving the previous account's notices behind; the pin hand-over; switching to an ended account; the /64 viewer; made-up tokens; the repeated alerts. Each mutation-checked RED for its own reason. |
||
|
peter
|
e7f38396a2 |
fix: review round 3 — a new tab keeps its account's notices, a revoked tab account lands on the default
Deep-review round 3 over main..improvement/watch-background-audio. Account boot - The notification history and settings, and the hints toggle, are read when their modules load, which in a new tab (bookmark, typed URL) is before BootGate pins the account: they read as nobody's, and the tab's first notice replaced the account's stored history (up to 100 entries) with a one-item list. They now read again when the tab is pinned (onAccountPinned, run by setActiveAccount); a notice raised before the pin stays, after the history. - A tab pinned to an account still in this browser's wallet but whose session ended (suspended, deleted, or revoked by "log out everywhere") got "signed out" from /api/me while the default account was still signed in, and signing in again as anyone else left it there. A refused per-tab account now leaves the wallet, and the probe answers as the default, so the tab reloads onto it. Share page (/watch) - The password unlock is limited per viewer (link + client IP), under a per-link ceiling that counts only wrong passwords. Keyed on the link alone, ten wrong guesses every five minutes locked out every viewer, the ones with the password included. client_ip moves to app.ratelimit so the public routes can key on it. - Its failure messages and the load-failure screen are announced (role=alert, the password input described by its error). The sign-in, passphrase, password-change and setup form errors get role=alert too. Comments that still described removed machinery are corrected: App's collapse flags, SettingsPanel's Save bar, ConfigPanel, authed_format_sig. Tests: the account's history and hints across the pin; a revoked and a suspended pinned account; the per-viewer limit and the ceiling; the limiter's exhausted(); the share page's announced failures; a share link's feed filters persisted. Each mutation-checked RED for its own reason. |
||
|
peter
|
f569b19c49 |
fix: review round 2 — a signed-out tab acts as nobody, honest share-page failures
Deep-review round 2 over main..improvement/watch-background-audio. Account resolution (server) - A request naming an account that is no longer signed in to this browser (X-Siftlode-Account or ?account=) now acts as nobody instead of the session's default account. A tab still pinned to an account signed out in another tab went on acting as the default: its player checkpoints and preference saves landed in that account, the messages socket and Plex playback ran as it, and its Sign out signed THAT account out. A hidden tab never polls /api/me, so a tab playing in the background kept doing it. The /api/me bootstrap probe alone keeps the fallback, which is how such a tab learns who it now is: the app's 401 handler reloads, and BootGate re-pins the tab. BootGate - On an identity change the tab reloads without moving its pin first. The tree being torn down still runs its unmount saves before the page goes, and with the pin already moved they wrote into the new account's local keys (a player volume still pending). The reloaded page corrects the pin. Share page (/watch) - Only a 404 says the link is gone. A 5xx, a dropped connection, and the unlock's own rate limit all rendered "no longer available ... expired or revoked" to a viewer with a valid link, with no way to try again. A load failure now offers Retry; too many password tries says so and keeps the form; an unlock that gets no answer says so instead of doing nothing. Strings in en/hu/de. Downloads - A cookie-authed asset is recognised under the current signature version and the one before it, so a job queued before a deploy that bumps the version still downloads with its cookie instead of failing on the login wall with an error that says no cookie was used. The app chunk is preloaded through lazyWithPreload, so a failed fetch no longer leaves an unhandled rejection beside the ErrorBoundary. The panel layout seed is one helper (seedLayout). Tests: the stale-tab refusal, its harmless Sign out and the probe fallback; the pre-bump authed asset; BootGate keeping the pin on reload and staying mounted through a same-account change; the reveal clock's later-commit hand-off; the prefs, panel layout and demo-scope seeds from the me cache; the share page's failure states. Each mutation-checked RED for its own reason. |
||
|
peter
|
6169e32ca5 |
fix(downloads): a signed-in Facebook reel keeps its title, and an mp3 stays an mp3
Since a Facebook cookie was added on prod, every reel came down titled "Video": measured on dev, a signed-in read gives yt-dlp the placeholder "Video" (an 18+ reel read as Chrome gives "Facebook") while the post's text is still in the description. A new SourceAdapter.title hook takes the whole first line of the description in that case, and both title writes (the early fill and the finished download) go through it, so the row, the file name and the share preview carry the post's text again. The line is not cut with an ellipsis: normalize_title's NFKC turns "…" into "..." and collapses it to ".", which read as a finished sentence (seen in dev UAT). The "Audio (MP3)" profile's extracted .mp3 was remuxed into an .mp4 by the browser-safe step (the audio spec is clamped to container mp4 too) and served as video/mp4, which iPad's WebKit will not play (prod job 268). Audio mode now skips that step, so an .mp3 is served as audio/mpeg and an .m4a as audio/mp4. Audio downloads version their cache identity on their own (v5), so a cached remuxed audio re-derives without re-downloading every cached video. |
||
|
peter
|
2a9ddb78ac |
fix(downloads): E11.5 epic-end review round 2
Pause now re-reads the job under the same asset-then-job row lock as resume, cancel, delete and redownload. Its unlocked write waited for a concurrent cancel's lock and then set the released job back to paused, a holding status, so a later delete released the same hold again and could free a file another user's job still held (measured: the overlapping-request test ends 'paused' instead of 'canceled' with the old read). The lock test now calls the route handlers, so it also pins which helper each route uses. Comments follow the code: resolve_url's contract names both verdicts and that a failed page read is raised as itself; the inherited-failure diagnosis asks the cookie gate as it stands when the job is claimed, so its advice fits the cookie that job's Retry would use; the undecryptable-cookie warning no longer claims a download goes ahead, since the worker parent reaches it only to diagnose. |
||
|
peter
|
e887233210 |
fix(downloads): E11.5 epic-end review round 1
A refused Facebook cookie on a group post is no longer "no video in this post":
measured on dev, Facebook answers with its login page (no post id at all), so the
adapter now raises its own POST_NOT_SHOWN verdict and the catalog explains it as
the refused cookie. The adapter owns both post verdicts and the "cannot parse
data" wall wording, so the Chrome retry and the catalog read one constant. The
group-post "no formats" entry is cookie-scoped (with a cookie yt-dlp only gets the
post's video link) and uses the adapter's own group-post test instead of a
"/groups/" substring.
Job transitions that move an asset's ref count (resume, cancel, delete,
redownload) re-read the job under a row lock, asset first, so two overlapping
requests can no longer release the same hold. get_or_create_asset loses an insert
race inside a SAVEPOINT instead of rolling back the caller's pending writes (Retry
lost its cleared error). Every worker-side job failure goes through
finalize_if_running and writes error_detail, so a cancel that landed first is kept
and a failure never shows an earlier one's copy. The worker asks the download
child's own cookie gate (credentials.usable_cookie) when a job inherits a failed
asset.
UI: the Facebook copy no longer points at a Retry button the failure window does
not have, says "paste" like Settings, and the HU wording is fixed
("Újraletöltés" on the re-download confirm). A category this client has no copy
for falls back to the generic sentence instead of a raw key.
Tests: route serialization of error_detail, the probed duration through a real
_download, the page-read cap and a failed read, a Retry off a pending asset, the
confirm-button labels, the lock and savepoint races.
|
||
|
peter
|
90ee34a1b4 |
feat(downloads): a Facebook group post downloads its own video, or says it has none
Given a group post's link, yt-dlp either found no formats or took another video from the page's feed data, so a "successful" retry of such a post was a wrong download. With the user's cookie the Facebook adapter now reads the post page, takes the single video node in the script blocks that carry the post id, and downloads that video's watch link (the Chrome retry does the rest). No single video in the post fails with its own message and the group-post copy, reworded to say the post has no video; the post link is never handed to yt-dlp as a fallback. Without a cookie nothing changes. |
||
|
peter
|
118fc70e08 |
fix(downloads): take the duration from the file when yt-dlp gives none
A Facebook or Instagram reel extracts with no duration, so the asset kept a NULL duration_s although the file plays for minutes, and the Library row had no trim action. The finalize probe now returns the duration with the codecs (the file wins, info fills a gap), and runs before the sidecars so the .nfo runtime gets it too. |
||
|
peter
|
1354f5b503 |
feat(downloads): a Facebook cookie download stuck at "Cannot parse data" is retried as Chrome
Measured on dev: with the user's cookie some reels still answer "Cannot parse data" and only extract when the request impersonates Chrome. Impersonation is a second attempt behind that one error, only when a cookie is attached — links that already download with the cookie keep their plain path, and without a cookie the same error is the login wall. curl_cffi is pinned inside yt-dlp's supported range (0.10-0.15; outside it every target silently reports "unavailable"); the lock only gains the new packages. A test fails the gate if the locked pair can no longer impersonate. |
||
|
peter
|
98ec1ffcab |
fix(downloads): Retry and Re-download take a cookie added after the failure
The login-wall copy tells the user to add their cookies and press Retry, but Retry kept the job on the shared asset the worker never attaches a cookie to, so it failed on the same wall. Resume now moves a pending/errored job onto the asset enqueue would pick with the cookie (never off a file or a download in flight). Re-download salted its fresh asset with the job's previous asset id, which the cookie gate cannot recompute, so even a cookie download rebuilt without the cookie. The rebuild salt is now the new asset's own id (plus the user's cookie salt when one applies), and is_authed_asset recognises it. |
||
|
peter
|
18c11ca437 |
feat(downloads): a per-site failure catalog — Facebook's walls say why and what to do
A failed Facebook download read "Download failed": yt-dlp words the login wall (18+ reel, closed group) as "Cannot parse data", which no category matched. Measured on dev with the owner's cookie before writing a rule — the same text is the login wall without a cookie and a refused cookie with one. - errors.diagnose(msg, url, cookie_attached) tries a host-scoped catalog first (all-of markers, cookie state, link fragment) and falls back to the category. First entries: facebook.login_wall, facebook.cookie_rejected, facebook.group_post_video. The module doc is the recipe for the next entry. - download_jobs.error_detail (0084) + backfill of the stored Facebook parse failures that were unknown. - The pipeline's failure branch passes the link and whether the cookie went with it; resume and redownload clear code and detail (redownload never cleared error_code). - A job meeting an already-failed asset got the text but no code, so it read "Download failed" where the first job said why. It is diagnosed again, with the cookie state from credentials.is_authed_asset (now shared with the cookie gate); the redacted network text classifies back to network. - FE: failCopyKeys gives the row and the failure window one lookup — catalog copy, else the category. Copy in EN and HU; a contract test pins every backend entry to both locales. |
||
|
peter
|
f130a1c05a |
fix(keys): epic-end review round 1 — Back, menus and dialogs keep the focus where it belongs
Deep-review of E11 (19 raised, 14 confirmed; 2 refuted, 3 latent test-hardening notes dropped). - Back from a sub-view (a Plex show page, a Messages thread) restores the item it was opened from even when focus was still inside the view: the first attempt now waits one task, past the sub-view's own popstate listener (measured in Chrome: it left focus on <body>). - The restore listener is installed at load, so the first Back after a reload finds it; a push cancels a restore still waiting for the last landing's lists. - Navigating over an overlay asks it again, a render apart, while its entry stays: the Plex player peels one panel per Back step, so Shift+W over info + stats closed one panel and dropped the navigation. - j/k: a native <button> item keeps its own Tab stop; landing twice on the same item no longer leaks its tabindex; after the focused item is removed (a hidden card) the next j goes on from its slot when that slot is on screen. - An abandoned `g` no longer swallows a modifier chord (g, then Mod+K opened the browser search). - A hover card above a dialog no longer switches its Tab trap off (player, Modal, wizard, Lightbox): new layerStack `anyBlockingLayerAbove`. - "Manage views…" closes its menu before opening the dialog, through a shared `closeThenAct` that ActionMenu now uses too. - Copy: the EN welcome says "Feed"; the EN unreachable reason no longer claims a change; stale comments about PlayerModal's scroll lock and dialog contract. |
||
|
peter
|
bbd31c7143 |
feat(keys): Back returns focus to the item a view was opened from; "Feed" everywhere; every backend id named
Owner answers to the E11 S5 raises (2026-09-13): 1 → A, 2 → "Feed" in every language, 3 → translate the
proxy label and fix every config key missing in BOTH languages ("maintenance", "tmdb" — old bugs).
Focus across a view change (1-A). Opening a Plex title from Home's rail, a channel from the Channels
table or a title from a Plex grid switches the page or the sub-view, which unmounts the list; Back used
to hand focus to <body>, so the next `j` started over at the top. The YouTube player (an overlay)
already restored onto the card.
- `lib/listNav`: `navPosition()` (list's place in reading order, the page's list count, item index)
and `restoreNavPosition()`: focuses the item once the page has as many lists on screen as when it was
left, polls on a timer (a hidden tab runs no animation frames), gives up after 2 s or at once when
something already holds focus, focuses without scrolling (the scroll restore owns the position).
- `lib/history`: `rememberNavPosition()` stamps `sfNav` onto the entry being left and returns the state
without it (a new entry must not inherit it); one popstate listener restores and spends the stamp,
so a later landing on the same entry (an overlay closing over it) cannot pull focus to an old item.
- Called before every view push: `setPage`, `openChannel`, `enterYtSearch`, `useHistorySubview.open`,
the Channels tab push. `navPosition.wiring.contract.test` fails on any `history.pushState(` without
it (the overlay stack's own push is the one exemption).
"Feed" (2). The module is "Feed" in both languages: every "hírfolyam" in the Hungarian UI (≈45 strings,
suffixed: Feedből, Feeded, Feedben…) and the prose "feed" in English that means the module ("your
Feed", "Hide from Feed"); "RSS feed" keeps its meaning. The welcome email says it too.
Backend ids without a name (3). The Configuration page showed `maintenance`, `tmdb`,
`maintenance_revalidate_batch`, `download_upload_max_bytes`, `audit_retention_days`, `plex_overlay_map`,
`tmdb_api_key` and `tmdb_language` as raw identifiers — the lookup falls back to the id, and the i18n
parity test stayed green because both languages missed the same keys. Added labels + hints (EN/HU),
the two groups (and their tab order), the missing `scheduler.jobDesc.demo_reset`, removed the stale
`plex_sync_interval_min` field, and "YouTube API kimenő proxy". `i18n/backendIds.contract.test` reads
`sysconfig.SPECS`, `scheduler.JOB_INTERVALS` and the audit/quota action enums from the backend sources
and fails on an id without its name in either language, or a name left for an id the backend dropped.
|
||
|
peter
|
8c51062e5b |
fix(api,ui): the demo refusal says why at every site, and the contract guard cannot fail open
Round 6 converted three of four `is_demo` 403s to a structured detail and wrote a docstring saying it
had converted all the sites there are. The account-delete route in `routes/me.py` still answered a
hand-written English sentence, which `notifyActionError` discards in favour of the caller's
translated fallback — so the demo user gets no hint that the demo account is the reason. It is
UI-gated today, so the string is unreachable; what was reachable was the false completeness claim,
which is precisely what stops the next reader from looking. The site is converted, and the list is
now DERIVED from the source: a new test walks `app/` for every `is_demo`-guarded 403, asserts each
raises a structured detail, and fails when the count moves — so this file cannot fall behind the code
the way it just did.
The same round's frontend contract guard was failing open. Its pattern required a bare identifier
before `.detail` and excluded `.detail` used as a last call argument, so `setErr(e.detail)`,
`(e as HttpError).detail`, `String(e.detail)` and four more idiomatic forms were invisible — and
because the file set is compared with `toEqual`, a file the pattern cannot see is simply never listed
and the gate stays green over a fresh copy of the contract. Widened to cover cast receivers, optional
chains and call-argument reads; the pinned set is unchanged, so the narrowing had removed no false
positive. A destructured `detail` stays out of reach (a pattern loose enough for it fires on
react-query's own `const { data: detail }`), and that hole is written into the test rather than
papered over with two innocent files.
Its anti-vacuity guard also did not guard anything: the comment says the glob must have matched
something, but the assertion under it read a sibling file by its own URL, which resolves whether or
not the glob enumerated anything. That only mattered while the pinned list was non-empty — the moment
the deferral sweep lands and it becomes [], the whole file would pass over a glob matching zero
files. The scan size is asserted now.
|
||
|
peter
|
71d7646bf1 |
fix(plex): a name this session cannot write fails at once, and the layout has one spelling
Round 6 turned `artifact_may_arrive` into a positive rule, but keyed it to the HLS vocabulary in
general rather than to what THIS session produces: the playlist regex accepted master/index/stream_*
for every session and the `seq == 0` branch accepted any variant token unconditionally. The two argv
branches are disjoint — a single-audio session is given `seg_%d.ts` + `index.m3u8` and no `%v` at
all, so `master.m3u8`, `stream_*.m3u8` and every `seg_<variant>_<n>.ts` are names it can never write,
and vice versa — so each of those was read as "an HLS name, so wait and see" and held a threadpool
worker, plus its pooled DB connection, for the full timeout. That is verbatim the round-5 outage the
route's own comment claims it prevents.
Measured through the real route against a live single-audio session, before: master.m3u8 404 after
30.0s, stream_q7z.m3u8 404 after 30.0s, seg_a0_0.ts / seg_zzzz_0.ts / seg_0_0.ts 404 after 20.1s
each. After, on the running localdev API: 404 in 44-316ms for all of them, while index.m3u8,
seg_0.ts and seg_1.ts still answer 200 — nothing is over-refused.
The session now carries the rendition names its ffmpeg was told to write, and `entry` is derived from
them instead of passed. And "one place" now means the argv BUILDS its patterns from the same
functions the parser reads back (`segment_name("%d", "%v")`, `variant_playlist_name("%v")`): the
layout was still spelled out independently in six places, with `_VIDEO_VARIANT` read by no consumer,
which is the drift that caused round 6's HIGH. A test asserts the argv's own patterns are what those
functions produce, and that every name they expand to is one the guard accepts.
The load-bearing MEASURED comment cited ffmpeg 8.1.1 — the Windows host's. The binary that writes
these files is the API container's 7.1.5; re-measured there, the conclusion holds.
|
||
|
peter
|
a0114deff1 |
fix(api,ui): a refused write says why, wherever the refusal happens
Round 5 taught `notifyActionError` to discard a plain-string `detail` in favour
of the caller's translated fallback, because `req` leaves 403/404/429 to the
caller and those routes hand-write English. Three places did not get the
message.
**The demo account lost its explanation.** `require_human` refused with a plain
string, so from that commit on the shared demo account got its caller's generic
"couldn't do that" and no hint that the demo account was the reason — and the
gate is not backed by a UI gate everywhere: the comfort shelf has no `is_demo`
check, so the pin button is live and simply fails. A structured `{code: "demo"}`
and an `errors.demo` string in both locales. The two auth routes that hand-roll
the same check were swept with it.
**`ChatThread` carried a second copy of the contract.** It filtered 404/429
itself and then did `e.detail || t(…)`, the pre-fix order, so the central fix
could not reach it and "That member hasn't set up messaging yet." stayed English
under a Hungarian UI. It goes through the shared helper now, and a contract test
pins the remaining raw-`detail` consumers as an explicit SET — never a count —
so a new copy fails the gate instead of shipping. The five on that list are
inline surfaces (a form field's error line, a setup banner) and are deferred
with the ~60 backend routes whose details they print.
**The Plex "Add to playlist" dialog was still silent.** Same shape as the
collection editor round 5 fixed and the YouTube twin that already had the fix:
no catch, so a playlist deleted on another device answered a structured 404 that
nothing reported — the tick never flipped, the gone row stayed, and the floating
promise became an unhandled rejection.
Also: the `ABORTED` sentinel's VALUE was asserted nowhere. The consumer test
hand-built the error it expected, so replacing `xhr.onabort`'s rejection with a
friendlier message left the lane green — and every upload the user cancels would
then raise a red "the upload failed" toast, which is what the sentinel prevents.
Both producer tests assert through `isAborted` now.
And two backend tests were pinning fewer sites than their own docstrings claimed:
the channel-404 loop said "all five" and issued three requests (explore and
subscribe unexercised, the excuse for subscribe naming a different call site
with a different reason slug), and the recipient/job test named three refusals
and requested two — `share_download`'s `not_found("recipient")` had no test in
the suite at all, nor did the locale keys added beside it. Every site named now
has a request behind it.
6 tests, mutation-checked against 7 mutations: the demo detail at both the gate
and an auth route, a re-hand-rolled ChatThread, the silent Plex dialog, the
abort sentinel, and each unpinned 404 — all red on the test that names them.
|
||
|
peter
|
7f17b17c13 |
fix(plex): an HLS session's file names are one vocabulary, and a name it cannot produce fails at once
Round 5 added a guard so a request for a segment beyond the produced region answers instead of polling out the full timeout — the route is a SYNC one, so every such wait parks a threadpool worker and, through `current_user`, a pooled DB connection. Measured against real ffmpeg, the guard could not fire where it mattered. `-hls_segment_filename seg_%v_%d.ts` substitutes the VARIANT first: a dual-audio session writes seg_0_0.ts (video) and seg_a0_0.ts / seg_a1_0.ts (the `name:aN` renditions). The regex read those as sequence-first, so every video segment parsed as segment 0 and every audio segment did not parse at all — the guard was inert on every multi-audio item, which is most of this library. `_seg()` carried the same inversion: the warm-up waited 8 s for `seg_1_0.ts`, a name ffmpeg never writes, on every multi-audio session start AND every seek-restart. The layout is now one constant plus one parser (`segment_name` / `_SEG_NAME_RE`), with a test pinning them to each other, instead of three comments that could drift — and they had drifted, all three ways. The guard is also a positive rule now. It was a blocklist, so anything that was not a single-audio segment name was waved through: `GET …/anything.m3u8` still parked a worker for 30 s, while the route's comment claimed a flood of hopeless names was what it prevented. A session produces its playlists and its segments and nothing else. `.vtt` leaves `_HLS_CT` with it — ffmpeg runs with `-sn` and subtitles are a separate route, so that extension could only ever name a file that will not arrive. Also here: - A 5000-digit segment index reached the client as a 500: `int()` on an unbounded digit run raises past 4300 digits and only OSError was caught. The name is bounded on the route and the index in the regex — the same bug class the subtitle routes were bounded for. - A session torn down between the wait and the read is a 404, not a 500. `_kill` rmtrees the directory and no lock spans those two steps; the player's recovery is keyed on the STATUS, so a 500 skipped the reclaim and ended playback with "couldn't play" instead of restarting at the playhead. The trigger is ordinary — `loadSession` awaits the new session before destroying the old hls instance. - Two comments claimed an open player checkpoints every 10 s "whether or not it is playing", which is the stated reason the 30 s finished-idle reap is safe. `flush` sends nothing once the finish latch is up, or before a position is known. The states it does cover — playing or paused mid-item — are the ones the rule needs; the comments now say that instead of more. - The route's tests leave the DB lane: the route reads nothing from the database since the auto-start was removed, and there those four regressions SKIP on a machine with no Postgres while pytest prints green. 14 tests, mutation-checked: the old parse, the old warm-up name, the blocklist, both bounds off the absurd name, the unguarded read, and the route dropping `proc` — each red on the test that names it. |
||
|
peter
|
1f81684b6e |
fix(ui,plex): a dialog keeps the focus its child asked for, and a hopeless segment fails at once
Both found by UAT in a real browser, after the tests for them were already green — which is the whole reason this repo settles focus and timing claims by running them. - **The autoFocus fix worked, and then the dialog took the focus back.** React 18's StrictMode runs every effect mount→cleanup→mount in development: the cleanup handed focus back to the trigger, and the second mount, finding the focus outside the dialog, moved it to the first focusable — off the `autoFocus` field a beat after the browser had put it there. Chrome measured it: INPUT (commitMount) → BODY (cleanup) → ✕ (re-mount). The effect is idempotent now — it remembers what the dialog was focused on and restores THAT on a re-mount — and the close-time restore is conditional on the dialog still holding the focus, which it tracks as it changes rather than computing at cleanup, where the panel is already out of the document. A dialog no longer yanks focus back from wherever the user has since moved it. - **Passing ffmpeg's handle to the segment wait only shortened the FINISHED case.** Measured against the running localdev API: a request for `seg_99999.ts` on a live session still took 20 195 ms, because that ffmpeg has not exited and the wait has no other reason to stop. ffmpeg writes segments in order, so a segment whose predecessor does not exist is not the next one and cannot arrive — and that is knowable at once. Same request now: 45 ms. Verified on localdev against the real library (Westworld S01E06, remux, multi-audio): Mod+K opens the command palette with the focus in its search field and typing reaches it; playback runs on the new session-scoped HLS URL; a request naming a stale session id is a 404; and a viewer paused for longer than the finished-session rule keeps their 2.4 GB session across two reaper passes — the progress checkpoint touches it — and resumes cleanly. vitest 1009, pytest 1127. |
||
|
peter
|
4a7b65d187 |
fix(api): a refused write is named, in the reader's language
Deep-review round 5, over round 4's own fixes.
- `notifyActionError` preferred the server's `detail` over the caller's translated fallback, and
`reportFailure` deliberately leaves 403/404/429 to the caller — so for exactly those statuses the
dozen call sites round 4 converted started printing whatever English a route hand-wrote into an
otherwise Hungarian UI ("No user with that email.", "Not subscribed to this channel", "Unknown
job"). A STRUCTURED detail is still preferred: `req` has already run it through `localizeDetail`,
so it is both localized and more specific. A plain string is not.
- The other half of the same defect, on the backend: round 4's `get_or_404(what="channel")` sweep
reached exactly ONE of the five identical sites in channels.py and shipped the HU/EN string with
it, so the Block button said it in Hungarian and the four controls beside it in English. All five
are converted, and the subscription / share-recipient / scheduler-job 404s with them (two new
reasons in both locales).
- `PlexCollectionEditor`'s add/remove, take-over and delete were try/finally with no catch — in the
very file round 4 edited to give `createNew` one. 403 and 404 are the caller's to report, so the
tick simply never appeared and a CONFIRMED "delete collection" did nothing, silently.
- `UploadDialog` is the only `uploadReq` caller and kept an empty catch whose comment still claimed
the global error modal had it. It had not since `reportFailure` became shared: a proxy 502 raised
only the self-clearing connection notice and a 413/403/404 nothing at all, so the dialog snapped
back to its pre-upload state with no error named. The user's own Cancel stays silent — and since
a cancel and a network drop both carry status 0, that difference is now a named `isAborted`
rather than a string literal at each end.
Regression tests for each (5 vitest, 2 pytest, plus two existing assertions corrected), all
mutation-checked: 6 mutations, each red on the assertion it is named after. vitest 1007,
pytest 1127.
|
||
|
peter
|
47af4d5c05 |
fix(plex): an HLS artifact names the session it belongs to, and a paused viewer says so
Deep-review round 5, over round 4's own fixes. - Sessions are keyed by rating_key alone while a session's DIRECTORY is per (key, offset, tag), and the artifact URL carried no session identity — so once a seek-restart or a second viewer of the same item took the registry slot, the old viewer's request for `seg_12.ts` was answered 200 with the new session's twelfth segment: bytes from a different `-ss` offset under `-copyts`, on a timeline the client is not on. Wrong content with a success status is the one failure a player cannot notice, and round 4's reclaim-on-404 could never fire on it. The id is in the path now (so the playlist's relative names resolve back into the same session), a mismatch is a 404, and the player reclaims at the playhead. The entry playlist's auto-start is gone with it: it could only ever begin at 0, silently discarding the viewer's position. - Round 4 exempted paused viewers from the reaper by keeping EVERY finished session for the full ten-minute idle window — one whole segmented copy of a film per item, on the volume the download center shares. The signal it needed already existed: an open player checkpoints its position every 10 s whether or not it is playing, so that route now touches the session and the periodic pass is back on the short rule. - POST /session's opportunistic reap was never scoped to the key being started: it walked the whole registry with the 30 s finished rule, so pressing play on ANY item — even a direct-play one, the call ran before that early return — killed another viewer's paused session. It is gone; the periodic pass owns reaping. - A segment that can never arrive no longer polls out the full 20 s: the producer's handle is passed to `wait_for`, so a finished ffmpeg answers at once. This is a sync route, so each such wait parked a threadpool worker AND a pooled DB connection, and an ordinary seek past the produced region was enough to do it. - The DataError handler's re-raise branch — the half that keeps a genuine DataError a 500 — is pinned, so simplifying it to an unconditional 422 cannot pass in silence. Six regression tests (two route-level, in the DB lane), each mutation-checked red on the assertion it is named after. pytest 1124, vitest 999. |
||
|
peter
|
6d49da17fe |
fix(api): a write that hit a connection failure is still reported, and every failure is reported once
Deep-review round 4, the "one report per failure" contract round 3 introduced. - `req` marked gateway (502/503/504) and network failures "already reported" because they raise the connection notice. But that notice is about connectivity and removes itself on the very next answer — which every caller's own resync refetch fetches at once — so a delete that hit a keepalive reset flashed the notice and was then reported nowhere, the list still there. Only the durable reports (the error dialog, the 401 sign-out) count now; the caller says the action failed. - A connection notice the user closed swallowed every later one for the rest of the visit; it is raised again. - `req` and `uploadReq` share one failure policy (`reportFailure`). The upload copy had drifted: a 502 opened the blocking "Server error (502)" dialog and cleared a notice that was still true. - A taken Plex collection name is a structured 409, localized, and the create/rename calls are quiet: it was an English dialog plus a Hungarian toast. `notifyCollectionEditError` is gone. - The mutation onErrors that toasted unconditionally after a non-quiet request (invite approve / deny / add, demo whitelist, audit clear, scheduler, share, full history, edit, config tests, tag attach) go through `notifyActionError`; the playlist delete's hand-rolled copy of it too (it takes a level now). The user role / suspend / delete actions had no onError at all: a user deleted in another tab failed silently and stayed listed — now said, and the list refetched. |
||
|
peter
|
31dc4184f5 |
fix(api): a paused viewer keeps its HLS session, a refused subtitle publish backs off, and a NUL is a 422 everywhere
Deep-review round 4, backend half. - The periodic HLS reaper (round 3) applied the "finished and unfetched for 30 s" rule that only a NEW session request may apply. A remux finishes long before the film, and a paused player with a full buffer fetches nothing, so a one-minute pause killed the session and playback died ~30 s after resuming. The periodic pass is `reap_abandoned` now (the long idle limit, finished or not). - A subtitle `publish` that could not keep its output (full / read-only cache volume) returned quietly, so every poll queued the window and the whole-container pass again. It reports failure and the pass is backed off like a failed read; a cached VTT that cannot be READ is answered `unavailable` (`cache_unreadable`) instead of scheduling reads that can never land. - The subtitle routes bound `rating_key` / `sub_ord` (they become a cache file name: a 250-digit one was ENAMETOOLONG, which pathlib's exists() re-raises -> a 500), and the cache helpers no longer throw on such a name. - One app-wide handler turns psycopg's "text fields cannot contain NUL" DataError into a 422: round 3 guarded only the by-rating_key lookup, and `contains=`, `contains_group=`, reorder/bulk key lists and the collection search still 500'd. - The hand-written `db.get(...) is None -> 404 "Unknown ..."` siblings use `get_or_404` with a `what` (channel / video / user), so the Hungarian UI no longer shows English for a gone row. |
||
|
peter
|
86971c574d |
fix(api): progress checkpoints past the int4 column are a 422, and idle HLS sessions are reaped on a timer
Deep-review round 3 of bug hunt 2026-09. - The YouTube and Plex progress bodies are bounded by POSITION_MAX_S (the int4 position columns): a finite position of 2^31 or more was a DataError, i.e. a 500 — the sibling of round 2's non-finite fix. - plex/stream.py promised "a periodic reaper", but reap_idle ran only inside the NEXT session request: after the last playback an abandoned session (its ffmpeg still working, its directory a whole segmented copy of the film) lived until someone pressed play again. A lifespan task reaps every 60 s. - Tests for what round 2 left untested: the 422's body keeps the list shape the UI localizes, and stream_session refuses non-finite start/aoff. |
||
|
peter
|
1cfe6a7f59 |
fix(api): a gone row and a refused playlist action are said in the reader's language, and a failure is reported once
Deep-review round 3 of bug hunt 2026-09.
- get_or_404 / owned_or_404 answer a structured {code: "not_found", reason: <what>}
(errors.notFound.* in HU+EN, generic fallback). The structured-404 fix of round 2 had gone into the
ad-hoc `_own_playlist` (CLAUDE.md's "Not" pattern) while the shared helpers stayed string-only, so
views, tags, pins, Plex playlists, invites and users still 404'd in English. `_own_playlist` now
goes through owned_or_404; admin's hand-rolled user lookups use get_or_404.
- A move into a list that is gone names the TARGET (reason "move_target") — "this playlist doesn't
exist any more" was read as the list on screen, which still exists.
- Every refusal in the playlists router is a {code: "playlist", reason} (playlists.errors.* HU+EN)
instead of a bare English string shown verbatim in the Hungarian UI.
- `req` / `uploadReq` mark the errors they already put on screen (the error dialog, the connection
notice, the 401 sign-out), and notifyActionError skips those. The watched toggles, the saved views
and a playlist delete toasted a fault the dialog was already showing; Playlists' two hand-written
status filters now go through the same check.
|
||
|
peter
|
4f37c41021 |
fix(plex): a subtitle failure about one position no longer ends the track, and windows are read ahead and never wait behind a full pass
Deep-review round 3 of bug hunt 2026-09. - `failed` is now final only for a track that can never be text (bitmap, no item/track/file). A Plex hiccup or a stretch ffmpeg could not read answers `unavailable`: said, polled again every 5 s, and the <track> stays mounted when cues exist elsewhere (gen > 0). Re-picking the selected track asks the server again at once. - Within LOOKAHEAD_S of a covered stretch's end the next window is read ahead, instead of only once the playhead has fallen off it (subtitles dropped out at every edge while the full pass was backed off or slower than a window lasts). - Windows and full passes run on separate pools with separate in-flight caps, so a window never queues behind (or is refused by) minute-long full passes — the pool comment promised that and nothing enforced it. - A pass that times out is TERMinated and keeps what it reached (published as partial) instead of being killed and discarded. - Lapsed failure memos are swept on every write (they were only pruned when the same key was read again, so polling fresh keys grew the process without bound). - `publish` no longer merges into a sidecar with no VTT behind it, or claims a region whose VTT write failed, or overwrites an unreadable track with one window. - A NUL in a rating_key names no row (it was a DataError, a 500, on every Plex route) and no longer 500s the status poll. Tests: 16 new, each mutation-checked (18 mutations, all red on the test that names them). |
||
|
peter
|
dcd8a9566a |
fix(playlists): the player's Remove follows the list it was launched from, the player survives a change of selection, and a gone list is said in the reader's language
- D3 snapshotted the removal's TARGET, but whether the player offered Remove still followed the
selected list: launched from Watch later (no Remove, by design), selecting another list under it
made Remove appear — and delete from Watch later. The snapshot now carries the launch list's
editability too.
- Selecting another list under an open player (the command palette works over it) put the page into
"Loading…", and the player was rendered only in the loaded branch — it vanished mid-playback and
came back at its launch video. It now sits at one place in the tree whatever the page is doing.
- A removal takes the video out of the CACHED list as well as the rows: the player's progress
checkpoint rewrote that entry before the refetch landed, handing the membership guard a list that
still held the video — the row came back and the undo history went.
- The watched-state callback refreshed the selected list, not the launch list (D3's sibling).
- `_own_playlist`'s 404 is structured (`{"code": "playlist", "reason": "not_found"}`) and localised:
the bare "Unknown playlist" reached the Hungarian UI in English, which also made round 1's new
`removeFailed` string unreachable. Both Remove and Move report through `notifyActionError` now.
- The "viewer moved on under the confirm" test stepped back only once, which lands on the same video
whether or not the removed one left the queue; it now takes the second step that tells them apart.
|
||
|
peter
|
7360339039 |
fix(api): a non-finite number in a JSON body is refused with a 422, not a 500
The JSON parser under FastAPI accepts `Infinity`/`NaN`, and both progress checkpoints truncate the position with `int()` — a hand-written checkpoint came back as a 500 with a traceback. Their bodies now refuse non-finite floats, and the app's validation handler — FastAPI's own shape — echoes a non-finite input by name: the default one failed to serialise its own error (JSON has no NaN), which was a 500 by another route. Sibling of the subtitle routes' non-finite query parameters. |
||
|
peter
|
473d9f02f2 |
fix(plex): subtitle windows land where the full pass does, and the extractor stops re-reading, respawning and outliving the process
Deep-review round 2 of the 2026-09 bug hunt, over the round-1 fix diff. - A window was LATE by the distance to the keyframe its input seek landed on, and then doubled for good once the full pass merged in: a cue already running at the window's start came out at a negative time, and the webvtt muxer's default shifted the whole output so it sat at 0. Measured on the real Westworld S01E06 at 25:00: every window cue +5.005 s, 682 cues merged instead of 608. `-avoid_negative_ts disabled` keeps the times exact (the parser reads ffmpeg's signed-field spelling of a negative time); now 608/608 on that file, and a real-ffmpeg test pins it. - A full pass that failed was re-armed every minute while the viewer watched what it did publish — a whole-container read per minute. It now backs off for an hour; windows keep running meanwhile. - A failed WINDOW set the track's failure memo, so any uncovered position read "failed" — a final answer that ends the player's poll, even with the full pass about to deliver, and one any account could plant with an absurd `at`. It now marks only the span it failed to read; "failed" is given once the full pass AND a window there have both failed. - Windows join by what they COVER, not a bucketed identity: a backward seek no longer waits behind a window that cannot cover it, and a playhead crossing a bucket edge no longer queues a second read. The queue is bounded per track and overall. - shutdown() sets a flag a worker checks before it spawns ffmpeg (one still in its Plex lookup used to start a read after the kill sweep); queued-job cancellation and the lifespan call are now tested. - A window starts where ffmpeg is told to seek (rounded to the ms it gets), a rounded-up millisecond carries into the minute (no more `00:00:60.000`), a key's first publish claims nothing before its VTT lands, and `at`/`offset`/`start`/`aoff` refuse non-finite values (`at` also an absurd one). |
||
|
peter
|
d2ba65e383 |
fix(plex): the subtitle poll backs off on errors, a failed <track> is replaced, and a resume asks for the right window
Deep-review R2, plus one defect found while verifying it on localdev. - The status poll re-armed every second after a FAILED request: `data` stays empty on an error, so it read as "still preparing" for as long as the player was open — and the request was not quiet, so a persistent 500 re-opened the global error dialog every second. It now backs off to 10 s (a blip still heals on its own) and is quiet; the player already shows the failure inline. - A <track> whose own VTT fetch failed never fetched again: with the server's answer unchanged its key was unchanged, so React kept the dead element — silently blank subtitles. The key (now built in the hook, where it is tested) carries a failed-load count; after two remounts the track is reported as unavailable instead. - Found in UAT: until the first timeupdate `abs` is 0, so after a resume every poll said at=0 and the first window was cut around the head of the film (a resume at 20:00 polled at=0 for as long as it stayed paused). The poll reports the session's start until playback ticks. And the window's dedup identity is the track plus a window-length bucket of its start — keyed on the track alone (R1), the right window queued behind that wrong one. The poll's start/stop rules finally have tests (every earlier one resolved a single fixed answer). Verified on localdev against Westworld S1E6 resumed at 20:00: the polls carried at=1197, one subtitle line on screen at 21:02, the poll stopped at `ready`; the real file's window (56 cues) + full pass (608) merged to 608 — the old merge would have served 664. 11 mutations, all red. |
||
|
peter
|
cf7ec5e9ec |
fix(plex): the subtitle extractor's seams — doubled cues, a truncated track cached as whole, a poll that respawned ffmpeg
Deep-review of the bug-hunt round (R1). The background extractor from
|
||
|
peter
|
889d2acf52 |
feat(plex): the collection page sorts, and a linked collection has a name
Two halves of the same report ("Collectionben nem működik a rendezés?"). The pasted browse URL
sorts correctly — that part was working. What did not:
The Collection INFO page had no sort control at all and a hardcoded release-year-ascending order.
It now shows the browse grid's Sort control, with the browse grid's vocabulary, because a second
sort vocabulary on a neighbouring surface is exactly the kind of drift this repo keeps paying for:
`PLEX_SORT_KEYS` is now one tuple in `plex/query.py`, and a test pins BOTH implementations to it —
browse's key→column map (extracted from the handler to be reachable) and the collection page's
key→member map. A key added to the vocabulary without teaching one of them fails the gate instead
of putting a dead option in the menu.
Ordering a collection cannot be a single ORDER BY: its two legs are different tables, so the merged
member list is sorted in Python. Members with no value for the key sink to the END in both
directions (a movie with no rating is not "the worst rated" — the browse grid's `nullslast()`), and
ties always resolve title-ascending, so flipping the direction reverses the ranking rather than the
alphabet inside it. `lastwatched` needed the show leg's `resumed_at`, so the page now takes ONE
rollup pass that yields both the badge status and the recency instead of the status alone — without
it that menu entry would have ordered every show as never-watched. The default stays release-year
ascending: a franchise reads in release order, and that is what the page did before.
The other half: a shared Plex link carries the collection filter as bare rating_keys, so a link
opened cold had no name for them — the sidebar token rendered `17045` and the chip could only say
"Collection". `GET /api/plex/collections` takes a `keys=` CSV now, and a display-only hook resolves
just the missing ids. It deliberately does NOT write them into `collectionTitles`: that map is
hashed into the Plex query keys, so storing a resolved label there would split the cache from the
identical state and refetch the grid for a piece of text.
Verified in the browser on localdev against the real Plex library: the movie collection's menu
offers Length and the show collection's does not (the same scope rule as browse); rating-ascending
reorders 8.1/7.4/7.2/6.6/6.5 correctly; `lastwatched` on a show collection brings the in-progress
shows up; the choice survives a reload (it is a server pref, on its own key so opening a collection
never re-sorts the library behind it); and the reported URL now reads "Pirates of the Caribbean
Collection" in both the sidebar and the chip.
12 pytest + 8 vitest, mutation-checked against 14 mutations (direction ignored, nulls first, no
stable tiebreak, unknown key untouched, wrong default, show recency dropped, empty `keys=` falling
through to everything, a key missing from either sort map, every id refetched, ids unsorted,
fetched titles unmerged, the chip ignoring them, the hook's result never reaching the builder) —
each red on exactly the test that names it.
|
||
|
peter
|
5a42d76e8a |
feat(plex): an off-by-default loudness normalisation toggle for playback
Bug hunt 2026-09, item 14 — "how does the multichannel downmix work, and is it
normalised?" It isn't: `plex/stream.py` has always encoded `-c:a aac -ac 2 -b:a
192k`, i.e. fixed stereo with ffmpeg's default downmix matrix and no loudness
handling at all, which is why a film mastered for a cinema arrives with whispered
dialogue and deafening action.
New admin knob `plex_audio_loudnorm` (Configuration → Plex), DEFAULT OFF: it makes
a film markedly louder and narrows its dynamic range, which is a taste call rather
than a fix. It sits at the end of the Plex group, after the software-transcode
knobs — it is an audio concern, not one of the libx264 ones.
-af aformat=channel_layouts=stereo,loudnorm=I=-16:TP=-1.5:LRA=11,aresample=48000
Three things about that chain are load-bearing and each has a test:
* the downmix is FORCED FIRST rather than left to `-ac 2` — measuring 5.1 and
folding it down afterwards would normalise something other than what is heard;
* the trailing resample is not decoration: `loudnorm` runs its own resampler at
192 kHz internally, which the AAC encoder will not take;
* one `-af`, no stream index, so EVERY audio rendition of a multi-audio file is
normalised — not just the first.
The knob is also part of the session directory name (`…_multi-ln_+0.00`), because
it changes the audio the segments contain, and a session left over from before the
admin flipped it must not be mistaken for one that honours it. The suffix is
appended with a DASH: `_SESSION_DIR_RE` treats the tag as one underscore-free
field, and that regex is the only thing keeping the orphan sweep off what may be a
shared scratch path. A test pins both halves — the name changes, and the sweep
still recognises it.
Measured end-to-end on the app's own HLS segments (a real 5.1 AC3 file, same item,
same 160 s offset, same ten segments): OFF -35.2 LUFS, ON -17.2 LUFS. On a 60 s
bench: I -27.6 → -16.6 LUFS with LRA 13.3 → 9.8 LU. A/V sync is untouched — under
this pipeline's own `-copyts` the audio start PTS is identical with and without the
filter, so the 3 s lookahead desyncs nothing.
Direct-played files are served untouched and stay so; a 5.1 mix is always `remux`,
so the reported case is fully covered.
|