4aaaaa786017278460e0b4f94cc0d3374e0e37e2
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 peter
|
4aaaaa7860 |
chore(comments): drop epic, sprint and review-round ids from the code
The code cited its plan instead of saying why: `E28 review J4`, `(R6 S3b)`, `S1 justified…`,
`deep-review round 5`. That plan lives in the private ops repo, so for every reader of this one the
citation was noise, and a sentence whose subject was the plan step ("round 2 showed…") hid its
lesson behind an id. Each one is gone and the "why" kept: a bare citation is deleted, a sentence
carrying a lesson is reworded around it ("a later attempt showed…").
Measured (`scripts/audit_metrics.py`, product code): 652 lines in 270 files carried an epic code by
the definition the next commit holds (507 / 241 by the old, two-digit-only one) -> 0. Across all
code, tests, scripts and configs: ~1100 epic-code lines plus ~320 sprint/round ids.
Behaviour unchanged, checked mechanically: every changed .py file has the same AST once docstrings
are blanked; css/sh/toml/yml/json files are identical once comments are stripped; the only
non-comment changes in TS/JS are test titles, one description string in a contract table, and five
ESLint messages, whose text is all the gates snapshot diff holds. Not touched: Markdown (docs and
the changelog keep their history), Alembic migrations, TV episode notation (`S2 · E5`), DESIGN.md
ids (`C1`, `B4`, `S3h`), tracker ids (`SEC-16`, `DX-17`).
One stale claim fixed on the way: test_hls_reaper's docstring said the player checkpoints "every
10 s whether or not it is playing", which plex/stream.py calls wrong; it now says what the code does.
|
||
|
peter
|
3c16012524 |
build(backend): close the lint gaps the E32 audit measured
ruff gains RUF100, PLW1510 and flake8-bugbear, every hit fixed rather than ignored. B008 stays on with FastAPI's Depends/Query declared immutable. basedpyright moves to standard mode, pytest to --strict-markers, and a hand run of pytest now requires the database unless SIFTLODE_TEST_REQUIRE_DB=0. PLW1510 found a real bug: the three ffprobe probes in plex/stream.py never looked at the exit status, so a probe that failed was read as the file's truth (an HDR source sent down the HW-safe path, a multi-audio file collapsed to one track, a subtitle offset taken from garbage) and the warning each except branch promises never fired. They now run with check=True. |
||
|
peter
|
7c0647f87a |
fix(uploads): finish the sweeps round 1 left half-done
Deep-review round 2, on round 1's own fix commit. It found MORE than round 1 did (28 raw, against 13), and most of it was mine -- a new piece of UI attracts findings a backend-only diff never does. The pattern in almost all of them is the same: a principle applied to the site that prompted it and not to its siblings. * `display_text` reached three of the FIVE writers of a user-supplied name. `POST /api/downloads` and `POST /api/downloads/edit` still handed Postgres an unbounded string, so a 300-character name pasted into the Add-download dialog or the editor was a bare 500 (main.py re-raises every DataError but the NUL case). Four of the seven reviewers found it independently. The bound now lives in `models.py`, the three columns are DECLARED with it, and the guard for the next writer is parsed rather than remembered: every `display_name` / `display_uploader` assignment under `app/routes/` must go through the helper. That guard immediately found two more -- the Google sign-in callback writes the profile name into the same 255-wide column -- so it is six, not five. * The connection release reached one of the THREE routes that read their own body. The screenshot route (10 MiB, from a phone) and the client log sink both still held a pooled connection idle-in-transaction for the length of the upload, which is the invariant db.py sizes the pool on. All three now call `db.release_read_connection`, whose precondition -- nothing pending on the shared session -- is CHECKED rather than described, because a whole-session rollback discarding a caller's staged write is a mistake this repo has already made once. `test_body_limit.py` holds the rule for the next such route. * The AST drift registry keyed its sites by (module, limit expression), so a second route copied from the first collapsed onto the same entry and went uncovered -- the exact hole the registry exists to close. Keyed per call site now, and it carries the release requirement too. * The dialog's own refusal was announced once and never again: two over-cap picks leave an identical node with identical text, and an untouched live region says nothing. It is keyed per pick now, the drop zone reads as invalid and points at the sentence, and the reason is reachable from the control rather than only from a one-shot announcement. * Two tests rebuilt the blind spot they were written to remove: the dialog's cap test sat nowhere near the boundary (a `>` loosened to `>=` survived it, which is the mutation round 1 fixed on the server side), and the header-precedence test only covered an exact-case duplicate while a differently-cased one would have been COMBINED onto the wire rather than replaced. Both fixed, and the merge now dedupes case-insensitively because header names are. Also: the upload dialog's unstubbed usage query was firing a real request from jsdom in a neighbouring suite and leaking a delayed "connection lost" toast across test boundaries; `GET /api/downloads/usage` had no test at all though the whole client-side check hangs off it; and the comment claiming the edge 413 is readable now says what was measured -- readable to 200 MiB over loopback, a bare `onerror` through a proxy -- so the pre-send check is named as the mechanism and that branch as the backstop it is. Every test added or changed here was mutation-checked. One mutation SURVIVED and is recorded rather than papered over: the `tooBig` term inside `submit()` cannot be reached, because `ui/Button` swallows the activation itself, so it is marked as belt and braces instead of being described as the guard. |
||
|
peter
|
efd7b217ca |
fix(uploads): make both caps answerable, and stop holding a connection
The epic-end deep review found three defects in P1 and four tests that did not hold what they claimed. They land together because they share the upload route and its test module; splitting by hunk would have produced commits that do not individually run green. * The edge ceiling and the route's own cap are the SAME number at the default configuration, so the edge always answers first: a declared Content-Length over the cap is refused before a byte is read, and an undeclared body trips the middleware on the same chunk the route would have tripped on. The route's numbered 422 -- and the drain written for it last week -- are reachable only once an admin LOWERS the knob, which is the one shape the browser UAT tested. Measured: headroom does not fix this, because the declared-length check fires first for any body big enough to matter and the drain pushes the count back over the edge for the rest. So the edge's own refusal is made readable instead: `errors.tooLarge` localizes `too_large` the way `upload_too_large` was already localized, and the dialog refuses an over-cap file BEFORE sending it, off the `upload_max_bytes` that GET /api/downloads/usage now reports. Measured in Chrome against the API directly: that 413 arrives as `onload` with a readable body in 43 ms. * Reading the body behind the auth dependency put the whole transfer inside the request's database session -- the auth SELECT autobegins a transaction, so one of the twenty pooled connections sat `idle in transaction`, with nothing to time it out, for as long as the client cared to keep sending. That is the invariant db.py sizes the pool on and plex/playback.py records an incident for. The read transaction now ends before the await. Measured on the dev server: an 1818 ms upload shows no idle-in-transaction sample while the bytes move, only afterwards, when the handler actually writes. * The display name moved into a request header, where nothing bounds its length, while the column stayed varchar(255) -- so a long enough name was a DataError 500 after the whole file had already transferred. One `display_text` helper now serves all three writers (the metadata PATCH already truncated, the upload ingest did not), and the dialog stops at the same number. * The gate invariant asked `route.dependant.body_params`, which holds only the route's OWN callable's parameters: a body declared in a SUB-DEPENDENCY -- the upload route's own idiom -- leaves it empty while FastAPI still reads the body before auth. It asks `route.body_field` now, which is what FastAPI itself consults. The AST registry of body-reading call sites had also never followed `capped_stream_to`, so the one route the whole invariant exists for was the single site the drift guard could not see. Four tests that passed over their own mutation are fixed: the cap boundary (a `>` loosened to `>=` survived the entire suite), the percent-decoding (every filename the suite sends is ASCII, where quote() is the identity function), the display-name header (no test at all), and the staging cleanup (deleting the unlink left the suite green while every refused upload leaked its partial bytes onto the volume). Every test added or changed here was mutation-checked. |
||
|
peter
|
89c77ae6b5 |
fix(logs): deep-review round 3 — a failure nobody could record is one scrubbed line
- The download and edit failure lines go back after their DB write: a job the write could not fail is not a failed job, and the worker's child_exit line already says it failed. - The download child's runner is the last stop before Python's hook: an escaping exception is one download.child.failed line with its chain scrubbed, then exit 1, instead of a raw stderr traceback carrying the URL. - A crash the database will not record carries its stack once, and the fail-write line has an event code. - The job's own generic-extractor name is taken out as a literal (formats.hiding), so brackets and colons in a direct link's file name no longer leak; the pattern stays as the fallback. `Downloading just the video <id>` is covered. - Tests: scrubbed_stack's cause, suppress, unprintable and text-format paths, the image steps' failure kinds, the watchdog error through the real _fail_running, exact poster/storyboard messages, ffmpeg skip marker. |
||
|
peter
|
0b672ed6e7 |
fix(logs): deep-review round 2 — a stack names code, not data, and survives a failed write
- A failure keeps its traceback unless it is yt-dlp's or our own StatedFailure (a message that is the whole story). A plain RuntimeError, RecursionError or OSError is also what a bug of ours raises, and its stack is the only lead. - The stack is scrubbed link by link (logs.scrubbed_stack): a chained DownloadError under a DB error no longer prints yt-dlp's text unscrubbed. - Engine hide_parameters: a DB error's message no longer carries the title or path it was writing. - The download and edit failure lines go out even when their DB write fails. - The worker's child_exit/crash/supervisor lines say download.job.failed only when the job failed there; a crash the DB will not record no longer ends the thread. Stopping a paused child is download.child.stopped, not a second abort. - log_safe also covers generic ids with spaces, playlist titles and bare ids. - The ffmpeg image steps log the failure kind, not the argv with its path. - The consent contract accepts only the body of a plain consented_trace() check, and any level that may be TRACE. - Tests for each; every one mutation-checked. |
||
|
peter
|
54d19cc386 |
fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
`formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
file paths, and the job-failed, refused, edit-failed and backfill lines use
it. Done lines log the asset id, not the title-built path. The PII contract
now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
a process's last batch closes the repeat window, so a short-lived download
child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
`download.job.failed` with a cause; they left no line at all before. The
crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
carry reason `transport`. Subscribe recognises a duplicate by reason code
only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
|
||
|
peter
|
04b58e5c6f |
feat(logs): give the log the events that no request line already carries
The gap this sprint was scoped against turned out to be smaller than the survey said, and in a way worth writing down: `log_requests` already records every HTTP call with its route template, status, duration, request id and user id, and the audit log already records every admin act on someone else's account. Putting a line in each of routes/downloads.py's twenty mutating endpoints would have added a second copy of the first and nothing else. So this instruments what neither of them can see. yt-dlp now explains itself to our log. `quiet: True` was doing more than quieting: with no `logger` its whole account of a download went to the child's stdout, so "Requested format is not available", a signature-extraction failure or a POT refusal reached us only as whatever text ended up in the raised exception — the real reason a download failed was routinely one line the app never kept. Measured against the pinned yt-dlp 2026.07.04 before wiring it: with `logger` set it calls `debug` for extraction and progress lines and `warning`/`error` for the rest, and neither `quiet` nor `no_warnings` gates those calls. `debug` lands at TRACE, being several lines a second, so the noisy half is the half the user's own consent gates. The full source URL is trimmed to its host on the way in — it is a row of someone's download history, and `logs.set_job` already puts the job id on every one of these lines. The option is set AFTER the adapter merge, like `cookiefile` and the size cap: a source adapter must not be able to take away the one account of what went wrong that a failed job leaves behind. An HLS session now says which pipeline it got (transcode/remux, hw, tonemap, variants, offset) — everything downstream is about that choice, and it was only inferable from an ffmpeg command line nobody keeps. A session culled by the concurrent-transcode cap says so: somebody's playback stops there, the route turns it into a retryable 404, and "my film keeps rebuffering" had no other explanation anywhere. The HW→SW fallback keeps its line and gains an event code. A live search refused by a quota cap says WHICH cap: the 429 is in the request line either way, but the per-user daily cap is answered by tomorrow and the shared budget by the admin. The download and edit job lifecycle lines gain event codes, which is what E22.6's console filters on — the job id was already a structured field. tests/test_ytdlp_logging.py; three of its four cases mutation-checked. A fourth (a `%` in a progress line) survived its mutation — stdlib only applies `%` formatting to a record that has args — so it is gone rather than kept as a case nothing can break; the reasoning moved into the comment on `_emit`. |
||
|
peter
|
6b0401b922 |
fix(logs): rule on every swallowed exception in the backend
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.
What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
degraded to a software transcode with nothing saying why; a failed audio probe
collapsed a multi-audio file to one track and the player showed no language
picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
own root, leaving every crashed session's segments on disk forever.
Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.
ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.
logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.
tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
|
||
|
peter
|
0cff964ff2 |
refactor(logs): name every logger after its module, and hold it there
25 literal names covered 34 modules. plex/client.py, plex/stream.py, plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves `siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`; routes/channels.py called itself `siftlode.api`. The `logger` column is in every stored row so E22.6's console can filter on it, and so an operator can pin a level to one part of the system — neither works when six modules share a name. `logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync. One name per module, and it cannot drift from the file it sits in. tests/test_logger_names.py pins the derivation, that everything stays under the one `siftlode` tree the handler and the tests attach to, and that no module hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`). Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic it carries rather than the module it lives in — it replaced uvicorn's access log and is what an operator pins when they want request lines and nothing else. app/scheduler.py called its logger `logger`; every other module calls it `log`. ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them: worker.py logged `str(exc)[:120]` where the traceback was the whole question, and three prints/f-strings that are deliberate now say so on the line. S110 and BLE001 stay off until S3 — 23 handlers are outstanding and each is a judgement about one of them; enabling the rules first would just mean 23 blanket noqas. |
||
|
peter
|
dc221338be |
fix(logs): keep what a person typed out of the backend log lines
The E22 redactor is a safety net over addresses and credential params, and every one of these call sites was leaning on it to do the whole job. What it does not cover went in whole, and what it does cover came out as a constant. Not covered, so a real leak: - routes/search.py logged the raw search term at ERROR. The privacy policy shipped in 0.88.0 says a search string is kept only at Teljes/TRACE, with the user's own consent. The ERROR line now carries the shape of the failure (source, candidate and collected counts) and the term follows on a TRACE line, which nothing emits unless that consent is in force. - downloads/backfill.py logged the full source URL at INFO — a row of the user's download history. Now the asset id and the host. - youtube/client.py logged the upstream error body, which is prose that quotes the request back. Every failing YouTube call now goes through one _upstream_failure() that logs the API's `reason` code, keeps the body for the same consented TRACE line, and raises a YouTubeError carrying `reason` as a field — callers put that error in log lines of their own (routes/channels.py) and in a 422 answered to the user, so the body must not travel in it either. Covered, so merely useless: routes/auth.py named the account by an address that reached both sinks as `[email]`, and app/email.py printed recipient lists that read as `['[email]']`. Both now carry the uid (or a count) and a reason code, which is also the first half of E22.5 S4's failed-sign-in coverage. routes/channels.py branched on the text of that error message to recognise an already-existing subscription. It now branches on `exc.reason`, with the substring check kept beside it for a body that carried no reason at all. tests/test_log_pii.py pins the call sites and the family: a source-level contract test walks every log call in app/ and fails on an argument naming user-typed content unless the line says why it is safe (`# log-pii-ok: <why>`). Eight sites carry that marker today, each with its reason. Gate 8/8 green; all five cases mutation-checked. |
||
|
peter
|
4ab128ccab | Merge branch 'main' into feature/e22-activity-trail | ||
|
peter
|
b59fd38a9c |
feat: store warnings and request lines in log_events
The log now has a stored, queryable copy for the diagnostics console: every WARNING and above plus the request line of every request, kept for log_retention_days (default 7, no "forever") and pruned hourly by the new log_gc job. Rows are built on the logging thread from the same redacted fields as the stdout line, queued without blocking and written in batches by one background thread on its own connection. A full queue or a failed write drops and reports instead of retrying. Flood control keeps 30 of any one message per minute plus a count of the rest; anonymous request lines are capped at 60 a minute per client, and the client address is never stored. Deleting an account deletes its stored events. Measured on prod first: a YouTube RSS outage wrote 11k warnings in four hours, one per channel per poll. The poll now writes one warning with per-reason counts and a few examples, and a failure on our side gets one traceback per poll instead of one per channel. |
||
|
peter
|
2b56c5c2eb |
fix: name the error yt-dlp actually raises for a cookie-emptied client set
A review round read yt-dlp's source and found the comment behind the previous commit's safety-net marker did not match it. Measured in the worker with a real account cookie and `player_client=android_vr`: yt-dlp logs `Skipping client "android_vr" since it does not support cookies`, drops it, and the request left with no client fails with "Requested format is not available" — not "No player clients have been requested". That raise sits EARLIER in yt-dlp, before the cookie-capability removal, so the cookie path can never reach it; and an unknown or renamed client is skipped with a warning and replaced by yt-dlp's defaults, so the other scenario the comment named cannot reach it either. So the marker bought nothing: the symptom that does occur was already retriable here before this branch. It is dropped, the comment now states what was measured, and the test asserts the real symptom instead of a message the code cannot produce. The cookie-capability filter itself is vindicated by the same measurement — without it every signed-in download would burn its first attempt on a client yt-dlp discards. Also from the round: an enqueue test for a `source_kind="youtube"` job, the one join nothing else made (a YouTube job stores a bare 11-char ref and the URL the cookie gate keys on is rebuilt by the adapter — both ends were pinned apart, the join was not, and a break there would stop every YouTube cookie silently while signed-out downloads kept working). And copy corrections: the group's only unhedged claim is hedged, "described the video" loses the pipeline vocabulary, and four Hungarian phrasings that read as translated English are rewritten. |
||
|
peter
|
c06a456a9a |
fix: a YouTube stream refused after extraction is the gate, not a network fault
Measured on the isolated test stack the same night, on the very same video ~90 minutes after it was answering with the bot message: yt-dlp got the player response and picked its formats, and the CDN then refused the stream itself — "unable to download video data: HTTP Error 403: Forbidden". Signed out it did that on every attempt; with the owner's cookie attached the video downloaded in full. So the wall moves between the player response and the stream, and the text it wears there was costing the download twice. retriable: the format URLs belong to the client that minted them, so the next client set is a real second chance — but this text classifies as `network` and nothing in the YouTube marker list matched it, so the ladder ended the job on attempt 1 and never tried the fallback. catalog: `youtube.media_forbidden` for the cookie-less case, so the UI stops calling it a network error when nothing was wrong with the network, and names what actually gets past it. Scoped exactly as narrowly as the measurement: another host keeps the generic `network` category, and so does a download that already carried a cookie — telling a signed-in user to sign in is the failure the catalog exists to avoid. EN + HU copy. |
||
|
peter
|
864f673ef5 |
fix: a YouTube video behind the bot wall is downloadable with cookies
A YouTube link can fail with "Sign in to confirm you're not a bot" while every other YouTube download works — YouTube gates INDIVIDUAL videos behind a signed-in session. Measured on prod job 273: all twelve of yt-dlp's player clients fail on that one video while its neighbours download fine, the bgutil PO-token sidecar is live and answering, and the newest yt-dlp fails identically. So the download is not broken by our stack or a stale yt-dlp, and no retry, client switch or upgrade gets past it. A cookie is the only way in. The failure copy already sends the user to Settings -> Account -> Download authentication, but YouTube was not in the cookie host registry, so that dropdown could not help them: the app asked for something it did not offer. credentials: YouTube joins SUPPORTED_HOSTS (youtube.com, youtu.be, youtube-nocookie.com). The Settings dropdown, the SEC-12 host scope and the private-asset cache salt all derive from the row, so that is the whole wiring. adapters: a signed-in YouTube download runs only cookie-capable player clients. Measured against yt-dlp's own INNERTUBE_CLIENTS table: with account cookies attached yt-dlp REMOVES every client whose SUPPORTS_COOKIES is False, and a set it empties that way raises "No player clients have been requested" — which is not a per-client flake it retries past. Our primary set is android_vr, one of those, so without this the cookie would have failed the download on attempt 1 and never reached the POT-backed web fallback it exists for. A test pins the client list against the installed yt-dlp so a version bump turns the gate red instead of silently un-authing every YouTube download, and the empty-set message joins the retriable markers as the second line of defence. errors: catalog entries for the wall with and without a cookie, so a user who has already added cookies is told they were rejected (YouTube rotates exported ones) rather than to add some. The marker is quote-agnostic: YouTube writes the apostrophe as U+2019, so "you're not a bot" with a straight quote never matches the real text. EN + HU copy. |
||
|
peter
|
6c63df6704 |
feat: one structured log for every process, with a request id
A deployment printed four different log formats and none of them could be joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in main.py for anything not started by uvicorn, logging.basicConfig() in the worker and in the download child (no timestamp at all), and alembic's fileConfig. app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log data model, no SDK): ts, level, severity, source, logger, event code, message, attributes, plus the correlation the context carries — request id, account, download job. Redaction is applied at the HANDLER, not at the call sites: every message, every attribute and every traceback is scrubbed on the way out, so a secret cannot leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies, secret config values and email addresses never reach a line. The access log is ours now (`http.request`): the route template instead of the concrete path, the status, the duration and the account — and never the query string, which uvicorn's access log printed on every single request (`?account=`, `?source=`, the `?token=` of a share link). A well-formed client request id is accepted and echoed in X-Request-Id, so a browser line and a server line join up. The account comes from `request.state`, filled by the auth dependency: the middleware is the outermost one, so it runs outside SessionMiddleware and in a different task context. `log_level` becomes an admin knob (sysconfig), applied without a restart and re-applied at startup — best-effort on purpose, since a logging knob must never be what stops the app from booting. Every compose service gets bounded json-file logs (3×50 MB): Docker keeps every line forever by default, and the API now writes one line per request. Verified on the localdev stack: JSON everywhere including alembic and the worker, the query strings gone, a supplied X-Request-Id echoed and logged, and three /healthz hits of which only the one made while the level was DEBUG produced a line. |
||
|
peter
|
e7f38396a2 |
fix: review round 3 — a new tab keeps its account's notices, a revoked tab account lands on the default
Deep-review round 3 over main..improvement/watch-background-audio. Account boot - The notification history and settings, and the hints toggle, are read when their modules load, which in a new tab (bookmark, typed URL) is before BootGate pins the account: they read as nobody's, and the tab's first notice replaced the account's stored history (up to 100 entries) with a one-item list. They now read again when the tab is pinned (onAccountPinned, run by setActiveAccount); a notice raised before the pin stays, after the history. - A tab pinned to an account still in this browser's wallet but whose session ended (suspended, deleted, or revoked by "log out everywhere") got "signed out" from /api/me while the default account was still signed in, and signing in again as anyone else left it there. A refused per-tab account now leaves the wallet, and the probe answers as the default, so the tab reloads onto it. Share page (/watch) - The password unlock is limited per viewer (link + client IP), under a per-link ceiling that counts only wrong passwords. Keyed on the link alone, ten wrong guesses every five minutes locked out every viewer, the ones with the password included. client_ip moves to app.ratelimit so the public routes can key on it. - Its failure messages and the load-failure screen are announced (role=alert, the password input described by its error). The sign-in, passphrase, password-change and setup form errors get role=alert too. Comments that still described removed machinery are corrected: App's collapse flags, SettingsPanel's Save bar, ConfigPanel, authed_format_sig. Tests: the account's history and hints across the pin; a revoked and a suspended pinned account; the per-viewer limit and the ceiling; the limiter's exhausted(); the share page's announced failures; a share link's feed filters persisted. Each mutation-checked RED for its own reason. |
||
|
peter
|
f569b19c49 |
fix: review round 2 — a signed-out tab acts as nobody, honest share-page failures
Deep-review round 2 over main..improvement/watch-background-audio. Account resolution (server) - A request naming an account that is no longer signed in to this browser (X-Siftlode-Account or ?account=) now acts as nobody instead of the session's default account. A tab still pinned to an account signed out in another tab went on acting as the default: its player checkpoints and preference saves landed in that account, the messages socket and Plex playback ran as it, and its Sign out signed THAT account out. A hidden tab never polls /api/me, so a tab playing in the background kept doing it. The /api/me bootstrap probe alone keeps the fallback, which is how such a tab learns who it now is: the app's 401 handler reloads, and BootGate re-pins the tab. BootGate - On an identity change the tab reloads without moving its pin first. The tree being torn down still runs its unmount saves before the page goes, and with the pin already moved they wrote into the new account's local keys (a player volume still pending). The reloaded page corrects the pin. Share page (/watch) - Only a 404 says the link is gone. A 5xx, a dropped connection, and the unlock's own rate limit all rendered "no longer available ... expired or revoked" to a viewer with a valid link, with no way to try again. A load failure now offers Retry; too many password tries says so and keeps the form; an unlock that gets no answer says so instead of doing nothing. Strings in en/hu/de. Downloads - A cookie-authed asset is recognised under the current signature version and the one before it, so a job queued before a deploy that bumps the version still downloads with its cookie instead of failing on the login wall with an error that says no cookie was used. The app chunk is preloaded through lazyWithPreload, so a failed fetch no longer leaves an unhandled rejection beside the ErrorBoundary. The panel layout seed is one helper (seedLayout). Tests: the stale-tab refusal, its harmless Sign out and the probe fallback; the pre-bump authed asset; BootGate keeping the pin on reload and staying mounted through a same-account change; the reveal clock's later-commit hand-off; the prefs, panel layout and demo-scope seeds from the me cache; the share page's failure states. Each mutation-checked RED for its own reason. |
||
|
peter
|
6169e32ca5 |
fix(downloads): a signed-in Facebook reel keeps its title, and an mp3 stays an mp3
Since a Facebook cookie was added on prod, every reel came down titled "Video": measured on dev, a signed-in read gives yt-dlp the placeholder "Video" (an 18+ reel read as Chrome gives "Facebook") while the post's text is still in the description. A new SourceAdapter.title hook takes the whole first line of the description in that case, and both title writes (the early fill and the finished download) go through it, so the row, the file name and the share preview carry the post's text again. The line is not cut with an ellipsis: normalize_title's NFKC turns "…" into "..." and collapses it to ".", which read as a finished sentence (seen in dev UAT). The "Audio (MP3)" profile's extracted .mp3 was remuxed into an .mp4 by the browser-safe step (the audio spec is clamped to container mp4 too) and served as video/mp4, which iPad's WebKit will not play (prod job 268). Audio mode now skips that step, so an .mp3 is served as audio/mpeg and an .m4a as audio/mp4. Audio downloads version their cache identity on their own (v5), so a cached remuxed audio re-derives without re-downloading every cached video. |
||
|
peter
|
2a9ddb78ac |
fix(downloads): E11.5 epic-end review round 2
Pause now re-reads the job under the same asset-then-job row lock as resume, cancel, delete and redownload. Its unlocked write waited for a concurrent cancel's lock and then set the released job back to paused, a holding status, so a later delete released the same hold again and could free a file another user's job still held (measured: the overlapping-request test ends 'paused' instead of 'canceled' with the old read). The lock test now calls the route handlers, so it also pins which helper each route uses. Comments follow the code: resolve_url's contract names both verdicts and that a failed page read is raised as itself; the inherited-failure diagnosis asks the cookie gate as it stands when the job is claimed, so its advice fits the cookie that job's Retry would use; the undecryptable-cookie warning no longer claims a download goes ahead, since the worker parent reaches it only to diagnose. |
||
|
peter
|
e887233210 |
fix(downloads): E11.5 epic-end review round 1
A refused Facebook cookie on a group post is no longer "no video in this post":
measured on dev, Facebook answers with its login page (no post id at all), so the
adapter now raises its own POST_NOT_SHOWN verdict and the catalog explains it as
the refused cookie. The adapter owns both post verdicts and the "cannot parse
data" wall wording, so the Chrome retry and the catalog read one constant. The
group-post "no formats" entry is cookie-scoped (with a cookie yt-dlp only gets the
post's video link) and uses the adapter's own group-post test instead of a
"/groups/" substring.
Job transitions that move an asset's ref count (resume, cancel, delete,
redownload) re-read the job under a row lock, asset first, so two overlapping
requests can no longer release the same hold. get_or_create_asset loses an insert
race inside a SAVEPOINT instead of rolling back the caller's pending writes (Retry
lost its cleared error). Every worker-side job failure goes through
finalize_if_running and writes error_detail, so a cancel that landed first is kept
and a failure never shows an earlier one's copy. The worker asks the download
child's own cookie gate (credentials.usable_cookie) when a job inherits a failed
asset.
UI: the Facebook copy no longer points at a Retry button the failure window does
not have, says "paste" like Settings, and the HU wording is fixed
("Újraletöltés" on the re-download confirm). A category this client has no copy
for falls back to the generic sentence instead of a raw key.
Tests: route serialization of error_detail, the probed duration through a real
_download, the page-read cap and a failed read, a Retry off a pending asset, the
confirm-button labels, the lock and savepoint races.
|
||
|
peter
|
90ee34a1b4 |
feat(downloads): a Facebook group post downloads its own video, or says it has none
Given a group post's link, yt-dlp either found no formats or took another video from the page's feed data, so a "successful" retry of such a post was a wrong download. With the user's cookie the Facebook adapter now reads the post page, takes the single video node in the script blocks that carry the post id, and downloads that video's watch link (the Chrome retry does the rest). No single video in the post fails with its own message and the group-post copy, reworded to say the post has no video; the post link is never handed to yt-dlp as a fallback. Without a cookie nothing changes. |
||
|
peter
|
118fc70e08 |
fix(downloads): take the duration from the file when yt-dlp gives none
A Facebook or Instagram reel extracts with no duration, so the asset kept a NULL duration_s although the file plays for minutes, and the Library row had no trim action. The finalize probe now returns the duration with the codecs (the file wins, info fills a gap), and runs before the sidecars so the .nfo runtime gets it too. |
||
|
peter
|
1354f5b503 |
feat(downloads): a Facebook cookie download stuck at "Cannot parse data" is retried as Chrome
Measured on dev: with the user's cookie some reels still answer "Cannot parse data" and only extract when the request impersonates Chrome. Impersonation is a second attempt behind that one error, only when a cookie is attached — links that already download with the cookie keep their plain path, and without a cookie the same error is the login wall. curl_cffi is pinned inside yt-dlp's supported range (0.10-0.15; outside it every target silently reports "unavailable"); the lock only gains the new packages. A test fails the gate if the locked pair can no longer impersonate. |
||
|
peter
|
98ec1ffcab |
fix(downloads): Retry and Re-download take a cookie added after the failure
The login-wall copy tells the user to add their cookies and press Retry, but Retry kept the job on the shared asset the worker never attaches a cookie to, so it failed on the same wall. Resume now moves a pending/errored job onto the asset enqueue would pick with the cookie (never off a file or a download in flight). Re-download salted its fresh asset with the job's previous asset id, which the cookie gate cannot recompute, so even a cookie download rebuilt without the cookie. The rebuild salt is now the new asset's own id (plus the user's cookie salt when one applies), and is_authed_asset recognises it. |
||
|
peter
|
18c11ca437 |
feat(downloads): a per-site failure catalog — Facebook's walls say why and what to do
A failed Facebook download read "Download failed": yt-dlp words the login wall (18+ reel, closed group) as "Cannot parse data", which no category matched. Measured on dev with the owner's cookie before writing a rule — the same text is the login wall without a cookie and a refused cookie with one. - errors.diagnose(msg, url, cookie_attached) tries a host-scoped catalog first (all-of markers, cookie state, link fragment) and falls back to the category. First entries: facebook.login_wall, facebook.cookie_rejected, facebook.group_post_video. The module doc is the recipe for the next entry. - download_jobs.error_detail (0084) + backfill of the stored Facebook parse failures that were unknown. - The pipeline's failure branch passes the link and whether the cookie went with it; resume and redownload clear code and detail (redownload never cleared error_code). - A job meeting an already-failed asset got the text but no code, so it read "Download failed" where the first job said why. It is diagnosed again, with the cookie state from credentials.is_authed_asset (now shared with the cookie gate); the redacted network text classifies back to network. - FE: failCopyKeys gives the row and the failure window one lookup — catalog copy, else the category. Copy in EN and HU; a contract test pins every backend entry to both locales. |
||
|
peter
|
9f28f46839 |
fix(e12): unbreak the gate, and stop printing a command that can strand the restore
Round five. 47 -> 34 -> 15 -> 7 -> 6 raw findings, and this round's sharpest was the loop's own blind spot: **the branch did not pass the gate.** `prettier --check` is one of its eight lanes and I had been running the other seven, so an unformatted `onError` from round one sat on the branch for four rounds — `siftlode publish` would have refused it, with no --skip flag to get past. **The restart hint could still strand the restored data.** Round four said the scripts no longer print a command they cannot resolve; they did. The COMPOSE_FILE fallback only fires when `docker-compose.yml` is ABSENT, so in a source checkout it never fires and the hint expanded to a bare, pasteable `docker compose up -d` — which reads the other file, whose volumes are `pgdata`/`downloads` rather than `siftlode_pgdata`/`siftlode_downloads`, and recreates the database on an empty one. When the file is unresolved the scripts now print `-f <the compose file you started with>`: not runnable, which is the point. **A comment naming the wrong owner.** `formats.py` still said a zero budget is "a refusal for enqueue to make" — the premise round one disproved, three files away from the two docstrings that now say the opposite. Anyone acting on it would delete the worker's refusal and re-open the original inversion, so it now says where the refusal actually lives. Also: the reset-flag rule was rewritten in three consecutive rounds and pinned by nothing — two tests now cover a fragment losing its token, both by a pasted verify link and by navigating away entirely; `backup.ps1` deletes a partially-copied dump before throwing, the guard round four gave the docs but not the script it mirrors; and one comment that read as if the asset were untouched by a line four below it. Gate green, all eight lanes: ruff, ruff format, 3x tsc, eslint, prettier, knip, vitest 687, pytest 950. |
||
|
peter
|
fc56f514b3 |
fix(e12): the restart hint pointed at the one command that undoes a rollback
Round three, on round two's diff. The trend across the loop is 47 -> 34 -> 15 raw findings; the ops half kept producing the sharpest ones because the gate cannot run it. **`docker compose start` ignores a newly pinned IMAGE_TAG.** Round two removed the automatic restart because it re-ran `alembic upgrade head` — and then told the operator to run that exact command by hand, in both scripts and the docs. `start` relaunches the container that already exists, which still carries the tag being rolled back FROM, so pinning 0.79.0 and following the printed instruction brings 0.80.0 back up and re-applies its migrations. Verified against a real daemon: after changing the pinned tag, `start` kept the old image and only `up -d` picked up the new one. Both scripts and the doc now say `up -d`, with the pin-first caveat where the operator reads it. **The reset flag outlived its token.** Reverting round one's "each link clears the other" went one step too far: a `#verify=` link pasted over a standing reset notice replaces the whole fragment, so the reset token is gone — but `resetPending` stayed true, and being checked first it hid the confirmation while promising a recovery that signing out could no longer complete. The flag is now cleared when the fragment it describes is gone, which is narrower than what was reverted and does not blank the signed-out card. **A test that could not fail.** The shared-asset regression test seeded the asset as `pending`, and `_reset_asset_pending` only acts on `downloading` — so the release half was a no-op and deleting it kept the suite green. Seeded as the worker actually leaves it now. Also: the malformed-URL guard answers with urlguard's structured detail rather than a bare English string, so both halves of one typo class get the same localized copy, and the third site in that module handing user input to `urlparse` was swept with it; the asset is released after the job is terminal, not before; the Stats 403 toast gets the wizard handle its own comment claimed it had; the pasted Windows recipe sets `$ErrorActionPreference` inside the block (a scriptblock inherits the caller's, so a `SilentlyContinue` profile swallowed every throw) and cleans up the container-side dump on the failure path too; the restore warning moved to the warning stream so a `> log` redirect cannot file it away; and two comments narrowed to what the code does. Gate green: vitest 685, pytest 950. |
||
|
peter
|
358d722ce9 |
fix(e12): close what round one broke, and revert two fixes that overreached
Round two of the review loop, on round one's own diff. Two of its findings were regressions the fixes themselves introduced, which is exactly what the loop is for: **A per-user refusal was writing onto a shared row.** MediaAssets are deduped across users, and the generic failure handler puts the message on the asset — right for an upstream failure, wrong for "your storage is full". Two users queuing the same video share one asset, so the first user hitting their cap failed the second user's job with the first one's private reason, telling someone with an empty library to free up space. The refusal now raises `_OwnerRefused`, caught separately: the job fails, the asset goes back to `pending` for whoever else holds it. Pinned by a test that drives `_download` directly — the refusal happens before extraction, so nothing needs stubbing. **The restore scripts restarted the app, undoing the restore.** `docker compose start` re-runs the container command, and that runs `alembic upgrade head` — so on the rollback path this documents as recommended (restore the dump, THEN pin the older tag) the newer image immediately re-applied the migrations the restore had just undone, and on a FAILED restore it put the app back on a half-restored schema. Both are the opposite of why the stop was added. Nothing is restarted automatically now; the scripts say what to run and when, and the docs' snippets say the same. **Reverted rather than patched:** the OnboardingWizard toast (the wizard already renders that failure inline, next to a Retry button — three recoveries for one failure), and `useVerifyEntry`'s "each new link clears the other". The latter blanked the signed-out card's only banner, where nothing renders the reset flag to replace it. The two states are independent again; what changes is that `dismiss` peels ONE layer, which is what the original finding was actually about. Also: `Stats` uses the YouTube helper rather than the bare one, so the route's English 403 detail no longer beats the translated fallback under a Hungarian UI; `urlparse` guarded alongside `.port` (an unclosed bracket raised just as late), the reason renamed `malformed` since "bad port" misdescribes it; three docstrings narrowed to what the code actually does (the worker's byte check is per-job, not an aggregate ledger, and not on the edit path; the redirect residual leaks a reachability oracle, which the note had denied; the invalidation helper does not cover suspend); the pasted Windows recipe wrapped so `throw` can stop it; the bash one-liner's failure branch no longer exits 0; and the tests that were missing for the round-one fixes — the account panel's error paths, the invalidation set, and the shared-asset case above. Gate green: vitest 685, pytest 950. |
||
|
peter
|
fe38dfbcca |
fix(e12): close the eighteen findings the E12 review round left standing
The deep review of the branch raised 47 findings, 18 of which survived adversarial verification. Grouped by what they actually were: **The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one user over quota was the only one downloading with no ceiling at all. `check_enqueue` could not catch it: it ran while the job was queued and the user was still under the cap. The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision lives, and where it can be tested), and the worker now fails the job outright rather than proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help until space is freed. **Silent failures in the new account panel.** All three actions went through `quiet: true` endpoints, which suppress the global error dialog on the promise that the caller reports inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On the broken-SMTP instance this panel exists for, an admin would read silence as "the mail went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`, shared with the pending-access queue, so the two copies of these mutations cannot drift apart again — which is how half of this was lost in the first place. **A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps back one migration, but a release routinely ships several (0.80.0 shipped four): it drops the newest one's columns and leaves the schema at a revision the older image also cannot resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is now the recommended path, and the alembic route asks the target version for its own head revision instead of counting backwards. Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as a 500; the pull cooldown extended to the sibling route spending the same QuotaAction; restore stops the app before dropping its tables; the PowerShell cleanup can no longer decide the script's verdict on 5.1, nor report the wrong exit code; the documented one-liners hardened to match the scripts; the registry's retention window stated where pinning is advertised; the two banners made mutually exclusive so neither can shadow the other; `hover:bg-hover` (a utility this project never generated) replaced by the `selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to the two callers that discarded it. Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset banner UAT'd on localdev. |
||
|
peter
|
0cc450d1c9 |
fix(downloads,sync): bound what one user can make the server fetch, download and spend
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches it from the WORKER, so any approved user could aim the server at the compose network, the LAN or a cloud metadata endpoint — and read the result off job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a port scanner. downloads/urlguard.py refuses a URL whose host resolves to a loopback/private/link-local/reserved address, at enqueue AND again in the worker (DNS can move in between, and the worker is where the socket opens). A network-class failure on a `url` job now reports one fixed sentence instead of the raw transport text; the error CODE is still classified from the original, so the localized explanation is unchanged. DNS rebinding is explicitly NOT closed — that needs the connection gated inside yt-dlp — and the module says so rather than implying more than it does. SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video, so a URL that is purely a playlist or channel still expanded into every entry — one enqueue, unbounded downloads. `playlist_items` pins it to one. Size was unbounded too: `check_enqueue` compares the CURRENT footprint against the cap, and a download's size is unknown until it lands, so one job could overshoot by any amount; the owner's remaining budget is now handed to yt-dlp as `max_filesize`, applied AFTER the adapter merge so no source adapter can raise a limit the quota layer set. A join's cut-list is capped at 100: an accurate join emits two filters per segment into one ffmpeg filter_complex. SEC-14: the YouTube budget is instance-wide, so one user holding down a manual sync spent the day for everyone. Per-user (not per-IP — the cost belongs to the account) 3-per-10-minutes limiters on the two bulk pulls, subscriptions and playlists, with separate budgets so one does not close the other. The review also listed /rss, /backfill and /enrich; E3 deleted them, and search stays unlimited on purpose — it is interactive, not a bulk pull. Gate: pytest 896 -> 924 (none skipped), vitest 671. |
||
|
peter
|
1102ad6a28 |
fix(downloads): make the probe verdict authoritative over yt-dlp info by probe-ran, not per-field
Deep-review round 1 on bug/download-mediakind-probe found a real HIGH the prior review missed: - resolve_media_facts used a per-field `probe or info` fallback, so a probe that RAN and deliberately found no video track (cover-art-only audio: attached_pic skipped; a still image: format-guarded) had its None video overridden by yt-dlp's stale vcodec. An audio-mode download from a progressive source (info carries the source's video codec, embed-thumbnail on) was then classified VIDEO -> a Play button + a broken in-app <video> with no video track. Only the download path regressed (upload calls probe_media directly). Fix: gate precedence on whether ffprobe RAN, not per-field. If the probe read the file, its codec verdict wins outright (a None means 'no such track'); only a full ffprobe failure falls back to info. Dimensions still gap-fill from info. Refactored the parsing into _facts_from_probe so resolve_media_facts can see the raw probe-ran signal; probe_media keeps its 5-tuple (upload + TestProbeClassifier unchanged). - tests: the cover-art-audio and still-image resurrection cases, a differing populated-info precedence case (the prior-flagged gap), and the ffprobe-failed info fallback — now stubbing run_ffprobe_json so the real parsing + precedence run together. Backend gate green (ruff + full pytest). |
||
|
peter
|
a86131e062 |
fix(downloads): classify media_kind from a file probe, not yt-dlp info
A downloaded video was silently demoted to a download-only "file" (no in-app play/edit) whenever yt-dlp's info dict omitted vcodec/acodec — which it does for Facebook and other sources. Finalize derived media_kind from those nulls (`derive(None, None)` -> file), so a genuine H.264/AAC MP4 became non-playable in the library. Fix: probe the finished file on disk (the ground truth) at finalize and derive from that, falling back to the info dict only for a field the probe couldn't read. This is the same ffprobe the upload ingest already used, now extracted to a shared downloads/probe.py so the download worker and the upload path share one classifier (and can't drift). Also backfills the stored vcodec/acodec/width/height from the real file. Tests: TestResolveMediaFacts pins probe-wins-over-info (incl. the Facebook no-codec case and yt-dlp's literal "none"); upload's probe tests repoint to the shared module. |
||
|
peter
|
614c5528ad |
fix(downloads): harden upload serving + ingest against stored-XSS and leaks (E7 review)
Deep-review remediation for the E7 upload path, all findings adversarially verified: - Serve every library file with a Content-Type chosen from the ffprobe-verified media_kind + a strict container allow-list (storage.served_media_type), never guessed from the user-controlled file extension. Closes a stored-XSS vector where a media/HTML polyglot uploaded as "evil.html" was served inline as text/html on the login-free /watch surface. Applied to both the authenticated /file and the public watch_file endpoints. - Classify uploaded still images (jpg/png/gif/webp/…) as media_kind='file' so they are force-downloaded and don't get a broken in-app video player. - Roll back the on-disk side of an upload when the ingest transaction fails after place_file (unlink the placed file + its deterministic poster path), so a failed commit can't orphan bytes in the GC-skipped .uploads tree. - Floor download_upload_max_bytes at min=1 so a stored 0 can't silently disable the per-file cap. - Dedupe the ffprobe-JSON invocation into edit.run_ffprobe_json (shared by probe_codecs and upload._probe) and the upload extension into storage._clean_upload_ext; drop the now-unused parse_probe_streams. Tests: public + authenticated force-download and the polyglot-not-served-as-HTML guard, the real _probe classifier (image/cover-art/dims/duration), the ingest rollback/cleanup path, the max_jobs branch, the cap-floor invariant, and run_ffprobe_json error handling. |
||
|
peter
|
fe0b34cd00 |
feat(downloads): local file upload ingest — media library backend (E7 S5a)
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.
- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
gates library actions. Set at every finalize (download probe, editor carries the
source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
places it in a per-user .uploads tree, and creates an already-ready asset + done job
(the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
(download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
application/octet-stream) on both the in-app and public endpoints, so uploaded
HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
|
||
|
peter
|
94fcc40bde |
feat(downloads): guarantee every download is browser-playable (mp4/H.264/AAC)
The Download module's purpose is download -> share -> play in-browser, so the output must play in EVERY browser (iOS/Safari/WebKit included). The only universally-safe combo is mp4 + H.264 + AAC/MP3. - formats.normalize() now CLAMPS every spec to that profile (container->mp4, vcodec->None, audio_format opus->m4a) at the single point all paths flow through (picker, stored profile, raw API) — not just by hiding UI options. A source exposing no H.264 is still caught by the worker's browser-safe re-encode, so playability is guaranteed end-to-end. - build_ydl_opts drops the now-unreachable non-compat branches (+ _VCODEC_SORT). - _SIG_VERSION 3->4 so assets cached under an old non-compat spec — or a stale non-H.264 mp4 that slipped the v3 net (the HEVC Facebook dls) — re-derive. - ProfileEditor drops the container/codec pickers and Opus; audio is M4A/MP3. - i18n: remove the now-unused container/vcodec/any keys (HU+EN parity). |
||
|
peter
|
bc33b2ed16 |
feat(downloads): per-user, per-host yt-dlp cookie/auth support (S6)
Each user stores their own login cookies (Netscape cookies.txt) for an auth-walled download host (Facebook, Instagram, X, TikTok, Reddit) in Settings; a download of an auth-walled link on that host is retried with the acting user's cookie. Two load-bearing security controls, both in downloads.credentials as the single source of truth so enqueue and the worker can't drift: - cache privacy: a cookie-authed download salts its MediaAsset format_sig with the user id, so the authed asset is private and never cache-served to another user. The worker attaches a cookie ONLY when the asset it is downloading into IS that private, salted identity. - host scope (SEC-12): a cookie is resolved only for a URL whose netloc the credential owns (netloc-scoped _host_is); a look-alike/unrelated host gets none. cookies_enc is Fernet-encrypted at rest, only decrypted in-worker into a 0600 file under staging/.auth/ (survives the retry wipe + _produced_file scan) and unlinked in finally. The blob is never returned by a route, logged, audited, or written to a job's error/snapshot. cookiefile is set after the adapter's extra merge so an adapter can't clobber it. Settings UI is write-only (host + set/updated only, never echoed). New download_credentials table + migration 0074; per-user routes; HU+EN i18n. Tests: host-scope, cache-privacy (no cross-user serve), encrypt-at-rest, write-only API, worker attach-gate; FE dom test for the write-only field. |
||
|
peter
|
9ae6c47d6c |
perf(downloads): de-prioritize worker transcodes + idle-back-off polls to stop LAN slowdown
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API, Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and starved web requests + playback until it finished. Full-scope S4 mitigations: - Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp descendant): CPU nice increment + best-effort idle ionice class + an OMP_NUM_THREADS budget, all config-gated and independently disable-able. - Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode reserves a core for foreground work instead of claiming all of them. - Throttle the retention GC's whole-tree orphan sweep (yield IO after each batch of files) so its rglob can't monopolize disk IO in the API process. - Idle-back-off the Download Center (2s→15s when nothing queued/running) and UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the next fetch after an enqueue (invalidateDownloads refetches immediately). Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence, _deprioritize (nice/ionice/OMP branches), GC sweep throttle batching. |
||
|
peter
|
b7a6451dba |
fix(downloads): iPad audio playback + honest non-retryable failure for auth-walled hosts
5a — the public watch page rendered every asset in a <video>, but WebKit (all iOS browsers) refuses to play an audio-only mp3 in <video>, so shared audio links were broken on iPad while working on desktop Chromium. public_meta now flags audio-only assets (codec present, no video stream) and WatchPage renders a native <audio> in a padded surface card (no artwork exists for these). 5c — a login-walled Facebook post failed as a bare 'unknown', which offered a misleading Retry. Classify closed-group / cookie-suggestion auth walls as login_required and genuinely unsupported sites as a new unsupported_host class (both non-retryable), and drop unknown from the modal's retryable set — an unclassified failure is most often an unparsed auth wall. The queue row keeps its own force-retry for real transients. |
||
|
peter
|
b1832afd18 |
style: apply ruff format across backend + scripts (line-length 100)
Mechanical, whole-tree `ruff format` pass — no behaviour change. Recorded in .git-blame-ignore-revs (next commit) so git blame skips it. |
||
|
peter
|
cff8bed62b |
fix(downloads): heartbeat only when no write touched the row this interval (E3 review R2)
Round-2 review of the R1 fixes found that the two-independent-limiter progress writer issued a redundant heartbeat every interval during an ACTIVE download: a percent change writes progress (which already refreshes updated_at, the stall watchdog's liveness source), then the sub-integer calls that follow fire the separate beat budget — ~2 DB writes/interval instead of the throttle's intended 1. Replace the second limiter with a shared `last_touch` clock (last DB write OR beat). Progress writes keep their own `allow_write` budget (so a beat still can't delay a real write — the R1 concern), but the heartbeat now fires only when nothing touched the row for a full interval. Active download → ~1 write/interval; a genuinely stuck re-encode → 1 heartbeat/interval. New regression test: an unchanged-percent call in the same interval as a write does NOT heartbeat. ruff clean, full DB-lane 365 pass. |
||
|
peter
|
bc79720e81 |
fix(e3): address /code-review high findings (5)
Round-1 review fixes on the E3 branch:
- prune partial-fetch (correctness): `_paginate` now reports whether it reached
the server's reported `total`; `_prune_stale` prunes only on a COMPLETE
non-empty listing. A truncated fetch (a mid-sequence empty/short page before
total — a transient Plex hiccup) no longer deletes the rows it merely failed
to list. New test: a listing that claims 5 but returns 1 prunes nothing.
- `_prune_stale` (robustness): `stats.get("pruned", 0) + 1` — no KeyError when a
caller's stats dict omits the key.
- service transitions (conventions): pause/resume/redownload use the JobStatus
StrEnum instead of loose "queued"/"paused"/"error" literals, matching cancel.
- ffmpeg progress writer (correctness): a stuck-percent heartbeat now runs on its
OWN 1/interval budget, so it can't consume the progress-write slot and delay a
real percent change by up to an interval.
- backend/app/README.md (conventions): reword the transaction-ownership rule to
quote and extend CLAUDE.md's canonical ("handler/job-wrapper commits; helpers
flush; only quota owns its session") instead of stating a contradicting
"helpers MAY commit" carve-out.
ruff clean, full DB-lane 364 pass.
|
||
|
peter
|
b2013a079a |
refactor(downloads): split worker.py parent/child into pipeline.py + jobdb.py (E3 S2, B-sub-9)
worker.py (~1090 L) mixed the PARENT supervisor (claim + supervise + the process
loop) with the CHILD pipeline (the yt-dlp/ffmpeg download+edit that runs in the
spawned subprocess). Extract them:
worker.py 1090 -> 475 parent: claim/supervise/loop
downloads/pipeline.py (new 592) the child pipeline (_download/_process_edit +
hooks/progress/ffmpeg helpers + _Aborted)
downloads/jobdb.py (new 73) the short-lived-session helpers both sides
share (_set_job/_job_status/_touch_job/
_requeue_if_running/_reset_asset_pending +
the staging-dir paths)
The graph is one-directional (runner -> pipeline -> jobdb; worker -> jobdb; no
worker<->pipeline edge) — the parent never calls the child directly, it spawns
runner.py as a subprocess. Consumers rewired: runner import, and two tests follow
the moved symbols (_media_info/_produced_file -> pipeline; the ffmpeg-callback
patches target pipeline where the names are looked up).
Pure code motion, verbatim bodies. ruff clean, full lane 363 pass (unchanged).
|
||
|
peter
|
4ace486618 |
refactor(downloads): service owns the job state transitions (E3 S2, B-sub-8)
pause/resume/cancel/delete/redownload each hand-rolled their asset bookkeeping inline in the route (release, failed-asset reset, the 20-line fresh-asset rebuild), duplicating logic the worker/GC also touch. Move each transition into a service.* method (service.pause/resume/cancel/delete/redownload) that owns the mutation + its commit; the routes keep only auth, the HTTP 409 guards, and serialization. Drops the now-dead _set_status route helper and two stale imports. Pure behaviour-preserving code motion (routes -56 lines). Green under the E3 S2 transition net + the full downloads suite (363 pass), ruff clean. |
||
|
peter
|
f0a0214d97 |
fix(downloads): atomic ref_count increment/decrement (E3 S2, B-sub-5)
MediaAsset.ref_count (how many jobs hold a shared cache asset) was a Python read-modify-write at four sites (enqueue, enqueue_edit, release_asset, the redownload route). Two concurrent enqueues of the same (source,format) both read the same value and write the same +1 → the count ends up one short → the first release drives it to zero and deletes the file out from under a live holder. Replace all four with atomic DB ops: incr_ref_count (`ref_count = ref_count + 1`) and decr_ref_count (`greatest(ref_count - 1, 0)` RETURNING the new value, so the release decides whether to free the asset from the post-decrement count while holding the row's write lock — Postgres serialises concurrent releases on that lock, closing the double-free). Pure DB-atomicity change, no semantic shift. Green under the E3 S2 ref_count net (5) + full lane (356 pass), ruff clean. |
||
|
peter
|
0340d0981c |
fix(e1): address /code-review high findings on the E1 stack
Review round on the whole E1 stack (main..HEAD). 6 findings, all low/medium, none blocking; fixed for a clean gate-green handoff. - plex/sync (correctness): the un-enriched cast guard compared the stored cast against THIS listing's length, so a trimmed Plex cast on a not-yet-enriched row was never applied. Compare against the listing CAP (_LISTING_CAST_CAP=20) instead: >cap = enriched (keep, survives a watermark reset), <=cap = listing-sourced (track edits in both directions). +2 regression tests. - main (security): add X-Frame-Options: DENY as the legacy-browser fallback for the CSP frame-ancestors clickjacking guard. - downloads/service (altitude): finalize_if_running now uses a SQLAlchemy 2.0 update() construct instead of a hand-built text() UPDATE string — same atomic WHERE status='running', idiomatic, no identifier interpolation, no manual StrEnum coercion. Drops the now-unused text import. - auth/admin (reuse): extract the SEC-10 "drop untrusted preset password + bump epoch" invariant into one helper (revoke_preset_password_if_unverified) so the Google-callback and admin-activation paths can't drift. - tests/migrations (robustness): guarantee the shared schema is restored to head in a finally, so a downgrade/re-upgrade failure can't cascade into later tests. - worker (simplification): align _finish_from_cache's display_name fill with the _download path's guard (no behavior change). Gate green: ruff clean, 334 pytest (0 skip), frontend unchanged. |
||
|
peter
|
4ce0a702f4 |
refactor(downloads): route worker terminal writes through an atomic finalize_if_running (B-sub-1 review)
Follow-up to the /code-review high on the S1 branch. The per-site `if job.status == 'running'` guards were a non-atomic read-then-write (TOCTOU): a cancel committing between the child's status read and its commit could still be overwritten with done/error, re-opening the double-release window. Introduce service.finalize_if_running(db, job_id, **fields): one atomic `UPDATE download_jobs SET ... WHERE status='running'` that returns whether it fired. It flushes pending asset writes first so the asset-then-job row-lock order matches the cancel route (no deadlock), and runs in the caller's session so it commits with the asset writes in one transaction. All five worker sites (download/edit completion, both error paths, cache finish) now go through it, so the invariant lives in one place. The 'job done' logs move inside the finalized branch so a job stopped mid-completion no longer logs a false 'done'. Add tests/test_downloads_finalize.py: running -> finalizes; canceled/paused -> no-op. |
||
|
peter
|
367bcebc9f |
fix(downloads): revive the worker cancel branch and guard terminal writes (B-sub-1)
The cancel route writes status 'canceled' but the supervisor compared against 'cancelled', so its "a route stopped it -> kill the child, free the asset" branch was dead code. A canceled job's child ran to completion and wrote 'done' unconditionally over the route's 'canceled'; since 'done' holds the asset but 'canceled' does not, a later delete released the asset a second time, dropping ref_count below the live holder count and deleting a file still in use. Add app/downloads/status.py as the single source of truth for the lifecycle literals (JobStatus StrEnum + the HOLDING / STOPPED_BY_ROUTE sets) and use it at the safety-critical sites: the supervisor stop check, the route cancel, and release_asset's holding set. Make the worker's terminal done/error writes conditional on the row still being 'running' so a child finishing just after a route cancel/pause can no longer resurrect it. Add the missing 'canceled' supervisor regression test. (The full literal sweep across the downloads surface is left for the E3 downloads owner-unification; this is the data-safety subset.) |
||
|
peter
|
e025e0f719 |
fix(downloads): floor max_concurrent + stable share order (R12 S5 review)
- Floor the effective max_concurrent at 1 on the read path (quota.resolve) and in the claim SQL (GREATEST(..., 1)). admin_set_quota already floors on write and the sysconfig default's min is 1, but a stray stored 0 (direct DB edit) would diverge the two "at limit" checks — the SQL claim-window exclusion can't see a user "capped" with zero running jobs, while at_concurrency_limit can — reopening the starvation the fairness fix closes. Keeping the effective limit >= 1 makes the two checks provably agree. - shared_with_me now orders by (created_at DESC, id DESC) to match list_downloads, so the newly-added 500 cap has a stable boundary among same-timestamp shares instead of a non-deterministic one. - 0062 upgrade loop: rename the unused loop var "table" to "_table" (matching the underscore-marked "_col"). Test: a stored max_concurrent=0 flooder no longer wedges the claim loop. |