fc56f514b33c2fde38cd5900fd3a5f2b54a64d24
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 peter
|
fc56f514b3 |
fix(e12): the restart hint pointed at the one command that undoes a rollback
Round three, on round two's diff. The trend across the loop is 47 -> 34 -> 15 raw findings; the ops half kept producing the sharpest ones because the gate cannot run it. **`docker compose start` ignores a newly pinned IMAGE_TAG.** Round two removed the automatic restart because it re-ran `alembic upgrade head` — and then told the operator to run that exact command by hand, in both scripts and the docs. `start` relaunches the container that already exists, which still carries the tag being rolled back FROM, so pinning 0.79.0 and following the printed instruction brings 0.80.0 back up and re-applies its migrations. Verified against a real daemon: after changing the pinned tag, `start` kept the old image and only `up -d` picked up the new one. Both scripts and the doc now say `up -d`, with the pin-first caveat where the operator reads it. **The reset flag outlived its token.** Reverting round one's "each link clears the other" went one step too far: a `#verify=` link pasted over a standing reset notice replaces the whole fragment, so the reset token is gone — but `resetPending` stayed true, and being checked first it hid the confirmation while promising a recovery that signing out could no longer complete. The flag is now cleared when the fragment it describes is gone, which is narrower than what was reverted and does not blank the signed-out card. **A test that could not fail.** The shared-asset regression test seeded the asset as `pending`, and `_reset_asset_pending` only acts on `downloading` — so the release half was a no-op and deleting it kept the suite green. Seeded as the worker actually leaves it now. Also: the malformed-URL guard answers with urlguard's structured detail rather than a bare English string, so both halves of one typo class get the same localized copy, and the third site in that module handing user input to `urlparse` was swept with it; the asset is released after the job is terminal, not before; the Stats 403 toast gets the wizard handle its own comment claimed it had; the pasted Windows recipe sets `$ErrorActionPreference` inside the block (a scriptblock inherits the caller's, so a `SilentlyContinue` profile swallowed every throw) and cleans up the container-side dump on the failure path too; the restore warning moved to the warning stream so a `> log` redirect cannot file it away; and two comments narrowed to what the code does. Gate green: vitest 685, pytest 950. |
||
|
peter
|
358d722ce9 |
fix(e12): close what round one broke, and revert two fixes that overreached
Round two of the review loop, on round one's own diff. Two of its findings were regressions the fixes themselves introduced, which is exactly what the loop is for: **A per-user refusal was writing onto a shared row.** MediaAssets are deduped across users, and the generic failure handler puts the message on the asset — right for an upstream failure, wrong for "your storage is full". Two users queuing the same video share one asset, so the first user hitting their cap failed the second user's job with the first one's private reason, telling someone with an empty library to free up space. The refusal now raises `_OwnerRefused`, caught separately: the job fails, the asset goes back to `pending` for whoever else holds it. Pinned by a test that drives `_download` directly — the refusal happens before extraction, so nothing needs stubbing. **The restore scripts restarted the app, undoing the restore.** `docker compose start` re-runs the container command, and that runs `alembic upgrade head` — so on the rollback path this documents as recommended (restore the dump, THEN pin the older tag) the newer image immediately re-applied the migrations the restore had just undone, and on a FAILED restore it put the app back on a half-restored schema. Both are the opposite of why the stop was added. Nothing is restarted automatically now; the scripts say what to run and when, and the docs' snippets say the same. **Reverted rather than patched:** the OnboardingWizard toast (the wizard already renders that failure inline, next to a Retry button — three recoveries for one failure), and `useVerifyEntry`'s "each new link clears the other". The latter blanked the signed-out card's only banner, where nothing renders the reset flag to replace it. The two states are independent again; what changes is that `dismiss` peels ONE layer, which is what the original finding was actually about. Also: `Stats` uses the YouTube helper rather than the bare one, so the route's English 403 detail no longer beats the translated fallback under a Hungarian UI; `urlparse` guarded alongside `.port` (an unclosed bracket raised just as late), the reason renamed `malformed` since "bad port" misdescribes it; three docstrings narrowed to what the code actually does (the worker's byte check is per-job, not an aggregate ledger, and not on the edit path; the redirect residual leaks a reachability oracle, which the note had denied; the invalidation helper does not cover suspend); the pasted Windows recipe wrapped so `throw` can stop it; the bash one-liner's failure branch no longer exits 0; and the tests that were missing for the round-one fixes — the account panel's error paths, the invalidation set, and the shared-asset case above. Gate green: vitest 685, pytest 950. |
||
|
peter
|
fe38dfbcca |
fix(e12): close the eighteen findings the E12 review round left standing
The deep review of the branch raised 47 findings, 18 of which survived adversarial verification. Grouped by what they actually were: **The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one user over quota was the only one downloading with no ceiling at all. `check_enqueue` could not catch it: it ran while the job was queued and the user was still under the cap. The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision lives, and where it can be tested), and the worker now fails the job outright rather than proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help until space is freed. **Silent failures in the new account panel.** All three actions went through `quiet: true` endpoints, which suppress the global error dialog on the promise that the caller reports inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On the broken-SMTP instance this panel exists for, an admin would read silence as "the mail went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`, shared with the pending-access queue, so the two copies of these mutations cannot drift apart again — which is how half of this was lost in the first place. **A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps back one migration, but a release routinely ships several (0.80.0 shipped four): it drops the newest one's columns and leaves the schema at a revision the older image also cannot resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is now the recommended path, and the alembic route asks the target version for its own head revision instead of counting backwards. Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as a 500; the pull cooldown extended to the sibling route spending the same QuotaAction; restore stops the app before dropping its tables; the PowerShell cleanup can no longer decide the script's verdict on 5.1, nor report the wrong exit code; the documented one-liners hardened to match the scripts; the registry's retention window stated where pinning is advertised; the two banners made mutually exclusive so neither can shadow the other; `hover:bg-hover` (a utility this project never generated) replaced by the `selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to the two callers that discarded it. Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset banner UAT'd on localdev. |
||
|
peter
|
0cc450d1c9 |
fix(downloads,sync): bound what one user can make the server fetch, download and spend
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches it from the WORKER, so any approved user could aim the server at the compose network, the LAN or a cloud metadata endpoint — and read the result off job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a port scanner. downloads/urlguard.py refuses a URL whose host resolves to a loopback/private/link-local/reserved address, at enqueue AND again in the worker (DNS can move in between, and the worker is where the socket opens). A network-class failure on a `url` job now reports one fixed sentence instead of the raw transport text; the error CODE is still classified from the original, so the localized explanation is unchanged. DNS rebinding is explicitly NOT closed — that needs the connection gated inside yt-dlp — and the module says so rather than implying more than it does. SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video, so a URL that is purely a playlist or channel still expanded into every entry — one enqueue, unbounded downloads. `playlist_items` pins it to one. Size was unbounded too: `check_enqueue` compares the CURRENT footprint against the cap, and a download's size is unknown until it lands, so one job could overshoot by any amount; the owner's remaining budget is now handed to yt-dlp as `max_filesize`, applied AFTER the adapter merge so no source adapter can raise a limit the quota layer set. A join's cut-list is capped at 100: an accurate join emits two filters per segment into one ffmpeg filter_complex. SEC-14: the YouTube budget is instance-wide, so one user holding down a manual sync spent the day for everyone. Per-user (not per-IP — the cost belongs to the account) 3-per-10-minutes limiters on the two bulk pulls, subscriptions and playlists, with separate budgets so one does not close the other. The review also listed /rss, /backfill and /enrich; E3 deleted them, and search stays unlimited on purpose — it is interactive, not a bulk pull. Gate: pytest 896 -> 924 (none skipped), vitest 671. |
||
|
peter
|
a86131e062 |
fix(downloads): classify media_kind from a file probe, not yt-dlp info
A downloaded video was silently demoted to a download-only "file" (no in-app play/edit) whenever yt-dlp's info dict omitted vcodec/acodec — which it does for Facebook and other sources. Finalize derived media_kind from those nulls (`derive(None, None)` -> file), so a genuine H.264/AAC MP4 became non-playable in the library. Fix: probe the finished file on disk (the ground truth) at finalize and derive from that, falling back to the info dict only for a field the probe couldn't read. This is the same ffprobe the upload ingest already used, now extracted to a shared downloads/probe.py so the download worker and the upload path share one classifier (and can't drift). Also backfills the stored vcodec/acodec/width/height from the real file. Tests: TestResolveMediaFacts pins probe-wins-over-info (incl. the Facebook no-codec case and yt-dlp's literal "none"); upload's probe tests repoint to the shared module. |
||
|
peter
|
fe0b34cd00 |
feat(downloads): local file upload ingest — media library backend (E7 S5a)
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.
- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
gates library actions. Set at every finalize (download probe, editor carries the
source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
places it in a per-user .uploads tree, and creates an already-ready asset + done job
(the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
(download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
application/octet-stream) on both the in-app and public endpoints, so uploaded
HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
|
||
|
peter
|
bc33b2ed16 |
feat(downloads): per-user, per-host yt-dlp cookie/auth support (S6)
Each user stores their own login cookies (Netscape cookies.txt) for an auth-walled download host (Facebook, Instagram, X, TikTok, Reddit) in Settings; a download of an auth-walled link on that host is retried with the acting user's cookie. Two load-bearing security controls, both in downloads.credentials as the single source of truth so enqueue and the worker can't drift: - cache privacy: a cookie-authed download salts its MediaAsset format_sig with the user id, so the authed asset is private and never cache-served to another user. The worker attaches a cookie ONLY when the asset it is downloading into IS that private, salted identity. - host scope (SEC-12): a cookie is resolved only for a URL whose netloc the credential owns (netloc-scoped _host_is); a look-alike/unrelated host gets none. cookies_enc is Fernet-encrypted at rest, only decrypted in-worker into a 0600 file under staging/.auth/ (survives the retry wipe + _produced_file scan) and unlinked in finally. The blob is never returned by a route, logged, audited, or written to a job's error/snapshot. cookiefile is set after the adapter's extra merge so an adapter can't clobber it. Settings UI is write-only (host + set/updated only, never echoed). New download_credentials table + migration 0074; per-user routes; HU+EN i18n. Tests: host-scope, cache-privacy (no cross-user serve), encrypt-at-rest, write-only API, worker attach-gate; FE dom test for the write-only field. |
||
|
peter
|
9ae6c47d6c |
perf(downloads): de-prioritize worker transcodes + idle-back-off polls to stop LAN slowdown
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API, Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and starved web requests + playback until it finished. Full-scope S4 mitigations: - Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp descendant): CPU nice increment + best-effort idle ionice class + an OMP_NUM_THREADS budget, all config-gated and independently disable-able. - Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode reserves a core for foreground work instead of claiming all of them. - Throttle the retention GC's whole-tree orphan sweep (yield IO after each batch of files) so its rglob can't monopolize disk IO in the API process. - Idle-back-off the Download Center (2s→15s when nothing queued/running) and UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the next fetch after an enqueue (invalidateDownloads refetches immediately). Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence, _deprioritize (nice/ionice/OMP branches), GC sweep throttle batching. |
||
|
peter
|
b1832afd18 |
style: apply ruff format across backend + scripts (line-length 100)
Mechanical, whole-tree `ruff format` pass — no behaviour change. Recorded in .git-blame-ignore-revs (next commit) so git blame skips it. |
||
|
peter
|
cff8bed62b |
fix(downloads): heartbeat only when no write touched the row this interval (E3 review R2)
Round-2 review of the R1 fixes found that the two-independent-limiter progress writer issued a redundant heartbeat every interval during an ACTIVE download: a percent change writes progress (which already refreshes updated_at, the stall watchdog's liveness source), then the sub-integer calls that follow fire the separate beat budget — ~2 DB writes/interval instead of the throttle's intended 1. Replace the second limiter with a shared `last_touch` clock (last DB write OR beat). Progress writes keep their own `allow_write` budget (so a beat still can't delay a real write — the R1 concern), but the heartbeat now fires only when nothing touched the row for a full interval. Active download → ~1 write/interval; a genuinely stuck re-encode → 1 heartbeat/interval. New regression test: an unchanged-percent call in the same interval as a write does NOT heartbeat. ruff clean, full DB-lane 365 pass. |
||
|
peter
|
bc79720e81 |
fix(e3): address /code-review high findings (5)
Round-1 review fixes on the E3 branch:
- prune partial-fetch (correctness): `_paginate` now reports whether it reached
the server's reported `total`; `_prune_stale` prunes only on a COMPLETE
non-empty listing. A truncated fetch (a mid-sequence empty/short page before
total — a transient Plex hiccup) no longer deletes the rows it merely failed
to list. New test: a listing that claims 5 but returns 1 prunes nothing.
- `_prune_stale` (robustness): `stats.get("pruned", 0) + 1` — no KeyError when a
caller's stats dict omits the key.
- service transitions (conventions): pause/resume/redownload use the JobStatus
StrEnum instead of loose "queued"/"paused"/"error" literals, matching cancel.
- ffmpeg progress writer (correctness): a stuck-percent heartbeat now runs on its
OWN 1/interval budget, so it can't consume the progress-write slot and delay a
real percent change by up to an interval.
- backend/app/README.md (conventions): reword the transaction-ownership rule to
quote and extend CLAUDE.md's canonical ("handler/job-wrapper commits; helpers
flush; only quota owns its session") instead of stating a contradicting
"helpers MAY commit" carve-out.
ruff clean, full DB-lane 364 pass.
|
||
|
peter
|
b2013a079a |
refactor(downloads): split worker.py parent/child into pipeline.py + jobdb.py (E3 S2, B-sub-9)
worker.py (~1090 L) mixed the PARENT supervisor (claim + supervise + the process
loop) with the CHILD pipeline (the yt-dlp/ffmpeg download+edit that runs in the
spawned subprocess). Extract them:
worker.py 1090 -> 475 parent: claim/supervise/loop
downloads/pipeline.py (new 592) the child pipeline (_download/_process_edit +
hooks/progress/ffmpeg helpers + _Aborted)
downloads/jobdb.py (new 73) the short-lived-session helpers both sides
share (_set_job/_job_status/_touch_job/
_requeue_if_running/_reset_asset_pending +
the staging-dir paths)
The graph is one-directional (runner -> pipeline -> jobdb; worker -> jobdb; no
worker<->pipeline edge) — the parent never calls the child directly, it spawns
runner.py as a subprocess. Consumers rewired: runner import, and two tests follow
the moved symbols (_media_info/_produced_file -> pipeline; the ffmpeg-callback
patches target pipeline where the names are looked up).
Pure code motion, verbatim bodies. ruff clean, full lane 363 pass (unchanged).
|