PlexLibrary, PlexShow, PlexSeason, PlexItem, PlexPerson, PlexCollection, PlexState, PlexLink,
PlexPlaylist and PlexPlaylistItem leave app/models.py for app/plex/models.py; app.tables registers
them. Every import is re-pointed; no schema change.
The nine Capture tables (files, jobs, folders, shares, links, quota, cookies, profiles, library
resume points) leave app/models.py for the unit's own module; app.tables registers every unit's
tables for the migrations, the test database and the ownership check. No schema change.
- Files toolbar link box (paste button, Enter); a lone http(s) link pasted onto any page opens the one
app-wide download dialog, which now names the link; the palette's "Download from a link…" focuses
the box on its history entry.
- The download tray, bottom right on every page: what waits, runs, is paused or failed (reason +
Retry, the queue's right-click / "…" menu), and what finished in this tab as "Done · Open" until
closed; folds to a pill (per-account pref); closed, it stays shut — across a reload — until the
work changes or something finishes. Back to top steps up over it, the page end keeps clear.
- A finish is announced once: "Downloaded: …", whether a phone plays it, Copy link + Open (toasts
and notices may carry a second action).
- The inbox's ready notice opens the file in its folder: download_done carries job_id + folder_id.
- GET /api/downloads is the queue only (in flight, failed, finished in the last 10 minutes), with
each row's source key.
The Files library starts here. A user's every download, clip and upload now
lands in a source folder made per origin (YouTube, Plex clips, uploads, one per
other site), decided once in downloads/folders.py at every place a job is made.
How long a file stays moves from the shared asset to the user's job: stamped
from its folder when it is done, never for a file the user keeps. The GC warns
each user once, then takes the expired file out of their library; the cache cap
never evicts a kept file, and an evicted one leaves every holder's library.
The media mount is /files now (FILES_ROOT, FILES_HOST_PATH, volume
siftlode_files); docs/self-hosting.md says how to carry an old volume over.
Migration 0102 places every existing job and carries each done file's expiry
over from its asset.
safe_abs_path takes a missing path and answers None, which drops one baselined
type error.
yt-dlp's `max_filesize` only stops a plain HTTP file that announces its size: YouTube's fragmented
formats never meet it — measured on localdev, a 718 MB download ran to the end with 1 MB left and
put its owner four times over the cap. The download now re-reads what is left once its file is made
(after the phone-safe pass) and refuses one larger than that as the owner's refusal; an editor clip,
which nothing checked, is held the same way, as a player clip already was.
yt-dlp refuses a file announced over `max_filesize` by logging one line and returning with no file
(measured, and pinned in test_formats against a local server): the worker found no output and
failed the job as `postprocess`. `formats.watching_size_cap` now sees that line through our yt-dlp
logger, and the download (and, should it ever happen, a YouTube clip's range fetch) fails as the
owner's refusal with DOWNLOAD_TOO_BIG_MESSAGE, classified `quota_exceeded`. The download's quota
copy says it does not fit in what is left — true at a full cap too — instead of "Storage full".
A pre-existing type error the change touched is fixed (baseline down by one).
- A converted upload over the space left (not zero) fails with CONVERTED_TOO_BIG_MESSAGE, not
"storage is full"; it classifies as quota_exceeded by the phrase it shares with the clip's.
- The public watch page says "preparing" only for a job a worker will take: a suspended owner's
queued job waits for as long as the suspension lasts, so the link says the video is gone, as
for a paused rebuild.
- The worker's claim docstring names the restart case: a running job put back in the queue waits.
A suspension now blocks the user's downloads as quota.resolve already advised: the worker leaves
their queued jobs unclaimed until it is lifted (the confirm says so). A cut clip larger than the
remaining storage is refused with its own message (not "storage is full"), and a clip's quota
refusal reads as a clip's in both languages. The cancel poll compares to JobStatus.RUNNING.
- The cut file is measured against what its owner has left, for a YouTube range (whose fetch never
reads yt-dlp's size ceiling) and for a Plex clip (which fetches nothing), before the fetch and
again before it is placed; a refusal leaves the asset pending for a Retry.
- A silent Plex file is cut silent; a file the probe cannot read says so.
- A second job on a failed or renamed clip gets the same YouTube diagnosis and the fetched title.
- The "too long" refusal names the server's cap; a contract test holds the client's copies of the
clip limits to clip.py; the shipped player clients are pinned against the installed yt-dlp.
- One HDR transfer set and one rating_key lookup; job status words from JobStatus; the display-name
guard sees subscript writes and the clip route; stale comments and types follow the clip job.
An optional crop in clip mode: a Crop toggle puts a frame on the picture (free, 16:9, 9:16 or 1:1;
the whole picture stays the default). The frame is sent in fractions of the box it was drawn in,
with that box's shape; the worker fits the real picture into the box by its display aspect (sample
aspect and rotation aware), drops the part over the bars, and crops first in the filter chain. A
frame covering the whole box is no crop at all, and one off the picture is refused by name.
The crop frame is a ui primitive (CropFrame) shared with the video editor, keyboard-operable.
YouTube: the wide side panel shuts while clipping and comes back after; maximised, the clip panel
docks in the stage. Plex: the picture sits above the clip panel so the frame lies on it.
A clip fetched its range signed out, so a video behind YouTube's per-video "not a bot" wall could
never be clipped while its download went through with the user's cookie - and the failure copy told
the user to add a cookie the clip would not use. The clip asset is private by construction
(`clip.asset_ref` names its user), so `usable_clip_cookie` attaches the user's YouTube cookie on that
identity, and a failed range fetch now carries the download catalog's diagnosis
(`youtube.bot_wall` / `youtube.cookie_rejected`) instead of a bare category.
POST /api/downloads/clip takes a source (a YouTube id or a Plex rating key), a range of at most
ten minutes and, for Plex, the audio track being heard. The clip becomes a per-user asset in the
library like an editor clip, shareable as soon as it is ready. A YouTube clip fetches only its
range (yt-dlp download_ranges: 3-7 s whatever the video's length); a Plex clip reads the library
file in place. Both then go through one re-encode: frame-exact at the marks, H.264 at most 1080p
with HDR tonemapped to SDR, stereo AAC, no chapter track, .mp4.
Job kinds are a JobKind StrEnum now, and Re-download refuses every kind but a download.
- The fast join gives each ffconcat entry its own media-only format whitelist, so a library file
can no longer be opened as a nested concat list (the list's whitelist was handed down to it).
- A file the guarded probe cannot read is a download-only file, not a guess from yt-dlp's info.
- A conversion's or an ingest's commit that raises is re-read in a fresh session before any file
is removed: a commit that landed keeps the file its row now names.
- An original that cannot be removed after conversion is logged; the upload stays ready.
- An upload whose conversion is cancelled or deleted midway is freed once nothing holds it; a
paused one is kept.
- A shared link says "preparing" only while its video's job is queued or running.
- The admin storage view counts every file on disk ("Files on disk"), not only ready ones.
- A clip's failure title, unknown failure and cancel dialog, and a clip's and an upload's delete
dialog speak of what the job made; an upload's quota refusal explains that the original is kept.
- /watch announces "preparing" as an alert and moves focus to the new Retry after an action.
- Every ffmpeg/ffprobe over a library file names the demuxers it may use
(edit.INPUT_GUARD; a downloaded thumbnail edit.THUMB_GUARD): a playlist-type
demuxer would read local paths named inside a user's upload.
- A re-encode ends on an even frame (edit.EVEN_FRAME): an odd 4:4:4/RGB source
failed its conversion; a clip stores the frame its file has.
- An upload counts against its owner's cap from the moment it lands (queued, or
kept after a failed conversion) — the footprint, the GC's cap and the admin
view read one ON_DISK predicate — and a conversion that outgrows the cap is
refused, keeping the original.
- The converted name keeps _a{id} (a dotted bare name once lost it and two
uploads shared a file); a converted file placed but not committed is removed,
and the orphan sweep enters .uploads; the LRU cap never evicts an upload.
- Ingest judges the extension the file is stored under; a failed conversion is
convert_failed / upload_missing, reports the "converting" phase, and notifies
as an upload. Job statuses in routes/downloads use JobStatus.
A share link is opened on whatever the friend holds. Measured on an iPhone and an iPad: AV1,
Matroska, HEVC tagged hev1 and MP3 inside an .mp4 fail there; Chrome and Firefox add HEVC and
10-bit / 4:4:4 H.264. The old check judged the codec NAME only and ran on downloads only, so a
10-bit H.264 passed, an upload went out exactly as it came (an iPhone's HEVC .mov, a webm, an mkv)
and a fast trim kept its source's codecs.
One rule now, `downloads/playable.py`: video is H.264 8-bit 4:2:0 + AAC in .mp4, audio-only is
.mp3 or AAC .m4a; `plan` says what must change, `build_cmd` re-encodes only the failing stream (or
remuxes). The worker applies it to a download, an editor clip and — new — an upload: media that
fails the rule is queued (asset pending, job queued) and the worker converts it, keeping only the
converted file (owner's choice). Until then the original sits at rel_path, so Retry starts from it
and a delete removes it; an upload that lost its last holder mid-conversion is reaped by the GC.
Every libx264 run (edit, concat, conversion) now pins `-pix_fmt yuv420p` — without it a 10-bit
source came out High 10. The download cache signature moves to v5 so a cached 10-bit asset
re-derives. The upload toast says when the file is still converting (HU/EN).
Tests: the plan for every measured class; real ffmpeg turning a 10-bit Matroska + Opus and an
mp3-in-mp4 into files the rule accepts (red without the pixel-format pin); the queued upload's
delete, cancel-then-GC, failed conversion and real worker conversion (the cleanup tests red
without their rules); the toast (red without the branch). Live on localdev: 9 fixture uploads
converted, every served file probed H.264 High yuv420p + AAC, or mp3 / m4a.
The code cited its plan instead of saying why: `E28 review J4`, `(R6 S3b)`, `S1 justified…`,
`deep-review round 5`. That plan lives in the private ops repo, so for every reader of this one the
citation was noise, and a sentence whose subject was the plan step ("round 2 showed…") hid its
lesson behind an id. Each one is gone and the "why" kept: a bare citation is deleted, a sentence
carrying a lesson is reworded around it ("a later attempt showed…").
Measured (`scripts/audit_metrics.py`, product code): 652 lines in 270 files carried an epic code by
the definition the next commit holds (507 / 241 by the old, two-digit-only one) -> 0. Across all
code, tests, scripts and configs: ~1100 epic-code lines plus ~320 sprint/round ids.
Behaviour unchanged, checked mechanically: every changed .py file has the same AST once docstrings
are blanked; css/sh/toml/yml/json files are identical once comments are stripped; the only
non-comment changes in TS/JS are test titles, one description string in a contract table, and five
ESLint messages, whose text is all the gates snapshot diff holds. Not touched: Markdown (docs and
the changelog keep their history), Alembic migrations, TV episode notation (`S2 · E5`), DESIGN.md
ids (`C1`, `B4`, `S3h`), tracker ids (`SEC-16`, `DX-17`).
One stale claim fixed on the way: test_hls_reaper's docstring said the player checkpoints "every
10 s whether or not it is playing", which plex/stream.py calls wrong; it now says what the code does.
Deep-review round 2, on round 1's own fix commit. It found MORE than round 1
did (28 raw, against 13), and most of it was mine -- a new piece of UI attracts
findings a backend-only diff never does. The pattern in almost all of them is
the same: a principle applied to the site that prompted it and not to its
siblings.
* `display_text` reached three of the FIVE writers of a user-supplied name.
`POST /api/downloads` and `POST /api/downloads/edit` still handed Postgres an
unbounded string, so a 300-character name pasted into the Add-download dialog
or the editor was a bare 500 (main.py re-raises every DataError but the NUL
case). Four of the seven reviewers found it independently. The bound now lives
in `models.py`, the three columns are DECLARED with it, and the guard for the
next writer is parsed rather than remembered: every `display_name` /
`display_uploader` assignment under `app/routes/` must go through the helper.
That guard immediately found two more -- the Google sign-in callback writes
the profile name into the same 255-wide column -- so it is six, not five.
* The connection release reached one of the THREE routes that read their own
body. The screenshot route (10 MiB, from a phone) and the client log sink both
still held a pooled connection idle-in-transaction for the length of the
upload, which is the invariant db.py sizes the pool on. All three now call
`db.release_read_connection`, whose precondition -- nothing pending on the
shared session -- is CHECKED rather than described, because a whole-session
rollback discarding a caller's staged write is a mistake this repo has already
made once. `test_body_limit.py` holds the rule for the next such route.
* The AST drift registry keyed its sites by (module, limit expression), so a
second route copied from the first collapsed onto the same entry and went
uncovered -- the exact hole the registry exists to close. Keyed per call site
now, and it carries the release requirement too.
* The dialog's own refusal was announced once and never again: two over-cap
picks leave an identical node with identical text, and an untouched live
region says nothing. It is keyed per pick now, the drop zone reads as invalid
and points at the sentence, and the reason is reachable from the control
rather than only from a one-shot announcement.
* Two tests rebuilt the blind spot they were written to remove: the dialog's cap
test sat nowhere near the boundary (a `>` loosened to `>=` survived it, which
is the mutation round 1 fixed on the server side), and the header-precedence
test only covered an exact-case duplicate while a differently-cased one would
have been COMBINED onto the wire rather than replaced. Both fixed, and the
merge now dedupes case-insensitively because header names are.
Also: the upload dialog's unstubbed usage query was firing a real request from
jsdom in a neighbouring suite and leaking a delayed "connection lost" toast
across test boundaries; `GET /api/downloads/usage` had no test at all though the
whole client-side check hangs off it; and the comment claiming the edge 413 is
readable now says what was measured -- readable to 200 MiB over loopback, a bare
`onerror` through a proxy -- so the pre-send check is named as the mechanism and
that branch as the backstop it is.
Every test added or changed here was mutation-checked. One mutation SURVIVED and
is recorded rather than papered over: the `tooBig` term inside `submit()` cannot
be reached, because `ui/Button` swallows the activation itself, so it is marked
as belt and braces instead of being described as the guard.
- The download and edit failure lines go back after their DB write: a job the
write could not fail is not a failed job, and the worker's child_exit line
already says it failed.
- The download child's runner is the last stop before Python's hook: an
escaping exception is one download.child.failed line with its chain scrubbed,
then exit 1, instead of a raw stderr traceback carrying the URL.
- A crash the database will not record carries its stack once, and the
fail-write line has an event code.
- The job's own generic-extractor name is taken out as a literal (formats.hiding),
so brackets and colons in a direct link's file name no longer leak; the
pattern stays as the fallback. `Downloading just the video <id>` is covered.
- Tests: scrubbed_stack's cause, suppress, unprintable and text-format paths,
the image steps' failure kinds, the watchdog error through the real
_fail_running, exact poster/storyboard messages, ffmpeg skip marker.
- A failure keeps its traceback unless it is yt-dlp's or our own StatedFailure (a
message that is the whole story). A plain RuntimeError, RecursionError or OSError
is also what a bug of ours raises, and its stack is the only lead.
- The stack is scrubbed link by link (logs.scrubbed_stack): a chained DownloadError
under a DB error no longer prints yt-dlp's text unscrubbed.
- Engine hide_parameters: a DB error's message no longer carries the title or path
it was writing.
- The download and edit failure lines go out even when their DB write fails.
- The worker's child_exit/crash/supervisor lines say download.job.failed only when
the job failed there; a crash the DB will not record no longer ends the thread.
Stopping a paused child is download.child.stopped, not a second abort.
- log_safe also covers generic ids with spaces, playlist titles and bare ids.
- The ffmpeg image steps log the failure kind, not the argv with its path.
- The consent contract accepts only the body of a plain consented_trace() check,
and any level that may be TRACE.
- Tests for each; every one mutation-checked.
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
`formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
file paths, and the job-failed, refused, edit-failed and backfill lines use
it. Done lines log the asset id, not the title-built path. The PII contract
now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
a process's last batch closes the repeat window, so a short-lived download
child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
`download.job.failed` with a cause; they left no line at all before. The
crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
carry reason `transport`. Subscribe recognises a duplicate by reason code
only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
The gap this sprint was scoped against turned out to be smaller than the survey
said, and in a way worth writing down: `log_requests` already records every HTTP
call with its route template, status, duration, request id and user id, and the
audit log already records every admin act on someone else's account. Putting a
line in each of routes/downloads.py's twenty mutating endpoints would have added
a second copy of the first and nothing else. So this instruments what neither of
them can see.
yt-dlp now explains itself to our log. `quiet: True` was doing more than quieting:
with no `logger` its whole account of a download went to the child's stdout, so
"Requested format is not available", a signature-extraction failure or a POT
refusal reached us only as whatever text ended up in the raised exception — the
real reason a download failed was routinely one line the app never kept. Measured
against the pinned yt-dlp 2026.07.04 before wiring it: with `logger` set it calls
`debug` for extraction and progress lines and `warning`/`error` for the rest, and
neither `quiet` nor `no_warnings` gates those calls. `debug` lands at TRACE, being
several lines a second, so the noisy half is the half the user's own consent
gates. The full source URL is trimmed to its host on the way in — it is a row of
someone's download history, and `logs.set_job` already puts the job id on every
one of these lines. The option is set AFTER the adapter merge, like `cookiefile`
and the size cap: a source adapter must not be able to take away the one account
of what went wrong that a failed job leaves behind.
An HLS session now says which pipeline it got (transcode/remux, hw, tonemap,
variants, offset) — everything downstream is about that choice, and it was only
inferable from an ffmpeg command line nobody keeps. A session culled by the
concurrent-transcode cap says so: somebody's playback stops there, the route turns
it into a retryable 404, and "my film keeps rebuffering" had no other explanation
anywhere. The HW→SW fallback keeps its line and gains an event code.
A live search refused by a quota cap says WHICH cap: the 429 is in the request
line either way, but the per-user daily cap is answered by tomorrow and the shared
budget by the admin.
The download and edit job lifecycle lines gain event codes, which is what E22.6's
console filters on — the job id was already a structured field.
tests/test_ytdlp_logging.py; three of its four cases mutation-checked. A fourth
(a `%` in a progress line) survived its mutation — stdlib only applies `%`
formatting to a record that has args — so it is gone rather than kept as a case
nothing can break; the reasoning moved into the comment on `_emit`.
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.
What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
degraded to a software transcode with nothing saying why; a failed audio probe
collapsed a multi-audio file to one track and the player showed no language
picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
own root, leaving every crashed session's segments on disk forever.
Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.
ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.
logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.
tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
25 literal names covered 34 modules. plex/client.py, plex/stream.py,
plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves
`siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`;
routes/channels.py called itself `siftlode.api`. The `logger` column is in every
stored row so E22.6's console can filter on it, and so an operator can pin a level
to one part of the system — neither works when six modules share a name.
`logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync.
One name per module, and it cannot drift from the file it sits in.
tests/test_logger_names.py pins the derivation, that everything stays under the
one `siftlode` tree the handler and the tests attach to, and that no module
hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`).
Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic
it carries rather than the module it lives in — it replaced uvicorn's access log
and is what an operator pins when they want request lines and nothing else.
app/scheduler.py called its logger `logger`; every other module calls it `log`.
ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them:
worker.py logged `str(exc)[:120]` where the traceback was the whole question, and
three prints/f-strings that are deliberate now say so on the line. S110 and BLE001
stay off until S3 — 23 handlers are outstanding and each is a judgement about one
of them; enabling the rules first would just mean 23 blanket noqas.
Since a Facebook cookie was added on prod, every reel came down titled
"Video": measured on dev, a signed-in read gives yt-dlp the placeholder
"Video" (an 18+ reel read as Chrome gives "Facebook") while the post's text
is still in the description. A new SourceAdapter.title hook takes the whole
first line of the description in that case, and both title writes (the early
fill and the finished download) go through it, so the row, the file name and
the share preview carry the post's text again. The line is not cut with an
ellipsis: normalize_title's NFKC turns "…" into "..." and collapses it to
".", which read as a finished sentence (seen in dev UAT).
The "Audio (MP3)" profile's extracted .mp3 was remuxed into an .mp4 by the
browser-safe step (the audio spec is clamped to container mp4 too) and served
as video/mp4, which iPad's WebKit will not play (prod job 268). Audio mode now
skips that step, so an .mp3 is served as audio/mpeg and an .m4a as audio/mp4.
Audio downloads version their cache identity on their own (v5), so a cached
remuxed audio re-derives without re-downloading every cached video.
Pause now re-reads the job under the same asset-then-job row lock as resume,
cancel, delete and redownload. Its unlocked write waited for a concurrent
cancel's lock and then set the released job back to paused, a holding status,
so a later delete released the same hold again and could free a file another
user's job still held (measured: the overlapping-request test ends 'paused'
instead of 'canceled' with the old read). The lock test now calls the route
handlers, so it also pins which helper each route uses.
Comments follow the code: resolve_url's contract names both verdicts and that a
failed page read is raised as itself; the inherited-failure diagnosis asks the
cookie gate as it stands when the job is claimed, so its advice fits the cookie
that job's Retry would use; the undecryptable-cookie warning no longer claims a
download goes ahead, since the worker parent reaches it only to diagnose.
A refused Facebook cookie on a group post is no longer "no video in this post":
measured on dev, Facebook answers with its login page (no post id at all), so the
adapter now raises its own POST_NOT_SHOWN verdict and the catalog explains it as
the refused cookie. The adapter owns both post verdicts and the "cannot parse
data" wall wording, so the Chrome retry and the catalog read one constant. The
group-post "no formats" entry is cookie-scoped (with a cookie yt-dlp only gets the
post's video link) and uses the adapter's own group-post test instead of a
"/groups/" substring.
Job transitions that move an asset's ref count (resume, cancel, delete,
redownload) re-read the job under a row lock, asset first, so two overlapping
requests can no longer release the same hold. get_or_create_asset loses an insert
race inside a SAVEPOINT instead of rolling back the caller's pending writes (Retry
lost its cleared error). Every worker-side job failure goes through
finalize_if_running and writes error_detail, so a cancel that landed first is kept
and a failure never shows an earlier one's copy. The worker asks the download
child's own cookie gate (credentials.usable_cookie) when a job inherits a failed
asset.
UI: the Facebook copy no longer points at a Retry button the failure window does
not have, says "paste" like Settings, and the HU wording is fixed
("Újraletöltés" on the re-download confirm). A category this client has no copy
for falls back to the generic sentence instead of a raw key.
Tests: route serialization of error_detail, the probed duration through a real
_download, the page-read cap and a failed read, a Retry off a pending asset, the
confirm-button labels, the lock and savepoint races.
Given a group post's link, yt-dlp either found no formats or took another
video from the page's feed data, so a "successful" retry of such a post was a
wrong download. With the user's cookie the Facebook adapter now reads the post
page, takes the single video node in the script blocks that carry the post id,
and downloads that video's watch link (the Chrome retry does the rest). No
single video in the post fails with its own message and the group-post copy,
reworded to say the post has no video; the post link is never handed to yt-dlp
as a fallback. Without a cookie nothing changes.
A Facebook or Instagram reel extracts with no duration, so the asset kept a
NULL duration_s although the file plays for minutes, and the Library row had
no trim action. The finalize probe now returns the duration with the codecs
(the file wins, info fills a gap), and runs before the sidecars so the .nfo
runtime gets it too.
Measured on dev: with the user's cookie some reels still answer "Cannot parse
data" and only extract when the request impersonates Chrome. Impersonation is
a second attempt behind that one error, only when a cookie is attached — links
that already download with the cookie keep their plain path, and without a
cookie the same error is the login wall.
curl_cffi is pinned inside yt-dlp's supported range (0.10-0.15; outside it
every target silently reports "unavailable"); the lock only gains the new
packages. A test fails the gate if the locked pair can no longer impersonate.
The login-wall copy tells the user to add their cookies and press Retry, but
Retry kept the job on the shared asset the worker never attaches a cookie to,
so it failed on the same wall. Resume now moves a pending/errored job onto the
asset enqueue would pick with the cookie (never off a file or a download in
flight).
Re-download salted its fresh asset with the job's previous asset id, which the
cookie gate cannot recompute, so even a cookie download rebuilt without the
cookie. The rebuild salt is now the new asset's own id (plus the user's cookie
salt when one applies), and is_authed_asset recognises it.
A failed Facebook download read "Download failed": yt-dlp words the login wall
(18+ reel, closed group) as "Cannot parse data", which no category matched.
Measured on dev with the owner's cookie before writing a rule — the same text is
the login wall without a cookie and a refused cookie with one.
- errors.diagnose(msg, url, cookie_attached) tries a host-scoped catalog first
(all-of markers, cookie state, link fragment) and falls back to the category.
First entries: facebook.login_wall, facebook.cookie_rejected,
facebook.group_post_video. The module doc is the recipe for the next entry.
- download_jobs.error_detail (0084) + backfill of the stored Facebook parse
failures that were unknown.
- The pipeline's failure branch passes the link and whether the cookie went
with it; resume and redownload clear code and detail (redownload never
cleared error_code).
- A job meeting an already-failed asset got the text but no code, so it read
"Download failed" where the first job said why. It is diagnosed again, with
the cookie state from credentials.is_authed_asset (now shared with the
cookie gate); the redacted network text classifies back to network.
- FE: failCopyKeys gives the row and the failure window one lookup — catalog
copy, else the category. Copy in EN and HU; a contract test pins every
backend entry to both locales.
Round five. 47 -> 34 -> 15 -> 7 -> 6 raw findings, and this round's sharpest was the
loop's own blind spot: **the branch did not pass the gate.** `prettier --check` is one of
its eight lanes and I had been running the other seven, so an unformatted `onError` from
round one sat on the branch for four rounds — `siftlode publish` would have refused it,
with no --skip flag to get past.
**The restart hint could still strand the restored data.** Round four said the scripts no
longer print a command they cannot resolve; they did. The COMPOSE_FILE fallback only fires
when `docker-compose.yml` is ABSENT, so in a source checkout it never fires and the hint
expanded to a bare, pasteable `docker compose up -d` — which reads the other file, whose
volumes are `pgdata`/`downloads` rather than `siftlode_pgdata`/`siftlode_downloads`, and
recreates the database on an empty one. When the file is unresolved the scripts now print
`-f <the compose file you started with>`: not runnable, which is the point.
**A comment naming the wrong owner.** `formats.py` still said a zero budget is "a refusal
for enqueue to make" — the premise round one disproved, three files away from the two
docstrings that now say the opposite. Anyone acting on it would delete the worker's refusal
and re-open the original inversion, so it now says where the refusal actually lives.
Also: the reset-flag rule was rewritten in three consecutive rounds and pinned by nothing —
two tests now cover a fragment losing its token, both by a pasted verify link and by
navigating away entirely; `backup.ps1` deletes a partially-copied dump before throwing, the
guard round four gave the docs but not the script it mirrors; and one comment that read as
if the asset were untouched by a line four below it.
Gate green, all eight lanes: ruff, ruff format, 3x tsc, eslint, prettier, knip,
vitest 687, pytest 950.
Round three, on round two's diff. The trend across the loop is 47 -> 34 -> 15 raw
findings; the ops half kept producing the sharpest ones because the gate cannot run it.
**`docker compose start` ignores a newly pinned IMAGE_TAG.** Round two removed the
automatic restart because it re-ran `alembic upgrade head` — and then told the operator to
run that exact command by hand, in both scripts and the docs. `start` relaunches the
container that already exists, which still carries the tag being rolled back FROM, so
pinning 0.79.0 and following the printed instruction brings 0.80.0 back up and re-applies
its migrations. Verified against a real daemon: after changing the pinned tag, `start` kept
the old image and only `up -d` picked up the new one. Both scripts and the doc now say
`up -d`, with the pin-first caveat where the operator reads it.
**The reset flag outlived its token.** Reverting round one's "each link clears the other"
went one step too far: a `#verify=` link pasted over a standing reset notice replaces the
whole fragment, so the reset token is gone — but `resetPending` stayed true, and being
checked first it hid the confirmation while promising a recovery that signing out could no
longer complete. The flag is now cleared when the fragment it describes is gone, which is
narrower than what was reverted and does not blank the signed-out card.
**A test that could not fail.** The shared-asset regression test seeded the asset as
`pending`, and `_reset_asset_pending` only acts on `downloading` — so the release half was a
no-op and deleting it kept the suite green. Seeded as the worker actually leaves it now.
Also: the malformed-URL guard answers with urlguard's structured detail rather than a bare
English string, so both halves of one typo class get the same localized copy, and the third
site in that module handing user input to `urlparse` was swept with it; the asset is
released after the job is terminal, not before; the Stats 403 toast gets the wizard handle
its own comment claimed it had; the pasted Windows recipe sets `$ErrorActionPreference`
inside the block (a scriptblock inherits the caller's, so a `SilentlyContinue` profile
swallowed every throw) and cleans up the container-side dump on the failure path too; the
restore warning moved to the warning stream so a `> log` redirect cannot file it away; and
two comments narrowed to what the code does.
Gate green: vitest 685, pytest 950.
Round two of the review loop, on round one's own diff. Two of its findings were
regressions the fixes themselves introduced, which is exactly what the loop is for:
**A per-user refusal was writing onto a shared row.** MediaAssets are deduped across
users, and the generic failure handler puts the message on the asset — right for an
upstream failure, wrong for "your storage is full". Two users queuing the same video share
one asset, so the first user hitting their cap failed the second user's job with the first
one's private reason, telling someone with an empty library to free up space. The refusal
now raises `_OwnerRefused`, caught separately: the job fails, the asset goes back to
`pending` for whoever else holds it. Pinned by a test that drives `_download` directly —
the refusal happens before extraction, so nothing needs stubbing.
**The restore scripts restarted the app, undoing the restore.** `docker compose start`
re-runs the container command, and that runs `alembic upgrade head` — so on the rollback
path this documents as recommended (restore the dump, THEN pin the older tag) the newer
image immediately re-applied the migrations the restore had just undone, and on a FAILED
restore it put the app back on a half-restored schema. Both are the opposite of why the
stop was added. Nothing is restarted automatically now; the scripts say what to run and
when, and the docs' snippets say the same.
**Reverted rather than patched:** the OnboardingWizard toast (the wizard already renders
that failure inline, next to a Retry button — three recoveries for one failure), and
`useVerifyEntry`'s "each new link clears the other". The latter blanked the signed-out
card's only banner, where nothing renders the reset flag to replace it. The two states are
independent again; what changes is that `dismiss` peels ONE layer, which is what the
original finding was actually about.
Also: `Stats` uses the YouTube helper rather than the bare one, so the route's English 403
detail no longer beats the translated fallback under a Hungarian UI; `urlparse` guarded
alongside `.port` (an unclosed bracket raised just as late), the reason renamed `malformed`
since "bad port" misdescribes it; three docstrings narrowed to what the code actually does
(the worker's byte check is per-job, not an aggregate ledger, and not on the edit path;
the redirect residual leaks a reachability oracle, which the note had denied; the
invalidation helper does not cover suspend); the pasted Windows recipe wrapped so `throw`
can stop it; the bash one-liner's failure branch no longer exits 0; and the tests that were
missing for the round-one fixes — the account panel's error paths, the invalidation set,
and the shared-asset case above.
Gate green: vitest 685, pytest 950.
The deep review of the branch raised 47 findings, 18 of which survived adversarial
verification. Grouped by what they actually were:
**The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user
already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one
user over quota was the only one downloading with no ceiling at all. `check_enqueue`
could not catch it: it ran while the job was queued and the user was still under the cap.
The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision
lives, and where it can be tested), and the worker now fails the job outright rather than
proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help
until space is freed.
**Silent failures in the new account panel.** All three actions went through `quiet: true`
endpoints, which suppress the global error dialog on the promise that the caller reports
inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On
the broken-SMTP instance this panel exists for, an admin would read silence as "the mail
went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`,
shared with the pending-access queue, so the two copies of these mutations cannot drift
apart again — which is how half of this was lost in the first place.
**A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps
back one migration, but a release routinely ships several (0.80.0 shipped four): it drops
the newest one's columns and leaves the schema at a revision the older image also cannot
resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is
now the recommended path, and the alembic route asks the target version for its own head
revision instead of counting backwards.
Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they
share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as
a 500; the pull cooldown extended to the sibling route spending the same QuotaAction;
restore stops the app before dropping its tables; the PowerShell cleanup can no longer
decide the script's verdict on 5.1, nor report the wrong exit code; the documented
one-liners hardened to match the scripts; the registry's retention window stated where
pinning is advertised; the two banners made mutually exclusive so neither can shadow the
other; `hover:bg-hover` (a utility this project never generated) replaced by the
`selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to
the two callers that discarded it.
Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset
banner UAT'd on localdev.
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches
it from the WORKER, so any approved user could aim the server at the compose
network, the LAN or a cloud metadata endpoint — and read the result off
job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a
port scanner. downloads/urlguard.py refuses a URL whose host resolves to a
loopback/private/link-local/reserved address, at enqueue AND again in the
worker (DNS can move in between, and the worker is where the socket opens).
A network-class failure on a `url` job now reports one fixed sentence instead
of the raw transport text; the error CODE is still classified from the
original, so the localized explanation is unchanged. DNS rebinding is
explicitly NOT closed — that needs the connection gated inside yt-dlp — and
the module says so rather than implying more than it does.
SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video,
so a URL that is purely a playlist or channel still expanded into every
entry — one enqueue, unbounded downloads. `playlist_items` pins it to one.
Size was unbounded too: `check_enqueue` compares the CURRENT footprint against
the cap, and a download's size is unknown until it lands, so one job could
overshoot by any amount; the owner's remaining budget is now handed to yt-dlp
as `max_filesize`, applied AFTER the adapter merge so no source adapter can
raise a limit the quota layer set. A join's cut-list is capped at 100: an
accurate join emits two filters per segment into one ffmpeg filter_complex.
SEC-14: the YouTube budget is instance-wide, so one user holding down a
manual sync spent the day for everyone. Per-user (not per-IP — the cost
belongs to the account) 3-per-10-minutes limiters on the two bulk pulls,
subscriptions and playlists, with separate budgets so one does not close the
other. The review also listed /rss, /backfill and /enrich; E3 deleted them,
and search stays unlimited on purpose — it is interactive, not a bulk pull.
Gate: pytest 896 -> 924 (none skipped), vitest 671.
A downloaded video was silently demoted to a download-only "file" (no
in-app play/edit) whenever yt-dlp's info dict omitted vcodec/acodec — which
it does for Facebook and other sources. Finalize derived media_kind from
those nulls (`derive(None, None)` -> file), so a genuine H.264/AAC MP4
became non-playable in the library.
Fix: probe the finished file on disk (the ground truth) at finalize and
derive from that, falling back to the info dict only for a field the probe
couldn't read. This is the same ffprobe the upload ingest already used, now
extracted to a shared downloads/probe.py so the download worker and the
upload path share one classifier (and can't drift). Also backfills the
stored vcodec/acodec/width/height from the real file.
Tests: TestResolveMediaFacts pins probe-wins-over-info (incl. the Facebook
no-codec case and yt-dlp's literal "none"); upload's probe tests repoint to
the shared module.
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.
- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
gates library actions. Set at every finalize (download probe, editor carries the
source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
places it in a per-user .uploads tree, and creates an already-ready asset + done job
(the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
(download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
application/octet-stream) on both the in-app and public endpoints, so uploaded
HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
Each user stores their own login cookies (Netscape cookies.txt) for an
auth-walled download host (Facebook, Instagram, X, TikTok, Reddit) in
Settings; a download of an auth-walled link on that host is retried with
the acting user's cookie.
Two load-bearing security controls, both in downloads.credentials as the
single source of truth so enqueue and the worker can't drift:
- cache privacy: a cookie-authed download salts its MediaAsset format_sig
with the user id, so the authed asset is private and never cache-served
to another user. The worker attaches a cookie ONLY when the asset it is
downloading into IS that private, salted identity.
- host scope (SEC-12): a cookie is resolved only for a URL whose netloc
the credential owns (netloc-scoped _host_is); a look-alike/unrelated
host gets none.
cookies_enc is Fernet-encrypted at rest, only decrypted in-worker into a
0600 file under staging/.auth/ (survives the retry wipe + _produced_file
scan) and unlinked in finally. The blob is never returned by a route,
logged, audited, or written to a job's error/snapshot. cookiefile is set
after the adapter's extra merge so an adapter can't clobber it. Settings
UI is write-only (host + set/updated only, never echoed). New
download_credentials table + migration 0074; per-user routes; HU+EN i18n.
Tests: host-scope, cache-privacy (no cross-user serve), encrypt-at-rest,
write-only API, worker attach-gate; FE dom test for the write-only field.
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API,
Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and
starved web requests + playback until it finished. Full-scope S4 mitigations:
- Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp
descendant): CPU nice increment + best-effort idle ionice class + an
OMP_NUM_THREADS budget, all config-gated and independently disable-able.
- Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode
reserves a core for foreground work instead of claiming all of them.
- Throttle the retention GC's whole-tree orphan sweep (yield IO after each
batch of files) so its rglob can't monopolize disk IO in the API process.
- Idle-back-off the Download Center (2s→15s when nothing queued/running) and
UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the
next fetch after an enqueue (invalidateDownloads refetches immediately).
Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence,
_deprioritize (nice/ionice/OMP branches), GC sweep throttle batching.
Round-2 review of the R1 fixes found that the two-independent-limiter progress
writer issued a redundant heartbeat every interval during an ACTIVE download: a
percent change writes progress (which already refreshes updated_at, the stall
watchdog's liveness source), then the sub-integer calls that follow fire the
separate beat budget — ~2 DB writes/interval instead of the throttle's intended 1.
Replace the second limiter with a shared `last_touch` clock (last DB write OR
beat). Progress writes keep their own `allow_write` budget (so a beat still can't
delay a real write — the R1 concern), but the heartbeat now fires only when
nothing touched the row for a full interval. Active download → ~1 write/interval;
a genuinely stuck re-encode → 1 heartbeat/interval. New regression test:
an unchanged-percent call in the same interval as a write does NOT heartbeat.
ruff clean, full DB-lane 365 pass.
Round-1 review fixes on the E3 branch:
- prune partial-fetch (correctness): `_paginate` now reports whether it reached
the server's reported `total`; `_prune_stale` prunes only on a COMPLETE
non-empty listing. A truncated fetch (a mid-sequence empty/short page before
total — a transient Plex hiccup) no longer deletes the rows it merely failed
to list. New test: a listing that claims 5 but returns 1 prunes nothing.
- `_prune_stale` (robustness): `stats.get("pruned", 0) + 1` — no KeyError when a
caller's stats dict omits the key.
- service transitions (conventions): pause/resume/redownload use the JobStatus
StrEnum instead of loose "queued"/"paused"/"error" literals, matching cancel.
- ffmpeg progress writer (correctness): a stuck-percent heartbeat now runs on its
OWN 1/interval budget, so it can't consume the progress-write slot and delay a
real percent change by up to an interval.
- backend/app/README.md (conventions): reword the transaction-ownership rule to
quote and extend CLAUDE.md's canonical ("handler/job-wrapper commits; helpers
flush; only quota owns its session") instead of stating a contradicting
"helpers MAY commit" carve-out.
ruff clean, full DB-lane 364 pass.
worker.py (~1090 L) mixed the PARENT supervisor (claim + supervise + the process
loop) with the CHILD pipeline (the yt-dlp/ffmpeg download+edit that runs in the
spawned subprocess). Extract them:
worker.py 1090 -> 475 parent: claim/supervise/loop
downloads/pipeline.py (new 592) the child pipeline (_download/_process_edit +
hooks/progress/ffmpeg helpers + _Aborted)
downloads/jobdb.py (new 73) the short-lived-session helpers both sides
share (_set_job/_job_status/_touch_job/
_requeue_if_running/_reset_asset_pending +
the staging-dir paths)
The graph is one-directional (runner -> pipeline -> jobdb; worker -> jobdb; no
worker<->pipeline edge) — the parent never calls the child directly, it spawns
runner.py as a subprocess. Consumers rewired: runner import, and two tests follow
the moved symbols (_media_info/_produced_file -> pipeline; the ffmpeg-callback
patches target pipeline where the names are looked up).
Pure code motion, verbatim bodies. ruff clean, full lane 363 pass (unchanged).
peter