The Files library starts here. A user's every download, clip and upload now
lands in a source folder made per origin (YouTube, Plex clips, uploads, one per
other site), decided once in downloads/folders.py at every place a job is made.
How long a file stays moves from the shared asset to the user's job: stamped
from its folder when it is done, never for a file the user keeps. The GC warns
each user once, then takes the expired file out of their library; the cache cap
never evicts a kept file, and an evicted one leaves every holder's library.
The media mount is /files now (FILES_ROOT, FILES_HOST_PATH, volume
siftlode_files); docs/self-hosting.md says how to carry an old volume over.
Migration 0102 places every existing job and carries each done file's expiry
over from its asset.
safe_abs_path takes a missing path and answers None, which drops one baselined
type error.
- The cut file is measured against what its owner has left, for a YouTube range (whose fetch never
reads yt-dlp's size ceiling) and for a Plex clip (which fetches nothing), before the fetch and
again before it is placed; a refusal leaves the asset pending for a Retry.
- A silent Plex file is cut silent; a file the probe cannot read says so.
- A second job on a failed or renamed clip gets the same YouTube diagnosis and the fetched title.
- The "too long" refusal names the server's cap; a contract test holds the client's copies of the
clip limits to clip.py; the shipped player clients are pinned against the installed yt-dlp.
- One HDR transfer set and one rating_key lookup; job status words from JobStatus; the display-name
guard sees subscript writes and the clip route; stale comments and types follow the clip job.
POST /api/downloads/clip takes a source (a YouTube id or a Plex rating key), a range of at most
ten minutes and, for Plex, the audio track being heard. The clip becomes a per-user asset in the
library like an editor clip, shareable as soon as it is ready. A YouTube clip fetches only its
range (yt-dlp download_ranges: 3-7 s whatever the video's length); a Plex clip reads the library
file in place. Both then go through one re-encode: frame-exact at the marks, H.264 at most 1080p
with HDR tonemapped to SDR, stereo AAC, no chapter track, .mp4.
Job kinds are a JobKind StrEnum now, and Re-download refuses every kind but a download.
A share link is opened on whatever the friend holds. Measured on an iPhone and an iPad: AV1,
Matroska, HEVC tagged hev1 and MP3 inside an .mp4 fail there; Chrome and Firefox add HEVC and
10-bit / 4:4:4 H.264. The old check judged the codec NAME only and ran on downloads only, so a
10-bit H.264 passed, an upload went out exactly as it came (an iPhone's HEVC .mov, a webm, an mkv)
and a fast trim kept its source's codecs.
One rule now, `downloads/playable.py`: video is H.264 8-bit 4:2:0 + AAC in .mp4, audio-only is
.mp3 or AAC .m4a; `plan` says what must change, `build_cmd` re-encodes only the failing stream (or
remuxes). The worker applies it to a download, an editor clip and — new — an upload: media that
fails the rule is queued (asset pending, job queued) and the worker converts it, keeping only the
converted file (owner's choice). Until then the original sits at rel_path, so Retry starts from it
and a delete removes it; an upload that lost its last holder mid-conversion is reaped by the GC.
Every libx264 run (edit, concat, conversion) now pins `-pix_fmt yuv420p` — without it a 10-bit
source came out High 10. The download cache signature moves to v5 so a cached 10-bit asset
re-derives. The upload toast says when the file is still converting (HU/EN).
Tests: the plan for every measured class; real ffmpeg turning a 10-bit Matroska + Opus and an
mp3-in-mp4 into files the rule accepts (red without the pixel-format pin); the queued upload's
delete, cancel-then-GC, failed conversion and real worker conversion (the cleanup tests red
without their rules); the toast (red without the branch). Live on localdev: 9 fixture uploads
converted, every served file probed H.264 High yuv420p + AAC, or mp3 / m4a.
Deep-review round 2, on round 1's own fix commit. It found MORE than round 1
did (28 raw, against 13), and most of it was mine -- a new piece of UI attracts
findings a backend-only diff never does. The pattern in almost all of them is
the same: a principle applied to the site that prompted it and not to its
siblings.
* `display_text` reached three of the FIVE writers of a user-supplied name.
`POST /api/downloads` and `POST /api/downloads/edit` still handed Postgres an
unbounded string, so a 300-character name pasted into the Add-download dialog
or the editor was a bare 500 (main.py re-raises every DataError but the NUL
case). Four of the seven reviewers found it independently. The bound now lives
in `models.py`, the three columns are DECLARED with it, and the guard for the
next writer is parsed rather than remembered: every `display_name` /
`display_uploader` assignment under `app/routes/` must go through the helper.
That guard immediately found two more -- the Google sign-in callback writes
the profile name into the same 255-wide column -- so it is six, not five.
* The connection release reached one of the THREE routes that read their own
body. The screenshot route (10 MiB, from a phone) and the client log sink both
still held a pooled connection idle-in-transaction for the length of the
upload, which is the invariant db.py sizes the pool on. All three now call
`db.release_read_connection`, whose precondition -- nothing pending on the
shared session -- is CHECKED rather than described, because a whole-session
rollback discarding a caller's staged write is a mistake this repo has already
made once. `test_body_limit.py` holds the rule for the next such route.
* The AST drift registry keyed its sites by (module, limit expression), so a
second route copied from the first collapsed onto the same entry and went
uncovered -- the exact hole the registry exists to close. Keyed per call site
now, and it carries the release requirement too.
* The dialog's own refusal was announced once and never again: two over-cap
picks leave an identical node with identical text, and an untouched live
region says nothing. It is keyed per pick now, the drop zone reads as invalid
and points at the sentence, and the reason is reachable from the control
rather than only from a one-shot announcement.
* Two tests rebuilt the blind spot they were written to remove: the dialog's cap
test sat nowhere near the boundary (a `>` loosened to `>=` survived it, which
is the mutation round 1 fixed on the server side), and the header-precedence
test only covered an exact-case duplicate while a differently-cased one would
have been COMBINED onto the wire rather than replaced. Both fixed, and the
merge now dedupes case-insensitively because header names are.
Also: the upload dialog's unstubbed usage query was firing a real request from
jsdom in a neighbouring suite and leaking a delayed "connection lost" toast
across test boundaries; `GET /api/downloads/usage` had no test at all though the
whole client-side check hangs off it; and the comment claiming the edge 413 is
readable now says what was measured -- readable to 200 MiB over loopback, a bare
`onerror` through a proxy -- so the pre-send check is named as the mechanism and
that branch as the backstop it is.
Every test added or changed here was mutation-checked. One mutation SURVIVED and
is recorded rather than papered over: the `tooBig` term inside `submit()` cannot
be reached, because `ui/Button` swallows the activation itself, so it is marked
as belt and braces instead of being described as the guard.
The epic-end deep review found three defects in P1 and four tests that did not
hold what they claimed. They land together because they share the upload route
and its test module; splitting by hunk would have produced commits that do not
individually run green.
* The edge ceiling and the route's own cap are the SAME number at the default
configuration, so the edge always answers first: a declared Content-Length
over the cap is refused before a byte is read, and an undeclared body trips
the middleware on the same chunk the route would have tripped on. The route's
numbered 422 -- and the drain written for it last week -- are reachable only
once an admin LOWERS the knob, which is the one shape the browser UAT tested.
Measured: headroom does not fix this, because the declared-length check fires
first for any body big enough to matter and the drain pushes the count back
over the edge for the rest. So the edge's own refusal is made readable
instead: `errors.tooLarge` localizes `too_large` the way `upload_too_large`
was already localized, and the dialog refuses an over-cap file BEFORE sending
it, off the `upload_max_bytes` that GET /api/downloads/usage now reports.
Measured in Chrome against the API directly: that 413 arrives as `onload`
with a readable body in 43 ms.
* Reading the body behind the auth dependency put the whole transfer inside the
request's database session -- the auth SELECT autobegins a transaction, so one
of the twenty pooled connections sat `idle in transaction`, with nothing to
time it out, for as long as the client cared to keep sending. That is the
invariant db.py sizes the pool on and plex/playback.py records an incident
for. The read transaction now ends before the await. Measured on the dev
server: an 1818 ms upload shows no idle-in-transaction sample while the bytes
move, only afterwards, when the handler actually writes.
* The display name moved into a request header, where nothing bounds its length,
while the column stayed varchar(255) -- so a long enough name was a DataError
500 after the whole file had already transferred. One `display_text` helper
now serves all three writers (the metadata PATCH already truncated, the upload
ingest did not), and the dialog stops at the same number.
* The gate invariant asked `route.dependant.body_params`, which holds only the
route's OWN callable's parameters: a body declared in a SUB-DEPENDENCY -- the
upload route's own idiom -- leaves it empty while FastAPI still reads the body
before auth. It asks `route.body_field` now, which is what FastAPI itself
consults. The AST registry of body-reading call sites had also never followed
`capped_stream_to`, so the one route the whole invariant exists for was the
single site the drift guard could not see.
Four tests that passed over their own mutation are fixed: the cap boundary (a
`>` loosened to `>=` survived the entire suite), the percent-decoding (every
filename the suite sends is ASCII, where quote() is the identity function), the
display-name header (no test at all), and the staging cleanup (deleting the
unlink left the suite green while every refused upload leaked its partial bytes
onto the volume). Every test added or changed here was mutation-checked.
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.
What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
degraded to a software transcode with nothing saying why; a failed audio probe
collapsed a multi-audio file to one track and the player showed no language
picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
own root, leaving every crashed session's segments on disk forever.
Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.
ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.
logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.
tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
A refused Facebook cookie on a group post is no longer "no video in this post":
measured on dev, Facebook answers with its login page (no post id at all), so the
adapter now raises its own POST_NOT_SHOWN verdict and the catalog explains it as
the refused cookie. The adapter owns both post verdicts and the "cannot parse
data" wall wording, so the Chrome retry and the catalog read one constant. The
group-post "no formats" entry is cookie-scoped (with a cookie yt-dlp only gets the
post's video link) and uses the adapter's own group-post test instead of a
"/groups/" substring.
Job transitions that move an asset's ref count (resume, cancel, delete,
redownload) re-read the job under a row lock, asset first, so two overlapping
requests can no longer release the same hold. get_or_create_asset loses an insert
race inside a SAVEPOINT instead of rolling back the caller's pending writes (Retry
lost its cleared error). Every worker-side job failure goes through
finalize_if_running and writes error_detail, so a cancel that landed first is kept
and a failure never shows an earlier one's copy. The worker asks the download
child's own cookie gate (credentials.usable_cookie) when a job inherits a failed
asset.
UI: the Facebook copy no longer points at a Retry button the failure window does
not have, says "paste" like Settings, and the HU wording is fixed
("Újraletöltés" on the re-download confirm). A category this client has no copy
for falls back to the generic sentence instead of a raw key.
Tests: route serialization of error_detail, the probed duration through a real
_download, the page-read cap and a failed read, a Retry off a pending asset, the
confirm-button labels, the lock and savepoint races.
The login-wall copy tells the user to add their cookies and press Retry, but
Retry kept the job on the shared asset the worker never attaches a cookie to,
so it failed on the same wall. Resume now moves a pending/errored job onto the
asset enqueue would pick with the cookie (never off a file or a download in
flight).
Re-download salted its fresh asset with the job's previous asset id, which the
cookie gate cannot recompute, so even a cookie download rebuilt without the
cookie. The rebuild salt is now the new asset's own id (plus the user's cookie
salt when one applies), and is_authed_asset recognises it.
A failed Facebook download read "Download failed": yt-dlp words the login wall
(18+ reel, closed group) as "Cannot parse data", which no category matched.
Measured on dev with the owner's cookie before writing a rule — the same text is
the login wall without a cookie and a refused cookie with one.
- errors.diagnose(msg, url, cookie_attached) tries a host-scoped catalog first
(all-of markers, cookie state, link fragment) and falls back to the category.
First entries: facebook.login_wall, facebook.cookie_rejected,
facebook.group_post_video. The module doc is the recipe for the next entry.
- download_jobs.error_detail (0084) + backfill of the stored Facebook parse
failures that were unknown.
- The pipeline's failure branch passes the link and whether the cookie went
with it; resume and redownload clear code and detail (redownload never
cleared error_code).
- A job meeting an already-failed asset got the text but no code, so it read
"Download failed" where the first job said why. It is diagnosed again, with
the cookie state from credentials.is_authed_asset (now shared with the
cookie gate); the redacted network text classifies back to network.
- FE: failCopyKeys gives the row and the failure window one lookup — catalog
copy, else the category. Copy in EN and HU; a contract test pins every
backend entry to both locales.
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.
- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
gates library actions. Set at every finalize (download probe, editor carries the
source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
places it in a per-user .uploads tree, and creates an already-ready asset + done job
(the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
(download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
application/octet-stream) on both the in-app and public endpoints, so uploaded
HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
Each user stores their own login cookies (Netscape cookies.txt) for an
auth-walled download host (Facebook, Instagram, X, TikTok, Reddit) in
Settings; a download of an auth-walled link on that host is retried with
the acting user's cookie.
Two load-bearing security controls, both in downloads.credentials as the
single source of truth so enqueue and the worker can't drift:
- cache privacy: a cookie-authed download salts its MediaAsset format_sig
with the user id, so the authed asset is private and never cache-served
to another user. The worker attaches a cookie ONLY when the asset it is
downloading into IS that private, salted identity.
- host scope (SEC-12): a cookie is resolved only for a URL whose netloc
the credential owns (netloc-scoped _host_is); a look-alike/unrelated
host gets none.
cookies_enc is Fernet-encrypted at rest, only decrypted in-worker into a
0600 file under staging/.auth/ (survives the retry wipe + _produced_file
scan) and unlinked in finally. The blob is never returned by a route,
logged, audited, or written to a job's error/snapshot. cookiefile is set
after the adapter's extra merge so an adapter can't clobber it. Settings
UI is write-only (host + set/updated only, never echoed). New
download_credentials table + migration 0074; per-user routes; HU+EN i18n.
Tests: host-scope, cache-privacy (no cross-user serve), encrypt-at-rest,
write-only API, worker attach-gate; FE dom test for the write-only field.
Round-1 review fixes on the E3 branch:
- prune partial-fetch (correctness): `_paginate` now reports whether it reached
the server's reported `total`; `_prune_stale` prunes only on a COMPLETE
non-empty listing. A truncated fetch (a mid-sequence empty/short page before
total — a transient Plex hiccup) no longer deletes the rows it merely failed
to list. New test: a listing that claims 5 but returns 1 prunes nothing.
- `_prune_stale` (robustness): `stats.get("pruned", 0) + 1` — no KeyError when a
caller's stats dict omits the key.
- service transitions (conventions): pause/resume/redownload use the JobStatus
StrEnum instead of loose "queued"/"paused"/"error" literals, matching cancel.
- ffmpeg progress writer (correctness): a stuck-percent heartbeat now runs on its
OWN 1/interval budget, so it can't consume the progress-write slot and delay a
real percent change by up to an interval.
- backend/app/README.md (conventions): reword the transaction-ownership rule to
quote and extend CLAUDE.md's canonical ("handler/job-wrapper commits; helpers
flush; only quota owns its session") instead of stating a contradicting
"helpers MAY commit" carve-out.
ruff clean, full DB-lane 364 pass.
pause/resume/cancel/delete/redownload each hand-rolled their asset bookkeeping
inline in the route (release, failed-asset reset, the 20-line fresh-asset
rebuild), duplicating logic the worker/GC also touch. Move each transition into a
service.* method (service.pause/resume/cancel/delete/redownload) that owns the
mutation + its commit; the routes keep only auth, the HTTP 409 guards, and
serialization. Drops the now-dead _set_status route helper and two stale imports.
Pure behaviour-preserving code motion (routes -56 lines). Green under the E3 S2
transition net + the full downloads suite (363 pass), ruff clean.
MediaAsset.ref_count (how many jobs hold a shared cache asset) was a Python
read-modify-write at four sites (enqueue, enqueue_edit, release_asset, the
redownload route). Two concurrent enqueues of the same (source,format) both read
the same value and write the same +1 → the count ends up one short → the first
release drives it to zero and deletes the file out from under a live holder.
Replace all four with atomic DB ops: incr_ref_count (`ref_count = ref_count + 1`)
and decr_ref_count (`greatest(ref_count - 1, 0)` RETURNING the new value, so the
release decides whether to free the asset from the post-decrement count while
holding the row's write lock — Postgres serialises concurrent releases on that
lock, closing the double-free). Pure DB-atomicity change, no semantic shift.
Green under the E3 S2 ref_count net (5) + full lane (356 pass), ruff clean.
Review round on the whole E1 stack (main..HEAD). 6 findings, all low/medium,
none blocking; fixed for a clean gate-green handoff.
- plex/sync (correctness): the un-enriched cast guard compared the stored cast
against THIS listing's length, so a trimmed Plex cast on a not-yet-enriched
row was never applied. Compare against the listing CAP (_LISTING_CAST_CAP=20)
instead: >cap = enriched (keep, survives a watermark reset), <=cap =
listing-sourced (track edits in both directions). +2 regression tests.
- main (security): add X-Frame-Options: DENY as the legacy-browser fallback for
the CSP frame-ancestors clickjacking guard.
- downloads/service (altitude): finalize_if_running now uses a SQLAlchemy 2.0
update() construct instead of a hand-built text() UPDATE string — same atomic
WHERE status='running', idiomatic, no identifier interpolation, no manual
StrEnum coercion. Drops the now-unused text import.
- auth/admin (reuse): extract the SEC-10 "drop untrusted preset password + bump
epoch" invariant into one helper (revoke_preset_password_if_unverified) so the
Google-callback and admin-activation paths can't drift.
- tests/migrations (robustness): guarantee the shared schema is restored to head
in a finally, so a downgrade/re-upgrade failure can't cascade into later tests.
- worker (simplification): align _finish_from_cache's display_name fill with the
_download path's guard (no behavior change).
Gate green: ruff clean, 334 pytest (0 skip), frontend unchanged.
Follow-up to the /code-review high on the S1 branch. The per-site
`if job.status == 'running'` guards were a non-atomic read-then-write (TOCTOU):
a cancel committing between the child's status read and its commit could still
be overwritten with done/error, re-opening the double-release window.
Introduce service.finalize_if_running(db, job_id, **fields): one atomic
`UPDATE download_jobs SET ... WHERE status='running'` that returns whether it
fired. It flushes pending asset writes first so the asset-then-job row-lock
order matches the cancel route (no deadlock), and runs in the caller's session
so it commits with the asset writes in one transaction. All five worker sites
(download/edit completion, both error paths, cache finish) now go through it,
so the invariant lives in one place. The 'job done' logs move inside the
finalized branch so a job stopped mid-completion no longer logs a false 'done'.
Add tests/test_downloads_finalize.py: running -> finalizes; canceled/paused ->
no-op.
The cancel route writes status 'canceled' but the supervisor compared against
'cancelled', so its "a route stopped it -> kill the child, free the asset"
branch was dead code. A canceled job's child ran to completion and wrote 'done'
unconditionally over the route's 'canceled'; since 'done' holds the asset but
'canceled' does not, a later delete released the asset a second time, dropping
ref_count below the live holder count and deleting a file still in use.
Add app/downloads/status.py as the single source of truth for the lifecycle
literals (JobStatus StrEnum + the HOLDING / STOPPED_BY_ROUTE sets) and use it at
the safety-critical sites: the supervisor stop check, the route cancel, and
release_asset's holding set. Make the worker's terminal done/error writes
conditional on the row still being 'running' so a child finishing just after a
route cancel/pause can no longer resurrect it. Add the missing 'canceled'
supervisor regression test.
(The full literal sweep across the downloads surface is left for the E3
downloads owner-unification; this is the data-safety subset.)
Consolidates source-specific policy (youtube / facebook / reddit / generic) that was scattered
across worker.py, backfill.py, service.py, formats.py and storage.py with no shared source
identity. app/downloads/adapters.py holds the base (= generic yt-dlp behaviour) + per-source
overrides; resolve_adapter(source_kind, source_ref, info) picks one by source_kind, the yt-dlp
extractor_key, or the URL host, falling back to generic. yt-dlp stays the engine — the adapter is
policy only.
Moved into adapters (behaviour-equivalent):
- uploader_url derivation (YouTube channel_url / Facebook uploader_id / reddit None) — removes the
worker/backfill duplication (build_uploader_url + _should_refetch gone).
- the player-client fallback sets + retriable markers + YouTube extractor_args/POT (build_ydl_opts
now takes source-agnostic opts plus an `extra` dict from the adapter).
- source_url reconstruction (service.source_url delegates to the adapter).
Two intentional (approved) behaviour changes:
- the player-client fallback is now YouTube-only; non-YouTube sources make a single extraction
attempt instead of a pointless double attempt with youtube-only client args.
- the .nfo <uniqueid type=...> follows the source (adapter.nfo_uniqueid_type) instead of the
hardcoded "youtube" that mislabelled every url-source download.
Deliberately left generic (documented non-moves): title cleaning (normalize_title is harmless
across sources), the classify marker table, thumbnail_to_jpg, the format selectors, and og.py's
YouTube-hqdefault og:image preference (a self-contained thumbnail heuristic on the share path).
Verified end-to-end on localdev across all four adapters: downloads succeed and uploader_url + the
.nfo uniqueid type are correct per source.
Layering cleanup, no behaviour change (ruff clean backend+scripts, pytest 193):
- Invert the scheduler->routes dependency: reset_demo_state (+ its demo-seed
helpers) move from routes/admin.py to a new service module app/demo.py.
Both routes/admin.py and scheduler.py now import it from app.demo, so the
scheduler no longer reaches into the routes layer — its lazy import (the
routes<->scheduler cycle workaround) is gone, hoisted to a top-level import.
- Publish the private symbols that were imported across module boundaries (the
underscore convention was lying): feed._serialize -> serialize,
feed._to_tsquery_str -> to_tsquery_str, videos._insert_stubs -> insert_stubs,
videos._stub_from_playlist_item -> stub_from_playlist_item,
worker._uploader_url -> uploader_url, plex._shared._item_or_404 ->
item_or_404. Scoped renames — the 5 other route modules keep their own truly
private _serialize helpers.
- _release_asset moves from routes/downloads.py to app/downloads/service.py as
the public release_asset (asset bookkeeping is a service concern); the routes
call it via service.release_asset.
- redownload: never clobber a co-held (deduped) asset. Drop the force-reset-to-pending;
_release_asset already deletes+recreates a solo asset (genuine re-fetch), while a shared
ready asset is left intact and the re-queued job re-points to it via _finish_from_cache —
so one user's re-download can't delete another's file mid-serve (finding #1).
- redownload: reset queue_pos to the tail so a re-download doesn't front-run newer queued
jobs on the worker's `ORDER BY queue_pos` claim (finding #3). Made service.next_queue_pos public.
- formats: video-only mp4 selector prefers every video-only option before any progressive
`best`, so a "Video only" download stays audio-free whenever a video-only stream exists (#2).
- titles: a title that is only the engagement prefix no longer resurrects the stripped stats
(return the stripped text / empty, not the raw) (#4).
- worker: probe the file's duration when yt-dlp gave none, so a long re-encode shows real
progress instead of a frozen 0% (#5); tighten the ensure_browser_playable docstring to its
actual mp4-profile scope (#6).
- B1 (sticky errored asset): get_or_create_asset now resets a reused status=='error'
asset back to 'pending' (+clears the error), so a fresh enqueue/edit of a once-failed
(source,format) pair actually re-downloads instead of the worker short-circuiting the
new job with the stale error. Errored assets carry no expires_at, so without this the
pair was permanently poisoned for all users until someone hit resume.
- B2 (ref_count leak): _release_asset counts 'error' as a holding state, so deleting an
errored job decrements the ref_count that enqueue always incremented (the worker never
decrements on failure). Errored rows are deliberately NOT deleted here — a concurrent
B1 reuse could otherwise be lost-updated + FK-nulled; the row is fileless and harmless.
- B3: admin storage dashboard reads sysconfig.get_int(db,'download_total_max_bytes')
(the admin-editable DB value GC enforces) instead of the raw env default.
- B4: the single-trim branch of normalize_edit_spec guards its float() coercion like the
crop/segments branches — a malformed trim now yields a 400, not an unhandled 500.
- B5: formats.normalize guards int(max_height) → falls back to "best" instead of 500.
Reviewed (race in an earlier B2 draft caught + fixed); localdev boots, B4/B5 unit-verified.
- C3: `_reference_url` (downloads.py) and public.py's inline source-URL block were
the same rule → extract `service.reference_url(job, asset)`; both surfaces now
share it so the "downloaded from" link can't drift between them.
- C4: Content-Disposition filename derivation (ext pick + doubled-ext strip + join)
was duplicated in download_file and watch_file → extract
`storage.download_filename(display_name, container, path)`.
- C5: inline the one-line `_clean_basename` passthrough (folded into C4).
- C6: the `db.get(MediaAsset, job.asset_id) if job.asset_id else None; _serialize(...)`
resolve-then-serialize dance was repeated across 6 single-job handlers → fold into
`_serialize_job(db, job)`. (File-serving handlers that use the asset for their own
checks keep their explicit resolve.)
Behavior-neutral; ruff/parse clean, localdev boots, downloads routes load.
A download whose source title exceeded 255 chars (e.g. a recipe baked into a
Facebook video's title) failed with a StringDataRightTruncation: the worker
auto-fills an empty display_name from the title, which overflows the 255-char
column. Clip the title to 255 at every point it flows into display_name (worker
completion paths + enqueue); the full title is untouched on the asset. Pre-existing
latent bug, surfaced by an unusually long title.
Release v0.31.1.
A queued/downloading row previously showed the bare video id and no thumbnail because asset
metadata was only filled when the download completed. Now:
- service.populate_from_catalog fills title/uploader/thumbnail/date/duration at enqueue from
our own Video/Channel catalog (0 network cost) — feed downloads look right instantly
- worker fills the same from yt-dlp's info_dict on the first progress event (covers ad-hoc
URLs / catalog misses), best-effort, never fails a download
Verified: a paused feed download now shows its real title, channel, thumbnail and duration.
The .gitignore 'downloads/' pattern for the DOWNLOAD_ROOT bind mount also matched the
backend/app/downloads source package, so formats.py/storage.py/service.py never got
committed with M2. Anchor the ignore to '/downloads/' (repo root only) and add the package.
storage.sanitize() now handles arbitrarily messy titles (emoji, ZWJ, fullwidth, clickbait
punctuation): NFKC-normalize, drop emoji/symbol/control unicode categories, collapse
repeated punctuation, underscore-join words -> space-free paths. Accents (HU/DE) preserved,
no ASCII folding. Plex layout uses _-_ separators + Season_{year}.
peter