The cancel route writes status 'canceled' but the supervisor compared against
'cancelled', so its "a route stopped it -> kill the child, free the asset"
branch was dead code. A canceled job's child ran to completion and wrote 'done'
unconditionally over the route's 'canceled'; since 'done' holds the asset but
'canceled' does not, a later delete released the asset a second time, dropping
ref_count below the live holder count and deleting a file still in use.
Add app/downloads/status.py as the single source of truth for the lifecycle
literals (JobStatus StrEnum + the HOLDING / STOPPED_BY_ROUTE sets) and use it at
the safety-critical sites: the supervisor stop check, the route cancel, and
release_asset's holding set. Make the worker's terminal done/error writes
conditional on the row still being 'running' so a child finishing just after a
route cancel/pause can no longer resurrect it. Add the missing 'canceled'
supervisor regression test.
(The full literal sweep across the downloads surface is left for the E3
downloads owner-unification; this is the data-safety subset.)
Consolidates source-specific policy (youtube / facebook / reddit / generic) that was scattered
across worker.py, backfill.py, service.py, formats.py and storage.py with no shared source
identity. app/downloads/adapters.py holds the base (= generic yt-dlp behaviour) + per-source
overrides; resolve_adapter(source_kind, source_ref, info) picks one by source_kind, the yt-dlp
extractor_key, or the URL host, falling back to generic. yt-dlp stays the engine — the adapter is
policy only.
Moved into adapters (behaviour-equivalent):
- uploader_url derivation (YouTube channel_url / Facebook uploader_id / reddit None) — removes the
worker/backfill duplication (build_uploader_url + _should_refetch gone).
- the player-client fallback sets + retriable markers + YouTube extractor_args/POT (build_ydl_opts
now takes source-agnostic opts plus an `extra` dict from the adapter).
- source_url reconstruction (service.source_url delegates to the adapter).
Two intentional (approved) behaviour changes:
- the player-client fallback is now YouTube-only; non-YouTube sources make a single extraction
attempt instead of a pointless double attempt with youtube-only client args.
- the .nfo <uniqueid type=...> follows the source (adapter.nfo_uniqueid_type) instead of the
hardcoded "youtube" that mislabelled every url-source download.
Deliberately left generic (documented non-moves): title cleaning (normalize_title is harmless
across sources), the classify marker table, thumbnail_to_jpg, the format selectors, and og.py's
YouTube-hqdefault og:image preference (a self-contained thumbnail heuristic on the share path).
Verified end-to-end on localdev across all four adapters: downloads succeed and uploader_url + the
.nfo uniqueid type are correct per source.
Layering cleanup, no behaviour change (ruff clean backend+scripts, pytest 193):
- Invert the scheduler->routes dependency: reset_demo_state (+ its demo-seed
helpers) move from routes/admin.py to a new service module app/demo.py.
Both routes/admin.py and scheduler.py now import it from app.demo, so the
scheduler no longer reaches into the routes layer — its lazy import (the
routes<->scheduler cycle workaround) is gone, hoisted to a top-level import.
- Publish the private symbols that were imported across module boundaries (the
underscore convention was lying): feed._serialize -> serialize,
feed._to_tsquery_str -> to_tsquery_str, videos._insert_stubs -> insert_stubs,
videos._stub_from_playlist_item -> stub_from_playlist_item,
worker._uploader_url -> uploader_url, plex._shared._item_or_404 ->
item_or_404. Scoped renames — the 5 other route modules keep their own truly
private _serialize helpers.
- _release_asset moves from routes/downloads.py to app/downloads/service.py as
the public release_asset (asset bookkeeping is a service concern); the routes
call it via service.release_asset.
- redownload: never clobber a co-held (deduped) asset. Drop the force-reset-to-pending;
_release_asset already deletes+recreates a solo asset (genuine re-fetch), while a shared
ready asset is left intact and the re-queued job re-points to it via _finish_from_cache —
so one user's re-download can't delete another's file mid-serve (finding #1).
- redownload: reset queue_pos to the tail so a re-download doesn't front-run newer queued
jobs on the worker's `ORDER BY queue_pos` claim (finding #3). Made service.next_queue_pos public.
- formats: video-only mp4 selector prefers every video-only option before any progressive
`best`, so a "Video only" download stays audio-free whenever a video-only stream exists (#2).
- titles: a title that is only the engagement prefix no longer resurrects the stripped stats
(return the stripped text / empty, not the raw) (#4).
- worker: probe the file's duration when yt-dlp gave none, so a long re-encode shows real
progress instead of a frozen 0% (#5); tighten the ensure_browser_playable docstring to its
actual mp4-profile scope (#6).
- B1 (sticky errored asset): get_or_create_asset now resets a reused status=='error'
asset back to 'pending' (+clears the error), so a fresh enqueue/edit of a once-failed
(source,format) pair actually re-downloads instead of the worker short-circuiting the
new job with the stale error. Errored assets carry no expires_at, so without this the
pair was permanently poisoned for all users until someone hit resume.
- B2 (ref_count leak): _release_asset counts 'error' as a holding state, so deleting an
errored job decrements the ref_count that enqueue always incremented (the worker never
decrements on failure). Errored rows are deliberately NOT deleted here — a concurrent
B1 reuse could otherwise be lost-updated + FK-nulled; the row is fileless and harmless.
- B3: admin storage dashboard reads sysconfig.get_int(db,'download_total_max_bytes')
(the admin-editable DB value GC enforces) instead of the raw env default.
- B4: the single-trim branch of normalize_edit_spec guards its float() coercion like the
crop/segments branches — a malformed trim now yields a 400, not an unhandled 500.
- B5: formats.normalize guards int(max_height) → falls back to "best" instead of 500.
Reviewed (race in an earlier B2 draft caught + fixed); localdev boots, B4/B5 unit-verified.
- C3: `_reference_url` (downloads.py) and public.py's inline source-URL block were
the same rule → extract `service.reference_url(job, asset)`; both surfaces now
share it so the "downloaded from" link can't drift between them.
- C4: Content-Disposition filename derivation (ext pick + doubled-ext strip + join)
was duplicated in download_file and watch_file → extract
`storage.download_filename(display_name, container, path)`.
- C5: inline the one-line `_clean_basename` passthrough (folded into C4).
- C6: the `db.get(MediaAsset, job.asset_id) if job.asset_id else None; _serialize(...)`
resolve-then-serialize dance was repeated across 6 single-job handlers → fold into
`_serialize_job(db, job)`. (File-serving handlers that use the asset for their own
checks keep their explicit resolve.)
Behavior-neutral; ruff/parse clean, localdev boots, downloads routes load.
A download whose source title exceeded 255 chars (e.g. a recipe baked into a
Facebook video's title) failed with a StringDataRightTruncation: the worker
auto-fills an empty display_name from the title, which overflows the 255-char
column. Clip the title to 255 at every point it flows into display_name (worker
completion paths + enqueue); the full title is untouched on the asset. Pre-existing
latent bug, surfaced by an unusually long title.
Release v0.31.1.
A queued/downloading row previously showed the bare video id and no thumbnail because asset
metadata was only filled when the download completed. Now:
- service.populate_from_catalog fills title/uploader/thumbnail/date/duration at enqueue from
our own Video/Channel catalog (0 network cost) — feed downloads look right instantly
- worker fills the same from yt-dlp's info_dict on the first progress event (covers ad-hoc
URLs / catalog misses), best-effort, never fails a download
Verified: a paused feed download now shows its real title, channel, thumbnail and duration.
The .gitignore 'downloads/' pattern for the DOWNLOAD_ROOT bind mount also matched the
backend/app/downloads source package, so formats.py/storage.py/service.py never got
committed with M2. Anchor the ignore to '/downloads/' (repo root only) and add the package.
storage.sanitize() now handles arbitrarily messy titles (emoji, ZWJ, fullwidth, clickbait
punctuation): NFKC-normalize, drop emoji/symbol/control unicode categories, collapse
repeated punctuation, underscore-join words -> space-free paths. Accents (HU/DE) preserved,
no ASCII folding. Plex layout uses _-_ separators + Season_{year}.
peter