- Correctness: _sync_overlay now first-wins on a guid (a `seen` set) so a duplicate HU entry
sharing a guid can't overwrite the applied Hungarian metadata or double-count `overlaid`.
- Test-coverage: cover the earlier dup-guid fix (both editions of a shared-guid pair get *_hu,
counted once) and the guid-keep guard (_apply_item preserves a prior guid when the listing
carries none, still updates on a real one).
Full backend suite 276 passed / 2 skipped.
- Correctness (stale HU): _sync_overlay is now a full reconcile — it clears every guid-bearing EN
row's *_hu first, then repopulates the matches, so a title whose HU twin vanished or was re-guid'd
falls back to EN instead of keeping stale Hungarian metadata (only rows that held HU data go dirty).
- Correctness (dup guid): by_guid maps a guid to the LIST of EN rows carrying it, so two editions of
one film both receive the overlay instead of just the last-iterated one.
- Robustness (guid wipe): _apply_item/_sync_shows keep a prior guid when a listing carries none
(`_canonical_guid(meta) or row.guid`), so a server ignoring includeGuids can't erase the match key.
- Reuse: merged _sync_overlay's movie/show branches (only the select + item_type differ).
- Test-coverage: a full-sync test drives sync() over a stub Plex (sections + section_items +
overlay_map) asserting pass-2 designation (metadata_lang/overlay_of), guid capture, the HU write,
and browse exclusion; plus a reconcile test that a vanished twin clears stale *_hu.
Full backend suite 274 passed / 2 skipped.
Adds the storage + sync for the Hungarian-metadata overlay (the browsable EN sections gain HU
twin fields written from a paired HU section, matched by guid; no browsable HU rows).
Schema (2 migrations, ADD-only; run before the first HU sync while *_hu is NULL so the generated
column build is trivial):
- 0063: plex_libraries.metadata_lang ('en' default) + self-FK overlay_of.
- 0064: plex_items/plex_shows.guid (indexed) + title_hu/summary_hu/genres_hu (+ GIN) + a STORED
generated search_vector_hu mirroring the EN vector (title_hu A / people_text B / summary_hu C) + GIN.
Sync:
- includeGuids=1 on section_items so the canonical guid rides the cheap listing.
- _canonical_guid(meta) (tmdb > imdb > tvdb) captured onto every EN row in _apply_item/_sync_shows.
- Two-pass sync(): pass 1 mirrors browsable (non-overlay) sections; pass 2 reads plex_overlay_map
('huKey:enKey', new sysconfig), designates each HU section (metadata_lang='hu' + overlay_of), and
_sync_overlay writes title_hu/summary_hu/genres_hu onto the guid-matched EN rows.
Browse: _content_libs now excludes overlay libraries (overlay_of is None); the unified_library and
facets inline library partitions fold into it, and list_libraries/collections exclude overlays too.
Tests: DB-lane coverage of the overlay write (matched vs unmatched rows, no browsable HU rows, the
generated search_vector_hu populates), the guid parse, and the browse exclusion. Full suite 272
passed; migrations verified reversible (upgrade→downgrade→upgrade). S5b (persist the live detail
fields + serve item/show detail from DB) is the follow-up.
- Efficiency: cap each person section at _PERSON_SECTION_MAX (200) so a prolific person can't
return an unbounded list the client mounts a card per row for.
- Reuse: extract the optimistic watch-toggle (flip → persist → resync) into a shared
toggleWatchState helper; the grid and the person page each supply their own cache updater.
- Correctness (scroll): onOpenPerson now stashes infoScrollRef ONLY when leaving an info page
(not from a show/grid context), and a jumped-to info page resets it — so a stale offset can't
wrongly restore a fresh info page.
- Test-coverage: add PlexPersonView.dom.test.tsx (export CastStrip + PlexPersonView) covering
the cast-row rewiring (open-person vs filter fallback) and the person page's sections + filter
+ empty state.
The name-as-identity finding is inherent to Plex cast carrying no per-person id (PlexPerson is
name-keyed too); documented in the endpoint docstring, not structurally fixable in S4.
Backend: new GET /api/plex/person/{name} returns a cast/crew member's cached headshot plus
the movies (PlexItem, kind=movie) and shows (PlexShow) in the enabled libraries whose
cast_names OR directors include the name — the inverse of the cast → filter link. Introduce a
shared _content_libs(db) helper (enabled-library partition) and reuse it in people_cards;
S5 will extend it with the HU-overlay exclusion. Watch-state comes from PlexState / rollup
(reusing _movie_card / _show_card / state_map). DB-lane tests cover cast+director credits,
library scoping, the kind filter, unknown-name empties, and the partition helper.
Frontend: a new `person` history-subview. A cast-row click (PlexInfo cast row + the show
page's CastStrip) now opens the person's filmography page instead of filtering; the filter
gesture moves to a "Filter library by {name}" button on the page itself. PlexPersonView
mirrors PlexInfoView's fetch-wrapper + glass hero (headshot + name + credit counts) and
renders Movies / Shows as PlexPosterCard grids, with an optimistic per-card watch toggle on
the person cache. New plexPerson query key + api client + PlexPersonDetail type; HU/EN copy.
Address review findings on the title-clamp: TITLE_MAX_LEN now reads the length straight
from Notification.title so the clamp can't drift from the column a migration would change
(single source of truth). Add a db-lane test that actually commits a >255-char title into
Postgres (end-to-end proof the truncation is gone, not just the clamp logic) and one that
asserts the constant tracks the column width.
A downloaded FB/IG reel's "title" is the whole caption (up to ~1900 chars). It flowed
untrimmed into Notification.title (String(255)); at the batched download_gc commit that
raised psycopg StringDataRightTruncation in pass 1, aborting the whole run so expiry
deletion, LRU eviction and orphan reap never executed (13/110 prod assets already exceed
the limit). Clamp the title to 255 (with an ellipsis) centrally in create_notification —
defending all six callers — since the full title already reaches the client via `data`
and the column is only an English fallback. Add a regression test.
Add a search scope-pill to the Plex header search box (GitHub-style, one
field-scope at a time) and an instant typeahead of the top matches below it.
Backend (unified_library):
- New `field` param: all | title | people | summary | episode. `all` is
byte-identical to the pre-scope behaviour (untouched weighted search_vector);
a specific field keeps the GIN `search_vector @@ tsq` prefilter AND ANDs a
per-column tsvector recheck, so scoped hits are a strict subset of Everywhere.
Per-field expressions mirror the search_vector Computed def (same TS config,
same left(summary,1000) truncation).
- `episode` scope empties the movie/show grid and title-only-matches the grouped
Episodes section; the title/people/summary scopes suppress that section.
- Unknown field falls back to `all`. New DB-lane characterization tests
(test_plex_field_scoped_search.py): each scope narrows correctly, field=all is
identical to today, episode empties the grid and needs shows in scope.
Frontend:
- PlexSearchBox wraps the shared SearchBar with the scope-pill (reusing
ActionMenu) and the typeahead; SearchBar gains generic leading/below/focus/
keydown slots so other modules' search boxes are untouched.
- Episode scope offered only when shows are in scope (movie-only hides it),
healed at the point of use like the sort state.
- Typeahead reuses /api/plex/library (limit=10), honours scope+field (not the
sidebar filters), shows poster+title+year, keyboard nav (arrows/Enter/Esc),
and jumps straight to a movie's info page or a show's drill-down via a new
provider jumpTo hand-off (mirrors the playlistOpen pattern).
- `field` persisted per-account; en/hu labels added.
- by_rating_key: generic-typed (type[_M] -> _M | None) so call sites keep their
Model | None static type.
- Migrate the last 2 open-coded PlexCollection-by-rating_key lookups (sync.py
resync_collection + delete_collection_local) to by_rating_key; zero open-coded
rating_key lookups now remain in the package.
- Add test_playlist_bulk_add_and_remove covering the bulk add/remove endpoints
and pinning the remove-bulk synchronize_session=False path.
Full suite 258 passed, 2 skipped; ruff clean.
- Restore synchronize_session=False on the remove-bulk Core delete (the 2.0
default is 'auto'; keep the pre-conversion contract explicit).
- _episode_nav: list(...scalars()) instead of [r[0] for r in ...all()], matching
the house style used across the conversion.
- Extract app/plex/lookups.by_rating_key() and migrate the get-by-rating_key
pattern at 13 sites (imgcache x4, _shared.item_or_404, playback show/season,
browse x6).
- Coverage: add TestFacetsWithData (the converted wsq aggregation with real
data — the highest-risk untested builder), TestPlaylistAndCollectionEndpoints
(playlist CRUD + collections list), and an item-state playback test.
Full suite 256 passed, 2 skipped; ruff clean.
Completes the plex-package 1.x->2.0 conversion, behaviour-equivalent (full
suite 253 passed, 2 skipped; the characterization net locks every path):
- routes/plex/browse.py (61 sites): libraries, facets (the wsq builder +
jsonb genre unnest), collections, playlists (+ bulk add/remove/reorder),
seasons, people/cards, collection strips, episode-nav, item detail, and
the unified_library union builder.
- plex/rollup.py: agg_subquery now returns a select(); state_map executes it
(the atomic unit coupled to unified_library via .subquery()).
Query builders -> select(); .filter/.filter_by -> .where; terminals ->
db.execute(...).scalars()/.all()/db.scalar(...); bulk .delete()/.update() ->
execute(delete()/update()). Zero db.query()/.filter_by remain in the package.
Mechanical 1.x->2.0 conversion of the self-contained plex modules, behaviour-
equivalent (the characterization net + existing plex tests stay green, 39
passed): app/plex/{sync(15),watch_sync(9),people(1),imgcache(4)} and
app/routes/plex/{admin(4),playback(8),_shared(1)}. Query builders become
select(); .first()/.all()->execute().scalars(); a bulk .update() ->
execute(update()); a single-column .first() -> db.scalar().
browse.py (61) + rollup.agg_subquery (the atomic unit with unified_library)
remain for wave 2.
Locks the observable behaviour of the previously-untested Plex query hot
paths before the 2.0 conversion, so it can be proven behaviour-equivalent:
- unified_library: both/movie/show scope, added-desc ordering, the four
watch-state branches, search rank + episode grouping, pagination.
- rollup.state_map: new / in_progress / watched rollup.
- sync upsert idempotency: _sync_movies + _sync_shows run twice = no dup
rows, in-place update, episode->show linking.
Green on the current 1.x code (9 passed). The conversion commits must keep
these identical.
Deep-review round 1 found the new tests asserted end-state only, so both
safety mechanisms this change adds were untested (a regression would ship
green):
- Per-page cursor durability: add test_a_mid_run_failure_leaves_the_prior_page
_durably_committed, which fails the 2nd page's fetch after page 1 committed
and asserts the cursor + rows are already persisted (re-read via db.expire).
Mutation-verified: reverting to a single end-of-loop commit makes it fail
while the pre-existing paging tests stay green — the property the original
test could not distinguish.
- Reserve guard: add test_backfill_reserve_reached_tracks_the_boundary, driving
real record_usage spend across the 9000/2000 default boundary and asserting
the un-mocked quota.backfill_reserve_reached() flips at exactly remaining<=reserve.
Full backend suite: 244 passed, 2 skipped (network-gated).
Addresses the code-review findings on the immediate full-history change:
- Extract quota.backfill_reserve_reached(): the remaining/reserve compare was
open-coded at seven sites (recent/deep/immediate backfill, explore, rolling
revalidation, and the new per-page guard). It reads in its OWN short-lived
session (like record_usage/log_action), so a per-page guard no longer holds
the caller's transaction open across the YouTube fetch that follows — the
400-page immediate run no longer pins a pooled connection idle-in-transaction
through each network round-trip. Migrated all call sites; dropped the now-dead
sysconfig imports in channels.py and maintenance.py.
- Simplify the deep-backfill loop: channel.backfill_done = not page_token (done
is False on entry) plus one break, replacing the doubled `if not page_token`.
- Tests patch the backfill_reserve_reached seam directly instead of mocking
remaining_today internals, removing the fragile dead-value budget iterator.
Full backend suite green; ruff clean.
The user-triggered "get full history" (backfill_channel_now) capped at
max_pages=60 = 3000 videos. A channel just over that (BBC Earth: 3403
uploads = 69 pages) stopped one page short, so its tail dribbled in over
the scheduler's paced 10-minute ticks (250 videos/tick) and the button
spinner looked stuck for ~18 minutes with no visible progress.
- Raise the immediate cap to max_pages=400 (20 000 videos) so any
realistic channel completes in the single background run.
- Make a high cap safe: backfill_channel_deep now re-checks the shared
quota before each page and yields at the backfill reserve (previously
quota was checked once by the caller before the whole run), and commits
the advanced cursor after every page so a restart resumes instead of
re-fetching — and re-charging quota for — pages already stored.
- Channel page: while loading, the button shows the live stored/total
count and a tooltip explaining the tail finishes in the background, so a
very large channel's paced remainder reads as progress, not a hang.
Adds deep-backfill paging tests (per-page cursor commit + resume, and the
per-page quota yield).
- Make the immediate-backfill guard SYMMETRIC: run_deep_backfill now also
registers the channel it pages in _backfill_in_progress, so it and
backfill_channel_now mutually exclude per-channel and never advance the
same cursor concurrently (whichever started first). The interactive cap
moves to a separate _immediate_running counter so the scheduler's channels
don't false-trip it.
- Attribute the immediate backfill's spend: wrap it in
quota.attribute(user_id, VIDEOS_BACKFILL_FULL) (route passes user.id), so
it shows under the user's full-backfill action, not the OTHER/system bucket.
- Tests: the update_channel->background-task wiring + the demo guard
(scheduled for a real user, NOT for demo); distinct N/S/L counts (2 short
+ 1 live) so a short/live swap fails; extracted channelDetailPollMs with a
pure unit test of the poll-or-stop predicate.
- a11y: role="img" on the fully-synced charm so screen readers announce it.
- list_channels no longer computes the Normal/Shorts/Live aggregation the
frontend stopped reading when R14 dropped the N/S/L table column; the dead
ManagedChannel.count_* fields are gone too.
- Immediate "get full history" backfill now coordinates with the paced
scheduler via a small in-flight set: capped at 2 concurrent + per-channel
dedup, and run_deep_backfill skips a channel an immediate backfill owns —
killing the cursor race and bounding the quota-drain burst, without losing
the speed win.
- Add test_channel_backfill.py: the channelDetail N/S/L counts + the guard's
no-op / cap / scheduler-skip behaviour.
- Inline the syncedChannels useMemo (its dep is a fresh array each render, so
it never cached); fix the stale "~2-min ticks" comment (it's 10 minutes).
Clicking "get full history" now kicks an immediate background deep
backfill of that one channel (backfill_channel_now: own session, up to
~3000 videos in one pass, quota-guarded) instead of waiting for the
scheduler's paced 10-minute ticks (5 pages/channel/tick). The request
returns instantly and the channel page's live poll shows the count fill
in seconds. A very large channel's tail still falls to the scheduler
(deep_requested stays set).
- DataTable: opt-in `fixedLayout` (colgroup + table-layout:fixed) so the
table fits its container at any zoom/text-size; only the channel manager
opts in — playlists/discovery/Plex keep auto-sizing.
- Remove the N/S/L and Sync columns from the manager table. Fully-synced
status becomes a ✓ charm on the Stored count; the sync filter moves to an
`applySyncFilter` pre-filter still driven by the top chips.
- Channel column is now fluid: ChannelLink truncates with the full name on
hover (title). Data columns get fixed widths so only Channel/About flex.
- N/S/L (normal/short/live) breakdown moves to the channel page: add the
three counts to the channelDetail endpoint (one grouped query, same
filters as the list) + the ChannelDetail type + a meta-line entry.
- Drop the now-dead sync/types i18n keys (en+hu).
- Floor the effective max_concurrent at 1 on the read path (quota.resolve)
and in the claim SQL (GREATEST(..., 1)). admin_set_quota already floors on
write and the sysconfig default's min is 1, but a stray stored 0 (direct DB
edit) would diverge the two "at limit" checks — the SQL claim-window
exclusion can't see a user "capped" with zero running jobs, while
at_concurrency_limit can — reopening the starvation the fairness fix closes.
Keeping the effective limit >= 1 makes the two checks provably agree.
- shared_with_me now orders by (created_at DESC, id DESC) to match
list_downloads, so the newly-added 500 cap has a stable boundary among
same-timestamp shares instead of a non-deterministic one.
- 0062 upgrade loop: rename the unused loop var "table" to "_table"
(matching the underscore-marked "_col").
Test: a stored max_concurrent=0 flooder no longer wedges the claim loop.
#2 — _claim_job excluded users at their concurrency limit only in Python,
AFTER a fixed LIMIT-50 fetch window. One user flooding the queue with >50
queued jobs while at their cap filled the whole window, chosen came back None,
and every other user's work past position 50 was never seen. Exclude capped
users IN THE SQL (per-user max_concurrent from the quota row, else the
sysconfig default; unlimited users never capped), keeping the Python re-check
as a race guard that can only reject, never over-claim.
#3 — list_downloads and shared_with_me loaded every job for a user with no
bound. Cap both at 500 newest-first (a shared _MAX_LIST, matching the admin
listing's existing cap); the Download Center renders newest-first and 500 is
far beyond any real manual-download history, so no frontend change is needed.
Tests: DB-backed — a capped user's 60-job flood no longer starves a user
behind the window; oldest-by-queue_pos claim, all-capped->None, unlimited
never excluded; and the list cap bounds the result set.
Each column already leads a composite UNIQUE constraint whose btree covers
leftmost-prefix lookups on it (and the ON DELETE CASCADE reverse lookups),
so the extra single-column index=True was pure write-amplification. Drop
index=True on the 10 columns + alembic 0062 drops the existing indexes
(DROP INDEX IF EXISTS; downgrade recreates). Verified: migration up/down
round-trips on localdev (10 indexes drop, composites intact), siftlode
check green (alembic upgrade head incl. 0062 + pytest 193). The 10 do not
appear in an alembic-check model/DB diff → models and migration agree.
R12 S5 item: the feed's FINISH_MARGIN_SECONDS (10) and the Plex player's
_FINISH_MARGIN_S (30) were flagged as drifted duplicates. They are a
deliberate divergence, not drift — the feed clears a short video's resume
bar, Plex marks long-form items (movies/episodes with credits) watched.
Cross-reference + document both so it reads as intentional.
resolve_adapter matched sources by loose substring over the netloc, so a
look-alike host ('myyoutube.com', 'youtube.com.evil') misresolved. Match the
host exactly or as a subdomain via a _host_is() helper, and identify from the
netloc of BOTH source_ref and yt-dlp's webpage_url — the latter by host only,
never a substring over its path. This also restores a host signal to the
early-fill path (resolve_adapter(None, None, info)), which has no source_ref,
so a Facebook asset's uploader_url no longer depends on extractor_key being
present in the progress-hook info_dict.
Tests: look-alike hosts stay generic while real subdomains resolve; a
webpage_url netloc identifies the source when extractor_key is absent.
- resolve_adapter: identify from the yt-dlp extractor_key only, not the full
webpage_url (a sibling source's name in a URL path no longer misresolves);
add youtube-nocookie.com to the YouTube host match so nocookie embeds keep
the POT/player-client extraction strategy.
- _download: re-resolve the adapter with the post-extraction info dict so the
.nfo uniqueid type and uploader_url agree on one source (dedups the inline
resolve too).
- MediaMeta.uniqueid_type defaults to source-neutral "generic" instead of
"youtube", so an un-updated caller can never falsely label a non-YouTube
download as youtube.
- Tests: lock extractor-key-only resolution + nocookie host; default nfo type.
write_sidecars / delete_asset_files computed `base = media.with_suffix("")` and then
`base.with_suffix(".nfo")`. Stripping the real `.mp4` first leaves a stem whose own last dot may
be inside the title (e.g. "Sonka Házilag."), which `with_suffix` then treats as the extension and
clobbers — writing the .nfo/.jpg under a TRUNCATED basename that no longer matches the media, so
Plex/Kodi never associate them (and delete_asset_files targets the wrong name, leaking orphans).
Swap the extension on the FULL media path via a shared `_sidecar` helper instead.
Pre-existing bug (predates the source-adapters work); found during its E2E and folded in here.
Verified end-to-end: a dotted-title Facebook reel now writes a single shared basename.
Consolidates source-specific policy (youtube / facebook / reddit / generic) that was scattered
across worker.py, backfill.py, service.py, formats.py and storage.py with no shared source
identity. app/downloads/adapters.py holds the base (= generic yt-dlp behaviour) + per-source
overrides; resolve_adapter(source_kind, source_ref, info) picks one by source_kind, the yt-dlp
extractor_key, or the URL host, falling back to generic. yt-dlp stays the engine — the adapter is
policy only.
Moved into adapters (behaviour-equivalent):
- uploader_url derivation (YouTube channel_url / Facebook uploader_id / reddit None) — removes the
worker/backfill duplication (build_uploader_url + _should_refetch gone).
- the player-client fallback sets + retriable markers + YouTube extractor_args/POT (build_ydl_opts
now takes source-agnostic opts plus an `extra` dict from the adapter).
- source_url reconstruction (service.source_url delegates to the adapter).
Two intentional (approved) behaviour changes:
- the player-client fallback is now YouTube-only; non-YouTube sources make a single extraction
attempt instead of a pointless double attempt with youtube-only client args.
- the .nfo <uniqueid type=...> follows the source (adapter.nfo_uniqueid_type) instead of the
hardcoded "youtube" that mislabelled every url-source download.
Deliberately left generic (documented non-moves): title cleaning (normalize_title is harmless
across sources), the classify marker table, thumbnail_to_jpg, the format selectors, and og.py's
YouTube-hqdefault og:image preference (a self-contained thumbnail heuristic on the share path).
Verified end-to-end on localdev across all four adapters: downloads succeed and uploader_url + the
.nfo uniqueid type are correct per source.
- worker.py: keep an absolute wall-clock backstop for ANY still-alive child. A child that
reached a terminal state (done/error) but wedges in teardown is now killed to free the worker
slot ("abandon"), with its DB status left intact; only a still-'running' over-cap job is failed.
Previously gating the wall-clock on status=='running' let a wedged finisher loop forever (#1).
- worker.py: reconcile the claimed asset on ANY clean child exit (not only non-zero), so a child
that exits without finalizing it — e.g. an early return on a job DELETEd mid-flight — no longer
leaves the asset stuck 'downloading' until boot recovery (#2).
- worker.py: extract _stop_and_free + _fail_after_kill; the exception / stall / timeout / cancel /
shutdown / abandon paths now share one kill+free (+fail) tail instead of three copies (#3).
- tests: a wedged-finished child is killed by the wall-clock without a status change; clean exits
now assert the reconciliation no-op (#1, #4).
- worker.py: the watchdog loop now treats only 'paused'/'cancelled' (or a vanished row) as a
stop; a terminal status (done/error, or a re-queued contended asset) means the child already
wrote its outcome and is exiting, so keep polling instead of killing it. Previously a
proc.wait timeout landing between the child's terminal commit and its exit was misread as a
route-cancel and SIGKILLed a successful download (dropping its completion notification). The
stall/wall-clock/shutdown checks are now gated on status=='running' (#1, #5 test).
- worker.py: guard the crash-path _job_liveness probe so a DB blip there can't propagate (the
child is already dead; boot recovery is the backstop) (#2).
- worker.py: drop _stop_child's redundant poll guard — escalate_kill no-ops on an exited proc
and getpgid yields None for a reaped pid (#3).
- worker.py: note the extraction heartbeat is per-attempt (a single over-timeout attempt is
still killed, bounded by socket_timeout) (#4).
- worker.py: wrap the watchdog loop so a failure in it (e.g. a transient DB error in
_job_liveness) still hard-kills the child + fails the job, instead of leaving the child
running unsupervised — the exact wedge the feature prevents (#1).
- worker.py: _touch_job heartbeat bumps updated_at at each extraction attempt, so a slow /
multi-client-set fallback extraction (which writes no progress, and re-writes identical
values so ORM onupdate does not fire) is not false-killed as "stalled" (#2).
- edit.py: extract escalate_kill (terminate->wait->kill escalation); _stop and worker's
_stop_child (process-group variant) now share it (#3).
- worker.py: document the edit path's second, tighter run_ffmpeg watchdog and that the
supervisor is POSIX/Linux-container-only (#4, #6).
- tests: a long download stays alive while progress is fresh, and a watchdog-loop error
still stops the child (#1, #5).
Defense-in-depth after the ReDoS incident. The heavy work (yt-dlp download / ffmpeg edit)
now runs in a child process (app.downloads.runner) supervised by the worker thread, not in
the thread itself. A thread can't be killed and a C-level spin can't even be cooperatively
cancelled, so one wedged download used to pin a core, hold the GIL and freeze the whole
queue; a child process can be SIGTERM/SIGKILLed.
The supervisor kills the child (and its process group, so ffmpeg grandchildren die too)
when: a route pauses/cancels the job, it makes no progress for download_stall_timeout_seconds
(150s), it runs past download_wall_clock_max_seconds (2h), the worker is shutting down, or
the child crashes without setting state. On a kill it frees the asset and fails the job with
a retryable network error (stalls are not auto-requeued). A clean exit is left alone — the
child already wrote the terminal state, including a cooperative pause via the progress hook.
- config.py: stall / wall-clock / supervisor-poll / kill-grace settings.
- worker.py: _run_supervised + _spawn_download_child / _job_liveness / _stop_child /
_cleanup_staging; _process_job claims the asset, then hands the heavy work to the child.
- app/downloads/runner.py: the child entrypoint.
- docker-compose.localdev.yml: init:true on the worker so tini (PID 1) reaps orphaned
ffmpeg grandchildren left by a hard-kill (the worker reaps its own direct children).
- tests: watchdog decision logic + SIGTERM->SIGKILL escalation (fake proc, no real subprocess).
- edit.py: extract _run_ffmpeg_best_effort (one run+timeout+swallow policy shared by
thumbnail_to_jpg / ensure_poster / ensure_storyboard) + a build_thumbnail_jpg_cmd
builder, matching the module's other ffmpeg commands.
- thumbnail_to_jpg: copy an already-JPEG thumbnail (Facebook, and Reddit's jpeg-under-
a-.png-name) instead of re-encoding it; only webp/png-proper are decoded via ffmpeg.
- errors.py: drop the redundant "account authentication" classify marker
("authentication is required" already covers the known Reddit message).
- formats.py: document that the opt-in EmbedThumbnail is unaffected by dropping the
convertor (it converts unsupported thumbs to PNG, not the crashing mjpeg2jpeg path).
- test_titles.py: guard the ReDoS regression with a thread + timeout (deterministic
"must not hang") instead of a wall-clock assertion.
Three prod-observed download bugs, root-caused from a frozen prod queue with the
worker pinned at 100% CPU:
- ReDoS in normalize_title: _TRAILING_HASHTAGS had two overlapping unbounded
quantifiers around the required letter, so a Facebook title with a long hashtag
cluster that is NOT at the end ("... #a #b #c | Page") backtracked catastrophically.
It runs inside the yt-dlp progress hook holding the GIL, so one such download froze
the whole worker (and could stall feed enrichment too - sync/videos.py calls it).
Rewrite with a zero-width "has a letter" lookahead + a possessive tag body: one
unambiguous split, nothing to backtrack. 40-tag stress case: ~0.1ms.
- Reddit download killed by a cosmetic thumbnail convert: yt-dlp's
FFmpegThumbnailsConvertor runs "-bsf:v mjpeg2jpeg" on a JPEG-served-under-a-.png-name
(Reddit) and fails; being a postprocessor, it failed the WHOLE download even though
the media was fine. Drop it and produce the <base>.jpg poster/og:image ourselves via
editmod.thumbnail_to_jpg (plain ffmpeg decode->mjpeg, best-effort -> ensure_poster
cuts a frame on failure). The FB Messenger / MS Teams share preview keeps working
(posters verified as real JPEGs).
- classify: map "authentication is required" (Reddit's auth wall) to login_required
instead of unknown, so the UI explains it rather than offering a pointless retry.
Tests: ReDoS-timing + numeric-marker regression, the Reddit classify case, and
thumbnail_to_jpg (real JPEG out / None on bad input, never raises).
The R12 S3 underscore-publish left worker.uploader_url() sharing a name with the
`uploader_url` model attribute and the yt-dlp `"uploader_url"` info-dict key. The
function is now build_uploader_url() — the attribute, dict key, log line and
docstrings keep the plain `uploader_url` name. Rename touches only the def, its
two call sites, and the backfill import; no behaviour change (ruff clean,
pytest 193).
Layering cleanup, no behaviour change (ruff clean backend+scripts, pytest 193):
- Invert the scheduler->routes dependency: reset_demo_state (+ its demo-seed
helpers) move from routes/admin.py to a new service module app/demo.py.
Both routes/admin.py and scheduler.py now import it from app.demo, so the
scheduler no longer reaches into the routes layer — its lazy import (the
routes<->scheduler cycle workaround) is gone, hoisted to a top-level import.
- Publish the private symbols that were imported across module boundaries (the
underscore convention was lying): feed._serialize -> serialize,
feed._to_tsquery_str -> to_tsquery_str, videos._insert_stubs -> insert_stubs,
videos._stub_from_playlist_item -> stub_from_playlist_item,
worker._uploader_url -> uploader_url, plex._shared._item_or_404 ->
item_or_404. Scoped renames — the 5 other route modules keep their own truly
private _serialize helpers.
- _release_asset moves from routes/downloads.py to app/downloads/service.py as
the public release_asset (asset bookkeeping is a service concern); the routes
call it via service.release_asset.
Cleanup only, no behaviour change (ruff clean + pytest 193):
- keep the whole OAuth scope vocabulary together in app/auth.py: the bundles
(BASE_SCOPES/READ_SCOPES/WRITE_SCOPES/_YOUTUBE_SCOPES) move back next to the
atoms (READ_SCOPE/WRITE_SCOPE) as core; routes/auth.py imports the bundles it
needs instead of rebuilding them from the atoms.
- compact the const groups the split generator spread out (the six rate
limiters, the TTLs, the demo/scope constants) back into blocks.
R12 S2. app/auth.py (1154 lines) mixed the core auth surface with 18 HTTP
endpoints on one /auth router. Split by dependency graph:
- app/auth.py (223 lines) keeps ONLY the core surface other modules import
(current_user, resolved_user_id, admin_user, require_*, has_*_scope,
purge_user, count_admins, get_or_create_demo_user, google_enabled,
is_allowed, operator_contact, optional_current_user) plus everything it
transitively depends on.
- app/routes/auth.py (883 lines, new) gets the router + all 18 endpoints +
their endpoint-only helpers, importing the core surface from app.auth.
- main.py includes auth_routes.router (was app.auth.router).
- test_oauth_redirect imports _oauth_redirect_uri from its new home.
The STAY/MOVE partition is driven by an AST scan of the COMPLETE external
surface (every `from app.auth import` incl. multiline + `from app import
auth; auth.X`), so no still-referenced symbol was moved. Behaviour-equivalent:
every symbol relocated verbatim (get_source_segment equality), route set
proven identical, no undefined names (symtable sweep).
Gate: ruff clean (backend + scripts), pytest 193 passed.
Cleanup only, no behaviour change (ruff clean + pytest 193):
- imports hoisted above `router =` and grouped/sorted (isort) across the split
files; ruff-format pass restores the 2-blank-line PEP8 spacing the generator
collapsed to 1.
- the shared on-disk cache root moves to a neutral app/plex/cache.py, so
subtitles.py no longer imports it from imgcache.py (drops the cross-import).
- the former plex.py module docstring's access-model note (read endpoints open
to any authed user; admin endpoints test/sync) is restored in the package
__init__.py.
R12 S1 step B. The 2011-line routes/plex.py becomes an app/routes/plex/
package:
- admin.py — connection test, catalog sync, owner watch-link config (5)
- browse.py — libraries, facets, collections, playlists, show/item, people (23)
- playback.py — image/subtitle proxies, HLS streaming, watch-state writes (10)
- _shared.py — the two genuinely cross-module helpers (_item_or_404, log)
- __init__.py — assembles `router` from the three sub-routers; the app wiring
(plex_routes.router in main.py) is unchanged.
Mechanically generated and proven behaviour-equivalent: every function/class
source relocated verbatim (get_source_segment equality) and the (method, path)
route set is identical to before. Endpoints keep the /api/plex prefix on each
sub-router. No SQL change (db.query untouched; 2.0 select() is the next step).
Gate: ruff clean, pytest 193 passed (pure-logic + DB lane).
R12 S1 step A. Move the pure, leaf-level helpers out of the 2227-line
routes/plex.py into the app/plex service package with public names:
- app/plex/imgcache.py — on-disk poster/art cache + proxy (was _img_cache_*,
_image_key, _img_response, _IMG_CACHE/_IMG_SIZES). Owns the shared cache dir.
- app/plex/subtitles.py — WebVTT convert/time-shift + subtitle cache (was
_srt_to_vtt, _shift_vtt, _ffmpeg_to_vtt, _sub_cache_*, _BITMAP_SUBS).
- app/plex/rollup.py — show watch-state aggregation (was _rollup,
_show_status, _show_agg_subq, _show_state_map).
Behaviour-equivalent move: bodies verbatim, only the public API renamed and
call sites re-qualified. No endpoint or SQL change (db.query kept as-is; the
2.0 select() conversion is a later step). plex.py 2227 -> 2011 lines.
Gate: ruff clean, pytest 193 passed (pure-logic + DB lane).
- admin._decide: add an idempotency guard (409 unless status was 'pending') so a
repeat/stale approve OR deny can't re-run its side effects — now that deny emails,
this stops duplicate denial mails and a rejection mail to an already-approved user.
- SavedViewsWidget: extract runViewMutation(fn, failMsg) (returns success) — dedups
the five try/catch+refresh blocks AND lets create/rename close their input only on
success, so a failed create keeps the user's typed name.
- worker._notify_download_done: take user_id/title captured BEFORE the caller's commit
(no post-commit refresh SELECT); add a so the inbox labels an edit job
'Edit ready' instead of 'Download ready' (new inbox.editDone i18n, HU+EN).
Gate green (ruff/tsc/eslint-0/prettier/knip/vitest 79/pytest 193); smoke-tested the
saved-views + notifications surfaces render clean on a fresh dev server.
Give equal-weight actions equal-weight feedback (R11).
S1 Plex error toasts: the watched-toggles (card / season mark-all / episode /
info-panel / show mark-all) used to fail silently on 403/404/429; each now surfaces
the server reason via a shared notifyActionError() + resyncs the optimistic flip.
The playback auto-mark-watched stays intentionally silent (a mid-play toast is
noise; the next sync corrects it). New generic lib/actionErrors.ts (Plex + saved
views reuse it; notifyYouTubeActionError delegates its non-403 case to it).
S2 download-done inbox notification: worker.py emits a best-effort type=download_done
notice to the job owner on real completion (regular + edit jobs, not instant
cache-hits), rendered/localized in the inbox with an Open-Downloads link.
S3 optimistic rollbacks: SavedViewsWidget create/rename/toggle-default/remove/reorder
and the inbox revert-state no longer swallow failures — they toast + refetch to
resync (the reorder's optimistic order reverts to server truth).
S4 access-request deny: deny_invite now emails the requester (send_access_denied),
mirroring approve — no more silent limbo.
1A ChannelPage "Full history" button no longer wraps (whitespace-nowrap).
1B Extract the manager's press-and-hold priority stepper into a shared
PriorityStepper and add it to the channel detail header (subscribed, non-demo);
backend channel_detail now returns the per-subscription `priority`.
2 Unify YouTube + Plex media cards onto one look/behaviour: drop the whole-card
hover LIFT (VideoCard) — which also fixes the clipped-top bug — and instead the
inner thumbnail/poster ZOOMS on hover (both); give the flat Plex posters the same
glass-card 3D depth + accent hover border as the feed cards (shadow-md is a
transparent no-op here, so it's glass-card that provides the depth); clamp Plex
card titles to a fixed 2-line height so 1- vs 2-line titles align.
3 Nav rail is now always a slim icon-only column, flush to the left edge; each icon
reveals a floating name label on hover (Tooltip gains `always` + `openDelay`, and
a right side). Removed the pin/unpin toggle, the expand-on-hover mode and the
navCollapsed pref plumbing.
4 Scope the side-panel peek-open HOVER to the tab handle + the card (not the whole
full-height slot next to the rail), so approaching the nav no longer pops the
filter open; focus stays container-wide. useHoverFocusWithin gains a hover-out
grace period + split hover/focus handler groups.
5 Plex art backdrop fills the whole viewport (a fixed inset-0 layer behind the
transparent app) instead of only the inset <main> box.
6 Plex season binge continues across seasons: the season view's play queue now
spans the whole show (allKeys), so a season's last episode rolls into the next
season's first (seasonKeys stays for "add season to playlist").
7 When a Plex video/episode ends with nothing queued next, the player returns to
the previous page (onEnded → onClose) instead of stranding a black screen.
A full cold sync crashed and left plex_people empty (cast/people missing in
every detail view). enrich_row_people added PlexPerson rows via the ORM with no
ON CONFLICT, and because SessionLocal is autoflush=False a person shared by two
items in the same enrich batch was queued twice -> the single batch commit
violated the unique index ix_plex_people_name and rolled the whole batch back.
Prod dodged it by building up incrementally; dev hits it on every reseed+sync.
Insert missing people via an immediately-executed pg_insert with
on_conflict_do_nothing (deduped within the cast by name, keeping the first
non-null thumb), mirroring app/sync/videos.py:_insert_stubs. The immediate
execute also lets a later row in the batch see the row via the existing-names
query and skip it; the conflict guard additionally covers the concurrent
info-page enrichment path.
Adds a DB-backed regression test: two items with overlapping cast now enrich
without a unique-violation, one row per name, and a prior-NULL headshot is
back-filled. Mutation-checked (reverting the fix reddens it).
Address the two remaining review findings on the tag work:
- extract useChannelTagActions — the optimistic attach/detach + create-and-attach flow that
the channel manager and the channel detail page had copy-pasted; the cache-shape-specific
optimistic update is injected (manager maps a ManagedChannel[]; detail patches one
ChannelDetail), so the two stay in sync;
- add a DB-lane test asserting GET /api/channels/{id} returns exactly this user's personal
tag links — not another user's, not the auto/system (user_id NULL) links.
Bring the same tag control to the channel page, next to Subscribe:
- backend: GET /api/channels/{id} now returns this user's `tag_ids` for the channel
(added to _channel_detail_dict + the ChannelDetail interface), matching the manager
list shape;
- ChannelPage renders the shared ChannelTagsControl, wired to attach/detach/create with
an optimistic update on the channel-detail cache (reverting on failure) and a manager
list invalidation so a tag added here shows there too.
Reuses the R8 popover platform + ChannelTagsControl — no new tag UI.
Facebook's scraper rejected our self-hosted poster as "Corrupted Image": the poster is an
ffmpeg-produced JPEG (non-standard marker order) that browsers/iOS/Twitter decode fine but FB's
stricter processor won't. For a YouTube source, point og:image at the video's stable, non-expiring
`i.ytimg.com/vi/<id>/hqdefault.jpg` (a Google-encoded JPEG FB decodes reliably) built from the id in
the stored thumbnail — instead of the stored `sqp`-signed variant, which can expire. Non-YouTube
sources keep the poster fallback (fine everywhere but FB). Declares 480×360 so FB includes it on the
first async scrape. og.py only — no schema change.
- og: guard safe_abs_path's None (missing/gc'd poster) before _jpeg_size, else the public /watch OG
render crashed with an uncaught AttributeError
- channels: hoist the priority step button to module scope so it no longer remounts on every
hold-repeat tick (which dropped the pointer capture and could let a hold run away on pointer drift)
- watch: only route audio through Web Audio on iOS (where a backgrounded <video> is suspended);
other platforms keep their reliable native audio, avoiding a foreground-silence risk if the
AudioContext can't resume
- share: drop the unstable `create` mutation object from the auto-create effect deps (it changes
every render); the links.data trigger + autoTried ref already fire it exactly once
The clipboard-on-focus finding is intended (the user explicitly asked for auto-paste on focus; the
read is best-effort and browsers reject it without a gesture, so no real prompt fires).
The catalog only stores the recent-backfill/RSS subset of a channel's uploads (video_count is
YouTube's raw total), so a subscribed channel looked like it was "missing" videos. Now the channel
page header shows "{stored} / {total} videos" when there's a gap, and a "Full history" action queues
the deep backfill (the same deep_requested trigger the channel manager uses) — showing a loading
state until done. Backend: channel_detail now returns deep_requested/backfill_done.
peter