resolve_adapter matched sources by loose substring over the netloc, so a
look-alike host ('myyoutube.com', 'youtube.com.evil') misresolved. Match the
host exactly or as a subdomain via a _host_is() helper, and identify from the
netloc of BOTH source_ref and yt-dlp's webpage_url — the latter by host only,
never a substring over its path. This also restores a host signal to the
early-fill path (resolve_adapter(None, None, info)), which has no source_ref,
so a Facebook asset's uploader_url no longer depends on extractor_key being
present in the progress-hook info_dict.
Tests: look-alike hosts stay generic while real subdomains resolve; a
webpage_url netloc identifies the source when extractor_key is absent.
- resolve_adapter: identify from the yt-dlp extractor_key only, not the full
webpage_url (a sibling source's name in a URL path no longer misresolves);
add youtube-nocookie.com to the YouTube host match so nocookie embeds keep
the POT/player-client extraction strategy.
- _download: re-resolve the adapter with the post-extraction info dict so the
.nfo uniqueid type and uploader_url agree on one source (dedups the inline
resolve too).
- MediaMeta.uniqueid_type defaults to source-neutral "generic" instead of
"youtube", so an un-updated caller can never falsely label a non-YouTube
download as youtube.
- Tests: lock extractor-key-only resolution + nocookie host; default nfo type.
write_sidecars / delete_asset_files computed `base = media.with_suffix("")` and then
`base.with_suffix(".nfo")`. Stripping the real `.mp4` first leaves a stem whose own last dot may
be inside the title (e.g. "Sonka Házilag."), which `with_suffix` then treats as the extension and
clobbers — writing the .nfo/.jpg under a TRUNCATED basename that no longer matches the media, so
Plex/Kodi never associate them (and delete_asset_files targets the wrong name, leaking orphans).
Swap the extension on the FULL media path via a shared `_sidecar` helper instead.
Pre-existing bug (predates the source-adapters work); found during its E2E and folded in here.
Verified end-to-end: a dotted-title Facebook reel now writes a single shared basename.
Consolidates source-specific policy (youtube / facebook / reddit / generic) that was scattered
across worker.py, backfill.py, service.py, formats.py and storage.py with no shared source
identity. app/downloads/adapters.py holds the base (= generic yt-dlp behaviour) + per-source
overrides; resolve_adapter(source_kind, source_ref, info) picks one by source_kind, the yt-dlp
extractor_key, or the URL host, falling back to generic. yt-dlp stays the engine — the adapter is
policy only.
Moved into adapters (behaviour-equivalent):
- uploader_url derivation (YouTube channel_url / Facebook uploader_id / reddit None) — removes the
worker/backfill duplication (build_uploader_url + _should_refetch gone).
- the player-client fallback sets + retriable markers + YouTube extractor_args/POT (build_ydl_opts
now takes source-agnostic opts plus an `extra` dict from the adapter).
- source_url reconstruction (service.source_url delegates to the adapter).
Two intentional (approved) behaviour changes:
- the player-client fallback is now YouTube-only; non-YouTube sources make a single extraction
attempt instead of a pointless double attempt with youtube-only client args.
- the .nfo <uniqueid type=...> follows the source (adapter.nfo_uniqueid_type) instead of the
hardcoded "youtube" that mislabelled every url-source download.
Deliberately left generic (documented non-moves): title cleaning (normalize_title is harmless
across sources), the classify marker table, thumbnail_to_jpg, the format selectors, and og.py's
YouTube-hqdefault og:image preference (a self-contained thumbnail heuristic on the share path).
Verified end-to-end on localdev across all four adapters: downloads succeed and uploader_url + the
.nfo uniqueid type are correct per source.
- worker.py: keep an absolute wall-clock backstop for ANY still-alive child. A child that
reached a terminal state (done/error) but wedges in teardown is now killed to free the worker
slot ("abandon"), with its DB status left intact; only a still-'running' over-cap job is failed.
Previously gating the wall-clock on status=='running' let a wedged finisher loop forever (#1).
- worker.py: reconcile the claimed asset on ANY clean child exit (not only non-zero), so a child
that exits without finalizing it — e.g. an early return on a job DELETEd mid-flight — no longer
leaves the asset stuck 'downloading' until boot recovery (#2).
- worker.py: extract _stop_and_free + _fail_after_kill; the exception / stall / timeout / cancel /
shutdown / abandon paths now share one kill+free (+fail) tail instead of three copies (#3).
- tests: a wedged-finished child is killed by the wall-clock without a status change; clean exits
now assert the reconciliation no-op (#1, #4).
- worker.py: the watchdog loop now treats only 'paused'/'cancelled' (or a vanished row) as a
stop; a terminal status (done/error, or a re-queued contended asset) means the child already
wrote its outcome and is exiting, so keep polling instead of killing it. Previously a
proc.wait timeout landing between the child's terminal commit and its exit was misread as a
route-cancel and SIGKILLed a successful download (dropping its completion notification). The
stall/wall-clock/shutdown checks are now gated on status=='running' (#1, #5 test).
- worker.py: guard the crash-path _job_liveness probe so a DB blip there can't propagate (the
child is already dead; boot recovery is the backstop) (#2).
- worker.py: drop _stop_child's redundant poll guard — escalate_kill no-ops on an exited proc
and getpgid yields None for a reaped pid (#3).
- worker.py: note the extraction heartbeat is per-attempt (a single over-timeout attempt is
still killed, bounded by socket_timeout) (#4).
- worker.py: wrap the watchdog loop so a failure in it (e.g. a transient DB error in
_job_liveness) still hard-kills the child + fails the job, instead of leaving the child
running unsupervised — the exact wedge the feature prevents (#1).
- worker.py: _touch_job heartbeat bumps updated_at at each extraction attempt, so a slow /
multi-client-set fallback extraction (which writes no progress, and re-writes identical
values so ORM onupdate does not fire) is not false-killed as "stalled" (#2).
- edit.py: extract escalate_kill (terminate->wait->kill escalation); _stop and worker's
_stop_child (process-group variant) now share it (#3).
- worker.py: document the edit path's second, tighter run_ffmpeg watchdog and that the
supervisor is POSIX/Linux-container-only (#4, #6).
- tests: a long download stays alive while progress is fresh, and a watchdog-loop error
still stops the child (#1, #5).
Defense-in-depth after the ReDoS incident. The heavy work (yt-dlp download / ffmpeg edit)
now runs in a child process (app.downloads.runner) supervised by the worker thread, not in
the thread itself. A thread can't be killed and a C-level spin can't even be cooperatively
cancelled, so one wedged download used to pin a core, hold the GIL and freeze the whole
queue; a child process can be SIGTERM/SIGKILLed.
The supervisor kills the child (and its process group, so ffmpeg grandchildren die too)
when: a route pauses/cancels the job, it makes no progress for download_stall_timeout_seconds
(150s), it runs past download_wall_clock_max_seconds (2h), the worker is shutting down, or
the child crashes without setting state. On a kill it frees the asset and fails the job with
a retryable network error (stalls are not auto-requeued). A clean exit is left alone — the
child already wrote the terminal state, including a cooperative pause via the progress hook.
- config.py: stall / wall-clock / supervisor-poll / kill-grace settings.
- worker.py: _run_supervised + _spawn_download_child / _job_liveness / _stop_child /
_cleanup_staging; _process_job claims the asset, then hands the heavy work to the child.
- app/downloads/runner.py: the child entrypoint.
- docker-compose.localdev.yml: init:true on the worker so tini (PID 1) reaps orphaned
ffmpeg grandchildren left by a hard-kill (the worker reaps its own direct children).
- tests: watchdog decision logic + SIGTERM->SIGKILL escalation (fake proc, no real subprocess).
- edit.py: extract _run_ffmpeg_best_effort (one run+timeout+swallow policy shared by
thumbnail_to_jpg / ensure_poster / ensure_storyboard) + a build_thumbnail_jpg_cmd
builder, matching the module's other ffmpeg commands.
- thumbnail_to_jpg: copy an already-JPEG thumbnail (Facebook, and Reddit's jpeg-under-
a-.png-name) instead of re-encoding it; only webp/png-proper are decoded via ffmpeg.
- errors.py: drop the redundant "account authentication" classify marker
("authentication is required" already covers the known Reddit message).
- formats.py: document that the opt-in EmbedThumbnail is unaffected by dropping the
convertor (it converts unsupported thumbs to PNG, not the crashing mjpeg2jpeg path).
- test_titles.py: guard the ReDoS regression with a thread + timeout (deterministic
"must not hang") instead of a wall-clock assertion.
Three prod-observed download bugs, root-caused from a frozen prod queue with the
worker pinned at 100% CPU:
- ReDoS in normalize_title: _TRAILING_HASHTAGS had two overlapping unbounded
quantifiers around the required letter, so a Facebook title with a long hashtag
cluster that is NOT at the end ("... #a #b #c | Page") backtracked catastrophically.
It runs inside the yt-dlp progress hook holding the GIL, so one such download froze
the whole worker (and could stall feed enrichment too - sync/videos.py calls it).
Rewrite with a zero-width "has a letter" lookahead + a possessive tag body: one
unambiguous split, nothing to backtrack. 40-tag stress case: ~0.1ms.
- Reddit download killed by a cosmetic thumbnail convert: yt-dlp's
FFmpegThumbnailsConvertor runs "-bsf:v mjpeg2jpeg" on a JPEG-served-under-a-.png-name
(Reddit) and fails; being a postprocessor, it failed the WHOLE download even though
the media was fine. Drop it and produce the <base>.jpg poster/og:image ourselves via
editmod.thumbnail_to_jpg (plain ffmpeg decode->mjpeg, best-effort -> ensure_poster
cuts a frame on failure). The FB Messenger / MS Teams share preview keeps working
(posters verified as real JPEGs).
- classify: map "authentication is required" (Reddit's auth wall) to login_required
instead of unknown, so the UI explains it rather than offering a pointless retry.
Tests: ReDoS-timing + numeric-marker regression, the Reddit classify case, and
thumbnail_to_jpg (real JPEG out / None on bad input, never raises).
The R12 S3 underscore-publish left worker.uploader_url() sharing a name with the
`uploader_url` model attribute and the yt-dlp `"uploader_url"` info-dict key. The
function is now build_uploader_url() — the attribute, dict key, log line and
docstrings keep the plain `uploader_url` name. Rename touches only the def, its
two call sites, and the backfill import; no behaviour change (ruff clean,
pytest 193).
Layering cleanup, no behaviour change (ruff clean backend+scripts, pytest 193):
- Invert the scheduler->routes dependency: reset_demo_state (+ its demo-seed
helpers) move from routes/admin.py to a new service module app/demo.py.
Both routes/admin.py and scheduler.py now import it from app.demo, so the
scheduler no longer reaches into the routes layer — its lazy import (the
routes<->scheduler cycle workaround) is gone, hoisted to a top-level import.
- Publish the private symbols that were imported across module boundaries (the
underscore convention was lying): feed._serialize -> serialize,
feed._to_tsquery_str -> to_tsquery_str, videos._insert_stubs -> insert_stubs,
videos._stub_from_playlist_item -> stub_from_playlist_item,
worker._uploader_url -> uploader_url, plex._shared._item_or_404 ->
item_or_404. Scoped renames — the 5 other route modules keep their own truly
private _serialize helpers.
- _release_asset moves from routes/downloads.py to app/downloads/service.py as
the public release_asset (asset bookkeeping is a service concern); the routes
call it via service.release_asset.
Cleanup only, no behaviour change (ruff clean + pytest 193):
- keep the whole OAuth scope vocabulary together in app/auth.py: the bundles
(BASE_SCOPES/READ_SCOPES/WRITE_SCOPES/_YOUTUBE_SCOPES) move back next to the
atoms (READ_SCOPE/WRITE_SCOPE) as core; routes/auth.py imports the bundles it
needs instead of rebuilding them from the atoms.
- compact the const groups the split generator spread out (the six rate
limiters, the TTLs, the demo/scope constants) back into blocks.
R12 S2. app/auth.py (1154 lines) mixed the core auth surface with 18 HTTP
endpoints on one /auth router. Split by dependency graph:
- app/auth.py (223 lines) keeps ONLY the core surface other modules import
(current_user, resolved_user_id, admin_user, require_*, has_*_scope,
purge_user, count_admins, get_or_create_demo_user, google_enabled,
is_allowed, operator_contact, optional_current_user) plus everything it
transitively depends on.
- app/routes/auth.py (883 lines, new) gets the router + all 18 endpoints +
their endpoint-only helpers, importing the core surface from app.auth.
- main.py includes auth_routes.router (was app.auth.router).
- test_oauth_redirect imports _oauth_redirect_uri from its new home.
The STAY/MOVE partition is driven by an AST scan of the COMPLETE external
surface (every `from app.auth import` incl. multiline + `from app import
auth; auth.X`), so no still-referenced symbol was moved. Behaviour-equivalent:
every symbol relocated verbatim (get_source_segment equality), route set
proven identical, no undefined names (symtable sweep).
Gate: ruff clean (backend + scripts), pytest 193 passed.
Cleanup only, no behaviour change (ruff clean + pytest 193):
- imports hoisted above `router =` and grouped/sorted (isort) across the split
files; ruff-format pass restores the 2-blank-line PEP8 spacing the generator
collapsed to 1.
- the shared on-disk cache root moves to a neutral app/plex/cache.py, so
subtitles.py no longer imports it from imgcache.py (drops the cross-import).
- the former plex.py module docstring's access-model note (read endpoints open
to any authed user; admin endpoints test/sync) is restored in the package
__init__.py.
R12 S1 step B. The 2011-line routes/plex.py becomes an app/routes/plex/
package:
- admin.py — connection test, catalog sync, owner watch-link config (5)
- browse.py — libraries, facets, collections, playlists, show/item, people (23)
- playback.py — image/subtitle proxies, HLS streaming, watch-state writes (10)
- _shared.py — the two genuinely cross-module helpers (_item_or_404, log)
- __init__.py — assembles `router` from the three sub-routers; the app wiring
(plex_routes.router in main.py) is unchanged.
Mechanically generated and proven behaviour-equivalent: every function/class
source relocated verbatim (get_source_segment equality) and the (method, path)
route set is identical to before. Endpoints keep the /api/plex prefix on each
sub-router. No SQL change (db.query untouched; 2.0 select() is the next step).
Gate: ruff clean, pytest 193 passed (pure-logic + DB lane).
R12 S1 step A. Move the pure, leaf-level helpers out of the 2227-line
routes/plex.py into the app/plex service package with public names:
- app/plex/imgcache.py — on-disk poster/art cache + proxy (was _img_cache_*,
_image_key, _img_response, _IMG_CACHE/_IMG_SIZES). Owns the shared cache dir.
- app/plex/subtitles.py — WebVTT convert/time-shift + subtitle cache (was
_srt_to_vtt, _shift_vtt, _ffmpeg_to_vtt, _sub_cache_*, _BITMAP_SUBS).
- app/plex/rollup.py — show watch-state aggregation (was _rollup,
_show_status, _show_agg_subq, _show_state_map).
Behaviour-equivalent move: bodies verbatim, only the public API renamed and
call sites re-qualified. No endpoint or SQL change (db.query kept as-is; the
2.0 select() conversion is a later step). plex.py 2227 -> 2011 lines.
Gate: ruff clean, pytest 193 passed (pure-logic + DB lane).
- admin._decide: add an idempotency guard (409 unless status was 'pending') so a
repeat/stale approve OR deny can't re-run its side effects — now that deny emails,
this stops duplicate denial mails and a rejection mail to an already-approved user.
- SavedViewsWidget: extract runViewMutation(fn, failMsg) (returns success) — dedups
the five try/catch+refresh blocks AND lets create/rename close their input only on
success, so a failed create keeps the user's typed name.
- worker._notify_download_done: take user_id/title captured BEFORE the caller's commit
(no post-commit refresh SELECT); add a so the inbox labels an edit job
'Edit ready' instead of 'Download ready' (new inbox.editDone i18n, HU+EN).
Gate green (ruff/tsc/eslint-0/prettier/knip/vitest 79/pytest 193); smoke-tested the
saved-views + notifications surfaces render clean on a fresh dev server.
Give equal-weight actions equal-weight feedback (R11).
S1 Plex error toasts: the watched-toggles (card / season mark-all / episode /
info-panel / show mark-all) used to fail silently on 403/404/429; each now surfaces
the server reason via a shared notifyActionError() + resyncs the optimistic flip.
The playback auto-mark-watched stays intentionally silent (a mid-play toast is
noise; the next sync corrects it). New generic lib/actionErrors.ts (Plex + saved
views reuse it; notifyYouTubeActionError delegates its non-403 case to it).
S2 download-done inbox notification: worker.py emits a best-effort type=download_done
notice to the job owner on real completion (regular + edit jobs, not instant
cache-hits), rendered/localized in the inbox with an Open-Downloads link.
S3 optimistic rollbacks: SavedViewsWidget create/rename/toggle-default/remove/reorder
and the inbox revert-state no longer swallow failures — they toast + refetch to
resync (the reorder's optimistic order reverts to server truth).
S4 access-request deny: deny_invite now emails the requester (send_access_denied),
mirroring approve — no more silent limbo.
1A ChannelPage "Full history" button no longer wraps (whitespace-nowrap).
1B Extract the manager's press-and-hold priority stepper into a shared
PriorityStepper and add it to the channel detail header (subscribed, non-demo);
backend channel_detail now returns the per-subscription `priority`.
2 Unify YouTube + Plex media cards onto one look/behaviour: drop the whole-card
hover LIFT (VideoCard) — which also fixes the clipped-top bug — and instead the
inner thumbnail/poster ZOOMS on hover (both); give the flat Plex posters the same
glass-card 3D depth + accent hover border as the feed cards (shadow-md is a
transparent no-op here, so it's glass-card that provides the depth); clamp Plex
card titles to a fixed 2-line height so 1- vs 2-line titles align.
3 Nav rail is now always a slim icon-only column, flush to the left edge; each icon
reveals a floating name label on hover (Tooltip gains `always` + `openDelay`, and
a right side). Removed the pin/unpin toggle, the expand-on-hover mode and the
navCollapsed pref plumbing.
4 Scope the side-panel peek-open HOVER to the tab handle + the card (not the whole
full-height slot next to the rail), so approaching the nav no longer pops the
filter open; focus stays container-wide. useHoverFocusWithin gains a hover-out
grace period + split hover/focus handler groups.
5 Plex art backdrop fills the whole viewport (a fixed inset-0 layer behind the
transparent app) instead of only the inset <main> box.
6 Plex season binge continues across seasons: the season view's play queue now
spans the whole show (allKeys), so a season's last episode rolls into the next
season's first (seasonKeys stays for "add season to playlist").
7 When a Plex video/episode ends with nothing queued next, the player returns to
the previous page (onEnded → onClose) instead of stranding a black screen.
A full cold sync crashed and left plex_people empty (cast/people missing in
every detail view). enrich_row_people added PlexPerson rows via the ORM with no
ON CONFLICT, and because SessionLocal is autoflush=False a person shared by two
items in the same enrich batch was queued twice -> the single batch commit
violated the unique index ix_plex_people_name and rolled the whole batch back.
Prod dodged it by building up incrementally; dev hits it on every reseed+sync.
Insert missing people via an immediately-executed pg_insert with
on_conflict_do_nothing (deduped within the cast by name, keeping the first
non-null thumb), mirroring app/sync/videos.py:_insert_stubs. The immediate
execute also lets a later row in the batch see the row via the existing-names
query and skip it; the conflict guard additionally covers the concurrent
info-page enrichment path.
Adds a DB-backed regression test: two items with overlapping cast now enrich
without a unique-violation, one row per name, and a prior-NULL headshot is
back-filled. Mutation-checked (reverting the fix reddens it).
Address the two remaining review findings on the tag work:
- extract useChannelTagActions — the optimistic attach/detach + create-and-attach flow that
the channel manager and the channel detail page had copy-pasted; the cache-shape-specific
optimistic update is injected (manager maps a ManagedChannel[]; detail patches one
ChannelDetail), so the two stay in sync;
- add a DB-lane test asserting GET /api/channels/{id} returns exactly this user's personal
tag links — not another user's, not the auto/system (user_id NULL) links.
Bring the same tag control to the channel page, next to Subscribe:
- backend: GET /api/channels/{id} now returns this user's `tag_ids` for the channel
(added to _channel_detail_dict + the ChannelDetail interface), matching the manager
list shape;
- ChannelPage renders the shared ChannelTagsControl, wired to attach/detach/create with
an optimistic update on the channel-detail cache (reverting on failure) and a manager
list invalidation so a tag added here shows there too.
Reuses the R8 popover platform + ChannelTagsControl — no new tag UI.
Facebook's scraper rejected our self-hosted poster as "Corrupted Image": the poster is an
ffmpeg-produced JPEG (non-standard marker order) that browsers/iOS/Twitter decode fine but FB's
stricter processor won't. For a YouTube source, point og:image at the video's stable, non-expiring
`i.ytimg.com/vi/<id>/hqdefault.jpg` (a Google-encoded JPEG FB decodes reliably) built from the id in
the stored thumbnail — instead of the stored `sqp`-signed variant, which can expire. Non-YouTube
sources keep the poster fallback (fine everywhere but FB). Declares 480×360 so FB includes it on the
first async scrape. og.py only — no schema change.
- og: guard safe_abs_path's None (missing/gc'd poster) before _jpeg_size, else the public /watch OG
render crashed with an uncaught AttributeError
- channels: hoist the priority step button to module scope so it no longer remounts on every
hold-repeat tick (which dropped the pointer capture and could let a hold run away on pointer drift)
- watch: only route audio through Web Audio on iOS (where a backgrounded <video> is suspended);
other platforms keep their reliable native audio, avoiding a foreground-silence risk if the
AudioContext can't resume
- share: drop the unstable `create` mutation object from the auto-create effect deps (it changes
every render); the links.data trigger + autoTried ref already fire it exactly once
The clipboard-on-focus finding is intended (the user explicitly asked for auto-paste on focus; the
read is best-effort and browsers reject it without a gesture, so no real prompt fires).
The catalog only stores the recent-backfill/RSS subset of a channel's uploads (video_count is
YouTube's raw total), so a subscribed channel looked like it was "missing" videos. Now the channel
page header shows "{stored} / {total} videos" when there's a gap, and a "Full history" action queues
the deep backfill (the same deep_requested trigger the channel manager uses) — showing a loading
state until done. Backend: channel_detail now returns deep_requested/backfill_done.
The FB Sharing Debugger confirmed the real cause of the missing desktop-Messenger preview: the
scraper fetches og:image ASYNCHRONOUSLY on a URL's first scrape ("The provided 'og:image'
properties are not yet available… specify the dimensions using 'og:image:width'/'og:image:height'"),
so a freshly shared /watch link unfurled title-only. Read the poster JPEG's real dimensions from its
SOF marker (Pillow isn't a dep; the poster aspect differs from the video's, so the stored video
width/height won't do) and emit og:image:width/height. Only runs on a crawler hit. Unit-tested.
The iPad Messenger app unfurls a share link on-device (fetches + renders the OG tags itself),
but messenger.com desktop shows only what Facebook's server-side scraper cached — which is
pickier. Add og:image:secure_url (public HTTPS) and og:image:type=image/jpeg for our poster so
the card renders even when the scraper won't sniff the bytes. Reachability/stale-cache is a
separate, live-diagnosis step.
Failed downloads stored the raw yt-dlp/ffmpeg text and dumped it as a truncated red
one-liner in the queue. Classify the failure into a category (unavailable / login_required /
geo_blocked / format_unavailable / drm / postprocess / source_missing / network / unknown)
and surface it properly:
- backend: app/downloads/errors.classify maps the raw message to a category code; the worker
stores it in the new download_jobs.error_code (migration 0061) at both failure sites; resume
clears it; the serializer exposes it. Raw text kept as technical detail.
- frontend: the queue row shows a clean localized reason + a "details" link opening a modal
with the explanation, the raw text behind a disclosure, and a Retry only for transient
categories (network/postprocess/unknown).
- tests: classify() unit coverage over verbatim yt-dlp/ffmpeg message shapes.
/code-review high on S2 — 4 findings, 2 fixed, 2 no-change:
- test_plex_sync_txn: a faked PlexClient fails on the second library; asserts the
first library's row survives (per-section commit). Mutation-checked — moving the
commit back out of the loop turns it red ([] == ['1']).
- dropped the `[[quota-owns-its-session]]` memory slugs from quota.py docstrings
(a dangling private-note reference in shipped code; the reasoning is already inline).
No change: record_usage's per-call second connection is a transient spike, fine for
this app's concurrency; the two-site `with SessionLocal()` duplication isn't worth a
helper until a third caller appears.
Gate: ruff clean; DB lane 181 green.
Transaction ownership (C-A5):
- quota.record_usage / log_action write in their OWN SessionLocal(), committed
immediately, and no longer take/commit the caller's db. The spend stays DURABLE
regardless of the caller (read-only YouTube paths never commit; a failed job rolls
back) — under-counting the shared daily budget would risk overspending the real
YouTube quota — AND quota stops flushing a job's partial state mid-run behind its
own `except: db.rollback()`. units_used_today switches to a scalar SELECT so
measured()'s before/after diff sees the separate session's commits (READ COMMITTED).
Call sites updated (youtube/client.py ×3, routes/search.py ×1).
- plex sync commits PER SECTION (moved the end-of-loop commit inside), so a PlexError
on a later library keeps the ones already mirrored (idempotent; next run reconciles).
The sync jobs did NOT rely on record_usage's incidental commit for persistence — their
per-item functions (apply_rss_feed, backfill_channel_recent, import_subscriptions,
sync_user_playlists, apply_channel_autotags) all commit internally, so the in-loop
rollback handlers already isolate a failed item. No loop restructuring needed.
DB-lane tests (test_quota_ownership, 4): spend survives a caller rollback while the
caller's uncommitted row does NOT leak; accumulation; measured-style mid-txn read.
Gate: ruff clean; DB lane 180 green.
/code-review high on S3 — 3 low findings, 2 fixed, 1 no-change:
- _db_reachable() now probes via a throwaway engine with connect_timeout=3, so a
slow/firewalled DATABASE_URL can't hang the whole suite at import (the app engine
has no connect timeout).
- test_plex_facets sets the "configured" gate flag via monkeypatch (auto-restored)
instead of poking state._configured_cache in a manual finally — no leak risk, and
the now-unnecessary mark_configured DB write is dropped.
No change: test_plex_jsonb_normalize copying migration 0060's SQL is deliberate —
migrations must stay self-contained (no app imports) and a shipped migration is
immutable, so there's no real drift to guard against.
Gate: ruff clean; DB lane 176 passed; pure-logic 172 passed / 4 skipped (fast).
First real customers of the DB lane:
- test_plex_facets: GET /api/plex/facets?scope=show returns 200 (not the
"cannot extract elements from a scalar" 500) for a show whose genres is a JSONB
scalar `null` — the 0.53.1 read-guard's regression test. Mutation-checked:
removing the jsonb_typeof='array' guard turns it red with that exact error.
- migration 0060 (S3a): one-time UPDATE …=NULL WHERE jsonb_typeof=… 'null' over the
plex_items/plex_shows tag columns, so pre-0.53.1 JSONB-null rows become uniform SQL
NULL and the read-guards go belt-only. test_plex_jsonb_normalize verifies scalar-null
→ SQL NULL while a real array is preserved.
Gate: ruff clean; DB lane 176 passed; pure-logic lane unchanged (skips the DB tests).
The suite was pure-logic only (R2 S2b deferred the DB lane). Add an opt-in `db`
fixture (tests/conftest.py) that hands a test a real session against an ephemeral
Postgres; schema is built with `alembic upgrade head` (the real prod schema needs
the unaccent extension + unaccent_simple TS config that create_all can't provide,
and this also proves the migration chain applies). Each test starts from a TRUNCATEd
clean DB so tests that COMMIT — the whole point, for R6 S2's transaction work —
don't leak.
backend/docker-compose.test.yml wires an ephemeral tmpfs pg + the test image. When
no Postgres is reachable (a plain `docker run`), the DB tests SKIP and the pure-logic
lane is unchanged: 172 passed, 2 skipped. With the DB: 174 passed.
test_db_lane.py smoke-proves commit round-trip + TRUNCATE isolation. The gate wiring
(siftlode.sh) lands in the ops repo.
/code-review high round 1 on S1b — 5 findings, 1 fixed, 4 by-design/deferred:
- FIXED: VideoProgressIn / ItemProgressIn position_seconds & duration_seconds
typed `int` would 422 a raw float currentTime on the 10s progress checkpoint
(the old `int(payload.get(...))` truncated it). Back to `float`, truncated in
the handler; +test. The live frontend floors so this was latent, not live.
By-design / deferred (no change): the 422 detail-array isn't localized by the FE
(filed under R7); PATCH is-not-none (channels/quota, null-safe) vs model_fields_set
(nullable-clear) is a deliberate split; missing-field→422 is the point of the epic;
me/switch_account's numeric-string coercion is more correct and still session-gated.
Gate: ruff clean, pytest 172 green.
Pure-logic tests that the three ROADMAP-named 500-producers (channel priority,
tag_id, setup password) and admin quota now reject bad input at the model
boundary, and that the PATCH models separate absent from explicit-null via
model_fields_set. Mutation-checked: reverting a field type turns them red.
All 11 download endpoints off payload:dict. Highlights:
- admin_set_quota: AdminSetQuotaIn — the `int(payload["max_bytes"])` 500 on a
non-number is now a 422; each field set only when provided (is not None).
- profiles/enqueue/edit: CreateProfileIn / UpdateProfileIn / EnqueueDownloadIn /
EnqueueEditIn; spec/edit_spec typed as dict (non-object → 422). _resolve_spec now
takes (profile_id, spec) instead of the raw payload.
- meta/share/links: UpdateDownloadMetaIn / ShareDownloadIn / CreateLinkIn /
UpdateLinkIn; _link_expiry takes an int|None instead of digging the payload.
PATCH endpoints read model_fields_set so an explicit null still clears a field.
Gate: ruff clean, pytest 164 green.
CreatePlaylistIn / RenamePlaylistIn (PATCH via model_fields_set) / ReorderItemsIn,
plus a shared VideoRefIn(video_id) for both add-video endpoints (watch-later +
add-item). A non-string video_id is now a 422 instead of reaching db.get.
Gate: ruff clean.
- setup: SetupAdminIn (KILLS the 3rd named 500 — a non-string password reaching
argon2/len()), SetupTestEmailIn; setup_config stays a typed dict[str, Any] (a
genuinely dynamic key→value settings map, not a fixed schema).
- admin: AddInviteIn / SetRoleIn / SetSuspendedIn / AddDemoWhitelistIn. The
role/status business-rule 400s are unchanged; type errors are now 422.
All 3 ROADMAP-named 500-producers now fixed (channels priority + tag_id, setup
password). Gate: ruff clean, pytest 164 green.
- saved_views: CreateViewIn / UpdateViewIn / ReorderViewsIn. PATCH presence via
model_fields_set; filters typed as dict (a non-object is now 422, an explicit
null on update stays a 400 "filters must be an object").
- feed: VideoStateIn(status) / VideoProgressIn(position_seconds,duration_seconds) —
the manual int()+400 in progress becomes a 422 on a non-number; the status
membership 400 and the video 404 are unchanged.
Gate: ruff clean.
update_channel: payload:dict → ChannelUpdateIn (priority/hidden/deep_requested,
all optional for PATCH semantics). Kills the `int(payload["priority"])` 500 the
prod report hit — a non-numeric/garbage value now returns 422, not 500.
attach_tag: payload:dict → AttachTagIn(tag_id:int) — a non-int tag_id was a 500
in `db.get(Tag, ...)`; the personal-tag ownership check reuses owned_or_404.
Round-1 review finding #4 named only audit.py/plex.py, but the same misplaced
`from app.utils import iso` sat in channels/downloads/feed/notifications too — the
round missed them (ruff enforces no import order here, so the gate was silent).
Move them to alphabetical position for consistency. No behaviour change.
/code-review high round 1 (5 findings):
- iso(dt) now typed `date | None` — MediaAsset.upload_date is a plain `date`, not a
datetime; the helper's signature was too narrow for the type-contract epic. +test.
- restore alphabetical app.* import order in audit.py and plex.py (the new iso/_common
imports had been dropped in mid-block).
The other 3 findings are deliberate design choices, left as-is for UAT: push/push-plan
now gate write-scope before the 404 (dependency ordering); the unified 403 message; and
owned_or_404's owner_attr kept for S1b's DownloadShare.to_user_id-style owners.
Gate: ruff clean, pytest 164 green.
Establish the R6 API-contract infra before the S1b payload:dict migration:
- iso(dt) in utils.py replaces the ~16 hand-rolled `x.isoformat() if x else None`
serializer idioms across the routes.
- get_or_404 / owned_or_404 in routes/_common.py — one 404 + existence-leak policy;
plex's _own_playlist_or_404 and saved_views' _own_view now delegate to it.
- require_write_scope dependency in auth.py collapses the inline has_write_scope→403
checks (channels subscribe/unsubscribe, playlists push/push-plan) that had drifted
into three different messages into one; the conditional delete_playlist guard stays
inline (its message was already canonical).
No response-shape change (iso == isoformat for a present value). Gate: ruff clean,
pytest 163 green (+ test_utils for iso).
A mirrored Shows library whose items have no genres stored `genres` as JSON
`null` — SQLAlchemy's JSONB persists a Python None as a JSON scalar, not SQL
NULL. /facets unnested it with jsonb_array_elements_text(), which raises
"cannot extract elements from a scalar", so a single bare library 500'd the
whole Plex module on open (the grid loads, the filter sidebar's facets call dies).
- Guard the /facets genre unnest with jsonb_typeof(genres)='array', mirroring
the guard people.py already applies to the same columns.
- Persist the JSONB tag columns (genres/directors/cast_names/collection_keys on
plex_items and plex_shows) with none_as_null=True, so an empty list round-trips
as SQL NULL and self-heals on the next sync — closing the trap for any future
unnest.
Verified against the real prod row (DumaTV show, genres = JSON null): the
unguarded query errors, the guarded one returns cleanly.
Found by running the round-8 fix in the real container instead of trusting the
unit test: a directory created 2ms AFTER `_PROCESS_START` was stamped with an
mtime 2ms BEFORE it, so the sweep deleted it as a leftover. The two sides come
from different clocks — a file's mtime from the kernel's coarse clock (one timer
tick of lag), `time.time()` from the fine one — and the unit test could not see it
because it supplies an artificial cutoff.
The window is small and lands exactly where it hurts: the sweep runs at boot, so
the session at risk is one started in the first instants after startup — the case
the guard exists for. Keep-side slack of 2s; the cost is that a leftover from the
last seconds before a restart survives until the next one.
Verified in the container: a backdated leftover is swept, a directory created by
this run is kept, and a registered live session with an old mtime is kept — 1 of 3.
Test pins the skew (mutation-checked: with the grace at 0 it goes red).
The worst one was, again, the previous round's own fix.
- Round 7 stopped awaiting the HLS sweep so startup could serve immediately — which
also made it run CONCURRENTLY with playback, and `sweep_orphans` snapshotted the
live-session set BEFORE listing the root and rmtree'd outside the lock. Since
`start_session` reuses a directory path for the same (key, offset, tag), a stale
name can become a live session at any moment. Two guards now: nothing whose mtime
is newer than this process's start is ever deleted, and the live check + rmtree
happen together under the lock, one directory at a time.
- The sweep task is awaited after cancel, so shutdown can't leave it pending.
- `_rate_limiter` used a truthiness test on its timestamp, so a `now()` of exactly
0.0 read as "never ran" — unreachable in production, but `now` exists to be
injected and a test clock starting at 0 is the obvious choice.
- The RSS total-outage error now carries the first underlying error: "all N channels
failed" is also what ONE dead feed id looks like on a small instance, and a red
card the user can't act on is its own kind of dishonest.
- The WebKit fullscreen handling moved into `lib/fullscreen` and PlexPlayer uses it
too — it had the same unprefixed-only reads in three places, including an Escape
branch that would have closed the whole player instead of leaving fullscreen.
- The `--max` CSS comment claimed `inset: 0` did the sizing; Tailwind's `w-full` is
still on the element and wins. Comment now states the real mechanism.
Verified in real Chrome (the pane can't composite or do native fullscreen):
- F fills the 1920x889 viewport; the exit pill renders top-right on the letterbox
band and overlaps NO YouTube control — the collision this round suspected does not
reproduce in this embed (its own buttons sit bottom-left / bottom-right).
- Four real ArrowDown presses while maximised: persisted volume 100 -> 80, with the
level flash visible on screen.
- A real click on the exit pill un-maximises and returns focus to the card.
- lib/fullscreen driven from a real click: enter (innerHeight 889 -> 1024), the
change listener fires exactly once per transition, exit returns to 889.
Backend suite 150 -> 158; the new sweep tests are mutation-checked, and they read
their cutoff from the filesystem rather than a clock (the container VM and the
bind-mounted host disagree by enough to make a clock-based assertion flaky).
peter