Compare commits

...
124 Commits
Author SHA1 Message Date
peter 3682e6a01f release: 0.76.0 (feature/e7-downloads-media-library)
gate / frontend (push) Failing after 1m36s
gate / backend (push) Successful in 2m26s
2026-08-29 21:54:19 +02:00
peter aca355f43b chore(release): 0.76.0 — E7 Downloads & Media Library
Bundles the whole E7 epic (library metadata, crop fix, share-by-default, in-app player,
browser-playable guarantee, local upload) plus the deep-review hardening. Bump VERSION and
add the 0.76.0 release-notes entry (RELEASE_NOTES[0] must match VERSION).
2026-08-29 21:52:21 +02:00
peter f41c81eb56 fix(downloads): media-library UI review fixes (E7)
Deep-review remediation on the frontend, all findings adversarially verified:

- LibraryPlayer picks audio vs video from the canonical media_kind (single source), not a
  hardcoded container whitelist, so an uploaded audio file with an uncommon/absent container
  renders in <audio> instead of a broken video box (legacy null media_kind keeps the heuristic).
- UploadDialog aborts the in-flight upload when the dialog is closed by X/ESC/backdrop/Back
  (not only the footer Cancel); drag highlight uses an enter/leave depth counter to stop the
  flicker; the progress bar gets a stable role=progressbar aria-label with the value on aria-valuenow.
- DownloadCenter's Shared tab renders honest loading/error states instead of a blank panel, and
  the library/shared row-action gating moves to a single tested helper (lib/downloadActions).

Tests: downloadActions gating matrix, the uploadReq XHR path (abort/401/422-localize/onerror),
and the media_kind-driven audio/video branch.

i18n: downloads.upload.uploadProgress (HU + EN).
2026-08-29 21:52:13 +02:00
peter 614c5528ad fix(downloads): harden upload serving + ingest against stored-XSS and leaks (E7 review)
Deep-review remediation for the E7 upload path, all findings adversarially verified:

- Serve every library file with a Content-Type chosen from the ffprobe-verified media_kind
  + a strict container allow-list (storage.served_media_type), never guessed from the
  user-controlled file extension. Closes a stored-XSS vector where a media/HTML polyglot
  uploaded as "evil.html" was served inline as text/html on the login-free /watch surface.
  Applied to both the authenticated /file and the public watch_file endpoints.
- Classify uploaded still images (jpg/png/gif/webp/…) as media_kind='file' so they are
  force-downloaded and don't get a broken in-app video player.
- Roll back the on-disk side of an upload when the ingest transaction fails after place_file
  (unlink the placed file + its deterministic poster path), so a failed commit can't orphan
  bytes in the GC-skipped .uploads tree.
- Floor download_upload_max_bytes at min=1 so a stored 0 can't silently disable the per-file cap.
- Dedupe the ffprobe-JSON invocation into edit.run_ffprobe_json (shared by probe_codecs and
  upload._probe) and the upload extension into storage._clean_upload_ext; drop the now-unused
  parse_probe_streams.

Tests: public + authenticated force-download and the polyglot-not-served-as-HTML guard, the
real _probe classifier (image/cover-art/dims/duration), the ingest rollback/cleanup path, the
max_jobs branch, the cap-floor invariant, and run_ffprobe_json error handling.
2026-08-29 21:52:02 +02:00
peter 104bd1e3a6 feat(downloads): public generic-file watch surface (E7 S5d)
The /watch/{token} SPA gets a third branch: media_kind==='file' renders an
info card (name / TYPE · size) + a Download button, no player. The <video>/
<audio> branches now guard on media_kind!=='file' so legacy links
(media_kind null) still play. Download is gated on the link's allow_download,
mirroring the media branch; the backend serves a file force-download
(attachment + octet-stream) regardless, so the gate only reflects owner ACL.

FE-only (public_meta already emits media_kind/size_bytes/container since S5a).
i18n HU/EN/DE 'file' fallback label. Two DOM tests: file -> info card +
Download, no player; view-only file -> no Download.
2026-08-29 19:51:58 +02:00
peter 584e9ced0c feat(downloads): local-upload UI — drag-drop dialog + progress (E7 S5c)
UploadDialog (drag-drop + native picker + optional display name) opened
from a new Upload button in the queue add-row. Uploads stream through a
new XHR helper (uploadReq in api/core) so the dialog shows real 0-100%
progress with a cancel — fetch/req can't report upload progress. The
upload lands as an already-done job; the S5b file-kind row gates it to
download+share. i18n HU/EN; upload cap/empty errors localize via the
shared structured-detail path.
2026-08-29 18:53:53 +02:00
peter b4ad59274b feat(downloads): gate library row actions by media_kind (E7 S5b)
Surface the S5a media_kind discriminator in the library UI. A generic
uploaded file (media_kind='file') is download+share only — no player and
no timeline to trim, so Play/Edit/Edit-details are media-only; it shows a
neutral file glyph instead of an (always-empty) thumbnail and no duration
chip (container/size still read). The "Shared with me" tab's file row is
gated the same way. Re-download is now hidden for edit clips and uploads
too (neither has a re-fetchable source — the backend already rejects both
with 409), removing a latent click-to-409 on edit rows.

FE-only: the serializer already emits media_kind (S5a). Adds media_kind to
the DownloadAsset type and "upload" to the job_kind union; DOM tests cover
the file-row presentation (glyph, no img, no duration).
2026-08-29 18:38:21 +02:00
peter fe0b34cd00 feat(downloads): local file upload ingest — media library backend (E7 S5a)
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.

- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
  gates library actions. Set at every finalize (download probe, editor carries the
  source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
  in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
  the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
  places it in a per-user .uploads tree, and creates an already-ready asset + done job
  (the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
  (download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
  charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
  application/octet-stream) on both the in-app and public endpoints, so uploaded
  HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
2026-08-29 08:02:11 +02:00
peter 9eaa01617c feat(downloads): in-app player for library items (E7 S4b)
Play a completed library item in-app: a Play button on Library rows (and
Shared rows with download access) opens a modal that streams the local file
into a native <video controls> (or <audio> for audio-only assets).

Rather than the YouTube IFrame PlayerModal or the Plex HLS PlexPlayer — both
bound to their own domain APIs and unable to point at a local file — this
reuses the shared Modal a11y shell (focus-trap, ESC, layer-stack, portal) and
the range-aware /api/downloads/{id}/file endpoint. S4a's universal-codec
guarantee (mp4/H.264/AAC + m4a/mp3) makes native playback always work, so seek,
volume, fullscreen and speed come for free with zero transcode.
2026-08-29 07:18:58 +02:00
peter fdca12968b feat(downloads): one-click Share + downloadable-by-default links (E7 S3)
D2-A: the library row's Share button becomes a single action — quickShareLink()
ensures a public link (minting on demand), copies it to the clipboard, and toasts
with an "Open in new tab" action. A copy failure still toasts (warning) with that
action so the link is never a dead end. Advanced controls (password/expiry/revoke/
user-ACL) stay in the existing ShareDialog, reached via a new "Manage sharing" row
button.

D1 (forward-only): CreateLinkIn.allow_download defaults to true, so a newly minted
link is downloadable by default. The default lives in the API, not a migration —
the download_links column default stays false, so no existing link flips. The
ShareDialog create-default is flipped to match.

Tests: TestCreateLinkDefault (default true; explicit false honored); 4 vitest cases
for quickShareLink (mint-vs-reuse, copy ok/fail, create failure). HU/EN parity.
2026-08-29 06:36:07 +02:00
peter 94fcc40bde feat(downloads): guarantee every download is browser-playable (mp4/H.264/AAC)
The Download module's purpose is download -> share -> play in-browser, so the
output must play in EVERY browser (iOS/Safari/WebKit included). The only
universally-safe combo is mp4 + H.264 + AAC/MP3.

- formats.normalize() now CLAMPS every spec to that profile (container->mp4,
  vcodec->None, audio_format opus->m4a) at the single point all paths flow
  through (picker, stored profile, raw API) — not just by hiding UI options. A
  source exposing no H.264 is still caught by the worker's browser-safe
  re-encode, so playability is guaranteed end-to-end.
- build_ydl_opts drops the now-unreachable non-compat branches (+ _VCODEC_SORT).
- _SIG_VERSION 3->4 so assets cached under an old non-compat spec — or a stale
  non-H.264 mp4 that slipped the v3 net (the HEVC Facebook dls) — re-derive.
- ProfileEditor drops the container/codec pickers and Opus; audio is M4A/MP3.
- i18n: remove the now-unused container/vcodec/any keys (HU+EN parity).
2026-08-29 06:08:21 +02:00
peter fb2c843fa9 fix(downloads): anchor crop overlay to the letterboxed picture (E7 S2)
The video preview is object-fit: contain (the browser default for <video>), so a
non-16:9 source is letter/pillarboxed inside the element box. The crop marquee was
drawn in fractions of the whole box (bars included) and those fractions were then
multiplied by the source resolution, so the emitted ffmpeg crop diverged from what
the selector showed. Anchor the overlay to the computed content rect (and set
object-contain explicitly) so a marquee fraction is a fraction of the real picture;
extract the display-px<->source-px math into cropGeometry.ts with unit tests, and
clamp the even-rounded window in-bounds so a full-frame crop can't overflow.
2026-08-29 05:43:41 +02:00
peter bb6cb6e335 style(downloads): bump library-row meta text + action buttons a step (E7 S1)
Enlarge every non-title text on the card (channel, status/format/size/date meta,
source-ref line, progress readout) one step (xs→sm) plus their inline icons, and
grow the row action buttons (padding + icon) via the actions wrapper so the shared
IconBtn stays smaller in the modals. Buttons keep the title's line — the wrapper is
shrink-0/no-wrap, so the text column shrinks instead.
2026-08-29 05:29:19 +02:00
peter bdbff89507 feat(downloads): show container/format chip in library rows (E7 S1)
The completed download rows already surfaced size + duration; add the concrete
container (MP4/M4A/WebM…) as a compact chip so the library reads as a media list
with a visible format — the audio-vs-video distinction the running-only phase pill
can't show at rest. Pure display of the already-serialized asset.container; locked
by a small DOM test (present when ready, absent before the worker fills it).
2026-08-29 05:21:54 +02:00
peter 9b87838e3a release: 0.75.0 (improvement/quickfix-batch-2)
gate / frontend (push) Failing after 1m34s
gate / backend (push) Successful in 2m16s
2026-08-29 03:55:15 +02:00
peter f014416e31 chore(release): 0.75.0 — quickfix batch #2 (Plex/downloads/playlists) 2026-08-29 03:54:08 +02:00
peter 3369587412 fix(review): resolve deep-review findings across quickfix batch #2
- Button: `loading` always disables (|| not ??) — stops a double-submit
  when a call-site passes an explicit boolean `disabled` (VideoEditor).
- playlists: track un-ingestable-but-live YouTube ids in the sync baseline
  so a linked playlist containing a private/deleted video converges instead
  of re-classifying it as a pending pull forever; re-attempt ingestion each
  sync so a restored (private->public) video heals, and prune ids that
  leave YouTube.
- SettingsPanel: download-auth section renders an honest error/loading
  state (StateMessage + retry) instead of a bare empty section on a failed
  credentials fetch; HU+EN loadError string.
- tests: reorder-order, partial-failure baseline hold, un-ingestable
  converge/heal/prune, Button loading-disables, download-auth error/retry.
2026-08-29 03:52:13 +02:00
peter 9af4e8f4ff feat(playlists): sentence-style two-way sync preview (only changed sides)
The confirm dialog spelled out both sides even when one was "+0 / −0", which read
as engineer shorthand. Build the message from one sentence per side that actually
changes and add a short "stays unchanged" line for the untouched side; the zero
side is no longer shown as a bare +0/−0 to decode. Assembled in the component
from granular i18n clauses (add/remove per side, reorder, rename) with count
plurals; HU+EN. Replaces the single pushPlanMerge template.
2026-08-28 05:20:34 +02:00
peter a0b725d779 feat(playlists): two-way (3-way) merge YouTube sync + mandatory preview (S7)
"Sync to YouTube" was a one-directional destructive push (local wins, deletes
from YouTube every item not in the local list), so items added on YouTube were
lost and a dirty pull was skipped entirely. Replace it with a real 3-way merge:
each item is classified against the last-synced baseline as a local-add, a
YouTube-add, or a genuine delete, so an add or delete on either side survives.

- Persist the ordered baseline id set (new playlists.synced_video_ids, migration
  0075); the fingerprint hash alone couldn't drive a merge. NULL baseline (fresh
  export / pre-S7 link) unions losslessly on the first sync and records one.
- Pure merge_plan(baseline, local, youtube) -> add/remove per side + merged order
  (local order, YouTube-only additions appended). merge_playlist applies the
  local side directly and the YouTube side through the failure breaker; the
  baseline only advances on full success so a partial failure re-converges.
- plan_merge/push-plan now report what each side gains and loses; the confirm
  dialog shows the two-way plan before applying (pushPlanMerge, HU+EN).
- Tests: 10 new (pure classification incl. delete-vs-stale-copy; DB apply proving
  a YouTube-add and a Siftlode-add both survive on both sides, a genuine delete
  propagates without collateral loss, no-baseline never deletes, create path).
2026-08-28 04:20:29 +02:00
peter bc33b2ed16 feat(downloads): per-user, per-host yt-dlp cookie/auth support (S6)
Each user stores their own login cookies (Netscape cookies.txt) for an
auth-walled download host (Facebook, Instagram, X, TikTok, Reddit) in
Settings; a download of an auth-walled link on that host is retried with
the acting user's cookie.

Two load-bearing security controls, both in downloads.credentials as the
single source of truth so enqueue and the worker can't drift:
- cache privacy: a cookie-authed download salts its MediaAsset format_sig
  with the user id, so the authed asset is private and never cache-served
  to another user. The worker attaches a cookie ONLY when the asset it is
  downloading into IS that private, salted identity.
- host scope (SEC-12): a cookie is resolved only for a URL whose netloc
  the credential owns (netloc-scoped _host_is); a look-alike/unrelated
  host gets none.

cookies_enc is Fernet-encrypted at rest, only decrypted in-worker into a
0600 file under staging/.auth/ (survives the retry wipe + _produced_file
scan) and unlinked in finally. The blob is never returned by a route,
logged, audited, or written to a job's error/snapshot. cookiefile is set
after the adapter's extra merge so an adapter can't clobber it. Settings
UI is write-only (host + set/updated only, never echoed). New
download_credentials table + migration 0074; per-user routes; HU+EN i18n.

Tests: host-scope, cache-privacy (no cross-user serve), encrypt-at-rest,
write-only API, worker attach-gate; FE dom test for the write-only field.
2026-08-28 03:53:04 +02:00
peter 5280949667 feat(ui): migrate the canonical action-button surface onto <Button>
Sweep the rounded-lg solid/ghost/outline action-button family onto the shared primitive:
ConfirmProvider, ErrorDialog, DownloadDialog, DownloadCenter, downloadModals, downloadAdmin,
ProfileEditor, PersonalViewsModal, SettingsPanel, WatchPage, DraftSaveBar. Deliberately left
on their own styles per the agreed scope: login/onboarding rounded-xl CTAs (Welcome/Onboarding/
Setup/KeyGate), glass Plex surfaces, segmented pills/toggles/switches/chips, dropdown triggers,
data-driven Banner/tone maps, text-xs dense buttons, and tiny inline p-1/p-1.5 icon controls —
none map 1:1 to a variant+size without a visual change, and the repo has no tailwind-merge.
2026-08-28 03:12:29 +02:00
peter 4892c38466 feat(ui): shared <Button> primitive + migrate ShareDialog/VideoEditor/ChannelPage proofs
Introduce frontend/src/components/ui/Button.tsx — the one action-button primitive
(variants primary|danger|outline|ghost|glass|danger-outline; sizes xs|sm|md|lg|icon;
icon/iconOnly/label/loading/fullWidth/forwardRef). No focus-visible (already centralized
in index.css @layer base). Migrate the cleanest proof sites off btnCls, and flip the
channel Subscribe button to the outline variant so it matches Full history (the user's ask).
2026-08-28 02:59:21 +02:00
peter 9ae6c47d6c perf(downloads): de-prioritize worker transcodes + idle-back-off polls to stop LAN slowdown
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API,
Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and
starved web requests + playback until it finished. Full-scope S4 mitigations:

- Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp
  descendant): CPU nice increment + best-effort idle ionice class + an
  OMP_NUM_THREADS budget, all config-gated and independently disable-able.
- Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode
  reserves a core for foreground work instead of claiming all of them.
- Throttle the retention GC's whole-tree orphan sweep (yield IO after each
  batch of files) so its rglob can't monopolize disk IO in the API process.
- Idle-back-off the Download Center (2s→15s when nothing queued/running) and
  UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the
  next fetch after an enqueue (invalidateDownloads refetches immediately).

Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence,
_deprioritize (nice/ionice/OMP branches), GC sweep throttle batching.
2026-08-28 02:37:56 +02:00
peter b7a6451dba fix(downloads): iPad audio playback + honest non-retryable failure for auth-walled hosts
5a — the public watch page rendered every asset in a <video>, but WebKit (all
iOS browsers) refuses to play an audio-only mp3 in <video>, so shared audio
links were broken on iPad while working on desktop Chromium. public_meta now
flags audio-only assets (codec present, no video stream) and WatchPage renders
a native <audio> in a padded surface card (no artwork exists for these).

5c — a login-walled Facebook post failed as a bare 'unknown', which offered a
misleading Retry. Classify closed-group / cookie-suggestion auth walls as
login_required and genuinely unsupported sites as a new unsupported_host class
(both non-retryable), and drop unknown from the modal's retryable set — an
unclassified failure is most often an unparsed auth wall. The queue row keeps
its own force-retry for real transients.
2026-08-28 02:22:28 +02:00
peter 11e32163b8 fix(ui): SortControl direction toggle no longer wraps on narrow popovers
The asc/desc buttons in the sort popover footer share one component across
Feed and Plex browse, but rendered differently: the Plex popover is narrower
(short option labels), so "Descending" wrapped icon-over-text and made the
active button taller. Add whitespace-nowrap to the direction buttons and bump
the popover min-width (min-w-52 -> min-w-56) so both call-sites render an
identical single-line toggle.
2026-08-28 02:06:06 +02:00
peter 257dc5741d fix(plex): resolve movies stuck "in progress" on the 15-min incremental sync
The incremental watch-sync (run_plex_watch_sync) learned a movie's watched
transition only from the history feed + onDeck. When a movie is finished on
Plex (commonly by scrubbing through the end credits) Plex sets viewCount>0,
clears viewOffset and DROPS it from Continue-Watching, so onDeck no longer
lists it (and onDeck can only ever ADD an in-progress row, never resolve one),
while a scrub-to-end frequently emits no history session row. The stale local
status='new', position>0 row then survived until the DAILY full reconcile —
which is why movies kept showing "in progress".

Add _reconcile_inprogress_movies: for each local in-progress MOVIE that Plex's
onDeck no longer lists, read its current per-account viewCount/viewOffset with
one metadata() call and apply it via the shared _pull_apply, flipping to
watched when Plex says so. Bounded by a probe cap (100), most-recently-touched
first; episodes are left to onDeck/history. No-signal rows are preserved (an
un-watch stays the daily reconcile's job).

Tests: watched-flip, onDeck+episode skip, and no-signal-preserved.
2026-08-28 01:34:02 +02:00
peter f52bc5898b release: 0.74.0 (improvement/quickfix-batch)
gate / frontend (push) Failing after 1m28s
gate / backend (push) Successful in 2m15s
2026-08-28 00:05:21 +02:00
peter d0033617e2 chore(release): 0.74.0 — quickfix batch (Plex sync/browse + Channel Manager) 2026-08-28 00:03:13 +02:00
peter f80ed115ad fix(plex,channels): deep-review fixes — durable watch-sync + no shared-link clobber
Deep-review of the quickfix batch surfaced two data/preference-loss bugs plus
smaller items; all fixed with tests.

- watch_sync _pull_apply: don't let the Plex-authoritative pull overwrite a DIRTY
  local row (a change whose immediate push failed) — that stranded the value
  forever since _repush_dirty only retries dirty rows. Leave it dirty for re-push.
- PlexProvider / ChannelsProvider: a partial shared link opened in a fresh tab
  persisted mount-time DEFAULTS over the account's stored scope/sort/filters
  (base read before the account was pinned). Persist/keep only URL-present dims;
  re-hydrate the omitted ones from storage.
- playback resume write-back: guard against out-of-order delivery (a superseded
  older push now drops) and evict the throttle map so it can't grow unbounded.
- PlexPlayer: announce mid-playback buffering to assistive tech (role=status +
  sr-only label); new plex.player.buffering (HU+EN).
- columnSort: hoist the shared key:dir sort codec used by both URL-state modules.
- Tests: dirty-guard + flag-settle + per-key throttle scoping + superseded-push
  (backend); hold-repeat/suppress-click/unmount + partial-link no-clobber +
  refetch-doesn't-clobber-undo (frontend). Backend pytest 475, FE vitest 519.
2026-08-27 23:54:44 +02:00
peter 7fbb996297 feat(channels): shareable manager view in the URL — tab+sort+column-filters (S8)
The Channels-manager content state now round-trips through the address bar,
so a manager view is bookmarkable/shareable — the S4 pattern applied to the
last content page the audit flagged.

- New lib/channelsUrlState.ts (channelsParamUpdates / channelsParamsToState /
  hasChannelsParams) under NAMESPACED keys ctab/csort/cf — channels-unique
  like ?v=/?pl/pscope, so no other provider reads or clears them and the
  feed's URL-stripping can't delete them. The mixed scalar/array FilterMap
  (sync/priority scalars + tags multi) rides in `cf` as a JSON blob so both
  shapes round-trip exactly. Only non-default values emit.
- ChannelsProvider mirrors {view, sort, filters} via useUrlMirror, page-gated
  on channels (off-page the keys clear; localStorage still holds the state and
  it re-mirrors on return). A shared link wins over stored state: readChannels
  UrlSeed captured once, applied before paint through the slice setters (so the
  account-pin re-hydrate sees them dirty and won't reload storage over the
  shared view), then persisted to the account once pinned so it survives a
  later plain reload — mirrors PlexProvider's seed/rehydrate. The name search
  stays ephemeral (out of the shared contract).
- NavigationProvider's page-switch nulls ctab/csort/cf alongside the plex keys
  so switching modules resets to the manager root.

Tests: channelsUrlState.dom.test.ts (9 round-trip) + ChannelsProvider.dom
seed/mirror (2); the existing tab-history test's beforeEach now resets the
query string too (the mirror writes it). FE vitest 503→514. FE-only.
2026-08-27 04:37:53 +02:00
peter 9cd4ae09d4 feat(views): editable saved-view detail — removable filter chips + undo + save-current (S7)
The Personal-views manager's detail pane becomes editable instead of
read-only:

- Captured filters render as REMOVABLE chips. Removing one prunes the set
  and writes it straight back to the stored view (PATCH filters), via the
  same live-row builders (useActiveFilters / buildPlexActiveFilters) that
  already compute the pruned snapshot — so the modal and the live chip row
  can't drift. The prune is wrapped in useUndoable + the shared UndoToolbar,
  so undo/redo re-persist the same way. The undoable draft is the single
  source of truth for the chip row, so a chip-prune refetch never clobbers
  the history.
- A "Save current" button beside Apply stores the module's currently-applied
  filters INTO the view (the live→stored inverse of Apply). Disabled when the
  view already equals the live filters (it's the active one). On save the
  draft re-seeds to the saved snapshot and clears the stale undo history.
- Rename gains a visible commit affordance: a check button beside the name
  input that enables once the name changes (Enter and blur still commit).

Provider gains two methods mirroring `rename`: updateFilters(id, filters)
and saveCurrent(id). One shared provider serves feed + plex, so both get it.
PATCH filters was already supported end-to-end — FE-only.

New i18n: views.saveCurrent / saveCurrentHint / saveCurrentUpToDate (HU+EN).
Tests: 5 new in PersonalViewsModal.dom.test.tsx (chip-remove persists, undo
re-persists, save-current fires + disables when up-to-date, rename commit).
2026-08-27 04:22:07 +02:00
peter 00527d0cf3 feat(channels): leading '+ Tag' pill + detail-page Unhide affordance (S6)
Item 6 — the tag add-trigger now LEADS the chip row (a stable left anchor)
instead of trailing it, so on the detail page (bounded=false) adding chips
grows the row to its right and the '+'/its editor popover no longer drift.
Compact icon in the width-bounded manager cell; a labelled '+ Tag' pill
where there's free width (the detail header).

Item 7 — surface the per-subscription 'hidden' flag on the detail GET
(channel_detail was omitting it) and add an Eye/EyeOff hide/unhide toggle
next to Block on the channel page, mirroring the manager. FE ChannelDetail
gains 'hidden'; new channel.* i18n keys (HU+EN).

Tests: +3 backend (channel_detail surfaces hidden, both states + unsubscribed
default). Gate green: backend pytest 471, FE vitest 498.
2026-08-27 04:02:25 +02:00
peter 8dc4976c2c feat(channels): horizontal −/+ priority stepper with pop-free inline edit (S5)
Redesign the shared PriorityStepper (channel manager cell + detail header)
from a value-plus-stacked-arrows control into a horizontal [−] value [+]
segmented stepper. The value now sits centred between the buttons and opens
an inline editor in a FIXED-width slot, so click-to-type no longer "pops"
the control wider (the old absolutely-positioned overlay grew w-6 → w-14).

One `size` prop drives the two surfaces: `sm` draws its own bordered pill
for the standalone table cell; `md` drops the outer border and keeps only
the segment dividers, blending into the detail header's lozenge instead of
double-bordering it. Hold-to-repeat, negatives, and the debounced-flush
persistence hook are unchanged.

The manager priority column widens 3.5rem → 5.5rem to fit the horizontal
control under fixed table layout.

Tests: 7 new DOM tests for the stepper interaction contract.
2026-08-27 03:29:42 +02:00
peter 56a2ecdec3 feat(plex): shareable Plex-browse state in the URL — scope+filters+sort+show (S4)
Mirror the Plex-browse content state into the address bar so a browse view is
bookmarkable/shareable, the Plex analogue of the feed's urlState. Keys are
NAMESPACED (pscope/psort/pshow/pf) rather than reusing the feed's bare
scope/sort/show: FeedFiltersProvider actively strips its own KEYS from the URL
whenever off the feed page, so a shared key would be deleted out from under the
Plex page. The namespaced keys are Plex-unique (like ?v=/?pl), so no other
provider ever reads or clears them.

- New lib/plexUrlState.ts: plexParamUpdates / plexParamsToState / hasPlexParams.
  The pf filter blob reuses the SAME appendPlexFilters wire serializer (new
  public plexFiltersToParams + inverse parsePlexFilters, kept next to each other
  so a new wire key can't be added on one side only).
- PlexProvider: URL-seed wins over localStorage at mount; the account-pin
  rehydrate persists the seed instead of clobbering it (mirrors feed's
  initialHadUrlFilters); useUrlMirror mirrors/clears the 4 keys, gated on
  page===plex. q/field stay out of the shareable contract.
- NavigationProvider: a page switch now also nulls the 4 plex keys.

Tests: 8 new round-trip cases in plexUrlState.dom.test.ts (FE vitest 483→491).
2026-08-27 03:14:54 +02:00
peter 8299ea72b7 feat(plex): 'last watched' browse sort + auto-apply in the in-progress view (S3)
Add a 'lastwatched' sort key to the unified Plex browse. Movies rank by
coalesce(progress_updated_at, watched_at); shows by max(coalesce(...)) across
their episodes (rollup.resumed_at, now broadened from progress-only so watched
episodes count too). Never-watched titles sink via nullslast.

Frontend auto-applies lastwatched:desc when entering the in-progress watch-state
view and restores the pre-entry sort on exit; a manual sort change while there
still wins. i18n HU+EN. Backend tests cover both union legs + mixed nullslast.
2026-08-27 02:50:41 +02:00
peter 6c918d1dff docs(plex): refresh watch-sync job docstrings for Plex-authoritative pull (S2)
The incremental/reconcile job docstrings still described last-write-wins and a
30-min cadence; update them to match the S2 behavior (Plex wins on divergence,
15-min incremental) so the "why" doesn't drift from the code.
2026-08-27 02:33:42 +02:00
peter 3f8257059d feat(plex): Plex-authoritative watch-sync — resume write-back + faster pull (S2)
Make Plex the source of truth for watch state:

- Write-back during playback: item_progress now pushes the resume position to
  Plex on the periodic checkpoint too (not only the final flush), throttled to
  one push per item every ~10s (in-memory, single-process rate limiter). The FE
  now also fires an immediate (debounced) push on seek.
- Faster incremental pull: 30 → 15 min (plex_watch_sync_interval_min).
- Plex-authoritative pull: _pull_apply resolves a genuine divergence to Plex's
  state instead of last-write-wins; the _same_state + synced_to_plex short-circuit
  still prevents a freshly-pushed row from bouncing. Removed the now-unused
  _LWW_TOLERANCE_S / _sift_ts.

Tests: 5 new in test_plex_watch_sync.py (Plex-wins resume/watched, no-bounce,
throttle final vs periodic).
2026-08-27 02:07:45 +02:00
peter 12d9561ba1 fix(plex): warm up HLS buffer on session start + player loading/buffering spinner
The Plex player stalled ~2s a few seconds into every auto-advanced next
episode. Root cause: each auto-advance spins up a cold ffmpeg HLS remux
session, and for a fresh start (start_s==0) start_session returned
immediately with zero buffer — hls.js drained the first segment, then
rebuffered waiting for the still-warming ffmpeg to write the next one.
Once ffmpeg gets ahead (stream-copy is faster than realtime) it never
stalls again.

- stream.py: before reporting a session ready, wait for the first
  _WARMUP_SEGMENTS (3) video segments to exist — the seg0 wait is now
  unconditional (was start_s>0 only) and warm-up runs for all starts.
  Best-effort and bounded: stop at the first segment not ready in time.
- PlexPlayer.tsx: a loading/buffering spinner. The existing overlay only
  covered the INITIAL load (!ready); add a `buffering` state driven by the
  video's waiting/stalled/seeking (set) and playing/canplay/seeked/pause
  (clear) events so a mid-playback stall or in-flight seek reads as
  "loading" instead of a frozen frame, and surface a fatal loadError even
  once playback has started. Spinner uses the shared Loader2.

Merges roadmap E6·item1 (spinner while loading/seeking) into this fix,
since the server-side warm-up deliberately holds ready a beat longer.

Tests: 3 new in test_plex_stream.py (warm-up waits for the first few
segments; stops at the first missing one; no warm-up when seg0 never
arrives). Gate: backend pytest 460 (Docker lane), FE tsc/lint/knip/
prettier + vitest 483.
2026-08-27 00:50:58 +02:00
peter fe09879edd release: 0.73.0 (feature/e8-feed-filtering)
gate / frontend (push) Failing after 1m22s
gate / backend (push) Successful in 2m4s
2026-08-26 05:28:11 +02:00
peter 7104767ff9 fix(e8): deep-review hardening + cut 0.73.0
Fixes from several multi-agent deep-review passes over the E8 feed-filtering &
channel-manager epic (S1-S4):
- feed_years buckets by UTC year to match the UTC-pinned year-filter bounds
- tag suggestions: examples ∩ subscribed (drop tagged-then-unsubscribed channels
  from the profile denominator); exclude hidden channels as suggestion targets
- channel manager: restore ghost chips + surface an error toast on a failed tag accept
- year filter: debounce the search term so the per-year facet doesn't refetch per keystroke
- channel tags: aria-label the "+N" overflow pills
- extract useTagSuggestions hook; add backend + hook tests (profile boundaries, cap,
  language exclusion, unsubscribed/hidden handling, UTC bucketing, bulk-accept)

Bump VERSION to 0.73.0 + release notes.
2026-08-26 05:25:47 +02:00
peter 17f8885fb1 fix(channels): collapse overflowing tag suggestions into a "+N" pill
With 2+ suggestions the narrow tags cell clipped the second ghost chip
mid-word right next to the edit "+", reading as the icon covering it. Route
the ghost chips through the same fit machinery the attached chips use: a
dedicated SuggestedChipsLane fits what the cell holds and collapses the rest
into a "+N" pill whose hover popover still one-tap accepts / dismisses each.

The lane is attached-XOR-ghost (suggestions are untagged-only, and accepting
one now drops the whole channel from the suggestion cache), so each lane owns
the full cell width instead of clipping the other.
2026-08-26 02:39:00 +02:00
peter 549aae67cc feat(channels): suggest your own tags for untagged channels from your tagging pattern (E8·S4)
Learn, per user tag, which auto TOPIC tags are characteristic of the channels
you've tagged it, then suggest that tag for still-untagged subscribed channels
whose topics cover enough of the profile. Language auto-tags are excluded from
the signal — near-universal within a language, they flooded every channel.

New GET /api/tags/suggestions (read-only, deterministic) backed by
app/sync/tagsuggest.py. Surfaced in the Channel Manager as dashed "ghost" chips
in the tags cell — one tap accepts (reuses the attach-tag endpoint), × dismisses
for the session — plus a header badge and a bulk "Accept N confident" (the
full-profile matches) behind a confirm. No silent writes (D4 = suggest-and-confirm).

Also reflow a pre-existing prettier drift in dateFilter.ts from E8·S2.
2026-08-26 02:18:34 +02:00
peter fbb961b5b7 feat(feed): hover infobox on feed cards — full title, description, tags (E8·S3) 2026-08-26 00:44:27 +02:00
peter edcea87ea5 feat(feed): filter by year — per-year count facet + year control (E8·S2)
Add a Year filter to the feed rail: quick chips for recent years plus a
'More…' popover listing all years with per-year video counts, sortable by
year or count. Selecting a year narrows the feed to that whole calendar
year (a published_after/before range applied in the api layer) and is
mutually exclusive with the Upload-date control.

Backend: new /api/feed/years endpoint + feed_years() — one grouped query
that honours every filter EXCEPT the publish-date narrowing (dropped), so
the year list stays a live drill-down and switching years never zeroes the
others. FE keys the list on the date-stripped filters, so picking a year
re-queries the feed but not the list.

Wires year through FeedFilters, the filterParams/URL (?year) round-trip,
the active-filter chip row, and the panel layout. i18n HU+EN.

Tests: feed_years count query (date narrowing ignored, other filters
honoured); dateFilter year-clearing; urlState year round-trip.
2026-08-25 02:55:12 +02:00
peter 2c843e549b fix(channels): tag-filter row folds overflow into "+N", keeps "…" pinned
The channel manager's tag-filter row wrapped the trailing "…" catalog-actions
menu onto a second line once the chips + Clear outgrew the width. Rework it as
one non-wrapping row: the TAGS editor trigger (left), a single-line chip lane
that collapses overflow into a "+N" pill whose popover holds the rest (still
toggleable), then Clear and the "…" menu pinned right — so "…" always stays on
the first line.

New TagFilterChips reuses the shared measure core (useChipOverflowFit +
fitChipsWithPill), the same compression the feed's applied-filter row uses.
2026-08-25 02:30:28 +02:00
peter 4ee0dab68f feat(feed,channels): "No tag" filter for untagged channels (E8·S1)
Filter to channels you haven't personally tagged in the channel manager
(no user-owned ChannelTag; auto language/topic tags don't count):

- Feed: a "No tag" sentinel in the "Your tags" sidebar combobox (exclusive
  with real tag picks) + a removable "Untagged" chip; new `untagged` bool
  feed param / `notag` URL key, persisted + coerced on every filter path.
- Channel manager: a virtual "No tag" chip in the tag-chip row filters the
  table to untagged channels (client-side, shared NO_TAG_SENTINEL).

Backend `untagged` is a plain channel-level narrowing, so feed_summary keeps
it applied in the base pass — the count/facet lockstep holds with no mirror.

Note: item E (per-tag counts) and item D (Any/All toggle) were already
shipped; S1 delivers only item B.
2026-08-25 01:44:34 +02:00
peter 5d7be687d3 test(views,layer): close the two polish-tail deep-review test gaps
UX-8 layerStack: add an integration test wired through the real useLayer
hook (popover document-listener + stamp) and a PlayerModal-shaped window
listener, driven by a real dispatched keydown. The popover close is wrapped
in flushSync so its layer is popped synchronously between the two listeners
-- the production mid-dispatch race -- making the escapeHandled stamp the
only thing that keeps the player from also closing. Mutation-checked:
removing the guard fails exactly this case.

SavedViews: cover feedCanSeedDefault's "yield to a bookmarked filter URL"
branch (a filter-bearing search string -> no seed) and the per-source
seededRef guard against the 60s refetch (a changed-list refetch re-runs the
seed effect; the guard blocks re-seed). The refetch test waits for the
changed list to reach the component so the effect has its chance to wrongly
re-fire. Both mutation-checked.

Frontend gate green: 464 vitest (+4). Backend untouched.
2026-08-24 23:41:49 +02:00
peter e2a8503496 release: 0.72.1 (improvement/polish-tail)
gate / frontend (push) Failing after 1m20s
gate / backend (push) Successful in 2m0s
2026-08-24 23:14:42 +02:00
peter a389306885 chore(release): 0.72.1 — polish tail (a11y, saved-view default, show-collection Play-all) 2026-08-24 23:14:07 +02:00
peter 32c05897f4 test(plex): cover Play-all leaf/show-expansion + honest empty notice
The S4 Play-all wrapper had no test — a regression inverting the empty-queue
guard or dropping the notice would re-introduce the silent no-op it was added
to fix, and stay green. Adds a PlexCollectionView DOM test (spied api +
notify) asserting: a leaf queues as itself, a show expands to its episodes in
season->episode order, and an empty expansion surfaces plex.collection.playEmpty
instead of no-op'ing.
2026-08-24 22:38:35 +02:00
peter 2e319be569 refactor(plex): share showEpisodeKeys + honest empty Play-all
Extract the show->ordered-episode-keys flatten into one pure, unit-tested
helper (lib/plexQueue.ts) and use it at all three sites — the collection
Play-all (S4) and the show/season play-through pages in plexViews — so the
season-then-episode queue-order invariant lives in one place instead of three
hand-rolled copies. Behaviour-identical (the helper's ?? [] is a safe superset
of the collection site's previously non-null read).

Also: a show-collection whose shows expand to zero episodes had an enabled
Play-all that silently no-op'd — the same 'does nothing' state S4 removed for
the empty-queue case. Surface an honest info notice (plex.collection.playEmpty,
HU+EN) instead.
2026-08-24 22:09:52 +02:00
peter 852b227ea2 fix(views): seed the default view only after the tab-account pins
The S2 seed effect ran on the saved-views query's first resolve, but that
query is ungated and — since SavedViewsProvider mounts above <App>, which pins
the tab-account only after /api/me — resolves before the account is known on a
fresh tab. It then evaluated canSeedDefault() against a null account (which
reports 'stored, do not seed') and burned its one-shot guard, so the starred
default was never applied until a reload: exactly the fresh-tab miss S2 set out
to fix. Gate the effect on the reactive account id so it defers instead of
consuming the guard, and re-fires when the account pins. Covers feed + plex.

Adds a dom test for the defer-then-seed-on-pin path the old test masked by
pre-pinning the account.
2026-08-24 22:09:39 +02:00
peter 06b85586e6 feat(plex): expand shows to episodes so a show-collection's Play-all plays (S4)
A collection's Play-all/Shuffle queue held only playable leaves
(movies/episodes), filtering out shows — so on a show-collection the queue
was empty and Play-all was hidden/dead. Build the queue by expanding each
show member into its episodes (season then episode order), concatenated in
member order; leaves still queue as themselves. Shows are fetched lazily on
click (parallel plexShow fetches), not on page load. Gate Play-all/Shuffle
and the poster on "has any playable member" (leaves OR shows), and disable
them while the queue builds. Scope: option A — no member manual-order
migration. Adds plex.collection.playFailed (HU/EN).
2026-08-23 23:30:21 +02:00
peter fc32188fd2 refactor(chips): share the chip-overflow measure core between the two lanes (S3)
ActiveFilterChips (feed applied-filter row) and ChannelTagsControl both
hand-rolled the same single-line chip-fit measurer: a hidden mirror + a
layout-effect + ResizeObserver reading chip right edges, the real rem gap,
the pill width and the lane width. The policy half was already extracted
(decideChipLayout); the DOM-measurement half was still duplicated.

Promote that half to useChipOverflowFit(rowRef, chipsRef, pillRef, deps,
decide, initial): it owns the effect + observer + initial measure and reads
the common geometry, then hands it to each caller's own policy. The feed
row layers its leading label + trailing "Clear all" reservation on top and
keeps decideChipLayout; the tag cell's chips-only policy is extracted as
fitChipsWithPill (pure, unit-tested) alongside it. Add a jsdom test for the
hook (stubbed offsets, re-measure on deps). Behaviour unchanged — verified
in the browser: feed all/count modes + popover, tag "+N" fit + hover reveal.
2026-08-23 23:22:52 +02:00
peter 9538fb1b7a fix(views): apply the starred default view on first resolve, for feed AND plex (S2)
The feed applied its starred default via a localStorage mirror
(LS.defaultViewFilters) written by SavedViewsProvider and read by
FeedFiltersProvider on load. That raced the login effect: on a genuinely
fresh browser (no LS.filters and no mirror yet) the feed could seed
DEFAULT_FILTERS and never re-apply the default until a reload. Plex had no
default-application at all.

Replace the mirror with a single path: SavedViewsProvider applies the
starred default via cfg.setFilters on the views query's FIRST resolve, but
only into a genuinely fresh module (no stored filters for the account, and
— feed only — no URL filter override). Applying persists it, so it happens
once and the last-view-wins path takes over, matching the documented
storage semantics. Because it runs off cfg.setFilters it delivers the Plex
port for free. Extract pickDefaultToSeed (pure, unit-tested) and cover the
first-resolve/stored/no-default cases for both modules in a provider test.

Removes the now-dead LS.defaultViewFilters key and its mirror read.
2026-08-23 23:05:24 +02:00
peter 5051809940 fix(a11y): touch/keyboard reveal tail + single-Escape layer coordination (UX-7/UX-8)
UX-8: one Escape must close only the topmost layer, never both an open
popover AND the player beneath it. The popover's Escape listener is on
`document` and fires before the player's on `window`; closing the popover
flushes its layer off the shared stack synchronously, so the player's
`anyLayerAbove()` lookup read "nothing above" and closed too. Add a
consumed-event flag to layerStack (`markEscapeHandled`/`escapeHandled`):
the acting topmost layer stamps the shared KeyboardEvent, and both players
check the stamp before the (now-mutated) stack. Immune to listener order
and mid-dispatch stack mutation. Cover with layerStack.dom.test.ts.

UX-7: the E4 `.reveal` sweep already covered the card action rows; two
residual hover-only traps remained — invisible-but-tappable/Tab-focusable
on touch. Add a group-agnostic `.reveal-hover` recipe (display:none under
`(hover: none)`) paired with each site's own hover/focus-within reveal:
- VideoCard play/Continue/Restart overlay: also reveal on
  group-focus-within (keyboard), and drop it on touch — the card tap
  already plays/resumes, so no permanent scrim over every mobile thumbnail.
- Carousel prev/next arrows: gone on touch, where the strip swipes.
2026-08-23 21:59:58 +02:00
peter 64dde6fd36 chore(format): fix prettier drift on plex pickers landed on main
`prettier --check` was red on main for these two files (drifted in the
collection-review-tail merge); no logic change, format-only.
2026-08-23 21:59:41 +02:00
peter 38921fb997 Merge chore/collection-review-tail: shared PlexLibraryPicker + scroll/create tests + 409->nameTaken sweep
gate / frontend (push) Failing after 36s
gate / backend (push) Successful in 2m3s
2026-08-23 18:42:56 +02:00
peter 2d1a7ff374 fix(plex): complete the 409->nameTaken sweep + cover PlexLibraryPicker
The notifyCollectionEditError helper localizes the backend's duplicate-name
409 to plex.collection.nameTaken; it was applied to the New-collection panel
but two sibling collection-write sites still routed 409 through
notifyActionError, surfacing the server's raw English detail under the HU UI:

- PlexCollectionView.doRename (rename collision)
- PlexCollectionEditor.createNew (create-in-editor duplicate name)

Both now use notifyCollectionEditError, matching the create path. Also adds a
direct dom test for the extracted PlexLibraryPicker covering the enabled gate,
emptyHint, results, honest error, and no-results branches.
2026-08-23 18:39:22 +02:00
peter 361ca7872e docs: add collaboration section to CLAUDE.md, document ruff format gate
gate / frontend (push) Failing after 1m16s
gate / backend (push) Successful in 2m16s
- CLAUDE.md: new "Collaboration" section (plan-first, options-with-marked-
  optimal-and-why, mockups for design decisions, challenge unsound requests,
  visual UAT before done, terse otherwise).
- CLAUDE.md: clarify release is user-owned in timing, agent-run in execution.
- CLAUDE.md + docs/development.md: document the ruff format --check gate step
  (already enforced in .gitea/workflows/check.yml, was missing from the docs).
- CLAUDE.md: refresh the stale footer to point at the now-existing
  docs/development.md and backend/app/README.md.
2026-08-23 17:17:02 +02:00
peter 81b7660232 test(plex): cover create write-through + the 409->nameTaken mapping (F-F)
Backend: create-collection happy path (write-through + editable) and the
create resync-failure 422 (honest reconcile detail), mirroring the rename/remove
tests. Frontend: extract notifyCollectionEditError (the 409->nameTaken vs
generic-fallback mapping that was inline in the New-collection create()) and
unit-test it; wire create() through it.
2026-08-23 08:08:34 +02:00
peter 7caa2636e4 test(nav): cover PageScroller save/restore keying (F12)
Renders PageScroller and drives restoreScroll's rAF synchronously with fake
metrics: asserts a forward nav (new entry id) opens at top and Back restores
the offset, and that a new content key within one entry resets to top while
returning restores — the per-(entry,key) contract only the id primitives were
covering before.
2026-08-23 08:04:00 +02:00
peter 1e894d378d refactor(plex): extract shared PlexLibraryPicker (F6)
AddItemPanel (Collection page) and NewCollectionPanel (browse) had near-
duplicate debounced library-search + result-list rendering. Extract it into
one PlexLibraryPicker that owns the query, the loading/error/empty states and
the row markup; callers keep their own chrome and pass term + renderAction.
No behaviour change.
2026-08-23 08:00:54 +02:00
peter 6fb369ffcc Merge chore/ruff-format: adopt ruff format (line-length 100)
gate / frontend (push) Failing after 1m24s
gate / backend (push) Successful in 2m3s
2026-08-23 07:50:25 +02:00
peter a3500340cf chore: add the ruff-format pass to .git-blame-ignore-revs
So git blame skips the mechanical line-length-100 reformat (b1832af).
2026-08-23 07:43:25 +02:00
peter b1832afd18 style: apply ruff format across backend + scripts (line-length 100)
Mechanical, whole-tree `ruff format` pass — no behaviour change. Recorded in
.git-blame-ignore-revs (next commit) so git blame skips it.
2026-08-23 07:43:06 +02:00
peter 434a98d6ed chore(lint): set ruff line-length 100 and gate ruff format --check
Adds ruff's formatter to the gate. line-length 100 matches the frontend
prettier printWidth. The reformat itself lands as the next commit (kept
separate so it can go in .git-blame-ignore-revs).
2026-08-23 07:42:42 +02:00
peter cc1f196a2c fix(build): drop the unused readMerged export so the publish gate passes
gate / frontend (push) Failing after 1m22s
gate / backend (push) Successful in 2m5s
knip flagged src/lib/storage.ts readMerged as an unused export — its only
caller is readAccountMerged in the same module, so it doesn't need to be
exported. This aborted the 0.72.0 publish gate after the merge; no behaviour
change.
2026-08-23 07:31:24 +02:00
peter e3dac34bb0 release: 0.72.0 (feature/plex-filter-bar-overhaul)
gate / frontend (push) Failing after 41s
gate / backend (push) Successful in 2m6s
2026-08-23 07:27:18 +02:00
peter 938135bb59 chore(release): 0.72.0 — Plex browsing overhaul + editable Collection page
Plex filter-bar overhaul, in-header Discover/Browse + Playlist/Collection
modes, filter-aware Discover rails, poster-collage tiles, Recently-watched
rail, capped unseen Surprise, saved views on Plex, per-page scroll memory, and
a full Collection info page with write-through Plex editing (create/rename/
add/remove/delete). Migration 0073 (saved_views.source). Two deep-review rounds.
2026-08-23 07:26:43 +02:00
peter 7d2bee59a3 fix(plex): round-2 deep-review follow-ups on the review fixes
- collections: the resync-failed-after-successful-write path now returns 422
  (honest "reconcile on next sync" detail), NOT 502. The frontend treats
  502/503/504 as gateway-unreachable and raises a sticky offline banner
  (api/core.ts RETRIABLE_GATEWAY); the app answered, so a 422 action-error
  toast is correct. Removes the false-offline regression the 502 introduced,
  and — since 422 isn't retried — the create double-write vector with it.
- tests: make the mutate-then-resync rollback assertion non-vacuous — the
  resync stub now flushes a partial local change before raising, so the test
  proves db.rollback() actually undoes it; split into a write-failed (422)
  case and a resync-failed (422 + reconcile detail + rollback) case.
- New-collection dup hint is now muted-tone with its own nameMaybeTaken copy
  (per-library uniqueness means the name may be free here) instead of the red
  authoritative nameTaken, which the still-enabled create flow contradicts.
- history: cover the navCounter post-reload collision guard (ensureNavId
  advances the counter past a higher persisted sfScroll id).
2026-08-23 07:22:53 +02:00
peter 6af238c2ea fix(plex): resolve deep-review findings on the Discover/Collections batch
Round of USER-triggered deep-review fixes (11 CONFIRMED + 1 PLAUSIBLE, all
verified) on the 22-commit branch:

- collections: split the mutate-then-resync error surface. A resync READ that
  fails after a successful Plex mutation no longer reports a false 422 "Plex
  write failed" + rollback; it returns 502 "sync pending" (the next full sync
  reconciles the mirror). New _write_or_pending_error helper; delete is exempt
  (no Plex read). Covered by new happy-path / 422-rollback / 502 tests.
- New-collection dup-name guard is now advisory only: existingNames is the
  search-scoped, all-library union, but the backend enforces uniqueness PER
  library — so it no longer blocks a valid same-name collection in another
  library, and the backend 409 is localized to nameTaken.
- member-remove now confirms before the write-through (matches delete-collection).
- add-title / new-collection library search render an error state on request
  failure instead of a misleading "No matches".
- history: seed navCounter past the current entry's persisted sfScroll on load
  so post-reload ids can't collide with back-stack entries and mis-restore scroll.
- PageScroller: LRU-cap the navId-keyed savedScroll map (was unbounded).
- share isIntrinsicReadOnly between the browse tile and the info page so the
  lock/edit affordance can't drift.
2026-08-23 06:57:33 +02:00
peter b71b33654a feat(plex): forbid duplicate collection names + a "New collection" button
Two follow-ups after testing the collection editing against real Plex, which revealed that
Plex CONFLATES same-named collections (two rating_keys resolve to one merged member list) —
a genuine footgun.

- Duplicate-name guard: create and rename now reject (409) a title that already exists in the
  target section (case-insensitive), before any Plex write. create() can also derive the
  library from the seed item, so the caller needn't know the section key. The create dialogs
  surface the 409, and the New-collection panel warns inline before submit.
- A "New collection" button on the Collections browse surface (admin only) opens a panel:
  a name field (with the live duplicate check) + a search to pick the FIRST title — Plex can't
  create an empty collection, it needs a seed. Creating opens the new collection's detail page
  straight away, so you can keep adding titles there. This is the B-route the user wanted,
  vs. the only prior path (a movie's "add to collection").

Verified live: the 409 guard (creating an existing name is blocked, no Plex write), and the
full New-collection flow (name + first title → create → lands on the detail page); test
collections created and deleted, real account left clean.

Gate: tsc/lint/knip/prettier clean, 424 vitest; backend ruff clean, 7 collections tests
(+ a duplicate-name-rejection assertion).
2026-08-23 05:51:48 +02:00
peter 5d99a5f253 feat(plex): Collection info page — hero, play/shuffle, and write-through editing (B)
Clicking a collection tile now opens a dedicated Collection info page (like the Plex-web
one) instead of just applying it as a filter. New GET /collections/{rk} returns the hero
aggregates (year range, most-restrictive member cert, total runtime) + the members ordered
by release year then title; a new PlexCollectionView renders the hero (art/title/meta/
summary+More) and the member grid, with a Play-all and Shuffle over the playable leaves
(reusing the player's existing queue + auto-advance).

For a non-external, admin-enabled collection the page offers rename / delete / add-title
(a search picker) / remove-item, all through the pre-existing admin endpoints that write
THROUGH to Plex (each edit awaits and re-fetches — no optimism, since a Plex write can 422).
An eligible collection shows "Enable editing" first (the DB editable flag). Read-only ones
(smart / imdb / tmdb / … auto-lists) are marked with a lock on both the info page AND the
browse tile, up front.

New `{ kind: "collection" }` sub-view wired through plexSubUrl (bookmarkable ?v=collection:rk),
PlexBrowse (render branch, Backspace back-nav, tile handler), with a round-trip test.

Verified live: tile→page, hero + chronological members (James Bond 1962→2021, PG-13, 55h),
Play/Shuffle, Enable editing → edit controls + remove-X, the add search (marks already-in
titles "Added"), and the read-only lock on external collections. Real Plex mutations wired
but not executed against the live account.

Gate: tsc/lint/knip/prettier clean, 424 vitest; backend ruff clean, 6 collections tests
(+ a detail-hero/ordering assertion).
2026-08-23 04:40:09 +02:00
peter c0d869089d feat(plex): add a "Recently watched" rail to Discover (D)
A new Discover rail listing what the user last FULLY watched, newest first: a movie as its
own card, a show as the specific EPISODE watched (carrying its show title via leaf_card). Sits
above Recently added. Movies honour the sidebar filters like the other rails; episodes carry no
filterable metadata so they don't. Clicking a card plays it (the existing onCard path already
routes episodes to the player). Bilingual heading added.

Verified live: the rail renders watched movies + episodes with Watched badges.

Gate: tsc/lint/knip/prettier clean, 423 vitest; backend ruff clean, 13 discover tests (+ a
recently-watched movie+episode assertion).
2026-08-23 04:08:28 +02:00
peter 15f5a5694a feat(plex): Surprise-me shows only unseen titles, capped 3 movies + 3 shows (C)
The Surprise rail drew a random 20+20 sample including already-watched titles. Its point is
to surface something to WATCH, so exclude seen titles (a watched movie, a fully-watched show
— a partially-watched or never-started show still counts as unseen) and cap at 3 movies + 3
shows (6 total), interleaved.

Verified live: /surprise returns 6 items, 3+3, none watched.

Gate: backend ruff clean, 12 discover tests (+ a watched-exclusion/cap assertion).
2026-08-23 04:01:22 +02:00
peter 14a44bff6e fix(plex): show-library collections get montage art too (A3)
The collection browse-tile montage queried only movie members, so show-library
collections (IMDb Popular, TMDb Trending, … in the TV library) rendered as bare colour
tiles. Union the art from PlexShow members as well (a collection lives in one library, so
exactly one side has art); fall back to the collection's own poster, then the colour hash.
Runtime stays movie-only (shows have no single duration).

Verified live: every collection now shows a montage (0 art-less tiles, was ~all the show
collections).

Gate: backend ruff clean, 5 collections tests (+ a show-collection art assertion).
2026-08-23 03:57:20 +02:00
peter 37d11b602f feat(plex): redesign browse tiles — 2-poster montage top, readable label band
The full-bleed 4-poster collage drowned the on-tile text. Split PosterTile in two: the top
half shows up to 2 montage posters, the bottom half is a solid colour-hash band that carries
the label, so the text is always readable. Genre tiles drop the count (just the label now);
collection tiles show the name on one truncated line plus a sub-line of item count · total
runtime. A group with no art falls back to an all-colour tile.

Backend: `/collections` cards gain `total_runtime_s` (a new `runtime_by_key` grouped sum) and
their montage art is capped at 2. Frontend: PosterTile reworked; a `collectionMeta` helper
builds the count·runtime sub-line via the existing formatRuntime.

Verified live: collection and genre tiles render 2 posters over a readable colour band with
the right sub-lines; colour fallback intact for art-less groups.

Gate: tsc/lint/knip/prettier clean, 423 vitest; backend ruff clean, collections tests green
(+ a new art/runtime assertion).
2026-08-23 03:52:21 +02:00
peter da3cb5e694 feat(plex): poster collages on the genre + collection browse tiles
The Browse-by-genre tiles and the Collections browse tiles were flat color-hash
gradients. Give each a 2×2 collage of its top films' poster art, with the color hash
kept as the fallback so a group with no matching art (a show-only or empty collection)
never renders blank.

Backend: a shared `top_art_keys` windowed query returns, per key (a genre value or a
collection rating_key), the top few items' rating_keys — best-rated then newest, in one
query. Wired into the Discover genre tiles and the collections union endpoint; the client
renders the posters from the existing /api/plex/image/{key} endpoint, so no new image
plumbing. Frontend: a shared PosterTile component (collage + scrim + label, color-hash
fallback) replaces the two ad-hoc colored-tile implementations.

Verified live on localdev: genre tiles and collection tiles render real poster collages,
with the color fallback correctly kicking in for groups without art.

Gate: tsc/lint/knip/prettier clean, 423 vitest; backend ruff clean, 11 discover tests
(incl. a new genre-art assertion).
2026-08-23 03:10:36 +02:00
peter 1e5232f47a feat(nav): restore scroll position per history entry, open forward views at the top
The app scrolls one persistent <main> (PageScroller); its saved offsets were keyed by a
content `scrollKey` that is the same for a Plex page's Discover and grid views, so opening
a Browse-by-genre tile inherited the scrolled-down Discover offset (landing mid-page) and
Back couldn't tell the two views apart.

Tie scroll memory to the history entry instead: every pushState that opens a main-content
view stamps a fresh `sfScroll` id into history.state (setPage, Plex sub-view open); the
initial/legacy entries get one lazily via replaceState. PageScroller now saves/restores per
`${entryId}:${scrollKey}` — the entry id makes Back restore exactly where you left an entry
and a forward navigation (new id) open at the top, while the scrollKey still resets a new
feed filter to the top within the same entry. A small nav bus lets a Plex sub-view open
(which doesn't re-render the scroller) trigger the restore; `history.scrollRestoration` is
set to manual since the browser can't see our inner scroller.

Verified live: genre tile opens the grid at the top and Back returns to the Discover scroll;
feed→plex→Back restores the feed scroll; a feed filter change still resets to the top.

Gate: tsc/lint/knip/prettier clean, 5 new nav-id unit tests, full vitest green.
2026-08-23 02:55:59 +02:00
peter dddaeb9a38 fix(plex): make the header Discover/Browse toggle clickable
The floating header wrapper is pointer-events-none (so clicks fall through its empty
area to the content below); every interactive child must re-enable pointer events.
PlexViewToggle rendered its SegmentedToggle without the pointer-events-auto wrapper the
sibling PlexScopeToggle has, so the whole Discover/Browse switch was click-dead — the
buttons rendered and were correctly wired, but no real mouse click ever reached them.

Add `className="pointer-events-auto shrink-0"` to the toggle and a regression test that
asserts the root re-enables pointer events (plus the render/onChange/hide-on-deeper-view
contract). Verified live on localdev: real clicks now switch Discover⟷Browse (?v=browse)
in both directions; a header-wide hit-test audit shows no other dead controls.

Gate: tsc/lint/knip/prettier clean, 418 vitest.
2026-08-23 02:23:09 +02:00
peter 70a74f8629 fix(plex): round-3 deep-review follow-ups
Third review round, scoped to the round-2 fix commit. The production ternary was
confirmed correct; two sibling-surface findings:

- The sidebar CollectionPicker's clear-all still wrote collectionTitles:{} while the
  chip-row remove (fixed in round 2) wrote undefined — a divergence that leaves the
  {}-vs-absent cache split on the picker path. Collapse an empty map to undefined there
  too, so both no-collection routes key-match the pristine state.
- The queryKeys lang test was near-tautological (a variadic-spread factory can't drop a
  passed arg), and nothing asserted the CALL SITE keys carry lang. Add a real guard in
  PlexDiscover.dom.test.tsx that inspects the registered query keys for scope+lang, and
  reword the factory test's comment to say it only shape-pins the segment pass-through.

Gate: tsc/lint/knip/prettier clean, 413 vitest.
2026-08-23 02:09:14 +02:00
peter 8f1309fa18 fix(plex): round-2 deep-review follow-ups on the review fixes
Second review round, scoped to the round-1 fix commit (where round-1's own fixes
introduced new code):

- The collection chip-remove pruned collectionTitles to an empty {} on last removal,
  which still hashes differently from the pristine no-collection state (key absent) in
  the React Query keys → a residual cache split. Collapse an empty map to undefined so
  the key-shape matches the pristine one.
- Replace the tautological "order-independent signature" test (both objects spread from
  EMPTY_PLEX_FILTERS inherited the same key order, so it proved nothing) with full
  literals in genuinely different insertion order that would fail an order-sensitive
  JSON.stringify signature.
- Cover the collectionTitles chip-remove prune (usePlexActiveFilters.dom.test.tsx) and
  the plexSurprise/plexDiscover lang-key split (queryKeys.test.ts) with regression tests.

Gate: tsc/lint/knip/prettier clean, 412 vitest.
2026-08-23 01:53:33 +02:00
peter b7a4eaff8d fix(plex): resolve deep-review findings on the filter-bar overhaul
Self-review of the branch diff (feature/plex-filter-bar-overhaul) against main:

- Surprise rail query key omitted `lang` while the backend /surprise now filters
  against the language-specific genre column → a HU/EN switch under an active genre
  filter left the rail stale. Key it by scope+lang+filters, matching plexDiscover.
- Chip-removing a collection left its entry in `collectionTitles` (hashed into the
  plex query keys), splitting the cache from the pristine no-collection state and
  growing the titles map unboundedly. Prune the title on remove, like CollectionPicker.
- The saved-views default-mirror effect gained `cfg` in its deps (re-memoized on every
  live filter edit) → a synchronous localStorage write per keystroke. Depend on the
  module-stable mirror fn instead.
- Collections-browse search input had no accessible name (placeholder only). Add
  aria-label + type="search".
- Extract the tile hue-hash + gradient duplicated between PlexDiscover and
  plexBrowseLists into a shared lib/hueTile so the two surfaces can't drift.
- Document migration 0073 downgrade() as lossy and de-dup non-feed rows before
  recreating the narrow (user_id, name) constraint (verified by the drift test).
- Add unit tests for coercePlexFilters (legacy single-collection migration) and
  plexFiltersSignature (order-independence, title-ignoring).

Gate: tsc/lint/knip/prettier clean, 410 vitest; backend ruff + migration drift +
saved-views tests green.
2026-08-23 01:39:29 +02:00
peter a20e27800c feat(plex): port the saved-views (personal views) UI to Plex (frontend)
Plex now has the same personal-views system as the feed — the rail header icon
opens the quick-pick popover, the rail title opens the manager modal (list /
create / rename / delete / reorder / set-default / apply), and the active view
leads the chip row with a "modified" dot.

One SavedViewsProvider serves both modules: it switches `source` (feed ⟷ plex)
by the active page (both filter providers are global, so it reads whichever the
page owns) and runs the same logic on per-source wiring — the live filters,
apply, a stable signature (feed filtersToParams / plex plexFiltersSignature),
coercion (asFilters / coercePlexFilters), and what to strip on save. Moved below
PlexProvider in the tree so it can read plex state; resets sticky/modal on a
source switch. The shared popover/modal/chip are unchanged; the modal describes
a plex view's captured filters via a pure buildPlexActiveFilters (extracted from
usePlexActiveFilters — no duplication). api.savedViews/createView take a source;
`SavedView.filters` is now an opaque blob. New coercePlexFilters/plexFilters-
Signature (also dedups PlexProvider's collection migration). Empty-state copy
genericised (no longer says "feed"). A Plex view captures the rail filters (not
watch-state/sort/scope).

Verified end-to-end on localdev: create/apply/preview/default/delete on plex,
feed views stay separate + unregressed (402 vitest), no console errors.
2026-08-22 19:16:30 +02:00
peter fa7a09a420 feat(saved-views): scope saved views by module source (backend)
Saved "smart views" were feed-only; the Plex module is getting them too. Add a
`source` discriminator to saved_views ("feed" | "plex"), default "feed" so every
existing row and any source-less client stays feed. Widen the per-user unique
name constraint to (user_id, source, name) so a feed view and a Plex view can
share a name, and scope create's position numbering + the is_default "clear the
others" to (user, source).

Migration 0073 (ADD source NOT NULL server_default 'feed' + swap the unique
constraint; reversible). Endpoints: GET/POST take `source` (default feed).
Tests: source isolation, per-source default, per-source name uniqueness.
2026-08-22 18:56:52 +02:00
peter f983a491c0 feat(plex): add Playlist/Collection browse modes to the header pill
The header library-scope pill gains two entry points — Playlist and Collection —
next to All/Movies/Shows, a second way into each area (the rail filter is the
other). Picking Playlist shows a grid of the user's playlists (a tile opens that
playlist's view); Collection shows a searchable grid of collections (a tile
filters the title grid by that collection). Pure front-end — reuses the existing
plexPlaylists / plexCollections(union) list endpoints.

Two new top-level sub-views (`playlists` / `collections`, bookmarkable as
?v=playlists / ?v=collections) rendered by PlexBrowse via the new plexBrowseLists
surfaces (hue-tiled like the Discover genre tiles). The PlexViewBridge now carries
the view `kind`; PlexScopeToggle is a 5-way that derives its active segment from
it (browse-mode when in one, else scope) and drives the two views through the
bridge — picking a scope from a browse mode returns to the title grid. The
Discover/Browse toggle hides in these modes; Backspace + the search→grid flip
cover them.

Tests: PlexScopeToggle (5-way + browse-mode drive/active), plexSubUrl round-trip
for the two new views. Gate: tsc/lint/knip/prettier + 402 vitest.
2026-08-22 18:19:08 +02:00
peter f85dc12819 feat(plex): move the Discover/Browse toggle into the header
The Discover ⟷ Browse switch now sits in the header next to the library-scope
pill (All / Movies / Shows), instead of on the chip row / atop the Discover
surface.

It's driven through a small one-way PlexViewBridge: PlexBrowse still OWNS the
sub-view (useHistorySubview) and publishes its top-level view + opener to the
bridge; the always-mounted Header reads it to render PlexViewToggle. A global
sub-view provider was deliberately avoided — PlexBrowse's hook remounts on every
plex entry and re-seeds from history (a page switch clears ?v + drops _sub), so
each entry starts at the Discover root; lifting it out would let the view go
stale across navigation. The toggle hides on deeper views (show/info/person/…)
and on non-plex pages (bridge cleared on unmount). sortView is just SortControl
now; the grid toolbar no longer carries the toggle.

Verified: toggle in the header, switches discover⟷grid, hides on info + on Feed,
and Feed→Plex resets to Discover (no stale view); no console errors.
2026-08-22 18:02:55 +02:00
peter 9798e506cf feat(plex): apply the sidebar filters to the Discover rails too
Discover now behaves like the Browse grid — the active metadata filters (genre /
rating / year / collection / studio / …) narrow every rail: Continue watching,
Recently added, Surprise me, the genre tiles, and Because you watched. Same
LibraryFilters / meta_filter_conds the grid uses, applied to each rail's movie +
show query (movie duration included); a rail may come back empty, which the client
already hides. Genre-language matches the grid.

Because Discover honours filters itself, adding a filter no longer force-flips the
Discover landing to the grid (only a header SEARCH does, since Discover has no
search rail) — so you stay on Discover with narrowed rails.

Frontend threads `filters` into api.plexDiscover/plexSurprise + their query keys
(plexSurprise is variadic now) and the watched-toggle setQueryData; PlexDiscover
reads them from usePlex. Tests: two discover filter-narrowing cases; PlexDiscover
dom test updated for the provider default + 3-arg call.
2026-08-22 17:35:37 +02:00
peter a60164edc4 feat(plex): make the Collection filter multi-select with Any/All
A title can belong to several Plex collections, so the collection filter becomes
multi-select combining with Any (OR, default) / All (AND) — the same shape as the
genre/actor filters.

Backend (query.py): `collection` (single) → `collections` CSV + `collection_mode`
on LibraryFilters; meta_filter_conds ORs/ANDs `collection_keys.contains` per id.
Frontend: PlexFilters `collection`/`collectionTitle` → `collections` + optional
`collectionMode` + `collectionTitles` map; CollectionPicker is now a multi-select
Combobox with the any/all match footer; one chip per collection (usePlexActive-
Filters); the info-page "Browse collection" strip unions into the list; applyFilter
now shallow-merges label maps so adding one keeps the others' titles. PlexProvider
migrates a legacy persisted single `collection` into the new shape on read.

Test: test_plex_collections_filter (single / any-union / all-intersection).
2026-08-22 08:18:25 +02:00
peter 6c7038cf1f feat(plex): make the playlists rail widget a compact combobox
The playlists section used to render an always-open list plus a dedicated
create input, growing the rail taller with every playlist. Replace it with a
single Combobox field (reusing the shared component, like the feed's "Your
tags"): the list of playlists only appears when the dropdown is opened, and
typing a name that doesn't exist offers a "Create …" row. Picking a playlist
opens it (the existing playlistOpen hand-off); the create row creates then opens
the new one. `selected` stays empty — this picks an action, not a persisted value.
2026-08-22 07:59:41 +02:00
peter f8e144a658 fix(combobox): stop a single chip wrapping the chevron to a second line
The field's input carried min-w-16 (64px). Flex line-packing sizes a flex item
at its min-width, so in a narrow rail field (~202px) one chip (~119px) + the
64px-reserved input + the chevron overflowed the line, wrapping the chevron
down while the (grown, empty-looking) input filled the first line — the "breaks
to two rows even though it'd fit" glitch seen on the Plex Collection widget and
the feed's channel picker (both are Combobox). Lowering the floor to min-w-8
(32px) keeps chip + input + chevron on one line; flex-1 still grows the input to
fill the remaining width. Verified in-browser: field height 54px→36px (one row).
2026-08-22 07:56:05 +02:00
peter b4984dc38b feat(plex): move Watch-state out of the rail into a toolbar pill
The "Show" (watch-state) filter leaves the left rail and becomes a FilterSelect
pill next to the item count, mirroring the feed's Watch-state control — a
top-level "what am I looking at" choice, always in reach, not buried in the rail.
It still emits a removable chip when non-default (same as the feed).
2026-08-22 07:51:54 +02:00
peter 21c9b34540 feat(plex): port the filter bar to the YT-feed pattern
Bring the Plex module's filter surface in line with the feed's final shape:

- Flatten the rail: PlexSidebar passes `collapsible={false}` to FilterRail so
  the widgets become plain always-open label + control rows (no collapse
  chevrons; reorder/hide in edit mode stay). Drop the rail's own "Clear all" —
  the chip row above the grid already owns it.
- Move Sort + the Discover/Browse toggle into the applied-filter chip row,
  pinned right directly above the grid (a `sortView` fragment, like the feed's),
  and give ActiveFilterChips `flex-1` so it gets the shared responsive overflow
  (wide: fit + "+N" pill; narrow: single "N filters" count pill) — the chips can
  no longer crowd the controls off-screen.

ItemCount leads its own toolbar row, matching the feed. No behaviour change to
Discover, which keeps its standalone view toggle.
2026-08-22 07:23:20 +02:00
peter 041cd526b7 Merge: YT feed filter-bar overhaul + 3 deep-review rounds
gate / frontend (push) Failing after 38s
gate / backend (push) Successful in 1m57s
Flattened, more compact YT feed filter rail and a responsive applied-filter row:
- Rail: removed the in-rail Clear all; Upload date is now a single-select dropdown
  (Any time + more presets + reveal-on-select Custom range); widgets are plain
  label + control rows (non-collapsible), with the "Your tags" title opening the
  tag manager.
- Sort + View moved into the applied-filter row (pinned right, above the grid)
  with responsive overflow: "+N" pill on wide lanes, a single count pill on narrow.
- Earlier branch work also merges here: typeable combobox filters, hidden-channel
  empty->Library hint, Mine/Library scope toggle in the header, per-category
  any/all tag match, and the saved-views redesign (popover + modal + chip).

Three USER-triggered deep-review rounds (trend 14 -> 4 -> 1); round 2 caught a real
coerce-on-seed migration bug. Gate-green: tsc, 397 vitest, lint, prettier, backend
pytest. Not yet released -- prod stays 0.71.0.
2026-08-22 07:09:25 +02:00
peter 9477fea3dc docs(test): correct the feed-views fixture comment on topic tag distribution
chA/chB each carry a different topic tag (T1 vs T2), not both — the old comment
implied they shared T1/T2. Assertions unchanged (equivalence test).
2026-08-22 07:04:00 +02:00
peter e4b19b5320 fix(feed): round-2 deep-review remediations (incl. a real coerce-on-seed bug)
The re-review of the fix round surfaced 4 findings; fixing the coverage one
exposed that the C1 migration was ineffective:
- FeedFiltersProvider.loadAccountFilters: run coerceTagModes on the RAW stored
  blob, not the object merged over DEFAULT_FILTERS. The merge injects
  DEFAULT_TAG_MODES (all "or"), which coerceTagModes prefers over the legacy
  scalar `tagMode` — so the migration never fired. Now a persisted legacy
  tagMode:'and' actually migrates into tagModes.topic (proven by a new test).
- test_feed_views: the tag_mode_other=and case was vacuous (only chC carried
  either "other" tag, so AND == OR). Give chA an "other" tag too so OR-other
  {chA,chC} differs from AND-other {chC} — the conjunctive path now narrows.
- DateFilterSelect: reset the custom reveal in useLayoutEffect (not useEffect)
  so the correction commits before paint — no one-frame "Custom range" flicker
  on the trigger after an external clear.
- SavedViewsProvider: extract the sticky-view tie-break to a pure, tested
  lib/savedViews.ts (preferSticky).

New tests: savedViews (preferSticky), FeedFiltersProvider.dom (legacy tagMode
migration on the localStorage/default-view seed).
2026-08-22 06:55:16 +02:00
peter 7692253636 test(feed): cover the filter-bar gaps deep-review flagged
- test_feed_views: extend _seed_tagged so a channel carries 2+ tags in a
  non-topic category, and add language-AND / other-AND equivalence cases — the
  per-category any/all generalization was only exercised for topic.
- tagModes.test: coerceTagModes migration (legacy global tagMode -> topic),
  per-category preservation, invalid-value coercion, defaults.
- urlState.dom.test: per-category tag-mode URL round-trip + legacy ?tagmode=and
  migration into the topic slot.
- Combobox.dom.test: the any/all match footer (hidden below 2 selections, reports
  a mode change at 2+).
- chipOverflow.test: the all/some/count fit boundaries and the +N reservation.
2026-08-22 06:33:48 +02:00
peter e90603d4db fix(feed): deep-review remediations for the filter-bar overhaul
Correctness / behaviour:
- Move the Personal Views modal from inside the collapsible filter rail to the
  feed page (moduleRegistry FeedPage), so opening it from the always-visible
  toolbar view chip works while the rail is unpinned (its default) instead of
  no-op'ing and latching modalOpen into a later stale pop-open.
- Migrate a persisted legacy global `tagMode` into per-category `tagModes` on the
  localStorage seed paths too (loadAccountFilters), matching the saved-view and
  URL paths — an upgrading user's topic any/all no longer silently reverts to 'or'.
- DateFilterSelect: reset the custom reveal on an external clear while the popover
  is closed (guard on !open, not maxAgeDays), so the trigger reverts to "Any time"
  instead of lingering on "Custom range" with empty fields.
- PersonalViewsModal delete: only advance the detail selection when the delete
  actually happened (remove() now resolves a boolean) — cancelling the confirm no
  longer switches the selected view.
- SavedViewsProvider: prefer the explicitly-applied (sticky) view among several
  that capture identical filters, so apply(B) isn't re-attributed to an earlier
  duplicate A (wrong "modified" chip name).

A11y:
- Give the clickable panel title a descriptive accessible name (titleActionLabel,
  wired to "Personal views") instead of announcing only "Filters".
- Give the "+N" overflow pill an aria-label ("N more filters").
- Use t("views.reorder") for the saved-view drag handle instead of a hardcoded
  English "reorder" literal.

Also extract the chip-fit decision into a pure, testable lib/chipOverflow.ts and
measure the inter-chip gap live (rem-safe) instead of a frozen 6px constant.
2026-08-22 06:33:48 +02:00
peter dd76e9dcf7 feat(feed): move sort + view into the chip row with responsive overflow
Sort and the view switcher are tightly bound to the grid, so pin them to the
right of the applied-filter row (directly above the cards) instead of the filter
toolbar; they render there on both the feed and channel-scoped views.

ActiveFilterChips now shares that row with the controls and trims itself so it
never crowds them, measuring fit off a hidden mirror of the full chip set:
- wide  -> show as many chips as fit, then a "+N" pill whose popover holds the rest;
- narrow -> collapse every chip into one "N filters" pill whose popover holds them all.
Both popovers list the (overflowed / all) chips, each removable, plus Clear all.
2026-08-22 05:42:45 +02:00
peter fca989c098 feat(feed): flatten the filter rail — dropdown upload-date, non-collapsible widgets, title triggers
Rework the feed filter rail into simpler label + control rows:
- Remove the rail's own "Clear all" (the chip row above the cards already
  surfaces one whenever any filter is active).
- Replace the Upload date chip group with a single-select dropdown
  (DateFilterSelect): an "Any time" clear, more presets (2 weeks / 3 months /
  2 years / 5 years) and a "Custom range" that reveals from/to on select; the
  trigger then reads the chosen range. Selection logic is the pure lib/dateFilter
  (applyDateChip -> selectDateChip, no toggle-off; tests updated).
- Widgets are no longer collapsible: PanelGroup gains `collapsible` (feed passes
  false) so groups render as always-open label + control rows; reorder + hide in
  edit mode stay. Other rails (Plex/Playlists) keep the collapsible islands.
- A non-collapsible group's title can act as its own trigger (PanelItem
  onTitleClick): "Your tags" opens the tag manager, replacing the inline Manage
  button.
2026-08-22 05:42:30 +02:00
peter 7ebbf45921 feat(feed): saved views as a header popover + Personal Views modal + a view chip
Retire the standalone 'Saved views' sidebar panel. The saved-views feature now
lives in three lighter surfaces, coordinated by a new SavedViewsProvider (query +
CRUD + the active/sticky view + modal state):

- the filter panel's header ICON opens a quick-pick popover (apply a view, active
  one checked, plus 'Manage views…');
- the header TITLE ('Filters') opens a new Personal Views modal — a master/detail
  manager: left = the reorderable list + a save-current-filters input, right = the
  selected view's name, its captured filters (shown as the same chips the toolbar
  row uses), default toggle, apply, and delete;
- a SelectedViewChip leads the applied-filter row, naming the active view with a
  warn-coloured 'modified' dot once the live filters diverge from it; clicking it
  reopens the picker.

SidePanel/FilterRail gain optional iconExpanded + onTitleClick slots (feed-only,
hidden for the demo account); the collapsed tab keeps its plain pin behaviour.
SavedViewsWidget is deleted and dropped from the panel layout.

Verified live: icon popover lists HH + Manage; applying HH surfaces the chip and
switches scope; removing a filter lights the dirty dot; the title opens the
master/detail modal with the captured-filters summary. 374 vitest, tsc, eslint
(0 err).
2026-08-22 04:03:58 +02:00
peter 829cadf5fb feat(feed): per-category any/all tag match, in each combobox's popover footer
The tag 'Match: Any/All' toggle was a single GLOBAL setting that only ever
affected topics, shown as its own row under the Topic widget. Make it
per-category — language, topic, and 'your tags' each get their own independent
any/all — and move it into each filter combobox's popover footer (like
SortControl's direction segment), reclaiming the standalone Match row.

Backend: the feed query already groups selected tags by category; replace the
single tag_mode with per-category tag_mode_language/topic/other params and apply
the conjunctive (HAVING all) branch per category. Mirrored across filtered_query,
feed_summary.passes(), and the /facets drill-down exclusion (kept in lockstep;
equivalence test updated to tag_mode_topic). No schema change.

Frontend: FeedFilters.tagMode -> tagModes {language,topic,other} (pure leaf
lib/api/tagModes with a coerceTagModes migration for the legacy global value);
serialized to the API and the shareable URL per category (legacy ?tagmode=and
still resolves into the topic slot); saved-view blobs migrate on read. Combobox
gains an optional matchMode/onMatchModeChange footer segment, shown once 2+
values are picked; the three tag comboboxes wire it to their category.

Verified: backend feed tests green (12, incl. the count/facets equivalence net);
live, the footer appears at 2+ selections, toggles aria-pressed, serializes
tagmode_topic=and to the URL, and the request carries tag_mode_topic=and (200).
374 vitest, tsc, ruff, eslint (0 err).
2026-08-22 03:48:23 +02:00
peter 8f08073564 feat(feed): move the Mine/Library scope toggle beside the header search box
The scope toggle used to sit at the top of the filter rail, which is buried when
the rail is collapsed/unpinned. Relocate it to the header, just right of the feed
search box — mirroring Plex's scope toggle placement — so the primary 'what am I
browsing' control is always visible.

New FeedScopeToggle adapter (SegmentedToggle over filters.scope, same as the Plex
one over its scope); rendered in Header on the feed page between the search box and
the sync chip. Removed the toggle from the Sidebar rail's now-unused `top` slot and
dropped the dead `top` prop from FilterRail. Same filters.scope field, so behaviour
is unchanged.

Verified live: the toggle sits beside the feed search and switches scope; the rail
no longer carries it.
2026-08-22 03:29:15 +02:00
peter af08a7850b feat(feed): guide a hidden-channel empty result to the Library
Filtering by a channel under the Mine scope returns nothing when that channel's
subscription is hidden (or you don't subscribe to it) — its videos only live in
the shared Library, since the Mine feed excludes hidden subscriptions by design.
Previously this left a bare 'No videos match these filters', which reads as a
bug when you know you're subscribed.

Add an empty-state hint + a 'View in Library' button (switches scope to all) when
there's a channel filter under Mine and zero results. Bilingual EN/HU.

Verified live: ?channel=<ColdFusion>&show=all under Mine shows the hint; the
button switches to Library and surfaces the 36 videos.
2026-08-22 03:25:49 +02:00
peter d2193ccd1b fix(combobox): merged input ref so the field is typeable by mouse
The <input> carried only floating-ui's reference ref, so the component's own
inputRef stayed null. The field container's onMouseDown then preventDefault()-ed
every press (its `e.target !== inputRef.current` guard was always true) and the
compensating inputRef.current?.focus() was a no-op — so clicking the box opened
the list but never focused the input, and mouse users could not type to filter.
Tab focus still worked, which is why it read as 'click, can't type'.

Feed a merged callback ref that assigns the node to both floating-ui's
setReference and inputRef. Fixes the Channel, Your tags, Topic, and Language
sidebar filters at once (all use Combobox); the channel list's 50-row cap is now
reachable past 50 via search. Single-select commit no longer re-focuses (that now
actually fires and would re-open the just-closed list). Regression test presses
the field chrome and asserts the input receives focus.

Verified live: clicking the Channel field focuses it and typing 'cold' narrows
the list to ColdFusion / wehatethecold.
2026-08-22 03:22:47 +02:00
peter cda70164ba Merge: toolbar/header UI-consistency refactor + a11y fixes
gate / frontend (push) Failing after 1m13s
gate / backend (push) Successful in 1m52s
Collects the toolbar/header/layout redesign and assorted bugfixes on the
bug/channel-back-header-overlap branch: new FilterSelect / ItemCount /
pill-SortControl primitives, removed ModuleName, the --sbw / --content-mw /
--rail-pinned-w chrome-alignment machinery, the centred search box, and the
two deep-review rounds (menu-role a11y, sort aria-pressed, rem rail threshold,
accessible trigger names, e2e helper fix, added coverage).

Not a release: VERSION stays 0.71.0 and this is not deployed to prod.
2026-08-22 02:07:20 +02:00
peter 5e1cc4d981 fix(a11y,test): deep-review round 2 — e2e helper attr, trigger names, observer test
Second gate round over the full branch surfaced 4 verified findings (1 was a
regression the round-1 role change introduced); a 5th (--sbw load jump) was
dropped as refuted (the header only mounts after auth+data, so --sbw is set
before it ever paints).

- e2e (medium): the round-1 switch of FilterSelect rows from role=option to
  menuitemcheckbox moved their state from aria-selected to aria-checked, but
  the Playwright helper expectContentType still asserted aria-selected, so the
  channel specs would fail whenever the e2e lane runs (not part of the fast
  gate, so it stayed green). Assert aria-checked.
- a11y (low): the FilterSelect and SortControl pill triggers set
  aria-label={label}, which overrides the visible "Label: value" text, so a
  screen reader announced only the group name, never the current selection.
  Carry the value (and, for sort, the direction) in the accessible name,
  matching ViewSwitcher. Updated the unit-test trigger name queries to match.
- tests (low): cover the PageShell font-scale MutationObserver recompute path
  (a post-mount root font-size change → --rail-pinned-w re-computes once, no
  loop) — previously only the pre-render and resize paths were exercised.

Verified live: the triggers now announce e.g. "Sort: Date, Descending" /
"Watch state: All". Gate green: tsc (app+node+e2e), 373 vitest, prettier,
eslint (0 err).
2026-08-22 01:06:22 +02:00
peter 4ee452b854 fix(a11y,layout): deep-review fixes — menu roles, sort aria-pressed, rem rail threshold
Epic-end deep-review of the branch surfaced six verified findings; fix them.

a11y (medium): FilterSelect and SortControl opened their AnchoredPopover in
mode:"listbox" but never passed a role, so the surface fell through to
role="menu" while the rows rendered role="option"+aria-selected — options
orphaned outside a listbox, so AT could not announce the selected value.
Switch both to the house menu pattern (as ViewSwitcher already uses): a menu
surface with menuitemradio (single) / menuitemcheckbox (multi) rows carrying
aria-checked, scoped by a role=group. Trigger stays a button+haspopup=menu.

a11y (low): the SortControl asc/desc direction segment signalled the active
direction by colour alone. Add aria-pressed so the current direction is
programmatically exposed, matching SegmentedToggle.

correctness (low): PageShell converted a rem maxWidth to px with a hardcoded
16 to decide the pinned-rail overlay threshold, but index.css scales the root
font by --font-scale (default 1.06 ~ 17px, slider 0.9-1.3), so the threshold
was wrong at every text size on the rem pages (playlists is the only one with
a pinned rail). Use the live root font-size, scale the 36rem slack the same
way, and recompute when --font-scale changes (a guarded MutationObserver — the
slider fires no resize).

tests: cover the two gaps the review flagged — ItemCount (plural, 1M+ overflow,
undefined icon-only, tooltip) and the PageShell rail-pinned-w/content-mw
threshold math incl. the font-scale scaling. Updated the FilterSelect/
SortControl role queries to the new menuitemradio/checkbox roles.

Verified in the running app: menu surfaces carry role=group + menuitemradio/
checkbox with 0 orphaned options, and the sort direction segment exposes
aria-pressed. Dropped two review findings (max-width duplication, pill-class
reuse) as refuted on adversarial verification. Gate green: tsc, 372 vitest,
prettier, eslint (0 err).
2026-08-21 22:41:57 +02:00
peter edbc9dede5 style(header): centre the search-box group over the page content
The floating header band already centres on --content-mw (the width the
page's cards use), but its search box + scope-toggle/sync-chip group was
packed to the left edge (default justify-start), leaving it offset from
the centred card grid on max-width pages (Plex/Channels/Playlists).

Add justify-center to the band so the group centres within it: once the
column is wider than the box's max-w, the box sits centred over the
cards instead of hugging the left edge. Shared header row, so it applies
to every module with a SearchBar. Verified at 1920px on Plex: group
centre == content-band centre (431px margin each side).
2026-08-21 22:08:23 +02:00
peter 53ca4e2849 feat(layout): a pinned filter rail no longer pushes content when there's room for it
On a page that centres its content at a max width, if the window is wide enough for the rail to fit
in the content's LEFT margin without overlapping it, pinning the rail now leaves it OVERLAY — the
content doesn't shift. It reverts to the old in-flow "push" only when there isn't room: a full-width
page (feed) or a window too narrow for the rail to clear the centred content.

PageShell computes --rail-pinned-w from the page's maxWidth vs the viewport (innerWidth >= maxWidth +
~576px → overlay 2.25rem, else in-flow 16.75rem) on a resize listener; the SidePanel slot reads it.

Verified at 1920px: pinning the playlists rail leaves the table at left 487 (moved 0); the rail sits
256px wide ending at 312 — 175px clear of the content. Feed (full width) still pins in-flow.
2026-08-21 21:53:24 +02:00
peter 1801a58f6b feat(layout): centre the header on each page's content width, centre the playlist table
Every page's floating header (search bar + sync chip) and the toolbar band now centre on the SAME
max-width the content column uses, instead of spanning the whole column while a max-w page centres
its grid. So the search bar lines up with the left edge of the cards/table on every page, and sits
centred with the content — not against the full page width.

- PageShell takes a `maxWidth`, published as --content-mw; the header adds max-w-[--content-mw] mx-auto
  inside its left-4 / right-(1rem+scrollbar) span, so it centres on that width (`none` = full width).
- moduleRegistry PAGE_CONTENT carries each page's content max width (plex 1600px, channels 96rem,
  playlists 64rem, downloads 56rem, notifications/users 48rem; feed stays full width) — one source.
- Playlists: its content (header, filter row, table) was left-aligned max-w-5xl → now mx-auto, so the
  table is centred and the header lines up with it.

Verified at 1920px: header left edge == content left edge — feed 340, channels 213, playlists 487.
2026-08-21 21:38:49 +02:00
peter 011acd919e feat(toolbar): lead the Plex toolbar with ItemCount too, unify its gaps
Plex browse now uses the same fixed-width ItemCount pill as the feed (its title count → "N items"),
placed as the leading element, and the toolbar gap tightened 2 → 1.5 to match. One consistent
toolbar shape across the source modules.
2026-08-21 20:37:36 +02:00
peter 45d8b47366 feat(toolbar): fixed-width ItemCount pill, lead the toolbar with it, align chrome to the cards
- New ItemCount: a generic, FIXED-WIDTH count pill (glass, non-interactive) — "192,050 items"
  (neutral noun, dynamic plural), min-w-[9rem] so a changing number never shifts the controls
  beside it; a million or more collapses to "1M+" with the exact figure in the title tooltip.
- Feed toolbar: ItemCount is the fixed leading element now; dropped the mx-1 separators so every
  gap is a uniform 1.5 (Content-types↔Source used to be wider than Watch-state↔Content-types).
- Chrome-to-card alignment (app-wide, via the shared Header + PageShell): the floating header and
  the toolbar band now span exactly the card grid's width — left-4, and a right edge of
  1rem + the scrollbar gutter (measured once into --sbw), so the fixed chrome lines up with the
  scrolled cards instead of running ~15px past them. Wraps cleanly to multiple rows on mobile.

Renames the count testid feed-result-count → item-count (E2Es updated).
2026-08-21 20:33:55 +02:00
peter c3820a0b44 refactor(header): remove the ModuleName module-switcher pill
The user no longer wants the header's "FEED" pill (module label + accent dot + ◀/▶ module-step
arrows). Drop it everywhere — the header now starts with the module's SearchBar; module switching
stays on the nav rail.

Deletes ModuleName.tsx, the now-unused stepModule() in lib/modules, and the Header's
module-order/label/setPage plumbing that only fed it.
2026-08-21 19:58:19 +02:00
peter c7b393901f style(type): correct the trigger text from 16px back to the view-switcher's real 14px
The earlier bump measured the view-switcher's BUTTON (16px, inherited) instead of its visible
label span (text-sm = 14px), and 14px is the size the user actually finds comfortable. Walk the
over-sized triggers back to text-sm: FilterSelect, SortControl, ViewSwitcher, and the Combobox
input (which was 14px before the bump anyway). The secondary text already at text-sm/text-xs
(count, chips, widget headers, section labels) was correct and stays.
2026-08-21 19:53:14 +02:00
peter f1e14c94c5 style(viewswitcher): unify the ViewSwitcher trigger with the FilterSelect pill
The user prefers FilterSelect's rounded-full glass pill over the ViewSwitcher's squarer
rounded-lg trigger. Give the ViewSwitcher trigger the same look — glass-card/glass-hover,
rounded-full, px-3 py-1, open→border-accent, and a 16px label (was text-sm) — so the two read
as one control family. Applies to every ViewSwitcher consumer (feed, channel manager, Plex
playlists).

Native <select>s, the Combobox input box, and playlist/list cards are a different control class
(form fields / variable-height inputs / cards) and keep their own shape.
2026-08-21 19:49:10 +02:00
peter ddb171fca3 style(type): raise the too-small toolbar and filter text toward the view-switcher size
The view-switcher's 16px reads comfortably; the toolbar and filter controls sat at 12px. Bring
the marked controls up to match, and lift the general scale on the shared filter surfaces.

- Toolbar (feed + channel + Plex, via the shared components): FilterSelect and SortControl trigger
  pills text-xs → text-base (16px, the view-switcher size); the result count → text-sm; the
  active-filter chips → text-sm with the "FILTERED BY" label text-2xs → text-xs.
- Filterbar: the Combobox picker input → text-base (matching the toolbar pills) + its chips → text-sm;
  the panel/widget headers (SidePanel collapsed-rail + panel title, PanelGroup widget titles) up one step.

Channel info and the manager tables were already at 14–16px (readable), so they were left as-is.
2026-08-21 19:37:11 +02:00
peter ec3bbc78dc style(layout): tighten the oversized vertical gaps around the page toolbars
The floating-header → toolbar → filter-chips → content stack carried ~2× the whitespace it
needed. Halve it, app-wide where the chrome is shared:
- PageShell: the --hdr-h gap below the floating header 8px → 4px (every page's header).
- Feed: toolbar wrapper pt-4 → pt-2, the toolbar's own pb-3 → pb-1.5, and the grid's top pt-2
  → pt-1. Measured on the feed: header→toolbar 24→12px, toolbar→chips 12→6px, chips→cards 12→8px.
- Plex / Playlists / ChannelDiscovery / DownloadCenter / Notifications / AdminUsers: toolbar
  wrapper pt-3 → pt-2, so every page's top band matches the tighter feed.

PageShell.dom.test updated for the 4px gap.
2026-08-21 19:13:38 +02:00
peter 1be1dc9d06 feat(sort): give SortControl the FilterSelect pill look with a footer direction segment
Rebuilds SortControl's internals (API unchanged, so all five consumers — feed, Plex,
playlists, and the two table/card ColumnSortControl sites — get it for free) as a labeled
"Label: key ↓ ▾" pill on the AnchoredPopover platform, matching FilterSelect. The listbox
lists the sort keys; an asc/desc segment sits in the panel footer (with the trailing slot —
feed reshuffle, playlists dirty-first — beside it).

Two-dimensional key+direction, so it stays its own component rather than reusing FilterSelect:
the trigger is a single plain button (no nested button), picking a key keeps the panel open so
key AND direction can be set in one visit, and picking "no sort" (tables' defaultLabel) closes
it. Directionless keys (shuffle) hide the segment and drop the trigger arrow.

Drops the now-redundant external "Sort" label spans in Feed and PlexBrowse (the trigger carries
the label now). Rewrites SortControl.dom.test for the popover UI.
2026-08-21 18:11:45 +02:00
peter 6109e0c6f1 feat(feed): convert the Source filter to the FilterSelect dropdown
Replaces the native <select> "Source" control with a labeled FilterSelect pill, matching
the Watch state / Content types pills — one consistent toolbar idiom. Single-select over
the three librarySource values (organic / all / search-only).

Sort and View are intentionally left as-is: Sort is a two-dimensional key+direction control
(plus reshuffle and a tables-only "no sort") shared across five surfaces, and View is already
on the same AnchoredPopover platform with an icon-only trigger tuned for this packed toolbar —
FilterSelect's text trigger would regress it. Neither fits FilterSelect's one-dimensional model.
2026-08-21 17:36:33 +02:00
peter 5851ddd329 feat(feed): compress the toolbar chip groups into labeled FilterSelect dropdowns
Adds a reusable FilterSelect: a "Label: value ▾" pill that opens a listbox (a checkmark
row for single-select, a checkbox row for multi), on the AnchoredPopover platform. It
replaces the two feed-toolbar chip walls:
  - Watch state (single-select, default All) — picking a row replaces + closes.
  - Content types (multi-select, default Normal; requireOne keeps at least one on) — rows
    toggle and the panel stays open.
The trigger summarises multi picks as "First +N". The component is source-agnostic, meant
for reuse anywhere a small known option set currently renders as a chip group.

FilterSelect takes a `testId` → the trigger gets it, each option gets `${testId}-${value}`,
so the feed/channel E2Es keep stable selectors. They now open the dropdown and assert the
option's aria-selected instead of the old inline aria-pressed chips (new expectContentType/
toggleContentType helpers).
2026-08-21 17:28:57 +02:00
peter ad0cf5827b feat(feed): use the Combobox picker for Channel/Your tags/Topic/Language filters
Replaces the hand-rolled channel search list and the flat tag chip walls with the
searchable, ARIA-correct Combobox built for the Plex filter bar (E5), so the YT feed
filters get the same input+listbox+chips UX with facet counts.

- ChannelPicker now wraps Combobox (selectedLabels keeps picks named); drops its bespoke
  input + scroll-fade list.
- Sidebar language/topic/tags: each Combobox owns only its slice of the shared
  filters.tags list (setTagSelection replaces that list's ids, leaves the rest). Topic
  keeps its any/all match toggle, Your tags keeps Manage. TagChip removed.
2026-08-21 17:01:41 +02:00
peter f6192aa6bf fix(feed): re-measure header on feed↔channel Back so --hdr-h can't stick at 8px
PageShell is reused (never remounted) across feed↔channel navigation — same
component position in App.tsx — so a one-shot useLayoutEffect kept its
ResizeObserver bound to the FIRST header node. On channel-open that node detached
and fired the observer with a 0 box, rewriting --hdr-h to 0+0+8=8px on the
still-mounted content div. Back restored the pt-(--hdr-h) pad class with that
stale value, sliding the content toolbar up under the header — stuck until reload.

Hold both the header and content nodes in state (callback refs) and key the
measurement effect to the live nodes, so it re-runs every time the header
mounts/unmounts; drop the stale pad when the header is absent (channel page).

Adds a regression test: header removed then restored must re-measure, not 8px.
2026-08-21 16:51:05 +02:00
429 changed files with 19621 additions and 4006 deletions
+3
View File
@@ -4,3 +4,6 @@
#
# Prettier one-time repo-wide format pass (R2 S1b, 2026-07-21)
51c36c2071728d2cd7ae580b7bfdf21f6867a347
# ruff format one-time backend + scripts pass (line-length 100, 2026-08-23)
b1832afd180866b922521148603db447020c57a4
+2
View File
@@ -59,6 +59,8 @@ jobs:
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
- name: ruff
run: python -m ruff check --config backend/ruff.toml backend scripts
- name: ruff format --check
run: python -m ruff format --check --config backend/ruff.toml backend scripts
- name: pytest
run: python -m pytest
working-directory: backend
+30 -8
View File
@@ -12,12 +12,12 @@ provider.
## The gate — never merge or ship red
The local gate is **ruff + typecheck (3 tsc projects) + eslint + prettier --check + knip + vitest +
pytest**. It runs on every `publish`, which **refuses to build on failure — there is deliberately no
The local gate is **ruff (`check` + `format --check`) + typecheck (3 tsc projects) + eslint + prettier
--check + knip + vitest + pytest**. It runs on every `publish`, which **refuses to build on failure — there is deliberately no
`--skip` flag.** Run it and get it green **before** any merge, release, or handing work to review.
- Full gate (agents with the private ops dispatcher): `siftlode check`.
- Directly, without the dispatcher: `cd frontend && npm run typecheck && npm run lint && npm run knip && npm run test`, plus backend `ruff check backend` and `pytest` (the DB-lane needs Postgres).
- Directly, without the dispatcher: `cd frontend && npm run typecheck && npm run lint && npm run knip && npm run test`, plus backend `ruff check backend` + `ruff format --check backend` and `pytest` (the DB-lane needs Postgres).
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
## Workflow
@@ -26,12 +26,33 @@ pytest**. It runs on every `publish`, which **refuses to build on failure — th
keep commits logical and Conventional-Commit-styled.
- **No AI attribution in commits.** No `Co-Authored-By` trailer, no "generated by" lines. (This file
being AI-written is the intended, visible exception; commits are not.)
- **Release is a deliberate, user-owned step** (`siftlode release` from the feature branch: merge to
`main` → publish → deploy → verify `/api/version`). Do not self-ship; bring work to a green,
committed state and stop for the user's go-ahead.
- **Release: user-owned in timing, agent-run in execution.** The user decides *when* to ship; on their
go-ahead *you* run `siftlode release` from the feature branch (merge to `main` → publish → deploy →
verify `/api/version`). Never self-ship — bring work to a green, committed state and stop for the
go-ahead first, then run it yourself.
- **Public repo.** Keep ops/topology/secrets out — those live in a separate private repo. Nothing here
should assume or leak them.
## Collaboration — how to work with the human
The human is the product owner: they bring ideas and requirements. You are the technical lead — you own
how it gets built. Concretely:
- **Plan before you build.** For anything non-trivial, present a short plan (sprint-level for larger
work) and get a go-ahead before writing code.
- **Offer choices, mark the optimal — and explain the trade-off.** When several solutions are viable,
list them and mark the recommended one. This is the *one* place to spend extra words: give the *why*
behind each option and the weight of the decision. These decision points are where the product owner
engages with the build and learns from it, so make them understandable — a reasoned menu, not a bare one.
- **Mockups for design/UX decisions.** For any visual or interaction choice, show mockup options to
choose from before building.
- **Challenge unsound requests.** If a request is technically ill-advised and you can back it with real
arguments, push back or propose an alternative — don't comply blindly.
- **Verify UI in the browser before calling it done.** tsc/unit/DOM tests miss visual and interaction
bugs; do a real visual check of any UI change.
- **Be terse everywhere else.** Outside those decision points, lead with the answer or decision. During
implementation and debugging especially: no restating the same point reworded, no filler.
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
@@ -79,5 +100,6 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
---
*A fuller written-conventions doc (`docs/development.md`) is planned; once it lands, this file slims to a
pointer plus the gate/workflow/no-attribution rules. Until then, the table above is the source of truth.*
*Run-it-locally and the full gate commands live in [`docs/development.md`](docs/development.md); backend
code conventions in [`backend/app/README.md`](backend/app/README.md). This file stays the source of
truth for the canonical-patterns table and the gate/workflow/collaboration rules above.*
+1 -1
View File
@@ -1 +1 @@
0.71.0
0.76.0
+1
View File
@@ -4,6 +4,7 @@ Revision ID: 0001_initial
Revises:
Create Date: 2026-06-11
"""
from typing import Sequence, Union
from alembic import op
@@ -4,6 +4,7 @@ Revision ID: 0002_ingest_core
Revises: 0001_initial
Create Date: 2026-06-11
"""
from typing import Sequence, Union
from alembic import op
+1
View File
@@ -4,6 +4,7 @@ Revision ID: 0003_tags
Revises: 0002_ingest_core
Create Date: 2026-06-11
"""
from typing import Sequence, Union
from alembic import op
@@ -4,6 +4,7 @@ Revision ID: 0004_feed_state_prefs
Revises: 0003_tags
Create Date: 2026-06-11
"""
from typing import Sequence, Union
from alembic import op
@@ -4,6 +4,7 @@ Revision ID: 0005_shorts_probed
Revises: 0004_feed_state_prefs
Create Date: 2026-06-11
"""
from typing import Sequence, Union
from alembic import op
@@ -4,6 +4,7 @@ Revision ID: 0006_app_state
Revises: 0005_shorts_probed
Create Date: 2026-06-11
"""
from typing import Sequence, Union
from alembic import op
@@ -4,6 +4,7 @@ Revision ID: 0007_deep_requested
Revises: 0006_app_state
Create Date: 2026-06-11
"""
from typing import Sequence, Union
from alembic import op
+2 -3
View File
@@ -4,6 +4,7 @@ Revision ID: 0008_invites
Revises: 0007_deep_requested
Create Date: 2026-06-12
"""
from typing import Sequence, Union
from alembic import op
@@ -22,9 +23,7 @@ def upgrade() -> None:
"invites",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("email", sa.String(length=320), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False, server_default="pending"
),
sa.Column("status", sa.String(length=16), nullable=False, server_default="pending"),
sa.Column("note", sa.Text(), nullable=True),
sa.Column(
"requested_at",
@@ -4,6 +4,7 @@ Revision ID: 0009_quota_events
Revises: 0008_invites
Create Date: 2026-06-12
"""
from typing import Sequence, Union
from alembic import op
@@ -4,6 +4,7 @@ Revision ID: 0010_watch_progress
Revises: 0009_quota_events
Create Date: 2026-06-14
"""
from typing import Sequence, Union
from alembic import op
+4 -9
View File
@@ -4,6 +4,7 @@ Revision ID: 0011_playlists
Revises: 0010_watch_progress
Create Date: 2026-06-15
"""
from typing import Sequence, Union
from alembic import op
@@ -26,16 +27,10 @@ def upgrade() -> None:
nullable=False,
),
sa.Column("name", sa.String(length=255), nullable=False),
sa.Column(
"kind", sa.String(length=16), nullable=False, server_default="user"
),
sa.Column(
"source", sa.String(length=16), nullable=False, server_default="local"
),
sa.Column("kind", sa.String(length=16), nullable=False, server_default="user"),
sa.Column("source", sa.String(length=16), nullable=False, server_default="local"),
sa.Column("yt_playlist_id", sa.String(length=64), nullable=True),
sa.Column(
"dirty", sa.Boolean(), nullable=False, server_default="false"
),
sa.Column("dirty", sa.Boolean(), nullable=False, server_default="false"),
sa.Column("position", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"created_at",
@@ -7,6 +7,7 @@ Create Date: 2026-06-15
Migrates the old per-video status='saved' into membership of a built-in 'watch_later'
playlist (one per user who had saved videos), then demotes those states to 'new'.
"""
from typing import Sequence, Union
from alembic import op
@@ -11,6 +11,7 @@ a YouTube read. Backfills the baseline for already-clean linked playlists (the c
state == the synced state for them); dirty ones are left NULL so they stay dirty until the
next real sync establishes a baseline.
"""
import hashlib
from typing import Sequence, Union
@@ -32,24 +33,18 @@ def _fingerprint(name: str, ids: list[str]) -> str:
def upgrade() -> None:
op.add_column(
"playlists", sa.Column("synced_fingerprint", sa.String(), nullable=True)
)
op.add_column("playlists", sa.Column("synced_fingerprint", sa.String(), nullable=True))
bind = op.get_bind()
# Backfill clean, YouTube-linked playlists: their current state IS the synced state.
rows = bind.execute(
sa.text(
"SELECT id, name FROM playlists "
"WHERE yt_playlist_id IS NOT NULL AND dirty = false"
)
sa.text("SELECT id, name FROM playlists WHERE yt_playlist_id IS NOT NULL AND dirty = false")
).fetchall()
for pid, name in rows:
ids = [
r[0]
for r in bind.execute(
sa.text(
"SELECT video_id FROM playlist_items "
"WHERE playlist_id = :pid ORDER BY position"
"SELECT video_id FROM playlist_items WHERE playlist_id = :pid ORDER BY position"
),
{"pid": pid},
).fetchall()
@@ -11,6 +11,7 @@ Adds the shared demo-account plumbing:
The demo user row itself is created lazily on first demo login (works across all three DBs
without a data migration), so this only adds the schema.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -47,9 +48,7 @@ def upgrade() -> None:
nullable=False,
),
)
op.create_index(
"ix_demo_whitelist_email", "demo_whitelist", ["email"], unique=True
)
op.create_index("ix_demo_whitelist_email", "demo_whitelist", ["email"], unique=True)
def downgrade() -> None:
@@ -7,6 +7,7 @@ Create Date: 2026-06-16
Adds scheduler_settings: a per-job run-interval override (minutes), editable from the admin
Scheduler dashboard. Absent row = use the env/config default, so this is purely additive.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -10,6 +10,7 @@ the validation lifecycle (last_checked_at for rolling re-validation; unavailable
unavailable_reason to flag a video as currently unplayable — hidden from the feed
immediately, hard-deleted after a grace period). Purely additive; all columns nullable.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -26,7 +27,9 @@ def upgrade() -> None:
op.add_column("videos", sa.Column("privacy_status", sa.String(length=16), nullable=True))
op.add_column("videos", sa.Column("upload_status", sa.String(length=16), nullable=True))
op.add_column("videos", sa.Column("last_checked_at", sa.DateTime(timezone=True), nullable=True))
op.add_column("videos", sa.Column("unavailable_since", sa.DateTime(timezone=True), nullable=True))
op.add_column(
"videos", sa.Column("unavailable_since", sa.DateTime(timezone=True), nullable=True)
)
op.add_column("videos", sa.Column("unavailable_reason", sa.String(length=24), nullable=True))
op.create_index("ix_videos_last_checked_at", "videos", ["last_checked_at"])
op.create_index("ix_videos_unavailable_since", "videos", ["unavailable_since"])
@@ -9,6 +9,7 @@ JSON payload column is named `data` (SQLAlchemy's declarative Base reserves `met
First producer is the maintenance job's batched "videos removed" notice. Coexists with the
existing client-side transient bell.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -36,18 +37,14 @@ def upgrade() -> None:
sa.Column("data", sa.JSON(), nullable=True),
sa.Column("read", sa.Boolean(), nullable=False, server_default="false"),
sa.Column("dismissed", sa.Boolean(), nullable=False, server_default="false"),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now()
),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
)
op.create_index("ix_notifications_user_id", "notifications", ["user_id"])
op.create_index("ix_notifications_type", "notifications", ["type"])
op.create_index("ix_notifications_read", "notifications", ["read"])
op.create_index("ix_notifications_created_at", "notifications", ["created_at"])
# Fast unread-badge count + inbox listing per user.
op.create_index(
"ix_notifications_user_read", "notifications", ["user_id", "read"]
)
op.create_index("ix_notifications_user_read", "notifications", ["user_id", "read"])
def downgrade() -> None:
@@ -8,6 +8,7 @@ Adds app_state.maintenance_revalidate_batch: an admin-tunable override (from the
dashboard) for how many videos the maintenance job re-validates per run. NULL = use the
env/config default. Purely additive.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -9,6 +9,7 @@ channels.list?mine=true and cached. Used to exclude the user's own channel from
playlist-discovery list (you don't subscribe to yourself). NULL = not resolved yet.
Purely additive.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -10,6 +10,7 @@ stored historical values to the canonical scheme defined in app.quota.QuotaActio
fully reversible. (Scheduler job ids and progress phase labels are a separate namespace and
are NOT touched.)
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -8,6 +8,7 @@ Adds the system_config table: admin overrides for operational config that otherw
from env/config defaults (see app.sysconfig for the registry of known keys). Secret values
are stored Fernet-encrypted (encrypted flag). Purely additive; absent rows = use defaults.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -9,6 +9,7 @@ google_sub nullable (password-only accounts have no Google sub until they link),
single-use auth_tokens table for email verification + password reset. Existing rows are all
Google/demo accounts → backfilled email_verified=true (is_active defaults true).
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -8,6 +8,7 @@ Adds users.is_suspended — an admin-controlled access block, distinct from is_a
approval lifecycle). A suspended account can't sign in (password or Google) and any live
session is rejected; the user is told why and given the operator's contact.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -9,6 +9,7 @@ email_verified=false (the callback didn't set it). Backfill those so the flag re
matters because password-login gates on email_verified, so a Google user who later adds a password
must count as verified. Demo/password-only rows (google_sub IS NULL) are left untouched.
"""
from typing import Sequence, Union
from alembic import op
@@ -8,6 +8,7 @@ Adds app_state.configured + setup_token_hash for the first-run install wizard. E
(any users present) are marked configured so they never see the wizard; a genuinely fresh DB (no
users) starts unconfigured and boots into setup mode.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -8,6 +8,7 @@ Adds the `messages` table for direct user-to-user messaging (notification module
A conversation is the set of rows between two users; both FKs cascade so a GDPR account
erasure removes that user's messages.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -12,6 +12,7 @@ hold each user's public key + passphrase-wrapped private key (the server can nev
Any pre-existing rows were plaintext with no keys, so they're dropped (the feature is unshipped;
prod has none).
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -11,6 +11,7 @@ the Library by default:
- channels.from_search — this channel exists only because search surfaced one of its videos
Both default false (every existing row is organic), indexed for the feed's exclude filter.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -25,16 +26,12 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"videos",
sa.Column(
"via_search", sa.Boolean(), nullable=False, server_default="false"
),
sa.Column("via_search", sa.Boolean(), nullable=False, server_default="false"),
)
op.create_index("ix_videos_via_search", "videos", ["via_search"])
op.add_column(
"channels",
sa.Column(
"from_search", sa.Boolean(), nullable=False, server_default="false"
),
sa.Column("from_search", sa.Boolean(), nullable=False, server_default="false"),
)
op.create_index("ix_channels_from_search", "channels", ["from_search"])
@@ -10,6 +10,7 @@ query wraps both sides in unaccent(); this migration just makes the function ava
`unaccent` ships with the official postgres image (postgresql-contrib). The DB role is a
superuser in our deployments, so CREATE EXTENSION succeeds; IF NOT EXISTS keeps it idempotent.
"""
from typing import Sequence, Union
from alembic import op
@@ -9,6 +9,7 @@ live YouTube search. This lets "videos I searched for" appear in that user's Min
Mine Source filter), independent of the global `videos.via_search` flag (which marks a video as
search-discovered by anyone, for the shared Library's Source filter).
"""
from typing import Sequence, Union
import sqlalchemy as sa
+2 -3
View File
@@ -15,6 +15,7 @@ search stays accent-insensitive ("tiesto" matches "Tiësto"). A GIN expression i
query uses the exact same expression so the planner picks the index. `unaccent` is already
enabled (migration 0029). The config-based 2-arg to_tsvector is IMMUTABLE, so it's index-safe.
"""
from typing import Sequence, Union
from alembic import op
@@ -26,9 +27,7 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"CREATE TEXT SEARCH CONFIGURATION public.unaccent_simple (COPY = pg_catalog.simple)"
)
op.execute("CREATE TEXT SEARCH CONFIGURATION public.unaccent_simple (COPY = pg_catalog.simple)")
op.execute(
"""
ALTER TEXT SEARCH CONFIGURATION public.unaccent_simple
@@ -21,6 +21,7 @@ title-only expression index from 0031 is dropped (superseded). Existing rows get
description indexed immediately; keywords/search_terms fill in as videos are (re-)enriched and
searched. Uses the `unaccent_simple` config from 0031 (accent-insensitive).
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -48,9 +49,7 @@ def upgrade() -> None:
f"ALTER TABLE videos ADD COLUMN search_vector tsvector "
f"GENERATED ALWAYS AS ({_SEARCH_VECTOR_EXPR}) STORED"
)
op.execute(
"CREATE INDEX ix_videos_search_vector ON videos USING gin (search_vector)"
)
op.execute("CREATE INDEX ix_videos_search_vector ON videos USING gin (search_vector)")
def downgrade() -> None:
@@ -17,6 +17,7 @@ un-subscribed channels):
access to its ephemeral videos and drives the grace-period purge of un-kept channels.
All new flags default false / NULL, so every existing row stays organic.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -30,9 +31,7 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("channels", sa.Column("total_view_count", sa.BigInteger(), nullable=True))
op.add_column(
"channels", sa.Column("published_at", sa.DateTime(timezone=True), nullable=True)
)
op.add_column("channels", sa.Column("published_at", sa.DateTime(timezone=True), nullable=True))
op.add_column("channels", sa.Column("banner_url", sa.String(length=1024), nullable=True))
op.add_column("channels", sa.Column("external_links", sa.JSON(), nullable=True))
op.add_column(
@@ -8,6 +8,7 @@ Adds the `blocked_channels` table: a per-user blocklist. Videos from a blocked c
never shown in that user's live YouTube search (and not ingested), and are hidden from their
feed / explore views. Per-user, so one user's block doesn't affect anyone else.
"""
from typing import Sequence, Union
import sqlalchemy as sa
+2 -3
View File
@@ -8,6 +8,7 @@ Adds the `saved_views` table: per-user named snapshots of the feed's filter/sort
row stores the serialized FeedFilters (JSON), a sidebar `position` for drag-reorder, and an
`is_default` flag (at most one per user, enforced by the API) for the view applied on load.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -26,9 +27,7 @@ def upgrade() -> None:
sa.Column("user_id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(length=80), nullable=False),
sa.Column("filters", sa.JSON(), nullable=False),
sa.Column(
"position", sa.Integer(), server_default=sa.text("0"), nullable=False
),
sa.Column("position", sa.Integer(), server_default=sa.text("0"), nullable=False),
sa.Column(
"is_default",
sa.Boolean(),
@@ -10,6 +10,7 @@ FOR UPDATE SKIP LOCKED, live progress written back for polling); `download_profi
reusable format presets (builtins seeded in the next migration); `download_shares` grant a file
to another user; `download_quotas` hold admin-set per-user limits (default via sysconfig).
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -47,9 +48,16 @@ def upgrade() -> None:
sa.Column("ref_count", sa.Integer(), server_default="0", nullable=False),
sa.Column("last_access_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("source_kind", "source_ref", "format_sig", name="uq_media_asset_cache_key"),
sa.UniqueConstraint(
"source_kind", "source_ref", "format_sig", name="uq_media_asset_cache_key"
),
)
op.create_index("ix_media_assets_status", "media_assets", ["status"])
@@ -61,7 +69,12 @@ def upgrade() -> None:
sa.Column("spec", sa.JSON(), nullable=False),
sa.Column("is_builtin", sa.Boolean(), server_default="false", nullable=False),
sa.Column("sort_order", sa.Integer(), server_default="0", nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
)
@@ -84,8 +97,18 @@ def upgrade() -> None:
sa.Column("phase", sa.String(length=32), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("queue_pos", sa.Integer(), server_default="0", nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["asset_id"], ["media_assets.id"], ondelete="SET NULL"),
sa.ForeignKeyConstraint(["profile_id"], ["download_profiles.id"], ondelete="SET NULL"),
@@ -102,7 +125,12 @@ def upgrade() -> None:
sa.Column("job_id", sa.Integer(), nullable=False),
sa.Column("from_user_id", sa.Integer(), nullable=False),
sa.Column("to_user_id", sa.Integer(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.ForeignKeyConstraint(["job_id"], ["download_jobs.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["from_user_id"], ["users.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["to_user_id"], ["users.id"], ondelete="CASCADE"),
@@ -120,7 +148,12 @@ def upgrade() -> None:
sa.Column("max_concurrent", sa.Integer(), nullable=False),
sa.Column("max_jobs", sa.Integer(), nullable=False),
sa.Column("unlimited", sa.Boolean(), server_default="false", nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("user_id"),
)
@@ -10,6 +10,7 @@ definition consumed by app.downloads.formats: mode ("av" audio+video / "v" video
the embed_*/sponsorblock postprocessor flags. Names are English (the UI default); the frontend
localizes builtin labels by a stable key derived from the name.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -7,6 +7,7 @@ Create Date: 2026-07-03
The download GC warns each holder once, a grace window before an asset's TTL expiry; this flag
records that so subsequent GC runs don't re-notify until the asset is deleted.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -9,6 +9,7 @@ display-friendly form (emoji stripped, ALL-CAPS de-shouted, trailing SEO hashtag
see app.titles). Reversible: original_title preserves the source, and title can be re-derived.
The generated search_vector column regenerates automatically as each title is updated.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -45,5 +46,7 @@ def upgrade() -> None:
def downgrade() -> None:
conn = op.get_bind()
conn.execute(sa.text("UPDATE videos SET title = original_title WHERE original_title IS NOT NULL"))
conn.execute(
sa.text("UPDATE videos SET title = original_title WHERE original_title IS NOT NULL")
)
op.drop_column("videos", "original_title")
@@ -11,6 +11,7 @@ rows so both fresh installs (0037 then this) and existing DBs converge on the sa
jobs are unaffected (they snapshot their spec; profile_id just SET NULLs if it pointed at a
replaced row).
"""
from typing import Sequence, Union
import sqlalchemy as sa
+6 -15
View File
@@ -10,6 +10,7 @@ still hangs off a `media_assets` row (with `source_kind='edit'` and a per-user c
existing file-serve / ref-count / GC / quota machinery is reused unchanged. `source_job_id` /
`source_asset_id` point at the parent download; `edit_spec` is the trim/crop recipe.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -24,16 +25,10 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"download_jobs",
sa.Column(
"job_kind", sa.String(length=16), server_default="download", nullable=False
),
)
op.add_column(
"download_jobs", sa.Column("source_job_id", sa.Integer(), nullable=True)
)
op.add_column(
"download_jobs", sa.Column("source_asset_id", sa.Integer(), nullable=True)
sa.Column("job_kind", sa.String(length=16), server_default="download", nullable=False),
)
op.add_column("download_jobs", sa.Column("source_job_id", sa.Integer(), nullable=True))
op.add_column("download_jobs", sa.Column("source_asset_id", sa.Integer(), nullable=True))
op.add_column("download_jobs", sa.Column("edit_spec", sa.JSON(), nullable=True))
op.create_foreign_key(
"fk_download_jobs_source_job",
@@ -54,12 +49,8 @@ def upgrade() -> None:
def downgrade() -> None:
op.drop_constraint(
"fk_download_jobs_source_asset", "download_jobs", type_="foreignkey"
)
op.drop_constraint(
"fk_download_jobs_source_job", "download_jobs", type_="foreignkey"
)
op.drop_constraint("fk_download_jobs_source_asset", "download_jobs", type_="foreignkey")
op.drop_constraint("fk_download_jobs_source_job", "download_jobs", type_="foreignkey")
op.drop_column("download_jobs", "edit_spec")
op.drop_column("download_jobs", "source_asset_id")
op.drop_column("download_jobs", "source_job_id")
@@ -9,6 +9,7 @@ can watch the file on the public `/watch/{token}` player page — no account nee
per-link controls: an expiry, an "allow download" toggle, and an (argon2-hashed) password. Revoke
= delete the row. Distinct from `download_shares` (an ACL grant to another *registered* user).
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -31,7 +32,12 @@ def upgrade() -> None:
sa.Column("password_hash", sa.Text(), nullable=True),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("view_count", sa.Integer(), server_default="0", nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.ForeignKeyConstraint(["job_id"], ["download_jobs.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["created_by"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
@@ -8,6 +8,7 @@ Stores yt-dlp's canonical `webpage_url` for a download's source, so the Download
clean "downloaded from" reference (copyable + openable). Nullable — filled by the worker on
extraction; existing/queued rows fall back to a URL derived from source_kind+source_ref.
"""
from typing import Sequence, Union
import sqlalchemy as sa
+67 -16
View File
@@ -18,6 +18,7 @@ Plex server (playback is from the local physical file; Plex is metadata only). T
The generated `search_vector` columns + their GIN indexes are added via raw SQL (same approach
as migration 0032), reusing the `unaccent_simple` text-search config from migration 0031.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -52,8 +53,12 @@ def upgrade() -> None:
sa.Column("kind", sa.String(length=16), nullable=False),
sa.Column("enabled", sa.Boolean(), server_default="true", nullable=False),
sa.Column("synced_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index("ix_plex_libraries_plex_key", "plex_libraries", ["plex_key"], unique=True)
@@ -61,7 +66,12 @@ def upgrade() -> None:
"plex_shows",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("rating_key", sa.String(length=32), nullable=False),
sa.Column("library_id", sa.Integer(), sa.ForeignKey("plex_libraries.id", ondelete="CASCADE"), nullable=False),
sa.Column(
"library_id",
sa.Integer(),
sa.ForeignKey("plex_libraries.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("title", sa.String(length=512), nullable=False),
sa.Column("summary", sa.Text(), nullable=True),
sa.Column("year", sa.Integer(), nullable=True),
@@ -69,8 +79,12 @@ def upgrade() -> None:
sa.Column("art_key", sa.String(length=512), nullable=True),
sa.Column("child_count", sa.Integer(), nullable=True),
sa.Column("added_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index("ix_plex_shows_rating_key", "plex_shows", ["rating_key"], unique=True)
op.create_index("ix_plex_shows_library_id", "plex_shows", ["library_id"])
@@ -81,12 +95,21 @@ def upgrade() -> None:
"plex_seasons",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("rating_key", sa.String(length=32), nullable=False),
sa.Column("show_id", sa.Integer(), sa.ForeignKey("plex_shows.id", ondelete="CASCADE"), nullable=False),
sa.Column(
"show_id",
sa.Integer(),
sa.ForeignKey("plex_shows.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("title", sa.String(length=255), nullable=True),
sa.Column("season_number", sa.Integer(), nullable=True),
sa.Column("thumb_key", sa.String(length=512), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index("ix_plex_seasons_rating_key", "plex_seasons", ["rating_key"], unique=True)
op.create_index("ix_plex_seasons_show_id", "plex_seasons", ["show_id"])
@@ -96,10 +119,25 @@ def upgrade() -> None:
"plex_items",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("rating_key", sa.String(length=32), nullable=False),
sa.Column("library_id", sa.Integer(), sa.ForeignKey("plex_libraries.id", ondelete="CASCADE"), nullable=False),
sa.Column(
"library_id",
sa.Integer(),
sa.ForeignKey("plex_libraries.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("kind", sa.String(length=16), nullable=False),
sa.Column("show_id", sa.Integer(), sa.ForeignKey("plex_shows.id", ondelete="CASCADE"), nullable=True),
sa.Column("season_id", sa.Integer(), sa.ForeignKey("plex_seasons.id", ondelete="CASCADE"), nullable=True),
sa.Column(
"show_id",
sa.Integer(),
sa.ForeignKey("plex_shows.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column(
"season_id",
sa.Integer(),
sa.ForeignKey("plex_seasons.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column("season_number", sa.Integer(), nullable=True),
sa.Column("episode_number", sa.Integer(), nullable=True),
sa.Column("title", sa.String(length=512), nullable=False),
@@ -119,8 +157,12 @@ def upgrade() -> None:
sa.Column("probed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("markers", sa.JSON(), nullable=True),
sa.Column("added_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index("ix_plex_items_rating_key", "plex_items", ["rating_key"], unique=True)
op.create_index("ix_plex_items_library_id", "plex_items", ["library_id"])
@@ -137,14 +179,23 @@ def upgrade() -> None:
op.create_table(
"plex_states",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("user_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False),
sa.Column("item_id", sa.Integer(), sa.ForeignKey("plex_items.id", ondelete="CASCADE"), nullable=False),
sa.Column(
"user_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
),
sa.Column(
"item_id",
sa.Integer(),
sa.ForeignKey("plex_items.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("status", sa.String(length=12), server_default="new", nullable=False),
sa.Column("position_seconds", sa.Integer(), server_default="0", nullable=False),
sa.Column("watched_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("progress_updated_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("synced_to_plex", sa.Boolean(), server_default="false", nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("user_id", "item_id", name="uq_plex_user_item"),
)
op.create_index("ix_plex_states_user_id", "plex_states", ["user_id"])
@@ -9,6 +9,7 @@ Plex section listing (no per-item API calls): rating (audienceRating ~ IMDb), co
release date, and GIN-indexed genre / director / cast arrays. These also double as the content
signals a future watch-habit recommender would use.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -33,9 +34,13 @@ def upgrade() -> None:
op.create_index("ix_plex_items_rating", "plex_items", ["rating"])
op.create_index("ix_plex_items_content_rating", "plex_items", ["content_rating"])
op.create_index("ix_plex_items_studio", "plex_items", ["studio"])
op.create_index("ix_plex_items_originally_available_at", "plex_items", ["originally_available_at"])
op.create_index(
"ix_plex_items_originally_available_at", "plex_items", ["originally_available_at"]
)
op.create_index("ix_plex_items_genres_gin", "plex_items", ["genres"], postgresql_using="gin")
op.create_index("ix_plex_items_directors_gin", "plex_items", ["directors"], postgresql_using="gin")
op.create_index(
"ix_plex_items_directors_gin", "plex_items", ["directors"], postgresql_using="gin"
)
op.create_index("ix_plex_items_cast_gin", "plex_items", ["cast_names"], postgresql_using="gin")
@@ -10,6 +10,7 @@ finds titles by an actor/director name and ranks those above films that merely m
their synopsis. A generated column's expression can't be altered in place, so we drop + recreate it
(and its GIN index). `people_text` is populated on the next Plex sync.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -10,6 +10,7 @@ A `plex_collections` table holds each collection's card metadata; membership is
combinable filter and a title's "collection strip" is a pure local lookup. `editable` marks the
(curated) collections we may later write back to Plex; everything is read-only by default.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -27,7 +28,12 @@ def upgrade() -> None:
"plex_collections",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("rating_key", sa.String(length=32), nullable=False),
sa.Column("library_id", sa.Integer(), sa.ForeignKey("plex_libraries.id", ondelete="CASCADE"), nullable=False),
sa.Column(
"library_id",
sa.Integer(),
sa.ForeignKey("plex_libraries.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("title", sa.String(length=512), nullable=False),
sa.Column("summary", sa.Text(), nullable=True),
sa.Column("thumb_key", sa.String(length=512), nullable=True),
@@ -35,16 +41,32 @@ def upgrade() -> None:
sa.Column("smart", sa.Boolean(), nullable=False, server_default="false"),
sa.Column("editable", sa.Boolean(), nullable=False, server_default="false"),
sa.Column("synced_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
)
op.create_index(
"ix_plex_collections_rating_key", "plex_collections", ["rating_key"], unique=True
)
op.create_index("ix_plex_collections_rating_key", "plex_collections", ["rating_key"], unique=True)
op.create_index("ix_plex_collections_library_id", "plex_collections", ["library_id"])
op.add_column("plex_items", sa.Column("collection_keys", postgresql.JSONB(), nullable=True))
op.create_index("ix_plex_items_collections_gin", "plex_items", ["collection_keys"], postgresql_using="gin")
op.create_index(
"ix_plex_items_collections_gin", "plex_items", ["collection_keys"], postgresql_using="gin"
)
op.add_column("plex_shows", sa.Column("collection_keys", postgresql.JSONB(), nullable=True))
op.create_index("ix_plex_shows_collections_gin", "plex_shows", ["collection_keys"], postgresql_using="gin")
op.create_index(
"ix_plex_shows_collections_gin", "plex_shows", ["collection_keys"], postgresql_using="gin"
)
def downgrade() -> None:
@@ -9,6 +9,7 @@ Plex account (like plex_states) — the "your own lists" counterpart to the shar
collections. `plex_playlists` is owned by a user; `plex_playlist_items` holds the ordered entries
(`position`). `plex_rating_key` is reserved for an optional later Plex-direction sync.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -24,11 +25,23 @@ def upgrade() -> None:
op.create_table(
"plex_playlists",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("user_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False),
sa.Column(
"user_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
),
sa.Column("title", sa.String(length=512), nullable=False),
sa.Column("plex_rating_key", sa.String(length=32), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
)
op.create_index("ix_plex_playlists_user_id", "plex_playlists", ["user_id"])
@@ -36,12 +49,16 @@ def upgrade() -> None:
"plex_playlist_items",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"playlist_id", sa.Integer(),
sa.ForeignKey("plex_playlists.id", ondelete="CASCADE"), nullable=False,
"playlist_id",
sa.Integer(),
sa.ForeignKey("plex_playlists.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"item_id", sa.Integer(),
sa.ForeignKey("plex_items.id", ondelete="CASCADE"), nullable=False,
"item_id",
sa.Integer(),
sa.ForeignKey("plex_items.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("position", sa.Integer(), nullable=False, server_default="0"),
sa.UniqueConstraint("playlist_id", "item_id", name="uq_plex_playlist_item"),
@@ -13,6 +13,7 @@ and captures yt-dlp's canonical channel/uploader URL so an auto-filled channel b
overrides (like display_name — display metadata only, never touches the on-disk file).
- download_jobs.extra_links: optional list of extra reference URLs shown on the card + watch page.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -26,7 +27,9 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("media_assets", sa.Column("uploader_url", sa.Text(), nullable=True))
op.add_column("download_jobs", sa.Column("display_uploader", sa.String(length=255), nullable=True))
op.add_column(
"download_jobs", sa.Column("display_uploader", sa.String(length=255), nullable=True)
)
op.add_column("download_jobs", sa.Column("display_uploader_url", sa.Text(), nullable=True))
op.add_column("download_jobs", sa.Column("extra_links", sa.JSON(), nullable=True))
@@ -9,6 +9,7 @@ download shows a blank image box. The worker now extracts a representative frame
saves it as a poster sidecar; `poster_path` (relative to download_root) records it so the card /
watch page / link preview can fall back to it. Nullable — only set when there was no thumbnail.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -5,6 +5,7 @@ state with a Plex account. The owner MVP uses the server admin token (uses_admin
token); a personal token column is reserved for later friend-account linking (PIN OAuth). See the
PlexLink model docstring.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -11,6 +11,7 @@ blob folded into the generated `search_vector` (weight B). This lets the TV grid
sorted like the movie grid. A generated column's expression can't be altered in place, so the
search_vector is dropped + recreated. All new columns are populated on the next Plex sync.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -57,9 +58,13 @@ def upgrade() -> None:
op.create_index("ix_plex_shows_rating", "plex_shows", ["rating"])
op.create_index("ix_plex_shows_content_rating", "plex_shows", ["content_rating"])
op.create_index("ix_plex_shows_studio", "plex_shows", ["studio"])
op.create_index("ix_plex_shows_originally_available_at", "plex_shows", ["originally_available_at"])
op.create_index(
"ix_plex_shows_originally_available_at", "plex_shows", ["originally_available_at"]
)
op.create_index("ix_plex_shows_genres_gin", "plex_shows", ["genres"], postgresql_using="gin")
op.create_index("ix_plex_shows_directors_gin", "plex_shows", ["directors"], postgresql_using="gin")
op.create_index(
"ix_plex_shows_directors_gin", "plex_shows", ["directors"], postgresql_using="gin"
)
op.create_index("ix_plex_shows_cast_gin", "plex_shows", ["cast_names"], postgresql_using="gin")
# NB: ix_plex_shows_collections_gin already exists (created with collections in 0047).
@@ -5,6 +5,7 @@ current_user rejects any cookie whose epoch is stale. Bumped on password reset,
or "log out everywhere" — so a stolen/copied client-side cookie dies on those events instead of
staying valid until it expires. Existing rows default to 0 (matching a fresh cookie's absent/0 epoch).
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -4,6 +4,7 @@ Revision ID: 0054_audit_log
Revises: 0053_user_session_epoch
Create Date: 2026-07-12
"""
from typing import Sequence, Union
from alembic import op
@@ -4,6 +4,7 @@ Revision ID: 0055_channel_keywords
Revises: 0054_audit_log
Create Date: 2026-07-12
"""
from typing import Sequence, Union
from alembic import op
+7 -2
View File
@@ -4,6 +4,7 @@ Revision ID: 0056_plex_people
Revises: 0055_channel_keywords
Create Date: 2026-07-19
"""
from typing import Sequence, Union
from alembic import op
@@ -23,8 +24,12 @@ def upgrade() -> None:
# A representative proxied headshot URL (or NULL when none is available). Captured
# opportunistically from an item's rich metadata (info-page view + the rich re-sync).
sa.Column("photo_thumb", sa.Text(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index("ix_plex_people_name", "plex_people", ["name"], unique=True)
@@ -4,6 +4,7 @@ Revision ID: 0057_plex_ppl_enriched
Revises: 0056_plex_people
Create Date: 2026-07-19
"""
from typing import Sequence, Union
from alembic import op
@@ -17,7 +18,9 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
for tbl in ("plex_items", "plex_shows"):
op.add_column(tbl, sa.Column("people_enriched_at", sa.DateTime(timezone=True), nullable=True))
op.add_column(
tbl, sa.Column("people_enriched_at", sa.DateTime(timezone=True), nullable=True)
)
def downgrade() -> None:
@@ -10,6 +10,7 @@ Best still one click away. Re-seeds the builtin rows so fresh installs and exist
(same mechanism as 0040). Existing jobs are unaffected (they snapshot their spec; a profile_id that
pointed at a replaced row just SET NULLs).
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -10,6 +10,7 @@ did NOT invalidate outstanding grants — they lived out their 6h TTL. Fold a pe
into the signed grant and bump it on every password change, so a rotation revokes old grants at once.
Small table (one row per share link), so this is a quick add-column with a default.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -9,6 +9,7 @@ This one-time data fixup rewrites those scalars to SQL NULL so the columns are u
forward-only: a SQL NULL is indistinguishable from the old JSON `null` to every reader (the guards
treat both as "not an array"), so there is nothing meaningful to downgrade. No schema change.
"""
from typing import Sequence, Union
from alembic import op
@@ -25,9 +26,7 @@ _COLS = ("genres", "directors", "cast_names", "collection_keys")
def upgrade() -> None:
for table in _TABLES:
for col in _COLS:
op.execute(
f"UPDATE {table} SET {col} = NULL WHERE jsonb_typeof({col}) = 'null'"
)
op.execute(f"UPDATE {table} SET {col} = NULL WHERE jsonb_typeof({col}) = 'null'")
def downgrade() -> None:
@@ -7,6 +7,7 @@ small category code (`unavailable` / `login_required` / `geo_blocked` / `format_
a retry makes sense, keeping the raw text as a technical-detail disclosure. Nullable; only set on the
error path. Forward-only — an existing errored row simply has NULL until its next failure/retry.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -17,6 +17,7 @@ Redundant index -> the composite that already covers it:
ix_plex_states_user_id <- uq_plex_user_item (user_id, item_id)
ix_plex_playlist_items_playlist_id <- uq_plex_playlist_item (playlist_id, item_id)
"""
from typing import Sequence, Union
from alembic import op
@@ -5,6 +5,7 @@ OVERLAY library is the Hungarian-metadata twin of another section: it is never b
its titles' HU fields are written onto the matched EN rows (paired by `guid`, see 0064). Cheap:
metadata-only, no table rewrite.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -9,6 +9,7 @@ Run BEFORE the first HU overlay sync: while the *_hu columns are all NULL the ge
is trivial. ADD only (no drop/rebuild of the EN search_vector). NOTE: adding a STORED generated
column + its GIN takes an ACCESS EXCLUSIVE lock and rewrites the table — brief, but flag it on prod.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -11,6 +11,7 @@ so the data step below resets `people_enriched_at` to NULL on movie/show rows: t
each once (60/run, self-throttled) and stamps the new columns. ADD-only, all nullable → no table
rewrite, no generated column, no ACCESS EXCLUSIVE lock (unlike 0064).
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -6,6 +6,7 @@ schema allowed NULL — a model/schema drift the new migration drift-test (E1 S2
server_default has filled every row since day one, so no NULLs exist; enforcing NOT NULL just makes the
schema match the contract the model already promises. Pure DDL, reversible.
"""
from typing import Sequence, Union
from alembic import op
@@ -22,6 +22,7 @@ CONCURRENTLY (in alembic's autocommit_block, since it can't run inside a transac
these without blocking reads/writes; downtime ~0. IF (NOT) EXISTS keeps the drift test's
downgrade->re-upgrade round-trip idempotent.
"""
from typing import Sequence, Union
from alembic import op
@@ -13,6 +13,7 @@ Note: `SET NOT NULL` validates with a brief ACCESS EXCLUSIVE lock + one table sc
CONCURRENTLY). On the prod row count (~235k) that is sub-second; still, it is a short write-blocking lock
rather than the ~0 of 0067's CONCURRENTLY index build.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -27,9 +28,7 @@ depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Backfill any stray NULLs first so SET NOT NULL can't fail (0 in prod, but be safe).
op.execute("UPDATE videos SET published_at = now() WHERE published_at IS NULL")
op.alter_column(
"videos", "published_at", nullable=False, server_default=sa.text("now()")
)
op.alter_column("videos", "published_at", nullable=False, server_default=sa.text("now()"))
def downgrade() -> None:
@@ -18,6 +18,7 @@ biggest table, same lookups. live_status is left alone (a separate finding).
All CONCURRENTLY (autocommit_block) — downtime ~0; IF (NOT) EXISTS keeps the drift test's
downgrade->re-upgrade idempotent.
"""
from typing import Sequence, Union
from alembic import op
@@ -5,6 +5,7 @@ hand-rolled override-or-default reader in `app.state`, duplicating the sysconfig
now a sysconfig key (`system_config` store), so every admin knob lives in one place. This migration
carries any existing override into the sysconfig store, then drops the dead column. Reversible.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -8,6 +8,7 @@ Plex-sourced `imdb_rating`/`rating`, which are left untouched. Plus `plex_people
foundation for S6's person bios). ADD-only, all nullable → no table rewrite, no ACCESS EXCLUSIVE lock.
No watermark reset needed: `tmdb_enriched_at` starts NULL, so every eligible row is picked up once.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -7,6 +7,7 @@ first time a person page is opened (from /person/{tmdb_person_id}), then SERVED
`tmdb_person_id` link itself already exists (0071). ADD-only, all nullable → no table rewrite, no
ACCESS EXCLUSIVE lock. `tmdb_enriched_at` starts NULL, so every person is enriched once on first view.
"""
from typing import Sequence, Union
import sqlalchemy as sa
@@ -0,0 +1,40 @@
"""saved_views.source (ask 4: per-module saved views)
Saved "smart views" were feed-only; the Plex module now has them too. Add a `source` discriminator
(default "feed" so every existing row stays a feed view) and widen the per-user unique name constraint
to (user_id, source, name) so a feed view and a Plex view can share a name. `source` is NOT NULL with a
server_default, so the ADD is a metadata-only rewrite-free change; the constraint swap is quick on a
small per-user table.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0073_saved_view_source"
down_revision: Union[str, None] = "0072_plex_person_bio"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"saved_views",
sa.Column("source", sa.String(length=16), nullable=False, server_default="feed"),
)
op.drop_constraint("uq_user_saved_view", "saved_views", type_="unique")
op.create_unique_constraint(
"uq_user_source_saved_view", "saved_views", ["user_id", "source", "name"]
)
def downgrade() -> None:
# LOSSY / effectively forward-only: upgrade lets a user hold a feed view and a plex view of the same
# name, so re-creating the narrow (user_id, name) constraint can raise IntegrityError on such a pair.
# De-dup by dropping the non-feed rows first (they can't exist in the pre-0073 schema anyway) so the
# narrow constraint re-applies cleanly for the drift test's downgrade→re-upgrade cycle.
op.execute("DELETE FROM saved_views WHERE source <> 'feed'")
op.drop_constraint("uq_user_source_saved_view", "saved_views", type_="unique")
op.create_unique_constraint("uq_user_saved_view", "saved_views", ["user_id", "name"])
op.drop_column("saved_views", "source")
@@ -0,0 +1,46 @@
"""download_credentials (S6: per-user, per-host yt-dlp cookie/auth store)
Each user stores their OWN login cookies (Netscape cookies.txt) for an auth-walled download host
(Facebook, Instagram, …), Fernet-encrypted at rest. A download of a yt-dlp-supported, auth-walled
link on that host is retried with the acting user's cookie. Unique per (user_id, host_key) so a
user holds at most one credential per host. Empty for every existing install; forward-only in
practice (a downgrade just drops the table + its secrets).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0074_download_credentials"
down_revision: Union[str, None] = "0073_saved_view_source"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"download_credentials",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("host_key", sa.String(length=64), nullable=False),
sa.Column("cookies_enc", sa.Text(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("user_id", "host_key", name="uq_download_credential"),
)
op.create_index("ix_download_credentials_user_id", "download_credentials", ["user_id"])
def downgrade() -> None:
op.drop_index("ix_download_credentials_user_id", table_name="download_credentials")
op.drop_table("download_credentials")
@@ -0,0 +1,26 @@
"""playlist synced_video_ids (S7: 3-way merge baseline for two-way YouTube sync)
The one-directional "local wins" push is replaced by a 3-way merge that classifies each item
against the last-synced state. That needs the raw ordered id set as a baseline, not just the
`synced_fingerprint` hash. Add a nullable `synced_video_ids` (JSON list) column; NULL on every
existing row means "no baseline yet" — the first merge then unions losslessly and records one.
Forward-only in practice (a downgrade just drops the column).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0075_playlist_sync_baseline"
down_revision: Union[str, None] = "0074_download_credentials"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("playlists", sa.Column("synced_video_ids", sa.Text(), nullable=True))
def downgrade() -> None:
op.drop_column("playlists", "synced_video_ids")
@@ -0,0 +1,41 @@
"""media_assets.media_kind (E7 S5a: the video|audio|file discriminator for the media library)
Adds a stored media-kind to each asset so the library can gate its actions on it (play/edit for
media, download-only for a generic uploaded file) instead of re-deriving from codecs in prose. The
column is the single home of the invariant `downloads.mediakind.derive` computes at finalize.
Backfill for existing rows: a stored video codec -> 'video', an audio-only codec -> 'audio', an
editor derivative (source_kind='edit', which carries no codecs but is always a video clip) ->
'video'. Rows still pending/errored (no codecs, not an edit) stay NULL — kind is set when they
finalize. Empty `media_kind` never reaches the UI: only `ready`/`done` rows render actions, and
those are all backfilled here.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0076_media_asset_kind"
down_revision: Union[str, None] = "0075_playlist_sync_baseline"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("media_assets", sa.Column("media_kind", sa.String(length=16), nullable=True))
op.execute(
"""
UPDATE media_assets
SET media_kind = CASE
WHEN vcodec IS NOT NULL THEN 'video'
WHEN acodec IS NOT NULL THEN 'audio'
WHEN source_kind = 'edit' THEN 'video'
ELSE NULL
END
"""
)
def downgrade() -> None:
op.drop_column("media_assets", "media_kind")
+4
View File
@@ -9,6 +9,7 @@ runs are always logged. Secrets are never stored (config changes log the key nam
mutation it describes, so the trail can't drift from reality (and a rolled-back mutation drops
its audit row too).
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
@@ -43,6 +44,9 @@ class AuditAction:
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
# The log itself was cleared (a single row survives the clear as tamper-evidence)
AUDIT_CLEAR = "audit_clear"
+6 -9
View File
@@ -41,8 +41,7 @@ log = logging.getLogger("siftlode.auth")
def google_enabled(db: Session) -> bool:
"""Whether Sign in with Google is available (both client id + secret resolve to non-empty)."""
return bool(
sysconfig.get_str(db, "google_client_id")
and sysconfig.get_str(db, "google_client_secret")
sysconfig.get_str(db, "google_client_id") and sysconfig.get_str(db, "google_client_secret")
)
@@ -125,7 +124,9 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
# ciphertext, so a token row with no refresh token silently failed to revoke). decrypt_optional so a
# TOKEN_ENCRYPTION_KEY mismatch can't 500 the purge — a best-effort revoke degrades to "skip". (B-core-2)
google_token = (
(decrypt_optional(tok.refresh_token_enc) or decrypt_optional(tok.access_token)) if tok else None
(decrypt_optional(tok.refresh_token_enc) or decrypt_optional(tok.access_token))
if tok
else None
)
# Raw delete so Postgres applies ON DELETE CASCADE / SET NULL on every dependent table.
db.execute(delete(User).where(User.id == user_id))
@@ -197,9 +198,7 @@ def current_user(request: Request, db: Session = Depends(get_db)) -> User:
return user
def optional_current_user(
request: Request, db: Session = Depends(get_db)
) -> User | None:
def optional_current_user(request: Request, db: Session = Depends(get_db)) -> User | None:
"""Like `current_user` but returns None instead of raising 401 when there's no valid session.
Used by the bootstrap /api/me probe so the logged-out landing doesn't log a failed request to
the browser console (a non-2xx fetch is a console error no matter how the app handles it)."""
@@ -215,9 +214,7 @@ def require_human(user: User = Depends(current_user)) -> User:
no token, so has_read_scope/has_write_scope are False); this makes the boundary explicit
where there's no scope check to lean on."""
if user.is_demo:
raise HTTPException(
status_code=403, detail="Not available in the demo account."
)
raise HTTPException(status_code=403, detail="Not available in the demo account.")
return user
+23 -1
View File
@@ -169,6 +169,7 @@ class Settings(BaseSettings):
@property
def player_client_fallback_list(self) -> list[str]:
return self._clients(self.download_player_clients_fallback)
# How many downloads the worker runs concurrently (claimed via FOR UPDATE SKIP LOCKED).
download_worker_concurrency: int = 2
# How often (empty pending queue) the worker polls for a new job, in seconds.
@@ -185,6 +186,21 @@ class Settings(BaseSettings):
download_supervisor_poll_seconds: int = 3
# Grace between SIGTERM and SIGKILL when stopping a child (matches run_ffmpeg's escalation).
download_kill_grace_seconds: int = 5
# --- worker-child resource de-prioritization ---
# The download/edit child (yt-dlp + ffmpeg) shares ONE LAN box with the API, Postgres, and Plex
# playback. Without a priority handicap a CPU re-encode pegs every core for ~30s and starves web
# requests + playback until it finishes. The child lowers its OWN priority at startup — inherited
# by every ffmpeg/yt-dlp descendant — so any foreground work always wins. Each knob is best-effort
# and independently disable-able with 0.
download_child_nice: int = 10 # CPU nice increment applied to the child; 0 = don't renice
download_child_ionice_class: int = 3 # best-effort IO class (3 = idle); 0 = leave IO untouched
# Cap the browser-safe re-encode's encoder threads so a transcode can't claim every core. 0 =
# reserve one core for foreground work (max(1, cpu_count - 1)); >0 pins exactly that many.
download_ffmpeg_threads: int = 0
# Throttle the retention GC's whole-tree orphan file sweep so its rglob can't monopolize disk IO
# in the API process: sleep `pause_ms` after every `batch` files touched. pause_ms=0 disables.
download_gc_sweep_pause_ms: int = 20
download_gc_sweep_batch: int = 400
# Total cache size cap across ALL users (bytes). When >0 and exceeded, the GC evicts the
# least-recently-accessed ready assets until under it. 0 = unlimited (disk is the limit).
download_total_max_bytes: int = 0
@@ -192,6 +208,10 @@ class Settings(BaseSettings):
download_default_max_bytes: int = 5_368_709_120 # 5 GiB
download_default_max_concurrent: int = 2
download_default_max_jobs: int = 50
# Per-file cap for local uploads (E7). Unlike a download (whose size is unknown until the worker
# finishes), an upload's size is known at ingest, so this bounds a single file directly; the
# user's overall byte footprint is still governed by download_default_max_bytes.
download_upload_max_bytes: int = 524_288_000 # 500 MiB
# Retention: an asset expires this many days after its last access; the GC job then
# deletes it (once no job references it) after an additional grace window.
download_retention_days: int = 30
@@ -231,7 +251,9 @@ class Settings(BaseSettings):
plex_sync_interval_min: int = 360
# How often the incremental Plex→Siftlode watch-state reconcile runs, in minutes (Phase C):
# a cheap history + on-deck pull plus a re-push of any states that failed their immediate push.
plex_watch_sync_interval_min: int = 30
# 15 min (S2): with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# Plex-authoritative pull), a faster incremental pull keeps Siftlode close to Plex-side changes.
plex_watch_sync_interval_min: int = 15
# How often the FULL watch-state reconcile runs, in minutes (Phase C). A whole-section rescan
# that catches what the incremental history feed can't — notably an un-watch done on the Plex
# side — so it runs far less often (daily) than the incremental pass.
+12 -7
View File
@@ -4,6 +4,7 @@ Shared by the admin's manual reset endpoint (routes/admin.py) and the scheduled
(scheduler.py). Lives in the service layer, not routes, so the scheduler can call it without an
`app.routes -> app.scheduler` import cycle (the reason its import used to be lazy).
"""
from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
@@ -49,13 +50,17 @@ def _seed_demo_subscriptions(db: Session, demo: User, n: int = 14) -> int:
Channel manager isn't empty — gives visitors something to explore and matches the landing-page
screenshot. Idempotent: clears the demo's subscriptions first, then re-adds the top N."""
db.execute(delete(Subscription).where(Subscription.user_id == demo.id))
top = db.execute(
select(Video.channel_id)
.where(Video.channel_id.is_not(None))
.group_by(Video.channel_id)
.order_by(func.count().desc())
.limit(n)
).scalars().all()
top = (
db.execute(
select(Video.channel_id)
.where(Video.channel_id.is_not(None))
.group_by(Video.channel_id)
.order_by(func.count().desc())
.limit(n)
)
.scalars()
.all()
)
for channel_id in top:
db.add(Subscription(user_id=demo.id, channel_id=channel_id))
return len(top)
+5 -2
View File
@@ -10,6 +10,7 @@ Adapters are stateless singletons; get one with `resolve_adapter(source_kind, so
Resolution works both before extraction (from source_kind / URL host) and after (from the yt-dlp
`extractor_key` in the info dict), so a non-YouTube "url" job still lands on the right adapter.
"""
from urllib.parse import urlparse
from app.config import settings
@@ -140,8 +141,10 @@ def resolve_adapter(
def host_is(*domains: str) -> bool:
return any(_host_is(h, *domains) for h in hosts)
if source_kind == "youtube" or "youtube" in extractor or host_is(
"youtube.com", "youtu.be", "youtube-nocookie.com"
if (
source_kind == "youtube"
or "youtube" in extractor
or host_is("youtube.com", "youtu.be", "youtube-nocookie.com")
):
return _YOUTUBE
if "facebook" in extractor or host_is("facebook.com", "fb.watch"):
+1
View File
@@ -10,6 +10,7 @@ URL may have expired — those just get a locally-generated poster instead).
Run once per environment:
docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
"""
import logging
import yt_dlp
+258
View File
@@ -0,0 +1,258 @@
"""Per-user, per-host download cookies — the single source of truth for "does the acting user's
cookie apply to this download, and how does that keep the result private".
Two call sites depend on ONE consistent answer (house rule: an invariant lives in shared code, not
in prose):
* `enqueue` salts the MediaAsset cache signature with the user id WHEN a cookie applies, so the
authed download becomes a PRIVATE asset that is never cache-served to another user.
* the worker attaches the cookie file to yt-dlp ONLY when a cookie still applies AND the asset it
is downloading into is that private, salted identity — so a cookie can never poison a shared
asset (the load-bearing cross-user privacy fix).
Both go through `cookie_salt` / `authed_format_sig` / `host_key_for_url` here, so they can't drift.
Security controls enforced here:
* **host scope (SEC-12)** — `host_key_for_url` matches the download URL's netloc against a curated
registry via the netloc-scoped `_host_is`, so a cookie is only ever offered to the exact site it
belongs to; an arbitrary `?url=` (or a look-alike host) resolves to no host_key and gets no
cookie.
* **secret handling** — cookies are Fernet-encrypted at rest; the blob is only ever decrypted
in-worker into a 0600 file that is unlinked in a `finally`, and is never returned by a route,
logged, or written to a job's error/snapshot.
"""
import logging
import os
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential
from app.security import decrypt_optional, encrypt
log = logging.getLogger("siftlode.downloads.credentials")
class HostSpec:
"""A supported cookie host: a short canonical `key` (stored on the credential + folded into the
cache salt), a human `label` for the UI, and the netloc `domains` it matches."""
__slots__ = ("key", "label", "domains")
def __init__(self, key: str, label: str, domains: tuple[str, ...]):
self.key = key
self.label = label
self.domains = domains
# Curated registry (user's call: seed the common auth-walled hosts now). One `key` can cover several
# domains of the same site. Add a host by adding a row — the FE dropdown, host-scope guard, and salt
# all derive from this, so there is nothing else to wire up.
SUPPORTED_HOSTS: tuple[HostSpec, ...] = (
HostSpec("facebook", "Facebook", ("facebook.com", "fb.watch")),
HostSpec("instagram", "Instagram", ("instagram.com",)),
HostSpec("twitter", "X (Twitter)", ("twitter.com", "x.com")),
HostSpec("tiktok", "TikTok", ("tiktok.com",)),
HostSpec("reddit", "Reddit", ("reddit.com",)),
)
_BY_KEY: dict[str, HostSpec] = {h.key: h for h in SUPPORTED_HOSTS}
def is_supported_host(host_key: str) -> bool:
return host_key in _BY_KEY
def host_label(host_key: str) -> str:
spec = _BY_KEY.get(host_key)
return spec.label if spec else host_key
def host_key_for_url(url: str | None) -> str | None:
"""The supported host_key whose domains own `url`'s netloc, or None. Netloc-scoped (via
`_host_is`), so a look-alike host ('myfacebook.com', 'facebook.com.evil') never matches. When
several specs could match, the most-specific (longest matched domain) wins so the result is
deterministic across the enqueue and worker call sites."""
if not url:
return None
from urllib.parse import urlparse
host = urlparse(url).netloc.lower()
if not host:
return None
best_key: str | None = None
best_len = -1
for spec in SUPPORTED_HOSTS:
for dom in spec.domains:
if _host_is(host, dom) and len(dom) > best_len:
best_key, best_len = spec.key, len(dom)
return best_key
def cookie_salt(user_id: int, host_key: str) -> str:
"""The cache-signature salt for a cookie-authed download. Keyed by user id + host so the authed
asset is private to that user (never dedup-shared) and stable across that user's own re-adds."""
return f"cookie:{user_id}:{host_key}"
def authed_format_sig(spec: dict, user_id: int, host_key: str) -> str:
"""The salted `format_sig` an authed download for (user, host) resolves to — the private asset
identity. Both enqueue (to create it) and the worker (to recognise it) compute it HERE."""
return formats.format_sig(spec, salt=cookie_salt(user_id, host_key))
def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
"""At enqueue: the cache salt to apply if this user has a cookie for `url`'s host, else None (a
normal shared download). Returns the salt string so the caller passes it straight to
`format_sig(spec, salt=...)`."""
host_key = host_key_for_url(url)
if host_key is None:
return None
if get_credential(db, user_id, host_key) is None:
return None
return cookie_salt(user_id, host_key)
# --- storage (CRUD) ---------------------------------------------------------------------------
def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredential | None:
return db.execute(
select(DownloadCredential).where(
DownloadCredential.user_id == user_id, DownloadCredential.host_key == host_key
)
).scalar_one_or_none()
class InvalidCookies(ValueError):
"""The pasted cookies aren't a usable Netscape cookies.txt — surfaced as a 400 by the route."""
def validate_cookies(raw: str) -> str:
"""Sanity-check a pasted cookies.txt (Netscape format) and return the cleaned text. yt-dlp reads
the Netscape format: comment/`#` lines plus TAB-separated cookie lines (domain, flag, path,
secure, expiry, name, value). We don't fully parse — just reject obvious non-cookie paste (empty,
or no tab-separated data line) so a mistaken paste fails HERE with a clear message rather than
silently producing a cookie file yt-dlp ignores."""
text = (raw or "").strip()
if not text:
raise InvalidCookies("empty")
has_data_line = any(
line.strip() and not line.lstrip().startswith("#") and "\t" in line
for line in text.splitlines()
)
if not has_data_line:
raise InvalidCookies("format")
# Netscape cookie files must end with a newline for some parsers; normalise.
return text if text.endswith("\n") else text + "\n"
def upsert_credential(db: Session, user_id: int, host_key: str, raw: str) -> DownloadCredential:
"""Create or replace this user's cookie blob for `host_key`. Caller has validated host_key is
supported and `raw` via `validate_cookies`. Does NOT commit (the route commits with the audit
row so the trail can't drift). The blob is encrypted here; the plaintext never leaves this call."""
cred = get_credential(db, user_id, host_key)
enc = encrypt(raw)
if cred is None:
cred = DownloadCredential(user_id=user_id, host_key=host_key, cookies_enc=enc)
db.add(cred)
else:
cred.cookies_enc = enc
return cred
def delete_credential(db: Session, user_id: int, host_key: str) -> bool:
"""Delete this user's cookie for `host_key`. Returns whether a row existed. Does NOT commit."""
cred = get_credential(db, user_id, host_key)
if cred is None:
return False
db.delete(cred)
return True
def status_list(db: Session, user_id: int) -> list[dict]:
"""The supported hosts with this user's configured/updated state — the write-only Settings view.
Never includes the blob."""
have = {
c.host_key: c
for c in db.execute(
select(DownloadCredential).where(DownloadCredential.user_id == user_id)
).scalars()
}
out = []
for spec in SUPPORTED_HOSTS:
cred = have.get(spec.key)
out.append(
{
"host_key": spec.key,
"label": spec.label,
"configured": cred is not None,
"updated_at": cred.updated_at.isoformat() if cred is not None else None,
}
)
return out
# --- worker: resolve + write the cookie file --------------------------------------------------
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset_format_sig: str,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download, or None.
The double gate that prevents cross-user leakage:
1. a credential must still exist for `url`'s host (host-scoped via `host_key_for_url`), AND
2. the asset being downloaded into must BE this user's private, salted identity
(`asset_format_sig == authed_format_sig(...)`).
If (2) fails the asset is the SHARED (unsalted) one — attaching the cookie would poison the
global cache with a private-group video, so we attach nothing. An unreadable blob (rotated key →
InvalidToken) also yields None; the download then fails honestly on the login wall (S3's
`login_required`)."""
host_key = host_key_for_url(url)
if host_key is None:
return None
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset_format_sig != authed_format_sig(spec, user_id, host_key):
return None # shared asset — never attach a cookie to a non-private identity
raw = decrypt_optional(cred.cookies_enc)
if not raw:
log.warning(
"download cookie for user %s host %s could not be decrypted (rotated key?) — "
"proceeding without it",
user_id,
host_key,
)
return None
return _write_cookiefile(raw, staging)
def _write_cookiefile(raw: str, staging: Path) -> Path:
"""Write the cookie blob to a private 0600 file under `staging/.auth/` and return its path.
Deliberately NOT in the staging root: the download retry loop unlinks every file it finds there,
and `_produced_file` picks the largest file there as the output — either would corrupt a
cookies.txt in the root. A `.auth` SUBDIR survives both (they skip directories) and is still
removed by the parent worker's staging cleanup on a hard kill. The caller unlinks it in a
`finally` on the normal/exception paths."""
auth_dir = staging / ".auth"
auth_dir.mkdir(mode=0o700, exist_ok=True)
path = auth_dir / "cookies.txt"
# Open with 0600 from the start (don't create world-readable then chmod).
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
try:
os.write(fd, raw.encode("utf-8"))
finally:
os.close(fd)
return path
+192 -46
View File
@@ -20,10 +20,12 @@ Also builds the editor **filmstrip** — a single tiled sprite of the source vid
scrub track. Cached on the source asset's `storyboard_path` (reserved in phase 1) and reusable
for a player hover-preview later.
"""
import hashlib
import json
import logging
import math
import os
import queue
import shutil
import subprocess
@@ -31,15 +33,28 @@ import threading
import time
from pathlib import Path
from app.config import settings
log = logging.getLogger("siftlode.edit")
def encode_thread_cap() -> int:
"""How many encoder threads a browser-safe re-encode may use. `download_ffmpeg_threads` when the
admin pinned one (>0), else reserve a core for foreground work (max(1, cpu_count - 1)). Shared by
the ffmpeg `-threads` flag and the child's `OMP_NUM_THREADS` so both honour the same budget."""
n = settings.download_ffmpeg_threads
if n > 0:
return n
return max(1, (os.cpu_count() or 2) - 1)
class EditAborted(Exception):
"""Raised by run_ffmpeg when the job was paused/canceled mid-encode."""
# --- edit spec ------------------------------------------------------------------------------
def normalize_edit_spec(spec: dict | None) -> dict:
"""Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
@@ -140,6 +155,7 @@ def clip_duration(spec: dict, source_duration: int | None) -> int | None:
# --- ffmpeg: trim / crop --------------------------------------------------------------------
def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]:
"""ffmpeg command for a trim/crop edit. `-progress pipe:1` streams machine-readable progress.
@@ -153,8 +169,15 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
reencode = needs_reencode(spec)
cmd = [
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
"-progress", "pipe:1", "-nostats",
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
]
if start:
cmd += ["-ss", f"{start:.3f}"]
@@ -170,8 +193,16 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
if crop:
cmd += ["-vf", f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}"]
cmd += [
"-c:v", "libx264", "-preset", "veryfast", "-crf", "20",
"-c:a", "aac", "-b:a", "192k",
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"20",
"-c:a",
"aac",
"-b:a",
"192k",
]
else:
cmd += ["-c", "copy", "-avoid_negative_ts", "make_zero"]
@@ -204,8 +235,15 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
crop = spec.get("crop")
prep: dict[Path, str] = {}
head = [
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
"-progress", "pipe:1", "-nostats",
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
]
if needs_reencode(spec):
@@ -218,9 +256,24 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
labels.append(f"[v{i}][a{i}]")
parts.append(f"{''.join(labels)}concat=n={len(segs)}:v=1:a=1[v][a]")
cmd = head + [
"-i", str(src), "-filter_complex", ";".join(parts),
"-map", "[v]", "-map", "[a]",
"-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-c:a", "aac", "-b:a", "192k",
"-i",
str(src),
"-filter_complex",
";".join(parts),
"-map",
"[v]",
"-map",
"[a]",
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"20",
"-c:a",
"aac",
"-b:a",
"192k",
]
else:
listp = staging / "concat.txt"
@@ -314,7 +367,9 @@ def _stop(proc: subprocess.Popen) -> None:
escalate_kill(proc, proc.terminate, proc.kill, grace_s=5, label="ffmpeg")
def run_ffmpeg(cmd, total_s, on_progress, should_cancel, stall_timeout_s=_FFMPEG_STALL_TIMEOUT_S) -> None:
def run_ffmpeg(
cmd, total_s, on_progress, should_cancel, stall_timeout_s=_FFMPEG_STALL_TIMEOUT_S
) -> None:
"""Run an ffmpeg edit, reporting 0–100 progress and honouring cooperative cancel.
Both pipes are drained by threads (see `_drain_tail`) and the main loop polls on a 1s tick, so
@@ -383,18 +438,42 @@ _BROWSER_SAFE_VCODEC = ("h264",)
_BROWSER_SAFE_ACODEC = ("aac", "mp3")
def parse_probe_streams(payload: str) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) out of ffprobe's JSON stream list, lowercased.
JSON, not the flat `default=nw=1` output: there the fields arrive in ffprobe's own struct
order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line "remember the last
codec_type" parser attributes every name to the PREVIOUS stream. Measured on real downloads:
an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download."""
def run_ffprobe_json(
path: Path,
*,
show_streams: bool = False,
show_format: bool = False,
entries: str | None = None,
) -> dict | None:
"""One ffprobe call → its parsed JSON dict, or None on any failure (timeout, non-zero exit,
missing binary, unparseable output). The SINGLE ffprobe-JSON-invocation point in the downloads
package: the browser-playability codec probe here and the upload-ingest probe (`upload._probe`)
both go through it, so the subprocess flags/timeout/error handling live in one place instead of
two drifting copies."""
args = ["ffprobe", "-v", "error"]
if show_streams:
args.append("-show_streams")
if show_format:
args.append("-show_format")
if entries:
args += ["-show_entries", entries]
args += ["-of", "json", str(path)]
try:
streams = json.loads(payload or "{}").get("streams") or []
except ValueError:
return None, None
out = subprocess.run(args, capture_output=True, text=True, timeout=30, check=True).stdout
return json.loads(out)
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
return None
def _codecs_from_streams(streams: list) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) lowercased from an ffprobe JSON stream list — the first of each kind.
Parsed from the JSON `streams` list, NOT ffprobe's flat `default=nw=1` output: there the fields
arrive in ffprobe's own struct order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line
"remember the last codec_type" parser attributes every name to the PREVIOUS stream. Measured on real
downloads: an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download. Reading each
stream dict by its own `codec_type` is immune to that ordering."""
vcodec = acodec = None
for s in streams:
name = (s.get("codec_name") or "").lower() or None
@@ -409,15 +488,10 @@ def parse_probe_streams(payload: str) -> tuple[str | None, str | None]:
def probe_codecs(path: Path) -> tuple[str | None, str | None]:
"""Return (video_codec, audio_codec) lowercased via one ffprobe call; None for a missing
stream or if ffprobe fails (a failed probe is treated as 'leave it alone' upstream)."""
try:
out = subprocess.run(
["ffprobe", "-v", "error", "-show_entries", "stream=codec_type,codec_name",
"-of", "json", str(path)],
capture_output=True, text=True, timeout=30, check=True,
).stdout
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError):
data = run_ffprobe_json(path, entries="stream=codec_type,codec_name")
if data is None:
return None, None
return parse_probe_streams(out)
return _codecs_from_streams(data.get("streams") or [])
def probe_duration(path: Path) -> int:
@@ -425,16 +499,29 @@ def probe_duration(path: Path) -> int:
re-encode when the yt-dlp info dict carried no `duration` (some URL/live extractions)."""
try:
out = subprocess.run(
["ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "default=nw=1:nk=1", str(path)],
capture_output=True, text=True, timeout=30, check=True,
[
"ffprobe",
"-v",
"error",
"-show_entries",
"format=duration",
"-of",
"default=nw=1:nk=1",
str(path),
],
capture_output=True,
text=True,
timeout=30,
check=True,
).stdout.strip()
return int(float(out)) if out else 0
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
return 0
def browser_transcode_flags(vcodec: str | None, acodec: str | None, is_mp4: bool) -> tuple[bool, bool, bool]:
def browser_transcode_flags(
vcodec: str | None, acodec: str | None, is_mp4: bool
) -> tuple[bool, bool, bool]:
"""Decide what (if anything) makes a file universally browser-playable. Returns
(needs_change, reencode_video, reencode_audio). needs_change=False => already H.264 + AAC/MP3
in mp4. A None codec (no such stream, or a failed probe) is never re-encoded."""
@@ -443,16 +530,34 @@ def browser_transcode_flags(vcodec: str | None, acodec: str | None, is_mp4: bool
return (rv or ra or not is_mp4), rv, ra
def build_browser_safe_cmd(src: Path, dest: Path, reencode_video: bool, reencode_audio: bool) -> list[str]:
def build_browser_safe_cmd(
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool, threads: int | None = None
) -> list[str]:
"""ffmpeg command that lands `src` as a browser-safe mp4 (faststart): re-encode only the
stream(s) that need it (libx264 / aac), stream-copy the rest, always into mp4. `-progress
pipe:1` streams machine-readable progress for run_ffmpeg."""
pipe:1` streams machine-readable progress for run_ffmpeg. `threads` caps the encoder so a
re-encode can't claim every core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
"-progress", "pipe:1", "-nostats", "-i", str(src),
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
"-i",
str(src),
]
cmd += ["-c:v", "libx264", "-preset", "veryfast", "-crf", "20"] if reencode_video else ["-c:v", "copy"]
cmd += (
["-c:v", "libx264", "-preset", "veryfast", "-crf", "20"]
if reencode_video
else ["-c:v", "copy"]
)
cmd += ["-c:a", "aac", "-b:a", "192k"] if reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
cmd += ["-movflags", "+faststart", str(dest)]
return cmd
@@ -486,25 +591,63 @@ def build_storyboard_cmd(src: Path, dest: Path, duration_s: int | None, geom: di
fps = geom["count"] / dur
vf = f"fps={fps:.6f},scale={_SB_TILE_W}:-2,tile={geom['cols']}x{geom['rows']}"
return [
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
"-i", str(src), "-frames:v", "1", "-q:v", "5", "-vf", vf, str(dest),
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-i",
str(src),
"-frames:v",
"1",
"-q:v",
"5",
"-vf",
vf,
str(dest),
]
def build_poster_cmd(src: Path, dest: Path, at_s: float) -> list[str]:
"""Grab one representative frame as a poster JPEG. `-ss` before `-i` seeks fast."""
return [
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
"-ss", f"{max(0.0, at_s):.3f}", "-i", str(src),
"-frames:v", "1", "-q:v", "3", "-vf", "scale=640:-2", str(dest),
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-ss",
f"{max(0.0, at_s):.3f}",
"-i",
str(src),
"-frames:v",
"1",
"-q:v",
"3",
"-vf",
"scale=640:-2",
str(dest),
]
def build_thumbnail_jpg_cmd(src: Path, dest: Path) -> list[str]:
"""Decode a downloaded thumbnail (webp/png/…) to a single baseline JPEG frame."""
return [
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
"-i", str(src), "-frames:v", "1", "-update", "1", str(dest),
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-i",
str(src),
"-frames:v",
"1",
"-update",
"1",
str(dest),
]
@@ -550,7 +693,10 @@ def thumbnail_to_jpg(src: Path | None, dest: Path, timeout_s: int = 30) -> Path
return None
err = _run_ffmpeg_best_effort(build_thumbnail_jpg_cmd(src, dest), timeout_s)
if err is not None:
log.info("thumbnail->jpg conversion failed (%s); poster falls back to a cut frame", str(err)[:120])
log.info(
"thumbnail->jpg conversion failed (%s); poster falls back to a cut frame",
str(err)[:120],
)
return None
return dest if dest.exists() else None
+14
View File
@@ -24,6 +24,16 @@ _RULES: tuple[tuple[str, str], ...] = (
# Some extractors (e.g. Reddit) gate content behind a logged-in account: "Account
# authentication is required." — an auth wall we deliberately don't cross, not a transient error.
("authentication is required", "login_required"),
# Closed-group / private-post auth walls (Facebook groups, Instagram private, …). yt-dlp phrases
# these many ways; the common thread is "you must be signed in" or its own suggestion to supply
# cookies. Treating the cookie suggestion as an auth signal is what makes a login-walled link read
# honestly (and stay non-retryable) when the acting user has no usable cookie for that host.
("log in to", "login_required"),
("login required", "login_required"),
("you must be logged in", "login_required"),
("only available for registered users", "login_required"),
("use --cookies", "login_required"),
("--cookies-from-browser", "login_required"),
# Geo-restricted.
("in your country", "geo_blocked"),
("available in your location", "geo_blocked"),
@@ -42,6 +52,10 @@ _RULES: tuple[tuple[str, str], ...] = (
("account associated with this video has been terminated", "unavailable"),
("this video is not available", "unavailable"),
("this video has been removed", "unavailable"),
# yt-dlp doesn't support this site at all ("Unsupported URL: …") — distinct from an auth wall on a
# supported site. Non-retryable: no cookie or retry can make an unsupported host downloadable.
("unsupported url", "unsupported_host"),
("no video could be found in this url", "unsupported_host"),
# DRM.
("drm protected", "drm"),
("drm-protected", "drm"),
+59 -51
View File
@@ -5,14 +5,18 @@ of truth for what file gets produced. `format_sig` hashes exactly the fields tha
OUTPUT BYTES, so two users picking equivalent settings share one cached MediaAsset.
`build_ydl_opts` turns a spec into a yt-dlp options dict (format selector + postprocessors).
spec shape (all keys optional; defaults applied by `normalize`):
spec shape (all keys optional; defaults + browser-playability clamp applied by `normalize`):
mode "av" (audio+video) | "v" (video-only) | "a" (audio-only)
max_height int | null (null = best)
container "mp4" | "mkv" | "webm" | null (merge/remux target for av/v)
audio_format "m4a" | "mp3" | "opus" | null (extract target for mode "a")
vcodec "h264" | "vp9" | "av1" | null (preference, best-effort)
container always coerced to "mp4" (the only universally browser-playable container)
audio_format "m4a" | "mp3" (mode "a" extract target; anything else — e.g. opus — → "m4a")
vcodec always coerced to null → H.264 selected (see `normalize`); no opt-out
embed_subs bool embed_chapters bool embed_thumbnail bool sponsorblock bool
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser (E7); the
`container`/`vcodec` keys survive only so an old stored profile deserializes without error.
"""
import hashlib
import json
@@ -49,10 +53,12 @@ _DEFAULTS = {
# prefer H.264+AAC (browser/iOS-compatible) rather than yt-dlp's default VP9/Opus best.
# v3: mp4 compat now *selects* H.264 (avc1) with a progressive fallback + re-encodes the rare
# AV1/VP9-only result to H.264, so cached AV1 assets must re-derive (fixes iOS broken-player).
_SIG_VERSION = 3
# v4: EVERY download is clamped to the mp4/H.264/AAC compat profile (no more mkv/webm/VP9/AV1/Opus
# opt-out), so any asset cached under an old non-compat spec — or a stale non-H.264 mp4 that slipped
# the v3 net — must re-derive through the current browser-safe guarantee.
_SIG_VERSION = 4
_SPONSORBLOCK_CATEGORIES = ["sponsor", "selfpromo", "interaction"]
_VCODEC_SORT = {"h264": "vcodec:h264", "vp9": "vcodec:vp9", "av1": "vcodec:av01"}
def normalize(spec: dict) -> dict:
@@ -75,6 +81,18 @@ def normalize(spec: dict) -> dict:
# Audio-only files can't embed subtitles/thumbnails as video; keep the flags meaningful.
if s["mode"] == "a":
s["embed_subs"] = False
# Browser-playability guarantee (E7): the Download module exists to download → share → PLAY
# in-browser, so every output must play in EVERY browser, iOS/Safari/WebKit included. The only
# universally-safe combo is mp4 + H.264 + AAC/MP3, so we CLAMP the spec to it HERE — the single
# point every path (picker, stored profile, raw API) flows through — instead of only hiding
# options in the UI. Non-mp4 containers (mkv doesn't play at all; webm/VP9/AV1 fail iOS) and any
# explicit vcodec are coerced to the compat profile; Opus audio (Safari-flaky as a bare file)
# falls back to AAC/m4a. A source that exposes no H.264 is still caught by the worker's
# browser-safe re-encode (`_ensure_browser_playable`), so playback is guaranteed end-to-end.
s["container"] = "mp4"
s["vcodec"] = None
if s["audio_format"] not in ("m4a", "mp3"):
s["audio_format"] = "m4a"
return s
@@ -97,12 +115,17 @@ def build_ydl_opts(
outtmpl: str,
progress_hook,
extra: dict | None = None,
cookiefile: str | None = None,
) -> dict:
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
`extra` is source-specific options contributed by the SourceAdapter (e.g. YouTube's
`extractor_args` with the player-client set + bgutil PO-token base_url); it is merged onto the
generic options below. Everything here is source-agnostic."""
generic options below. Everything here is source-agnostic.
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
merge so a source adapter can never clobber it."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
@@ -131,6 +154,11 @@ def build_ydl_opts(
if extra:
opts.update(extra)
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
# be able to override the authenticated session the worker resolved for this specific user.
if cookiefile:
opts["cookiefile"] = cookiefile
if s["mode"] == "a":
opts["format"] = "bestaudio/best"
opts["postprocessors"].append(
@@ -142,51 +170,33 @@ def build_ydl_opts(
)
else:
hcap = f"[height<=?{h}]" if h else ""
# Compatibility-first for mp4 (the universal container): produce H.264 video + AAC audio
# so shared/downloaded files play in EVERY browser, including iOS/Safari. WebKit (which
# every iOS browser, Brave included, is forced to use) decodes only H.264/H.265 + AAC
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there, even though Chromium
# plays it fine. A custom profile that explicitly sets `vcodec` opts out.
compat = s["container"] == "mp4" and s["vcodec"] is None
# Every video download is the mp4/H.264 compat profile (normalize() clamps container→mp4,
# vcodec→None), so shared/downloaded files play in EVERY browser, iOS/Safari included. WebKit
# (which every iOS browser, Brave included, is forced to use) decodes only H.264/H.265 + AAC
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there — so H.264 is selected at the
# SELECTION level, not merely preferred: a sort key can't conjure an H.264 stream a source
# doesn't expose as video-only. A source that ships only non-H.264 is caught after download by
# the worker's browser-safe re-encode (`_ensure_browser_playable`).
if s["mode"] == "v":
if compat:
# Video-only: keep it audio-free. Prefer an H.264 *video-only* stream, then any
# video-only, and only fall to a progressive (audio-bearing) `best` as a last
# resort — same audio-free-when-possible guarantee as the old `bestvideo/best`.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}/bestvideo{hcap}"
f"/best[vcodec^=avc1]{hcap}/best{hcap}"
)
else:
opts["format"] = f"bestvideo{hcap}/best{hcap}"
# Video-only: keep it audio-free. Prefer an H.264 video-only stream, then any video-only,
# and only fall to a progressive (audio-bearing) `best` as a last resort.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}/bestvideo{hcap}/best[vcodec^=avc1]{hcap}/best{hcap}"
)
else: # av
if compat:
# Prefer H.264 at the SELECTION level, not just via format_sort: a sort key is only
# a *preference* and can't conjure an H.264 stream a source doesn't expose as
# video-only. A non-YouTube source (e.g. Facebook) may ship AV1 as the best
# video-only stream and H.264 only as a *progressive* (a+v) format — which
# `bestvideo` skips — so `bestvideo+bestaudio` picked AV1 and iOS broke. This chain
# tries avc1 video-only, then the avc1 progressive, then anything.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}+bestaudio[acodec^=mp4a]"
f"/best[vcodec^=avc1]{hcap}"
f"/bestvideo{hcap}+bestaudio/best{hcap}"
)
else:
opts["format"] = f"bestvideo{hcap}+bestaudio/best{hcap}"
if s["container"]:
opts["merge_output_format"] = s["container"]
sort = []
# Keep the codec preference as a tiebreaker (ranks H.264/AAC above resolution among the
# selected candidates); the selector above is the hard guarantee, this is belt-and-braces.
if compat:
sort += ["vcodec:h264", "acodec:aac"]
# avc1 video-only + m4a audio, then the avc1 progressive, then anything (re-encoded).
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}+bestaudio[acodec^=mp4a]"
f"/best[vcodec^=avc1]{hcap}"
f"/bestvideo{hcap}+bestaudio/best{hcap}"
)
opts["merge_output_format"] = "mp4"
# Codec preference tiebreaker (ranks H.264/AAC among the selected candidates); the selector
# above is the hard guarantee, this is belt-and-braces.
sort = ["vcodec:h264", "acodec:aac"]
if h:
sort.append(f"res:{h}")
if s["vcodec"] in _VCODEC_SORT:
sort.append(_VCODEC_SORT[s["vcodec"]])
if sort:
opts["format_sort"] = sort
opts["format_sort"] = sort
# SponsorBlock first (marks segments), then strip them from the media + chapters.
if s["sponsorblock"]:
@@ -215,8 +225,6 @@ def build_ydl_opts(
# jpg/png thumbnail directly and converts anything else to PNG (not the crashing mjpeg2jpeg-to-jpg
# path), so it never needed the convertor and does not reintroduce the crash.
if s["embed_thumbnail"]:
opts["postprocessors"].append(
{"key": "EmbedThumbnail", "already_have_thumbnail": True}
)
opts["postprocessors"].append({"key": "EmbedThumbnail", "already_have_thumbnail": True})
return opts
+73 -35
View File
@@ -15,6 +15,7 @@ Five passes each run:
Runs in the API process (which mounts DOWNLOAD_ROOT). Pure disk/DB work — no YouTube quota.
"""
import logging
import time
from datetime import datetime, timedelta, timezone
@@ -40,9 +41,7 @@ _ORPHAN_MIN_AGE_S = 3600 # only sweep files older than this, so an in-flight do
def _holders(db: Session, asset_id: int) -> list[int]:
return list(
db.execute(
select(DownloadJob.user_id)
.where(DownloadJob.asset_id == asset_id)
.distinct()
select(DownloadJob.user_id).where(DownloadJob.asset_id == asset_id).distinct()
).scalars()
)
@@ -87,11 +86,25 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
cutoff = time.time() - _ORPHAN_MIN_AGE_S
removed = 0
# Throttle the whole-tree walk: this runs in the API process, and an unpaced rglob+stat over a
# large download root monopolizes disk IO long enough to make web requests + playback stutter.
# Yield the IO channel after every `batch` files touched so foreground work interleaves.
pause_s = max(0, settings.download_gc_sweep_pause_ms) / 1000.0
batch = max(1, settings.download_gc_sweep_batch)
touched = 0
def _throttle() -> None:
nonlocal touched
touched += 1
if pause_s and touched % batch == 0:
time.sleep(pause_s)
for entry in root.iterdir():
if entry.name.startswith("."): # .edits / .plex-* / .staging / .storyboards + any dotfile
continue
files = [entry] if entry.is_file() else [p for p in entry.rglob("*") if p.is_file()]
for f in files:
_throttle()
try:
if f.resolve() in keep or f.stat().st_mtime > cutoff:
continue
@@ -101,7 +114,9 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
pass
if entry.is_dir(): # prune emptied dirs (deepest first), then the channel dir itself
for d in sorted(
(p for p in entry.rglob("*") if p.is_dir()), key=lambda p: len(p.parts), reverse=True
(p for p in entry.rglob("*") if p.is_dir()),
key=lambda p: len(p.parts),
reverse=True,
):
try:
d.rmdir()
@@ -121,15 +136,19 @@ def run_download_gc(db: Session) -> dict:
# 1. Pre-expiry warnings.
warn_before = now + timedelta(days=grace_days)
soon = db.execute(
select(MediaAsset).where(
MediaAsset.status == "ready",
MediaAsset.gc_notified.is_(False),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at > now,
MediaAsset.expires_at <= warn_before,
soon = (
db.execute(
select(MediaAsset).where(
MediaAsset.status == "ready",
MediaAsset.gc_notified.is_(False),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at > now,
MediaAsset.expires_at <= warn_before,
)
)
).scalars().all()
.scalars()
.all()
)
for asset in soon:
_notify_holders(db, asset, "download_expiring")
asset.gc_notified = True
@@ -138,13 +157,17 @@ def run_download_gc(db: Session) -> dict:
db.commit()
# 2. Expiry deletion (ready assets past TTL, and any errored leftovers with a TTL).
dead = db.execute(
select(MediaAsset).where(
MediaAsset.status.in_(("ready", "error")),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at <= now,
dead = (
db.execute(
select(MediaAsset).where(
MediaAsset.status.in_(("ready", "error")),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at <= now,
)
)
).scalars().all()
.scalars()
.all()
)
for asset in dead:
_delete_asset(db, asset, "download_expired")
expired += 1
@@ -154,17 +177,24 @@ def run_download_gc(db: Session) -> dict:
# 3. LRU eviction over the total cache cap.
cap = sysconfig.get_int(db, "download_total_max_bytes")
if cap > 0:
total = db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(
MediaAsset.status == "ready"
)
).scalar() or 0
total = (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(
MediaAsset.status == "ready"
)
).scalar()
or 0
)
if total > cap:
candidates = db.execute(
select(MediaAsset)
.where(MediaAsset.status == "ready")
.order_by(MediaAsset.last_access_at.asc().nulls_first())
).scalars().all()
candidates = (
db.execute(
select(MediaAsset)
.where(MediaAsset.status == "ready")
.order_by(MediaAsset.last_access_at.asc().nulls_first())
)
.scalars()
.all()
)
for asset in candidates:
if total <= cap:
break
@@ -176,13 +206,17 @@ def run_download_gc(db: Session) -> dict:
# 4. Reap orphaned errored rows (no file, no holder). get_or_create_asset resets a still-
# referenced errored row to pending on re-enqueue, so these only accumulate once every
# holder is gone; they carry no expires_at, so passes 1-3 never touch them.
orphans = db.execute(
select(MediaAsset).where(
MediaAsset.status == "error",
MediaAsset.ref_count == 0,
MediaAsset.rel_path.is_(None),
orphans = (
db.execute(
select(MediaAsset).where(
MediaAsset.status == "error",
MediaAsset.ref_count == 0,
MediaAsset.rel_path.is_(None),
)
)
).scalars().all()
.scalars()
.all()
)
for asset in orphans:
db.delete(asset) # no file to unlink; FK SET NULL clears any stray job.asset_id
reaped += 1
@@ -194,7 +228,11 @@ def run_download_gc(db: Session) -> dict:
swept = _sweep_orphan_files(db, settings.download_root)
result = {
"warned": warned, "expired": expired, "evicted": evicted, "reaped": reaped, "swept": swept
"warned": warned,
"expired": expired,
"evicted": evicted,
"reaped": reaped,
"swept": swept,
}
log.info("download_gc %s", result)
return result
+2
View File
@@ -1,4 +1,5 @@
"""Short-lived-session DB helpers + staging paths shared by the worker parent and the child pipeline."""
from pathlib import Path
from sqlalchemy import text
@@ -10,6 +11,7 @@ from app.models import DownloadJob, MediaAsset
# --- small DB helpers (short-lived sessions; the download itself holds no transaction) -------
def _set_job(job_id: int, **fields) -> None:
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
+16 -5
View File
@@ -6,6 +6,7 @@ require a short-lived signed **grant** on the file request — because a `<video
header, the password is exchanged once at `/unlock` for an HMAC grant appended to the file URL,
so the raw password never rides in the media URL (or server logs).
"""
import hashlib
import hmac
import secrets
@@ -27,10 +28,9 @@ def is_expired(link: DownloadLink) -> bool:
# --- signed grants (password-protected file access) ----------------------------------------
def _sign(msg: str) -> str:
return hmac.new(
settings.secret_key.encode(), msg.encode(), hashlib.sha256
).hexdigest()[:32]
return hmac.new(settings.secret_key.encode(), msg.encode(), hashlib.sha256).hexdigest()[:32]
def make_grant(token: str, password_version: int, now: float | None = None) -> str:
@@ -58,6 +58,7 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
# --- serialization -------------------------------------------------------------------------
def owner_view(link: DownloadLink, base_url: str = "") -> dict:
"""What the file owner sees when managing their links."""
return {
@@ -72,14 +73,20 @@ def owner_view(link: DownloadLink, base_url: str = "") -> dict:
}
def public_meta(link: DownloadLink, job, asset, file_url: str, poster_url: str | None = None) -> dict:
def public_meta(
link: DownloadLink, job, asset, file_url: str, poster_url: str | None = None
) -> dict:
"""Metadata for the public watch page (only ever returned once access is authorized).
Resolves the owner's per-download display overrides (custom title/channel) over the asset's
auto-extracted values, and surfaces the source + any extra reference links as clickable URLs."""
from app.downloads import service
from app.downloads import mediakind, service
source_url = service.reference_url(job, asset)
# Audio-only assets (mp3 rips, extracted audio) can't play inside a <video> element on WebKit
# (all iOS browsers), so the watch page renders a native <audio> instead. A generic uploaded
# file (media_kind='file') has no player at all — the watch page shows file info + a download.
# Both branches read the stored media_kind (the single source), not a re-derived codec check.
return {
"title": (job and job.display_name) or asset.title,
"uploader": (job and job.display_uploader) or asset.uploader,
@@ -89,6 +96,10 @@ def public_meta(link: DownloadLink, job, asset, file_url: str, poster_url: str |
"duration_s": asset.duration_s,
"width": asset.width,
"height": asset.height,
"audio": asset.media_kind == mediakind.AUDIO,
"media_kind": asset.media_kind,
"size_bytes": asset.size_bytes,
"container": asset.container,
"allow_download": link.allow_download,
"file_url": file_url,
"poster_url": poster_url,
+23
View File
@@ -0,0 +1,23 @@
"""The media-kind discriminator for a library asset — the single home of one invariant.
`MediaAsset.media_kind` is `"video" | "audio" | "file"`. It decides which library actions apply:
media (video/audio) gets play/edit/share; a generic uploaded file gets download + share only, and
is never rendered inline (see the force-download serving in the file/watch routes). Every finalize
path computes it HERE — the download worker + the upload ingest via `derive`, the editor by
carrying the source's kind over — so the rule lives in code, not in a comment that can drift (an
earlier ad-hoc `vcodec is None and acodec is not None` audio check lived only in links.py).
"""
VIDEO = "video"
AUDIO = "audio"
FILE = "file"
def derive(vcodec: str | None, acodec: str | None) -> str:
"""Media kind from the probed codecs: a video stream -> video, audio-only -> audio, neither
(a non-media upload: pdf/zip/docx/…) -> file."""
if vcodec:
return VIDEO
if acodec:
return AUDIO
return FILE
+2 -3
View File
@@ -6,6 +6,7 @@ This injects per-video OG/Twitter tags into the served HTML so a shared link unf
video's title, channel and thumbnail. Real browsers ignore the extra tags and hydrate the SPA as
usual. Password-protected / expired / invalid links fall back to the generic card (no metadata leak).
"""
import html
import re
from pathlib import Path
@@ -71,9 +72,7 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
def _watch_tags(token: str, db: Session) -> str | None:
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
link = db.execute(
select(DownloadLink).where(DownloadLink.token == token)
).scalar_one_or_none()
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
if link is None or linksmod.is_expired(link):
return None
url = f"{settings.app_base}/watch/{token}"
+93 -25
View File
@@ -1,4 +1,5 @@
"""Child-side download/edit pipeline — the yt-dlp download + ffmpeg edit that runs in the spawned subprocess."""
import logging
import re
import shutil
@@ -7,13 +8,14 @@ from datetime import date, datetime, timedelta, timezone
from pathlib import Path
import yt_dlp
from sqlalchemy import select
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import errors as dlerrors
from app.downloads import adapters, formats, service, storage
from app.downloads import adapters, credentials, formats, mediakind, service, storage
from app.downloads.jobdb import (
_edit_staging_dir,
_job_status,
@@ -57,9 +59,7 @@ _YTDLP_TEMP_RE = re.compile(
)
def _notify_download_done(
db, user_id: int | None, title: str | None, job_id, *, kind: str
) -> None:
def _notify_download_done(db, user_id: int | None, title: str | None, job_id, *, kind: str) -> None:
"""Post a best-effort 'ready' notice to the owner's inbox — downloads/edits used to finish
silently while scheduler jobs and playlist-push already notified (R11 feedback symmetry). A
notification failure must never mask the completed job, so it is isolated + rolled back (mirrors
@@ -113,7 +113,7 @@ def _fill_asset_meta_early(asset_id: int, info: dict) -> None:
def _make_progress_hook(job_id: int, asset_id: int):
state = {"meta_done": False}
cancelled = _cancel_poll(job_id) # same policy as the ffmpeg path
cancelled = _cancel_poll(job_id) # same policy as the ffmpeg path
allow_write = _rate_limiter(_DB_WRITE_INTERVAL_S)
def hook(d: dict) -> None:
@@ -247,7 +247,9 @@ def _make_pp_hook(job_id: int):
def hook(d: dict) -> None:
if d.get("status") in ("started", "processing"):
_set_job(job_id, phase=_pp_phase(d.get("postprocessor") or ""), speed_bps=None, eta_s=None)
_set_job(
job_id, phase=_pp_phase(d.get("postprocessor") or ""), speed_bps=None, eta_s=None
)
return hook
@@ -269,7 +271,9 @@ def _produced_file(info: dict, staging: Path) -> Path:
back to the largest file left in staging that is neither a thumbnail nor one of yt-dlp's own
working files (see `_YTDLP_TEMP_RE`)."""
req = (info.get("requested_downloads") or [{}])[0]
fp = req.get("filepath") or req.get("_filename") or info.get("filepath") or info.get("_filename")
fp = (
req.get("filepath") or req.get("_filename") or info.get("filepath") or info.get("_filename")
)
if fp:
produced = Path(str(fp))
if produced.is_file():
@@ -301,17 +305,23 @@ def _media_info(info: dict) -> dict:
return {**info, **chosen}
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter):
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter, cookiefile=None):
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
error); other sources make a single attempt. Returns the info dict of the successful attempt."""
error); other sources make a single attempt. Returns the info dict of the successful attempt.
`cookiefile` (S6) is the acting user's per-host cookies.txt, already gated by
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated."""
client_sets = adapter.client_sets()
outtmpl = str(staging / "%(id)s.%(ext)s")
last_exc: Exception | None = None
for i, clients in enumerate(client_sets):
opts = formats.build_ydl_opts(
spec, outtmpl, _make_progress_hook(job_id, asset_id),
spec,
outtmpl,
_make_progress_hook(job_id, asset_id),
extra=adapter.ydl_opts_extra(clients),
cookiefile=cookiefile,
)
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
_touch_job(job_id) # stall-watchdog heartbeat: extraction writes no progress, so a slow /
@@ -324,7 +334,12 @@ def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter):
except yt_dlp.utils.DownloadError as exc:
last_exc = exc
if i < len(client_sets) - 1 and adapter.retriable(str(exc)):
log.info("job %s: clients %s failed (%s); retrying with fallback", job_id, clients, str(exc)[:70])
log.info(
"job %s: clients %s failed (%s); retrying with fallback",
job_id,
clients,
str(exc)[:70],
)
for p in list(staging.iterdir()): # drop partials before the retry
try:
p.unlink()
@@ -363,13 +378,17 @@ def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict
dest = produced.with_name(produced.stem + ".browser.mp4")
action = "re-encode" if (reenc_v or reenc_a) else "remux"
log.info("job %s: %s (v=%s a=%s mp4=%s) -> browser-safe mp4", job_id, action, vcodec, acodec, is_mp4)
log.info(
"job %s: %s (v=%s a=%s mp4=%s) -> browser-safe mp4", job_id, action, vcodec, acodec, is_mp4
)
_set_job(job_id, phase="processing", progress=0, speed_bps=None, eta_s=None)
# Prefer the info duration; probe the file when the extractor didn't provide one, so a long
# re-encode still shows real progress instead of a frozen 0%.
total_s = int(info["duration"]) if info.get("duration") else editmod.probe_duration(produced)
editmod.run_ffmpeg(
editmod.build_browser_safe_cmd(produced, dest, reenc_v, reenc_a),
editmod.build_browser_safe_cmd(
produced, dest, reenc_v, reenc_a, threads=editmod.encode_thread_cap()
),
total_s=total_s,
on_progress=_ffmpeg_progress_writer(job_id, "processing"),
should_cancel=_cancel_poll(job_id),
@@ -383,8 +402,11 @@ def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict
return dest
def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spec: dict) -> None:
def _download(
job_id: int, asset_id: int, source_kind: str, source_ref: str, spec: dict, user_id: int
) -> None:
staging = _staging_dir(asset_id)
cookiefile: Path | None = None
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
@@ -393,7 +415,27 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
_set_job(job_id, phase="downloading", progress=0)
adapter = adapters.resolve_adapter(source_kind, source_ref)
url = service.source_url(source_kind, source_ref)
info = _media_info(_extract_with_fallback(job_id, asset_id, url, spec, staging, adapter))
# S6: attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
# this asset is that user's private, salted identity (the double gate lives in credentials).
with SessionLocal() as cdb:
asset_sig = cdb.execute(
select(MediaAsset.format_sig).where(MediaAsset.id == asset_id)
).scalar_one_or_none()
if asset_sig is not None:
cookiefile = credentials.resolve_cookiefile(
cdb, user_id, url, asset_sig, spec, staging
)
info = _media_info(
_extract_with_fallback(
job_id,
asset_id,
url,
spec,
staging,
adapter,
cookiefile=str(cookiefile) if cookiefile else None,
)
)
# Re-resolve now that yt-dlp's info reveals the true source (a bare/redirected url only
# identifies post-extraction) so the .nfo uniqueid type + uploader_url agree on one source.
adapter = adapters.resolve_adapter(source_kind, source_ref, info)
@@ -436,8 +478,13 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
asset.width = int(info["width"]) if info.get("width") else None
asset.height = int(info["height"]) if info.get("height") else None
asset.container = ext
asset.vcodec = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
asset.acodec = info.get("acodec") if info.get("acodec") not in (None, "none") else None
asset.vcodec = (
info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
)
asset.acodec = (
info.get("acodec") if info.get("acodec") not in (None, "none") else None
)
asset.media_kind = mediakind.derive(asset.vcodec, asset.acodec)
asset.title = meta.title
asset.uploader = meta.uploader
asset.uploader_url = adapter.uploader_url(info)
@@ -456,8 +503,13 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
job = db.get(DownloadJob, job_id)
# Capture the owner before commit expires the row (avoids a refresh SELECT just to notify).
owner_id = job.user_id if job else None
fields = {"status": JobStatus.DONE, "progress": 100, "phase": None, "speed_bps": None,
"eta_s": None}
fields = {
"status": JobStatus.DONE,
"progress": 100,
"phase": None,
"speed_bps": None,
"eta_s": None,
}
if job and not job.display_name and meta.title:
# display_name is VARCHAR(255) — clip a long title (see _fill_asset_meta_early).
fields["display_name"] = meta.title[:255]
@@ -484,12 +536,20 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
# Atomic fail: only while still running, so a route-set cancel/pause is never overwritten
# with `error` (error also holds the asset → would re-open the double-release class).
service.finalize_if_running(
db, job_id, status=JobStatus.ERROR, error=msg,
error_code=dlerrors.classify(msg), phase=None,
db,
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
phase=None,
)
db.commit()
log.warning("job %s failed: %s", job_id, msg)
finally:
# Unlink the decrypted cookie file explicitly (the rmtree below removes the whole staging
# tree too, but never leave a plaintext session cookie on disk a moment longer than needed).
if cookiefile is not None:
cookiefile.unlink(missing_ok=True)
shutil.rmtree(staging, ignore_errors=True)
@@ -501,8 +561,12 @@ def _fail_edit(job_id: int, asset_id: int, msg: str) -> None:
asset.error = msg
# Atomic fail: only while still running (see the download error path).
service.finalize_if_running(
db, job_id, status=JobStatus.ERROR, error=msg,
error_code=dlerrors.classify(msg), phase=None,
db,
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
phase=None,
)
db.commit()
log.warning("edit job %s failed: %s", job_id, msg)
@@ -517,7 +581,9 @@ def _process_edit(job_id: int, asset_id: int) -> None:
user_id = job.user_id if job else None
src_ready = src is not None and src.status == "ready" and bool(src.rel_path)
src_rel = src.rel_path if src else None
src_w, src_h, src_dur = (src.width, src.height, src.duration_s) if src else (None, None, None)
src_w, src_h, src_dur = (
(src.width, src.height, src.duration_s) if src else (None, None, None)
)
if not src_ready or not src_rel:
_fail_edit(job_id, asset_id, "The source download is no longer available.")
@@ -541,7 +607,9 @@ def _process_edit(job_id: int, asset_id: int) -> None:
_set_job(job_id, phase="editing", progress=0, speed_bps=None, eta_s=None)
if edit_spec.get("segments"):
cmd, prep = editmod.build_concat_plan(src_path, dest_staging, edit_spec, out_ext, staging)
cmd, prep = editmod.build_concat_plan(
src_path, dest_staging, edit_spec, out_ext, staging
)
for p, content in prep.items():
p.write_text(content, encoding="utf-8")
else:
+17
View File
@@ -8,6 +8,7 @@ Limits come from a per-user DownloadQuota row if the admin set one, else the sys
Enforcement split: max_jobs + max_bytes are checked at enqueue (they bound holdings); per-user
max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads).
"""
from dataclasses import dataclass
from sqlalchemy import func, select
@@ -121,6 +122,22 @@ def check_enqueue(db: Session, user_id: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, used)
def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
"""Like check_enqueue, but for an upload whose size is KNOWN at ingest: also reject when adding
this file would push the footprint over max_bytes. A download can't pre-check its size (unknown
until the worker finishes) so check_enqueue only compares the current footprint; an upload can,
so it's charged against the cap before it lands. Raises QuotaExceeded(max_jobs|max_bytes)."""
lim = resolve(db, user_id)
if lim.unlimited:
return
n = active_jobs(db, user_id)
if n >= lim.max_jobs:
raise QuotaExceeded("max_jobs", lim.max_jobs, n)
projected = footprint(db, user_id) + add_bytes
if projected > lim.max_bytes:
raise QuotaExceeded("max_bytes", lim.max_bytes, projected)
def usage(db: Session, user_id: int) -> dict:
"""Compact usage snapshot for the UI (footprint + counts vs limits)."""
lim = resolve(db, user_id)
+35 -1
View File
@@ -10,17 +10,50 @@ Contract: the asset is already claimed ('downloading') by the parent. This proce
writes the terminal job/asset state itself (done / error, or a cooperative pause via the in-flight
progress hook), then exits 0. An unexpected crash exits non-zero and the parent fails the job.
"""
import logging
import os
import shutil
import subprocess
import sys
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.models import DownloadJob
log = logging.getLogger("siftlode.worker.runner")
def _deprioritize() -> None:
"""Lower THIS child's CPU + IO priority so its yt-dlp/ffmpeg work — every descendant inherits the
handicap — can never starve the co-located API, Postgres, or Plex playback. Run once at startup,
before any heavy work. Every step is best-effort: a failed renice must never fail a download."""
nice_inc = settings.download_child_nice
if nice_inc > 0:
try:
os.nice(nice_inc) # relative increment; the child starts at the worker's nice (0)
except OSError as exc:
log.info("could not renice download child: %s", str(exc)[:120])
cls = settings.download_child_ionice_class
if cls > 0 and shutil.which("ionice"): # util-linux; absent on some minimal images
try:
subprocess.run(
["ionice", "-c", str(cls), "-p", str(os.getpid())],
capture_output=True,
timeout=5,
check=False,
)
except (OSError, subprocess.SubprocessError) as exc:
log.info("could not ionice download child: %s", str(exc)[:120])
# Cap OpenMP-parallel ffmpeg filters to the same budget as the encoder threads, so a filter
# graph can't fan out across every core behind ffmpeg's own `-threads` cap.
os.environ.setdefault("OMP_NUM_THREADS", str(editmod.encode_thread_cap()))
def main() -> None:
logging.basicConfig(level=logging.INFO)
_deprioritize()
if len(sys.argv) != 2:
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
raise SystemExit(2)
@@ -34,6 +67,7 @@ def main() -> None:
job_kind = job.job_kind
spec = dict(job.profile_snapshot or {})
source_kind, source_ref = job.source_kind, job.source_ref
user_id = job.user_id
if asset_id is None:
return
@@ -44,7 +78,7 @@ def main() -> None:
if job_kind == "edit":
_process_edit(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec)
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
if __name__ == "__main__":
+18 -3
View File
@@ -6,6 +6,7 @@ cache hit (asset already ready) — marks the job done instantly with no re-down
handles everything else. State transitions used by the routes (pause/resume/cancel/delete) and
the worker live in `worker.py` / the routes; this module owns enqueue + asset bookkeeping.
"""
from datetime import datetime, timezone
from sqlalchemy import func, select, update
@@ -14,8 +15,9 @@ from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import adapters
from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import formats, quota, storage
from app.downloads import formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobStatus
from app.models import Channel, DownloadJob, MediaAsset, Video
@@ -154,7 +156,12 @@ def enqueue(
Raises quota.QuotaExceeded if the user is at their job-count or footprint cap."""
quota.check_enqueue(db, user_id)
spec = formats.normalize(spec)
sig = formats.format_sig(spec)
# If this user has a stored cookie for the download's host, salt the cache signature with their
# id so the cookie-authed result is a PRIVATE asset — never dedup-shared with, or cache-served
# to, another user (the load-bearing privacy fix; the worker attaches the cookie to exactly this
# salted identity). A normal download (no matching cookie) gets salt=None and shares as before.
salt = credentials.salt_for_enqueue(db, user_id, source_url(source_kind, source_ref))
sig = formats.format_sig(spec, salt=salt)
asset = get_or_create_asset(db, source_kind, source_ref, sig)
# Show a real title + thumbnail the moment it's queued (feed downloads are catalog videos).
populate_from_catalog(db, asset)
@@ -166,7 +173,8 @@ def enqueue(
source_ref=source_ref,
profile_id=profile_id,
profile_snapshot=spec,
display_name=display_name or (asset.title[:255] if asset.status == "ready" and asset.title else None),
display_name=display_name
or (asset.title[:255] if asset.status == "ready" and asset.title else None),
status="queued",
queue_pos=next_queue_pos(db),
)
@@ -208,6 +216,10 @@ def enqueue_edit(
src_asset = db.get(MediaAsset, source_job.asset_id) if source_job.asset_id else None
if src_asset is None or src_asset.status != "ready" or not src_asset.rel_path:
raise EditError("source_not_ready")
# A generic uploaded file (media_kind='file') has no timeline to trim/crop — the editor path is
# media-only. Reject here so an upload's file never reaches the ffmpeg edit worker.
if src_asset.media_kind == mediakind.FILE:
raise EditError("not_media")
spec = editmod.normalize_edit_spec(edit_spec)
if not spec:
@@ -224,6 +236,9 @@ def enqueue_edit(
asset.thumbnail_url = src_asset.thumbnail_url
asset.upload_date = src_asset.upload_date
asset.duration_s = editmod.clip_duration(spec, src_asset.duration_s)
# A clip inherits its source's kind (a video edit stays video, an audio trim stays audio);
# the ffmpeg edit worker doesn't re-probe codecs, so set it here where the source is known.
asset.media_kind = src_asset.media_kind or mediakind.VIDEO
job = DownloadJob(
user_id=user_id,
+101 -7
View File
@@ -10,6 +10,7 @@ Plus a Kodi/Plex-readable `<basename>.nfo` (episodedetails) and `<basename>.jpg`
a Plex library using a metadata agent picks up rich info. Channel = "show", video = episode,
date-based — the community-standard mapping for YouTube-in-Plex.
"""
import re
import shutil
import unicodedata
@@ -18,6 +19,8 @@ from datetime import date
from pathlib import Path
from xml.sax.saxutils import escape
from app.downloads import mediakind
# Reserve room for the "_id.ext" suffix + parent dirs within the 255-char component limit.
_MAX_TITLE = 120
# User-facing (Content-Disposition) names can be longer than an on-disk component.
@@ -36,9 +39,27 @@ _DROP_CATEGORIES = {"So", "Sk", "Cc", "Cf", "Cs", "Co", "Cn"}
# that accented names show in non-UTF-8 tools.
_FOLD_MAP = str.maketrans(
{
"ß": "ss", "ẞ": "SS", "ø": "o", "Ø": "O", "ł": "l", "Ł": "L", "đ": "d", "Đ": "D",
"ð": "d", "Ð": "D", "þ": "th", "Þ": "Th", "æ": "ae", "Æ": "AE", "œ": "oe", "Œ": "OE",
"ı": "i", "İ": "I", "·": " ", "–": "-", "—": "-",
"ß": "ss",
"ẞ": "SS",
"ø": "o",
"Ø": "O",
"ł": "l",
"Ł": "L",
"đ": "d",
"Đ": "D",
"ð": "d",
"Ð": "D",
"þ": "th",
"Þ": "Th",
"æ": "ae",
"Æ": "AE",
"œ": "oe",
"Œ": "OE",
"ı": "i",
"İ": "I",
"·": " ",
"–": "-",
"—": "-",
}
)
@@ -128,6 +149,81 @@ def edit_rel_path(user_id: int, asset_id: int, ext: str) -> str:
return f".edits/{user_id}/clip_{asset_id}.{ext}"
def _clean_upload_ext(filename: str) -> str:
"""The upload's extension reduced to a shell-safe, lowercased, bounded token (``""`` when the
upload has no extension). The SINGLE definition feeding both the on-disk suffix and the stored
`container` — they must agree (the served Content-Disposition suffix has to match the container),
so the sanitize/truncation rule lives in one place, not two drifting copies."""
return _UNSAFE.sub("", Path(filename).suffix.lstrip(".")).lower()[:16]
def upload_rel_path(user_id: int, asset_id: int, filename: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a locally-uploaded file.
Kept in a per-user `.uploads/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (an upload is arbitrary user content, not a canonical episode; no .nfo/poster
episode sidecars). The name is system-decided and bound to the asset id (`_a{id}` keeps two
same-named uploads apart); the original filename is display metadata only. The extension is
preserved (shell-safe, bounded) so the served file + its Content-Disposition carry a correct
suffix; an extensionless upload stays extensionless."""
stem = sanitize(Path(filename).stem, 80)
ext = _clean_upload_ext(filename)
suffix = f".{ext}" if ext else ""
return f".uploads/{user_id}/{stem}_a{asset_id}{suffix}"
def upload_ext(filename: str) -> str | None:
"""The stored `container` for an uploaded file: its shell-safe lowercased extension, or None when
the upload has no extension (a bare name). Shares `_clean_upload_ext` with `upload_rel_path` so the
stored container and the on-disk suffix can never diverge."""
return _clean_upload_ext(filename) or None
# The Content-Type used to serve an inline-playable media asset, keyed by its stored `container`.
# DELIBERATELY a strict allow-list of known audio/video containers: the served type is chosen from
# the asset's ffprobe-verified media_kind + this table, NEVER guessed from the raw (user-controlled)
# upload extension. Without this, an uploaded media/HTML polyglot named "evil.html" — a real MP3/MP4
# that ffprobe classifies as media, so it dodges the generic-file force-download — would be served
# with a Content-Type Starlette guesses from ".html" (text/html) and execute as stored XSS on the
# serving origin. A media asset whose container isn't in this table falls back to a neutral
# force-download attachment (see `served_media_type`), so nothing script-executable is ever inline.
_MEDIA_MIME: dict[str, str] = {
# video containers
"mp4": "video/mp4",
"m4v": "video/mp4",
"mov": "video/quicktime",
"webm": "video/webm",
"mkv": "video/x-matroska",
"ogv": "video/ogg",
"avi": "video/x-msvideo",
# audio containers
"m4a": "audio/mp4",
"mp3": "audio/mpeg",
"aac": "audio/aac",
"ogg": "audio/ogg",
"oga": "audio/ogg",
"opus": "audio/ogg",
"weba": "audio/webm",
"wav": "audio/wav",
"flac": "audio/flac",
}
def served_media_type(media_kind: str | None, container: str | None) -> str | None:
"""The Content-Type a library file MUST be served with — a real audio/video MIME for a recognised
media container, else ``None`` meaning "serve as a neutral force-download attachment".
Returns None for a generic ``file`` kind AND for any media whose container we don't positively
recognise as safe. Callers serve None as ``application/octet-stream`` + ``attachment`` so the
served Content-Type is NEVER derived from the (attacker-controlled) file extension — the guard
that stops uploaded HTML/SVG or a media/HTML polyglot from executing inline on the serving
origin. A ``None`` media_kind (a legacy asset before the discriminator was backfilled) is treated
as media so an old download keeps its inline player."""
if media_kind == mediakind.FILE:
return None
return _MEDIA_MIME.get((container or "").lower())
def download_filename(display_name: str, container: str | None, path: Path) -> str:
"""The Content-Disposition filename for a served download: the cleaned display name carrying a single
correct extension — the asset container, else the file's own suffix — without doubling it when the
@@ -174,7 +270,7 @@ def _nfo_xml(meta: MediaMeta) -> str:
f" <premiered>{aired}</premiered>",
f" <year>{year}</year>",
f" <plot>{escape(meta.description or '')}</plot>",
f" <uniqueid type=\"{escape(meta.uniqueid_type)}\" default=\"true\">{escape(meta.video_id)}</uniqueid>",
f' <uniqueid type="{escape(meta.uniqueid_type)}" default="true">{escape(meta.video_id)}</uniqueid>',
]
if meta.duration_s:
parts.append(f" <runtime>{round(meta.duration_s / 60)}</runtime>")
@@ -190,9 +286,7 @@ def _sidecar(media: Path, ext: str) -> Path:
return media.with_suffix(ext)
def write_sidecars(
download_root: str, rel: str, meta: MediaMeta, thumb_src: Path | None
) -> bool:
def write_sidecars(download_root: str, rel: str, meta: MediaMeta, thumb_src: Path | None) -> bool:
"""Write `<basename>.nfo` + `<basename>.jpg` next to the media file. Returns True if written."""
media = abs_path(download_root, rel)
try:

Some files were not shown because too many files have changed in this diff Show More