Compare commits

..
106 Commits
Author SHA1 Message Date
peter 7339a0b84b release: 0.79.0 (improvement/e9-playlist)
gate / frontend (push) Failing after 1m43s
gate / backend (push) Successful in 2m48s
2026-09-03 21:59:31 +02:00
peter c960c07a6a release: 0.79.0 notes + version bump
E9 Playlist module: total playtime in both playlist headers, the header
compacted to two rows, and the sticky column header sitting flush. Plus the
six defects the epic-end deep review confirmed.
2026-09-03 21:59:09 +02:00
peter 3cfdb36f06 test(format): pin the sub-minute truncation (E9 R2)
Round 2 mutation-checked round 1's own fix and found the one line it left
uncovered: every table row was an integer, so `Math.floor(sec)` in the new
seconds tier could be deleted or swapped for round/ceil with the suite still
green — exactly the "later cleanup removes it as redundant" failure the fix's
comment claims to prevent. Two fractional rows kill all three mutants.
2026-09-03 21:51:14 +02:00
peter ada95d04a4 fix(playlists): one rendering for a playlist total, honest below a minute (E9 R1)
Epic-end deep review, round 1. Six findings survived verification; five of them
were one defect wearing different hats.

formatRuntime floored 1-59s to "0m". Harmless while it only ever formatted a
Plex movie's length, but 12D repurposed it for an aggregate, and a total is the
only input that can land under a minute: a playlist of Shorts reported "0m"
while the rail beside it read "0:45". Fixed in the function, not behind a
`>= 60` guard at each of the seven call sites — the invariant belongs where it
cannot be forgotten. Its contract comment claimed "Plex movie/episode lengths"
and was two epics stale.

The rail rendered the same playlist total as formatDuration (1:23:45) while the
new header rendered formatRuntime (1h 23m) — two shapes of one number, side by
side on one screen. The rail now uses formatRuntime too; formatTotalHours had
already recorded that H:MM:SS is wrong for an aggregate.

The rename input kept min-w-0 without flex-1 in the compacted row, so it was
the only shrinkable child beside a shrink-0 action cluster. The Plex half of
this same epic already had the fix; this restores the symmetry.

plex.browse.items had no plural forms, so a one-item Plex playlist read
"1 items". Pre-existing, but the new header is the call site where count=1 is
ordinary. Both languages get both forms, no key drift.

formatRuntime had no unit tests at all across seven call sites. Added a tier
table plus a sweep asserting no non-zero second ever renders as "0m".

Dropped after 3/3 adversarial refutations: the meta-line segment ordering
(the filtered count is labelled, and count-then-runtime is the convention) and
the null-duration lower bound (the server's own SUM has skipped NULLs since
long before this branch; changing only the client would split them).
2026-09-03 21:43:04 +02:00
peter 1b004e6ccf feat(plex): total playtime in the playlist header (E9 S3)
Parity with the YouTube playlist header (E9 12D): the Plex playlist detail
now carries a meta line under the title — "N items · Xh Ym", summed from the
items' duration_seconds. No backend change: leaf_card already serializes
duration_seconds for both movies and episodes, and no new i18n key either —
the count reuses plex.browse.items, the same key the collection header uses.

The title/rename input moved into a min-w-0 flex-1 column so the meta line
sits under both; the line is hidden on an empty playlist.
2026-09-02 18:30:06 +02:00
peter 95c33e0b0f refactor(playlists): compact the playlist header to two rows (E9 12A)
The header portals into the shell's FIXED toolbar band, so its height is
list height the user never gets back — and it was spending four stacked
blocks on it: title, meta line, an action row, then a separate view-control
row under the cover. Option B collapses that into two tight rows beside a
smaller cover: title + rename/YouTube-link on the first, with Play-all,
push/revert/delete and the unsynced badge pushed right; the meta line
(count · playtime · filtered) and the view controls (group-by, reset view,
save order, view status, undo/redo) sharing the second.

Every control and its gating is unchanged — the view-control group still
appears only for `items.length > 1 || viewActive`, so a one-item playlist
shows just the meta line.

Measured on a 32-item playlist: the band goes 170px -> 88px at rest
(122px while a view is active, where the status sentence wraps), which is
~80px more visible list.
2026-09-02 03:45:07 +02:00
peter 4e6bcbc078 fix(playlists): stop the table header shifting up on first scroll (E9 12B)
The playlist content wrapper carried `pt-3`, so its sticky column header sat
12px below the scroller top at rest and jumped up to `top:0` on the first
scroll. Drop the top padding so the header is flush from rest and never
shifts — the fixed toolbar band above supplies the breathing room, matching
the channel table's already-correct pattern.
2026-09-02 03:39:17 +02:00
peter e17fb5bbf5 feat(playlists): total playtime in header + sticky-header fade fix (E9 S1)
Surface the playlist's total playtime beside the item count in the detail
header (12D) — sums every item's duration (formatRuntime "h m", matching the
approved mockup), hidden when nothing sums. Data is already client-side; no
backend change.

Pass fadeTop:false for the playlists page (12B) so the shell's top edge-fade
no longer sits over the already-sticky column header (tableHeader.tsx:257),
matching the channels/audit pages.
2026-09-02 03:39:17 +02:00
peter a9044ece50 release: 0.78.8 (improvement/email-html-alternative)
gate / frontend (push) Failing after 1m40s
gate / backend (push) Successful in 2m40s
2026-09-02 03:24:59 +02:00
peter 473aafd1c8 release: bump VERSION to 0.78.8 + release notes 2026-09-02 03:24:56 +02:00
peter 7019e3afdf fix(email): give every mail an HTML alternative so its link is clickable
The bodies are text/plain only, so whether the URL in a verification or
password-reset mail is a link at all was the client's decision. Gmail, Outlook and
Apple Mail linkify plain text; lighter web clients render the body verbatim and leave
the link as dead text — in a message whose entire purpose is that one link.

_send now attaches an HTML part generated from the SAME plain body, rather than
per-template HTML that would have to be kept in sync with it: one converter, all
eleven templates, and text/plain stays the primary part and the source of truth.
Escaping runs before linkification — bodies interpolate untrusted values (a
requester's address, the operator contact) — so no caller can inject markup, and a
query string's `&` is already `&` when it reaches the href, which is what an HTML
attribute wants. Sentence-final punctuation is trimmed out of the href while
url-safe token characters (`-`, `_`) are left alone. Inline styles only, no images,
no external CSS, no tracking.
2026-09-02 03:13:23 +02:00
peter c81b110637 release: 0.78.7 (bug/verify-link-and-approval-lockout)
gate / frontend (push) Failing after 1m44s
gate / backend (push) Successful in 2m42s
2026-09-02 02:31:19 +02:00
peter 2ecc70ca6d release: bump VERSION to 0.78.7 + release notes 2026-09-02 02:31:14 +02:00
peter f8c63e80c2 fix(auth): widen the 0078 backfill so revoke victims who verified are rescued too
Review found the backfill's fourth predicate `AND email_verified = false` strands
exactly the subset that needs it most. The revoke does not invalidate the account's
outstanding verification token, so clicking the emailed link is the natural next
thing such a user does — and it sets email_verified=True while password_hash stays
NULL. That row then has no route back at all: reset ineligible, resend refuses an
already-verified address, set-password needs a session the account cannot obtain,
and nothing re-stamps it because the revoke itself is guarded on being unverified.

The remaining three predicates already pin the victim set exactly: registration and
the setup wizard always set a hash, every Google row carries a google_sub (adoption
assigns it before revoking), and nothing in the codebase ever clears google_sub — so
a local, non-demo, passwordless row can only have come from the revoke. Widening is
therefore safe, and it is the argument the docstring was already making while the
SQL contradicted it.

Also corrects that docstring: set-password is unreachable because the account cannot
authenticate at all, not because the epoch bump killed a session — an unverified
account is refused at login before a session is ever minted, so it never had one.
2026-09-02 02:21:46 +02:00
peter 3ac3d382e7 fix(auth): stop the address-bar mirror eating emailed auth tokens, and unlock revoked accounts
Two independent bugs made an email+password registration unrecoverable in prod.

The verification link did nothing. urlWithParams/clearParams rebuilt the URL as
pathname+query and dropped the fragment — but the one-time auth tokens ride the
fragment (#verify=/#reset=, SB3), and the providers that mirror nav params
(useUrlMirror in FeedFilters/Playlists/Plex/Channels) sit ABOVE the lazily-loaded
<App> in main.tsx. Their mount effect replaceState'd the token away before the auth
card's chunk had loaded, so the card read an empty hash: the link opened a bare
sign-in page, no request, no banner. Both helpers now carry the fragment over.

The account was then locked out for good. SEC-10 nulls password_hash when an
unverified account is attested (here: admin approval landing before the user could
verify — which the bug above made impossible). That is indistinguishable from a
Google-only account, and reset eligibility keyed off password_hash alone, so no
reset link was ever issued: no password to sign in with, and no way to get one.
users.password_revoked_at records the revocation, reset eligibility honours it, and
setting a password again (reset-confirm or set-password) clears it. An account that
never had a password stays ineligible on purpose — its owner adds one from Settings.

/auth/verify now also answers password_reset_required so the card can say why the
registered password stopped working, instead of leaving the user at the uniform
"Invalid email or password". Telling the holder of a valid verify token leaks
nothing: they have already proved they own the mailbox.

The 0078 migration backfills the stamp for rows carrying the exact lockout signature
(local, non-demo, no password, unverified), so existing victims recover unaided.
2026-09-02 01:28:45 +02:00
peter 94c62ca699 release: 0.78.6 (feat/admin-invite-delete-email-health)
gate / frontend (push) Failing after 1m43s
gate / backend (push) Successful in 2m53s
2026-09-02 00:36:52 +02:00
peter 1fe24d374a docs(tests): correct the deny-vs-delete rationale in the invites test file
Round 4's only finding: the file header and one comment still said Deny stops a
refused person re-asking. It does not — upsert_pending_invite reopens a denied
row as pending — and the delete_invite docstring in the same branch already says
so. The real difference is that Deny records an auditable decision and mails the
requester; Delete forgets the address.
2026-09-02 00:15:21 +02:00
peter 2cba6b9fca docs: round-3 copy corrections, and pin the two behaviours nothing was guarding
Round 3's correctness/security pass came back clean; what was left said more than
the code does.

- emailOff promised the access-request list is where sign-ups show up. A sign-up
  on an address that is already approved or already pending adds nothing to it
  (roadmap UX-25), and "only" was falsified by this release's own inbox notice.
- deleteConfirm claimed the admin was dropping "an approval you already gave" —
  it is the confirmation for pending and denied rows too.
- sendFailed blamed the mail server for rejecting a message; the reason covers
  connect, DNS, TLS and login failures alike, so it now describes the symptom and
  leaves the diagnosis to the log.
- ADMIN_EMAILS re-promotion happens in the Google callback only, not on any
  sign-in.

Tests: the decided-list copy of the delete affordance was never rendered (both
new cases fail if that branch is dropped), and the throttle release — a
deliberate behaviour the notes advertise — was asserted by nothing; a drop with
no admin yet, or with a failing write, must not silence the following hour.
2026-09-02 00:06:01 +02:00
peter 8d7aaad9a8 fix: warn before the delete, not after — plus round-2 copy and a11y corrections
Round 2 on the feature diff (15 → 12):

- still_allowed_by_env now rides the invite LIST as well as the delete response,
  so the confirmation warns while the admin can still change their mind. It
  mattered most for a DENIED row on an env-allow-listed address: deleting it
  silently converted a refusal into a grant, and the only warning arrived after
  the irreversible click.
- Both inbox bodies claimed to name "the most recent" dropped message. The
  throttle records the FIRST of each hour and never updates it, so they now say
  "one of them", and the not-configured body finally names where to fix it.
- settings.invites.emailOff told the admin they "get no notice when someone
  signs up" — rendered directly above the queue that shows exactly that. Only
  the email is lost, and it now says so.
- The disabled delete buttons put their reason out of reach: `disabled` drops a
  control from the tab order and suppresses its title. Switched to
  aria-disabled + preventDefault, the pattern Playlists.tsx already established
  for the same reason.
- deleteInvite is quiet, so the 409 the UI handles with a translated toast no
  longer also raises the server's English detail as a modal.
- After a completed password reset, Back landed on the reset form again with a
  consumed token — the history change made that state reachable. The token is
  cleared on success so the reset view stops being the card's root.
- The hu deleteHasAccount named a tab that renders under a different label.

Tests: a dom test for the Access-requests panel covering both has_account
variants and all three delete outcomes, /api/me flag cases, an ADMIN_EMAILS
case, and the env-allowlist test now patches the raw setting so the parsing
stays under test.
2026-09-01 23:50:29 +02:00
peter 85043013f0 fix: address the feature review — accurate copy, reachable controls, real coverage
Round 1 on the feature diff found 15; the substantive ones:

- The delete control VANISHED on any invite whose address had registered, which
  is the ordinary sign-up shape (register writes the User and the pending invite
  together). The guard stays broad — that row is the account's approval gate, and
  orphaning it strands someone who can then neither sign in nor be approved — but
  the control is now a disabled button carrying the reason, on both lists, so an
  admin never sees a row silently missing a button its neighbours have.
- Deleting an entry is not always a revoke: is_allowed falls back to the env
  ALLOWED_EMAILS / ADMIN_EMAILS lists once no row exists, and an ADMIN_EMAILS
  address is force-promoted on its next Google sign-in. The route now reports
  still_allowed_by_env and the UI says so instead of claiming access was removed.
- "Deny keeps the row so they can't immediately re-ask" was simply false —
  upsert_pending_invite reopens a denied row. Corrected everywhere it was stated.
- The Configuration warning reads me.email_configured, which save() did not
  invalidate, so it kept warning about fields the admin had just filled in.
- The read-only variant was an aria-hidden span explained only by a Tooltip that
  the hints toggle can switch off; now a disabled button with title + aria-label.
- The delete error path threw away the 409's reason (the stale-list case) and
  left the stale row on screen.
- The throttle window was spent even when no notice was written, silencing the
  next hour on the path whose job is to stop silent drops.
- email_enabled() opened a second session and decrypted the SMTP password on
  every /api/me poll; it now takes the caller's session.
- welcomeMode.dom.test.ts never rendered the card, so reverting the history
  wiring to useState left it green. A render test now drives the real links and
  fails on that revert (verified).

Plus /api/me flag tests, the registration-shaped pending case, the env-allowlist
case, and a release note that matches what the code does.
2026-09-01 23:31:49 +02:00
peter 44f626f732 feat(admin): delete a whitelist entry, and stop hiding a broken mail setup
Three operator gaps, all found the same way — a person could not sign up and the
app showed nobody anything.

- DELETE /api/admin/invites/{id} with a delete control on both the pending
  requests and the previously-decided list. Deny keeps the row so the person
  cannot immediately re-ask; delete forgets the address. Refused with 409 while
  a User row exists for it, because is_allowed reads that row and removing it
  would revoke a live user's access with no other sign — the list ships
  has_account so the UI withholds the button instead of offering one that fails.
- Email is fail-soft by design, which had come to mean invisible. _send() now
  records an email_undelivered notification for every admin when a message is
  skipped or rejected, naming the message that was lost, throttled to one an
  hour. /api/me carries email_configured, driving a standing warning on
  Configuration → Email / SMTP and above the access-request list.
- The sign-in card's Create account / Forgot password / Didn't get the email
  views are history entries (useHistorySubview), so Back returns to the previous
  view instead of only moving the address bar.

Also: the per-test rate-limiter reset sweeps every loaded app.* module rather
than a hand-kept list of three — this change added a limiter outside it.
2026-09-01 23:09:25 +02:00
peter 24c93cbb83 docs: name the Configuration tab as it is rendered, drop an unused fixture
Round 6's two remaining nits. The operator note said "Configuration → Email"
but the group renders as "Email / SMTP" in both locales, and
test_an_admin_can_activate_a_pre_approved_registrant requested make_user without
using it — a leftover from the reverted rounds that made the test look like it
seeded an admin when it did not.
2026-09-01 22:15:56 +02:00
peter cb1f4f00b0 docs(auth): correct the last round of over-claiming comments and notes
Round 5's correctness/security pass came back clean; what was left was text that
said more than the code does.

- The register-path notify comment (and its test) claimed the admin "is still
  told". The call fires, but on a no-SMTP instance _send drops it — which is the
  open half of the problem, not something this branch fixed. Both now say so.
- The release notes tell admins what to actually do: an account whose address was
  whitelisted in advance has no pending request to approve, so the whitelist box
  is what activates it. That instruction only existed in the roadmap and in
  chores, which the dialog does not render.
- "That one line is the whole production change" now accounts for the test-only
  RateLimiter.reset() that also ships.
- The rate-limiter test claimed to pin that blocked attempts are not counted, on
  an assertion that held either way. Replaced with one that drives the clock
  across the window boundary and fails if hammering extends it.
2026-09-01 22:02:43 +02:00
peter f6e1d1e69f fix(auth): keep the admin notice unconditional on the register path
Round 4 showed the `inv is not None` gate added in b5a1002 removed a signal that
matters: for an address the admin had already approved, that email is their only
notice that someone has now created an account on it — the invite is not pending
so nothing lands in the Access-requests queue either. Suppressing it left the
registrant at "awaiting admin approval" with the admin unaware they existed. The
duplicate it avoided (Request access, then sign up) is mild noise by comparison.

The production change is now one db.commit(), which is all the reported bug ever
needed. Release notes corrected again: a previously-denied address was also
unaffected, the pending-request case never lost its queue entry (only the
account behind it), and the test inventory was missing a guard.
2026-09-01 21:48:40 +02:00
peter b64c3f514d fix(auth): drop the invite reopen — keep the fix to the lost account alone
Round 3 of the review killed the reopen from b5a1002: flipping an approved
invite back to pending let an unauthenticated POST /auth/register REVOKE a
standing approval for any whitelisted-but-unclaimed address, which then fails
is_allowed() and locks the real owner out of Google sign-in — the primary way
people get in here. It also re-armed the pre-emption window it was meant to
close, because with no SMTP email_verified is set True and the SEC-10 guard the
design leaned on never fires.

Two attempts at fixing the stranded-account UX in the same change both came back
worse than the wart. What remains is the narrow, real bug: the account was
silently rolled back and is now committed. An admin still binds the approval to
the account, exactly as before this branch. UX-25 in the roadmap records the
follow-up and both dead ends so they are not retried.

Also from round 3: the upsert docstring no longer claims the admin UI cannot
reach these accounts (re-entering the address in the same panel's whitelist box
does, and a test now pins it), the release notes are scoped to what the diff
actually changes with the operator note moved where the dialog renders it, and
RateLimiter has unit tests so reset() is not production code only a fixture
calls.
2026-09-01 21:34:38 +02:00
peter b5a1002412 fix(auth): reopen a pre-dated approval instead of auto-granting it
Replaces the activation added in ea931dc, which round 2 of the review showed was
a real weakening: on a no-SMTP instance it made a standing whitelist entry a
bearer grant — the first anonymous party to register an approved-but-unclaimed
address got an immediately usable account under that identity, with no
attestation, no admin action and no signal anywhere. The verify-time activation
had the same shape one step removed, and neither site ran the SEC-10 preset-
password guard or left an audit trail.

Keep a human in the loop instead. An approval granted before any account existed
was a decision about the ADDRESS, so registration reopens the invite
(upsert_pending_invite gains revive_approved, used only by the register worker).
The request lands in the Access-requests queue, approving it runs
_activate_user_by_email — and with it revoke_preset_password_if_unverified — and
the account activates. One mechanism, already audited, no new grant path, and
the "awaiting admin approval" copy is honest again.

The explicit commit from the previous commit still matters: with an invite
already pending, upsert returns without committing and _issue_token is gated on
SMTP, so a no-SMTP registration was still being rolled back.

Also from round 2: gate the admin notice on a fresh decision (matching the two
other upsert callers) so a retry no longer re-pings every admin, and reset the
module-level rate limiters between tests — the suite pooled every login into one
300s window and was two logins short of failing an unrelated test with a 429.
2026-09-01 21:18:14 +02:00
peter ea931dca30 fix(auth): honour an approval granted before sign-up instead of parking the account
Follow-up to the commit fix, from the deep-review. Persisting the account was
only half the story: when the admin had approved the address BEFORE the person
registered, the account landed is_active=False against an invite that is no
longer pending — absent from the Access-requests queue, and approve_invite 409s
on an already-decided invite. The sign-up worked and still could not sign in.

Derive activation from is_allowed() at the moment ownership is attested:
- verify_email_confirm activates a pre-approved account when the emailed link is
  confirmed (only the mailbox owner can follow it), so SEC-10 stays intact;
- _register_account activates only when there is no SMTP to verify with, where
  the admin whitelist is by design the sole gate — the same one add_invite
  already activates on.

Tests: the pre-approved no-SMTP case and the verify-activates case are real
regression tests (both fail without this change); guards cover the unknown-email
and denied->pending branches, verification of an unapproved account, and the
first end-to-end coverage of the /auth/register route and its
allow_registration gate. Release notes rewritten — the previous entry miscounted
its regression tests and advertised an email fix this release has no code for.
2026-09-01 20:56:43 +02:00
peter d954e71a77 fix(auth): commit the account on registration so a pre-approved sign-up isn't lost
_register_account() flushed the new User but relied on someone else to commit:
upsert_pending_invite() commits only when it actually writes an invite row, and
_issue_token() runs only when SMTP is enabled. For an email an admin had ALREADY
approved, on an instance where email is unavailable, neither fired — the session
closed and the account was rolled back. The would-be user got the uniform "ok"
and no account, could not sign in, and re-registering was a silent no-op.

Commit explicitly after the invite upsert (a no-op when upsert already committed
the same transaction). Three DB-lane regression tests cover the pre-approved
no-SMTP path (fails without the fix), the unknown-email path that always worked,
and the pre-approved path with SMTP on.
2026-09-01 16:25:05 +02:00
peter 602f62967a release: 0.78.4 (bug/plex-transcode-hls-startposition)
gate / frontend (push) Failing after 1m55s
gate / backend (push) Successful in 2m57s
2026-08-30 20:33:15 +02:00
peter ba00b50dea release: 0.78.4 — fix Plex transcode playback stall/rebuffer
hls.js startPosition:0 so the transcode/remux HLS session starts VOD-style
at the requested position instead of chasing the live edge of the growing
EVENT playlist.
2026-08-30 20:32:56 +02:00
peter 92adb593c8 fix(plex): start hls.js at position 0 for the transcode/remux HLS session
Our HLS is a growing EVENT playlist with no ENDLIST, which hls.js reads as
LIVE. Its default startPosition (-1) then starts at the live edge and hugs it
(no forward buffer → rebuffer spinners), and on a resume/seek session the
segments' absolute PTS (-copyts, offset by the server's -ss) don't line up
with the live-edge timeline → 0 fragments load → permanent stall. The
seek-restart model makes fragment 0 the requested content position, so start
VOD-style at 0 and buffer forward. Fixes the transcode playback stall/rebuffer
found in 0.78.2 UAT.
2026-08-30 20:09:00 +02:00
peter 66132b6dda release: 0.78.3 (feature/plex-playback-class-filter)
gate / frontend (push) Failing after 1m36s
gate / backend (push) Successful in 2m35s
2026-08-30 19:09:22 +02:00
peter d09f021f0c release: 0.78.3 — filter the Plex feed by playback class 2026-08-30 19:09:08 +02:00
peter d7e9240e54 merge main (0.78.2) into playback-class before release
# Conflicts:
#	backend/app/plex/stream.py
2026-08-30 19:08:08 +02:00
peter f2fa49b778 release: 0.78.2 (feature/plex-hdr-hw-tonemap)
gate / frontend (push) Failing after 1m37s
gate / backend (push) Successful in 2m24s
2026-08-30 18:56:32 +02:00
peter d4ed45b2eb release: 0.78.2 — HW HDR→SDR tonemap for 10-bit/HDR Plex transcode 2026-08-30 18:56:18 +02:00
peter 375f6e58be merge main (0.78.1) into tonemap before release 2026-08-30 18:55:14 +02:00
peter 1a102ce81a release: 0.78.1 (bug/download-mediakind-probe)
gate / frontend (push) Failing after 1m36s
gate / backend (push) Successful in 2m22s
2026-08-30 18:47:05 +02:00
peter bfd45a155e release: 0.78.1 — download media_kind from a file probe (BUG 2) 2026-08-30 18:46:33 +02:00
peter 1102ad6a28 fix(downloads): make the probe verdict authoritative over yt-dlp info by probe-ran, not per-field
Deep-review round 1 on bug/download-mediakind-probe found a real HIGH the prior
review missed:

- resolve_media_facts used a per-field `probe or info` fallback, so a probe that
  RAN and deliberately found no video track (cover-art-only audio: attached_pic
  skipped; a still image: format-guarded) had its None video overridden by
  yt-dlp's stale vcodec. An audio-mode download from a progressive source (info
  carries the source's video codec, embed-thumbnail on) was then classified VIDEO
  -> a Play button + a broken in-app <video> with no video track. Only the
  download path regressed (upload calls probe_media directly).

  Fix: gate precedence on whether ffprobe RAN, not per-field. If the probe read
  the file, its codec verdict wins outright (a None means 'no such track'); only a
  full ffprobe failure falls back to info. Dimensions still gap-fill from info.
  Refactored the parsing into _facts_from_probe so resolve_media_facts can see the
  raw probe-ran signal; probe_media keeps its 5-tuple (upload + TestProbeClassifier
  unchanged).

- tests: the cover-art-audio and still-image resurrection cases, a differing
  populated-info precedence case (the prior-flagged gap), and the ffprobe-failed
  info fallback — now stubbing run_ffprobe_json so the real parsing + precedence
  run together.

Backend gate green (ruff + full pytest).
2026-08-30 18:09:26 +02:00
peter c1a3b7b935 fix(plex): gate the part-key re-probe to single-version items; drop dead i18n key; cover cross-facet narrowing
Deep-review round 2 on the round-1 fixes (all low):

- part_key churn (state): round 1's part-key reset re-probed EVERY sync for a
  multi-version item (an original + a Plex 'Optimized' copy), because
  _media_facts reads media[0]/part[0] whose order Plex doesn't guarantee stable,
  oscillating transcode_hw between the two versions. Gate the part-key signal on
  a single-version item (len(Media)<=1); codec-change reset still applies to all.
  A multi-version same-codec re-encode not self-healing is an acceptable rare
  edge; a per-sync re-probe loop is not.

- dead i18n key: this diff's playableClass() collapses to transcode_hw/sw, so the
  bare plex.playable.transcode key it orphaned is now unused (knip can't see
  dynamic keys). Dropped from both locales, parity kept.

- test: cross-facet narrowing by an active playable class (a co-facet's year
  range collapses to the selected subset) — the flip side of self-exclusion,
  previously unpinned.

The aria-label+title double-announcement on the card dot is a deliberate accepted
trade-off (reliable SR name + mouse tooltip). Backend + FE gates green.
2026-08-30 17:58:38 +02:00
peter bf98b48517 fix(plex): re-probe on file replace, HU parity, public class const, playback-class test gaps
Deep-review round 1 on feature/plex-playback-class-filter:

- Sticky watermark (medium): _apply_item reset the bit-depth probe watermark
  only on a codec-string change, so a same-codec re-encode (8-bit->10-bit HEVC)
  kept a stale video_bit_depth/transcode_hw forever (the backfill only re-probes
  media_probed_at IS NULL rows). Also reset when the Plex Part key changes (the
  file was replaced), not just the codec.

- i18n parity (medium): the HU playback-class labels were mixed-language (only
  'direct' translated). Translated remux/transcode_hw/transcode_sw and the
  tooltips consistently to Hungarian (Kozvetlen streameles / Atkodolas).

- Layering (low): _PLAYBACK_CLASSES was module-private yet imported across into
  routes.plex.browse; renamed to public PLAYBACK_CLASSES (it is the shared
  source of truth for facet order + zero-fill).

- a11y (low): the browse-card playback-class dot conveyed meaning only via a
  mouse-hover title; added role=img + aria-label so SR/touch users get the class.

- tests: the NULL/unprobed transcode_hw -> SW fold (filter + facet), facet
  self-exclusion under an active playable filter, the migration codec-set ==
  HW_DECODABLE_CODECS drift guard, the probe-batch bound + oldest-first order,
  the transient probe-failure retry path, and the same-codec re-encode re-probe.

Backend gate green (ruff + full pytest); FE gate green (tsc/lint/knip/prettier/
vitest). NOTE: the facet-vs-runtime disagreement for 10-bit HEVC (C1) is the
known merge-order constraint — this branch's runtime is codec-only; the facet
becomes truthful once BUG1/tonemap's bit-depth-aware routing lands with it. Left
as a sequencing constraint, not code-changed here.
2026-08-30 17:51:33 +02:00
peter b563155a37 docs(plex): correct the cull-yield fallback comment to name the real route 404s
Deep-review round 3: the only surviving finding — a comment-drift nit. The
fallback's cull-yield comment named the route's 'Not available yet' path, but a
culled-key segment request actually returns 'No active playback session' (the
substantive invariant — never a None that 404s as file-not-found — is unchanged).
Reworded to match the route. Concurrency + correctness came back clean.
2026-08-30 15:18:26 +02:00
peter 915319fa35 fix(plex): make the HW->SW fallback supersede-check atomic; honest cull yield
Deep-review round 2 on the round-1 fix:

- Race (medium): the round-1 identity guard did the supersede check and the
  respawn in TWO separate _lock sections. A newer same-key session installed in
  the gap was still torn down by _spawn's unconditional pop+kill (the round-1
  comment even had the direction backwards). Fold the check into _spawn via a
  new replace_only= param: the identity test and the pop+replace now share ONE
  lock, so the respawn is a true no-op when superseded/culled.

- Honest states (low): the superseded branch returned current_session(key),
  which is None when the key was cap-culled during our out-of-lock wait; the
  route turns None into a dishonest 'Media file not found on disk' 404 for a
  file that exists. Yield 'current_session(key) or s' so a culled session
  degrades to the route's retryable 'not available yet' path instead.

- Hygiene: _want_hw docstring corrected (it is a test-only shim now; production
  decides via _select_encode) and the two stale _want_hw references refreshed
  (stream.py header, config.py plex_transcode_hw).

- tests: the cap-cull no-respawn/no-None sub-case; the _opencl_present except
  OSError branch (glob raising, not just empty); tightened a tautological -vf
  assertion. Backend gate green (ruff + full pytest).
2026-08-30 15:11:37 +02:00
peter a2a8f5681d fix(plex): don't tonemap 10-bit SDR; identity-guard the HW→SW fallback
Deep-review round 1 on feature/plex-hdr-hw-tonemap:

- F1 (correctness): _select_encode gated tonemap on bit-depth alone, so a
  10-bit *SDR* (bt709) source got the HDR hable tone curve applied, visibly
  altering an SDR picture where only a 10->8-bit downconvert was needed. Split
  the ffprobe gate into a 3-way classifier (_probe_video_kind: hwsafe / hdr /
  sdr10); only true-HDR (PQ/HLG) takes the OpenCL tonemap graph, 10-bit SDR
  drops to software libx264 (an exact downconvert). Applies to both auto and on.

- F2 (race): the runtime HW->SW fallback popped+killed _sessions[key] with no
  identity check. Because wait_for blocks OUTSIDE _lock, a concurrent same-key
  _spawn (rapid seek-restart) or _enforce_cap cull could kill this session; the
  fallback then respawned software over the NEWER session and breached the
  transcode cap. Guard the respawn: only fall back if _sessions[key] is still
  this session, else yield the currently-registered one.

- tests: 3-way classifier (incl. HLG + empty-probe), real _opencl_present glob,
  real wait_for proc fast-fail branch, tonemap-variant fallback, the F1
  regressions (auto + on), the supersede/no-respawn race, and the on/opencl
  matrix cell. Backend gate green (ruff + full pytest).
2026-08-30 15:02:41 +02:00
peter 4827aebb7f feat(plex): HW HDR→SDR tonemap for 10-bit/HDR transcode (OpenCL on the iGPU)
Upgrade mode-3's 10-bit/HDR branch from "route to software libx264" (BUG 1's
behaviour) to a full-GPU pipeline: VAAPI decode → OpenCL tonemap (hable,
HDR10/HLG → bt709 SDR nv12) → h264_vaapi. Gen 9.5 (UHD 630) can't tonemap in
the VAAPI VPP (tonemap_vaapi fails), so OpenCL is the only HW route — proven on
the prod iGPU at ~4.2x realtime, ~1 CPU core.

- stream.py: `_select_encode` returns (hw, tonemap); a 10-bit/HDR HW-decodable
  source picks the tonemap graph when the OpenCL runtime is present, else drops
  to software. `_opencl_present` gates on a registered ICD. `_venc_opts` /
  `_hwaccel_in_opts` gain the tonemap variant (named vaapi+opencl devices,
  `-filter_hw_device ocl`, hwmap→tonemap_opencl→hwmap reverse). `_want_hw` kept
  as the hw-half wrapper. The runtime HW→SW fallback still backstops a session
  that emits no segment.
- Dockerfile: add the Intel NEO OpenCL runtime (intel-opencl-icd). It was removed
  from Debian trixie, so it's pulled from bookworm (22.43 still supports Gen 9.5;
  the .deb runs on the trixie base — verified live). Harmless without /dev/dri.
- route + FE: thread `tonemap` through the session response; the stats overlay
  mode row reads "transcode · GPU · HDR→SDR" for the tonemap path.
- tests: tonemap command branch, the (hw, tonemap) selection matrix (auto/on ×
  8-bit/10-bit × opencl), start_session tonemap wiring, and the fmtMode tag.
2026-08-30 06:02:45 +02:00
peter 840526b1b8 feat(plex): filter the feed by playback class (Direct Play / Direct Stream / Transcode HW/SW)
Adds a movie-scoped "Playback" filter to the Plex feed so a user can isolate items by how
they play — the groundwork for collecting HW-transcodable (or no-transcode) versions.

Four classes derived from the stored `playable` + a new `transcode_hw` facet:
- Direct Play (direct), Direct Stream (remux)
- Transcode · HW  — playable=transcode AND the iGPU can hardware-transcode it
- Transcode · SW  — playable=transcode, software libx264 only

The HW/SW split is precise, not codec-only: a 10-bit/HDR HEVC is HW-decodable yet breaks the
VAAPI nv12 pipeline (the same case plex/stream special-cases), so it must count as SW.

Backend:
- New PlexItem columns (migration 0077): transcode_hw, video_bit_depth, media_probed_at.
  Migration backfills transcode_hw codec-only so the filter works immediately.
- `plex.sync.transcode_hw_capable` is the single classifier; the HW-decodable codec set moves
  to sync as `HW_DECODABLE_CODECS` and plex.stream imports it, so the catalog facet and the
  runtime encoder choice can't disagree by codec.
- Bit-depth backfill (`_probe_media_batch`, wired into sync): probes only HW-decodable transcode
  items' video Stream for bit-depth + HDR and refines a 10-bit/HDR HEVC to software. Bounded
  per run, watermarked by media_probed_at. A codec change resets the watermark (stale probe).
- Filter + self-excluded facet on a shared `playback_class_expr` CASE (movie-leg only — a show
  is an aggregate with no single class). transcode_hw threaded onto the feed/detail/card JSON.

Frontend:
- Multi-select "Playback" Combobox in the sidebar (movie scope, facet-empty drop-out), active
  chips, 4-way corner-dot badge (green/blue/amber/red) + tooltip, and an info-page playback chip
  with a codec · bit-depth subtitle. New `bg-info` (blue) token, i18n HU/EN parity.
- Fix a regression where the card's playback dot lost its hover tooltip: the full-card reveal
  overlay swallowed the cursor; the dot now sits above it (z-20).

Gate: backend full lane + frontend typecheck/lint/knip/prettier/vitest all green. Browser-UAT'd
on localdev (filter narrows to the SW set, facet counts, badge colour + tooltip, info chip).
2026-08-30 04:45:11 +02:00
peter 10d69cb316 fix(plex): route 10-bit/HDR HEVC to software + runtime VAAPI→libx264 fallback
A 10-bit/HDR HEVC source (e.g. an HDR10 x265 file) died at scale_vaapi=format=nv12
on the UHD 630 (Gen 9.5 VPP: "VAProfile not supported"); ffmpeg produced 0 segments
and the item never played, with no fallback — a retry just re-picked the failing HW
path. An 8-bit HEVC and the same file on libx264 both work.

Two defenses:
- `_want_hw` auto mode now also gates on an 8-bit-SDR source via a new
  `_probe_video_hwsafe` ffprobe (bit depth from pix_fmt/bits_per_raw_sample, HDR from
  color_transfer) — a 10-bit/HDR source takes the software libx264 path, which
  downconverts to yuv420p cleanly. The Plex mirror stores no bit-depth signal, so this
  is probed at session start. "on" still trusts the admin and skips the gate.
- `start_session` adds a runtime HW→SW fallback: a HW session that emits no first
  segment respawns on libx264 and reports hw=False (the route/stats overlay honestly
  show CPU). `wait_for` gained an optional `proc` so a dead ffmpeg fails fast instead
  of burning the 25s budget before the fallback fires.

Tests: TestProbeVideoHwsafe (parsing + probe-failure-assumes-safe), the 10-bit auto
gate + "on" skip in TestWantHw, and the no-segment HW→SW respawn in
TestTranscodeStartSession. Backend lane green.
2026-08-30 03:53:31 +02:00
peter a86131e062 fix(downloads): classify media_kind from a file probe, not yt-dlp info
A downloaded video was silently demoted to a download-only "file" (no
in-app play/edit) whenever yt-dlp's info dict omitted vcodec/acodec — which
it does for Facebook and other sources. Finalize derived media_kind from
those nulls (`derive(None, None)` -> file), so a genuine H.264/AAC MP4
became non-playable in the library.

Fix: probe the finished file on disk (the ground truth) at finalize and
derive from that, falling back to the info dict only for a field the probe
couldn't read. This is the same ffprobe the upload ingest already used, now
extracted to a shared downloads/probe.py so the download worker and the
upload path share one classifier (and can't drift). Also backfills the
stored vcodec/acodec/width/height from the real file.

Tests: TestResolveMediaFacts pins probe-wins-over-info (incl. the Facebook
no-codec case and yt-dlp's literal "none"); upload's probe tests repoint to
the shared module.
2026-08-30 03:32:14 +02:00
peter a84f2db310 release: 0.78.0 (feature/e6-plex-player-polish)
gate / frontend (push) Failing after 1m35s
gate / backend (push) Successful in 2m20s
2026-08-30 02:26:25 +02:00
peter 872d91f540 release: 0.78.0 — Plex transcode playback + player polish (E6) 2026-08-30 02:25:20 +02:00
peter 4eaa68ceeb feat(plex): S1a — add intel-media-va-driver for VAAPI transcode (E6)
The runtime image ships ffmpeg with h264_vaapi/libva2 but no VA driver, so
h264_vaapi can't initialise and a Plex transcode silently falls back to
libx264. Add the iHD driver (intel-media-va-driver) so the UHD 630 iGPU can
hardware-encode once /dev/dri is passed into LXC 118 (host + compose side,
tracked in siftlode-ops). Harmless where no render node is present.
2026-08-30 02:14:06 +02:00
peter 00539a4d97 fix(plex): deep-review follow-ups on E6 transcode + player polish
Eight verified findings from the sprint deep-review:

- Transcode cap: _enforce_cap now counts only transcode sessions, so a
  live remux (stream-copy) stream can no longer be LRU-evicted mid-play —
  matching the config comment + the HU/EN Config hint that promise remux
  is uncapped.
- Player Back: the browser/mouse Back interceptor (BackClose) now also
  mounts for the stats overlay and up-next card, so Back closes them first
  instead of ejecting the whole player (keyboard/Esc already did).
- Honest cold-start: loadSession clears sessionIsTranscode up front, so a
  transcode->direct/remux switch no longer flashes a stale "Preparing
  stream..." during the session request.
- Stats overlay a11y: the close (X) buttons in the stats, help and info
  overlays now read as "Close"/"Bezaras" (new plex.player.close key), not
  "Back"; dropped aria-live from the per-second stat grid so a screen
  reader is not flooded with telemetry.

Tests: route-level transcode routing (transcode item -> HLS re-encode
session, remux stays copy, direct opens none); start_session threading the
preset/crf/max_height knobs + hw/preexec into the ffmpeg argv; UpNextCard
countdown auto-advance, Play-now, Cancel and the S?-E? heading fallback.

Also brings UpNextCard.tsx / plexStats.ts / plexStats.test.ts to prettier.
2026-08-30 02:04:17 +02:00
peter dc24d353e0 feat(plex): honest transcode cold-start + encoder in stats (E6 S1d)
Surface the transcode session's runtime facts in the player. The session
response's transcode/hw flags now thread through: the stats overlay's mode row
reads "transcode · GPU" (VAAPI) or "transcode · CPU" (libx264) — the at-a-glance
confirmation of which S1 path engaged — and the initial-load overlay says
"Preparing stream…" instead of "Loading…" for a transcode session, since its
first segment is genuinely slower than a remux and shouldn't read as a hang.

fmtMode (pure, unit-tested) appends the encoder only for a transcode with a known
encoder. i18n HU/EN for the new preparing label. Browser-UAT'd on the XviD
fixture: mode row reads "transcode · CPU" live (software, no render node here).
2026-08-30 01:30:53 +02:00
peter ce96032edb feat(plex): transcode resource discipline + admin knobs (E6 S1c)
The software-fallback re-encode now runs deprioritized and admin-configurable.
The SW transcode ffmpeg child is niced (preexec_fn → os.nice, applied per-child
because the Popen lives in the long-lived API process, which must NOT itself be
niced — unlike the download worker's whole-process _deprioritize) so a CPU-bound
re-encode can't starve request handling or playback; the VAAPI path needs none of
it.

New sysconfig knobs (plex group, rendered on the admin Configuration page, i18n
HU/EN): plex_transcode_hw (auto/on/off, layered onto _want_hw), plex_transcode_
preset, plex_transcode_crf, plex_transcode_max_height (software-only downscale-to
cap; the GPU path encodes at source resolution). plex_max_transcodes stays the
concurrency cap (comment corrected — the fallback is shipped, not "later").

pytest extends the command-builder + _want_hw coverage (preset/crf/max_height
plumbing, the on/off/auto policy). Browser-UAT'd: max_height=240 drops the live
resolution to 330x240; the SW ffmpeg child runs at nice 10 (ppid=api process).
2026-08-30 01:22:30 +02:00
peter 0658f9550b feat(plex): transcode playback mode — VAAPI/libx264 re-encode (E6 S1b)
Non-h264 items (playable="transcode": HEVC/VP9/AV1/MPEG-2/XviD…) now play by
re-encoding the video to H.264 inside the existing HLS session, instead of the
old HTTP 501. VAAPI hardware encode (h264_vaapi, full GPU decode→scale→encode
pipeline) where the render node is present and the source is HW-decodable;
libx264 software fallback otherwise (yuv420p 8-bit, -threads reserve-one-core).

The re-encode mints a keyframe at every segment boundary (-force_key_frames) so
the HLS muxer still cuts independent segments and seeking works; everything else
— -copyts, the HLS tail, multi-audio renditions, the AAC audio path — is the
same machinery as remux, so hls.js/subtitles/seeking are unchanged. The session
route returns transcode/hw so the player can surface the active mode.

pytest pins both command branches (software + VAAPI shape) and the HW-selection
gate (render node present AND source HW-decodable). Browser-UAT'd on the XviD
fixture (Winning London): 501→plays, real 528×384 H.264 frames, 0 dropped, seek
lands clean; stats overlay Mode row reads "transcode".
2026-08-30 01:06:08 +02:00
peter f162788135 feat(plex): up-next end-card at episode end (E6 S4)
At episode end, when a next item exists and the pref is on, show an up-next
card (poster + title + ~10s countdown + Cancel) instead of the old instant jump,
so binge stays in the viewer's control (D6-A). Countdown or Play now advances;
Cancel/✕ dismisses and stays on the finished frame. Lives only on the
play-to-the-end path — credits-auto-skip advances earlier and never reaches the
card, so the two reconcile by construction (no double-advance). New pref
'upNext' (default on) with a Playback-tab toggle; the card closes on item change
and joins the Back/Escape cascade. i18n HU/EN.
2026-08-30 00:28:50 +02:00
peter 968674bc6a feat(plex): stats-for-nerds debug overlay (E6 S2)
Add a toggleable stats overlay to the Plex player: active mode
(direct/remux/transcode), transport, resolution, codec, bitrate, HLS level,
dropped frames, buffer-ahead, session offset and position — sampled once a
second off the live videoRef + hlsRef (no data layer). Non-modal corner panel,
aria-live polite, dismissable; toggled with 'd', a gauge control-bar button, or
its X. Number-crunching lives in a pure, unit-tested plexStats module. Doubles
as the S1 transcode-verification surface. i18n HU/EN.
2026-08-30 00:17:34 +02:00
peter 90ae9f6e88 feat(plex): on-screen ±10s skip buttons flanking play/pause (E6 S3)
Add discoverable rewind-10 / forward-10 controls to the Plex player control
bar, flanking play/pause (D5-A). New Skip10 glyph: an outline circular arrow
with a centred '10', deliberately distinct from the episode-level
SkipBack/SkipForward and from Restart's plain RotateCcw. Reuses seekTo; mirrors
the keyboard ←/→ (seekStep=10). i18n HU/EN aria-labels.
2026-08-30 00:09:36 +02:00
peter 3682e6a01f release: 0.76.0 (feature/e7-downloads-media-library)
gate / frontend (push) Failing after 1m36s
gate / backend (push) Successful in 2m26s
2026-08-29 21:54:19 +02:00
peter aca355f43b chore(release): 0.76.0 — E7 Downloads & Media Library
Bundles the whole E7 epic (library metadata, crop fix, share-by-default, in-app player,
browser-playable guarantee, local upload) plus the deep-review hardening. Bump VERSION and
add the 0.76.0 release-notes entry (RELEASE_NOTES[0] must match VERSION).
2026-08-29 21:52:21 +02:00
peter f41c81eb56 fix(downloads): media-library UI review fixes (E7)
Deep-review remediation on the frontend, all findings adversarially verified:

- LibraryPlayer picks audio vs video from the canonical media_kind (single source), not a
  hardcoded container whitelist, so an uploaded audio file with an uncommon/absent container
  renders in <audio> instead of a broken video box (legacy null media_kind keeps the heuristic).
- UploadDialog aborts the in-flight upload when the dialog is closed by X/ESC/backdrop/Back
  (not only the footer Cancel); drag highlight uses an enter/leave depth counter to stop the
  flicker; the progress bar gets a stable role=progressbar aria-label with the value on aria-valuenow.
- DownloadCenter's Shared tab renders honest loading/error states instead of a blank panel, and
  the library/shared row-action gating moves to a single tested helper (lib/downloadActions).

Tests: downloadActions gating matrix, the uploadReq XHR path (abort/401/422-localize/onerror),
and the media_kind-driven audio/video branch.

i18n: downloads.upload.uploadProgress (HU + EN).
2026-08-29 21:52:13 +02:00
peter 614c5528ad fix(downloads): harden upload serving + ingest against stored-XSS and leaks (E7 review)
Deep-review remediation for the E7 upload path, all findings adversarially verified:

- Serve every library file with a Content-Type chosen from the ffprobe-verified media_kind
  + a strict container allow-list (storage.served_media_type), never guessed from the
  user-controlled file extension. Closes a stored-XSS vector where a media/HTML polyglot
  uploaded as "evil.html" was served inline as text/html on the login-free /watch surface.
  Applied to both the authenticated /file and the public watch_file endpoints.
- Classify uploaded still images (jpg/png/gif/webp/…) as media_kind='file' so they are
  force-downloaded and don't get a broken in-app video player.
- Roll back the on-disk side of an upload when the ingest transaction fails after place_file
  (unlink the placed file + its deterministic poster path), so a failed commit can't orphan
  bytes in the GC-skipped .uploads tree.
- Floor download_upload_max_bytes at min=1 so a stored 0 can't silently disable the per-file cap.
- Dedupe the ffprobe-JSON invocation into edit.run_ffprobe_json (shared by probe_codecs and
  upload._probe) and the upload extension into storage._clean_upload_ext; drop the now-unused
  parse_probe_streams.

Tests: public + authenticated force-download and the polyglot-not-served-as-HTML guard, the
real _probe classifier (image/cover-art/dims/duration), the ingest rollback/cleanup path, the
max_jobs branch, the cap-floor invariant, and run_ffprobe_json error handling.
2026-08-29 21:52:02 +02:00
peter 104bd1e3a6 feat(downloads): public generic-file watch surface (E7 S5d)
The /watch/{token} SPA gets a third branch: media_kind==='file' renders an
info card (name / TYPE · size) + a Download button, no player. The <video>/
<audio> branches now guard on media_kind!=='file' so legacy links
(media_kind null) still play. Download is gated on the link's allow_download,
mirroring the media branch; the backend serves a file force-download
(attachment + octet-stream) regardless, so the gate only reflects owner ACL.

FE-only (public_meta already emits media_kind/size_bytes/container since S5a).
i18n HU/EN/DE 'file' fallback label. Two DOM tests: file -> info card +
Download, no player; view-only file -> no Download.
2026-08-29 19:51:58 +02:00
peter 584e9ced0c feat(downloads): local-upload UI — drag-drop dialog + progress (E7 S5c)
UploadDialog (drag-drop + native picker + optional display name) opened
from a new Upload button in the queue add-row. Uploads stream through a
new XHR helper (uploadReq in api/core) so the dialog shows real 0-100%
progress with a cancel — fetch/req can't report upload progress. The
upload lands as an already-done job; the S5b file-kind row gates it to
download+share. i18n HU/EN; upload cap/empty errors localize via the
shared structured-detail path.
2026-08-29 18:53:53 +02:00
peter b4ad59274b feat(downloads): gate library row actions by media_kind (E7 S5b)
Surface the S5a media_kind discriminator in the library UI. A generic
uploaded file (media_kind='file') is download+share only — no player and
no timeline to trim, so Play/Edit/Edit-details are media-only; it shows a
neutral file glyph instead of an (always-empty) thumbnail and no duration
chip (container/size still read). The "Shared with me" tab's file row is
gated the same way. Re-download is now hidden for edit clips and uploads
too (neither has a re-fetchable source — the backend already rejects both
with 409), removing a latent click-to-409 on edit rows.

FE-only: the serializer already emits media_kind (S5a). Adds media_kind to
the DownloadAsset type and "upload" to the job_kind union; DOM tests cover
the file-row presentation (glyph, no img, no duration).
2026-08-29 18:38:21 +02:00
peter fe0b34cd00 feat(downloads): local file upload ingest — media library backend (E7 S5a)
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.

- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
  gates library actions. Set at every finalize (download probe, editor carries the
  source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
  in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
  the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
  places it in a per-user .uploads tree, and creates an already-ready asset + done job
  (the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
  (download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
  charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
  application/octet-stream) on both the in-app and public endpoints, so uploaded
  HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
2026-08-29 08:02:11 +02:00
peter 9eaa01617c feat(downloads): in-app player for library items (E7 S4b)
Play a completed library item in-app: a Play button on Library rows (and
Shared rows with download access) opens a modal that streams the local file
into a native <video controls> (or <audio> for audio-only assets).

Rather than the YouTube IFrame PlayerModal or the Plex HLS PlexPlayer — both
bound to their own domain APIs and unable to point at a local file — this
reuses the shared Modal a11y shell (focus-trap, ESC, layer-stack, portal) and
the range-aware /api/downloads/{id}/file endpoint. S4a's universal-codec
guarantee (mp4/H.264/AAC + m4a/mp3) makes native playback always work, so seek,
volume, fullscreen and speed come for free with zero transcode.
2026-08-29 07:18:58 +02:00
peter fdca12968b feat(downloads): one-click Share + downloadable-by-default links (E7 S3)
D2-A: the library row's Share button becomes a single action — quickShareLink()
ensures a public link (minting on demand), copies it to the clipboard, and toasts
with an "Open in new tab" action. A copy failure still toasts (warning) with that
action so the link is never a dead end. Advanced controls (password/expiry/revoke/
user-ACL) stay in the existing ShareDialog, reached via a new "Manage sharing" row
button.

D1 (forward-only): CreateLinkIn.allow_download defaults to true, so a newly minted
link is downloadable by default. The default lives in the API, not a migration —
the download_links column default stays false, so no existing link flips. The
ShareDialog create-default is flipped to match.

Tests: TestCreateLinkDefault (default true; explicit false honored); 4 vitest cases
for quickShareLink (mint-vs-reuse, copy ok/fail, create failure). HU/EN parity.
2026-08-29 06:36:07 +02:00
peter 94fcc40bde feat(downloads): guarantee every download is browser-playable (mp4/H.264/AAC)
The Download module's purpose is download -> share -> play in-browser, so the
output must play in EVERY browser (iOS/Safari/WebKit included). The only
universally-safe combo is mp4 + H.264 + AAC/MP3.

- formats.normalize() now CLAMPS every spec to that profile (container->mp4,
  vcodec->None, audio_format opus->m4a) at the single point all paths flow
  through (picker, stored profile, raw API) — not just by hiding UI options. A
  source exposing no H.264 is still caught by the worker's browser-safe
  re-encode, so playability is guaranteed end-to-end.
- build_ydl_opts drops the now-unreachable non-compat branches (+ _VCODEC_SORT).
- _SIG_VERSION 3->4 so assets cached under an old non-compat spec — or a stale
  non-H.264 mp4 that slipped the v3 net (the HEVC Facebook dls) — re-derive.
- ProfileEditor drops the container/codec pickers and Opus; audio is M4A/MP3.
- i18n: remove the now-unused container/vcodec/any keys (HU+EN parity).
2026-08-29 06:08:21 +02:00
peter fb2c843fa9 fix(downloads): anchor crop overlay to the letterboxed picture (E7 S2)
The video preview is object-fit: contain (the browser default for <video>), so a
non-16:9 source is letter/pillarboxed inside the element box. The crop marquee was
drawn in fractions of the whole box (bars included) and those fractions were then
multiplied by the source resolution, so the emitted ffmpeg crop diverged from what
the selector showed. Anchor the overlay to the computed content rect (and set
object-contain explicitly) so a marquee fraction is a fraction of the real picture;
extract the display-px<->source-px math into cropGeometry.ts with unit tests, and
clamp the even-rounded window in-bounds so a full-frame crop can't overflow.
2026-08-29 05:43:41 +02:00
peter bb6cb6e335 style(downloads): bump library-row meta text + action buttons a step (E7 S1)
Enlarge every non-title text on the card (channel, status/format/size/date meta,
source-ref line, progress readout) one step (xs→sm) plus their inline icons, and
grow the row action buttons (padding + icon) via the actions wrapper so the shared
IconBtn stays smaller in the modals. Buttons keep the title's line — the wrapper is
shrink-0/no-wrap, so the text column shrinks instead.
2026-08-29 05:29:19 +02:00
peter bdbff89507 feat(downloads): show container/format chip in library rows (E7 S1)
The completed download rows already surfaced size + duration; add the concrete
container (MP4/M4A/WebM…) as a compact chip so the library reads as a media list
with a visible format — the audio-vs-video distinction the running-only phase pill
can't show at rest. Pure display of the already-serialized asset.container; locked
by a small DOM test (present when ready, absent before the worker fills it).
2026-08-29 05:21:54 +02:00
peter 9b87838e3a release: 0.75.0 (improvement/quickfix-batch-2)
gate / frontend (push) Failing after 1m34s
gate / backend (push) Successful in 2m16s
2026-08-29 03:55:15 +02:00
peter f014416e31 chore(release): 0.75.0 — quickfix batch #2 (Plex/downloads/playlists) 2026-08-29 03:54:08 +02:00
peter 3369587412 fix(review): resolve deep-review findings across quickfix batch #2
- Button: `loading` always disables (|| not ??) — stops a double-submit
  when a call-site passes an explicit boolean `disabled` (VideoEditor).
- playlists: track un-ingestable-but-live YouTube ids in the sync baseline
  so a linked playlist containing a private/deleted video converges instead
  of re-classifying it as a pending pull forever; re-attempt ingestion each
  sync so a restored (private->public) video heals, and prune ids that
  leave YouTube.
- SettingsPanel: download-auth section renders an honest error/loading
  state (StateMessage + retry) instead of a bare empty section on a failed
  credentials fetch; HU+EN loadError string.
- tests: reorder-order, partial-failure baseline hold, un-ingestable
  converge/heal/prune, Button loading-disables, download-auth error/retry.
2026-08-29 03:52:13 +02:00
peter 9af4e8f4ff feat(playlists): sentence-style two-way sync preview (only changed sides)
The confirm dialog spelled out both sides even when one was "+0 / −0", which read
as engineer shorthand. Build the message from one sentence per side that actually
changes and add a short "stays unchanged" line for the untouched side; the zero
side is no longer shown as a bare +0/−0 to decode. Assembled in the component
from granular i18n clauses (add/remove per side, reorder, rename) with count
plurals; HU+EN. Replaces the single pushPlanMerge template.
2026-08-28 05:20:34 +02:00
peter a0b725d779 feat(playlists): two-way (3-way) merge YouTube sync + mandatory preview (S7)
"Sync to YouTube" was a one-directional destructive push (local wins, deletes
from YouTube every item not in the local list), so items added on YouTube were
lost and a dirty pull was skipped entirely. Replace it with a real 3-way merge:
each item is classified against the last-synced baseline as a local-add, a
YouTube-add, or a genuine delete, so an add or delete on either side survives.

- Persist the ordered baseline id set (new playlists.synced_video_ids, migration
  0075); the fingerprint hash alone couldn't drive a merge. NULL baseline (fresh
  export / pre-S7 link) unions losslessly on the first sync and records one.
- Pure merge_plan(baseline, local, youtube) -> add/remove per side + merged order
  (local order, YouTube-only additions appended). merge_playlist applies the
  local side directly and the YouTube side through the failure breaker; the
  baseline only advances on full success so a partial failure re-converges.
- plan_merge/push-plan now report what each side gains and loses; the confirm
  dialog shows the two-way plan before applying (pushPlanMerge, HU+EN).
- Tests: 10 new (pure classification incl. delete-vs-stale-copy; DB apply proving
  a YouTube-add and a Siftlode-add both survive on both sides, a genuine delete
  propagates without collateral loss, no-baseline never deletes, create path).
2026-08-28 04:20:29 +02:00
peter bc33b2ed16 feat(downloads): per-user, per-host yt-dlp cookie/auth support (S6)
Each user stores their own login cookies (Netscape cookies.txt) for an
auth-walled download host (Facebook, Instagram, X, TikTok, Reddit) in
Settings; a download of an auth-walled link on that host is retried with
the acting user's cookie.

Two load-bearing security controls, both in downloads.credentials as the
single source of truth so enqueue and the worker can't drift:
- cache privacy: a cookie-authed download salts its MediaAsset format_sig
  with the user id, so the authed asset is private and never cache-served
  to another user. The worker attaches a cookie ONLY when the asset it is
  downloading into IS that private, salted identity.
- host scope (SEC-12): a cookie is resolved only for a URL whose netloc
  the credential owns (netloc-scoped _host_is); a look-alike/unrelated
  host gets none.

cookies_enc is Fernet-encrypted at rest, only decrypted in-worker into a
0600 file under staging/.auth/ (survives the retry wipe + _produced_file
scan) and unlinked in finally. The blob is never returned by a route,
logged, audited, or written to a job's error/snapshot. cookiefile is set
after the adapter's extra merge so an adapter can't clobber it. Settings
UI is write-only (host + set/updated only, never echoed). New
download_credentials table + migration 0074; per-user routes; HU+EN i18n.

Tests: host-scope, cache-privacy (no cross-user serve), encrypt-at-rest,
write-only API, worker attach-gate; FE dom test for the write-only field.
2026-08-28 03:53:04 +02:00
peter 5280949667 feat(ui): migrate the canonical action-button surface onto <Button>
Sweep the rounded-lg solid/ghost/outline action-button family onto the shared primitive:
ConfirmProvider, ErrorDialog, DownloadDialog, DownloadCenter, downloadModals, downloadAdmin,
ProfileEditor, PersonalViewsModal, SettingsPanel, WatchPage, DraftSaveBar. Deliberately left
on their own styles per the agreed scope: login/onboarding rounded-xl CTAs (Welcome/Onboarding/
Setup/KeyGate), glass Plex surfaces, segmented pills/toggles/switches/chips, dropdown triggers,
data-driven Banner/tone maps, text-xs dense buttons, and tiny inline p-1/p-1.5 icon controls —
none map 1:1 to a variant+size without a visual change, and the repo has no tailwind-merge.
2026-08-28 03:12:29 +02:00
peter 4892c38466 feat(ui): shared <Button> primitive + migrate ShareDialog/VideoEditor/ChannelPage proofs
Introduce frontend/src/components/ui/Button.tsx — the one action-button primitive
(variants primary|danger|outline|ghost|glass|danger-outline; sizes xs|sm|md|lg|icon;
icon/iconOnly/label/loading/fullWidth/forwardRef). No focus-visible (already centralized
in index.css @layer base). Migrate the cleanest proof sites off btnCls, and flip the
channel Subscribe button to the outline variant so it matches Full history (the user's ask).
2026-08-28 02:59:21 +02:00
peter 9ae6c47d6c perf(downloads): de-prioritize worker transcodes + idle-back-off polls to stop LAN slowdown
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API,
Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and
starved web requests + playback until it finished. Full-scope S4 mitigations:

- Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp
  descendant): CPU nice increment + best-effort idle ionice class + an
  OMP_NUM_THREADS budget, all config-gated and independently disable-able.
- Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode
  reserves a core for foreground work instead of claiming all of them.
- Throttle the retention GC's whole-tree orphan sweep (yield IO after each
  batch of files) so its rglob can't monopolize disk IO in the API process.
- Idle-back-off the Download Center (2s→15s when nothing queued/running) and
  UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the
  next fetch after an enqueue (invalidateDownloads refetches immediately).

Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence,
_deprioritize (nice/ionice/OMP branches), GC sweep throttle batching.
2026-08-28 02:37:56 +02:00
peter b7a6451dba fix(downloads): iPad audio playback + honest non-retryable failure for auth-walled hosts
5a — the public watch page rendered every asset in a <video>, but WebKit (all
iOS browsers) refuses to play an audio-only mp3 in <video>, so shared audio
links were broken on iPad while working on desktop Chromium. public_meta now
flags audio-only assets (codec present, no video stream) and WatchPage renders
a native <audio> in a padded surface card (no artwork exists for these).

5c — a login-walled Facebook post failed as a bare 'unknown', which offered a
misleading Retry. Classify closed-group / cookie-suggestion auth walls as
login_required and genuinely unsupported sites as a new unsupported_host class
(both non-retryable), and drop unknown from the modal's retryable set — an
unclassified failure is most often an unparsed auth wall. The queue row keeps
its own force-retry for real transients.
2026-08-28 02:22:28 +02:00
peter 11e32163b8 fix(ui): SortControl direction toggle no longer wraps on narrow popovers
The asc/desc buttons in the sort popover footer share one component across
Feed and Plex browse, but rendered differently: the Plex popover is narrower
(short option labels), so "Descending" wrapped icon-over-text and made the
active button taller. Add whitespace-nowrap to the direction buttons and bump
the popover min-width (min-w-52 -> min-w-56) so both call-sites render an
identical single-line toggle.
2026-08-28 02:06:06 +02:00
peter 257dc5741d fix(plex): resolve movies stuck "in progress" on the 15-min incremental sync
The incremental watch-sync (run_plex_watch_sync) learned a movie's watched
transition only from the history feed + onDeck. When a movie is finished on
Plex (commonly by scrubbing through the end credits) Plex sets viewCount>0,
clears viewOffset and DROPS it from Continue-Watching, so onDeck no longer
lists it (and onDeck can only ever ADD an in-progress row, never resolve one),
while a scrub-to-end frequently emits no history session row. The stale local
status='new', position>0 row then survived until the DAILY full reconcile —
which is why movies kept showing "in progress".

Add _reconcile_inprogress_movies: for each local in-progress MOVIE that Plex's
onDeck no longer lists, read its current per-account viewCount/viewOffset with
one metadata() call and apply it via the shared _pull_apply, flipping to
watched when Plex says so. Bounded by a probe cap (100), most-recently-touched
first; episodes are left to onDeck/history. No-signal rows are preserved (an
un-watch stays the daily reconcile's job).

Tests: watched-flip, onDeck+episode skip, and no-signal-preserved.
2026-08-28 01:34:02 +02:00
peter f52bc5898b release: 0.74.0 (improvement/quickfix-batch)
gate / frontend (push) Failing after 1m28s
gate / backend (push) Successful in 2m15s
2026-08-28 00:05:21 +02:00
peter d0033617e2 chore(release): 0.74.0 — quickfix batch (Plex sync/browse + Channel Manager) 2026-08-28 00:03:13 +02:00
peter f80ed115ad fix(plex,channels): deep-review fixes — durable watch-sync + no shared-link clobber
Deep-review of the quickfix batch surfaced two data/preference-loss bugs plus
smaller items; all fixed with tests.

- watch_sync _pull_apply: don't let the Plex-authoritative pull overwrite a DIRTY
  local row (a change whose immediate push failed) — that stranded the value
  forever since _repush_dirty only retries dirty rows. Leave it dirty for re-push.
- PlexProvider / ChannelsProvider: a partial shared link opened in a fresh tab
  persisted mount-time DEFAULTS over the account's stored scope/sort/filters
  (base read before the account was pinned). Persist/keep only URL-present dims;
  re-hydrate the omitted ones from storage.
- playback resume write-back: guard against out-of-order delivery (a superseded
  older push now drops) and evict the throttle map so it can't grow unbounded.
- PlexPlayer: announce mid-playback buffering to assistive tech (role=status +
  sr-only label); new plex.player.buffering (HU+EN).
- columnSort: hoist the shared key:dir sort codec used by both URL-state modules.
- Tests: dirty-guard + flag-settle + per-key throttle scoping + superseded-push
  (backend); hold-repeat/suppress-click/unmount + partial-link no-clobber +
  refetch-doesn't-clobber-undo (frontend). Backend pytest 475, FE vitest 519.
2026-08-27 23:54:44 +02:00
peter 7fbb996297 feat(channels): shareable manager view in the URL — tab+sort+column-filters (S8)
The Channels-manager content state now round-trips through the address bar,
so a manager view is bookmarkable/shareable — the S4 pattern applied to the
last content page the audit flagged.

- New lib/channelsUrlState.ts (channelsParamUpdates / channelsParamsToState /
  hasChannelsParams) under NAMESPACED keys ctab/csort/cf — channels-unique
  like ?v=/?pl/pscope, so no other provider reads or clears them and the
  feed's URL-stripping can't delete them. The mixed scalar/array FilterMap
  (sync/priority scalars + tags multi) rides in `cf` as a JSON blob so both
  shapes round-trip exactly. Only non-default values emit.
- ChannelsProvider mirrors {view, sort, filters} via useUrlMirror, page-gated
  on channels (off-page the keys clear; localStorage still holds the state and
  it re-mirrors on return). A shared link wins over stored state: readChannels
  UrlSeed captured once, applied before paint through the slice setters (so the
  account-pin re-hydrate sees them dirty and won't reload storage over the
  shared view), then persisted to the account once pinned so it survives a
  later plain reload — mirrors PlexProvider's seed/rehydrate. The name search
  stays ephemeral (out of the shared contract).
- NavigationProvider's page-switch nulls ctab/csort/cf alongside the plex keys
  so switching modules resets to the manager root.

Tests: channelsUrlState.dom.test.ts (9 round-trip) + ChannelsProvider.dom
seed/mirror (2); the existing tab-history test's beforeEach now resets the
query string too (the mirror writes it). FE vitest 503→514. FE-only.
2026-08-27 04:37:53 +02:00
peter 9cd4ae09d4 feat(views): editable saved-view detail — removable filter chips + undo + save-current (S7)
The Personal-views manager's detail pane becomes editable instead of
read-only:

- Captured filters render as REMOVABLE chips. Removing one prunes the set
  and writes it straight back to the stored view (PATCH filters), via the
  same live-row builders (useActiveFilters / buildPlexActiveFilters) that
  already compute the pruned snapshot — so the modal and the live chip row
  can't drift. The prune is wrapped in useUndoable + the shared UndoToolbar,
  so undo/redo re-persist the same way. The undoable draft is the single
  source of truth for the chip row, so a chip-prune refetch never clobbers
  the history.
- A "Save current" button beside Apply stores the module's currently-applied
  filters INTO the view (the live→stored inverse of Apply). Disabled when the
  view already equals the live filters (it's the active one). On save the
  draft re-seeds to the saved snapshot and clears the stale undo history.
- Rename gains a visible commit affordance: a check button beside the name
  input that enables once the name changes (Enter and blur still commit).

Provider gains two methods mirroring `rename`: updateFilters(id, filters)
and saveCurrent(id). One shared provider serves feed + plex, so both get it.
PATCH filters was already supported end-to-end — FE-only.

New i18n: views.saveCurrent / saveCurrentHint / saveCurrentUpToDate (HU+EN).
Tests: 5 new in PersonalViewsModal.dom.test.tsx (chip-remove persists, undo
re-persists, save-current fires + disables when up-to-date, rename commit).
2026-08-27 04:22:07 +02:00
peter 00527d0cf3 feat(channels): leading '+ Tag' pill + detail-page Unhide affordance (S6)
Item 6 — the tag add-trigger now LEADS the chip row (a stable left anchor)
instead of trailing it, so on the detail page (bounded=false) adding chips
grows the row to its right and the '+'/its editor popover no longer drift.
Compact icon in the width-bounded manager cell; a labelled '+ Tag' pill
where there's free width (the detail header).

Item 7 — surface the per-subscription 'hidden' flag on the detail GET
(channel_detail was omitting it) and add an Eye/EyeOff hide/unhide toggle
next to Block on the channel page, mirroring the manager. FE ChannelDetail
gains 'hidden'; new channel.* i18n keys (HU+EN).

Tests: +3 backend (channel_detail surfaces hidden, both states + unsubscribed
default). Gate green: backend pytest 471, FE vitest 498.
2026-08-27 04:02:25 +02:00
peter 8dc4976c2c feat(channels): horizontal −/+ priority stepper with pop-free inline edit (S5)
Redesign the shared PriorityStepper (channel manager cell + detail header)
from a value-plus-stacked-arrows control into a horizontal [−] value [+]
segmented stepper. The value now sits centred between the buttons and opens
an inline editor in a FIXED-width slot, so click-to-type no longer "pops"
the control wider (the old absolutely-positioned overlay grew w-6 → w-14).

One `size` prop drives the two surfaces: `sm` draws its own bordered pill
for the standalone table cell; `md` drops the outer border and keeps only
the segment dividers, blending into the detail header's lozenge instead of
double-bordering it. Hold-to-repeat, negatives, and the debounced-flush
persistence hook are unchanged.

The manager priority column widens 3.5rem → 5.5rem to fit the horizontal
control under fixed table layout.

Tests: 7 new DOM tests for the stepper interaction contract.
2026-08-27 03:29:42 +02:00
peter 56a2ecdec3 feat(plex): shareable Plex-browse state in the URL — scope+filters+sort+show (S4)
Mirror the Plex-browse content state into the address bar so a browse view is
bookmarkable/shareable, the Plex analogue of the feed's urlState. Keys are
NAMESPACED (pscope/psort/pshow/pf) rather than reusing the feed's bare
scope/sort/show: FeedFiltersProvider actively strips its own KEYS from the URL
whenever off the feed page, so a shared key would be deleted out from under the
Plex page. The namespaced keys are Plex-unique (like ?v=/?pl), so no other
provider ever reads or clears them.

- New lib/plexUrlState.ts: plexParamUpdates / plexParamsToState / hasPlexParams.
  The pf filter blob reuses the SAME appendPlexFilters wire serializer (new
  public plexFiltersToParams + inverse parsePlexFilters, kept next to each other
  so a new wire key can't be added on one side only).
- PlexProvider: URL-seed wins over localStorage at mount; the account-pin
  rehydrate persists the seed instead of clobbering it (mirrors feed's
  initialHadUrlFilters); useUrlMirror mirrors/clears the 4 keys, gated on
  page===plex. q/field stay out of the shareable contract.
- NavigationProvider: a page switch now also nulls the 4 plex keys.

Tests: 8 new round-trip cases in plexUrlState.dom.test.ts (FE vitest 483→491).
2026-08-27 03:14:54 +02:00
peter 8299ea72b7 feat(plex): 'last watched' browse sort + auto-apply in the in-progress view (S3)
Add a 'lastwatched' sort key to the unified Plex browse. Movies rank by
coalesce(progress_updated_at, watched_at); shows by max(coalesce(...)) across
their episodes (rollup.resumed_at, now broadened from progress-only so watched
episodes count too). Never-watched titles sink via nullslast.

Frontend auto-applies lastwatched:desc when entering the in-progress watch-state
view and restores the pre-entry sort on exit; a manual sort change while there
still wins. i18n HU+EN. Backend tests cover both union legs + mixed nullslast.
2026-08-27 02:50:41 +02:00
peter 6c918d1dff docs(plex): refresh watch-sync job docstrings for Plex-authoritative pull (S2)
The incremental/reconcile job docstrings still described last-write-wins and a
30-min cadence; update them to match the S2 behavior (Plex wins on divergence,
15-min incremental) so the "why" doesn't drift from the code.
2026-08-27 02:33:42 +02:00
peter 3f8257059d feat(plex): Plex-authoritative watch-sync — resume write-back + faster pull (S2)
Make Plex the source of truth for watch state:

- Write-back during playback: item_progress now pushes the resume position to
  Plex on the periodic checkpoint too (not only the final flush), throttled to
  one push per item every ~10s (in-memory, single-process rate limiter). The FE
  now also fires an immediate (debounced) push on seek.
- Faster incremental pull: 30 → 15 min (plex_watch_sync_interval_min).
- Plex-authoritative pull: _pull_apply resolves a genuine divergence to Plex's
  state instead of last-write-wins; the _same_state + synced_to_plex short-circuit
  still prevents a freshly-pushed row from bouncing. Removed the now-unused
  _LWW_TOLERANCE_S / _sift_ts.

Tests: 5 new in test_plex_watch_sync.py (Plex-wins resume/watched, no-bounce,
throttle final vs periodic).
2026-08-27 02:07:45 +02:00
peter 12d9561ba1 fix(plex): warm up HLS buffer on session start + player loading/buffering spinner
The Plex player stalled ~2s a few seconds into every auto-advanced next
episode. Root cause: each auto-advance spins up a cold ffmpeg HLS remux
session, and for a fresh start (start_s==0) start_session returned
immediately with zero buffer — hls.js drained the first segment, then
rebuffered waiting for the still-warming ffmpeg to write the next one.
Once ffmpeg gets ahead (stream-copy is faster than realtime) it never
stalls again.

- stream.py: before reporting a session ready, wait for the first
  _WARMUP_SEGMENTS (3) video segments to exist — the seg0 wait is now
  unconditional (was start_s>0 only) and warm-up runs for all starts.
  Best-effort and bounded: stop at the first segment not ready in time.
- PlexPlayer.tsx: a loading/buffering spinner. The existing overlay only
  covered the INITIAL load (!ready); add a `buffering` state driven by the
  video's waiting/stalled/seeking (set) and playing/canplay/seeked/pause
  (clear) events so a mid-playback stall or in-flight seek reads as
  "loading" instead of a frozen frame, and surface a fatal loadError even
  once playback has started. Spinner uses the shared Loader2.

Merges roadmap E6·item1 (spinner while loading/seeking) into this fix,
since the server-side warm-up deliberately holds ready a beat longer.

Tests: 3 new in test_plex_stream.py (warm-up waits for the first few
segments; stops at the first missing one; no warm-up when seg0 never
arrives). Gate: backend pytest 460 (Docker lane), FE tsc/lint/knip/
prettier + vitest 483.
2026-08-27 00:50:58 +02:00
peter fe09879edd release: 0.73.0 (feature/e8-feed-filtering)
gate / frontend (push) Failing after 1m22s
gate / backend (push) Successful in 2m4s
2026-08-26 05:28:11 +02:00
peter 7104767ff9 fix(e8): deep-review hardening + cut 0.73.0
Fixes from several multi-agent deep-review passes over the E8 feed-filtering &
channel-manager epic (S1-S4):
- feed_years buckets by UTC year to match the UTC-pinned year-filter bounds
- tag suggestions: examples ∩ subscribed (drop tagged-then-unsubscribed channels
  from the profile denominator); exclude hidden channels as suggestion targets
- channel manager: restore ghost chips + surface an error toast on a failed tag accept
- year filter: debounce the search term so the per-year facet doesn't refetch per keystroke
- channel tags: aria-label the "+N" overflow pills
- extract useTagSuggestions hook; add backend + hook tests (profile boundaries, cap,
  language exclusion, unsubscribed/hidden handling, UTC bucketing, bulk-accept)

Bump VERSION to 0.73.0 + release notes.
2026-08-26 05:25:47 +02:00
peter 17f8885fb1 fix(channels): collapse overflowing tag suggestions into a "+N" pill
With 2+ suggestions the narrow tags cell clipped the second ghost chip
mid-word right next to the edit "+", reading as the icon covering it. Route
the ghost chips through the same fit machinery the attached chips use: a
dedicated SuggestedChipsLane fits what the cell holds and collapses the rest
into a "+N" pill whose hover popover still one-tap accepts / dismisses each.

The lane is attached-XOR-ghost (suggestions are untagged-only, and accepting
one now drops the whole channel from the suggestion cache), so each lane owns
the full cell width instead of clipping the other.
2026-08-26 02:39:00 +02:00
peter 549aae67cc feat(channels): suggest your own tags for untagged channels from your tagging pattern (E8·S4)
Learn, per user tag, which auto TOPIC tags are characteristic of the channels
you've tagged it, then suggest that tag for still-untagged subscribed channels
whose topics cover enough of the profile. Language auto-tags are excluded from
the signal — near-universal within a language, they flooded every channel.

New GET /api/tags/suggestions (read-only, deterministic) backed by
app/sync/tagsuggest.py. Surfaced in the Channel Manager as dashed "ghost" chips
in the tags cell — one tap accepts (reuses the attach-tag endpoint), × dismisses
for the session — plus a header badge and a bulk "Accept N confident" (the
full-profile matches) behind a confirm. No silent writes (D4 = suggest-and-confirm).

Also reflow a pre-existing prettier drift in dateFilter.ts from E8·S2.
2026-08-26 02:18:34 +02:00
peter fbb961b5b7 feat(feed): hover infobox on feed cards — full title, description, tags (E8·S3) 2026-08-26 00:44:27 +02:00
peter edcea87ea5 feat(feed): filter by year — per-year count facet + year control (E8·S2)
Add a Year filter to the feed rail: quick chips for recent years plus a
'More…' popover listing all years with per-year video counts, sortable by
year or count. Selecting a year narrows the feed to that whole calendar
year (a published_after/before range applied in the api layer) and is
mutually exclusive with the Upload-date control.

Backend: new /api/feed/years endpoint + feed_years() — one grouped query
that honours every filter EXCEPT the publish-date narrowing (dropped), so
the year list stays a live drill-down and switching years never zeroes the
others. FE keys the list on the date-stripped filters, so picking a year
re-queries the feed but not the list.

Wires year through FeedFilters, the filterParams/URL (?year) round-trip,
the active-filter chip row, and the panel layout. i18n HU+EN.

Tests: feed_years count query (date narrowing ignored, other filters
honoured); dateFilter year-clearing; urlState year round-trip.
2026-08-25 02:55:12 +02:00
peter 2c843e549b fix(channels): tag-filter row folds overflow into "+N", keeps "…" pinned
The channel manager's tag-filter row wrapped the trailing "…" catalog-actions
menu onto a second line once the chips + Clear outgrew the width. Rework it as
one non-wrapping row: the TAGS editor trigger (left), a single-line chip lane
that collapses overflow into a "+N" pill whose popover holds the rest (still
toggleable), then Clear and the "…" menu pinned right — so "…" always stays on
the first line.

New TagFilterChips reuses the shared measure core (useChipOverflowFit +
fitChipsWithPill), the same compression the feed's applied-filter row uses.
2026-08-25 02:30:28 +02:00
peter 4ee0dab68f feat(feed,channels): "No tag" filter for untagged channels (E8·S1)
Filter to channels you haven't personally tagged in the channel manager
(no user-owned ChannelTag; auto language/topic tags don't count):

- Feed: a "No tag" sentinel in the "Your tags" sidebar combobox (exclusive
  with real tag picks) + a removable "Untagged" chip; new `untagged` bool
  feed param / `notag` URL key, persisted + coerced on every filter path.
- Channel manager: a virtual "No tag" chip in the tag-chip row filters the
  table to untagged channels (client-side, shared NO_TAG_SENTINEL).

Backend `untagged` is a plain channel-level narrowing, so feed_summary keeps
it applied in the base pass — the count/facet lockstep holds with no mirror.

Note: item E (per-tag counts) and item D (Any/All toggle) were already
shipped; S1 delivers only item B.
2026-08-25 01:44:34 +02:00
peter 5d7be687d3 test(views,layer): close the two polish-tail deep-review test gaps
UX-8 layerStack: add an integration test wired through the real useLayer
hook (popover document-listener + stamp) and a PlayerModal-shaped window
listener, driven by a real dispatched keydown. The popover close is wrapped
in flushSync so its layer is popped synchronously between the two listeners
-- the production mid-dispatch race -- making the escapeHandled stamp the
only thing that keeps the player from also closing. Mutation-checked:
removing the guard fails exactly this case.

SavedViews: cover feedCanSeedDefault's "yield to a bookmarked filter URL"
branch (a filter-bearing search string -> no seed) and the per-source
seededRef guard against the 60s refetch (a changed-list refetch re-runs the
seed effect; the guard blocks re-seed). The refetch test waits for the
changed list to reach the component so the effect has its chance to wrongly
re-fire. Both mutation-checked.

Frontend gate green: 464 vitest (+4). Backend untouched.
2026-08-24 23:41:49 +02:00
217 changed files with 15408 additions and 1142 deletions
+20 -4
View File
@@ -45,18 +45,34 @@ RUN adduser --disabled-password --gecos "" appuser \
&& mkdir -p /downloads \
&& chown appuser /app /downloads
# ffmpeg: merge separate video+audio streams + postprocessing. Deno: the JavaScript runtime
# yt-dlp uses to solve YouTube's "n" signature challenge for the web player clients (required
# alongside the PO token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode (E6 S1).
# intel-media-va-driver (iHD): the VA driver ffmpeg's h264_vaapi needs — libva2 ships in the base but
# no driver, so without this `vainfo`/`h264_vaapi` fail and transcode silently falls back to libx264.
# Harmless on hosts with no /dev/dri (the driver just goes unused). Deno: the JavaScript runtime yt-dlp
# uses to solve YouTube's "n" signature challenge for the web player clients (required alongside the PO
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg unzip curl ca-certificates \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
&& unzip -o /tmp/deno.zip -d /usr/local/bin \
&& chmod +x /usr/local/bin/deno \
&& rm /tmp/deno.zip \
&& rm -rf /var/lib/apt/lists/*
# Intel NEO OpenCL runtime (intel-opencl-icd) — needed by the Plex transcode's HDR/10-bit HW path, which
# tone-maps HDR→SDR on the GPU via `tonemap_opencl` (Gen 9.5 / UHD 630 can't tonemap in the VAAPI VPP, so
# OpenCL is the only HW route; without this the 10-bit branch drops to CPU libx264). NEO was REMOVED from
# Debian trixie (this base = python:3.12-slim = trixie), so it's pulled from **bookworm** — the 22.43 there
# still supports Gen 9.5 and its .deb runs on trixie (verified live on the prod UHD 630). The bookworm apt
# source is added only for this one install and removed after, so it can't drift the rest of the image.
# Harmless on hosts with no /dev/dri or no OpenCL use (the runtime just sits unused).
RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nComponents: main\nSigned-By: /usr/share/keyrings/debian-archive-keyring.pgp\n' > /etc/apt/sources.list.d/bookworm.sources \
&& apt-get update \
&& apt-get install -y --no-install-recommends -t bookworm intel-opencl-icd \
&& rm /etc/apt/sources.list.d/bookworm.sources \
&& rm -rf /var/lib/apt/lists/*
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
+1 -1
View File
@@ -1 +1 @@
0.72.1
0.79.0
@@ -0,0 +1,46 @@
"""download_credentials (S6: per-user, per-host yt-dlp cookie/auth store)
Each user stores their OWN login cookies (Netscape cookies.txt) for an auth-walled download host
(Facebook, Instagram, …), Fernet-encrypted at rest. A download of a yt-dlp-supported, auth-walled
link on that host is retried with the acting user's cookie. Unique per (user_id, host_key) so a
user holds at most one credential per host. Empty for every existing install; forward-only in
practice (a downgrade just drops the table + its secrets).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0074_download_credentials"
down_revision: Union[str, None] = "0073_saved_view_source"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"download_credentials",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("host_key", sa.String(length=64), nullable=False),
sa.Column("cookies_enc", sa.Text(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("user_id", "host_key", name="uq_download_credential"),
)
op.create_index("ix_download_credentials_user_id", "download_credentials", ["user_id"])
def downgrade() -> None:
op.drop_index("ix_download_credentials_user_id", table_name="download_credentials")
op.drop_table("download_credentials")
@@ -0,0 +1,26 @@
"""playlist synced_video_ids (S7: 3-way merge baseline for two-way YouTube sync)
The one-directional "local wins" push is replaced by a 3-way merge that classifies each item
against the last-synced state. That needs the raw ordered id set as a baseline, not just the
`synced_fingerprint` hash. Add a nullable `synced_video_ids` (JSON list) column; NULL on every
existing row means "no baseline yet" — the first merge then unions losslessly and records one.
Forward-only in practice (a downgrade just drops the column).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0075_playlist_sync_baseline"
down_revision: Union[str, None] = "0074_download_credentials"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("playlists", sa.Column("synced_video_ids", sa.Text(), nullable=True))
def downgrade() -> None:
op.drop_column("playlists", "synced_video_ids")
@@ -0,0 +1,41 @@
"""media_assets.media_kind (E7 S5a: the video|audio|file discriminator for the media library)
Adds a stored media-kind to each asset so the library can gate its actions on it (play/edit for
media, download-only for a generic uploaded file) instead of re-deriving from codecs in prose. The
column is the single home of the invariant `downloads.mediakind.derive` computes at finalize.
Backfill for existing rows: a stored video codec -> 'video', an audio-only codec -> 'audio', an
editor derivative (source_kind='edit', which carries no codecs but is always a video clip) ->
'video'. Rows still pending/errored (no codecs, not an edit) stay NULL — kind is set when they
finalize. Empty `media_kind` never reaches the UI: only `ready`/`done` rows render actions, and
those are all backfilled here.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0076_media_asset_kind"
down_revision: Union[str, None] = "0075_playlist_sync_baseline"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("media_assets", sa.Column("media_kind", sa.String(length=16), nullable=True))
op.execute(
"""
UPDATE media_assets
SET media_kind = CASE
WHEN vcodec IS NOT NULL THEN 'video'
WHEN acodec IS NOT NULL THEN 'audio'
WHEN source_kind = 'edit' THEN 'video'
ELSE NULL
END
"""
)
def downgrade() -> None:
op.drop_column("media_assets", "media_kind")
@@ -0,0 +1,54 @@
"""plex_items playback-class facet (E7: filter the feed by Direct Play / Direct Stream / Transcode)
Adds the columns the playback-class filter needs:
* transcode_hw — for a `playable="transcode"` item, whether the prod iGPU can hardware-transcode it
(codec HW-decodable AND 8-bit SDR) vs software-only. NULL for non-transcode items.
* video_bit_depth — the source video bit depth (from the Plex media Stream), NULL until probed.
* media_probed_at — watermark for the bit-depth backfill (NULL = not yet probed).
Backfill for existing rows: `transcode_hw` is derived from the ALREADY-stored `codec_video` alone (the
same HW-decodable allow-list `plex.sync.HW_DECODABLE_CODECS` / `plex.stream` use), so the filter works
immediately after migration. Bit depth is unknown for existing rows (the cheap Plex listing doesn't
carry it), so a 10-bit/HDR HEVC is provisionally 'hw' here and later refined to 'sw' by the bit-depth
backfill (`plex.sync._probe_media_batch`) once its Stream is probed. `video_bit_depth`/`media_probed_at`
start NULL.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0077_plex_playback_class"
down_revision: Union[str, None] = "0076_media_asset_kind"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("plex_items", sa.Column("transcode_hw", sa.Boolean(), nullable=True))
op.add_column("plex_items", sa.Column("video_bit_depth", sa.Integer(), nullable=True))
op.add_column(
"plex_items", sa.Column("media_probed_at", sa.DateTime(timezone=True), nullable=True)
)
op.create_index("ix_plex_items_transcode_hw", "plex_items", ["transcode_hw"])
# Codec-only derivation for existing rows — keep this codec set identical to
# plex.sync.HW_DECODABLE_CODECS. Bit depth is unknown here, so provisionally 8-bit (→ hw for a
# decodable codec); the bit-depth backfill downgrades 10-bit/HDR HEVC to sw later.
op.execute(
"""
UPDATE plex_items
SET transcode_hw = CASE
WHEN playable = 'transcode'
THEN lower(codec_video) IN ('h264','hevc','h265','vp9','vc1','mpeg2video')
ELSE NULL
END
"""
)
def downgrade() -> None:
op.drop_index("ix_plex_items_transcode_hw", table_name="plex_items")
op.drop_column("plex_items", "media_probed_at")
op.drop_column("plex_items", "video_bit_depth")
op.drop_column("plex_items", "transcode_hw")
@@ -0,0 +1,58 @@
"""users.password_revoked_at — remember that SEC-10 dropped a preset password
Revision ID: 0078_password_revoked_at
Revises: 0077_plex_playback_class
Create Date: 2026-09-02
SEC-10 (`auth.revoke_preset_password_if_unverified`) nulls `password_hash` when an account whose email
was never verified is attested (Google adoption, or admin approval). That left NO trace, and
password-reset eligibility keyed off `password_hash IS NOT NULL` — so such an account could neither
sign in (no password) nor request a reset link (not "eligible"): a permanent lockout with a silent,
uniform response at every step.
This column is that trace. A row with `password_revoked_at` set is a local account that HAD a password
and had it revoked, so the reset flow treats it as eligible again. Setting a password (reset-confirm or
/auth/set-password) clears the stamp. A Google-only account that never had a password keeps it NULL and
stays reset-ineligible by design — its owner adds a password from Settings while signed in.
Backfill: the signature is exact and narrow — a local (google_sub IS NULL), non-demo account with NO
password could ONLY have got there through the revoke. Registration always sets a hash, the setup
wizard sets one too, every Google row carries a google_sub (the adoption path assigns it BEFORE
revoking), and nothing in the codebase ever clears google_sub. Stamp those with the current time so
they become recoverable without operator intervention.
Deliberately NOT also requiring `email_verified = false`, even though the revoke only fires on an
unverified account: verifying is exactly what such a victim does NEXT. The revoke does not invalidate
the outstanding verification token, so clicking the emailed link sets email_verified=True while
password_hash stays NULL — and that row has no route back at all (reset ineligible, resend refuses an
already-verified address, and set-password needs a signed-in session the account can no longer obtain:
no password to sign in with, and no Google identity). Those are the victims who most need the stamp,
so the predicate must not exclude them.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0078_password_revoked_at"
down_revision: Union[str, None] = "0077_plex_playback_class"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"users",
sa.Column("password_revoked_at", sa.DateTime(timezone=True), nullable=True),
)
op.execute(
"UPDATE users SET password_revoked_at = now() "
"WHERE password_hash IS NULL "
"AND google_sub IS NULL "
"AND is_demo = false"
)
def downgrade() -> None:
op.drop_column("users", "password_revoked_at")
+4
View File
@@ -30,6 +30,7 @@ class AuditAction:
INVITE_APPROVE = "invite_approve"
INVITE_DENY = "invite_deny"
INVITE_ADD = "invite_add"
INVITE_DELETE = "invite_delete"
DEMO_ADD = "demo_add"
DEMO_REMOVE = "demo_remove"
DEMO_RESET = "demo_reset"
@@ -44,6 +45,9 @@ class AuditAction:
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
# The log itself was cleared (a single row survives the clear as tamper-evidence)
AUDIT_CLEAR = "audit_clear"
+37 -3
View File
@@ -186,6 +186,21 @@ class Settings(BaseSettings):
download_supervisor_poll_seconds: int = 3
# Grace between SIGTERM and SIGKILL when stopping a child (matches run_ffmpeg's escalation).
download_kill_grace_seconds: int = 5
# --- worker-child resource de-prioritization ---
# The download/edit child (yt-dlp + ffmpeg) shares ONE LAN box with the API, Postgres, and Plex
# playback. Without a priority handicap a CPU re-encode pegs every core for ~30s and starves web
# requests + playback until it finishes. The child lowers its OWN priority at startup — inherited
# by every ffmpeg/yt-dlp descendant — so any foreground work always wins. Each knob is best-effort
# and independently disable-able with 0.
download_child_nice: int = 10 # CPU nice increment applied to the child; 0 = don't renice
download_child_ionice_class: int = 3 # best-effort IO class (3 = idle); 0 = leave IO untouched
# Cap the browser-safe re-encode's encoder threads so a transcode can't claim every core. 0 =
# reserve one core for foreground work (max(1, cpu_count - 1)); >0 pins exactly that many.
download_ffmpeg_threads: int = 0
# Throttle the retention GC's whole-tree orphan file sweep so its rglob can't monopolize disk IO
# in the API process: sleep `pause_ms` after every `batch` files touched. pause_ms=0 disables.
download_gc_sweep_pause_ms: int = 20
download_gc_sweep_batch: int = 400
# Total cache size cap across ALL users (bytes). When >0 and exceeded, the GC evicts the
# least-recently-accessed ready assets until under it. 0 = unlimited (disk is the limit).
download_total_max_bytes: int = 0
@@ -193,6 +208,10 @@ class Settings(BaseSettings):
download_default_max_bytes: int = 5_368_709_120 # 5 GiB
download_default_max_concurrent: int = 2
download_default_max_jobs: int = 50
# Per-file cap for local uploads (E7). Unlike a download (whose size is unknown until the worker
# finishes), an upload's size is known at ingest, so this bounds a single file directly; the
# user's overall byte footprint is still governed by download_default_max_bytes.
download_upload_max_bytes: int = 524_288_000 # 500 MiB
# Retention: an asset expires this many days after its last access; the GC job then
# deletes it (once no job references it) after an additional grace window.
download_retention_days: int = 30
@@ -232,14 +251,29 @@ class Settings(BaseSettings):
plex_sync_interval_min: int = 360
# How often the incremental Plex→Siftlode watch-state reconcile runs, in minutes (Phase C):
# a cheap history + on-deck pull plus a re-push of any states that failed their immediate push.
plex_watch_sync_interval_min: int = 30
# 15 min (S2): with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# Plex-authoritative pull), a faster incremental pull keeps Siftlode close to Plex-side changes.
plex_watch_sync_interval_min: int = 15
# How often the FULL watch-state reconcile runs, in minutes (Phase C). A whole-section rescan
# that catches what the incremental history feed can't — notably an un-watch done on the Plex
# side — so it runs far less often (daily) than the incremental pass.
plex_watch_reconcile_interval_min: int = 1440
# Max concurrent transcodes for the P3 fallback. Low by default — CPU-only transcode is
# expensive; direct-serve (browser-compatible files) has no such limit.
# Max concurrent transcode sessions (playable="transcode"). Low by default — the software-fallback
# re-encode is CPU-heavy (the VAAPI hardware path is far cheaper and could sustain more); direct-serve
# and remux (stream-copy) are not capped by this.
plex_max_transcodes: int = 4
# --- Transcode (playable="transcode") re-encode tuning (E6 S1) ---
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
plex_transcode_hw: str = "auto"
# Software-fallback libx264 tuning (IGNORED on the VAAPI path). `preset` trades CPU for size; `crf`
# trades quality for size (lower = better + bigger). `max_height` caps the SOFTWARE transcode's output
# height (0 = keep source) so a CPU re-encode of a tall source stays affordable — the GPU path always
# encodes at source resolution.
plex_transcode_preset: str = "veryfast"
plex_transcode_crf: int = 20
plex_transcode_max_height: int = 1080
# Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
# dir under download_root. On localdev the download_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's download_root is fast
+258
View File
@@ -0,0 +1,258 @@
"""Per-user, per-host download cookies — the single source of truth for "does the acting user's
cookie apply to this download, and how does that keep the result private".
Two call sites depend on ONE consistent answer (house rule: an invariant lives in shared code, not
in prose):
* `enqueue` salts the MediaAsset cache signature with the user id WHEN a cookie applies, so the
authed download becomes a PRIVATE asset that is never cache-served to another user.
* the worker attaches the cookie file to yt-dlp ONLY when a cookie still applies AND the asset it
is downloading into is that private, salted identity — so a cookie can never poison a shared
asset (the load-bearing cross-user privacy fix).
Both go through `cookie_salt` / `authed_format_sig` / `host_key_for_url` here, so they can't drift.
Security controls enforced here:
* **host scope (SEC-12)** — `host_key_for_url` matches the download URL's netloc against a curated
registry via the netloc-scoped `_host_is`, so a cookie is only ever offered to the exact site it
belongs to; an arbitrary `?url=` (or a look-alike host) resolves to no host_key and gets no
cookie.
* **secret handling** — cookies are Fernet-encrypted at rest; the blob is only ever decrypted
in-worker into a 0600 file that is unlinked in a `finally`, and is never returned by a route,
logged, or written to a job's error/snapshot.
"""
import logging
import os
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential
from app.security import decrypt_optional, encrypt
log = logging.getLogger("siftlode.downloads.credentials")
class HostSpec:
"""A supported cookie host: a short canonical `key` (stored on the credential + folded into the
cache salt), a human `label` for the UI, and the netloc `domains` it matches."""
__slots__ = ("key", "label", "domains")
def __init__(self, key: str, label: str, domains: tuple[str, ...]):
self.key = key
self.label = label
self.domains = domains
# Curated registry (user's call: seed the common auth-walled hosts now). One `key` can cover several
# domains of the same site. Add a host by adding a row — the FE dropdown, host-scope guard, and salt
# all derive from this, so there is nothing else to wire up.
SUPPORTED_HOSTS: tuple[HostSpec, ...] = (
HostSpec("facebook", "Facebook", ("facebook.com", "fb.watch")),
HostSpec("instagram", "Instagram", ("instagram.com",)),
HostSpec("twitter", "X (Twitter)", ("twitter.com", "x.com")),
HostSpec("tiktok", "TikTok", ("tiktok.com",)),
HostSpec("reddit", "Reddit", ("reddit.com",)),
)
_BY_KEY: dict[str, HostSpec] = {h.key: h for h in SUPPORTED_HOSTS}
def is_supported_host(host_key: str) -> bool:
return host_key in _BY_KEY
def host_label(host_key: str) -> str:
spec = _BY_KEY.get(host_key)
return spec.label if spec else host_key
def host_key_for_url(url: str | None) -> str | None:
"""The supported host_key whose domains own `url`'s netloc, or None. Netloc-scoped (via
`_host_is`), so a look-alike host ('myfacebook.com', 'facebook.com.evil') never matches. When
several specs could match, the most-specific (longest matched domain) wins so the result is
deterministic across the enqueue and worker call sites."""
if not url:
return None
from urllib.parse import urlparse
host = urlparse(url).netloc.lower()
if not host:
return None
best_key: str | None = None
best_len = -1
for spec in SUPPORTED_HOSTS:
for dom in spec.domains:
if _host_is(host, dom) and len(dom) > best_len:
best_key, best_len = spec.key, len(dom)
return best_key
def cookie_salt(user_id: int, host_key: str) -> str:
"""The cache-signature salt for a cookie-authed download. Keyed by user id + host so the authed
asset is private to that user (never dedup-shared) and stable across that user's own re-adds."""
return f"cookie:{user_id}:{host_key}"
def authed_format_sig(spec: dict, user_id: int, host_key: str) -> str:
"""The salted `format_sig` an authed download for (user, host) resolves to — the private asset
identity. Both enqueue (to create it) and the worker (to recognise it) compute it HERE."""
return formats.format_sig(spec, salt=cookie_salt(user_id, host_key))
def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
"""At enqueue: the cache salt to apply if this user has a cookie for `url`'s host, else None (a
normal shared download). Returns the salt string so the caller passes it straight to
`format_sig(spec, salt=...)`."""
host_key = host_key_for_url(url)
if host_key is None:
return None
if get_credential(db, user_id, host_key) is None:
return None
return cookie_salt(user_id, host_key)
# --- storage (CRUD) ---------------------------------------------------------------------------
def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredential | None:
return db.execute(
select(DownloadCredential).where(
DownloadCredential.user_id == user_id, DownloadCredential.host_key == host_key
)
).scalar_one_or_none()
class InvalidCookies(ValueError):
"""The pasted cookies aren't a usable Netscape cookies.txt — surfaced as a 400 by the route."""
def validate_cookies(raw: str) -> str:
"""Sanity-check a pasted cookies.txt (Netscape format) and return the cleaned text. yt-dlp reads
the Netscape format: comment/`#` lines plus TAB-separated cookie lines (domain, flag, path,
secure, expiry, name, value). We don't fully parse — just reject obvious non-cookie paste (empty,
or no tab-separated data line) so a mistaken paste fails HERE with a clear message rather than
silently producing a cookie file yt-dlp ignores."""
text = (raw or "").strip()
if not text:
raise InvalidCookies("empty")
has_data_line = any(
line.strip() and not line.lstrip().startswith("#") and "\t" in line
for line in text.splitlines()
)
if not has_data_line:
raise InvalidCookies("format")
# Netscape cookie files must end with a newline for some parsers; normalise.
return text if text.endswith("\n") else text + "\n"
def upsert_credential(db: Session, user_id: int, host_key: str, raw: str) -> DownloadCredential:
"""Create or replace this user's cookie blob for `host_key`. Caller has validated host_key is
supported and `raw` via `validate_cookies`. Does NOT commit (the route commits with the audit
row so the trail can't drift). The blob is encrypted here; the plaintext never leaves this call."""
cred = get_credential(db, user_id, host_key)
enc = encrypt(raw)
if cred is None:
cred = DownloadCredential(user_id=user_id, host_key=host_key, cookies_enc=enc)
db.add(cred)
else:
cred.cookies_enc = enc
return cred
def delete_credential(db: Session, user_id: int, host_key: str) -> bool:
"""Delete this user's cookie for `host_key`. Returns whether a row existed. Does NOT commit."""
cred = get_credential(db, user_id, host_key)
if cred is None:
return False
db.delete(cred)
return True
def status_list(db: Session, user_id: int) -> list[dict]:
"""The supported hosts with this user's configured/updated state — the write-only Settings view.
Never includes the blob."""
have = {
c.host_key: c
for c in db.execute(
select(DownloadCredential).where(DownloadCredential.user_id == user_id)
).scalars()
}
out = []
for spec in SUPPORTED_HOSTS:
cred = have.get(spec.key)
out.append(
{
"host_key": spec.key,
"label": spec.label,
"configured": cred is not None,
"updated_at": cred.updated_at.isoformat() if cred is not None else None,
}
)
return out
# --- worker: resolve + write the cookie file --------------------------------------------------
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset_format_sig: str,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download, or None.
The double gate that prevents cross-user leakage:
1. a credential must still exist for `url`'s host (host-scoped via `host_key_for_url`), AND
2. the asset being downloaded into must BE this user's private, salted identity
(`asset_format_sig == authed_format_sig(...)`).
If (2) fails the asset is the SHARED (unsalted) one — attaching the cookie would poison the
global cache with a private-group video, so we attach nothing. An unreadable blob (rotated key →
InvalidToken) also yields None; the download then fails honestly on the login wall (S3's
`login_required`)."""
host_key = host_key_for_url(url)
if host_key is None:
return None
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset_format_sig != authed_format_sig(spec, user_id, host_key):
return None # shared asset — never attach a cookie to a non-private identity
raw = decrypt_optional(cred.cookies_enc)
if not raw:
log.warning(
"download cookie for user %s host %s could not be decrypted (rotated key?) — "
"proceeding without it",
user_id,
host_key,
)
return None
return _write_cookiefile(raw, staging)
def _write_cookiefile(raw: str, staging: Path) -> Path:
"""Write the cookie blob to a private 0600 file under `staging/.auth/` and return its path.
Deliberately NOT in the staging root: the download retry loop unlinks every file it finds there,
and `_produced_file` picks the largest file there as the output — either would corrupt a
cookies.txt in the root. A `.auth` SUBDIR survives both (they skip directories) and is still
removed by the parent worker's staging cleanup on a hard kill. The caller unlinks it in a
`finally` on the normal/exception paths."""
auth_dir = staging / ".auth"
auth_dir.mkdir(mode=0o700, exist_ok=True)
path = auth_dir / "cookies.txt"
# Open with 0600 from the start (don't create world-readable then chmod).
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
try:
os.write(fd, raw.encode("utf-8"))
finally:
os.close(fd)
return path
+56 -32
View File
@@ -25,6 +25,7 @@ import hashlib
import json
import logging
import math
import os
import queue
import shutil
import subprocess
@@ -32,9 +33,21 @@ import threading
import time
from pathlib import Path
from app.config import settings
log = logging.getLogger("siftlode.edit")
def encode_thread_cap() -> int:
"""How many encoder threads a browser-safe re-encode may use. `download_ffmpeg_threads` when the
admin pinned one (>0), else reserve a core for foreground work (max(1, cpu_count - 1)). Shared by
the ffmpeg `-threads` flag and the child's `OMP_NUM_THREADS` so both honour the same budget."""
n = settings.download_ffmpeg_threads
if n > 0:
return n
return max(1, (os.cpu_count() or 2) - 1)
class EditAborted(Exception):
"""Raised by run_ffmpeg when the job was paused/canceled mid-encode."""
@@ -425,18 +438,42 @@ _BROWSER_SAFE_VCODEC = ("h264",)
_BROWSER_SAFE_ACODEC = ("aac", "mp3")
def parse_probe_streams(payload: str) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) out of ffprobe's JSON stream list, lowercased.
JSON, not the flat `default=nw=1` output: there the fields arrive in ffprobe's own struct
order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line "remember the last
codec_type" parser attributes every name to the PREVIOUS stream. Measured on real downloads:
an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download."""
def run_ffprobe_json(
path: Path,
*,
show_streams: bool = False,
show_format: bool = False,
entries: str | None = None,
) -> dict | None:
"""One ffprobe call → its parsed JSON dict, or None on any failure (timeout, non-zero exit,
missing binary, unparseable output). The SINGLE ffprobe-JSON-invocation point in the downloads
package: the browser-playability codec probe here and the shared media probe (`probe.probe_media`,
used by both the download worker's finalize and the upload ingest) both go through it, so the
subprocess flags/timeout/error handling live in one place instead of drifting copies."""
args = ["ffprobe", "-v", "error"]
if show_streams:
args.append("-show_streams")
if show_format:
args.append("-show_format")
if entries:
args += ["-show_entries", entries]
args += ["-of", "json", str(path)]
try:
streams = json.loads(payload or "{}").get("streams") or []
except ValueError:
return None, None
out = subprocess.run(args, capture_output=True, text=True, timeout=30, check=True).stdout
return json.loads(out)
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
return None
def _codecs_from_streams(streams: list) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) lowercased from an ffprobe JSON stream list — the first of each kind.
Parsed from the JSON `streams` list, NOT ffprobe's flat `default=nw=1` output: there the fields
arrive in ffprobe's own struct order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line
"remember the last codec_type" parser attributes every name to the PREVIOUS stream. Measured on real
downloads: an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download. Reading each
stream dict by its own `codec_type` is immune to that ordering."""
vcodec = acodec = None
for s in streams:
name = (s.get("codec_name") or "").lower() or None
@@ -451,26 +488,10 @@ def parse_probe_streams(payload: str) -> tuple[str | None, str | None]:
def probe_codecs(path: Path) -> tuple[str | None, str | None]:
"""Return (video_codec, audio_codec) lowercased via one ffprobe call; None for a missing
stream or if ffprobe fails (a failed probe is treated as 'leave it alone' upstream)."""
try:
out = subprocess.run(
[
"ffprobe",
"-v",
"error",
"-show_entries",
"stream=codec_type,codec_name",
"-of",
"json",
str(path),
],
capture_output=True,
text=True,
timeout=30,
check=True,
).stdout
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError):
data = run_ffprobe_json(path, entries="stream=codec_type,codec_name")
if data is None:
return None, None
return parse_probe_streams(out)
return _codecs_from_streams(data.get("streams") or [])
def probe_duration(path: Path) -> int:
@@ -510,11 +531,12 @@ def browser_transcode_flags(
def build_browser_safe_cmd(
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool, threads: int | None = None
) -> list[str]:
"""ffmpeg command that lands `src` as a browser-safe mp4 (faststart): re-encode only the
stream(s) that need it (libx264 / aac), stream-copy the rest, always into mp4. `-progress
pipe:1` streams machine-readable progress for run_ffmpeg."""
pipe:1` streams machine-readable progress for run_ffmpeg. `threads` caps the encoder so a
re-encode can't claim every core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
@@ -534,6 +556,8 @@ def build_browser_safe_cmd(
else ["-c:v", "copy"]
)
cmd += ["-c:a", "aac", "-b:a", "192k"] if reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
cmd += ["-movflags", "+faststart", str(dest)]
return cmd
+14
View File
@@ -24,6 +24,16 @@ _RULES: tuple[tuple[str, str], ...] = (
# Some extractors (e.g. Reddit) gate content behind a logged-in account: "Account
# authentication is required." — an auth wall we deliberately don't cross, not a transient error.
("authentication is required", "login_required"),
# Closed-group / private-post auth walls (Facebook groups, Instagram private, …). yt-dlp phrases
# these many ways; the common thread is "you must be signed in" or its own suggestion to supply
# cookies. Treating the cookie suggestion as an auth signal is what makes a login-walled link read
# honestly (and stay non-retryable) when the acting user has no usable cookie for that host.
("log in to", "login_required"),
("login required", "login_required"),
("you must be logged in", "login_required"),
("only available for registered users", "login_required"),
("use --cookies", "login_required"),
("--cookies-from-browser", "login_required"),
# Geo-restricted.
("in your country", "geo_blocked"),
("available in your location", "geo_blocked"),
@@ -42,6 +52,10 @@ _RULES: tuple[tuple[str, str], ...] = (
("account associated with this video has been terminated", "unavailable"),
("this video is not available", "unavailable"),
("this video has been removed", "unavailable"),
# yt-dlp doesn't support this site at all ("Unsupported URL: …") — distinct from an auth wall on a
# supported site. Non-retryable: no cookie or retry can make an unsupported host downloadable.
("unsupported url", "unsupported_host"),
("no video could be found in this url", "unsupported_host"),
# DRM.
("drm protected", "drm"),
("drm-protected", "drm"),
+57 -48
View File
@@ -5,13 +5,16 @@ of truth for what file gets produced. `format_sig` hashes exactly the fields tha
OUTPUT BYTES, so two users picking equivalent settings share one cached MediaAsset.
`build_ydl_opts` turns a spec into a yt-dlp options dict (format selector + postprocessors).
spec shape (all keys optional; defaults applied by `normalize`):
spec shape (all keys optional; defaults + browser-playability clamp applied by `normalize`):
mode "av" (audio+video) | "v" (video-only) | "a" (audio-only)
max_height int | null (null = best)
container "mp4" | "mkv" | "webm" | null (merge/remux target for av/v)
audio_format "m4a" | "mp3" | "opus" | null (extract target for mode "a")
vcodec "h264" | "vp9" | "av1" | null (preference, best-effort)
container always coerced to "mp4" (the only universally browser-playable container)
audio_format "m4a" | "mp3" (mode "a" extract target; anything else — e.g. opus — → "m4a")
vcodec always coerced to null → H.264 selected (see `normalize`); no opt-out
embed_subs bool embed_chapters bool embed_thumbnail bool sponsorblock bool
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser (E7); the
`container`/`vcodec` keys survive only so an old stored profile deserializes without error.
"""
import hashlib
@@ -50,10 +53,12 @@ _DEFAULTS = {
# prefer H.264+AAC (browser/iOS-compatible) rather than yt-dlp's default VP9/Opus best.
# v3: mp4 compat now *selects* H.264 (avc1) with a progressive fallback + re-encodes the rare
# AV1/VP9-only result to H.264, so cached AV1 assets must re-derive (fixes iOS broken-player).
_SIG_VERSION = 3
# v4: EVERY download is clamped to the mp4/H.264/AAC compat profile (no more mkv/webm/VP9/AV1/Opus
# opt-out), so any asset cached under an old non-compat spec — or a stale non-H.264 mp4 that slipped
# the v3 net — must re-derive through the current browser-safe guarantee.
_SIG_VERSION = 4
_SPONSORBLOCK_CATEGORIES = ["sponsor", "selfpromo", "interaction"]
_VCODEC_SORT = {"h264": "vcodec:h264", "vp9": "vcodec:vp9", "av1": "vcodec:av01"}
def normalize(spec: dict) -> dict:
@@ -76,6 +81,18 @@ def normalize(spec: dict) -> dict:
# Audio-only files can't embed subtitles/thumbnails as video; keep the flags meaningful.
if s["mode"] == "a":
s["embed_subs"] = False
# Browser-playability guarantee (E7): the Download module exists to download → share → PLAY
# in-browser, so every output must play in EVERY browser, iOS/Safari/WebKit included. The only
# universally-safe combo is mp4 + H.264 + AAC/MP3, so we CLAMP the spec to it HERE — the single
# point every path (picker, stored profile, raw API) flows through — instead of only hiding
# options in the UI. Non-mp4 containers (mkv doesn't play at all; webm/VP9/AV1 fail iOS) and any
# explicit vcodec are coerced to the compat profile; Opus audio (Safari-flaky as a bare file)
# falls back to AAC/m4a. A source that exposes no H.264 is still caught by the worker's
# browser-safe re-encode (`_ensure_browser_playable`), so playback is guaranteed end-to-end.
s["container"] = "mp4"
s["vcodec"] = None
if s["audio_format"] not in ("m4a", "mp3"):
s["audio_format"] = "m4a"
return s
@@ -98,12 +115,17 @@ def build_ydl_opts(
outtmpl: str,
progress_hook,
extra: dict | None = None,
cookiefile: str | None = None,
) -> dict:
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
`extra` is source-specific options contributed by the SourceAdapter (e.g. YouTube's
`extractor_args` with the player-client set + bgutil PO-token base_url); it is merged onto the
generic options below. Everything here is source-agnostic."""
generic options below. Everything here is source-agnostic.
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
merge so a source adapter can never clobber it."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
@@ -132,6 +154,11 @@ def build_ydl_opts(
if extra:
opts.update(extra)
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
# be able to override the authenticated session the worker resolved for this specific user.
if cookiefile:
opts["cookiefile"] = cookiefile
if s["mode"] == "a":
opts["format"] = "bestaudio/best"
opts["postprocessors"].append(
@@ -143,51 +170,33 @@ def build_ydl_opts(
)
else:
hcap = f"[height<=?{h}]" if h else ""
# Compatibility-first for mp4 (the universal container): produce H.264 video + AAC audio
# so shared/downloaded files play in EVERY browser, including iOS/Safari. WebKit (which
# every iOS browser, Brave included, is forced to use) decodes only H.264/H.265 + AAC
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there, even though Chromium
# plays it fine. A custom profile that explicitly sets `vcodec` opts out.
compat = s["container"] == "mp4" and s["vcodec"] is None
# Every video download is the mp4/H.264 compat profile (normalize() clamps container→mp4,
# vcodec→None), so shared/downloaded files play in EVERY browser, iOS/Safari included. WebKit
# (which every iOS browser, Brave included, is forced to use) decodes only H.264/H.265 + AAC
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there — so H.264 is selected at the
# SELECTION level, not merely preferred: a sort key can't conjure an H.264 stream a source
# doesn't expose as video-only. A source that ships only non-H.264 is caught after download by
# the worker's browser-safe re-encode (`_ensure_browser_playable`).
if s["mode"] == "v":
if compat:
# Video-only: keep it audio-free. Prefer an H.264 *video-only* stream, then any
# video-only, and only fall to a progressive (audio-bearing) `best` as a last
# resort — same audio-free-when-possible guarantee as the old `bestvideo/best`.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}/bestvideo{hcap}"
f"/best[vcodec^=avc1]{hcap}/best{hcap}"
)
else:
opts["format"] = f"bestvideo{hcap}/best{hcap}"
# Video-only: keep it audio-free. Prefer an H.264 video-only stream, then any video-only,
# and only fall to a progressive (audio-bearing) `best` as a last resort.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}/bestvideo{hcap}/best[vcodec^=avc1]{hcap}/best{hcap}"
)
else: # av
if compat:
# Prefer H.264 at the SELECTION level, not just via format_sort: a sort key is only
# a *preference* and can't conjure an H.264 stream a source doesn't expose as
# video-only. A non-YouTube source (e.g. Facebook) may ship AV1 as the best
# video-only stream and H.264 only as a *progressive* (a+v) format — which
# `bestvideo` skips — so `bestvideo+bestaudio` picked AV1 and iOS broke. This chain
# tries avc1 video-only, then the avc1 progressive, then anything.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}+bestaudio[acodec^=mp4a]"
f"/best[vcodec^=avc1]{hcap}"
f"/bestvideo{hcap}+bestaudio/best{hcap}"
)
else:
opts["format"] = f"bestvideo{hcap}+bestaudio/best{hcap}"
if s["container"]:
opts["merge_output_format"] = s["container"]
sort = []
# Keep the codec preference as a tiebreaker (ranks H.264/AAC above resolution among the
# selected candidates); the selector above is the hard guarantee, this is belt-and-braces.
if compat:
sort += ["vcodec:h264", "acodec:aac"]
# avc1 video-only + m4a audio, then the avc1 progressive, then anything (re-encoded).
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}+bestaudio[acodec^=mp4a]"
f"/best[vcodec^=avc1]{hcap}"
f"/bestvideo{hcap}+bestaudio/best{hcap}"
)
opts["merge_output_format"] = "mp4"
# Codec preference tiebreaker (ranks H.264/AAC among the selected candidates); the selector
# above is the hard guarantee, this is belt-and-braces.
sort = ["vcodec:h264", "acodec:aac"]
if h:
sort.append(f"res:{h}")
if s["vcodec"] in _VCODEC_SORT:
sort.append(_VCODEC_SORT[s["vcodec"]])
if sort:
opts["format_sort"] = sort
opts["format_sort"] = sort
# SponsorBlock first (marks segments), then strip them from the media + chapters.
if s["sponsorblock"]:
+14
View File
@@ -86,11 +86,25 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
cutoff = time.time() - _ORPHAN_MIN_AGE_S
removed = 0
# Throttle the whole-tree walk: this runs in the API process, and an unpaced rglob+stat over a
# large download root monopolizes disk IO long enough to make web requests + playback stutter.
# Yield the IO channel after every `batch` files touched so foreground work interleaves.
pause_s = max(0, settings.download_gc_sweep_pause_ms) / 1000.0
batch = max(1, settings.download_gc_sweep_batch)
touched = 0
def _throttle() -> None:
nonlocal touched
touched += 1
if pause_s and touched % batch == 0:
time.sleep(pause_s)
for entry in root.iterdir():
if entry.name.startswith("."): # .edits / .plex-* / .staging / .storyboards + any dotfile
continue
files = [entry] if entry.is_file() else [p for p in entry.rglob("*") if p.is_file()]
for f in files:
_throttle()
try:
if f.resolve() in keep or f.stat().st_mtime > cutoff:
continue
+9 -1
View File
@@ -80,9 +80,13 @@ def public_meta(
Resolves the owner's per-download display overrides (custom title/channel) over the asset's
auto-extracted values, and surfaces the source + any extra reference links as clickable URLs."""
from app.downloads import service
from app.downloads import mediakind, service
source_url = service.reference_url(job, asset)
# Audio-only assets (mp3 rips, extracted audio) can't play inside a <video> element on WebKit
# (all iOS browsers), so the watch page renders a native <audio> instead. A generic uploaded
# file (media_kind='file') has no player at all — the watch page shows file info + a download.
# Both branches read the stored media_kind (the single source), not a re-derived codec check.
return {
"title": (job and job.display_name) or asset.title,
"uploader": (job and job.display_uploader) or asset.uploader,
@@ -92,6 +96,10 @@ def public_meta(
"duration_s": asset.duration_s,
"width": asset.width,
"height": asset.height,
"audio": asset.media_kind == mediakind.AUDIO,
"media_kind": asset.media_kind,
"size_bytes": asset.size_bytes,
"container": asset.container,
"allow_download": link.allow_download,
"file_url": file_url,
"poster_url": poster_url,
+23
View File
@@ -0,0 +1,23 @@
"""The media-kind discriminator for a library asset — the single home of one invariant.
`MediaAsset.media_kind` is `"video" | "audio" | "file"`. It decides which library actions apply:
media (video/audio) gets play/edit/share; a generic uploaded file gets download + share only, and
is never rendered inline (see the force-download serving in the file/watch routes). Every finalize
path computes it HERE — the download worker + the upload ingest via `derive`, the editor by
carrying the source's kind over — so the rule lives in code, not in a comment that can drift (an
earlier ad-hoc `vcodec is None and acodec is not None` audio check lived only in links.py).
"""
VIDEO = "video"
AUDIO = "audio"
FILE = "file"
def derive(vcodec: str | None, acodec: str | None) -> str:
"""Media kind from the probed codecs: a video stream -> video, audio-only -> audio, neither
(a non-media upload: pdf/zip/docx/…) -> file."""
if vcodec:
return VIDEO
if acodec:
return AUDIO
return FILE
+48 -14
View File
@@ -8,13 +8,14 @@ from datetime import date, datetime, timedelta, timezone
from pathlib import Path
import yt_dlp
from sqlalchemy import select
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import errors as dlerrors
from app.downloads import adapters, formats, service, storage
from app.downloads import adapters, credentials, formats, mediakind, probe, service, storage
from app.downloads.jobdb import (
_edit_staging_dir,
_job_status,
@@ -304,10 +305,13 @@ def _media_info(info: dict) -> dict:
return {**info, **chosen}
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter):
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter, cookiefile=None):
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
error); other sources make a single attempt. Returns the info dict of the successful attempt."""
error); other sources make a single attempt. Returns the info dict of the successful attempt.
`cookiefile` (S6) is the acting user's per-host cookies.txt, already gated by
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated."""
client_sets = adapter.client_sets()
outtmpl = str(staging / "%(id)s.%(ext)s")
last_exc: Exception | None = None
@@ -317,6 +321,7 @@ def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter):
outtmpl,
_make_progress_hook(job_id, asset_id),
extra=adapter.ydl_opts_extra(clients),
cookiefile=cookiefile,
)
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
_touch_job(job_id) # stall-watchdog heartbeat: extraction writes no progress, so a slow /
@@ -381,7 +386,9 @@ def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict
# re-encode still shows real progress instead of a frozen 0%.
total_s = int(info["duration"]) if info.get("duration") else editmod.probe_duration(produced)
editmod.run_ffmpeg(
editmod.build_browser_safe_cmd(produced, dest, reenc_v, reenc_a),
editmod.build_browser_safe_cmd(
produced, dest, reenc_v, reenc_a, threads=editmod.encode_thread_cap()
),
total_s=total_s,
on_progress=_ffmpeg_progress_writer(job_id, "processing"),
should_cancel=_cancel_poll(job_id),
@@ -395,8 +402,11 @@ def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict
return dest
def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spec: dict) -> None:
def _download(
job_id: int, asset_id: int, source_kind: str, source_ref: str, spec: dict, user_id: int
) -> None:
staging = _staging_dir(asset_id)
cookiefile: Path | None = None
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
@@ -405,7 +415,27 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
_set_job(job_id, phase="downloading", progress=0)
adapter = adapters.resolve_adapter(source_kind, source_ref)
url = service.source_url(source_kind, source_ref)
info = _media_info(_extract_with_fallback(job_id, asset_id, url, spec, staging, adapter))
# S6: attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
# this asset is that user's private, salted identity (the double gate lives in credentials).
with SessionLocal() as cdb:
asset_sig = cdb.execute(
select(MediaAsset.format_sig).where(MediaAsset.id == asset_id)
).scalar_one_or_none()
if asset_sig is not None:
cookiefile = credentials.resolve_cookiefile(
cdb, user_id, url, asset_sig, spec, staging
)
info = _media_info(
_extract_with_fallback(
job_id,
asset_id,
url,
spec,
staging,
adapter,
cookiefile=str(cookiefile) if cookiefile else None,
)
)
# Re-resolve now that yt-dlp's info reveals the true source (a bare/redirected url only
# identifies post-extraction) so the .nfo uniqueid type + uploader_url agree on one source.
adapter = adapters.resolve_adapter(source_kind, source_ref, info)
@@ -445,15 +475,15 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
asset.rel_path = rel
asset.size_bytes = size
asset.duration_s = meta.duration_s
asset.width = int(info["width"]) if info.get("width") else None
asset.height = int(info["height"]) if info.get("height") else None
# Codecs/dims come from a PROBE of the finished file — the ground truth — NOT yt-dlp's
# `info` dict, whose codec fields are absent for some sources (Facebook among them):
# deriving media_kind from those nulls misclassified a real H.264 video as a
# download-only 'file' with no in-app play/edit. Same probe the upload ingest uses.
asset.vcodec, asset.acodec, asset.width, asset.height = probe.resolve_media_facts(
final, info
)
asset.container = ext
asset.vcodec = (
info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
)
asset.acodec = (
info.get("acodec") if info.get("acodec") not in (None, "none") else None
)
asset.media_kind = mediakind.derive(asset.vcodec, asset.acodec)
asset.title = meta.title
asset.uploader = meta.uploader
asset.uploader_url = adapter.uploader_url(info)
@@ -515,6 +545,10 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
db.commit()
log.warning("job %s failed: %s", job_id, msg)
finally:
# Unlink the decrypted cookie file explicitly (the rmtree below removes the whole staging
# tree too, but never leave a plaintext session cookie on disk a moment longer than needed).
if cookiefile is not None:
cookiefile.unlink(missing_ok=True)
shutil.rmtree(staging, ignore_errors=True)
+107
View File
@@ -0,0 +1,107 @@
"""Ground-truth media probe for a produced library file — the SINGLE place both ingest paths ask
"what is actually on disk?".
`media_kind` (and the stored `vcodec`/`acodec`/`width`/`height`) must reflect the real file, not an
extractor's metadata: yt-dlp's `info` dict omits codec fields for some sources (e.g. Facebook), and
deriving the kind from those nulls misclassified a genuine H.264 video as a download-only `file`
(no in-app play). So the download worker (`pipeline`) and the local upload ingest (`upload`) both
probe the finished file here; `mediakind.derive` then turns the probed codecs into the kind.
"""
from pathlib import Path
from app.downloads import edit as editmod
# ffprobe container formats that mean "this is a still image / image sequence", NOT playable media.
# A standalone jpg/png/gif/webp/bmp reports a real video stream (codec mjpeg/png/…) with no
# `attached_pic` flag, so without this guard it would classify as media_kind='video' and get a Play
# button + a broken in-app <video> player (and inline serving on the public share). Keyed on the
# container format so an animated gif/webp also lands as a generic 'file' (no browser plays those in
# a <video> either). SVG and anything else ffprobe can't decode already fails the probe → 'file'.
_IMAGE_FORMATS = {
"image2",
"image2pipe",
"png_pipe",
"mjpeg",
"mjpeg_pipe",
"jpeg_pipe",
"gif",
"webp_pipe",
"bmp_pipe",
"tiff_pipe",
"ico",
"svg_pipe",
}
def _facts_from_probe(
data: dict,
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) parsed from a NON-None ffprobe JSON dict. A still
image / image-sequence container yields no vcodec (→ generic 'file'); a video stream flagged
`attached_pic` (embedded cover art in an mp3/m4a) is skipped — it's a thumbnail, not a real video
track, so an audio file with album art still classifies as audio."""
fmt = data.get("format") or {}
fmt_names = {n.strip() for n in (fmt.get("format_name") or "").split(",")}
if fmt_names & _IMAGE_FORMATS:
return None, None, None, None, None
vcodec = acodec = None
width = height = None
for stream in data.get("streams", []):
kind = stream.get("codec_type")
if kind == "video" and vcodec is None:
if (stream.get("disposition") or {}).get("attached_pic"):
continue # cover art, not a video track
vcodec = stream.get("codec_name")
width = stream.get("width")
height = stream.get("height")
elif kind == "audio" and acodec is None:
acodec = stream.get("codec_name")
duration = None
try:
raw = fmt.get("duration")
if raw is not None:
duration = int(float(raw))
except (TypeError, ValueError):
duration = None
return vcodec, acodec, width, height, duration
def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) from one ffprobe call; all None if ffprobe fails
or the file isn't playable media (an unprobeable file — or a still image — is treated as a generic
non-media file, so it becomes a download-only 'file' rather than a broken 'video')."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
if data is None:
return None, None, None, None, None
return _facts_from_probe(data)
def resolve_media_facts(
path: Path, info: dict
) -> tuple[str | None, str | None, int | None, int | None]:
"""(vcodec, acodec, width, height) for the stored download asset: PROBE the finished file on disk
(the ground truth). yt-dlp's `info` omits codec/dimension fields for some sources (Facebook among
them), so deriving media_kind from it there labelled a real H.264 video a download-only 'file'.
Precedence is by whether the probe RAN, not per-field: if ffprobe read the file, its codec verdict
is authoritative — INCLUDING a deliberate None (cover-art-only audio, a still image). A per-field
`probe or info` fallback would let `info`'s stale vcodec resurrect a video track the file doesn't
have (yt-dlp reports a progressive `vcodec` for an audio-mode download whose video it then strips),
mislabelling an audio download as a broken in-app 'video'. Only when ffprobe could NOT read the
file at all do we fall back to `info` (the pre-probe behaviour, a safety net); the cosmetic
dimensions still gap-fill from `info` in either case."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
info_v = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
info_a = info.get("acodec") if info.get("acodec") not in (None, "none") else None
info_w = int(info["width"]) if info.get("width") else None
info_h = int(info["height"]) if info.get("height") else None
if data is None:
# ffprobe could not read the file — fall back to the extractor's info entirely.
return info_v, info_a, info_w, info_h
pv, pa, pw, ph, _dur = _facts_from_probe(data)
# Probe ran → its codecs win outright (a None means "no such track", not "unknown"); info only
# gap-fills the cosmetic width/height the probe couldn't read.
return pv, pa, pw if pw is not None else info_w, ph if ph is not None else info_h
+16
View File
@@ -122,6 +122,22 @@ def check_enqueue(db: Session, user_id: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, used)
def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
"""Like check_enqueue, but for an upload whose size is KNOWN at ingest: also reject when adding
this file would push the footprint over max_bytes. A download can't pre-check its size (unknown
until the worker finishes) so check_enqueue only compares the current footprint; an upload can,
so it's charged against the cap before it lands. Raises QuotaExceeded(max_jobs|max_bytes)."""
lim = resolve(db, user_id)
if lim.unlimited:
return
n = active_jobs(db, user_id)
if n >= lim.max_jobs:
raise QuotaExceeded("max_jobs", lim.max_jobs, n)
projected = footprint(db, user_id) + add_bytes
if projected > lim.max_bytes:
raise QuotaExceeded("max_bytes", lim.max_bytes, projected)
def usage(db: Session, user_id: int) -> dict:
"""Compact usage snapshot for the UI (footprint + counts vs limits)."""
lim = resolve(db, user_id)
+34 -1
View File
@@ -12,16 +12,48 @@ progress hook), then exits 0. An unexpected crash exits non-zero and the parent
"""
import logging
import os
import shutil
import subprocess
import sys
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.models import DownloadJob
log = logging.getLogger("siftlode.worker.runner")
def _deprioritize() -> None:
"""Lower THIS child's CPU + IO priority so its yt-dlp/ffmpeg work — every descendant inherits the
handicap — can never starve the co-located API, Postgres, or Plex playback. Run once at startup,
before any heavy work. Every step is best-effort: a failed renice must never fail a download."""
nice_inc = settings.download_child_nice
if nice_inc > 0:
try:
os.nice(nice_inc) # relative increment; the child starts at the worker's nice (0)
except OSError as exc:
log.info("could not renice download child: %s", str(exc)[:120])
cls = settings.download_child_ionice_class
if cls > 0 and shutil.which("ionice"): # util-linux; absent on some minimal images
try:
subprocess.run(
["ionice", "-c", str(cls), "-p", str(os.getpid())],
capture_output=True,
timeout=5,
check=False,
)
except (OSError, subprocess.SubprocessError) as exc:
log.info("could not ionice download child: %s", str(exc)[:120])
# Cap OpenMP-parallel ffmpeg filters to the same budget as the encoder threads, so a filter
# graph can't fan out across every core behind ffmpeg's own `-threads` cap.
os.environ.setdefault("OMP_NUM_THREADS", str(editmod.encode_thread_cap()))
def main() -> None:
logging.basicConfig(level=logging.INFO)
_deprioritize()
if len(sys.argv) != 2:
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
raise SystemExit(2)
@@ -35,6 +67,7 @@ def main() -> None:
job_kind = job.job_kind
spec = dict(job.profile_snapshot or {})
source_kind, source_ref = job.source_kind, job.source_ref
user_id = job.user_id
if asset_id is None:
return
@@ -45,7 +78,7 @@ def main() -> None:
if job_kind == "edit":
_process_edit(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec)
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
if __name__ == "__main__":
+15 -2
View File
@@ -15,8 +15,9 @@ from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import adapters
from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import formats, quota, storage
from app.downloads import formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobStatus
from app.models import Channel, DownloadJob, MediaAsset, Video
@@ -155,7 +156,12 @@ def enqueue(
Raises quota.QuotaExceeded if the user is at their job-count or footprint cap."""
quota.check_enqueue(db, user_id)
spec = formats.normalize(spec)
sig = formats.format_sig(spec)
# If this user has a stored cookie for the download's host, salt the cache signature with their
# id so the cookie-authed result is a PRIVATE asset — never dedup-shared with, or cache-served
# to, another user (the load-bearing privacy fix; the worker attaches the cookie to exactly this
# salted identity). A normal download (no matching cookie) gets salt=None and shares as before.
salt = credentials.salt_for_enqueue(db, user_id, source_url(source_kind, source_ref))
sig = formats.format_sig(spec, salt=salt)
asset = get_or_create_asset(db, source_kind, source_ref, sig)
# Show a real title + thumbnail the moment it's queued (feed downloads are catalog videos).
populate_from_catalog(db, asset)
@@ -210,6 +216,10 @@ def enqueue_edit(
src_asset = db.get(MediaAsset, source_job.asset_id) if source_job.asset_id else None
if src_asset is None or src_asset.status != "ready" or not src_asset.rel_path:
raise EditError("source_not_ready")
# A generic uploaded file (media_kind='file') has no timeline to trim/crop — the editor path is
# media-only. Reject here so an upload's file never reaches the ffmpeg edit worker.
if src_asset.media_kind == mediakind.FILE:
raise EditError("not_media")
spec = editmod.normalize_edit_spec(edit_spec)
if not spec:
@@ -226,6 +236,9 @@ def enqueue_edit(
asset.thumbnail_url = src_asset.thumbnail_url
asset.upload_date = src_asset.upload_date
asset.duration_s = editmod.clip_duration(spec, src_asset.duration_s)
# A clip inherits its source's kind (a video edit stays video, an audio trim stays audio);
# the ffmpeg edit worker doesn't re-probe codecs, so set it here where the source is known.
asset.media_kind = src_asset.media_kind or mediakind.VIDEO
job = DownloadJob(
user_id=user_id,
+77
View File
@@ -19,6 +19,8 @@ from datetime import date
from pathlib import Path
from xml.sax.saxutils import escape
from app.downloads import mediakind
# Reserve room for the "_id.ext" suffix + parent dirs within the 255-char component limit.
_MAX_TITLE = 120
# User-facing (Content-Disposition) names can be longer than an on-disk component.
@@ -147,6 +149,81 @@ def edit_rel_path(user_id: int, asset_id: int, ext: str) -> str:
return f".edits/{user_id}/clip_{asset_id}.{ext}"
def _clean_upload_ext(filename: str) -> str:
"""The upload's extension reduced to a shell-safe, lowercased, bounded token (``""`` when the
upload has no extension). The SINGLE definition feeding both the on-disk suffix and the stored
`container` — they must agree (the served Content-Disposition suffix has to match the container),
so the sanitize/truncation rule lives in one place, not two drifting copies."""
return _UNSAFE.sub("", Path(filename).suffix.lstrip(".")).lower()[:16]
def upload_rel_path(user_id: int, asset_id: int, filename: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a locally-uploaded file.
Kept in a per-user `.uploads/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (an upload is arbitrary user content, not a canonical episode; no .nfo/poster
episode sidecars). The name is system-decided and bound to the asset id (`_a{id}` keeps two
same-named uploads apart); the original filename is display metadata only. The extension is
preserved (shell-safe, bounded) so the served file + its Content-Disposition carry a correct
suffix; an extensionless upload stays extensionless."""
stem = sanitize(Path(filename).stem, 80)
ext = _clean_upload_ext(filename)
suffix = f".{ext}" if ext else ""
return f".uploads/{user_id}/{stem}_a{asset_id}{suffix}"
def upload_ext(filename: str) -> str | None:
"""The stored `container` for an uploaded file: its shell-safe lowercased extension, or None when
the upload has no extension (a bare name). Shares `_clean_upload_ext` with `upload_rel_path` so the
stored container and the on-disk suffix can never diverge."""
return _clean_upload_ext(filename) or None
# The Content-Type used to serve an inline-playable media asset, keyed by its stored `container`.
# DELIBERATELY a strict allow-list of known audio/video containers: the served type is chosen from
# the asset's ffprobe-verified media_kind + this table, NEVER guessed from the raw (user-controlled)
# upload extension. Without this, an uploaded media/HTML polyglot named "evil.html" — a real MP3/MP4
# that ffprobe classifies as media, so it dodges the generic-file force-download — would be served
# with a Content-Type Starlette guesses from ".html" (text/html) and execute as stored XSS on the
# serving origin. A media asset whose container isn't in this table falls back to a neutral
# force-download attachment (see `served_media_type`), so nothing script-executable is ever inline.
_MEDIA_MIME: dict[str, str] = {
# video containers
"mp4": "video/mp4",
"m4v": "video/mp4",
"mov": "video/quicktime",
"webm": "video/webm",
"mkv": "video/x-matroska",
"ogv": "video/ogg",
"avi": "video/x-msvideo",
# audio containers
"m4a": "audio/mp4",
"mp3": "audio/mpeg",
"aac": "audio/aac",
"ogg": "audio/ogg",
"oga": "audio/ogg",
"opus": "audio/ogg",
"weba": "audio/webm",
"wav": "audio/wav",
"flac": "audio/flac",
}
def served_media_type(media_kind: str | None, container: str | None) -> str | None:
"""The Content-Type a library file MUST be served with — a real audio/video MIME for a recognised
media container, else ``None`` meaning "serve as a neutral force-download attachment".
Returns None for a generic ``file`` kind AND for any media whose container we don't positively
recognise as safe. Callers serve None as ``application/octet-stream`` + ``attachment`` so the
served Content-Type is NEVER derived from the (attacker-controlled) file extension — the guard
that stops uploaded HTML/SVG or a media/HTML polyglot from executing inline on the serving
origin. A ``None`` media_kind (a legacy asset before the discriminator was backfilled) is treated
as media so an old download keeps its inline player."""
if media_kind == mediakind.FILE:
return None
return _MEDIA_MIME.get((container or "").lower())
def download_filename(display_name: str, container: str | None, path: Path) -> str:
"""The Content-Disposition filename for a served download: the cleaned display name carrying a single
correct extension — the asset container, else the file's own suffix — without doubling it when the
+122
View File
@@ -0,0 +1,122 @@
"""Local file ingest for the download center (E7 S5) — the non-yt-dlp path into the media library.
A user-uploaded file joins the SAME library as a downloaded one, so it inherits the whole function
set already built (metadata chips, share links, and — for media — in-app play/edit). Unlike a
download it bypasses yt-dlp AND the worker entirely: the file is already on the host, so ingest
probes it, places it on disk, and creates an already-`ready` MediaAsset + `done` DownloadJob in one
step — the same "the file exists, mark it done now" shortcut `service.enqueue` takes on a cache hit.
Kinds (the invariant lives in `mediakind`): a probe with a real video stream -> video, audio-only
-> audio, neither -> a generic 'file' (pdf/zip/docx/…) that is download-only and never rendered
inline (see the force-download serving in the file/watch routes). This module is the ingest seam a
future sub-function (e.g. convert) plugs a new adapter into, mirroring how it sidesteps the worker.
An upload does NOT get an `expires_at`: it's user-provided and may be irreplaceable (there's no
source to re-download), so it doesn't age out on the retention clock the way a re-fetchable download
does — it lives until the user deletes it.
"""
import secrets
from datetime import datetime, timezone
from pathlib import Path
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import edit as editmod
from app.downloads import mediakind, probe, quota, service, storage
from app.models import DownloadJob, MediaAsset
def _title_from_filename(filename: str) -> str:
"""A human display title from the original upload name: the stem with separators tidied to
spaces (kept as-is otherwise — this is display metadata, not the shell-safe on-disk name)."""
stem = Path(filename).stem.replace("_", " ").replace(".", " ").strip()
return (stem or "Upload")[:255]
def ingest_upload(
db: Session,
user_id: int,
*,
filename: str,
src_path: Path,
size: int,
display_name: str | None = None,
) -> DownloadJob:
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return the `done`
job. `src_path` is MOVED into place on success (the caller stages it to a temp file first).
Raises quota.QuotaExceeded (holdings/footprint cap) — checked BEFORE any disk write, so a
rejected upload leaves nothing behind and the caller just cleans up its temp file."""
quota.check_upload(db, user_id, size) # the file's size is known → charge it up front
vcodec, acodec, width, height, duration = probe.probe_media(src_path)
kind = mediakind.derive(vcodec, acodec)
# An upload never dedups (each file is its own asset): a random ref makes the cache key unique,
# so `format_sig` is a constant marker rather than a format hash.
ref = f"{user_id}:{secrets.token_hex(8)}"[:512]
asset = MediaAsset(
source_kind="upload", source_ref=ref, format_sig="upload", status="pending", ref_count=0
)
db.add(asset)
db.flush() # need asset.id for the on-disk path
rel = storage.upload_rel_path(user_id, asset.id, filename)
storage.place_file(src_path, settings.download_root, rel)
# From here the file is on disk under `.uploads/`. If the transaction fails before it commits, the
# asset/job rows roll back but the placed file would be ORPHANED — nothing references it, the
# per-user delete path keys off asset.rel_path (which is gone), and the GC orphan sweep skips the
# dot-prefixed `.uploads/` tree. So roll back the on-disk side too on any failure.
try:
final = storage.abs_path(settings.download_root, rel)
actual_size = final.stat().st_size if final.exists() else size
asset.status = "ready"
asset.rel_path = rel
asset.size_bytes = actual_size
asset.container = storage.upload_ext(filename)
asset.vcodec = vcodec
asset.acodec = acodec
asset.width = width
asset.height = height
asset.duration_s = duration
asset.media_kind = kind
asset.title = (display_name or "").strip() or _title_from_filename(filename)
asset.last_access_at = datetime.now(timezone.utc)
# No expires_at — see the module docstring (an upload is irreplaceable, so it doesn't age out).
if kind == mediakind.VIDEO:
# A frame poster gives the library row a thumbnail + the share link an og:image. Audio has
# no frame and a generic file no player, so neither gets one. Best-effort (ffmpeg decodes
# server side even for a codec the browser can't play).
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind="upload",
source_kind="upload",
source_ref=ref,
profile_snapshot={},
display_name=(display_name or "").strip() or None,
status="done",
progress=100,
queue_pos=service.next_queue_pos(db),
)
db.add(job)
service.incr_ref_count(db, asset.id)
db.commit()
except Exception:
db.rollback()
# The rows are gone; remove the now-unreferenced file so no bytes leak. Clean the poster by its
# DETERMINISTIC path (rel with a .jpg suffix, the same path ensure_poster writes to) rather than
# ensure_poster's return value: it writes the frame directly to that path and returns None on an
# ffmpeg failure, so a partial poster can be on disk while the return is None — gating on the
# return would skip that orphan. unlink is missing_ok, so the poster path is a harmless no-op
# when none was generated (audio/file, or a video whose poster never started).
for leaked in (rel, str(Path(rel).with_suffix(".jpg"))):
storage.abs_path(settings.download_root, leaked).unlink(missing_ok=True)
raise
db.refresh(job)
return job
+128 -14
View File
@@ -3,36 +3,103 @@
Gmail SMTP + App Password by default. Deliberately fail-soft: if SMTP isn't configured
or a send errors, we log and return False so the caller can fall back to in-app
notifications — email is never load-bearing for the app's core flow.
Fail-soft must not mean invisible, though: every drop also leaves an in-app notice in the admins'
inbox (see `_alert_admins`). An instance whose SMTP was never filled in silently swallows
verification links, password resets and access-request notices, and the operator's first clue used
to be a person telling them nothing arrived.
"""
import html
import logging
import re
import smtplib
import ssl
from email.message import EmailMessage
from email.utils import formatdate
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.models import User
from app.notifications import create_notification
from app.ratelimit import RateLimiter
log = logging.getLogger("siftlode.email")
def _smtp() -> dict:
"""Effective SMTP config (admin DB override over env defaults). Opens its own short
session because email is often sent from a BackgroundTask, outside a request's db."""
with SessionLocal() as db:
return {
"host": sysconfig.get_str(db, "smtp_host"),
"port": sysconfig.get_int(db, "smtp_port"),
"user": sysconfig.get_str(db, "smtp_user"),
"password": sysconfig.get_str(db, "smtp_password"),
"from": sysconfig.get_str(db, "smtp_from"),
}
# One admin notice an hour per reason. A relay that is down fails every send, and the inbox must
# stay readable; the log keeps the full record either way.
_alert_throttle = RateLimiter(max_events=1, window_seconds=3600)
def email_enabled() -> bool:
c = _smtp()
def _alert_admins(reason: str, subject: str, recipients: list[str]) -> None:
"""Record an in-app notice that a message did NOT go out. Best effort in every direction: it
opens its own session (callers are often background tasks), swallows its own errors, and never
reports failure to the caller — this is the fallback path, it must not become a new way to
break a send."""
if not _alert_throttle.allow(reason):
return
try:
with SessionLocal() as db:
admins = (
db.execute(
select(User.id).where(
User.role == "admin",
User.is_active.is_(True),
User.is_suspended.is_(False),
)
)
.scalars()
.all()
)
if not admins:
# Nothing was recorded, so don't let this cost the next hour of alerts on the very
# path whose job is to stop drops going unseen. Same on the failure branch below.
_alert_throttle.reset()
return
for uid in admins:
create_notification(
db,
user_id=uid,
type="email_undelivered",
# English fallback; the inbox renders a translated version from `data`.
title="Email could not be sent",
body=subject,
data={"reason": reason, "subject": subject, "recipients": recipients},
commit=False,
)
db.commit()
except Exception:
_alert_throttle.reset()
log.exception("Could not record the undelivered-email notice (%r)", subject)
def _read_smtp(db: Session) -> dict:
return {
"host": sysconfig.get_str(db, "smtp_host"),
"port": sysconfig.get_int(db, "smtp_port"),
"user": sysconfig.get_str(db, "smtp_user"),
"password": sysconfig.get_str(db, "smtp_password"),
"from": sysconfig.get_str(db, "smtp_from"),
}
def _smtp(db: Session | None = None) -> dict:
"""Effective SMTP config (admin DB override over env defaults). Opens its own short session when
the caller has none, because email is usually sent from a BackgroundTask outside a request's db.
A caller that IS inside a request should pass its session: reading the password decrypts a Fernet
value, and /api/me polls this for every admin tab."""
if db is not None:
return _read_smtp(db)
with SessionLocal() as own:
return _read_smtp(own)
def email_enabled(db: Session | None = None) -> bool:
c = _smtp(db)
return bool(c["host"] and c["user"] and c["password"])
@@ -40,6 +107,48 @@ def _admin_contact() -> str | None:
return next(iter(sorted(settings.admin_email_set)), None)
# A bare URL in a text/plain mail is only clickable if the CLIENT decides to linkify it. Gmail,
# Outlook and Apple Mail do; plenty of lighter web clients render the body verbatim and leave the
# link as dead text — which, for a verification or password-reset mail, is the whole message. So
# every mail also carries an HTML alternative built from the SAME plain body (below), rather than
# per-template HTML that would have to be kept in sync with it.
#
# `[^\s<]` stops at whitespace and at the `<` of any following markup; the trailing-punctuation trim
# keeps a sentence-final period out of the href. `;` and `:` are deliberately NOT trimmed — escaping
# turns `&` into `&amp;`, so a query string can legitimately end in a semicolon.
_URL_RE = re.compile(r"https?://[^\s<]+")
def _html_body(text: str) -> str:
"""Render a plain-text mail body as minimal HTML: escaped, blank-line-separated paragraphs, with
real links. Escaping happens FIRST and the linkifier then runs over the escaped text — bodies
interpolate untrusted values (a requester's email address, the operator contact), so no caller
can inject markup, and `&` inside a URL is already `&amp;` by the time it reaches the href, which
is what an HTML attribute wants anyway."""
def _anchor(m: re.Match[str]) -> str:
url = m.group(0)
trail = ""
while url and url[-1] in ".,!?)":
url, trail = url[:-1], url[-1] + trail
if not url:
return trail
return f'<a href="{url}">{url}</a>{trail}'
paragraphs = [p for p in html.escape(text).split("\n\n") if p.strip()]
rendered = "".join(
f"<p>{_URL_RE.sub(_anchor, p).replace(chr(10), '<br>')}</p>" for p in paragraphs
)
# Inline styles only — many clients drop <style> blocks. No images, no external CSS, no tracking:
# the HTML part exists to make the link clickable, not to become a template to maintain.
return (
'<html><body style="margin:0;padding:16px;font-family:system-ui,-apple-system,'
"'Segoe UI',sans-serif;font-size:15px;line-height:1.6\">"
f'<div style="max-width:560px">{rendered}</div>'
"</body></html>"
)
def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -> bool:
recipients = [e for e in to if e]
if not recipients:
@@ -47,6 +156,7 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
c = _smtp()
if not (c["host"] and c["user"] and c["password"]):
log.info("SMTP not configured; skipping email %r to %s", subject, recipients)
_alert_admins("not_configured", subject, recipients)
return False
msg = EmailMessage()
msg["Subject"] = subject
@@ -57,7 +167,10 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
msg["Date"] = formatdate(localtime=True)
if reply_to:
msg["Reply-To"] = reply_to
# text/plain stays the primary part (it is the source of truth, and a text-only client still
# gets the full message); the HTML alternative below is what makes the link clickable everywhere.
msg.set_content(body)
msg.add_alternative(_html_body(body), subtype="html")
try:
with smtplib.SMTP(c["host"], c["port"], timeout=20) as s:
s.starttls(context=ssl.create_default_context())
@@ -67,6 +180,7 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
return True
except Exception:
log.exception("SMTP send failed (%r to %s)", subject, recipients)
_alert_admins("send_failed", subject, recipients)
return False
+48 -4
View File
@@ -100,6 +100,11 @@ def serialize(row) -> dict:
}
def _tag_visible(user: User):
"""A ChannelTag row this user may see: a shared system tag (user_id NULL) or their own."""
return or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
def tags_by_category(db: Session, tag_ids: list[int]) -> dict[str, list[int]]:
"""Group tag ids by their category (language/topic/other). The single source for the category split
shared by the feed's tag WHERE (filtered_query) and the facet/count summary (feed_summary), so the
@@ -118,6 +123,7 @@ def filtered_query(
*,
tags: list[int],
tag_modes: dict[str, str],
untagged: bool = False,
channel_ids: list[str],
q: str | None,
min_duration: int | None,
@@ -294,7 +300,7 @@ def filtered_query(
# selections but still honours the other categories' filters).
if exclude_tag_category:
by_category.pop(exclude_tag_category, None)
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
visible = _tag_visible(user)
for category, ids in by_category.items():
if tag_modes.get(category, "or") == "and" and len(ids) > 1:
sub = (
@@ -307,6 +313,18 @@ def filtered_query(
sub = select(ChannelTag.channel_id).where(ChannelTag.tag_id.in_(ids), visible)
query = query.where(Video.channel_id.in_(sub))
# "No tag" filter (item B): keep only channels the USER has not personally tagged in the channel
# manager — i.e. channels with no ChannelTag row owned by this user (user_id == me, source "user").
# System/auto tags (user_id NULL) don't count: this is "channels I haven't tagged yet". It's a plain
# channel-level narrowing (like channel_ids/date), NOT a tag-category selection, so feed_summary
# leaves it applied in the base pass and needs no mirror in passes().
if untagged:
query = query.where(
Video.channel_id.notin_(
select(ChannelTag.channel_id).where(ChannelTag.user_id == user.id)
)
)
# Content type: Normal / Shorts / Live·Upcoming as a union of enabled types.
# Explicit Watched/Saved/Hidden views show every type so nothing goes missing.
explicit_view = show in ("watched", "hidden")
@@ -355,8 +373,9 @@ def feed_summary(db: Session, user: User, **params) -> dict:
within, a category in "and" mode requires ALL its selected tags)."""
tag_modes = params.get("tag_modes", {})
# ONE pass: per-channel video counts for the base filter, tags stripped so every candidate channel
# is present to facet over.
# ONE pass: per-channel video counts for the base filter, the positive tag filter stripped so every
# candidate channel is present to facet over. The "No tag" (untagged) filter is a plain channel-level
# narrowing, so it STAYS applied here (not stripped) and the facets naturally drill down within it.
base, _rank = filtered_query(db, user, **{**params, "tags": []})
ch_col = base.with_only_columns(Video.channel_id).subquery()
channel_counts: dict[str, int] = dict(
@@ -386,7 +405,8 @@ def feed_summary(db: Session, user: User, **params) -> dict:
def passes(ch_tags: set[int], exclude_cat: str | None) -> bool:
# The Python mirror of filtered_query's SQL tag block (AND across categories, OR within,
# topic-AND requires all selected) — keep in lockstep; see the ⚠️ note there.
# topic-AND requires all selected) — keep in lockstep; see the ⚠️ note there. (The "No tag"
# filter is applied in the base query above, so it needs no mirror here.)
for cat, sel in selected_by_cat.items():
if cat == exclude_cat:
continue
@@ -417,6 +437,30 @@ def feed_summary(db: Session, user: User, **params) -> dict:
return {"count": count, "facets": facets}
def feed_years(db: Session, user: User, **params) -> list[dict]:
"""Per-year video counts for the current filter context (item 8 / E8·S2): powers the year control's
quick chips + the "more…" per-year-counts dialog. Every OTHER filter (tags, channels, content type,
search, watch state, scope) still applies, but the publish-date narrowing itself is DROPPED —
published_after/before and max_age_days — so the year list is the full set of years available and
switching years stays a live drill-down (the same principle as the tag facets dropping their own
category). One grouped query, newest year first."""
base, _rank = filtered_query(
db,
user,
**{**params, "published_after": None, "published_before": None, "max_age_days": None},
)
pub = base.with_only_columns(Video.published_at.label("pub")).subquery()
# Bucket by UTC year to stay in lockstep with the year filter's boundaries (published_after/before
# are pinned to UTC in filtered_query). published_at is timestamptz, so a bare extract('year', …)
# would use the DB session timezone and, on a non-UTC session, bucket a video near a year boundary
# into a different year than the filter includes — the count and the drill-down feed would disagree.
year = func.extract("year", func.timezone("UTC", pub.c.pub))
rows = db.execute(
select(year.label("y"), func.count()).group_by(year).order_by(year.desc())
).all()
return [{"year": int(y), "count": n} for y, n in rows if y is not None]
# --- Keyset (cursor) pagination ---------------------------------------------
#
# Each sort is described as an ordered list of key columns (expression + direction
+61
View File
@@ -47,6 +47,12 @@ class User(Base, TimestampMixin):
email: Mapped[str] = mapped_column(String(320), unique=True, index=True)
# argon2id hash for email+password login; NULL for Google-only / demo accounts.
password_hash: Mapped[str | None] = mapped_column(String(255))
# When SEC-10 dropped a preset password (see auth.revoke_preset_password_if_unverified). Without
# this trace a revoked account was indistinguishable from a Google-only one, so it failed the
# password-reset eligibility check and could never be recovered — it could neither sign in nor
# reset. Set = "had a password, lost it to the guard" -> reset-eligible. Cleared whenever a
# password is set again (reset-confirm / set-password).
password_revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Email ownership proven (Google sign-in is implicitly verified; password registration
# verifies via a link). Login requires this AND is_active.
email_verified: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
@@ -551,6 +557,10 @@ class Playlist(Base, TimestampMixin, UpdatedAtMixin):
# SHA-256 of name + ordered video ids as last synced from / pushed to YouTube; the
# baseline `dirty` is derived against. NULL = no baseline yet (treated as dirty).
synced_fingerprint: Mapped[str | None] = mapped_column(String)
# JSON list of the ordered video ids as of the last sync — the 3-way-merge baseline the
# two-way YouTube sync classifies each item against (local-add vs YT-add vs real delete).
# NULL = no baseline yet; the first merge then does a lossless union (never deletes).
synced_video_ids: Mapped[str | None] = mapped_column(Text)
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
items: Mapped[list["PlaylistItem"]] = relationship(
@@ -704,6 +714,12 @@ class MediaAsset(Base, TimestampMixin):
container: Mapped[str | None] = mapped_column(String(16))
vcodec: Mapped[str | None] = mapped_column(String(32))
acodec: Mapped[str | None] = mapped_column(String(32))
# "video" | "audio" | "file" — the media-kind discriminator that drives which library actions
# apply (play/edit for media; download-only for a generic uploaded file). Null while an asset is
# still pending (kind is known once the file is probed at finalize). It is the single stored home
# of the invariant `downloads.mediakind.derive` used to compute — never re-derived in prose (an
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Set once the GC has warned holders of the upcoming expiry, so it doesn't warn every run.
gc_notified: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
@@ -867,6 +883,35 @@ class DownloadLink(Base, TimestampMixin):
view_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class DownloadCredential(Base, TimestampMixin, UpdatedAtMixin):
"""A user's stored login cookies for ONE auth-walled download host (Facebook, Instagram, …).
Per-user, per-host: each user pastes their OWN browser cookies.txt (Netscape format) for a host
where they're logged in (e.g. a closed Facebook group they belong to), and a download of a
yt-dlp-supported, auth-walled link on that host is retried with the ACTING user's cookie. The
blob is Fernet-encrypted at rest (`cookies_enc`) and is NEVER returned by any API/log/audit —
the routes are write-only (host + updated_at only). See `downloads.credentials`.
Two load-bearing security controls live around this row, NOT on it:
* **cache privacy** — a cookie-authed download SALTS its MediaAsset `format_sig` with the
user id, so the authed asset is private to that user and can never be cache-served to
another user (`downloads.credentials.cookie_salt`).
* **host scope (SEC-12)** — a cookie is attached ONLY when the download URL's netloc matches
this row's `host_key` (`downloads.credentials.host_key_for_url`), so an arbitrary `?url=`
can't exfiltrate the user's session to a look-alike host.
`host_key` is a short canonical key from the supported-host registry (e.g. "facebook"), not the
raw netloc — one credential can cover several domains of the same site (facebook.com/fb.watch)."""
__tablename__ = "download_credentials"
__table_args__ = (UniqueConstraint("user_id", "host_key", name="uq_download_credential"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
host_key: Mapped[str] = mapped_column(String(64))
cookies_enc: Mapped[str] = mapped_column(Text) # Fernet ciphertext — never returned/logged
class MessageKey(Base, TimestampMixin):
"""A user's end-to-end-encryption key material for direct messaging (phase 2).
@@ -1088,6 +1133,22 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
# Browser-playability class from ffprobe: "direct" | "remux" | "transcode" (None = unprobed).
playable: Mapped[str | None] = mapped_column(String(12), index=True)
probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Transcode HW-capability facet (E7 playback-class filter). NULL for non-transcode items; for a
# `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
# AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
# breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
# set codec-only at sync (bit depth unknown from the cheap listing), then refined by the bit-depth
# backfill (`_probe_media_batch`) which downgrades a 10-bit/HDR HEVC to False. The one filterable
# home of the invariant `_want_hw`'s source gate computes at play time — kept in sync via the shared
# classifier so the catalog facet and the runtime encoder choice never disagree by codec.
transcode_hw: Mapped[bool | None] = mapped_column(Boolean, index=True)
# Video bit depth (8/10/12…) from the Plex media Stream, filled by the bit-depth backfill. NULL =
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw`.
video_bit_depth: Mapped[int | None] = mapped_column(Integer)
# Watermark for the bit-depth backfill: when this item's media Stream was last probed for
# bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are
# probed (the rest are SW-only by codec and need no bit-depth); see `_probe_media_batch`.
media_probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Intro/credit markers: [{"type": "intro"|"credits", "start_s": int, "end_s": int}, ...]
markers: Mapped[list | None] = mapped_column(JSON)
added_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
+2
View File
@@ -33,6 +33,7 @@ def movie_card(it: PlexItem, st: PlexState | None) -> dict:
"thumb": f"/api/plex/image/{it.rating_key}",
"art": _art_url(it.rating_key, it.art_key),
"playable": it.playable,
"transcode_hw": it.transcode_hw, # HW/SW transcode split → the 4-way corner-dot badge
"status": st.status if st else "new",
"position_seconds": st.position_seconds if st else 0,
}
@@ -65,6 +66,7 @@ def episode_card(e: PlexItem, st: PlexState | None) -> dict:
"duration_seconds": e.duration_s,
"thumb": f"/api/plex/image/{e.rating_key}",
"playable": e.playable,
"transcode_hw": e.transcode_hw, # HW/SW transcode split → the 4-way corner-dot badge
"status": st.status if st else "new",
"position_seconds": st.position_seconds if st else 0,
}
+31
View File
@@ -9,6 +9,8 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import and_, case, func, or_, select, type_coerce
from sqlalchemy.dialects.postgresql import JSONB
from app.models import PlexItem
TS_CONFIG = "public.unaccent_simple"
# --- Language-aware column pickers (E5 S6) ----------------------------------
@@ -206,6 +208,33 @@ def apply_meta_filters(q, model, p: dict, lang: str = "en"):
return q.where(*conds) if conds else q
# --- Playback-class filter (E7) — MOVIE-ONLY (a show is an aggregate with no single class) ---------
# The 4 classes the feed filter + facet expose, derived from the stored `playable` + `transcode_hw`.
# ONE definition (this SQL CASE) so the filter WHERE and the facet counts can't drift: `playable_conds`
# filters on it, `facets` groups by it. transcode_hw IS NOT True groups NULL (unprobed) with software.
# Public — it is the shared source of truth imported by routes.plex.browse (facet order + zero-fill).
PLAYBACK_CLASSES = ("direct", "remux", "transcode_hw", "transcode_sw")
def playback_class_expr():
"""SQL CASE mapping a PlexItem (movie) to its playback-class string, or NULL. The HW branch must
precede the generic transcode branch (ordered evaluation)."""
return case(
(PlexItem.playable == "direct", "direct"),
(PlexItem.playable == "remux", "remux"),
(and_(PlexItem.playable == "transcode", PlexItem.transcode_hw.is_(True)), "transcode_hw"),
(PlexItem.playable == "transcode", "transcode_sw"),
else_=None,
)
def playable_conds(sel: list[str]) -> list:
"""A one-element condition list (or empty) selecting the given playback classes — an OR via
`class IN (...)` over the shared CASE. Unknown keys are dropped."""
keep = [c for c in sel if c in PLAYBACK_CLASSES]
return [playback_class_expr().in_(keep)] if keep else []
@dataclass
class LibraryFilters:
"""The shared sidebar filter query-params for /library + /facets, declared once and injected via
@@ -228,6 +257,7 @@ class LibraryFilters:
keywords: str | None = None # TMDB keyword chips (E5 S5), CSV; language-neutral
collections: str | None = None # CSV of collection rating_keys
collection_mode: str = "any" # how multiple collections combine: any (OR, default) | all (AND)
playable: str | None = None # CSV of playback classes: direct|remux|transcode_hw|transcode_sw
def as_dict(self) -> dict:
return {
@@ -247,6 +277,7 @@ class LibraryFilters:
"keywords": self.keywords,
"collections": self.collections,
"collection_mode": self.collection_mode,
"playable": self.playable,
}
+7 -3
View File
@@ -43,8 +43,10 @@ def rollup(cards: list[dict]) -> dict:
def agg_subquery(db: Session, user_id: int, include_resumed: bool = False):
"""Per-show episode counts for a user: total, watched, in-progress. Reused to (a) filter the show
grid by aggregate watch-state, (b) compute each card's badge, and (c) drive the Discover
continue-watching rail. `include_resumed` adds `resumed_at` = max(progress_updated_at) across the
show's episodes, so the rail can order shows by how recently they were last progressed."""
continue-watching rail, and (d) the browse "last watched" sort. `include_resumed` adds `resumed_at`
= max(coalesce(progress_updated_at, watched_at)) across the show's episodes — i.e. how recently ANY
episode was last touched (resumed OR marked watched) — so callers can order shows by recency. A
fully-watched show (no resume progress) still gets a timestamp from its episodes' watched_at."""
ps = aliased(PlexState)
cols = [
PlexItem.show_id.label("show_id"),
@@ -63,7 +65,9 @@ def agg_subquery(db: Session, user_id: int, include_resumed: bool = False):
).label("inprog"),
]
if include_resumed:
cols.append(func.max(ps.progress_updated_at).label("resumed_at"))
cols.append(
func.max(func.coalesce(ps.progress_updated_at, ps.watched_at)).label("resumed_at")
)
return (
select(*cols)
.outerjoin(ps, and_(ps.item_id == PlexItem.id, ps.user_id == user_id))
+443 -44
View File
@@ -3,7 +3,10 @@
Playback is from the LOCAL physical file. Browser-compatible files (playable="direct") are served
raw with HTTP range requests. Everything else that is h264 video (playable="remux") is remuxed on
the fly to HLS with **video stream-copy** (cheap, I/O-bound — no video re-encode) and audio to AAC
when needed. HEVC/VP9 (playable="transcode") needs a full re-encode and is deferred to P3.
when needed. Non-h264 video (playable="transcode": HEVC/VP9/AV1/MPEG-2/XviD…) needs a full re-encode
to H.264: on the prod host that runs on the Intel iGPU via VAAPI (`h264_vaapi`, near-zero CPU), with
software libx264 as the automatic fallback where the render node is absent or the source codec isn't
hardware-decodable. Same HLS session/seek machinery either way — only the video encoder differs.
Seek model (Jellyfin-style): one ffmpeg session per item, started at a given offset via `-ss`
(fast keyframe seek). A seek beyond the generated region restarts the session at the new offset,
@@ -16,6 +19,7 @@ CPU-only prod host; only P3's full transcode is CPU-heavy.
"""
import logging
import os
import re
import shutil
import subprocess
@@ -27,8 +31,10 @@ from sqlalchemy.orm import Session as DbSession
from app import sysconfig
from app.config import settings
from app.downloads.edit import encode_thread_cap
from app.models import PlexItem
from app.plex import paths
from app.plex.sync import HW_DECODABLE_CODECS
log = logging.getLogger("siftlode.plex")
@@ -38,6 +44,17 @@ _HLS_ROOT = (
else Path(settings.download_root) / ".plex-hls"
)
_SEG_SECONDS = 6
# Cold-start warm-up. A freshly-spawned ffmpeg has produced only the first segment (or two) by the
# time we could report the session ready; hls.js parses the EVENT playlist, drains that tiny buffer,
# then STALLS a few seconds into playback waiting for the next segment the still-warming ffmpeg
# hasn't written yet. Very visible on auto-advance to the next episode (start_s==0, which previously
# returned with ZERO buffer). So before reporting ready we wait for the first few segments to exist:
# the client then starts with ~_WARMUP_SEGMENTS×_SEG_SECONDS of buffer and, since stream-copy runs
# faster than realtime, ffmpeg stays ahead from there and never stalls again. Best-effort and bounded
# — we stop at the first segment that isn't ready within the per-segment budget (a slow miss is no
# worse than the old zero-buffer behaviour).
_WARMUP_SEGMENTS = 3
_WARMUP_SEG_TIMEOUT_S = 8.0
_SESSION_IDLE_S = 600 # reap a session with no access for this long
# A session directory is named `{rating_key}_{int(start_s)}_{tag}_{aoff:+.2f}` (see start_session);
# before multi-audio it was just `{rating_key}_{int(start_s)}`. `sweep_orphans` deletes ONLY names
@@ -72,11 +89,22 @@ _sessions: dict[str, "HlsSession"] = {}
class HlsSession:
def __init__(
self, key: str, directory: Path, proc: subprocess.Popen, start_s: float, entry: str
self,
key: str,
directory: Path,
proc: subprocess.Popen,
start_s: float,
entry: str,
transcode: bool = False,
hw: bool = False,
tonemap: bool = False,
):
self.key = key
self.dir = directory
self.proc = proc
self.transcode = transcode # video re-encoded (non-h264 source), not stream-copied
self.hw = hw # transcode ran on the VAAPI hardware encoder (else libx264 software)
self.tonemap = tonemap # HW HDR/10-bit path: OpenCL tonemap between VAAPI decode and encode
self.start_s = start_s # the REQUESTED seek offset
# The REAL media start: with `-ss X -c:v copy` ffmpeg can only cut at the nearest earlier
# keyframe K<=X, and hls.js zero-bases the timeline to K (so video.currentTime=0 is K, not X).
@@ -116,6 +144,269 @@ _HLS_TAIL = [
]
# --- Transcode (playable="transcode") video re-encode -----------------------------------------
# The remux path stream-copies the video (`-c:v copy`); a non-h264 source can't be copied and must be
# re-encoded to H.264. The encoder + tonemap are chosen per-session by `_select_encode`:
# * VAAPI hardware (`h264_vaapi`) — the prod default once the iGPU is passed in. A full GPU pipeline
# (decode + scale + encode all on-device) offloads the CPU almost entirely, so several concurrent
# 1080p transcodes are fine and the API stays responsive.
# * libx264 software — the automatic fallback where `/dev/dri` is absent (dev box, or prod before the
# iGPU enablement) or the source codec isn't hardware-decodable (AV1/XviD on Gen 9.5). CPU-bound, so
# it reserves a core via `-threads` (see encode_thread_cap).
# Everything else — the `-copyts`/`_HLS_TAIL`/segment/seek-restart model and the AAC audio path — is
# unchanged, so hls.js, subtitles, multi-audio and seeking keep working exactly as for remux.
_VAAPI_DEVICE = "/dev/dri/renderD128"
# Where an OpenCL ICD registers its driver. The Intel NEO runtime (intel-opencl-icd) drops an .icd file
# here; its presence is the precondition for the HW HDR→SDR tonemap graph (see _opencl_present).
_OPENCL_ICD_DIR = Path("/etc/OpenCL/vendors")
# Codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for the
# full-GPU VAAPI pipeline. A source outside it (AV1, MPEG-4 ASP/XviD…) falls back to software even with
# the render node present, because the GPU can't decode it to feed its own encoder. THE single source is
# plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't disagree by
# codec; aliased to the old name to keep the call sites below unchanged.
_HW_DECODABLE = HW_DECODABLE_CODECS
# Color-transfer values that mark a true-HDR source (PQ / HLG). A HW-decodable HDR source can't take the
# plain scale_vaapi=format=nv12 path (Gen 9.5 VPP refuses it) — it needs the OpenCL tonemap graph
# (_venc_opts tonemap branch) when the OpenCL runtime is present, else software libx264. A 10-bit source
# WITHOUT one of these transfers is SDR, not HDR, and must not be tone-mapped. See _probe_video_kind.
_HDR_TRANSFERS = {"smpte2084", "arib-std-b67"}
# The three source classes that drive the transcode-pipeline choice (see _probe_video_kind).
_VK_HWSAFE = "hwsafe" # 8-bit SDR — the plain scale_vaapi=format=nv12 GPU path
_VK_HDR = "hdr" # PQ/HLG HDR — needs the OpenCL tonemap graph (or software if no runtime)
_VK_SDR10 = "sdr10" # 10-bit *SDR* — a plain 10→8-bit downconvert, NEVER the HDR tone curve
def _probe_video_kind(src: Path) -> str:
"""Classify the source for the transcode pipeline — one of `_VK_HWSAFE` / `_VK_HDR` / `_VK_SDR10`:
* `_VK_HWSAFE` — 8-bit SDR: safe for the plain VAAPI path (`scale_vaapi=format=nv12` + h264_vaapi).
* `_VK_HDR` — PQ/HLG HDR (any depth): the plain scale path breaks on Gen 9.5 VPP, so it needs the
OpenCL tonemap graph (`tonemap_opencl=hable`) when the runtime is present, else software libx264.
* `_VK_SDR10` — 10-bit but *SDR* (bt709 etc.): NOT HDR. Applying the HDR hable tone curve here would
visibly alter an SDR picture (crushed highlights/midtones) where only a 10→8-bit downconvert was
needed, so it must NOT take the tonemap graph — it goes to software libx264, which CPU-decodes and
downconverts to 8-bit yuv420p exactly. (Kept off the plain scale_vaapi 10-bit path too: that path
is unverified on Gen 9.5 VPP, and software is the known-correct downconvert.)
Ground truth from ffprobe (bit depth via pix_fmt / bits_per_raw_sample, HDR via color_transfer); the
Plex mirror doesn't store these. Returns `_VK_HWSAFE` on any probe failure: the runtime HW→SW fallback
in start_session backstops a HW session that still produces nothing, so a probe miss only costs the
rare wasted first attempt, never a stranded item."""
try:
out = subprocess.run(
[
"ffprobe",
"-v",
"error",
"-select_streams",
"v:0",
"-show_entries",
"stream=pix_fmt,bits_per_raw_sample,color_transfer",
"-of",
"default=noprint_wrappers=1",
str(src),
],
capture_output=True,
text=True,
timeout=10,
)
except (subprocess.SubprocessError, OSError):
return _VK_HWSAFE
pix_fmt = bits = transfer = ""
for line in out.stdout.splitlines():
k, _, v = line.strip().partition("=")
v = v.strip().lower()
if k == "pix_fmt":
pix_fmt = v
elif k == "bits_per_raw_sample":
bits = v
elif k == "color_transfer":
transfer = v
if transfer in _HDR_TRANSFERS:
return _VK_HDR # PQ/HLG — needs a real HDR→SDR tonemap, not a plain downconvert
# >8-bit but not HDR = 10-bit SDR. bits_per_raw_sample OR the pix_fmt bit-depth suffix
# (yuv420p10le, p010le, yuv444p12le…) signals >8-bit even when the other is absent.
if (bits.isdigit() and int(bits) > 8) or re.search(r"(?:10|12|16)(?:le|be)$", pix_fmt):
return _VK_SDR10
return _VK_HWSAFE
def _opencl_present() -> bool:
"""Whether an OpenCL runtime (Intel NEO / intel-opencl-icd) is installed — the precondition for the
HW HDR→SDR tonemap graph. Gen 9.5 VPP can't tonemap natively (tonemap_vaapi fails), so the working
HW path for a 10-bit/HDR source is VAAPI-decode → OpenCL tonemap → VAAPI-encode, which needs an ICD
registered here. Absent on the dev box and until the image bakes NEO in → those sources take
software libx264. Its own function so it's monkeypatchable in tests, mirroring _render_node_present."""
try:
return any(_OPENCL_ICD_DIR.glob("*.icd"))
except OSError:
return False
def _hwaccel_in_opts(hw: bool, tonemap: bool = False) -> list[str]:
"""VAAPI decode options, placed before `-i` so the source is decoded (and kept) on the GPU.
`-hwaccel_output_format vaapi` keeps frames as GPU surfaces all the way to `h264_vaapi` — no
round-trip to system memory. Empty for the software path.
The tonemap variant additionally derives an OpenCL device FROM the VAAPI one (`opencl=ocl@va`) so
tonemap_opencl can map the decoded surfaces zero-copy (see _venc_opts); it needs named devices, so it
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`."""
if not hw:
return []
if tonemap:
return [
"-init_hw_device",
f"vaapi=va:{_VAAPI_DEVICE}",
"-init_hw_device",
"opencl=ocl@va",
"-hwaccel",
"vaapi",
"-hwaccel_output_format",
"vaapi",
"-hwaccel_device",
"va",
]
return ["-hwaccel", "vaapi", "-hwaccel_output_format", "vaapi", "-vaapi_device", _VAAPI_DEVICE]
def _venc_opts(
transcode: bool,
hw: bool,
tonemap: bool = False,
preset: str = "veryfast",
crf: int = 20,
max_height: int = 0,
) -> list[str]:
"""The video-encoder args that replace `-c:v copy` for a transcode session (returns the copy args
when not transcoding). A re-encode must MINT its own keyframes at the segment boundaries — the copy
path leaned on the source's keyframes, but a re-encoded stream has none the HLS muxer can cut on
unless we force them (`-force_key_frames` every `_SEG_SECONDS`), else segments aren't independent and
seeking breaks. `preset`/`crf`/`max_height` (software only) come from admin sysconfig."""
if not transcode:
return ["-c:v", "copy"]
force_kf = ["-force_key_frames", f"expr:gte(t,n_forced*{_SEG_SECONDS})"]
if hw and tonemap:
# HDR/10-bit HW path: Gen 9.5 VPP can't tonemap (scale_vaapi=format=nv12 fails), so the decoded
# VAAPI surfaces are mapped to OpenCL (hwmap=derive_device=opencl), tone-mapped HDR→SDR to 8-bit
# nv12 there (tonemap_opencl, hable curve, bt709 out), then mapped back to VAAPI (reverse=1) for
# h264_vaapi. `-filter_hw_device ocl` tells the filter graph which device to run the OpenCL
# filters on. Full GPU pipeline — ~1 CPU core, several×realtime on the UHD 630 (measured).
return [
"-filter_hw_device",
"ocl",
"-vf",
"hwmap=derive_device=opencl,"
"tonemap_opencl=tonemap=hable:transfer=bt709:matrix=bt709:primaries=bt709:format=nv12,"
"hwmap=derive_device=vaapi:reverse=1",
"-c:v",
"h264_vaapi",
*force_kf,
]
if hw:
# scale_vaapi with format=nv12 downconverts a 10-bit (p010) HEVC source to 8-bit nv12 so
# h264_vaapi (8-bit High) can encode it; for an 8-bit source it's a cheap format pass-through.
# The GPU encode is cheap, so it runs at source resolution (max_height is a software-only knob).
return ["-vf", "scale_vaapi=format=nv12", "-c:v", "h264_vaapi", *force_kf]
# Software fallback: same browser-safe recipe as downloads/edit.py (libx264). yuv420p forces 8-bit
# High (not High10) so any browser decodes it; -threads reserves a core so the CPU-bound encode can't
# starve request handling / playback on the shared box.
args: list[str] = []
if max_height and max_height > 0:
# Downscale-ONLY to the height cap (min(ih,H)), keeping aspect with an even width (-2), so a tall
# source doesn't make the software re-encode ruinously expensive. The comma in min() is escaped
# (\,) so ffmpeg's filter parser reads one filter, not two. Never upscales a smaller source.
args += ["-vf", f"scale=-2:min(ih\\,{max_height})"]
args += [
"-c:v",
"libx264",
"-preset",
preset,
"-crf",
str(crf),
"-pix_fmt",
"yuv420p",
"-threads",
str(encode_thread_cap()),
*force_kf,
]
return args
def _render_node_present() -> bool:
"""Whether the VAAPI render node is available — i.e. `/dev/dri` was passed into the container.
Present on prod after the iGPU enablement; absent on the CPU-only dev box and in the localdev
Docker. Its own function so the selection is monkeypatchable in tests and one place for S1c's knob."""
return Path(_VAAPI_DEVICE).exists()
def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> tuple[bool, bool]:
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap)`:
* `(False, False)` → software libx264 (the fallback: no render node, non-decodable source, or a
10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
* `(True, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR source.
* `(True, True)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG) source, when
the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
A 10-bit *SDR* source is deliberately NOT a `(True, True)` case: it needs a plain 10→8-bit downconvert,
not the HDR tone curve (which would visibly darken/flatten an SDR picture), so it drops to software.
Honours the admin `plex_transcode_hw` policy: "off" → always software; "on" → HW wherever the render
node exists (trusting the admin, skipping the decodability gate — the runtime HW→SW fallback still
backstops a session that produces nothing — but still routing a true-HDR source through the tonemap
graph so it doesn't hit the scale_vaapi wall); "auto" (default) → HW only when the render node is
present AND the iGPU can hardware-decode the source, with tonemap chosen only for a true-HDR (PQ/HLG)
source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
if not transcode:
return (False, False)
mode = (sysconfig.get_str(db, "plex_transcode_hw") or "auto").lower()
if mode == "off" or not _render_node_present():
return (False, False)
if mode == "on":
# Trust the admin: HW wherever the render node exists, skipping the decodability gate (the runtime
# backstop catches a miss). Only a true-HDR (PQ/HLG) source needs the tonemap graph — a 10-bit SDR
# source takes the plain scale path here, NOT the HDR tone curve. Probe only when OpenCL exists
# (short-circuit), so "on" stays probe-free where tonemap can't apply anyway.
tonemap = _opencl_present() and _probe_video_kind(src) == _VK_HDR
return (True, tonemap)
# auto
if (item.codec_video or "").lower() not in _HW_DECODABLE:
return (False, False)
kind = _probe_video_kind(src)
if kind == _VK_HWSAFE:
return (True, False) # 8-bit SDR → full-GPU scale path
if kind == _VK_HDR:
# PQ/HLG: HW OpenCL tonemap when the runtime is present, else software libx264.
return (True, True) if _opencl_present() else (False, False)
# _VK_SDR10 — 10-bit SDR: a plain 10→8-bit downconvert, done exactly on the CPU. Never the HDR tone
# curve (it would alter an SDR picture), and off the unverified scale_vaapi 10-bit path — software.
return (False, False)
def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
"""The HW gate on its own (the `hw` half of `_select_encode`). A readability shim for the tests that
assert only whether a session runs on the GPU (production decides via `_select_encode` directly)."""
return _select_encode(db, item, transcode, src)[0]
# CPU-nice increment for the SOFTWARE transcode child. The Plex ffmpeg is a Popen in the long-lived API
# process (which must NOT itself be niced), so — unlike the download worker's _deprioritize (run once in
# its own main()) — we lower priority per-child via preexec_fn in the fork. The GPU path needs none of
# this. Reuses the download child's nice default so all co-located re-encoders share one handicap.
_SW_TRANSCODE_NICE = settings.download_child_nice
def _sw_child_preexec() -> None:
"""Runs in the forked child (POSIX) before exec: renice this software-transcode ffmpeg down so it
can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
playback."""
try:
os.nice(_SW_TRANSCODE_NICE)
except OSError:
pass
def _probe_audio_langs(src: Path) -> list[str]:
"""One entry per audio stream (its language tag, or "" if untagged). len() = audio-track count.
Used to build the multi-rendition var_stream_map. Empty on probe failure → single-audio path."""
@@ -149,8 +440,19 @@ def _ffmpeg_cmd(
audio_ord: int | None,
aoff: float = 0.0,
audio_langs: list[str] | None = None,
transcode: bool = False,
hw: bool = False,
tonemap: bool = False,
preset: str = "veryfast",
crf: int = 20,
max_height: int = 0,
) -> list[str]:
args = ["ffmpeg", "-nostdin", "-loglevel", "error"]
# For a transcode session the video is re-encoded (VAAPI or libx264) instead of stream-copied; the
# hwaccel decode options go before -i, the encoder args replace `-c:v copy` below. Everything else is
# identical to remux. `tonemap` is the HDR/10-bit HW variant (OpenCL tonemap between VAAPI decode and
# encode) — it needs matching decode-side (named devices) and encode-side (the OpenCL filter graph) opts.
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height)
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap)]
# Keep ORIGINAL timestamps on the output so the first segment carries the true absolute PTS of the
# keyframe ffmpeg actually seeked to. hls.js still zero-bases playback to that PTS; we read it back
# (see _probe_first_pts) to learn the real content offset. Without -copyts the segment PTS is
@@ -171,7 +473,7 @@ def _ffmpeg_cmd(
args += ["-map", "0:v:0"]
for i in range(len(audio_langs)):
args += ["-map", f"{ai}:a:{i}?"]
args += ["-sn", "-c:v", "copy", "-c:a", "aac", "-ac", "2", "-b:a", "192k"]
args += ["-sn", *venc, "-c:a", "aac", "-ac", "2", "-b:a", "192k"]
var = ["v:0,agroup:aud"]
for i, lang in enumerate(audio_langs):
entry = f"a:{i},agroup:aud,name:a{i}"
@@ -192,7 +494,7 @@ def _ffmpeg_cmd(
else:
ao = audio_ord if audio_ord is not None else 0
args += ["-map", "0:v:0", "-map", f"{ai}:a:{ao}?", "-sn"]
args += ["-c:v", "copy", "-c:a", "aac", "-ac", "2", "-b:a", "192k"]
args += [*venc, "-c:a", "aac", "-ac", "2", "-b:a", "192k"]
args += _HLS_TAIL + [
"-hls_segment_filename",
str(out_dir / "seg_%d.ts"),
@@ -247,12 +549,14 @@ def _kill(s: HlsSession) -> None:
def _enforce_cap(cap: int) -> None:
# Caller holds _lock. Drop the least-recently-accessed sessions over the cap.
if len(_sessions) <= cap:
# Caller holds _lock. Only TRANSCODE sessions count toward plex_max_transcodes — remux
# (stream-copy) and direct-serve are cheap and deliberately uncapped (see config.py:
# plex_max_transcodes, and the user-facing config hint). Drop the least-recently-accessed
# transcode sessions over the cap, leaving any concurrent remux streams untouched.
tx = [(key, s) for key, s in _sessions.items() if s.transcode]
if len(tx) <= cap:
return
for key, s in sorted(_sessions.items(), key=lambda kv: kv[1].last_access)[
: len(_sessions) - cap
]:
for key, s in sorted(tx, key=lambda kv: kv[1].last_access)[: len(tx) - cap]:
_kill(s)
_sessions.pop(key, None)
@@ -264,11 +568,14 @@ def start_session(
audio_ord: int | None = None,
aoff: float = 0.0,
multi: bool = False,
transcode: bool = False,
) -> HlsSession | None:
"""(Re)start the HLS remux for an item at the given offset. `multi` (item has >1 audio track) maps
every audio track as an HLS rendition so the client switches audio without a restart; otherwise a
single audio track (`audio_ord`) is muxed. Subtitles are separate WebVTT tracks (not muxed here).
Returns None if the local file can't be read."""
"""(Re)start the HLS session for an item at the given offset. `transcode` (item is non-h264,
playable="transcode") re-encodes the video to H.264 (VAAPI or libx264) instead of stream-copying it;
otherwise the video is copied (remux). `multi` (item has >1 audio track) maps every audio track as an
HLS rendition so the client switches audio without a restart; otherwise a single audio track
(`audio_ord`) is muxed. Subtitles are separate WebVTT tracks (not muxed here). Returns None if the
local file can't be read."""
src = paths.local_media_path(db, item.file_path)
if src is None:
return None
@@ -276,43 +583,125 @@ def start_session(
start_s = max(0.0, float(start_s))
audio_langs = _probe_audio_langs(src) if multi else []
is_multi = len(audio_langs) > 1 # only worth a master playlist when there really are ≥2 tracks
with _lock:
old = _sessions.pop(key, None)
if old is not None:
_kill(old)
tag = "multi" if is_multi else str(audio_ord)
directory = _HLS_ROOT / f"{key}_{int(start_s)}_{tag}_{aoff:+.2f}"
shutil.rmtree(directory, ignore_errors=True)
directory.mkdir(parents=True, exist_ok=True)
proc = subprocess.Popen(
_ffmpeg_cmd(
src, start_s, directory, audio_ord, aoff, audio_langs if is_multi else None
),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
tag = "multi" if is_multi else str(audio_ord)
directory = _HLS_ROOT / f"{key}_{int(start_s)}_{tag}_{aoff:+.2f}"
def _seg(i: int) -> Path:
# multi-audio: the video rendition is variant v0 → seg_{i}_0.ts; single-audio → seg_{i}.ts
return directory / (f"seg_{i}_0.ts" if is_multi else f"seg_{i}.ts")
def _spawn(
hw: bool, tonemap: bool = False, *, replace_only: "HlsSession | None" = None
) -> "HlsSession | None":
# `replace_only` makes the supersede identity-aware for the runtime fallback: the whole
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
# seek-restart) installed between the fallback's decision and here is never clobbered. When the
# registered session is no longer `replace_only`, we abort and return None (the caller yields to
# whoever is registered now). The INITIAL spawn passes replace_only=None and always supersedes —
# that is the seek-restart's intended last-writer-wins.
# Software-fallback encode tuning — read only on the SW transcode path (the VAAPI path and
# remux ignore preset/crf/max_height, and remux must not touch the DB for them).
preset, crf, max_height = "veryfast", 20, 0
if transcode and not hw:
preset = sysconfig.get_str(db, "plex_transcode_preset") or "veryfast"
crf = sysconfig.get_int(db, "plex_transcode_crf")
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
# can't starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a
# platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
with _lock:
if replace_only is not None and _sessions.get(key) is not replace_only:
return None # superseded/culled while we waited out of lock — don't clobber the newer one
old = _sessions.pop(key, None)
if old is not None:
_kill(old)
shutil.rmtree(directory, ignore_errors=True)
directory.mkdir(parents=True, exist_ok=True)
proc = subprocess.Popen(
_ffmpeg_cmd(
src,
start_s,
directory,
audio_ord,
aoff,
audio_langs if is_multi else None,
transcode=transcode,
hw=hw,
tonemap=tonemap,
preset=preset,
crf=crf,
max_height=max_height,
),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
preexec_fn=preexec,
)
s = HlsSession(
key,
directory,
proc,
start_s,
"master.m3u8" if is_multi else "index.m3u8",
transcode=transcode,
hw=hw,
tonemap=tonemap,
)
_sessions[key] = s
# Admin-configurable concurrency cap (Configuration → Plex → Max concurrent transcodes);
# at least 1 so playback can't be capped to zero.
_enforce_cap(max(1, sysconfig.get_int(db, "plex_max_transcodes")))
return s
s = _spawn(*_select_encode(db, item, transcode, src))
# Wait for the first video segment outside the lock (this blocks on ffmpeg, and we must not hold up
# other sessions). Pass the proc so a HW session that dies at VAAPI init fails fast (not after 25s).
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
if not ready and s.hw:
# We waited on _seg(0) OUTSIDE _lock, so `s.proc` dying is ambiguous: it may have failed on its
# own (VAProfile/init/tonemap refusal), OR a concurrent same-key _spawn (a rapid seek-restart) or
# _enforce_cap may have killed it to install/keep a newer session. Respawn on libx264 (no HW, no
# tonemap) so a genuinely-failed GPU session never strands playback — but pass `replace_only=s` so
# the respawn is a no-op if we've been superseded: the identity check and the pop+replace share
# ONE lock in _spawn, so we can never tear down the newer session. s.hw is now False, so the route
# honestly reports "CPU" and the player's Preparing-stream state stays correct.
log.warning(
"plex transcode: HW session produced no segments — falling back to software key=%s", key
)
s = HlsSession(key, directory, proc, start_s, "master.m3u8" if is_multi else "index.m3u8")
_sessions[key] = s
# Admin-configurable concurrency cap (Configuration → Plex → Max concurrent transcodes);
# at least 1 so playback can't be capped to zero.
_enforce_cap(max(1, sysconfig.get_int(db, "plex_max_transcodes")))
# Learn the REAL start offset K from the first VIDEO segment (outside the lock — this blocks on
# ffmpeg producing it, and we must not hold up other sessions). The client reads `media_start_s`, so
# its absolute clock + subtitle shift key off the true keyframe, not the requested offset. On
# timeout/probe failure we keep the provisional requested offset (old behaviour, small drift).
if start_s > 0:
seg0 = directory / ("seg_0_0.ts" if is_multi else "seg_0.ts") # multi: video variant = v0
if wait_for(seg0, timeout=25.0):
k = _probe_first_pts(seg0)
respawned = _spawn(False, False, replace_only=s)
if respawned is None:
# Superseded/culled mid-fallback — yield whoever is registered now (a newer seek-restart), or
# our own now-dead session if the key was cap-culled. Never None here: the POST /session route
# turns a None return into a "Media file not found on disk" 404 — dishonest for a file that
# exists. A (dead) session instead degrades to a retryable 404 ("No active playback session" /
# "Not available yet") the client re-requests.
return current_session(key) or s
s = respawned
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
if ready:
# For a seek (start_s>0) learn the REAL start offset K from the first segment's absolute PTS,
# so the client's absolute clock + subtitle shift key off the true keyframe, not the requested
# offset. On probe failure we keep the provisional requested offset (old behaviour, small drift).
if start_s > 0:
k = _probe_first_pts(_seg(0))
if k is not None and k >= 0:
s.media_start_s = k
# Warm up the buffer (see _WARMUP_SEGMENTS) so the client doesn't outrun the cold ffmpeg. Stop
# at the first segment that isn't ready in time — ffmpeg produces them faster than realtime once
# warm, so a miss means something slow and we proceed rather than block the request further.
for i in range(1, _WARMUP_SEGMENTS):
if not wait_for(_seg(i), timeout=_WARMUP_SEG_TIMEOUT_S, proc=s.proc):
break
log.info(
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s",
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s tonemap=%s",
key,
start_s,
s.media_start_s,
is_multi,
audio_ord,
transcode,
s.hw,
s.tonemap,
)
return s
@@ -325,10 +714,14 @@ def current_session(key: str) -> HlsSession | None:
return s
def wait_for(path: Path, timeout: float = 20.0) -> bool:
def wait_for(path: Path, timeout: float = 20.0, proc: subprocess.Popen | None = None) -> bool:
"""Wait until a session file (playlist / segment) exists and is non-empty. Segments are
produced ~faster than realtime, so a segment just ahead of playback appears quickly; a segment
far beyond the generated region won't (the frontend restarts the session at a seek instead)."""
far beyond the generated region won't (the frontend restarts the session at a seek instead).
With `proc` given, also stop as soon as that ffmpeg exits: a VAAPI session that fails to init
(10-bit/HDR VAProfile refusal) dies within ~1s producing nothing, and without this we'd block the
whole `timeout` before the HW→SW fallback could kick in — turning the fix into a stall."""
end = time.time() + timeout
while time.time() < end:
try:
@@ -336,6 +729,12 @@ def wait_for(path: Path, timeout: float = 20.0) -> bool:
return True
except OSError:
pass
if proc is not None and proc.poll() is not None:
# the producer exited — no more segments are coming; one last check then give up.
try:
return path.exists() and path.stat().st_size > 0
except OSError:
return False
time.sleep(0.15)
return path.exists()
+130 -3
View File
@@ -18,7 +18,7 @@ item-detail/player path (P2), not here.
import logging
from datetime import datetime, timezone
from sqlalchemy import select
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import sysconfig
@@ -58,6 +58,57 @@ def classify_playable(vcodec: str | None, acodec: str | None, container: str | N
return "direct"
# Video codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for
# the full-GPU VAAPI transcode pipeline. THE single source of this set: plex.stream imports it here so
# the runtime encoder choice (`_want_hw`) and this catalog facet can never disagree by codec. A source
# outside it (AV1, MPEG-4 ASP/XviD…) is software-only even on the iGPU (it can't decode it to feed its
# own encoder).
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"})
# Color-transfer values marking an HDR source (PQ / HLG). A 10-bit/HDR source breaks the VAAPI
# scale=nv12 path on Gen 9.5 (see plex.stream) → software transcode, so it counts as NOT HW-capable.
_HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
def transcode_hw_capable(
playable: str | None,
vcodec: str | None,
bit_depth: int | None = None,
hdr: bool = False,
) -> bool | None:
"""The stored `PlexItem.transcode_hw` facet: for a `playable="transcode"` item, whether the prod
iGPU can HARDWARE-transcode it (else software libx264 only); None for a non-transcode item. HW needs
a HW-decodable codec AND an 8-bit SDR source — a 10-bit/HDR source (bit_depth>8 or an HDR transfer)
breaks the VAAPI nv12 pipeline and falls back to software. `bit_depth`/`hdr` are unknown at sync
time (the cheap listing carries no Stream), so they default to 8-bit SDR (codec-only) and the
bit-depth backfill refines a 10-bit/HDR HEVC to False later."""
if playable != "transcode":
return None
if (vcodec or "").lower() not in HW_DECODABLE_CODECS:
return False
if bit_depth is not None and bit_depth > 8:
return False
return not hdr
def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
"""(bit_depth, is_hdr) from a FULL Plex metadata dict's video Stream (Media→Part→Stream, streamType
1). Only the rich metadata call carries Stream; the cheap listing does not — so this is read by the
bit-depth backfill, not the listing sync. (None, False) when absent/unparseable."""
for m in meta.get("Media") or []:
for part in m.get("Part") or []:
for s in part.get("Stream") or []:
if s.get("streamType") != 1: # 1 = video
continue
bd = s.get("bitDepth")
try:
depth = int(bd) if bd else None
except (TypeError, ValueError):
depth = None
hdr = str(s.get("colorTrc") or "").lower() in _HDR_TRANSFERS
return depth, hdr
return None, False
def _epoch(v) -> datetime | None:
try:
return datetime.fromtimestamp(int(v), tz=timezone.utc) if v else None
@@ -217,6 +268,13 @@ def sync(db: Session) -> dict:
stats["people_enriched"] = _enrich_people_batch(db, plex)
except Exception as e: # best-effort — never fail the sync over the backfill
log.warning("Plex people enrich batch failed: %s", e)
# Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
# playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
try:
stats["media_probed"] = _probe_media_batch(db, plex)
except Exception as e: # best-effort — never fail the sync over the backfill
log.warning("Plex media probe batch failed: %s", e)
except PlexNotConfigured as e:
return {"skipped": str(e)}
except PlexError as e:
@@ -236,6 +294,49 @@ def sync(db: Session) -> dict:
# How many titles to rich-fetch per sync run for the cast backfill — bounds the extra Plex load; the
# whole library is covered over successive runs (each fetched item is stamped people_enriched_at once).
_ENRICH_BATCH = 60
# Same bounded-per-run pattern for the bit-depth probe. Only HW-decodable-codec transcode items are
# probed (typically a small minority — the bulk of transcodes are old codecs that are SW by codec), so
# a modest batch covers the work-list in a run or two; a stuck item retries next run (watermark unset).
_PROBE_BATCH = 60
def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH) -> int:
"""Rich-fetch up to `limit` not-yet-probed HW-decodable transcode items, read the video Stream's
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit/HDR HEVC →
False/software). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
so an item is probed once and not re-fetched every sync; a transient fetch FAILURE leaves the
watermark unset to retry. Oldest-id first. Returns how many items were probed this run."""
rows = (
db.execute(
select(PlexItem)
.where(
PlexItem.playable == "transcode",
func.lower(PlexItem.codec_video).in_(HW_DECODABLE_CODECS),
PlexItem.media_probed_at.is_(None),
)
.order_by(PlexItem.id)
.limit(limit)
)
.scalars()
.all()
)
now = datetime.now(timezone.utc)
n = 0
for row in rows:
try:
meta = plex.metadata(row.rating_key) or {}
except Exception as e: # transient — leave media_probed_at NULL so it retries next run
log.debug("Plex media probe fetch failed for %s: %s", row.rating_key, e)
continue
bit_depth, hdr = _video_stream_props(meta)
row.video_bit_depth = bit_depth
row.transcode_hw = transcode_hw_capable("transcode", row.codec_video, bit_depth, hdr)
row.media_probed_at = (
now # probed (even if Plex carried no Stream — don't re-fetch forever)
)
n += 1
db.commit()
return n
def _enrich_people_batch(db: Session, plex: PlexClient, limit: int = _ENRICH_BATCH) -> int:
@@ -369,15 +470,41 @@ def _apply_item(row: PlexItem, lib_id: int, kind: str, meta: dict) -> None:
row.added_at = _epoch(meta.get("addedAt"))
_apply_facet_fields(row, meta) # filterable/orderable extras (shared with shows)
mf = _media_facts(meta)
new_vcodec = mf.get("codec_video")
new_part_key = mf.get("part_key")
# The physical file was replaced when its video codec changes OR — for a SINGLE-version item — when its
# Plex Part key changes (a re-encode / quality upgrade gets a new part id). Either way any probed
# bit-depth is stale, so reset the probe watermark and re-derive transcode_hw from codec alone until the
# backfill re-probes. Codec alone is NOT enough: a same-codec bit-depth re-encode (8-bit→10-bit HEVC)
# keeps the codec `hevc` but flips the HW class, and only the part-key move catches it.
#
# The part-key check is gated on single-version because `_media_facts` reads media[0]/part[0], and for a
# MULTI-version item (an original + a Plex "Optimized" copy) that order isn't guaranteed stable across
# listings — an unguarded part-key compare would then reset+re-probe every sync and oscillate the
# transcode_hw bucket between the two versions. A multi-version same-codec re-encode not self-healing is
# an acceptable rare edge; a per-sync re-probe loop is not.
single_version = len(meta.get("Media") or []) <= 1
codec_changed = (row.codec_video or "").lower() != (new_vcodec or "").lower()
if codec_changed or (single_version and row.part_key != new_part_key):
row.video_bit_depth = None
row.media_probed_at = None
row.file_path = mf.get("file_path")
row.part_key = mf.get("part_key")
row.part_key = new_part_key
row.container = mf.get("container")
row.codec_video = mf.get("codec_video")
row.codec_video = new_vcodec
row.codec_audio = mf.get("codec_audio")
row.size_bytes = mf.get("size_bytes")
row.playable = classify_playable(
mf.get("codec_video"), mf.get("codec_audio"), mf.get("container")
)
# transcode_hw: keep the bit-depth backfill's authoritative value (it folds in bit-depth + HDR) once
# probed; otherwise (re)derive the provisional codec-only class. A non-transcode item is always None,
# even if it had been probed as a transcode before (an audio/container flip to remux without a codec
# change wouldn't reset the watermark, so pin it here).
if row.playable != "transcode":
row.transcode_hw = None
elif row.media_probed_at is None:
row.transcode_hw = transcode_hw_capable("transcode", row.codec_video, row.video_bit_depth)
def _prune_stale(
+109 -22
View File
@@ -11,11 +11,14 @@ already pages) and upsert them into the owner's `plex_states`. Plex wins on over
exist only in Siftlode are left untouched (union, Plex-authoritative on the intersection).
Phases B (Siftlode→Plex push via scrobble/timeline) and C (incremental Plex→Siftlode via the watch
history + onDeck, last-write-wins) build on this and land in later ships.
history + onDeck) build on this and land in later ships. As of S2 Plex is the source of truth: the
player pushes the resume position back to Plex on the periodic checkpoint (not just the final flush),
and the incremental pull resolves a genuine divergence to Plex's state (see `_pull_apply`) rather than
running a last-write-wins comparison.
"""
import logging
from datetime import datetime, timedelta, timezone
from datetime import datetime, timezone
from sqlalchemy import func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
@@ -293,12 +296,6 @@ def push_bulk_state_to_plex(user_id: int, changes: list[tuple[int, str, str]]) -
# --- Phase C: incremental + full Plex ↔ Siftlode reconcile (scheduler jobs) ------------------------
# Clock-skew / round-trip slack when comparing a Plex timestamp to a Siftlode one. A state we just
# pushed (Plex stamps viewedAt a beat after our watched_at) must not read as "newer on Plex" and get
# pulled back with a different shape. Genuine Plex-side edits are minutes from our writes — far
# outside this — so the window only absorbs skew, never real changes.
_LWW_TOLERANCE_S = 90
def _active_sync_links(db: Session) -> list[PlexLink]:
"""Every link with two-way sync actually running (owner MVP = one row). Iterated uniformly so
@@ -338,11 +335,6 @@ def _resolve_account_id(db: Session, plex: PlexClient, link: PlexLink) -> int:
return acct_id
def _sift_ts(st: PlexState) -> datetime | None:
"""The freshest Siftlode-side timestamp on a row, for last-write-wins."""
return max([t for t in (st.watched_at, st.progress_updated_at) if t], default=None)
def _same_state(st: PlexState | None, status: str, pos: int) -> bool:
"""Whether a row already matches a target (status, pos) — the core ping-pong guard: when nothing
would change we skip the write entirely (no row churn, no timestamp bump, no re-pull)."""
@@ -370,9 +362,20 @@ def _pull_apply(
if st is not None and not st.synced_to_plex:
st.synced_to_plex = True # already matches Plex → just settle the flag, no churn
return None
# Conflict: keep Siftlode when it's newer than Plex (plus skew slack).
sts = _sift_ts(st) if st else None
if sts and plex_ts and plex_ts <= sts + timedelta(seconds=_LWW_TOLERANCE_S):
# Plex-authoritative (S2): on a genuine divergence for an item Plex knows about, Plex wins — no
# last-write-wins tug-of-war. This is safe because prompt write-back (playback.py pushes the resume
# position to Plex on the ~10s checkpoint, not just on the final flush) keeps Plex current, so a
# local edit that's "newer" than Plex has almost always already been pushed and matched above by
# `_same_state`; a row that reaches here really is out of step with the source of truth. The
# `_same_state` + `synced_to_plex` short-circuit above still absorbs the freshly-pushed case (Plex
# echoes the same shape a beat later) so a just-pushed row never bounces.
if st is not None and not st.synced_to_plex:
# Durability guard: this row holds a local change that never reached Plex — its immediate push
# failed, so it's still dirty. Do NOT let the Plex-authoritative pull overwrite it and clear the
# flag: that would strand the local value forever, since `_repush_dirty` only retries dirty rows.
# Leave it dirty; `_repush_dirty` (later in this same run) pushes the local value up to Plex. This
# closes the "Plex flapped mid-session" lost-update — the one case the Plex-wins comment's
# "almost always already pushed" assumption above excludes.
return None
if st is None:
st = PlexState(user_id=user_id, item_id=item_id)
@@ -436,14 +439,89 @@ def _repush_dirty(db: Session, plex: PlexClient, link: PlexLink) -> int:
return n
# How many stale in-progress movies we probe per incremental run (one metadata() call each). The set
# is normally tiny (movies started but no longer on Plex's Continue-Watching); the daily full reconcile
# is the backstop for anything beyond the cap. Most-recently-touched are probed first.
_MOVIE_PROBE_CAP = 100
def _reconcile_inprogress_movies(
db: Session, plex: PlexClient, link: PlexLink, on_deck_ids: set[int]
) -> dict[str, int]:
"""Close the movie "stuck in progress" gap that the history + onDeck feeds structurally miss.
When a movie is finished on Plex (often by scrubbing through the end credits) Plex sets
``viewCount>0``, clears ``viewOffset`` and DROPS the movie from Continue-Watching. So onDeck no
longer lists it (and onDeck can only ever ADD an in-progress row, never resolve one to watched —
see `_pull_apply`), and a scrub-to-end frequently emits no `/status/sessions/history` session row,
so the history feed misses it too. The stale local ``status='new', position>0`` row then survives
until the *daily* full reconcile. Movies are hit hardest: a movie is a single leaf with no
episode-rollup to mask the stuck row, and once watched it leaves onDeck entirely.
Fix: for each local in-progress MOVIE that Plex's onDeck no longer lists, read the item's current
per-(token-)account ``viewCount``/``viewOffset`` with one cheap `metadata()` call and apply it via
the shared `_pull_apply` — flipping to watched when Plex says so. Returns ``{"watched", "in_progress",
"probed"}`` counts. Episodes are intentionally left to onDeck/history (they promote the next episode
and aren't affected). Note: like the rest of this job, `metadata()` reads the TOKEN account (the
owner) — a HTPC playing under a managed Plex user (accountID>1) is out of scope here as elsewhere."""
conds = [
PlexState.user_id == link.user_id,
PlexState.status.notin_(("watched", "hidden")),
PlexState.position_seconds > _PROGRESS_MIN_S,
PlexItem.kind == "movie",
]
if on_deck_ids:
conds.append(PlexState.item_id.notin_(on_deck_ids))
rows = db.execute(
select(PlexState, PlexItem.rating_key)
.join(PlexItem, PlexItem.id == PlexState.item_id)
.where(*conds)
.order_by(PlexState.progress_updated_at.desc().nullslast())
.limit(_MOVIE_PROBE_CAP + 1)
).all()
if len(rows) > _MOVIE_PROBE_CAP:
rows = rows[:_MOVIE_PROBE_CAP]
log.warning(
"in-progress movie probe hit the %d cap (user %s); reconcile catches the tail",
_MOVIE_PROBE_CAP,
link.user_id,
)
out = {"watched": 0, "in_progress": 0, "probed": 0}
for st, rk in rows:
try:
meta = plex.metadata(rk)
except PlexError as e:
log.warning(
"movie watch-probe failed (user %s, item %s): %s", link.user_id, st.item_id, e
)
continue
out["probed"] += 1
if meta is None:
continue
res = _plex_watch_to_state(
meta.get("viewCount"), meta.get("viewOffset"), meta.get("lastViewedAt")
)
if res is None:
# Plex holds no watch signal now (viewCount 0, no offset) — don't clear it here; an un-watch
# is the daily reconcile's job, and a Siftlode-only in-progress row must be preserved.
continue
status, pos, watched_at, prog_at = res
ch = _pull_apply(db, link.user_id, st.item_id, status, pos, watched_at or prog_at)
if ch:
out[ch] += 1
return out
def run_plex_watch_sync(db: Session) -> dict:
"""Scheduler job (default 30 min) — the cheap incremental pass. Pulls Plex-side changes since the
high-water mark (watch history + on-deck, filtered to the owner account) into Siftlode under
last-write-wins, then re-pushes any still-unsynced local states. Does NOT propagate un-watches
(the history feed only carries positive views) — that's `run_plex_watch_reconcile`'s job."""
"""Scheduler job (default 15 min) — the cheap incremental pass. Pulls Plex-side changes since the
high-water mark (watch history + on-deck, filtered to the owner account) into Siftlode, resolving a
divergence to Plex (the source of truth — see `_pull_apply`), reconciles stale in-progress movies
Plex has since finished (see `_reconcile_inprogress_movies`), then re-pushes any still-unsynced
local states. Does NOT propagate un-watches (the history feed only carries positive views) — that's
`run_plex_watch_reconcile`'s job."""
if not sysconfig.get_bool(db, "plex_enabled"):
return {"skipped": "disabled"}
total = {"links": 0, "watched": 0, "in_progress": 0, "repushed": 0}
total = {"links": 0, "watched": 0, "in_progress": 0, "repushed": 0, "movie_probes": 0}
for link in _active_sync_links(db):
total["links"] += 1
try:
@@ -463,10 +541,14 @@ def run_plex_watch_sync(db: Session) -> dict:
if ch:
total[ch] += 1
# onDeck resolves resume-position changes AND tells us which items Plex still considers
# in-progress (needed by the movie reconcile below to know which local rows are stale).
on_deck_ids: set[int] = set()
for row in plex.on_deck():
iid = rk_to_id.get(str(row.get("ratingKey") or ""))
if iid is None:
continue
on_deck_ids.add(iid)
off = int(row.get("viewOffset") or 0) // 1000
if off < _PROGRESS_MIN_S:
continue
@@ -476,6 +558,11 @@ def run_plex_watch_sync(db: Session) -> dict:
if ch:
total[ch] += 1
mv = _reconcile_inprogress_movies(db, plex, link, on_deck_ids)
total["watched"] += mv["watched"]
total["in_progress"] += mv["in_progress"]
total["movie_probes"] += mv["probed"]
total["repushed"] += _repush_dirty(db, plex, link)
link.last_watch_sync_at = now
db.commit()
@@ -491,7 +578,7 @@ def run_plex_watch_sync(db: Session) -> dict:
def run_plex_watch_reconcile(db: Session) -> dict:
"""Scheduler job (default daily) — the authoritative full reconcile. Rescans every section and,
per item, uses `synced_to_plex` to settle what the incremental feed can't:
- Plex has a state, Siftlode differs → last-write-wins (Plex usually the newer here).
- Plex has a state, Siftlode differs → Plex wins (the source of truth — see `_pull_apply`).
- Siftlode has a state, Plex has NO record:
· synced_to_plex=True → Plex un-watched it since our last sync → clear it (un-watch).
· synced_to_plex=False → a local change that never reached Plex → re-push it (via
+7
View File
@@ -18,6 +18,13 @@ class RateLimiter:
self._hits: dict[str, list[float]] = {}
self._lock = threading.Lock()
def reset(self) -> None:
"""Forget every recorded attempt. Each limiter is a module-level singleton, so a test suite
sharing one process would otherwise pool all of its attempts into a single window and
eventually trip an unrelated test with a 429."""
with self._lock:
self._hits.clear()
def allow(self, key: str) -> bool:
"""Record an attempt for ``key`` and return whether it is within the limit.
+74 -5
View File
@@ -1,6 +1,7 @@
"""Admin-only onboarding: review access requests (Invites), approve/deny, manual add.
Also the demo-account admin surface: the demo email whitelist + a manual state reset."""
from collections.abc import Iterable
from datetime import datetime, timezone
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
@@ -18,6 +19,7 @@ from app.auth import (
operator_contact,
purge_user,
)
from app.config import settings
from app.db import get_db
from app.demo import reset_demo_state
from app.models import (
@@ -31,7 +33,26 @@ from app.utils import iso, valid_email
router = APIRouter(prefix="/api/admin", tags=["admin"])
def _serialize(inv: Invite) -> dict:
def _accounts_for(db: Session, emails: Iterable[str]) -> set[str]:
"""Which of these addresses already have an account. Batched, so listing the invites costs one
extra query. An invite whose address has an account must not be deletable: `is_allowed` reads
this row, so removing it would silently revoke a live user's access."""
wanted = {e.lower() for e in emails if e}
if not wanted:
return set()
rows = db.execute(select(User.email).where(User.email.in_(wanted))).scalars()
return {e.lower() for e in rows if e}
def _env_allowed(email: str) -> bool:
"""Whether the env bootstrap lists still let this address in. `is_allowed` consults them only
when no Invite row exists, so this is exactly what deleting the row would fall back to — and the
admin has to know that BEFORE they confirm, not from a toast afterwards."""
email = (email or "").lower()
return email in settings.allowed_email_set or email in settings.admin_email_set
def _serialize(inv: Invite, *, has_account: bool = False) -> dict:
return {
"id": inv.id,
"email": inv.email,
@@ -40,6 +61,11 @@ def _serialize(inv: Invite) -> dict:
"requested_at": iso(inv.requested_at),
"decided_at": iso(inv.decided_at),
"decided_by": inv.decided_by,
# Drives the delete affordance: an entry backed by an account is not removable here.
"has_account": has_account,
# …and whether removing it would actually take access away, or leave the env allow-list
# granting it anyway (for an ADMIN_EMAILS address, as an admin).
"still_allowed_by_env": _env_allowed(inv.email),
}
@@ -56,7 +82,8 @@ def list_invites(
rows = db.execute(stmt.order_by(Invite.requested_at.desc())).scalars().all()
order = {"pending": 0, "approved": 1, "denied": 2}
rows.sort(key=lambda i: order.get(i.status, 9))
return [_serialize(i) for i in rows]
accounts = _accounts_for(db, (i.email for i in rows))
return [_serialize(i, has_account=(i.email or "").lower() in accounts) for i in rows]
def _activate_user_by_email(db: Session, email: str) -> None:
@@ -107,7 +134,7 @@ def approve_invite(
inv = _decide(db, invite_id, admin, approved=True)
_activate_user_by_email(db, inv.email)
background.add_task(email_mod.send_access_approved, inv.email)
return _serialize(inv)
return _serialize(inv, has_account=bool(_accounts_for(db, [inv.email])))
@router.post("/invites/{invite_id}/deny")
@@ -121,7 +148,7 @@ def deny_invite(
# account is activated, so this is an email only (the requester has no in-app inbox yet).
inv = _decide(db, invite_id, admin, approved=False)
background.add_task(email_mod.send_access_denied, inv.email)
return _serialize(inv)
return _serialize(inv, has_account=bool(_accounts_for(db, [inv.email])))
class AddInviteIn(BaseModel):
@@ -150,7 +177,49 @@ def add_invite(
)
db.commit()
_activate_user_by_email(db, email)
return _serialize(inv)
return _serialize(inv, has_account=bool(_accounts_for(db, [email])))
@router.delete("/invites/{invite_id}")
def delete_invite(
invite_id: int,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Remove a whitelist/request entry outright — the counterpart to add_invite, for an address that
never became an account (an approval added by mistake, or one that is no longer wanted). Deny
records a decision and keeps the row for the audit trail; delete forgets the address. Neither
stops the person asking again: `upsert_pending_invite` reopens a denied row, and a deleted one
simply comes back as a fresh pending request.
Refused once an account exists for the address, whatever its state: this row is the account's
approval gate, and orphaning the account from it would leave a person who can neither sign in nor
be approved. Delete the USER instead — purge_user removes the invite along with it.
Reports `still_allowed_by_env`, because deleting the row is NOT always a revoke: `is_allowed`
falls back to the env ALLOWED_EMAILS / ADMIN_EMAILS bootstrap lists once no invite exists, so for
an address on those the deletion leaves access intact (and an ADMIN_EMAILS address is still
promoted to admin on its next Google sign-in). The admin has to hear that, not guess it."""
inv = db.get(Invite, invite_id)
if inv is None:
raise HTTPException(status_code=404, detail="No such invite")
if _accounts_for(db, [inv.email]):
raise HTTPException(
status_code=409,
detail="That address already has an account — delete the account instead.",
)
still_allowed_by_env = _env_allowed(inv.email)
audit.record(
db,
admin.id,
AuditAction.INVITE_DELETE,
target_type="invite",
target_id=inv.id,
summary=inv.email,
)
db.delete(inv)
db.commit()
return {"deleted": invite_id, "still_allowed_by_env": still_allowed_by_env}
# --- Users & roles --------------------------------------------------------------------
+45 -7
View File
@@ -636,12 +636,25 @@ def _register_account(email: str, password: str) -> None:
# no-op as the pre-check hit — roll back and stop.
db.rollback()
return
upsert_pending_invite(db, email) # admin-approval gate (commits)
upsert_pending_invite(db, email) # admin-approval gate
# Persist the account ourselves. upsert_pending_invite commits only when it actually writes an
# invite row — not when this address already has a pending or approved one — and _issue_token
# below only runs with SMTP on. So for an address the admin had already approved (or one whose
# request was still pending), on an instance with no working SMTP, NOTHING along this path
# committed and the flushed user was silently rolled back when the session closed: the
# would-be user got the uniform "ok" and no account, and re-registering did the same again.
# Commit unconditionally; it is a no-op when upsert already committed the same transaction.
db.commit()
if email_ok:
raw = _issue_token(db, user, "verify", VERIFY_TTL)
# Fragment (#), not a query token: keeps the token out of proxy/access logs + Referer (SB3).
email_mod.send_verify_email(email, f"{_app_base()}/#verify={raw}")
admins = admin_notify_emails(db)
# Fired unconditionally, unlike the other two callers which gate on a fresh invite row: for
# an address the admin already approved, nothing lands in the Access-requests queue, so this
# is the only place the account's existence is announced at all. Delivery is a separate
# matter — with no working SMTP `_send` drops it, which is why the pre-approved registrant
# still needs the admin to notice them (roadmap UX-25).
if admins:
email_mod.send_admin_new_request(admins, email)
@@ -649,15 +662,22 @@ def _register_account(email: str, password: str) -> None:
@router.post("/verify")
def verify_email_confirm(body: TokenIn, db: Session = Depends(get_db)) -> dict:
"""Confirm an email-verification token the SPA read from the URL fragment (SB3). Uniform 400
on any invalid/expired/used token."""
on any invalid/expired/used token.
`password_reset_required` tells the SPA to send the user straight to the reset flow: an admin who
approved this account BEFORE it was verified tripped SEC-10, so the password chosen at registration
is gone and sign-in would answer with the deliberately uniform "Invalid email or password" — true,
but unreadable to the person who just proved they own the mailbox. Saying so leaks nothing: holding
a valid single-use verify token for this account already proves that much."""
user = _consume_token(db, body.token, "verify")
if user is None:
raise HTTPException(
status_code=400, detail="This verification link is invalid or has expired."
)
user.email_verified = True
needs_password = user.password_hash is None and user.password_revoked_at is not None
db.commit()
return {"ok": True}
return {"ok": True, "password_reset_required": needs_password}
@router.post("/verify/resend")
@@ -701,8 +721,10 @@ def verify_email(token: str, db: Session = Depends(get_db)):
if user is None:
return RedirectResponse(url="/?verify=invalid")
user.email_verified = True
needs_password = user.password_hash is None and user.password_revoked_at is not None
db.commit()
return RedirectResponse(url="/?verify=ok")
# Same two outcomes as the POST flow above, so an old link in flight lands on the same guidance.
return RedirectResponse(url="/?verify=" + ("ok_reset" if needs_password else "ok"))
@router.post("/password-login")
@@ -767,10 +789,18 @@ def _send_reset_if_eligible(email: str) -> None:
"""Background worker for password_reset_request: issue a reset token + email the link, but ONLY
for a real (non-demo) password account. Runs after the response with its own DB session, so the
request's timing never reveals whether the account exists. Token rides the URL FRAGMENT (#) so it
can't leak into proxy/access logs or a Referer (SB3)."""
can't leak into proxy/access logs or a Referer (SB3).
"Password account" is `password_hash OR password_revoked_at` — an account SEC-10 stripped had a
password and must be able to get a new one, or it is locked out permanently (no password to sign in
with, and formerly no reset link either). A Google-only account that never had a password stays
ineligible on purpose: its owner adds one from Settings (/auth/set-password) while signed in, which
proves identity through the session rather than by mail."""
with SessionLocal() as db:
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
if user is not None and user.password_hash and not user.is_demo:
if user is None or user.is_demo:
return
if user.password_hash or user.password_revoked_at:
raw = _issue_token(db, user, "reset", RESET_TTL)
email_mod.send_password_reset(email, f"{_app_base()}/#reset={raw}")
@@ -789,6 +819,7 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
if user is None:
raise HTTPException(status_code=400, detail="This reset link is invalid or has expired.")
user.password_hash = hash_password(new_password)
user.password_revoked_at = None # the account has a password again — SEC-10's trace is spent
# A valid reset link proves control of the mailbox, so completing a reset also verifies the
# email — this rescues an account that registered but never clicked the verification link (it
# can then sign in). The separate admin-approval gate (is_active) is deliberately NOT touched
@@ -831,9 +862,15 @@ def revoke_preset_password_if_unverified(user: User) -> None:
ownership is finally attested (Google adoption, or admin approval), drop that password and bump the
epoch to revoke any session it minted; the owner re-sets one via password-reset. No-op for an
already-verified account, so a legit no-SMTP registration is unaffected. The single source for this
rule — both the Google-callback and admin-activation paths call it so they can't drift."""
rule — both the Google-callback and admin-activation paths call it so they can't drift.
`password_revoked_at` is what makes "the owner re-sets one via password-reset" actually true. The
revoke leaves `password_hash IS NULL`, which is also what a Google-only account looks like — and
_send_reset_if_eligible keyed off exactly that, so a revoked account silently got no reset link and
was locked out for good (it could not sign in either). The stamp separates the two cases."""
if not user.email_verified:
user.password_hash = None
user.password_revoked_at = datetime.now(timezone.utc)
bump_session_epoch(user)
@@ -903,6 +940,7 @@ def set_password(
if not verify_password(body.current_password or "", user.password_hash):
raise HTTPException(status_code=403, detail="Current password is incorrect.")
user.password_hash = hash_password(new_password)
user.password_revoked_at = None # same as the reset path: a password exists again
# SA4: a password change logs out every OTHER session; re-stamp this cookie at the new epoch so
# the person making the change stays signed in here. Commit BEFORE re-stamping so a failed commit
# can't leave the cookie at a newer epoch than the DB (which would wrongly 401 THIS session).
+5
View File
@@ -229,6 +229,7 @@ def _channel_detail_dict(
count_live: int,
deep_requested: bool,
priority: int,
hidden: bool,
tag_ids: list[int],
) -> dict:
return {
@@ -262,6 +263,9 @@ def _channel_detail_dict(
# This user's per-subscription feed priority (0 when not subscribed) so the channel page can
# offer the same priority stepper as the manager. Editable only while subscribed.
"priority": priority,
# This user's per-subscription "hide from feed" flag (False when not subscribed), so the
# channel page can offer the same Unhide toggle the manager has. Editable only while subscribed.
"hidden": hidden,
"backfill_done": channel.backfill_done,
"from_explore": channel.from_explore,
"details_synced": channel.details_synced_at is not None,
@@ -344,6 +348,7 @@ def channel_detail(
count_live=count_live,
deep_requested=bool(sub.deep_requested) if sub else False,
priority=int(sub.priority) if sub else 0,
hidden=bool(sub.hidden) if sub else False,
tag_ids=tag_ids,
)
+149 -7
View File
@@ -6,23 +6,28 @@ worker is a separate process); this module never downloads — it enqueues, mana
serves finished files (range-aware, with the user's custom display name), and shares them.
"""
import os
import re
import tempfile
from datetime import datetime, timedelta, timezone
from pathlib import Path
from urllib.parse import parse_qs, urlparse
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import sysconfig
from app import audit, sysconfig
from app.auth import admin_user, require_human
from app.config import settings
from app.db import get_db
from app.downloads import credentials as dlcreds
from app.downloads import edit as editmod
from app.downloads import links as linksmod
from app.downloads import quota, service, storage
from app.downloads import upload as uploadmod
from app.models import (
DownloadJob,
DownloadLink,
@@ -122,6 +127,9 @@ def _serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
"width": asset.width,
"height": asset.height,
"container": asset.container,
# "video" | "audio" | "file" — the library gates play/edit on this (media only); a
# generic uploaded file is download-only. Null while still in flight (set at finalize).
"media_kind": asset.media_kind,
"uploader_url": asset.uploader_url,
"thumbnail_url": asset.thumbnail_url,
"expires_at": iso(asset.expires_at),
@@ -247,6 +255,65 @@ def delete_profile(
return {"deleted": profile_id}
# --- per-user download auth (cookies) ------------------------------------------------------
# Write-only surface: the stored blob is NEVER returned — only the supported hosts + whether each
# is configured + when. See downloads.credentials for the host-scope + cache-privacy guarantees.
@router.get("/credentials")
def list_credentials(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
return {"hosts": dlcreds.status_list(db, user.id)}
class SetCredentialIn(BaseModel):
cookies: str
@router.put("/credentials/{host_key}")
def set_credential(
host_key: str,
body: SetCredentialIn,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Store (encrypted) this user's cookies.txt for one supported host. Write-only: the value is
never echoed back, logged, or audited — only the fact that it changed."""
if not dlcreds.is_supported_host(host_key):
raise HTTPException(status_code=404, detail="Unknown host")
try:
cleaned = dlcreds.validate_cookies(body.cookies)
except dlcreds.InvalidCookies as e:
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)})
dlcreds.upsert_credential(db, user.id, host_key, cleaned)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_CREDENTIAL_SET,
target_type="download_credential",
target_id=host_key,
)
db.commit()
return {"ok": True, "hosts": dlcreds.status_list(db, user.id)}
@router.delete("/credentials/{host_key}")
def delete_credential(
host_key: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
if not dlcreds.is_supported_host(host_key):
raise HTTPException(status_code=404, detail="Unknown host")
if dlcreds.delete_credential(db, user.id, host_key):
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_CREDENTIAL_DELETE,
target_type="download_credential",
target_id=host_key,
)
db.commit()
return {"ok": True, "hosts": dlcreds.status_list(db, user.id)}
def _resolve_spec(
db: Session, user: User, profile_id: int | None, spec: dict | None
) -> tuple[dict, int | None]:
@@ -294,6 +361,61 @@ def enqueue_download(
return _serialize_job(db, job)
# Read the upload in 1 MiB chunks so a large file never sits fully in memory, and the size cap is
# enforced on the ACTUAL bytes read (a client-set Content-Length can't be trusted to gate it).
_UPLOAD_CHUNK = 1024 * 1024
@router.post("/upload")
def upload_download(
file: UploadFile = File(...),
display_name: str | None = Form(None),
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Ingest a local file into the caller's library (E7 S5). Any file type is accepted; media
(video/audio) gets the full function set, a generic file is download-only. The file is streamed
to a temp path under the download volume (enforcing the per-file cap as it goes), then handed to
the non-yt-dlp ingest which probes + places it and returns an already-`done` job.
A sync (threadpool) handler on purpose: the ingest does blocking work — an ffprobe of up to 30s
plus sync DB writes — so it must NOT run on the event loop. `file.file` is the sync file object
Starlette spools the multipart body into."""
cap = sysconfig.get_int(db, "download_upload_max_bytes")
staging = Path(settings.download_root) / ".staging" / "uploads"
staging.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(staging))
tmp_path = Path(tmp)
try:
size = 0
with os.fdopen(fd, "wb") as out:
while chunk := file.file.read(_UPLOAD_CHUNK):
size += len(chunk)
if cap and size > cap:
raise HTTPException(
status_code=422, detail={"code": "upload_too_large", "limit": cap}
)
out.write(chunk)
if size == 0:
raise HTTPException(status_code=400, detail={"code": "upload_empty"})
try:
job = uploadmod.ingest_upload(
db,
user.id,
filename=file.filename or "upload",
src_path=tmp_path,
size=size,
display_name=(display_name or "").strip() or None,
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
return _serialize_job(db, job)
finally:
# Ingest MOVES the temp file into place on success (unlink is then a no-op); on any early exit
# — cap hit, empty, quota, probe/move error — this removes the abandoned staging file.
tmp_path.unlink(missing_ok=True)
class EnqueueEditIn(BaseModel):
source_job_id: int = 0
edit_spec: dict
@@ -484,8 +606,10 @@ def redownload_download(
never deletes a file another job/user still shares: the old asset is only removed if THIS job
was its last holder."""
job = _own_job(db, user, job_id)
if job.job_kind == "edit":
raise HTTPException(status_code=409, detail="Editor clips can't be re-downloaded.")
if job.job_kind in ("edit", "upload"):
# Neither has a re-fetchable source: an editor clip is derived locally, an upload IS the
# source. Re-running yt-dlp against source_kind='upload'/'edit' would just fail.
raise HTTPException(status_code=409, detail="This item can't be re-downloaded.")
if job.status in ("queued", "running", "paused"):
raise HTTPException(status_code=409, detail="This download is still in progress.")
service.redownload(db, job)
@@ -550,8 +674,21 @@ def download_file(
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
# Starlette's FileResponse honours the Range header (206 partial content) for seek/resume.
return FileResponse(path, filename=filename)
# The served Content-Type comes from the ffprobe-verified media_kind + a container allow-list
# (storage.served_media_type), NEVER guessed from the file's own extension. A generic 'file' (and
# any media whose container we don't recognise — e.g. a media/HTML polyglot named "evil.html")
# returns None → a neutral force-download attachment, so uploaded HTML/SVG can't execute inline on
# this origin. Recognised media is served with its real type so the in-app player streams it. Both
# are range-aware (FileResponse → 206).
media_type = storage.served_media_type(asset.media_kind, asset.container)
if media_type is None:
return FileResponse(
path,
filename=filename,
media_type="application/octet-stream",
content_disposition_type="attachment",
)
return FileResponse(path, filename=filename, media_type=media_type)
# --- editor filmstrip (scrub track) --------------------------------------------------------
@@ -763,7 +900,12 @@ def list_links(
class CreateLinkIn(BaseModel):
password: str = ""
allow_download: bool = False
# D1 (E7 S3): a newly shared download is downloadable by default — the common intent is "let the
# recipient have the file", not stream-only. Forward-only: this only sets the default for links
# minted from now on; existing links keep whatever `allow_download` they were created with (the
# column default stays False, so no back-catalogue link silently flips). The owner can still
# uncheck it in the Share dialog, and toggle any link stream-only afterwards.
allow_download: bool = True
expires_days: int | None = None
+17
View File
@@ -14,6 +14,7 @@ from app.feedquery import (
decode_cursor,
encode_cursor,
feed_summary,
feed_years,
filtered_query,
keyset_predicate,
order_by,
@@ -53,6 +54,8 @@ def _feed_params(
tag_mode_language: str = "or",
tag_mode_topic: str = "or",
tag_mode_other: str = "or",
# "No tag" filter: keep only channels the user has not personally tagged (no user-owned ChannelTag).
untagged: bool = False,
channel_ids: list[str] = Query(default=[]),
q: str | None = None,
min_duration: int | None = None,
@@ -74,6 +77,7 @@ def _feed_params(
"topic": tag_mode_topic,
"other": tag_mode_other,
},
"untagged": untagged,
"channel_ids": channel_ids,
"q": q,
"min_duration": min_duration,
@@ -203,6 +207,19 @@ def get_feed_summary(
return feed_summary(db, user, **params)
@router.get("/feed/years")
def get_feed_years(
params: dict = Depends(_feed_params),
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Per-year video counts for the year control (item 8 / E8·S2): the quick recent-year chips and the
"more…" per-year-counts dialog. Every filter EXCEPT the publish-date narrowing applies, so picking a
year re-queries the feed but not this list. Response: {"years": [{"year": <int>, "count": <int>}]},
newest year first."""
return {"years": feed_years(db, user, **params)}
class VideoStateIn(BaseModel):
status: str
+7
View File
@@ -13,6 +13,7 @@ from app.auth import (
optional_current_user,
purge_user,
)
from app import email as email_mod
from app import sysconfig
from app.db import get_db
from app.models import Invite, User
@@ -98,11 +99,16 @@ def get_me(
if user is None:
return {"authenticated": False}
pending_invites = 0
# Admin-only, because it drives an operator warning: email is fail-soft, so a half-filled SMTP
# config costs verification links, password resets and access-request notices with nothing
# visible anywhere. Surfacing it here puts the warning on the screens where onboarding happens.
email_configured = True
if user.role == "admin":
pending_invites = (
db.scalar(select(func.count()).select_from(Invite).where(Invite.status == "pending"))
or 0
)
email_configured = email_mod.email_enabled(db) # this request's session; no second connect
return {
"authenticated": True,
"id": user.id,
@@ -121,6 +127,7 @@ def get_me(
"can_read": has_read_scope(user),
"can_write": has_write_scope(user),
"pending_invites": pending_invites,
"email_configured": email_configured,
"preferences": user.preferences or {},
}
+12 -12
View File
@@ -14,8 +14,8 @@ from app.db import get_db
from app.feedquery import feed_columns, serialize
from app.models import Channel, Playlist, PlaylistItem, User, Video, VideoState
from app.sync.playlists import (
plan_push,
push_playlist,
merge_playlist,
plan_merge,
recompute_dirty,
repull_playlist,
sync_user_playlists,
@@ -329,13 +329,13 @@ def push_plan(
user: User = Depends(require_write_scope),
db: Session = Depends(get_db),
) -> dict:
"""Dry run: what 'Sync to YouTube' would do + the quota estimate, so the UI can show a
confirm. For a linked playlist this reads the live YouTube state (cheap, 1 unit/page).
Write scope is required (require_write_scope) — syncing is a YouTube write."""
"""Dry run: what a two-way 'Sync to YouTube' merge would do (what each side gains/loses) +
the quota estimate, so the UI can show a confirm. For a linked playlist this reads the live
YouTube state (cheap, 1 unit/page). Write scope is required — syncing is a YouTube write."""
pl = _pushable(db, user, playlist_id)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
plan = plan_push(db, user, pl)
plan = plan_merge(db, user, pl)
except YouTubeError:
raise HTTPException(status_code=422, detail="Couldn't read the playlist on YouTube.")
plan["remaining_today"] = quota.remaining_today(db)
@@ -349,14 +349,14 @@ def push(
user: User = Depends(require_write_scope),
db: Session = Depends(get_db),
) -> dict:
"""Push a local playlist to YouTube (create it on first push, else reconcile to match
local — local wins). Refuses if the estimated quota exceeds what's left today, so a
big playlist can't strand half-pushed. Write scope required (require_write_scope)."""
"""Two-way sync a local playlist with YouTube (create it on first push, else 3-way merge so
neither side loses an item). Refuses if the estimated quota exceeds what's left today, so a
big sync can't strand half-applied. Write scope required (require_write_scope)."""
pl = _pushable(db, user, playlist_id)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
# Read the live playlist ONCE (linked playlists only) and share it with both the
# plan and the push, halving read-quota and closing the second read→write TOCTOU
# plan and the merge, halving read-quota and closing the second read→write TOCTOU
# window. A fresh export (no yt_playlist_id) reads nothing.
live = None
if pl.yt_playlist_id:
@@ -365,13 +365,13 @@ def push(
list(yt.iter_playlist_items_with_ids(pl.yt_playlist_id)),
yt.get_playlist_snippet(pl.yt_playlist_id),
)
plan = plan_push(db, user, pl, live=live)
plan = plan_merge(db, user, pl, live=live)
if plan["units_estimate"] > quota.remaining_today(db):
raise HTTPException(
status_code=429,
detail="Not enough YouTube quota left today for this sync.",
)
return push_playlist(db, user, pl, live=live)
return merge_playlist(db, user, pl, live=live)
except YouTubeError:
db.rollback()
raise HTTPException(status_code=422, detail="YouTube playlist sync failed.")
+50 -1
View File
@@ -2,6 +2,7 @@
from fastapi import APIRouter, Depends, Query
from sqlalchemy import (
Boolean,
Integer,
String,
and_,
@@ -27,6 +28,7 @@ from app.plex import rollup
from app.plex.cards import episode_card
from app.plex.query import (
FIELD_COLUMNS,
PLAYBACK_CLASSES,
TS_CONFIG,
LibraryFilters,
apply_meta_filters,
@@ -36,6 +38,8 @@ from app.plex.query import (
genres_col,
meta_filter_conds,
movie_status_case,
playable_conds,
playback_class_expr,
show_status_case,
split_csv,
summary_col,
@@ -135,9 +139,13 @@ def unified_library(
PlexItem.originally_available_at.label("rel"),
PlexItem.duration_s.label("duration_s"),
PlexItem.playable.label("playable"),
PlexItem.transcode_hw.label("transcode_hw"),
cast(null(), Integer).label("season_count"),
func.coalesce(st.position_seconds, 0).label("position_seconds"),
m_status.label("status"),
# "last watched" sort key: when the movie was last resumed, else when it was marked
# watched. NULL for never-touched movies → nullslast sinks them (see the sort dict).
func.coalesce(st.progress_updated_at, st.watched_at).label("last_watched"),
(
func.ts_rank(vector_col(PlexItem, lang), tsq)
if tsq is not None
@@ -153,13 +161,15 @@ def unified_library(
mq = mq.where(PlexItem.duration_s >= f.duration_min)
if f.duration_max is not None:
mq = mq.where(PlexItem.duration_s <= f.duration_max)
# Playback-class filter (movie-only — a show is an aggregate with no single class).
mq = mq.where(*playable_conds(split_csv(f.playable)))
if tsq is not None:
mq = mq.where(vector_match(PlexItem, lang, tsq))
if recheck is not None:
mq = mq.where(recheck(PlexItem, field, tsq, lang))
selects.append(mq)
if want_shows and grid_field:
agg = rollup.agg_subquery(db, user.id).subquery()
agg = rollup.agg_subquery(db, user.id, include_resumed=True).subquery()
tot = func.coalesce(agg.c.total, 0)
wat = func.coalesce(agg.c.watched, 0)
inp = func.coalesce(agg.c.inprog, 0)
@@ -176,9 +186,13 @@ def unified_library(
PlexShow.originally_available_at.label("rel"),
cast(null(), Integer).label("duration_s"),
cast(null(), String).label("playable"),
cast(null(), Boolean).label("transcode_hw"),
PlexShow.child_count.label("season_count"),
literal(0).label("position_seconds"),
s_status.label("status"),
# Show leg's "last watched": max(resume/watched time) across the show's episodes (see
# rollup.agg_subquery include_resumed). NULL for a never-watched show → nullslast.
agg.c.resumed_at.label("last_watched"),
(
func.ts_rank(vector_col(PlexShow, lang), tsq)
if tsq is not None
@@ -212,6 +226,7 @@ def unified_library(
"rating": u.c.rating,
"duration": u.c.duration_s,
"release": u.c.rel,
"lastwatched": u.c.last_watched,
}.get(sort, u.c.added_at)
order = [(col.asc() if sort_dir == "asc" else col.desc()).nullslast()]
order.append(u.c.rk.desc())
@@ -230,6 +245,7 @@ def unified_library(
"duration_seconds": r.duration_s,
"thumb": f"/api/plex/image/{r.rk}",
"playable": r.playable,
"transcode_hw": r.transcode_hw,
"status": r.status,
"position_seconds": r.position_seconds,
}
@@ -305,6 +321,7 @@ def facets(
"genres": [],
"content_ratings": [],
"studios": [],
"playable": [],
"year_min": None,
"year_max": None,
"rating_max": None,
@@ -331,6 +348,9 @@ def facets(
query = query.where(model.duration_s >= pp["duration_min"])
if with_dur and pp.get("duration_max") is not None:
query = query.where(model.duration_s <= pp["duration_max"])
# Playback-class filter (movie-only) — applied for the OTHER groups so an active class
# narrows them; the playable facet itself passes pp with playable=None (self-exclusion).
query = query.where(*playable_conds(split_csv(pp.get("playable"))))
# Join watch-state and reuse the grid's exact movie state filter (unified_library) —
# hidden movies never appear in the grid, so they must not inflate facet counts/bounds either.
st = aliased(PlexState)
@@ -349,6 +369,7 @@ def facets(
genre_counts: dict[str, int] = {}
cr_counts: dict[str, int] = {}
studio_counts: dict[str, int] = {}
playable_counts: dict[str, int] = {}
years: list[int] = []
ratings: list[float] = []
durs: list[int] = []
@@ -413,6 +434,23 @@ def facets(
)
for s, c in db.execute(stq).all():
studio_counts[s] = studio_counts.get(s, 0) + c
# Playback class (movie-only) — exclude its own so each class stays an addable OR option.
if kind_movie:
pcls = playback_class_expr()
pcq = (
wsq(
select(pcls.label("cls"), func.count()),
model,
ids,
{**p, "playable": None},
kind_movie,
True,
)
.where(pcls.isnot(None))
.group_by(pcls)
)
for cls, c in db.execute(pcq).all():
playable_counts[cls] = playable_counts.get(cls, 0) + c
# Year bounds — exclude the year filter.
by = db.execute(
wsq(
@@ -470,6 +508,11 @@ def facets(
# Same for the actively-selected studios, so a selected studio stays removable even at zero matches.
for s in split_csv(f.studios):
studio_counts.setdefault(s, 0)
# Keep an actively-selected playback class visible even at zero matches (the sidebar hides the whole
# section when its facet list is empty, which would trap a zero-result selection with no way to undo).
for c in split_csv(f.playable):
if c in PLAYBACK_CLASSES:
playable_counts.setdefault(c, 0)
return {
"genres": [
{"value": g, "count": c} for g, c in sorted(genre_counts.items(), key=lambda kv: kv[0])
@@ -481,6 +524,12 @@ def facets(
{"value": s, "count": c}
for s, c in sorted(studio_counts.items(), key=lambda kv: (-kv[1], kv[0]))
],
# Fixed good→heavy order (not by count), so the class list reads as a severity scale.
"playable": [
{"value": c, "count": playable_counts[c]}
for c in PLAYBACK_CLASSES
if c in playable_counts
],
"year_min": min(years) if years else None,
"year_max": max(years) if years else None,
"rating_max": max(ratings) if ratings else None,
+4
View File
@@ -411,6 +411,10 @@ def item_detail(
"year": it.year,
"duration_seconds": it.duration_s,
"playable": it.playable,
# Playback-class detail: HW/SW transcode split + the codec/bit-depth the info-page chip shows.
"transcode_hw": it.transcode_hw,
"codec_video": it.codec_video,
"video_bit_depth": it.video_bit_depth,
"thumb": f"/api/plex/image/{it.rating_key}",
"art": f"/api/plex/image/{it.rating_key}?variant=art",
"library": _lib_key(db, it.library_id),
+101 -16
View File
@@ -3,7 +3,8 @@
import hashlib
import os
import tempfile
from datetime import datetime, timezone
import threading
from datetime import datetime, timedelta, timezone
import httpx
from fastapi import (
@@ -155,6 +156,47 @@ _PROGRESS_MIN_S = 5
_FINISH_MARGIN_S = 30
# S2 write-back throttle: the resume position is now pushed to Plex on the periodic (non-final)
# checkpoint too — not only on the final pause/unmount flush — so Plex (the source of truth) stays
# current in near-real-time. To keep a single watch from spraying `/:/timeline` calls, a push is
# rate-limited to at most one per item per user every _PUSH_THROTTLE_S seconds; a `final` checkpoint
# (pause / seek / stop) always pushes immediately and resets the window. This is an in-memory, single-
# process rate-limiter (the deployment is one FastAPI process): after a restart the map is empty, so at
# most one extra push happens on the first checkpoint — harmless. It is NOT durable domain state.
_PUSH_THROTTLE_S = 10
_last_push_at: dict[tuple[int, int], datetime] = {}
_push_throttle_lock = threading.Lock()
def _should_push_resume(user_id: int, item_id: int, now: datetime, final: bool) -> bool:
"""Whether to mirror this resume checkpoint to Plex now. `final` always pushes; a periodic tick
pushes only when ≥_PUSH_THROTTLE_S has elapsed since the last push for this (user, item). Records
`now` as the last-push time whenever it returns True so the window is measured from the real push
(and so `_resume_push_superseded` can drop an older, still-pending push — see there)."""
with _push_throttle_lock:
last = _last_push_at.get((user_id, item_id))
if final or last is None or (now - last).total_seconds() >= _PUSH_THROTTLE_S:
_last_push_at[(user_id, item_id)] = now
# Opportunistic eviction so the map can't grow unbounded over a long-lived process (one
# entry per item ever checkpointed). Anything older than a few windows is dead — a push
# completes in seconds, so a stale entry protects nothing.
if len(_last_push_at) > 2048:
cutoff = now - timedelta(seconds=_PUSH_THROTTLE_S * 6)
for k in [k for k, t in _last_push_at.items() if t < cutoff]:
del _last_push_at[k]
return True
return False
def _resume_push_superseded(user_id: int, item_id: int, intent: datetime) -> bool:
"""True if a newer resume push for this (user, item) was scheduled after `intent`. Called at the
HTTP boundary of a background resume push: because periodic and seek/pause pushes are independent
background tasks with no ordering guard, an older still-pending periodic push could otherwise land
at Plex AFTER a newer backward-seek push and — with the Plex-authoritative pull — durably revive the
stale position. Skipping a superseded push keeps the last-scheduled position the one Plex settles on."""
with _push_throttle_lock:
return _last_push_at.get((user_id, item_id), intent) > intent
@router.get("/person-image")
def person_image(u: str, request: Request) -> Response:
@@ -178,6 +220,26 @@ def person_image(u: str, request: Request) -> Response:
return imgcache.response(r.content, ct)
def _push_resume_guarded(
user_id: int,
item_id: int,
rating_key: str,
time_ms: int,
duration_ms: int,
expect_ts: datetime | None,
intent: datetime,
) -> None:
"""Background entrypoint for a resume push that drops itself if a newer resume push for the same
(user, item) was scheduled after it (see `_resume_push_superseded`), then delegates to the real
push. This narrows the periodic-vs-seek write-ordering race to the (rare) case where the older push
has already left the process before the newer one is scheduled."""
if _resume_push_superseded(user_id, item_id, intent):
return
plex_watch.push_state_to_plex(
user_id, item_id, rating_key, "resume", time_ms, duration_ms, expect_ts
)
def _push_watch(
background: BackgroundTasks,
db: Session,
@@ -187,10 +249,14 @@ def _push_watch(
action: str,
time_ms: int = 0,
duration_ms: int = 0,
intent: datetime | None = None,
) -> None:
"""Schedule a best-effort Siftlode→Plex watch push, but only for a user whose owner link has
push enabled — non-owner users never touch Plex. Guarding here (with the request session) avoids
spinning up a throwaway background session on every state change for the common no-link case."""
spinning up a throwaway background session on every state change for the common no-link case.
A `resume` push with an `intent` timestamp is routed through `_push_resume_guarded` so a superseded
(older) push can't land out of order; `watched`/`unwatched` scrobbles go straight through."""
if plex_watch.link_for_push(db, user_id) is None:
return
# Capture the row's freshest timestamp NOW so the background push can tell if a newer local edit
@@ -202,6 +268,12 @@ def _push_watch(
.scalars()
.first()
)
marker = plex_watch._state_marker(st)
if action == "resume" and intent is not None:
background.add_task(
_push_resume_guarded, user_id, item_id, rating_key, time_ms, duration_ms, marker, intent
)
return
background.add_task(
plex_watch.push_state_to_plex,
user_id,
@@ -210,7 +282,7 @@ def _push_watch(
action,
time_ms,
duration_ms,
plex_watch._state_marker(st),
marker,
)
@@ -232,9 +304,10 @@ def item_progress(
) -> dict:
"""Checkpoint the resume position while playing (near the end → mark watched + clear it).
`final` (sent on the pause/pagehide/unmount flush, not the periodic 10s checkpoint) is what
debounces the Plex resume push: we mirror the settled position to Plex only on a final flush, so
a single watch doesn't spray `/:/timeline` calls at the server every 10 seconds."""
S2 write-back: the resume position is mirrored to Plex (the source of truth) on the periodic
checkpoint too, not only on the final flush — throttled to ≥_PUSH_THROTTLE_S per item (see
`_should_push_resume`). `final` (sent on pause / seek / pagehide / unmount) pushes immediately and
resets the throttle window, so Plex sees a settled position the instant the user pauses or seeks."""
it = item_or_404(db, rating_key)
position = max(0, int(body.position_seconds))
duration = int(body.duration_seconds)
@@ -259,6 +332,10 @@ def item_progress(
st.position_seconds = 0
st.progress_updated_at = now
st.synced_to_plex = False
# Terminal state for resume write-back: drop the throttle/ordering entry so the map doesn't
# retain a dead key for every finished item.
with _push_throttle_lock:
_last_push_at.pop((user.id, it.id), None)
db.commit()
# Immediate on the watched transition; skip a repeat scrobble if it was already watched
# (a later checkpoint of the same finished item would otherwise bump Plex's viewCount again).
@@ -280,10 +357,13 @@ def item_progress(
db.add(st)
st.position_seconds = position
st.progress_updated_at = now
if final:
push_now = _should_push_resume(user.id, it.id, now, final)
if push_now:
# Mark dirty before the (background) push so the reconcile knows to re-push if it fails, and so
# `push_state_to_plex` can settle synced_to_plex once Plex confirms.
st.synced_to_plex = False
db.commit()
if final:
if push_now:
_push_watch(
background,
db,
@@ -293,6 +373,7 @@ def item_progress(
"resume",
position * 1000,
duration * 1000,
intent=now,
)
return {"position_seconds": position}
@@ -451,15 +532,13 @@ def stream_session(
db: Session = Depends(get_db),
) -> dict:
"""Prepare playback from `start` seconds. Direct-playable files are served raw; remux files
(re)start an HLS session at the offset (seek-restart); transcode (HEVC/VP9) is P3. Selecting a
non-default audio track (`audio` = stream ordinal) forces the HLS path so the chosen audio can be
muxed in. Subtitles are independent WebVTT tracks (GET /subtitle) — no session restart."""
(re)start an HLS session at the offset (seek-restart, video stream-copied); transcode files (non-h264:
HEVC/VP9/AV1/…) take the same HLS path but with the video re-encoded to H.264 (VAAPI or libx264).
Selecting a non-default audio track (`audio` = stream ordinal) forces the HLS path so the chosen audio
can be muxed in. Subtitles are independent WebVTT tracks (GET /subtitle) — no session restart."""
plex_stream.reap_idle() # opportunistic cleanup of idle sessions on each new playback
it = item_or_404(db, rating_key)
if it.playable == "transcode":
raise HTTPException(
status_code=501, detail="This format needs transcoding (a later phase)."
)
transcode = it.playable == "transcode"
# `multi` (item has ≥2 audio tracks) forces the HLS path so ALL audio tracks ship as renditions and
# the client can switch audio without a restart — even for otherwise direct-playable files.
if it.playable == "direct" and audio is None and not aoff and not multi:
@@ -470,15 +549,21 @@ def stream_session(
}
if plex_paths.local_media_path(db, it.file_path) is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
s = plex_stream.start_session(db, it, start, audio, aoff, multi)
s = plex_stream.start_session(db, it, start, audio, aoff, multi, transcode=transcode)
if s is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
# Report the REAL media start (the keyframe ffmpeg landed on, which hls.js zero-bases to), not the
# requested offset — the client's absolute clock and the subtitle cue-shift both key off this.
# `transcode`/`hw`/`tonemap` let the player show an honest cold-start ("Preparing stream…") and
# surface the active mode + encoder in the stats overlay (`tonemap` = the HW HDR→SDR path); both
# segment streams are otherwise identical.
return {
"mode": "hls",
"start_s": s.media_start_s,
"url": f"/api/plex/stream/{it.rating_key}/hls/{s.entry}",
"transcode": s.transcode,
"hw": s.hw,
"tonemap": s.tonemap,
}
+18 -2
View File
@@ -122,9 +122,25 @@ def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
raise HTTPException(status_code=404, detail="This video is no longer available.")
filename = storage.download_filename(asset.title or "video", asset.container, path)
# The served Content-Type comes from the asset's ffprobe-verified media_kind + a container
# allow-list (storage.served_media_type) — NEVER guessed from the file's own extension. A generic
# 'file' (and any media whose container we don't positively recognise, e.g. a media/HTML polyglot
# named "evil.html") returns None → a neutral force-download attachment, so arbitrary uploaded
# HTML/SVG can't execute inline on this login-free public origin. A recognised media container is
# served with its real audio/video type, inline unless the link forces a download so it can play
# in the watch page. Both are range-aware (FileResponse → 206) for seeking/resume.
media_type = storage.served_media_type(asset.media_kind, asset.container)
if media_type is None:
return FileResponse(
path,
filename=filename,
media_type="application/octet-stream",
content_disposition_type="attachment",
)
disposition = "attachment" if link.allow_download else "inline"
# Starlette's FileResponse honours the Range header (206) for seeking/scrubbing.
return FileResponse(path, filename=filename, content_disposition_type=disposition)
return FileResponse(
path, filename=filename, media_type=media_type, content_disposition_type=disposition
)
@router.get("/watch/{token}/poster.jpg")
+12
View File
@@ -8,10 +8,22 @@ from app.auth import current_user
from app.db import get_db
from app.models import ChannelTag, Tag, User
from app.routes._common import owned_or_404
from app.sync.tagsuggest import compute_tag_suggestions
router = APIRouter(prefix="/api/tags", tags=["tags"])
@router.get("/suggestions")
def tag_suggestions(
user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict[str, list[dict]]:
"""Suggested user tags for this user's still-untagged subscribed channels, learned from the
auto-tag pattern of the channels they HAVE tagged (see app.sync.tagsuggest). Read-only — the
Channel Manager shows these as one-tap-accept chips; accepting posts to the normal attach-tag
endpoint. Keyed by channel id; channels with no suggestion are omitted."""
return compute_tag_suggestions(db, user)
@router.get("")
def list_tags(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
rows = db.execute(
+240 -71
View File
@@ -1,14 +1,16 @@
"""YouTube playlist sync, both directions. Read: mirror each user's own YouTube playlists
into local `source='youtube'` playlists (kept fresh by the scheduler). Write-back: push local
edits out to YouTube (`plan_push` / `push_playlist`), behind the optional write scope.
into local `source='youtube'` playlists (kept fresh by the scheduler). Write-back: two-way sync
local edits with YouTube (`plan_merge` / `merge_playlist`), behind the optional write scope.
YouTube's special Watch Later / History playlists are not exposed by the Data API and are
never synced. Videos in a playlist that aren't in our shared catalog yet are fetched and
ingested (with a stub channel) so the mirror is faithful."""
import hashlib
import json
import logging
from collections.abc import Callable
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import TypeVar
@@ -42,6 +44,57 @@ def current_fingerprint(db: Session, pl: Playlist) -> str:
return fingerprint(pl.name, _local_video_ids(db, pl))
def _dump_ids(ids: list[str]) -> str:
"""Serialize the ordered id list stored as the 3-way-merge baseline (`synced_video_ids`)."""
return json.dumps(ids)
def _load_baseline(pl: Playlist) -> tuple[list[str] | None, set[str]]:
"""Parse `synced_video_ids` into (ordered id set, un-ingestable id set). Two on-disk forms:
a plain JSON list (legacy / no un-ingestable ids) or `{"ids": [...], "unavailable": [...]}`.
`unavailable` = YouTube-only videos that exist on YouTube but the Data API won't return
(private/deleted/region-locked), so they never became local rows — tracked so the merge can
skip them instead of forever re-classifying them as a YouTube-add (they'd never converge)."""
if not pl.synced_video_ids:
return None, set()
try:
val = json.loads(pl.synced_video_ids)
except (ValueError, TypeError):
return None, set()
if isinstance(val, list):
return [str(v) for v in val], set()
if isinstance(val, dict):
raw_ids = val.get("ids")
ids = [str(v) for v in raw_ids] if isinstance(raw_ids, list) else None
raw_un = val.get("unavailable")
un = {str(v) for v in raw_un} if isinstance(raw_un, list) else set()
return ids, un
return None, set()
def parse_baseline(pl: Playlist) -> list[str] | None:
"""The last-synced ordered id set, or None when no baseline has been recorded yet (a fresh
export, or a link created before S7). None makes the first merge union losslessly."""
return _load_baseline(pl)[0]
def parse_unavailable(pl: Playlist) -> set[str]:
"""The last-synced set of un-ingestable-but-live YouTube ids (see `_load_baseline`)."""
return _load_baseline(pl)[1]
def _set_baseline(pl: Playlist, ids: list[str], unavailable: list[str] | None = None) -> None:
"""Record `ids` as both the hash (drives `dirty`, over the local ids only) and the raw ordered
set (drives merge). `unavailable` = un-ingestable-but-live YouTube ids to carry forward so the
next merge skips them; stored only when non-empty (keeps the common case a plain list)."""
pl.synced_fingerprint = fingerprint(pl.name, ids)
if unavailable:
pl.synced_video_ids = json.dumps({"ids": ids, "unavailable": list(unavailable)})
else:
pl.synced_video_ids = _dump_ids(ids)
pl.dirty = False
def recompute_dirty(db: Session, pl: Playlist) -> None:
"""Set `dirty` by comparing the current state to the synced baseline. Unlinked lists
(and Watch later) are never dirty; a missing baseline counts as dirty (unknown YT state)."""
@@ -116,12 +169,18 @@ def _replace_items_from_youtube(
if v in present and v not in seen:
seen.add(v)
ordered.append(v)
_set_local_items(db, pl, ordered)
# After a pull/reset local == YouTube, so this ordered set IS the new merge baseline.
_set_baseline(pl, ordered)
return ordered
def _set_local_items(db: Session, pl: Playlist, ordered: list[str]) -> None:
"""Authoritatively replace `pl`'s local items with `ordered` (positions 0..n). The caller
is responsible for having ensured every id exists in the catalog."""
db.execute(delete(PlaylistItem).where(PlaylistItem.playlist_id == pl.id))
for pos, vid in enumerate(ordered):
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=pos))
pl.synced_fingerprint = fingerprint(pl.name, ordered)
pl.dirty = False
return ordered
def sync_user_playlists(db: Session, user: User) -> dict:
@@ -216,22 +275,91 @@ def _reorder_moves(current: list[str], desired: list[str]) -> int:
return moves
def plan_push(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Compute (without writing) what pushing this local playlist to YouTube would do:
create vs update, how many inserts/deletes/reorders, the quota estimate, and whether
YouTube has diverged (items present there that local will remove). Costs read units
(1/page) for a linked playlist; nothing for a fresh export. `live` = a pre-fetched
(current_items, snippet) tuple so the caller can share one read with push_playlist."""
desired = _local_video_ids(db, pl)
@dataclass
class MergePlan:
"""The 3-way merge of a linked playlist: what each side gains and loses so neither side
loses an item, plus the merged order. `final` is the id set both sides end up with."""
final: list[str] # merged order (local order, YouTube-only adds appended)
add_to_yt: list[str] # local-adds to push out to YouTube
add_to_local: list[str] # YouTube-adds to pull into Siftlode
remove_from_yt: list[str] # items deleted locally that are still on YouTube
remove_from_local: list[str] # items deleted on YouTube that are still local
reorders: int # estimated YouTube playlistItems.update moves to reach `final`
baseline_known: bool
def merge_plan(
baseline: list[str] | None,
local: list[str],
youtube: list[str],
unavailable: set[str] | None = None,
) -> MergePlan:
"""Pure 3-way merge. Classify each id against the last-synced `baseline`:
- in local but not youtube, not deleted → add to YouTube (a local-add)
- in youtube but not local, not deleted → add to Siftlode (a YouTube-add)
- in baseline but gone from local → a genuine local delete → propagate to YouTube
- in baseline but gone from youtube → a genuine YouTube delete → propagate to Siftlode
With NO baseline (first-ever merge) we can't tell an add from a delete, so we UNION both
sides (never delete) — the lossless default. Merged order: local order first, YouTube-only
additions appended in their YouTube order.
`unavailable` = known un-ingestable-but-live YouTube ids: dropped from the YouTube side up
front so they're never re-classified as a YouTube-add (they can't ingest) nor as a local
delete (they were never local) — otherwise such a video would make the playlist never
converge. They stay in place on YouTube, untouched by this merge."""
un = set(unavailable or ())
L = list(dict.fromkeys(local))
Y = [v for v in dict.fromkeys(youtube) if v not in un]
Lset, Yset = set(L), set(Y)
if baseline is None:
add_to_yt = [v for v in L if v not in Yset]
add_to_local = [v for v in Y if v not in Lset]
remove_from_yt: list[str] = []
remove_from_local: list[str] = []
final = L + [v for v in Y if v not in Lset]
else:
B = set(baseline)
deleted_local = B - Lset # was synced, removed on our side
deleted_yt = B - Yset # was synced, removed on YouTube's side
gone = deleted_local | deleted_yt
remove_from_yt = [v for v in Y if v in deleted_local]
remove_from_local = [v for v in L if v in deleted_yt]
add_to_yt = [v for v in L if v not in Yset and v not in gone]
add_to_local = [v for v in Y if v not in Lset and v not in gone]
final = [v for v in L if v not in gone] + [v for v in Y if v not in Lset and v not in gone]
final_set = set(final)
# On YouTube, after membership reconciliation the order is current-kept + inserts appended;
# the reorder pass then moves items into `final`.
after_membership = [v for v in Y if v in final_set] + add_to_yt
reorders = _reorder_moves(after_membership, final)
return MergePlan(
final=final,
add_to_yt=add_to_yt,
add_to_local=add_to_local,
remove_from_yt=remove_from_yt,
remove_from_local=remove_from_local,
reorders=reorders,
baseline_known=baseline is not None,
)
def plan_merge(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Compute (without writing) what a two-way sync of this playlist would do: create vs
update, how many items each side gains/loses, reorders, and the quota estimate (YouTube
writes only — local edits are free). Reads live YouTube state (1 unit/page) for a linked
playlist; nothing for a fresh export. `live` = a pre-fetched (current_items, snippet) tuple
so the caller can share one read with merge_playlist."""
local = _local_video_ids(db, pl)
if not pl.yt_playlist_id:
ops = 1 + len(desired) # create the playlist + one insert per video
ops = 1 + len(local) # create the playlist + one insert per video
return {
"action": "create",
"to_insert": len(desired),
"to_delete": 0,
"add_yt": len(local),
"add_local": 0,
"remove_yt": 0,
"remove_local": 0,
"to_reorder": 0,
"rename": False,
"yt_extra": 0,
"units_estimate": ops * WRITE_UNIT_COST,
}
if live is not None:
@@ -241,23 +369,17 @@ def plan_push(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
current = list(yt.iter_playlist_items_with_ids(pl.yt_playlist_id))
snippet = yt.get_playlist_snippet(pl.yt_playlist_id)
rename = bool(snippet) and (snippet.get("title") or "") != pl.name
cur_vids = [it["video_id"] for it in current]
cur_set = set(cur_vids)
desired_set = set(desired)
to_insert = [v for v in desired if v not in cur_set]
to_delete = [v for v in cur_vids if v not in desired_set]
# After membership reconciliation, the YT order is current-minus-deletes plus inserts
# appended; the reorder pass then moves items into the desired order.
after_membership = [v for v in cur_vids if v in desired_set] + to_insert
reorders = _reorder_moves(after_membership, desired)
ops = len(to_insert) + len(to_delete) + reorders + (1 if rename else 0)
yt_ids = [it["video_id"] for it in current]
mp = merge_plan(parse_baseline(pl), local, yt_ids, parse_unavailable(pl))
ops = len(mp.add_to_yt) + len(mp.remove_from_yt) + mp.reorders + (1 if rename else 0)
return {
"action": "update",
"to_insert": len(to_insert),
"to_delete": len(to_delete),
"to_reorder": reorders,
"add_yt": len(mp.add_to_yt),
"add_local": len(mp.add_to_local),
"remove_yt": len(mp.remove_from_yt),
"remove_local": len(mp.remove_from_local),
"to_reorder": mp.reorders,
"rename": rename,
"yt_extra": len(to_delete),
"units_estimate": ops * WRITE_UNIT_COST,
}
@@ -327,13 +449,16 @@ class _Breaker:
return out
def push_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Push a local playlist to YouTube: create it if needed, then reconcile membership
and order so YouTube matches local (local wins). Marks the playlist clean on success.
def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Two-way sync a local playlist with YouTube: create it on first push, else 3-way MERGE
both sides so an item added (or deleted) on either side survives — no side wins wholesale.
Local edits are applied directly; YouTube edits go through the failure breaker. On full
success the merged set becomes the new baseline and the playlist is marked clean; on any
YouTube failure the baseline is left untouched so the next sync retries and converges.
Caller must have checked the write scope and the quota budget. `live` = a pre-fetched
(current_items, snippet) tuple (from plan_push's read) to avoid a second live page-through."""
desired = _local_video_ids(db, pl)
inserted = deleted = reordered = created = 0
(current_items, snippet) tuple (from plan_merge's read) to avoid a second page-through."""
local = _local_video_ids(db, pl)
inserted = deleted = reordered = created = pulled = removed_local = 0
failures: list[str] = []
breaker = _Breaker()
with YouTubeClient(db, user) as yt:
@@ -341,8 +466,8 @@ def push_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
pl.yt_playlist_id = yt.create_playlist(pl.name)
db.commit()
created = 1
# Brand-new playlist: just append every video in order.
for vid in desired:
# Brand-new playlist: just append every local video in order.
for vid in local:
added = breaker.run(
lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
lambda v=vid: failures.append(v),
@@ -350,60 +475,95 @@ def push_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
if not isinstance(added, _Failed):
inserted += 1
if not failures:
pl.synced_fingerprint = fingerprint(pl.name, desired)
pl.dirty = False
_set_baseline(pl, local)
db.commit()
return {
"created": created,
"inserted": inserted,
"deleted": deleted,
"reordered": reordered,
"pulled": pulled,
"removed_local": removed_local,
"failures": failures,
"aborted": breaker.tripped,
"yt_playlist_id": pl.yt_playlist_id,
}
# Existing link: diff against the live YouTube state (reuse plan_push's read if given).
# Existing link: 3-way merge against the live YouTube state (reuse plan_merge's read).
if live is not None:
current, snippet = live
else:
snippet = yt.get_playlist_snippet(pl.yt_playlist_id)
current = list(yt.iter_playlist_items_with_ids(pl.yt_playlist_id))
item_id_by_vid = {it["video_id"]: it["item_id"] for it in current}
yt_ids = [it["video_id"] for it in current]
# Re-attempt ingestion of ids previously marked un-ingestable that are still on YouTube: a
# video that has since become available (e.g. private→public) should re-enter the merge, not
# stay skipped forever. Those that still won't ingest remain filtered (`effective_unavailable`).
prev_unavailable = parse_unavailable(pl)
still_listed = [v for v in yt_ids if v in prev_unavailable]
if still_listed:
_ensure_videos(db, yt, still_listed)
healed = set(
db.execute(select(Video.id).where(Video.id.in_(still_listed))).scalars().all()
)
effective_unavailable = prev_unavailable - healed
else:
effective_unavailable = prev_unavailable
mp = merge_plan(parse_baseline(pl), local, yt_ids, effective_unavailable)
# --- Siftlode side (free, direct): pull YouTube-adds in, drop YouTube-deletes, land the
# merged order authoritatively. Do this first so local reflects the merge even if a later
# YouTube write trips the breaker.
_ensure_videos(db, yt, mp.add_to_local)
# Only the ids that actually exist in the catalog can become local rows (a private/deleted
# YouTube video won't ingest); keep the merged order, dropping any that didn't materialize.
present = (
set(db.execute(select(Video.id).where(Video.id.in_(mp.final))).scalars().all())
if mp.final
else set()
)
final_local = [v for v in mp.final if v in present]
pulled = len([v for v in mp.add_to_local if v in present])
removed_local = len(mp.remove_from_local)
_set_local_items(db, pl, final_local)
db.commit()
# --- YouTube side (metered, breaker-guarded).
if snippet is not None and (snippet.get("title") or "") != pl.name:
try:
yt.update_playlist_title(pl.yt_playlist_id, pl.name)
except YouTubeError:
failures.append("title")
item_id_by_vid = {it["video_id"]: it["item_id"] for it in current}
cur_vids = [it["video_id"] for it in current]
desired_set = set(desired)
# Delete extras (local is authoritative).
for it in current:
if it["video_id"] not in desired_set:
dropped = breaker.run(
lambda i=it: yt.delete_playlist_item(i["item_id"]),
lambda i=it: failures.append(i["video_id"]),
)
if not isinstance(dropped, _Failed):
deleted += 1
# Insert missing (append; the reorder pass below fixes positions).
cur_set = set(cur_vids)
for vid in desired:
if vid not in cur_set:
item_id = breaker.run(
lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
lambda v=vid: failures.append(v),
)
if not isinstance(item_id, _Failed):
item_id_by_vid[vid] = item_id
inserted += 1
# Reorder to match `desired` via insertion-sort moves over a local model.
model = [v for v in cur_vids if v in desired_set] + [
v for v in desired if v not in cur_set and v in item_id_by_vid
# Propagate genuine local deletes.
for vid in mp.remove_from_yt:
item_id = item_id_by_vid.get(vid)
if item_id is None:
continue
dropped = breaker.run(
lambda i=item_id: yt.delete_playlist_item(i),
lambda v=vid: failures.append(v),
)
if not isinstance(dropped, _Failed):
deleted += 1
item_id_by_vid.pop(vid, None)
# Insert local-adds (append; the reorder pass below fixes positions).
for vid in mp.add_to_yt:
item_id = breaker.run(
lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
lambda v=vid: failures.append(v),
)
if not isinstance(item_id, _Failed):
item_id_by_vid[vid] = item_id
inserted += 1
# Reorder YouTube to match `final` via insertion-sort moves over a local model.
final_set = set(mp.final)
model = [v for v in yt_ids if v in final_set and v in item_id_by_vid] + [
v for v in mp.add_to_yt if v in item_id_by_vid
]
for i, want in enumerate(desired):
for i, want in enumerate(mp.final):
if want not in item_id_by_vid:
continue # failed insert; skip
continue # deleted-from-YT or failed insert; skip
if i < len(model) and model[i] == want:
continue
moved = breaker.run(
@@ -419,14 +579,23 @@ def push_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
model.pop(j)
model.insert(i, want)
if not failures:
pl.synced_fingerprint = fingerprint(pl.name, desired)
pl.dirty = False
# Carry forward un-ingestable-but-live YouTube ids so the next merge skips them
# (see merge_plan): this run's failed pulls, plus prior ones still on YouTube, minus
# any that have since left YouTube. Recomputed each success → self-healing.
yt_id_set = set(yt_ids)
unavailable = sorted(
(effective_unavailable | {v for v in mp.add_to_local if v not in present})
& yt_id_set
)
_set_baseline(pl, final_local, unavailable)
db.commit()
return {
"created": created,
"inserted": inserted,
"deleted": deleted,
"reordered": reordered,
"pulled": pulled,
"removed_local": removed_local,
"failures": failures,
"aborted": breaker.tripped,
"yt_playlist_id": pl.yt_playlist_id,
+131
View File
@@ -0,0 +1,131 @@
"""Suggest a user's own manual tags for channels they haven't tagged yet.
The signal is the auto-tagger's *topic* output: each channel carries ``source='auto'`` topic
ChannelTags (see ``autotag.py``). For every user tag T the user has applied to enough channels,
we learn which topics are *characteristic* of T (present in a high fraction of the channels the
user tagged T), then suggest T for any still-untagged channel whose own topics cover enough of
that profile.
Language auto-tags are deliberately EXCLUDED from the signal: they are near-universal within a
language ("English" is on most channels), so they don't discriminate — including them floods the
result (measured on real data: every untagged channel got a suggestion, most on a shared language
alone). Topic overlap is what makes "you tagged similar channels this way" meaningful.
Purely read-only and deterministic — the Channel Manager surfaces the result as one-tap-accept
"ghost" chips; nothing is written here (D4 = suggest-and-confirm, no silent auto-apply). The
thresholds are tuning constants rather than admin knobs on purpose: they mirror how ``autotag.py``
keeps its mappings inline, and sysconfig only stores int/str/bool (no float), so a knob would mean
storing percentages — not worth the surface for values an operator has no reason to tune per-install.
"""
from collections import Counter
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.models import ChannelTag, Subscription, Tag, User
# A user tag needs at least this many manually-tagged example channels before we trust its
# profile — fewer is noise (one gaming-cooking channel shouldn't make "cooking" mean "gaming").
MIN_EXAMPLES = 3
# A topic counts as *characteristic* of a user tag when it appears in ≥60% of that tag's example
# channels. High enough that an incidental shared topic (one channel's stray category) doesn't
# enter the profile, low enough to survive a couple of outliers in a small example set.
CHAR_SUPPORT = 0.6
# Suggest tag T for channel C when C's topics cover ≥60% of T's characteristic topics. The
# intersection-over-profile shape means a partial match scores proportionally: 1 topic of a
# 2-topic profile = 0.5 (dropped), 2 of 3 = 0.67 (kept). Tuned against real data to surface
# strong topical matches without flooding — 0.5 let a single shared topic carry a suggestion.
SUGGEST_THRESHOLD = 0.6
# A suggestion at/above this score is "high confidence" — a full-profile topic match — and is the
# only kind the UI's "Accept all confident" action touches. Never applied silently (see docstring).
AUTOAPPLY_THRESHOLD = 0.9
# Cap suggestions per channel so a busy channel's chip row stays legible.
MAX_PER_CHANNEL = 3
def compute_tag_suggestions(db: Session, user: User) -> dict[str, list[dict]]:
"""Map each of the user's still-untagged subscribed channels to suggested user tags.
Returns ``{channel_id: [{"tag_id": int, "confidence": float}, ...]}`` — ordered best-first,
capped at ``MAX_PER_CHANNEL``. Channels with no suggestion are omitted (empty dict when the
user has no tag with a usable profile). Deterministic for a fixed DB state.
"""
# Candidate universe: the channels this user is subscribed to. `subscribed` (incl. hidden) is the
# SIGNAL set — a hidden channel is still a valid example / topic-feature source, since hiding only
# affects the feed, not the tagging pattern. `hidden` is excluded as a suggestion TARGET below (the
# Channel Manager shows ghost chips only on non-hidden rows, so suggesting for hidden channels would
# inflate the header count past what's shown and let bulk-accept tag channels the user can't see).
subs = db.execute(
select(Subscription.channel_id, Subscription.hidden).where(Subscription.user_id == user.id)
).all()
subscribed: set[str] = {cid for cid, _ in subs}
hidden: set[str] = {cid for cid, h in subs if h}
if not subscribed:
return {}
# The user's manual tag links: tag_id -> example channels, and each channel's user-tag set (so
# a channel with ANY user tag is excluded from "untagged"). Intersect with `subscribed`: unsubscribe
# deletes the Subscription but NOT the source='user' ChannelTag, so a tagged-then-unsubscribed
# channel would otherwise stay an "example" — inflating a profile's CHAR_SUPPORT denominator while
# contributing zero features (features are scanned over `subscribed` only), dropping characteristic
# topics. Keeping examples ⊆ subscribed makes the support fraction honest.
examples: dict[int, set[str]] = {}
user_tagged: set[str] = set()
for tag_id, cid in db.execute(
select(ChannelTag.tag_id, ChannelTag.channel_id).where(
ChannelTag.user_id == user.id, ChannelTag.source == "user"
)
).all():
if cid not in subscribed:
continue
examples.setdefault(tag_id, set()).add(cid)
user_tagged.add(cid)
# Topic features per channel: the auto TOPIC tags on the candidate channels. Language auto-tags
# are joined out (Tag.category) — they don't discriminate (see the module docstring).
features: dict[str, set[int]] = {}
for cid, tid in db.execute(
select(ChannelTag.channel_id, ChannelTag.tag_id)
.join(Tag, Tag.id == ChannelTag.tag_id)
.where(
ChannelTag.user_id.is_(None),
ChannelTag.source == "auto",
ChannelTag.channel_id.in_(subscribed),
Tag.category == "topic",
)
).all():
features.setdefault(cid, set()).add(tid)
# Characteristic-feature profile per user tag with enough examples.
profiles: dict[int, set[int]] = {}
for tag_id, ex in examples.items():
if len(ex) < MIN_EXAMPLES:
continue
counts: Counter[int] = Counter()
for cid in ex:
counts.update(features.get(cid, ()))
threshold = CHAR_SUPPORT * len(ex)
chars = {f for f, c in counts.items() if c >= threshold}
if chars:
profiles[tag_id] = chars
if not profiles:
return {}
suggestions: dict[str, list[dict]] = {}
for cid in subscribed:
if cid in hidden or cid in user_tagged: # hidden → no chips shown; tagged → not "untagged"
continue
cfeat = features.get(cid)
if not cfeat:
continue
scored = [(tag_id, len(cfeat & chars) / len(chars)) for tag_id, chars in profiles.items()]
hits = [(tid, s) for tid, s in scored if s >= SUGGEST_THRESHOLD]
if not hits:
continue
# Best score first; tag_id as a stable tie-break so the order is deterministic.
hits.sort(key=lambda x: (-x[1], x[0]))
suggestions[cid] = [
{"tag_id": tid, "confidence": round(s, 3)} for tid, s in hits[:MAX_PER_CHANNEL]
]
return suggestions
+12
View File
@@ -127,6 +127,10 @@ SPECS: tuple[ConfigSpec, ...] = (
min=1,
max=100_000,
),
# min=1, NOT 0: the per-file cap is enforced as `if cap and size > cap`, so a stored 0 would read
# as falsy and SILENTLY disable the limit (an unbounded streamed write to the volume). Floor it at
# 1 so the cap is always active; the overall footprint is separately bounded by the byte quota.
ConfigSpec("download_upload_max_bytes", "int", "downloads", "download_upload_max_bytes", min=1),
ConfigSpec(
"download_retention_days", "int", "downloads", "download_retention_days", min=0, max=3_650
),
@@ -155,6 +159,14 @@ SPECS: tuple[ConfigSpec, ...] = (
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
# (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
# Transcode re-encode tuning (E6 S1). hw = "auto"|"on"|"off" (free-text, like search_source); the
# libx264 knobs apply only to the software fallback. crf 0–51 (x264 range); max_height 0 = source.
ConfigSpec("plex_transcode_hw", "str", "plex", "plex_transcode_hw"),
ConfigSpec("plex_transcode_preset", "str", "plex", "plex_transcode_preset"),
ConfigSpec("plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51),
ConfigSpec(
"plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
),
# NOTE: plex_watch_sync_interval_min is a config.py default read by the watch-sync scheduler job
# (scheduler.py); it is intentionally NOT a runtime admin knob, so it stays out of SPECS.
# --- TMDB enrichment (E5 S2; optional) — the API key is the only secret ---
+1
View File
@@ -43,6 +43,7 @@ pydantic-settings==2.15.0
pydantic==2.13.4
pydantic_core==2.46.4
python-dotenv==1.2.3
python-multipart==0.0.32
requests==2.34.2
starlette==1.6.0
typing-inspection==0.4.4
+3
View File
@@ -1,4 +1,7 @@
fastapi>=0.115,<1.0
# Multipart form parsing — required by Starlette/FastAPI to accept file uploads (UploadFile/Form).
# The download center's local-file Upload (E7) is the only multipart endpoint in the app.
python-multipart>=0.0.9,<1.0
uvicorn[standard]>=0.30,<1.0
sqlalchemy>=2.0,<2.1
psycopg[binary]>=3.2,<4.0
+21
View File
@@ -189,3 +189,24 @@ def as_user(db, override_dep, _configured):
return TestClient(app, headers={"x-test-user": str(user.id)})
return _as
@pytest.fixture(autouse=True)
def _reset_rate_limiters():
"""Every RateLimiter is a module-level singleton keyed by client IP, and TestClient always
presents the same host — so without this the whole suite's login/register attempts pool into one
300s window and, past the cap, some unrelated test starts failing with a 429 depending on order.
Cleared before each test; a test that wants to exercise throttling can still exhaust one itself.
Sweeps every loaded `app.*` module rather than a hand-kept list, so a limiter added in a new
module is covered the day it lands instead of silently pooling until something fails."""
import sys
from app.ratelimit import RateLimiter
for name, mod in list(sys.modules.items()):
if name != "app" and not name.startswith("app."):
continue
for obj in vars(mod).values():
if isinstance(obj, RateLimiter):
obj.reset()
+170
View File
@@ -0,0 +1,170 @@
"""Deleting a whitelist / access-request entry (admin).
Deny records an auditable decision and keeps the row; delete forgets the address outright, for an
approval added by mistake or one that is no longer wanted. Neither stops the person asking again —
`upsert_pending_invite` reopens a denied row as pending, and a deleted one comes back as a fresh
request. The one hard rule is that an entry backed by an account is NOT deletable here: this row is
that account's approval gate, and orphaning it would strand someone who could then neither sign in
nor be approved.
"""
from sqlalchemy import select
from app.models import Invite, User
def _invites(client):
r = client.get("/api/admin/invites")
assert r.status_code == 200
return {i["email"]: i for i in r.json()}
def test_admin_deletes_a_whitelist_entry_that_never_became_an_account(db, make_user, as_user):
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="mistake@example.com", status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
listed = _invites(client)["mistake@example.com"]
assert listed["has_account"] is False # drives the delete affordance in the UI
r = client.delete(f"/api/admin/invites/{listed['id']}")
assert r.status_code == 200 and r.json()["deleted"] == listed["id"]
db.expire_all()
assert (
db.execute(select(Invite).where(Invite.email == "mistake@example.com")).scalar_one_or_none()
is None
)
def test_deleting_an_entry_that_has_an_account_is_refused(db, make_user, as_user):
# The guard that matters: this row is what is_allowed() reads, so deleting it would revoke the
# user's access without touching the account or saying so.
admin = make_user(email="admin@example.com", role="admin")
member = make_user(email="member@example.com")
db.add(Invite(email=member.email, status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
listed = _invites(client)[member.email]
assert listed["has_account"] is True
r = client.delete(f"/api/admin/invites/{listed['id']}")
assert r.status_code == 409
db.expire_all()
assert (
db.execute(select(Invite).where(Invite.email == member.email)).scalar_one_or_none()
is not None
)
def test_a_pending_request_from_a_registration_is_not_deletable(db, make_user, as_user):
# The shape the ordinary sign-up produces: _register_account writes the User (inactive) AND the
# pending Invite together. The invite is that account's approval gate, so deleting it would
# strand the person — they could neither sign in nor be approved, and re-registering is a silent
# no-op. Managed from Users & roles instead, which is what the UI and the 409 both say.
admin = make_user(email="admin@example.com", role="admin")
make_user(email="signup@example.com", is_active=False, email_verified=False)
db.add(Invite(email="signup@example.com", status="pending"))
db.commit()
client = as_user(admin)
listed = _invites(client)["signup@example.com"]
assert listed["has_account"] is True # the UI shows a disabled control, not a missing one
r = client.delete(f"/api/admin/invites/{listed['id']}")
assert r.status_code == 409
db.expire_all()
assert (
db.execute(select(Invite).where(Invite.email == "signup@example.com")).scalar_one_or_none()
is not None
)
def test_deleting_an_env_allow_listed_address_reports_that_it_kept_access(
db, make_user, as_user, monkeypatch
):
# Removing the row is NOT always a revoke: is_allowed falls back to the env bootstrap lists once
# no invite exists. The admin has to be told, or they think they removed access and did not.
from app.config import settings
# Patch the raw env field, not the derived property: that keeps the split/strip/lower parsing
# under test, and matches how the rest of the suite stubs settings.
monkeypatch.setattr(settings, "allowed_emails", " OPS@Example.com , someone@else.test ")
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="ops@example.com", status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
listed = _invites(client)["ops@example.com"]
# The LIST carries it too, so the confirm dialog can warn before the delete happens.
assert listed["still_allowed_by_env"] is True
r = client.delete(f"/api/admin/invites/{listed['id']}")
assert r.status_code == 200
assert r.json()["still_allowed_by_env"] is True
def test_an_admin_emails_address_is_also_reported(db, make_user, as_user, monkeypatch):
# ADMIN_EMAILS is the sharper case: deleting the row not only keeps access, the next Google
# sign-in re-promotes the address to admin.
from app.config import settings
monkeypatch.setattr(settings, "admin_emails", "boss@example.com")
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="boss@example.com", status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
assert _invites(client)["boss@example.com"]["still_allowed_by_env"] is True
def test_deleting_a_pending_request_forgets_it(db, make_user, as_user):
# A pending row is deletable too — "drop this without recording a decision" is a different
# choice from Deny, which stores a decision (status, decided_by) and mails the requester.
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="asker@example.com", status="pending"))
db.commit()
client = as_user(admin)
r = client.delete(f"/api/admin/invites/{_invites(client)['asker@example.com']['id']}")
assert r.status_code == 200
assert r.json()["still_allowed_by_env"] is False # nothing in the env allow-lists
assert "asker@example.com" not in _invites(client)
def test_deleting_a_missing_invite_is_404(db, make_user, as_user):
admin = make_user(email="admin@example.com", role="admin")
assert as_user(admin).delete("/api/admin/invites/9999").status_code == 404
def test_a_regular_user_cannot_delete_an_invite(db, make_user, as_user):
make_user(email="admin@example.com", role="admin")
plain = make_user(email="plain@example.com")
db.add(Invite(email="mistake@example.com", status="approved"))
db.commit()
inv_id = db.execute(select(Invite.id).where(Invite.email == "mistake@example.com")).scalar_one()
assert as_user(plain).delete(f"/api/admin/invites/{inv_id}").status_code == 403
db.expire_all()
assert db.get(Invite, inv_id) is not None
def test_deleting_the_user_also_clears_the_way_to_re_invite(db, make_user, as_user):
# The refusal above tells the admin to delete the account instead; purge_user takes the invite
# with it, so the address really is forgotten and a fresh request starts over as pending.
admin = make_user(email="admin@example.com", role="admin")
member = make_user(email="member@example.com")
db.add(Invite(email=member.email, status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
assert client.delete(f"/api/admin/users/{member.id}").status_code == 200
db.expire_all()
assert db.execute(select(User).where(User.email == member.email)).scalar_one_or_none() is None
assert (
db.execute(select(Invite).where(Invite.email == member.email)).scalar_one_or_none() is None
)
+345
View File
@@ -127,3 +127,348 @@ def test_activating_a_verified_account_keeps_its_password(db, make_user):
_activate_user_by_email(db, "legit@example.com")
assert user.is_active is True
assert user.password_hash is not None
def _register_and_fetch(db, email: str):
"""Run the /register background worker, then read the users/invites rows back through a FRESH
query. The worker owns its own session, so anything it failed to commit is simply not there."""
from sqlalchemy import select as _select
from app.models import Invite, User
from app.routes.auth import _register_account
_register_account(email, _PW)
db.expire_all()
user = db.execute(_select(User).where(User.email == email)).scalar_one_or_none()
invite = db.execute(_select(Invite).where(Invite.email == email)).scalar_one_or_none()
return user, invite
def _no_smtp(monkeypatch):
"""Put the worker on the no-SMTP path and capture the admin notice instead of sending it."""
from app.routes import auth as auth_mod
notified: list[str] = []
monkeypatch.setattr(auth_mod.email_mod, "email_enabled", lambda: False)
monkeypatch.setattr(
auth_mod.email_mod, "send_admin_new_request", lambda admins, to: notified.append(to) or True
)
return notified
def test_registration_of_a_pre_approved_email_persists_the_account_without_smtp(
db, make_user, monkeypatch
):
# THE regression. An admin whitelists an address, the person registers, and the account silently
# never existed: upsert_pending_invite leaves an already-approved row alone and returns WITHOUT
# committing, and the only other commit on this path (_issue_token) is gated on SMTP — so with no
# working mail the flushed user was rolled back at session close. The sign-up answered "ok",
# created nothing, and re-registering did the same again.
from app.models import Invite
notified = _no_smtp(monkeypatch)
make_user(email="admin@example.com", role="admin")
db.add(Invite(email="roland@example.com", status="approved", decided_by="admin@example.com"))
db.commit()
user, invite = _register_and_fetch(db, "roland@example.com")
assert user is not None, "pre-approved registration was rolled back — the account is gone"
assert user.password_hash is not None
assert user.email_verified is True # no SMTP => ownership can't be proven by link
assert (
user.is_active is False
) # approval is bound to the account by an admin, not by signing up
# The admin's standing decision is left exactly as it was — is_allowed() still reads "approved",
# so their Google sign-in keeps working and an anonymous sign-up cannot revoke access.
assert invite.status == "approved"
assert invite.decided_by == "admin@example.com"
# The notify path still fires for an already-approved address — nothing lands in the queue, so
# it is the only announcement there is. (Whether it is DELIVERED is separate: on a real no-SMTP
# instance `_send` drops it, which is the open half of the problem, roadmap UX-25.)
assert notified == ["roland@example.com"]
def test_registration_with_an_already_pending_request_persists_the_account(
db, make_user, monkeypatch
):
# The same rollback, reached the other way: someone uses Request access, then registers before
# the admin has decided. upsert_pending_invite leaves the pending row alone and does not commit.
from app.models import Invite
notified = _no_smtp(monkeypatch)
make_user(email="admin@example.com", role="admin")
db.add(Invite(email="pia@example.com", status="pending"))
db.commit()
user, invite = _register_and_fetch(db, "pia@example.com")
assert user is not None, "registration was rolled back — the account is gone"
assert user.is_active is False
assert invite.status == "pending"
assert notified == ["pia@example.com"]
def test_an_admin_can_activate_a_pre_approved_registrant(db, monkeypatch):
# The recovery path for the account above, pinned so it cannot quietly break: re-entering the
# address in the Access-requests whitelist box (POST /api/admin/invites) runs
# _activate_user_by_email, which is also what keeps the SEC-10 preset-password guard in play.
from app.models import Invite
from app.routes.admin import _activate_user_by_email
_no_smtp(monkeypatch)
db.add(Invite(email="roland@example.com", status="approved", decided_by="admin@example.com"))
db.commit()
user, _ = _register_and_fetch(db, "roland@example.com")
_activate_user_by_email(db, "roland@example.com")
db.expire_all()
assert user.is_active is True
assert user.password_hash is not None # a verified account keeps its own password
def test_registration_of_an_unknown_email_still_persists_account_and_pending_invite(
db, make_user, monkeypatch
):
# Guard, not a regression test: this branch always worked (upsert INSERTs the invite and commits
# the same transaction). It must keep working — and it is the case that DOES notify the admin.
notified = _no_smtp(monkeypatch)
make_user(email="admin@example.com", role="admin")
user, invite = _register_and_fetch(db, "stranger@example.com")
assert user is not None
assert user.is_active is False
assert invite is not None and invite.status == "pending"
assert notified == ["stranger@example.com"]
def test_registration_of_a_previously_denied_email_revives_the_request(db, make_user, monkeypatch):
# Guard: a denied person may ask again — the invite flips back to pending (and commits) so the
# admin sees a fresh request. The stale decision must not leak into the new account's state.
from app.models import Invite
notified = _no_smtp(monkeypatch)
make_user(email="admin@example.com", role="admin")
db.add(Invite(email="denis@example.com", status="denied", decided_by="admin@example.com"))
db.commit()
user, invite = _register_and_fetch(db, "denis@example.com")
assert user is not None
assert user.is_active is False
assert invite.status == "pending"
assert invite.decided_at is None and invite.decided_by is None
assert notified == ["denis@example.com"]
def test_registration_of_a_pre_approved_email_persists_the_account_with_smtp(db, monkeypatch):
# Guard, not a regression test: with SMTP on, _issue_token's own commit already persisted the
# user, so this branch was never broken and must stay unbroken.
from app.models import Invite
from app.routes import auth as auth_mod
sent: list[str] = []
monkeypatch.setattr(auth_mod.email_mod, "email_enabled", lambda: True)
monkeypatch.setattr(auth_mod.email_mod, "send_verify_email", lambda to, link: sent.append(to))
monkeypatch.setattr(auth_mod.email_mod, "send_admin_new_request", lambda admins, to: True)
db.add(Invite(email="rita@example.com", status="approved", decided_by="admin@example.com"))
db.commit()
user, invite = _register_and_fetch(db, "rita@example.com")
assert user is not None
assert user.email_verified is False # must confirm the address
assert user.is_active is False
assert invite.status == "approved"
assert sent == ["rita@example.com"]
def test_verifying_an_email_never_grants_access_on_its_own(client, db, make_user):
# Verification attests the address; it does not decide access. The two gates stay separate.
from app.models import Invite
from app.routes.auth import _issue_token
user = make_user(email="rita@example.com", password=_PW, email_verified=False, is_active=False)
db.add(Invite(email="rita@example.com", status="approved", decided_by="admin@example.com"))
db.commit()
raw = _issue_token(db, user, "verify", timedelta(hours=1))
assert client.post("/auth/verify", json={"token": raw}).status_code == 200
db.expire_all()
assert user.email_verified is True
assert user.is_active is False
assert _login(client, "rita@example.com", _PW).status_code == 403
def test_register_route_creates_the_account_end_to_end(client, db, monkeypatch):
# The tests above call _register_account directly, so nothing covered the ROUTE: its
# allow_registration gate, its rate limiter and the background.add_task wiring that makes the
# whole flow reachable at all. TestClient runs BackgroundTasks after the response, so a single
# POST exercises the real path end to end.
from sqlalchemy import select as _select
from app.models import User
_no_smtp(monkeypatch)
r = client.post("/auth/register", json={"email": "newbie@example.com", "password": _PW})
assert r.status_code == 200 and r.json() == {"status": "ok"}
db.expire_all()
user = db.execute(_select(User).where(User.email == "newbie@example.com")).scalar_one_or_none()
assert user is not None, "POST /auth/register answered ok but created nothing"
assert user.password_hash is not None
def test_register_route_is_refused_when_registration_is_closed(client, db, monkeypatch):
# The gate is an explicit 403 with nothing created — not a uniform "ok" that silently drops it.
from sqlalchemy import select as _select
from app import sysconfig
from app.models import User
_no_smtp(monkeypatch)
sysconfig.set_value(db, "allow_registration", False)
r = client.post("/auth/register", json={"email": "closed@example.com", "password": _PW})
assert r.status_code == 403
db.expire_all()
assert (
db.execute(_select(User).where(User.email == "closed@example.com")).scalar_one_or_none()
is None
)
# --- SEC-10 lockout: a revoked preset password must stay RECOVERABLE ---------------------------
#
# Regression pack for the prod incident of 2026-09-02. An account registered with a password, then
# approved by an admin BEFORE its email was verified, ended up with password_hash = NULL. That is
# also what a Google-only account looks like, and reset eligibility keyed off `password_hash` alone —
# so the account could neither sign in (no password) nor obtain a reset link, and every response on
# the way was a deliberately uniform "ok"/401. `password_revoked_at` is the trace that tells the two
# apart; these pin each step of the way back in.
def _capture_reset_links(monkeypatch) -> list[str]:
"""Patch the reset mailer and collect the recipients it is asked to write to."""
from app.routes import auth as auth_mod
sent: list[str] = []
monkeypatch.setattr(auth_mod.email_mod, "send_password_reset", lambda to, link: sent.append(to))
return sent
def test_the_approval_revoke_stamps_password_revoked_at(db, make_user):
from app.routes.admin import _activate_user_by_email
victim = make_user(
email="stamped@example.com",
password="attacker-chosen",
email_verified=False,
is_active=False,
)
assert victim.password_revoked_at is None
_activate_user_by_email(db, "stamped@example.com")
assert victim.password_hash is None
assert victim.password_revoked_at is not None # …so the reset flow can tell it HAD a password
def test_an_account_whose_password_was_revoked_can_still_get_a_reset_link(
db, make_user, monkeypatch
):
# THE lockout bug: without the stamp this sent nothing, and the account was unrecoverable.
from app.routes.admin import _activate_user_by_email
from app.routes.auth import _send_reset_if_eligible
make_user(
email="lockedout@example.com",
password="chosen at registration",
email_verified=False,
is_active=False,
)
_activate_user_by_email(db, "lockedout@example.com")
sent = _capture_reset_links(monkeypatch)
_send_reset_if_eligible("lockedout@example.com")
assert sent == ["lockedout@example.com"]
def test_a_google_only_account_is_still_not_reset_eligible(db, make_user, monkeypatch):
# The other side of the rule: an account that NEVER had a password gets no reset link — its owner
# adds one from Settings while signed in (/auth/set-password), which proves identity by session.
from app.routes.auth import _send_reset_if_eligible
make_user(email="googler@example.com", password=None, google_sub="sub-123")
sent = _capture_reset_links(monkeypatch)
_send_reset_if_eligible("googler@example.com")
assert sent == []
def test_completing_a_reset_clears_the_revocation_stamp(client, db, make_user):
from app.routes.admin import _activate_user_by_email
from app.routes.auth import _issue_token
user = make_user(
email="recovered@example.com",
password="chosen at registration",
email_verified=False,
is_active=False,
)
_activate_user_by_email(db, "recovered@example.com")
raw = _issue_token(db, user, "reset", timedelta(hours=1))
db.commit()
ok = client.post(
"/auth/password-reset/confirm", json={"token": raw, "password": "a brand new one"}
)
assert ok.status_code == 200
db.expire_all()
assert user.password_hash is not None
assert user.password_revoked_at is None # spent — not a permanently reset-eligible account
assert _login(client, "recovered@example.com", "a brand new one").status_code == 200
def test_verify_tells_a_revoked_account_that_it_needs_a_new_password(client, db, make_user):
# The banner's input. Saying so leaks nothing: the caller just proved mailbox control by holding a
# valid single-use verify token. Without it the user meets only the uniform "Invalid email or
# password" and has no way to learn why the password they chose stopped working.
from app.routes.admin import _activate_user_by_email
from app.routes.auth import _issue_token
user = make_user(
email="needspw@example.com",
password="chosen at registration",
email_verified=False,
is_active=False,
)
_activate_user_by_email(db, "needspw@example.com")
raw = _issue_token(db, user, "verify", timedelta(hours=1))
db.commit()
r = client.post("/auth/verify", json={"token": raw})
assert r.status_code == 200
assert r.json() == {"ok": True, "password_reset_required": True}
def test_verify_reports_no_reset_needed_for_an_intact_account(client, db, make_user):
from app.routes.auth import _issue_token
user = make_user(email="intact@example.com", email_verified=False, is_active=False)
raw = _issue_token(db, user, "verify", timedelta(hours=1))
db.commit()
r = client.post("/auth/verify", json={"token": raw})
assert r.status_code == 200
assert r.json() == {"ok": True, "password_reset_required": False}
+43
View File
@@ -55,6 +55,49 @@ def test_channel_detail_returns_normal_shorts_live_counts(db, monkeypatch):
assert body["count_normal"] == 2
@pytest.mark.parametrize("hidden", [True, False])
def test_channel_detail_surfaces_subscription_hidden_flag(db, monkeypatch, hidden):
# S6: the detail GET must expose the per-subscription `hidden` flag so the channel page can offer
# the same Unhide toggle the manager has (the flag was previously omitted from the detail dict).
ch = Channel(id="UC_hid", title="Hidden?", published_at=_PUB)
db.add(ch)
me = User(email="hid@example.com")
db.add(me)
db.flush()
db.add(Subscription(user_id=me.id, channel_id=ch.id, hidden=hidden))
db.commit()
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: me
try:
resp = TestClient(app).get(f"/api/channels/{ch.id}")
finally:
app.dependency_overrides.clear()
assert resp.status_code == 200, resp.text
assert resp.json()["hidden"] is hidden
def test_channel_detail_hidden_defaults_false_when_not_subscribed(db, monkeypatch):
# No subscription row → the flag can't be True; it must serialize as False, not raise.
ch = Channel(id="UC_hid_ns", title="Unsub", published_at=_PUB)
me = User(email="hidns@example.com")
db.add_all([ch, me])
db.commit()
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: me
try:
resp = TestClient(app).get(f"/api/channels/{ch.id}")
finally:
app.dependency_overrides.clear()
assert resp.status_code == 200, resp.text
assert resp.json()["hidden"] is False
def _setup_channel_and_sub(db, *, email, is_demo=False):
ch = Channel(
id="UC_wire",
+257
View File
@@ -0,0 +1,257 @@
"""S6 — per-user, per-host download cookies: host-scope, cache privacy, encryption, write-only API.
The two load-bearing security properties are covered as behaviour, not prose:
* **cache privacy** — a cookie-authed download is a PRIVATE, salted asset; a second user without a
cookie enqueuing the same URL gets the shared (unsalted) asset and can never be cache-served the
authed one, and the worker attaches a cookie ONLY to that private identity.
* **host scope (SEC-12)** — a cookie is resolved ONLY for a URL whose netloc the credential owns; a
look-alike or unrelated host resolves to no host_key and gets no cookie.
Pure-logic parts run with no DB; the CRUD / enqueue-salting / API parts use the `db`/`as_user` lane.
"""
import os
from pathlib import Path
import pytest
from cryptography.fernet import Fernet
from app import security
from app.downloads import credentials, formats, service
from app.models import DownloadCredential, MediaAsset, User
@pytest.fixture(autouse=True)
def _fernet(monkeypatch):
# A throwaway key so encrypt/decrypt work without TOKEN_ENCRYPTION_KEY in the env (mirrors
# test_security). Autouse: every credential write in this module encrypts.
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
COOKIES = "# Netscape HTTP Cookie File\n.facebook.com\tTRUE\t/\tTRUE\t0\tc_user\t123\n"
# --- host scope (pure logic) ------------------------------------------------------------------
class TestHostKeyForUrl:
def test_matches_a_supported_host(self):
assert credentials.host_key_for_url("https://www.facebook.com/watch?v=1") == "facebook"
assert credentials.host_key_for_url("https://fb.watch/abc/") == "facebook"
assert credentials.host_key_for_url("https://x.com/i/status/1") == "twitter"
def test_look_alike_host_never_matches(self):
assert credentials.host_key_for_url("https://myfacebook.com/x") is None
assert credentials.host_key_for_url("https://facebook.com.evil.test/x") is None
def test_unrelated_or_empty_url_has_no_host_key(self):
assert credentials.host_key_for_url("https://www.youtube.com/watch?v=abc") is None
assert credentials.host_key_for_url("") is None
assert credentials.host_key_for_url(None) is None
class TestValidateCookies:
def test_accepts_a_netscape_file(self):
assert credentials.validate_cookies(COOKIES).endswith("\n")
def test_rejects_empty(self):
with pytest.raises(credentials.InvalidCookies):
credentials.validate_cookies(" ")
def test_rejects_non_cookie_paste(self):
# Comments only, no tab-separated data line -> not a usable cookies.txt.
with pytest.raises(credentials.InvalidCookies):
credentials.validate_cookies("just some pasted text\n# a comment")
def test_authed_sig_differs_per_user_and_from_the_shared_sig():
spec = formats.normalize({})
shared = formats.format_sig(spec)
a = credentials.authed_format_sig(spec, 1, "facebook")
b = credentials.authed_format_sig(spec, 2, "facebook")
assert a != shared and b != shared and a != b
def test_write_cookiefile_is_private_and_survives_the_retry_wipe(tmp_path):
staging = tmp_path / "staging"
staging.mkdir()
path = credentials._write_cookiefile(COOKIES, staging)
assert path.read_text() == COOKIES
# 0600 (owner-only) — POSIX perms only (the worker is Linux-only; Windows has no mode bits).
if os.name == "posix":
assert (path.stat().st_mode & 0o777) == 0o600
# It lives in a SUBDIR so the download retry loop (which unlinks files in the staging ROOT) and
# _produced_file (which scans the root) never see it.
assert path.parent == staging / ".auth"
assert not any(p.is_file() for p in staging.iterdir())
# --- CRUD + encryption at rest (DB) -----------------------------------------------------------
def _user(db, email="cred@example.com") -> User:
u = User(email=email)
db.add(u)
db.flush()
return u
class TestCrud:
def test_upsert_stores_ciphertext_not_plaintext(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
row = db.get(DownloadCredential, 1)
assert row.cookies_enc != COOKIES # encrypted at rest
assert COOKIES not in row.cookies_enc
assert security.decrypt(row.cookies_enc) == COOKIES
def test_upsert_replaces_in_place(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
credentials.upsert_credential(db, u.id, "facebook", COOKIES.replace("123", "999"))
db.commit()
rows = db.query(DownloadCredential).filter_by(user_id=u.id, host_key="facebook").all()
assert len(rows) == 1
assert "999" in security.decrypt(rows[0].cookies_enc)
def test_status_list_never_leaks_the_blob(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
status = credentials.status_list(db, u.id)
fb = next(h for h in status if h["host_key"] == "facebook")
assert fb["configured"] is True and fb["updated_at"] is not None
ig = next(h for h in status if h["host_key"] == "instagram")
assert ig["configured"] is False and ig["updated_at"] is None
# No field anywhere carries the cookie value.
assert all(COOKIES not in str(v) for h in status for v in h.values())
def test_delete(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
assert credentials.delete_credential(db, u.id, "facebook") is True
db.commit()
assert credentials.get_credential(db, u.id, "facebook") is None
assert credentials.delete_credential(db, u.id, "facebook") is False
# --- the crux: enqueue salts privately, no cross-user cache serve (DB) -------------------------
FB_URL = "https://www.facebook.com/watch?v=100"
class TestCachePrivacy:
def test_enqueue_with_a_cookie_creates_a_private_salted_asset(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
job = service.enqueue(db, u.id, "url", FB_URL, {})
asset = db.get(MediaAsset, job.asset_id)
spec = formats.normalize({})
assert asset.format_sig == credentials.authed_format_sig(spec, u.id, "facebook")
assert asset.format_sig != formats.format_sig(spec)
def test_enqueue_without_a_cookie_uses_the_shared_sig(self, db):
u = _user(db)
job = service.enqueue(db, u.id, "url", FB_URL, {})
asset = db.get(MediaAsset, job.asset_id)
assert asset.format_sig == formats.format_sig(formats.normalize({}))
def test_two_users_never_share_the_authed_asset(self, db):
a = _user(db, "a@example.com")
b = _user(db, "b@example.com")
credentials.upsert_credential(db, a.id, "facebook", COOKIES)
db.commit()
ja = service.enqueue(db, a.id, "url", FB_URL, {}) # A: authed, private
jb = service.enqueue(db, b.id, "url", FB_URL, {}) # B: no cookie, shared
assert ja.asset_id != jb.asset_id
# B's asset is the shared identity — A's private group video can never be served to B.
assert db.get(MediaAsset, jb.asset_id).format_sig == formats.format_sig(
formats.normalize({})
)
# --- the worker gate: attach only to the private identity, host-scoped (DB) --------------------
class TestResolveCookiefile:
def _asset_sig(self, sig: str) -> MediaAsset:
return MediaAsset(
source_kind="url", source_ref=FB_URL, format_sig=sig, status="downloading"
)
def test_attaches_to_the_private_salted_asset(self, db, tmp_path):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
spec = formats.normalize({})
salted = credentials.authed_format_sig(spec, u.id, "facebook")
path = credentials.resolve_cookiefile(db, u.id, FB_URL, salted, spec, tmp_path)
assert path is not None and Path(path).read_text() == COOKIES
def test_never_attaches_to_a_shared_asset(self, db, tmp_path):
# Cookie exists, but the asset is the UNSALTED shared identity -> attaching would poison the
# global cache with a private-group video. Must return None.
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
spec = formats.normalize({})
shared = formats.format_sig(spec)
assert credentials.resolve_cookiefile(db, u.id, FB_URL, shared, spec, tmp_path) is None
def test_non_matching_host_gets_no_cookie(self, db, tmp_path):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
spec = formats.normalize({})
# A YouTube URL has no supported host_key -> no cookie regardless of what's stored.
yt = "https://www.youtube.com/watch?v=abc"
salted = credentials.authed_format_sig(spec, u.id, "facebook")
assert credentials.resolve_cookiefile(db, u.id, yt, salted, spec, tmp_path) is None
def test_no_credential_gets_no_cookie(self, db, tmp_path):
u = _user(db)
spec = formats.normalize({})
salted = credentials.authed_format_sig(spec, u.id, "facebook")
assert credentials.resolve_cookiefile(db, u.id, FB_URL, salted, spec, tmp_path) is None
# --- API: write-only, never echoes the blob (DB) ----------------------------------------------
class TestApi:
def test_put_get_delete_roundtrip_never_returns_the_blob(self, db, as_user):
u = _user(db)
db.commit()
client = as_user(u)
r = client.put("/api/downloads/credentials/facebook", json={"cookies": COOKIES})
assert r.status_code == 200
body = r.json()
assert body["ok"] is True
assert COOKIES not in r.text # the response must never carry the value
fb = next(h for h in body["hosts"] if h["host_key"] == "facebook")
assert fb["configured"] is True
r = client.get("/api/downloads/credentials")
assert r.status_code == 200 and COOKIES not in r.text
r = client.delete("/api/downloads/credentials/facebook")
assert r.status_code == 200
assert credentials.get_credential(db, u.id, "facebook") is None
def test_unknown_host_is_404(self, db, as_user):
u = _user(db)
db.commit()
r = as_user(u).put("/api/downloads/credentials/nope", json={"cookies": COOKIES})
assert r.status_code == 404
def test_bad_cookies_are_rejected(self, db, as_user):
u = _user(db)
db.commit()
r = as_user(u).put("/api/downloads/credentials/facebook", json={"cookies": " "})
assert r.status_code == 400
assert r.json()["detail"]["code"] == "cookies"
+103
View File
@@ -0,0 +1,103 @@
"""Worker-child resource de-prioritization (S4): the encoder thread cap, the `-threads` flag on the
browser-safe re-encode, and the child's startup renice/ionice/OMP handicap. These keep a CPU-heavy
transcode from pegging every core on the box it shares with the API, Postgres, and Plex playback."""
import subprocess
import pytest
from app.config import settings
from app.downloads import edit as editmod
from app.downloads import runner
class TestEncodeThreadCap:
def test_pinned_setting_wins(self, monkeypatch):
monkeypatch.setattr(settings, "download_ffmpeg_threads", 3)
assert editmod.encode_thread_cap() == 3
def test_zero_reserves_one_core(self, monkeypatch):
monkeypatch.setattr(settings, "download_ffmpeg_threads", 0)
monkeypatch.setattr(editmod.os, "cpu_count", lambda: 8)
assert editmod.encode_thread_cap() == 7
def test_single_core_box_never_caps_below_one(self, monkeypatch):
monkeypatch.setattr(settings, "download_ffmpeg_threads", 0)
monkeypatch.setattr(editmod.os, "cpu_count", lambda: 1)
assert editmod.encode_thread_cap() == 1
def test_unknown_cpu_count_falls_back(self, monkeypatch):
monkeypatch.setattr(settings, "download_ffmpeg_threads", 0)
monkeypatch.setattr(editmod.os, "cpu_count", lambda: None)
assert editmod.encode_thread_cap() == 1 # max(1, 2 - 1)
class TestBrowserSafeThreads:
def test_threads_flag_added_when_capped(self):
cmd = editmod.build_browser_safe_cmd(
"/in.mkv", "/out.mp4", reencode_video=True, reencode_audio=False, threads=2
)
assert "-threads" in cmd
assert cmd[cmd.index("-threads") + 1] == "2"
# The cap precedes the output file, so it binds the encoder (not input decoding).
assert cmd.index("-threads") < cmd.index("/out.mp4")
@pytest.mark.parametrize("threads", [0, None])
def test_no_threads_flag_when_uncapped(self, threads):
cmd = editmod.build_browser_safe_cmd(
"/in.mkv", "/out.mp4", reencode_video=True, reencode_audio=False, threads=threads
)
assert "-threads" not in cmd
class TestDeprioritize:
def test_renices_by_configured_increment(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 12)
monkeypatch.setattr(settings, "download_child_ionice_class", 0) # skip ionice branch
calls = []
monkeypatch.setattr(runner.os, "nice", lambda n: calls.append(n))
monkeypatch.setattr(runner.os, "environ", {})
runner._deprioritize()
assert calls == [12]
def test_nice_zero_does_not_renice(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 0)
monkeypatch.setattr(settings, "download_child_ionice_class", 0)
calls = []
monkeypatch.setattr(runner.os, "nice", lambda n: calls.append(n))
monkeypatch.setattr(runner.os, "environ", {})
runner._deprioritize()
assert calls == []
def test_sets_omp_budget_to_thread_cap(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 0)
monkeypatch.setattr(settings, "download_child_ionice_class", 0)
monkeypatch.setattr(settings, "download_ffmpeg_threads", 4)
env: dict = {}
monkeypatch.setattr(runner.os, "environ", env)
runner._deprioritize()
assert env["OMP_NUM_THREADS"] == "4"
def test_ionice_invoked_when_binary_present(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 0)
monkeypatch.setattr(settings, "download_child_ionice_class", 3)
monkeypatch.setattr(runner.os, "environ", {})
monkeypatch.setattr(runner.shutil, "which", lambda name: "/usr/bin/ionice")
ran = []
monkeypatch.setattr(
runner.subprocess,
"run",
lambda cmd, **kw: ran.append(cmd) or subprocess.CompletedProcess(cmd, 0),
)
runner._deprioritize()
assert ran and ran[0][:3] == ["ionice", "-c", "3"]
def test_ionice_skipped_when_binary_absent(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 0)
monkeypatch.setattr(settings, "download_child_ionice_class", 3)
monkeypatch.setattr(runner.os, "environ", {})
monkeypatch.setattr(runner.shutil, "which", lambda name: None)
ran = []
monkeypatch.setattr(runner.subprocess, "run", lambda cmd, **kw: ran.append(cmd))
runner._deprioritize()
assert ran == []
+7
View File
@@ -14,6 +14,13 @@ def test_real_ytdlp_messages_map_to_categories():
"ERROR: [youtube] abc: Sign in to confirm you're not a bot": "login_required",
"ERROR: [youtube] abc: Join this channel to get access to members-only content": "login_required",
"ERROR: [Reddit] 1vi1x56: Account authentication is required.": "login_required",
# Closed-group / private-post auth walls: yt-dlp either says to sign in or suggests cookies.
"ERROR: [facebook] 123: You must be logged in to see this post": "login_required",
"ERROR: [facebook] 123: Login required. Use --cookies for the authentication.": "login_required",
"ERROR: [instagram] abc: This content is only available for registered users": "login_required",
"ERROR: [youtube] abc: Sign in to confirm... Use --cookies-from-browser or --cookies": "login_required",
# A site yt-dlp has no extractor for at all — distinct from an auth wall, still non-retryable.
"ERROR: Unsupported URL: https://example.com/whatever": "unsupported_host",
"ERROR: [youtube] abc: The uploader has not made this video available in your country": "geo_blocked",
"ERROR: [youtube] abc: Requested format is not available": "format_unavailable",
"ERROR: [youtube] abc: Only images are available for download": "format_unavailable",
+58 -21
View File
@@ -6,7 +6,7 @@ with "PosixPath('.') has an empty name" instead of downloading."""
import pytest
from app.downloads.edit import _concat_quote, parse_probe_streams
from app.downloads.edit import _codecs_from_streams, _concat_quote, run_ffprobe_json
from app.downloads.pipeline import _media_info, _produced_file
from pathlib import Path, PurePosixPath
@@ -105,39 +105,76 @@ class TestMediaInfo:
assert _media_info(info)["id"] == "vid"
class TestParseProbeStreams:
class TestCodecsFromStreams:
# ffprobe emits codec_name BEFORE codec_type, so the old line parser shifted every name onto
# the previous stream and an H.264+AAC mp4 came back as (aac, None) → a needless re-encode.
# _codecs_from_streams reads each stream dict by its own codec_type, immune to that ordering.
def test_reads_each_stream_by_its_own_type(self):
payload = '{"streams":[{"codec_name":"h264","codec_type":"video"},{"codec_name":"aac","codec_type":"audio"}]}'
assert parse_probe_streams(payload) == ("h264", "aac")
streams = [
{"codec_name": "h264", "codec_type": "video"},
{"codec_name": "aac", "codec_type": "audio"},
]
assert _codecs_from_streams(streams) == ("h264", "aac")
def test_audio_first_ordering_is_handled(self):
payload = '{"streams":[{"codec_name":"AAC","codec_type":"audio"},{"codec_name":"H264","codec_type":"video"}]}'
assert parse_probe_streams(payload) == ("h264", "aac")
streams = [
{"codec_name": "AAC", "codec_type": "audio"},
{"codec_name": "H264", "codec_type": "video"},
]
assert _codecs_from_streams(streams) == ("h264", "aac")
def test_extra_streams_do_not_win_over_the_first_of_their_kind(self):
payload = (
'{"streams":[{"codec_name":"h264","codec_type":"video"},'
'{"codec_name":"vp9","codec_type":"video"},'
'{"codec_name":"mov_text","codec_type":"subtitle"},'
'{"codec_name":"opus","codec_type":"audio"}]}'
)
assert parse_probe_streams(payload) == ("h264", "opus")
streams = [
{"codec_name": "h264", "codec_type": "video"},
{"codec_name": "vp9", "codec_type": "video"},
{"codec_name": "mov_text", "codec_type": "subtitle"},
{"codec_name": "opus", "codec_type": "audio"},
]
assert _codecs_from_streams(streams) == ("h264", "opus")
def test_audio_only_and_video_only(self):
assert parse_probe_streams('{"streams":[{"codec_name":"opus","codec_type":"audio"}]}') == (
assert _codecs_from_streams([{"codec_name": "opus", "codec_type": "audio"}]) == (
None,
"opus",
)
assert parse_probe_streams('{"streams":[{"codec_name":"av1","codec_type":"video"}]}') == (
"av1",
None,
)
assert _codecs_from_streams([{"codec_name": "av1", "codec_type": "video"}]) == ("av1", None)
def test_garbage_is_not_a_crash(self):
assert parse_probe_streams("") == (None, None)
assert parse_probe_streams("not json") == (None, None)
def test_empty_stream_list(self):
assert _codecs_from_streams([]) == (None, None)
class TestRunFfprobeJson:
"""The single ffprobe-JSON invocation point (shared by probe_codecs + probe.probe_media). A failed
probe or unparseable output must degrade to None, not raise — the malformed-input safety the old
parse_probe_streams string parser used to own now lives here."""
def test_unparseable_output_returns_none(self, monkeypatch):
import subprocess
class _R:
stdout = "not json"
monkeypatch.setattr(subprocess, "run", lambda *a, **k: _R())
assert run_ffprobe_json(Path("x")) is None
def test_ffprobe_failure_returns_none(self, monkeypatch):
import subprocess
def _boom(*a, **k):
raise subprocess.CalledProcessError(1, "ffprobe")
monkeypatch.setattr(subprocess, "run", _boom)
assert run_ffprobe_json(Path("x")) is None
def test_parses_streams_from_valid_json(self, monkeypatch):
import subprocess
class _R:
stdout = '{"streams":[{"codec_name":"h264","codec_type":"video"}]}'
monkeypatch.setattr(subprocess, "run", lambda *a, **k: _R())
data = run_ffprobe_json(Path("x"), entries="stream=codec_type,codec_name")
assert _codecs_from_streams(data["streams"]) == ("h264", None)
class TestConcatQuote:
+149
View File
@@ -0,0 +1,149 @@
"""An undelivered email leaves a trace in the admins' inbox.
Sending is deliberately fail-soft — a missing SMTP setting or a rejecting relay logs and returns
False so nothing user-facing breaks. The cost of that used to be total silence: an instance whose
SMTP was never filled in swallowed verification links, password resets and access-request notices,
and the operator's first clue was somebody saying nothing arrived.
"""
import smtplib
import pytest
from sqlalchemy import select
from app import email as email_mod
from app.models import Notification
@pytest.fixture(autouse=True)
def _no_smtp_config(monkeypatch):
"""Every test here drives the real `_send`, so make sure it never reaches a socket."""
monkeypatch.setattr(
email_mod,
"_smtp",
lambda: {"host": "", "port": 587, "user": "", "password": "", "from": ""},
)
def _notifications(db, user_id: int) -> list[Notification]:
return list(
db.execute(
select(Notification)
.where(Notification.user_id == user_id, Notification.type == "email_undelivered")
.order_by(Notification.id)
)
.scalars()
.all()
)
def test_a_skipped_email_notifies_every_admin(db, make_user):
admin_a = make_user(email="admin-a@example.com", role="admin")
admin_b = make_user(email="admin-b@example.com", role="admin")
plain = make_user(email="plain@example.com")
assert email_mod.send_test("someone@example.com") is False # fail-soft, as before
db.expire_all()
for admin in (admin_a, admin_b):
(n,) = _notifications(db, admin.id)
assert n.data["reason"] == "not_configured"
assert n.data["subject"] == "Siftlode SMTP test"
assert n.data["recipients"] == ["someone@example.com"]
assert _notifications(db, plain.id) == [] # not an operator's problem to see
def test_the_notice_is_throttled_so_a_dead_relay_cannot_flood_the_inbox(db, make_user):
admin = make_user(email="admin@example.com", role="admin")
for _ in range(5):
email_mod.send_test("someone@example.com")
db.expire_all()
assert len(_notifications(db, admin.id)) == 1
def test_a_send_failure_is_reported_separately_from_a_missing_config(db, make_user, monkeypatch):
# A configured-but-rejecting relay is a different operator problem from an empty config, and the
# inbox copy differs — so the reason has to survive to the notification.
admin = make_user(email="admin@example.com", role="admin")
monkeypatch.setattr(
email_mod,
"_smtp",
lambda: {
"host": "smtp.example.com",
"port": 587,
"user": "u",
"password": "p",
"from": "Siftlode <u@example.com>",
},
)
def _boom(*a, **kw):
raise smtplib.SMTPException("nope")
monkeypatch.setattr(email_mod.smtplib, "SMTP", _boom)
assert email_mod.send_test("someone@example.com") is False
db.expire_all()
(n,) = _notifications(db, admin.id)
assert n.data["reason"] == "send_failed"
def test_no_admins_means_no_notification_and_no_error(db, make_user):
# A fresh instance can have no admin yet (the wizard has not run). The alert must stay silent
# rather than blowing up inside the fail-soft path it is supposed to make visible.
plain = make_user(email="plain@example.com")
assert email_mod.send_test("someone@example.com") is False
db.expire_all()
assert _notifications(db, plain.id) == []
def test_a_drop_with_nobody_to_tell_does_not_burn_the_hour(db, make_user):
# The throttle window must only be spent on a notice that was actually written. Otherwise a drop
# during setup — before any admin exists — silences the next hour for the admin created a minute
# later, which is exactly the blind spot this alert exists to remove.
email_mod.send_test("someone@example.com") # no admins yet
admin = make_user(email="admin@example.com", role="admin")
assert email_mod.send_test("someone@example.com") is False
db.expire_all()
assert len(_notifications(db, admin.id)) == 1
def test_a_failed_alert_does_not_burn_the_hour_either(db, make_user, monkeypatch):
admin = make_user(email="admin@example.com", role="admin")
boom = {"on": True}
def _maybe_boom(*a, **kw):
if boom["on"]:
raise RuntimeError("notification store is down")
from app.notifications import create_notification as real
return real(*a, **kw)
monkeypatch.setattr(email_mod, "create_notification", _maybe_boom)
email_mod.send_test("someone@example.com") # the write blows up; nothing recorded
boom["on"] = False
assert email_mod.send_test("someone@example.com") is False
db.expire_all()
assert len(_notifications(db, admin.id)) == 1
def test_a_broken_alert_never_breaks_the_send(db, make_user, monkeypatch):
# The alert is the fallback path; if IT fails, the caller must still get the ordinary fail-soft
# False rather than an exception surfacing from a background task.
make_user(email="admin@example.com", role="admin")
def _boom(*a, **kw):
raise RuntimeError("notification store is down")
monkeypatch.setattr(email_mod, "create_notification", _boom)
assert email_mod.send_test("someone@example.com") is False
+119
View File
@@ -0,0 +1,119 @@
"""Every outbound mail carries an HTML alternative, so its link is actually clickable.
The bodies are plain text and stay that way — but a bare URL in a text/plain mail is only a link if
the CLIENT chooses to linkify it. Gmail/Outlook/Apple Mail do; lighter web clients render the body
verbatim and leave the verification or password-reset link as dead text, which is the entire point of
those two messages. So `_send` now attaches an HTML part generated from the same plain body.
These are pure-logic tests except the last one, which drives the real `_send` against a fake SMTP so
the wire format itself is pinned, not just the helper.
"""
import pytest
from app import email as email_mod
_SMTP_OK = {
"host": "smtp.example.com",
"port": 587,
"user": "u",
"password": "p",
"from": "Siftlode <u@example.com>",
}
def test_a_bare_url_becomes_a_real_link():
# THE regression: this is what a client that does not linkify plain text was leaving as dead text.
out = email_mod._html_body(
"Confirm your email:\n\nhttps://siftlode.example/#verify=tok123\n\n— Siftlode"
)
assert '<a href="https://siftlode.example/#verify=tok123">' in out
def test_interpolated_content_cannot_inject_markup():
# Bodies carry untrusted values — a requester's address in the admin notice, the operator contact.
# Escaping must run BEFORE linkification, or a crafted address becomes markup in every admin's mail.
out = email_mod._html_body('A request from <script>alert("x")</script> & co')
assert "<script>" not in out
assert "&lt;script&gt;" in out
assert "&amp; co" in out
def test_a_query_string_keeps_its_escaped_ampersand_inside_the_href():
# `&` is escaped to `&amp;` before the linkifier sees it, which is also what an HTML attribute
# wants — so the href must carry the escaped form and nothing may be trimmed off it.
out = email_mod._html_body("https://siftlode.example/x?a=1&b=2")
assert (
'<a href="https://siftlode.example/x?a=1&amp;b=2">https://siftlode.example/x?a=1&amp;b=2</a>'
in out
)
@pytest.mark.parametrize(
"body,href,trail",
[
("See https://siftlode.example/x.", "https://siftlode.example/x", "."),
("See https://siftlode.example/x, then sign in", "https://siftlode.example/x", ","),
("See (https://siftlode.example/x)", "https://siftlode.example/x", ")"),
],
)
def test_sentence_punctuation_stays_out_of_the_href(body, href, trail):
out = email_mod._html_body(body)
assert f'<a href="{href}">{href}</a>{trail}' in out
def test_a_token_ending_in_a_url_safe_character_is_not_trimmed():
# token_urlsafe output can end in `-` or `_`; trimming either would hand the user a dead link.
for tok in ("abc-", "abc_"):
out = email_mod._html_body(f"https://siftlode.example/#reset={tok}")
assert f'href="https://siftlode.example/#reset={tok}"' in out
def test_blank_lines_become_paragraphs_and_single_breaks_stay_inline():
out = email_mod._html_body("Hi,\n\nline one\nline two\n\n— Siftlode")
assert out.count("<p>") == 3
assert "<p>line one<br>line two</p>" in out
def test_the_sent_message_is_multipart_with_the_plain_body_intact(monkeypatch):
# The wire format, not just the helper: text/plain stays the primary part and the source of
# truth, with text/html alongside it.
monkeypatch.setattr(email_mod, "_smtp", lambda: _SMTP_OK)
sent = []
class _FakeSMTP:
def __init__(self, *a, **kw):
pass
def __enter__(self):
return self
def __exit__(self, *a):
return False
def starttls(self, *a, **kw):
pass
def login(self, *a, **kw):
pass
def send_message(self, msg):
sent.append(msg)
monkeypatch.setattr(email_mod.smtplib, "SMTP", _FakeSMTP)
assert (
email_mod.send_verify_email("someone@example.com", "https://siftlode.example/#verify=tok")
is True
)
(msg,) = sent
assert msg.get_content_type() == "multipart/alternative"
plain = msg.get_body(preferencelist=("plain",))
rich = msg.get_body(preferencelist=("html",))
assert plain is not None and rich is not None
# The plain part must be byte-for-byte the template's own body — the HTML part is derived from
# it, never a second copy that could drift.
assert "https://siftlode.example/#verify=tok" in plain.get_content()
assert "<a href=" not in plain.get_content()
assert '<a href="https://siftlode.example/#verify=tok">' in rich.get_content()
+143
View File
@@ -162,11 +162,154 @@ _SUMMARY_CASES = [
"show=all&tags={O1}&tags={O2}&tag_mode_other=and",
"show=all&tags={U1}", # user-owned tag (visibility == user.id branch)
"show=all&tags={U1}&tags={L1}", # user tag AND a system tag, cross-category
# "No tag" filter (item B): channels with no USER tag. It stays applied in the base pass, so the
# facet/count lockstep must still hold with it on (alone, and combined with a positive tag filter).
"show=all&untagged=true",
"show=all&untagged=true&tags={L1}",
"show=all&q=clip",
"show=unwatched",
]
def test_untagged_filter_keeps_only_channels_the_user_has_not_tagged(db, as_user, make_user):
"""The "No tag" filter (item B): `untagged=true` keeps only channels with NO tag the USER personally
attached (no user-owned ChannelTag). System/auto tags (user_id NULL) don't count, and another user's
tag on the same channel doesn't either."""
user = make_user(email="untagged@example.com")
other = make_user(email="someone-else@example.com")
mine = Tag(user_id=user.id, name="Fav", category="other")
sys_topic = Tag(user_id=None, name="Tech", category="topic")
theirs = Tag(user_id=other.id, name="TheirTag", category="other")
db.add_all([mine, sys_topic, theirs])
chans = {c: Channel(id=c, title=c, subscriber_count=10) for c in ("chMine", "chSys", "chNone")}
db.add_all(chans.values())
db.flush()
# chMine: the user's own tag → tagged. chSys: only a system tag + another user's tag → still
# "untagged" for THIS user. chNone: nothing at all → untagged.
db.add(ChannelTag(channel_id="chMine", tag_id=mine.id, user_id=user.id, source="user"))
db.add(ChannelTag(channel_id="chSys", tag_id=sys_topic.id, user_id=None, source="auto"))
db.add(ChannelTag(channel_id="chSys", tag_id=theirs.id, user_id=other.id, source="user"))
for cid in chans:
db.add(Subscription(user_id=user.id, channel_id=cid, priority=0, hidden=False))
db.add(_video(f"{cid}v", cid, f"{cid} clip"))
db.commit()
client = as_user(user)
# Off: every subscribed channel shows.
assert _feed_ids(client, "show=all") == {"chMinev", "chSysv", "chNonev"}
# On: only channels the user has NOT personally tagged (a system tag / another user's tag don't count).
assert _feed_ids(client, "show=all&untagged=true") == {"chSysv", "chNonev"}
def test_feed_years_counts_ignore_the_date_narrowing_but_honour_other_filters(
db, as_user, make_user
):
"""The year facet (item 8 / E8·S2): `/api/feed/years` returns a per-year video count, newest first,
with the publish-date narrowing DROPPED (so switching years stays a drill-down) but every other
filter still applied (here: a channel pick)."""
user = make_user(email="years@example.com")
chans = {c: Channel(id=c, title=c, subscriber_count=10) for c in ("chX", "chY")}
db.add_all(chans.values())
db.flush()
# chX: 2 videos in 2024, 1 in 2022; chY: 1 in 2023 (should drop once we scope to chX).
spread = {
"chX": [(2024, 1), (2024, 2), (2022, 5)],
"chY": [(2023, 3)],
}
for cid, dates in spread.items():
db.add(Subscription(user_id=user.id, channel_id=cid, priority=0, hidden=False))
for i, (yr, day) in enumerate(dates):
db.add(
Video(
id=f"{cid}v{i}",
channel_id=cid,
published_at=datetime(yr, 1, day, tzinfo=timezone.utc),
title=f"{cid} clip {i}",
is_short=False,
live_status="none",
via_search=False,
via_explore=False,
)
)
db.commit()
client = as_user(user)
# All channels: every year present, counted, newest first.
resp = client.get("/api/feed/years?show=all")
assert resp.status_code == 200, resp.text
assert resp.json()["years"] == [
{"year": 2024, "count": 2},
{"year": 2023, "count": 1},
{"year": 2022, "count": 1},
]
# A channel filter narrows the year list; a date narrowing is ignored (the point of the facet — so
# a selected year never zeroes out the others). Scoping to chX drops 2023, keeps 2024 (×2) + 2022.
resp = client.get("/api/feed/years?show=all&channel_ids=chX&published_after=2024-01-01")
assert resp.status_code == 200, resp.text
assert resp.json()["years"] == [
{"year": 2024, "count": 2},
{"year": 2022, "count": 1},
]
def test_feed_years_buckets_by_utc_year_under_a_non_utc_session(db, make_user):
"""feed_years must bucket by UTC year to stay in lockstep with the year filter's UTC-pinned
published_after/before bounds. published_at is timestamptz; a bare extract('year', …) would use the
DB session timezone, so a video just before midnight UTC on New Year's Eve would bucket into the
wrong (local) year on a non-UTC session and disagree with the feed the year chip drills into. Called
directly on this session (the HTTP client uses a separate connection SET TIME ZONE wouldn't reach)."""
from sqlalchemy import text
from app.feedquery import feed_years
user = make_user(email="tz-years@example.com")
db.add(Channel(id="chTZ", title="chTZ", subscriber_count=1))
db.add(Subscription(user_id=user.id, channel_id="chTZ", priority=0, hidden=False))
# 2023-12-31 23:30 UTC — a positive-offset session tz local-shifts this into 2024.
db.add(
Video(
id="tzv0",
channel_id="chTZ",
published_at=datetime(2023, 12, 31, 23, 30, tzinfo=timezone.utc),
title="NYE clip",
is_short=False,
live_status="none",
via_search=False,
via_explore=False,
)
)
db.commit()
params: dict = {
"tags": [],
"tag_modes": {"language": "or", "topic": "or", "other": "or"},
"untagged": False,
"channel_ids": [],
"q": None,
"min_duration": None,
"max_duration": None,
"max_age_days": None,
"published_after": None,
"published_before": None,
"show_normal": True,
"include_shorts": False,
"include_live": False,
"show": "all",
"scope": "my",
"library_source": "organic",
}
db.execute(
text("SET TIME ZONE 'Asia/Tokyo'")
) # +9: pushes the video's local wall-clock into 2024
try:
years = feed_years(db, user, **params)
finally:
db.execute(text("SET TIME ZONE 'UTC'")) # SET (not SET LOCAL) survives rollback — reset it
# UTC bucketing keeps the boundary video in 2023; a bare session-tz extract would report 2024.
assert years == [{"year": 2023, "count": 1}]
@pytest.mark.parametrize("case", _SUMMARY_CASES)
def test_feed_summary_matches_count_plus_facets(db, as_user, make_user, case):
# /feed/summary must return byte-identical numbers to the old /feed/count + /facets it replaces,
+32 -10
View File
@@ -1,6 +1,7 @@
"""build_ydl_opts: the mp4 compat profile must SELECT H.264 (not merely prefer it), so shared
downloads play on iOS/Safari, while custom vcodec / non-mp4 profiles opt out. See the iPad AV1
broken-player fix."""
"""build_ydl_opts: EVERY download is clamped to the mp4/H.264 compat profile (E7 browser-
playability guarantee) and must SELECT H.264 (not merely prefer it), so shared downloads play in
every browser incl. iOS/Safari. Non-mp4 containers and non-H.264 vcodecs are coerced, not honoured.
See the iPad AV1 broken-player fix + E7 format narrowing."""
from app.downloads import edit as editmod
from app.downloads import formats
@@ -56,16 +57,37 @@ def test_max_height_caps_every_branch():
assert fmt.count("[height<=?720]") >= 3 # applied to each avc1/fallback branch
def test_explicit_vcodec_opts_out_of_avc1_selector():
def test_explicit_vcodec_is_clamped_to_the_compat_selector():
# E7 browser-playability guarantee: a non-H.264 vcodec no longer opts out — normalize() coerces
# vcodec→None, so the avc1 selector (and mp4 merge) is used regardless of what was requested.
opts = _opts({"mode": "av", "container": "mp4", "vcodec": "vp9"})
assert "vcodec^=avc1" not in opts["format"]
assert opts["format"] == "bestvideo+bestaudio/best"
assert "vcodec:vp9" in opts["format_sort"]
assert opts["format"].startswith("bestvideo[vcodec^=avc1]")
assert opts["merge_output_format"] == "mp4"
assert "vcodec:vp9" not in opts["format_sort"]
def test_non_mp4_container_opts_out():
fmt = _opts({"mode": "av", "container": "webm"})["format"]
assert "vcodec^=avc1" not in fmt
def test_non_mp4_container_is_clamped_to_mp4():
# A webm/mkv request is coerced to the mp4/H.264 compat profile (nothing plays mkv in-browser;
# webm/VP9 fails iOS), so the avc1 selector is used and the merge target is mp4.
opts = _opts({"mode": "av", "container": "webm"})
assert opts["format"].startswith("bestvideo[vcodec^=avc1]")
assert opts["merge_output_format"] == "mp4"
def test_normalize_clamps_to_browser_playable():
n = formats.normalize({"container": "mkv", "vcodec": "av1", "audio_format": "opus"})
assert n["container"] == "mp4"
assert n["vcodec"] is None
assert n["audio_format"] == "m4a"
# A playable audio choice is preserved.
assert formats.normalize({"audio_format": "mp3"})["audio_format"] == "mp3"
def test_non_playable_specs_dedup_to_the_compat_identity():
# Two requests that differ only in (now-clamped) non-playable fields share ONE cache asset.
a = formats.format_sig({"mode": "av", "max_height": 720, "container": "webm", "vcodec": "av1"})
b = formats.format_sig({"mode": "av", "max_height": 720, "container": "mp4"})
assert a == b
def test_audio_only_unaffected():
+37
View File
@@ -5,6 +5,8 @@ import os
import time
from pathlib import Path
from app.config import settings
from app.downloads import gc as gcmod
from app.downloads.gc import _sweep_orphan_files
@@ -69,3 +71,38 @@ def test_sweep_removes_orphans_but_keeps_owned_system_and_fresh(tmp_path: Path):
assert not (root / "OldChan").exists()
assert sysfile.exists() # system tree untouched
assert fresh.exists() # too new to sweep
def test_sweep_throttles_after_each_batch(tmp_path: Path, monkeypatch):
"""The whole-tree walk yields the IO channel after every `batch` files so it can't monopolize
disk IO in the API process. batch=2 over 5 touched files → 2 pauses (at file 2 and 4)."""
root = tmp_path
d = root / "Chan"
d.mkdir()
for i in range(5):
(d / f"orphan_{i}.dat").write_bytes(b"x")
for f in d.iterdir():
_aged(f, 5) # old enough that every file is "touched" by the walk
monkeypatch.setattr(settings, "download_gc_sweep_pause_ms", 10)
monkeypatch.setattr(settings, "download_gc_sweep_batch", 2)
pauses = []
monkeypatch.setattr(gcmod.time, "sleep", lambda s: pauses.append(s))
_sweep_orphan_files(_StubDB([]), str(root))
assert pauses == [0.01, 0.01] # 5 files, batch 2 → yield at #2 and #4
def test_sweep_no_throttle_when_pause_disabled(tmp_path: Path, monkeypatch):
(tmp_path / "Chan").mkdir()
f = tmp_path / "Chan" / "orphan.dat"
f.write_bytes(b"x")
_aged(f, 5)
monkeypatch.setattr(settings, "download_gc_sweep_pause_ms", 0)
pauses = []
monkeypatch.setattr(gcmod.time, "sleep", lambda s: pauses.append(s))
_sweep_orphan_files(_StubDB([]), str(tmp_path))
assert pauses == []
+69 -1
View File
@@ -4,7 +4,14 @@ token, rides in the file URL (a <video src> can't send a header), and expires.""
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
from app.downloads.links import _GRANT_TTL, check_grant, is_expired, make_grant, new_token
from app.downloads.links import (
_GRANT_TTL,
check_grant,
is_expired,
make_grant,
new_token,
public_meta,
)
def test_new_token_is_unguessable_and_unique():
@@ -48,6 +55,67 @@ class TestGrant:
assert check_grant("tok", bad, 0) is False
def _asset(**over):
base = dict(
title="Clip",
uploader="Someone",
uploader_url=None,
duration_s=42,
width=1280,
height=720,
vcodec="h264",
acodec="aac",
media_kind="video",
size_bytes=1000,
container="mp4",
source_webpage_url=None,
source_kind="url",
)
base.update(over)
return SimpleNamespace(**base)
class TestPublicMetaAudioFlag:
"""The watch page picks <audio> vs <video> from this flag; WebKit refuses an audio-only mp3 in a
<video>, so an audio-only asset (media_kind='audio') must report audio=True. The flag now reads
the stored media_kind (the single source), not a re-derived codec check."""
_link = SimpleNamespace(allow_download=True)
def test_audio_only_asset_is_flagged_audio(self):
asset = _asset(media_kind="audio", vcodec=None, acodec="mp3", width=None, height=None)
meta = public_meta(self._link, None, asset, "/f.mp3")
assert meta["audio"] is True
def test_a_video_asset_is_not_audio(self):
meta = public_meta(self._link, None, _asset(), "/f.mp4")
assert meta["audio"] is False
def test_a_generic_file_is_not_audio(self):
# A non-media upload (media_kind='file') has no player at all — never the audio bar.
meta = public_meta(self._link, None, _asset(media_kind="file"), "/f.bin")
assert meta["audio"] is False
assert meta["media_kind"] == "file"
class TestCreateLinkDefault:
"""D1 (E7 S3): a newly minted share link is downloadable by default. The one-click Share relies on
this — it POSTs no `allow_download`, so the API default decides. Forward-only lives here (the API
default), NOT in a migration: the DownloadLink column default stays False, so no existing back-
catalogue link silently flips to downloadable."""
def test_omitted_allow_download_defaults_downloadable(self):
from app.routes.downloads import CreateLinkIn
assert CreateLinkIn().allow_download is True
def test_explicit_false_is_respected(self):
# The owner can still uncheck it to mint a stream-only link.
from app.routes.downloads import CreateLinkIn
assert CreateLinkIn(allow_download=False).allow_download is False
class TestIsExpired:
def test_none_expiry_never_expires(self):
assert is_expired(SimpleNamespace(expires_at=None)) is False
+58
View File
@@ -0,0 +1,58 @@
"""/api/me carries whether this instance can actually send email.
It drives the operator warnings on the Configuration and Access-requests screens. Admin-only,
because it is an operator concern and a regular user has nothing to do with it.
"""
from app import email as email_mod
_PW = "correct horse battery"
def _signed_in(client, make_user, **kw):
"""/api/me resolves its user through optional_current_user, which the as_user override does not
cover — so sign in for real and read the bootstrap payload the SPA actually gets."""
user = make_user(password=_PW, **kw)
assert (
client.post("/auth/password-login", json={"email": user.email, "password": _PW}).status_code
== 200
)
r = client.get("/api/me")
assert r.status_code == 200 and r.json()["authenticated"] is True
return r.json()
def test_an_admin_is_told_when_email_is_not_configured(client, make_user, monkeypatch):
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: False)
me = _signed_in(client, make_user, email="admin@example.com", role="admin")
assert me["email_configured"] is False
def test_an_admin_sees_nothing_wrong_when_email_works(client, make_user, monkeypatch):
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
me = _signed_in(client, make_user, email="admin@example.com", role="admin")
assert me["email_configured"] is True
def test_a_regular_user_is_never_warned_about_the_mail_setup(client, make_user, monkeypatch):
# Not their problem, and the flag must not become a way to probe the instance's configuration.
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: False)
me = _signed_in(client, make_user, email="plain@example.com")
assert me["email_configured"] is True
def test_the_flag_reuses_the_request_session(client, make_user, monkeypatch):
# /api/me is polled once a minute per open tab; reading the SMTP settings decrypts a Fernet
# value, so it must not also check out a second pooled connection each time.
seen: list[object] = []
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: seen.append(db) or True)
_signed_in(client, make_user, email="admin@example.com", role="admin")
assert seen and seen[0] is not None
+360
View File
@@ -0,0 +1,360 @@
"""S7 — two-way (3-way) YouTube playlist merge.
The load-bearing property: an item added (or deleted) on EITHER side survives the sync — the old
push was one-directional (local wins, deletes everything not local from YouTube), so YouTube-side
adds were lost. Pure-merge classification runs with no DB; the apply path uses the `db` lane with a
fake YouTube client so both the local rows and the simulated YouTube state can be asserted.
"""
from datetime import datetime, timezone
import pytest
from app.models import Channel, Playlist, PlaylistItem, Video
from app.sync import playlists as pl_mod
from app.sync.playlists import (
merge_plan,
merge_playlist,
parse_baseline,
parse_unavailable,
plan_merge,
)
from app.youtube.client import YouTubeError
# --- pure merge classification (no DB) --------------------------------------------------------
class TestMergePlan:
def test_no_baseline_unions_losslessly(self):
# First-ever merge: can't tell an add from a delete, so keep everything, delete nothing.
mp = merge_plan(None, local=["a", "b"], youtube=["b", "c"])
assert mp.add_to_yt == ["a"] # local-only pushed out
assert mp.add_to_local == ["c"] # youtube-only pulled in
assert mp.remove_from_yt == [] and mp.remove_from_local == []
assert mp.final == ["a", "b", "c"] # local order, youtube-only appended
assert mp.baseline_known is False
def test_adds_on_both_sides_both_survive(self):
# baseline b,c ; local added 'a' ; youtube added 'd' — nothing deleted.
mp = merge_plan(["b", "c"], local=["a", "b", "c"], youtube=["b", "c", "d"])
assert mp.add_to_yt == ["a"]
assert mp.add_to_local == ["d"]
assert mp.remove_from_yt == [] and mp.remove_from_local == []
assert mp.final == ["a", "b", "c", "d"]
def test_genuine_local_delete_propagates_to_youtube(self):
# baseline a,b,c ; local removed 'b' ; youtube unchanged -> remove b from youtube.
mp = merge_plan(["a", "b", "c"], local=["a", "c"], youtube=["a", "b", "c"])
assert mp.remove_from_yt == ["b"]
assert mp.add_to_yt == [] and mp.add_to_local == []
assert mp.final == ["a", "c"]
def test_genuine_youtube_delete_propagates_to_local(self):
# baseline a,b,c ; youtube removed 'b' ; local unchanged -> remove b from local.
mp = merge_plan(["a", "b", "c"], local=["a", "b", "c"], youtube=["a", "c"])
assert mp.remove_from_local == ["b"]
assert mp.final == ["a", "c"]
def test_delete_on_both_sides_is_gone_not_a_conflict(self):
mp = merge_plan(["a", "b", "c"], local=["a", "c"], youtube=["a", "c"])
assert mp.remove_from_yt == [] and mp.remove_from_local == []
assert mp.final == ["a", "c"]
def test_a_local_delete_beats_a_stale_youtube_copy(self):
# 'b' was synced, deleted locally, still present on youtube -> it must be removed there,
# NOT resurrected into local as a youtube-add.
mp = merge_plan(["a", "b"], local=["a"], youtube=["a", "b"])
assert mp.add_to_local == []
assert mp.remove_from_yt == ["b"]
assert "b" not in mp.final
# --- apply path (db lane + fake YouTube) ------------------------------------------------------
class FakeYouTube:
"""Enough of YouTubeClient for merge_playlist: an in-memory playlist of (item_id, video_id)
and the write ops that mutate it. Every catalog video is pre-seeded so _ensure_videos never
needs get_videos."""
def __init__(
self,
*,
title: str,
items: list[str],
unavailable: set[str] | None = None,
fail_add: set[str] | None = None,
):
self.title = title
self._seq = 0
self.items = [(self._new_id(), v) for v in items]
# Ids the Data API won't return (private/deleted) — get_videos omits them, so _ensure_videos
# can't ingest them. They still live in the playlist (`items`) like a real un-ingestable video.
self.unavailable = set(unavailable or ())
# Ids whose add_playlist_item raises, to exercise the partial-failure baseline-hold path.
self.fail_add = set(fail_add or ())
def _new_id(self) -> str:
self._seq += 1
return f"it{self._seq}"
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def create_playlist(self, title, description="", privacy="private"):
self.title = title
return "PLnew"
def get_playlist_snippet(self, pid):
return {"title": self.title}
def iter_playlist_items_with_ids(self, pid):
return [{"item_id": iid, "video_id": v} for iid, v in self.items]
def get_videos(self, ids):
# Mirror the Data API: a minimal ingestable record for each id, EXCEPT the un-ingestable
# ones (private/deleted), which are simply absent from the response.
return [
{
"id": v,
"snippet": {
"channelId": "UCtest",
"channelTitle": "Test",
"publishedAt": "2020-01-01T00:00:00Z",
},
}
for v in ids
if v not in self.unavailable
]
def update_playlist_title(self, pid, title):
self.title = title
def add_playlist_item(self, pid, vid, position=None):
if vid in self.fail_add:
raise YouTubeError("simulated add failure")
iid = self._new_id()
self.items.append((iid, vid))
return iid
def delete_playlist_item(self, item_id):
self.items = [(iid, v) for iid, v in self.items if iid != item_id]
def move_playlist_item(self, item_id, pid, vid, position):
row = next((r for r in self.items if r[0] == item_id), None)
if row is not None:
self.items.remove(row)
self.items.insert(position, row)
@property
def video_ids(self) -> list[str]:
return [v for _, v in self.items]
def _seed_videos(db, ids):
db.add(Channel(id="UCtest", title="Test"))
now = datetime.now(timezone.utc)
for v in ids:
db.add(Video(id=v, channel_id="UCtest", published_at=now))
db.commit()
def _linked_playlist(db, user, *, local_ids, baseline):
pl = Playlist(
user_id=user.id,
name="Mix",
source="local",
yt_playlist_id="PLyt",
synced_video_ids=None if baseline is None else pl_mod._dump_ids(baseline),
)
db.add(pl)
db.flush()
for pos, v in enumerate(local_ids):
db.add(PlaylistItem(playlist_id=pl.id, video_id=v, position=pos))
db.commit()
return pl
def _local_ids(db, pl):
return pl_mod._local_video_ids(db, pl)
@pytest.fixture
def _fake_yt(monkeypatch):
holder = {}
def install(fake):
holder["fake"] = fake
monkeypatch.setattr(pl_mod, "YouTubeClient", lambda db, user: fake)
return fake
return install
def test_adds_on_both_sides_survive_on_both_sides(db, make_user, _fake_yt):
user = make_user()
_seed_videos(db, ["a", "b", "c", "d"])
# baseline b,c ; local added 'a' ; youtube added 'd'
pl = _linked_playlist(db, user, local_ids=["a", "b", "c"], baseline=["b", "c"])
fake = _fake_yt(FakeYouTube(title="Mix", items=["b", "c", "d"]))
r = merge_playlist(db, user, pl)
assert r["failures"] == []
assert set(_local_ids(db, pl)) == {"a", "b", "c", "d"}, "youtube-add 'd' pulled into Siftlode"
assert set(fake.video_ids) == {"a", "b", "c", "d"}, "local-add 'a' pushed to YouTube"
assert r["inserted"] == 1 and r["pulled"] == 1
# Baseline advanced to the merged set so a re-sync is a no-op.
assert set(parse_baseline(pl)) == {"a", "b", "c", "d"}
def test_a_genuine_delete_propagates_without_collateral_loss(db, make_user, _fake_yt):
user = make_user()
_seed_videos(db, ["a", "b", "c", "d"])
# baseline a,b,c ; local removed 'b' ; youtube added 'd' (a real add, must NOT be collateral)
pl = _linked_playlist(db, user, local_ids=["a", "c"], baseline=["a", "b", "c"])
fake = _fake_yt(FakeYouTube(title="Mix", items=["a", "b", "c", "d"]))
r = merge_playlist(db, user, pl)
assert r["failures"] == []
assert set(fake.video_ids) == {"a", "c", "d"}, "'b' removed on YouTube, 'd' kept"
assert set(_local_ids(db, pl)) == {"a", "c", "d"}, "'d' pulled in, 'b' stays deleted"
assert r["deleted"] == 1
def test_first_merge_with_no_baseline_never_deletes(db, make_user, _fake_yt):
user = make_user()
_seed_videos(db, ["a", "b", "c"])
pl = _linked_playlist(db, user, local_ids=["a", "b"], baseline=None)
fake = _fake_yt(FakeYouTube(title="Mix", items=["b", "c"]))
r = merge_playlist(db, user, pl)
assert r["deleted"] == 0 and r["removed_local"] == 0
assert set(_local_ids(db, pl)) == {"a", "b", "c"}
assert set(fake.video_ids) == {"a", "b", "c"}
assert parse_baseline(pl) is not None, "a baseline is recorded after the first union"
def test_create_path_appends_and_records_baseline(db, make_user, _fake_yt):
user = make_user()
_seed_videos(db, ["a", "b"])
pl = Playlist(user_id=user.id, name="Fresh", source="local")
db.add(pl)
db.flush()
for pos, v in enumerate(["a", "b"]):
db.add(PlaylistItem(playlist_id=pl.id, video_id=v, position=pos))
db.commit()
_fake_yt(FakeYouTube(title="Fresh", items=[]))
r = merge_playlist(db, user, pl)
assert r["created"] == 1 and r["inserted"] == 2
assert pl.yt_playlist_id # created and linked
assert parse_baseline(pl) == ["a", "b"]
def test_reorder_reaches_the_merged_order_on_youtube(db, make_user, _fake_yt):
# Same membership on both sides, but the user reordered locally to c,a,b. The apply-side
# insertion-sort must drive YouTube to that exact order — asserted as an ordered list, not a
# set, so an off-by-one or a dropped reorder pass can't pass unnoticed.
user = make_user()
_seed_videos(db, ["a", "b", "c"])
pl = _linked_playlist(db, user, local_ids=["c", "a", "b"], baseline=["a", "b", "c"])
fake = _fake_yt(FakeYouTube(title="Mix", items=["a", "b", "c"]))
r = merge_playlist(db, user, pl)
assert r["failures"] == []
assert fake.video_ids == ["c", "a", "b"], "YouTube order matches the merged (local) order"
assert r["reordered"] >= 1
assert parse_baseline(pl) == ["c", "a", "b"]
def test_partial_youtube_failure_holds_the_baseline_for_retry(db, make_user, _fake_yt):
# A local-add whose YouTube insert fails must NOT advance the baseline — otherwise the retry
# sync would read the un-pushed add as a genuine YouTube-side delete and silently drop it.
user = make_user()
_seed_videos(db, ["a", "b", "c"])
pl = _linked_playlist(db, user, local_ids=["a", "b", "c"], baseline=["b", "c"])
fake = _fake_yt(FakeYouTube(title="Mix", items=["b", "c"], fail_add={"a"}))
r = merge_playlist(db, user, pl)
assert r["failures"] == ["a"], "the failed local-add is reported"
assert "a" not in fake.video_ids, "the add never landed on YouTube"
# Baseline is UNCHANGED so the next sync re-classifies 'a' as a local-add and re-pushes it,
# rather than treating it as a YouTube delete and removing it locally.
assert parse_baseline(pl) == ["b", "c"]
assert "a" in set(_local_ids(db, pl)), "'a' stays local for the retry"
def test_unavailable_youtube_video_converges(db, make_user, _fake_yt):
# A YouTube-only video that can't be ingested (private/deleted) must not make the playlist
# loop forever re-classifying it as a pending pull. It is recorded as un-ingestable and skipped
# by the next merge; the playlist reports "up to date".
user = make_user()
_seed_videos(db, ["a", "b"]) # note: 'x' is NOT in the catalog and can't be ingested
pl = _linked_playlist(db, user, local_ids=["a", "b"], baseline=["a", "b"])
fake = _fake_yt(FakeYouTube(title="Mix", items=["a", "b", "x"], unavailable={"x"}))
r1 = merge_playlist(db, user, pl)
assert r1["failures"] == []
assert r1["pulled"] == 0, "the un-ingestable video did not become a local row"
assert set(_local_ids(db, pl)) == {"a", "b"}
assert parse_unavailable(pl) == {"x"}, "'x' recorded as un-ingestable"
assert "x" in fake.video_ids, "'x' left untouched on YouTube"
# Second sync: the preview reports nothing pending (plan_merge filters 'x' via the stored
# unavailable set). The sync itself re-attempts ingestion of 'x' (the heal path), but since it
# still won't ingest, 'x' stays recorded as unavailable — no perpetual pending pull.
plan = plan_merge(db, user, pl)
assert plan["add_local"] == 0, "no perpetual pending pull — the playlist is up to date"
r2 = merge_playlist(db, user, pl)
assert r2["pulled"] == 0 and r2["failures"] == []
assert parse_unavailable(pl) == {"x"}
assert "x" in fake.video_ids
def test_unavailable_video_that_becomes_available_is_pulled_in(db, make_user, _fake_yt):
# A video marked un-ingestable (e.g. private) that later becomes available (public) must
# re-enter the merge on a normal sync — not stay skipped forever.
user = make_user()
_seed_videos(db, ["a", "b"])
pl = _linked_playlist(db, user, local_ids=["a", "b"], baseline=["a", "b"])
fake = _fake_yt(FakeYouTube(title="Mix", items=["a", "b", "x"], unavailable={"x"}))
merge_playlist(db, user, pl)
assert parse_unavailable(pl) == {"x"}, "recorded un-ingestable on the first sync"
# 'x' becomes available; the next sync re-attempts ingestion and pulls it in.
fake.unavailable.clear()
r2 = merge_playlist(db, user, pl)
assert r2["failures"] == []
assert "x" in set(_local_ids(db, pl)), "the now-available video is pulled into Siftlode"
assert parse_unavailable(pl) == set(), "'x' is no longer marked un-ingestable"
def test_unavailable_id_that_leaves_youtube_is_pruned(db, make_user, _fake_yt):
# A recorded un-ingestable id that is later removed from the YouTube playlist must be pruned
# from the persisted `unavailable` set (the `& yt_id_set` self-heal), not carried forever.
user = make_user()
_seed_videos(db, ["a", "b"])
pl = _linked_playlist(db, user, local_ids=["a", "b"], baseline=["a", "b"])
fake = _fake_yt(FakeYouTube(title="Mix", items=["a", "b", "x"], unavailable={"x"}))
merge_playlist(db, user, pl)
assert parse_unavailable(pl) == {"x"}, "recorded un-ingestable on the first sync"
# 'x' is removed from the YouTube playlist entirely; the next sync must drop it.
fake.items = [row for row in fake.items if row[1] != "x"]
r2 = merge_playlist(db, user, pl)
assert r2["failures"] == []
assert parse_unavailable(pl) == set(), "'x' pruned once it left YouTube"
@@ -0,0 +1,158 @@
"""S3: the "last watched" browse sort (`sort=lastwatched`). Movies rank by
coalesce(progress_updated_at, watched_at); shows by max(coalesce(...)) across their episodes
(rollup.resumed_at). Never-watched titles sink to the bottom (nullslast). DB lane (`db` fixture) —
skipped without Postgres.
"""
from datetime import datetime, timedelta, timezone
from fastapi.testclient import TestClient
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import PlexItem, PlexLibrary, PlexShow, PlexState, User
NOW = datetime(2024, 6, 1, tzinfo=timezone.utc)
def _seed(db):
lib_m = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
lib_s = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add_all([lib_m, lib_s])
db.flush()
# Movies: newest resume, a watched-only (coalesce falls back to watched_at), an older resume, and a
# never-touched one.
mo_new = PlexItem(rating_key="mo_new", library_id=lib_m.id, kind="movie", title="Newest Resume")
mo_watched = PlexItem(
rating_key="mo_watched", library_id=lib_m.id, kind="movie", title="Watched Only"
)
mo_old = PlexItem(rating_key="mo_old", library_id=lib_m.id, kind="movie", title="Older Resume")
mo_never = PlexItem(rating_key="mo_never", library_id=lib_m.id, kind="movie", title="Never")
# Shows: a recently-resumed episode, a watched-only episode, and a never-watched show.
sh_recent = PlexShow(
rating_key="sh_recent", library_id=lib_s.id, title="Recent Show", child_count=1
)
sh_watched = PlexShow(
rating_key="sh_watched", library_id=lib_s.id, title="Watched Show", child_count=1
)
sh_never = PlexShow(
rating_key="sh_never", library_id=lib_s.id, title="Never Show", child_count=1
)
db.add_all([mo_new, mo_watched, mo_old, mo_never, sh_recent, sh_watched, sh_never])
db.flush()
ep_recent = PlexItem(
rating_key="sh_recent_e1",
library_id=lib_s.id,
kind="episode",
title="E1",
show_id=sh_recent.id,
season_number=1,
episode_number=1,
)
ep_watched = PlexItem(
rating_key="sh_watched_e1",
library_id=lib_s.id,
kind="episode",
title="E1",
show_id=sh_watched.id,
season_number=1,
episode_number=1,
)
db.add_all([ep_recent, ep_watched])
user = User(email="lastwatched@example.com")
db.add(user)
db.flush()
db.add_all(
[
PlexState(
user_id=user.id,
item_id=mo_new.id,
status="new",
position_seconds=100,
progress_updated_at=NOW,
),
PlexState(
user_id=user.id,
item_id=mo_watched.id,
status="watched",
watched_at=NOW - timedelta(hours=1),
),
PlexState(
user_id=user.id,
item_id=mo_old.id,
status="new",
position_seconds=100,
progress_updated_at=NOW - timedelta(hours=2),
),
# Show episodes.
PlexState(
user_id=user.id,
item_id=ep_recent.id,
status="new",
position_seconds=50,
progress_updated_at=NOW - timedelta(minutes=30),
),
PlexState(
user_id=user.id,
item_id=ep_watched.id,
status="watched",
watched_at=NOW - timedelta(minutes=90),
),
]
)
db.commit()
return user
def _get(db, user, monkeypatch, path):
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
try:
return TestClient(app).get(path)
finally:
app.dependency_overrides.clear()
def _ids(resp):
return [i["id"] for i in resp.json()["items"]]
def test_movies_ordered_by_last_watched_desc(db, monkeypatch):
user = _seed(db)
resp = _get(
db, user, monkeypatch, "/api/plex/library?scope=movie&sort=lastwatched&sort_dir=desc"
)
assert resp.status_code == 200, resp.text
# Newest resume, then the watched-only (coalesce → watched_at), then the older resume, then never.
assert _ids(resp) == ["mo_new", "mo_watched", "mo_old", "mo_never"]
def test_shows_ordered_by_last_watched_desc(db, monkeypatch):
user = _seed(db)
resp = _get(
db, user, monkeypatch, "/api/plex/library?scope=show&sort=lastwatched&sort_dir=desc"
)
assert resp.status_code == 200, resp.text
# Recent resume first, then the watched-only show, then the never-watched one (nullslast).
assert _ids(resp) == ["sh_recent", "sh_watched", "sh_never"]
def test_last_watched_mixed_scope_nullslast(db, monkeypatch):
user = _seed(db)
resp = _get(
db, user, monkeypatch, "/api/plex/library?scope=both&sort=lastwatched&sort_dir=desc"
)
assert resp.status_code == 200, resp.text
ids = _ids(resp)
# Global recency order across both legs: mo_new (NOW) > sh_recent (−30m) > mo_watched (−1h) >
# sh_watched (−90m) > mo_old (−2h); the two never-watched titles sink to the bottom.
assert ids[:5] == ["mo_new", "sh_recent", "mo_watched", "sh_watched", "mo_old"]
assert set(ids[5:]) == {"mo_never", "sh_never"}
+495
View File
@@ -0,0 +1,495 @@
"""The playback-class filter (E7): filter the Plex feed by Direct Play / Direct Stream / Transcode·HW /
Transcode·SW. Covers the shared classifier + Stream parsing (unit), the sync derivation + codec-change
reset, the bit-depth backfill, and the /library filter + /facets counts (DB lane).
The HW/SW split is derived from the stored `transcode_hw` facet (codec at sync, refined by the bit-depth
backfill). These tests pin that a 10-bit/HDR HEVC lands in SW even though its codec is HW-decodable —
the exact case the runtime VAAPI path (plex.stream) also special-cases.
"""
import re
from pathlib import Path
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import PlexItem, PlexLibrary, User
from app.plex import sync
from fastapi.testclient import TestClient
class TestTranscodeHwCapable:
"""The single classifier both the catalog facet and (by codec) the runtime encoder choice key off."""
def test_non_transcode_items_are_none(self):
assert sync.transcode_hw_capable("direct", "h264") is None
assert sync.transcode_hw_capable("remux", "h264") is None
assert sync.transcode_hw_capable(None, "hevc") is None
def test_hw_decodable_8bit_sdr_is_hw(self):
assert sync.transcode_hw_capable("transcode", "hevc") is True
assert sync.transcode_hw_capable("transcode", "vp9", bit_depth=8) is True
def test_non_decodable_codec_is_sw(self):
# AV1 / MPEG-4 ASP (XviD) can't be HW-decoded on Gen 9.5 → software regardless of bit depth.
assert sync.transcode_hw_capable("transcode", "av1") is False
assert sync.transcode_hw_capable("transcode", "mpeg4", bit_depth=8) is False
def test_10bit_or_hdr_hevc_is_sw(self):
# A HW-decodable codec that is 10-bit or HDR still breaks the VAAPI nv12 pipeline → software.
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=10) is False
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=8, hdr=True) is False
class TestVideoStreamProps:
def test_reads_bit_depth_and_hdr_from_the_video_stream(self):
meta = {
"Media": [
{
"Part": [
{
"Stream": [
{"streamType": 2, "codec": "eac3"}, # audio — skipped
{"streamType": 1, "bitDepth": 10, "colorTrc": "smpte2084"},
]
}
]
}
]
}
assert sync._video_stream_props(meta) == (10, True)
def test_8bit_sdr_stream(self):
meta = {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]}
assert sync._video_stream_props(meta) == (8, False)
def test_absent_stream_is_none_false(self):
assert sync._video_stream_props({}) == (None, False)
assert sync._video_stream_props({"Media": [{"Part": [{}]}]}) == (None, False)
class TestSyncDerivation:
"""`_apply_item` sets transcode_hw codec-only at sync (bit depth unknown from the listing), and a
changed video codec resets the probe watermark so a stale bit-depth can't linger."""
def _meta(self, vcodec, *, key="/parts/1/file.mkv", file="/m.mkv"):
return {
"title": "X",
"Media": [{"videoCodec": vcodec, "Part": [{"file": file, "key": key}]}],
}
def test_codec_only_derivation_at_sync(self):
row = PlexItem(rating_key="x", kind="movie")
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc"))
assert row.playable == "transcode" and row.transcode_hw is True # decodable, 8-bit assumed
row2 = PlexItem(rating_key="y", kind="movie")
sync._apply_item(row2, lib_id=1, kind="movie", meta=self._meta("av1"))
assert row2.transcode_hw is False # av1 not HW-decodable
def test_probed_value_survives_resync_when_codec_unchanged(self):
row = PlexItem(rating_key="x", kind="movie")
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc"))
# Simulate the backfill downgrading this 10-bit HEVC to software.
row.transcode_hw = False
row.video_bit_depth = 10
row.media_probed_at = sync.datetime.now(sync.timezone.utc)
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc"))
assert row.transcode_hw is False # NOT re-derived back to True
assert row.video_bit_depth == 10 and row.media_probed_at is not None
def test_codec_change_resets_the_probe(self):
row = PlexItem(rating_key="x", kind="movie")
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc"))
row.transcode_hw = False
row.video_bit_depth = 10
row.media_probed_at = sync.datetime.now(sync.timezone.utc)
# The file was replaced with an h264 remux — the stale 10-bit probe must be cleared.
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("h264"))
assert row.video_bit_depth is None and row.media_probed_at is None
def test_same_codec_reencode_with_a_new_part_resets_the_probe(self):
# A same-codec bit-depth re-encode (e.g. 8-bit→10-bit HEVC) keeps codec 'hevc' but replaces the
# physical file, so its Plex Part key changes. Codec-only reset would miss this and keep the stale
# 8-bit/HW verdict forever (the backfill only re-probes media_probed_at IS NULL rows) — the
# part-key move must reset the watermark so the new bit depth is re-probed.
row = PlexItem(rating_key="x", kind="movie")
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc", key="/parts/1/f.mkv"))
row.transcode_hw = True
row.video_bit_depth = 8
row.media_probed_at = sync.datetime.now(sync.timezone.utc)
# Re-encoded in place: same codec 'hevc', but a new Part key (new file).
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc", key="/parts/2/f.mkv"))
assert row.video_bit_depth is None and row.media_probed_at is None # re-probe scheduled
def test_migration_codec_set_matches_hw_decodable_codecs():
# Migration 0077 hardcodes the HW-decodable codec set in raw SQL and its comment mandates it stay
# identical to plex.sync.HW_DECODABLE_CODECS. A comment isn't enforcement (house rule) — pin the
# equality so adding/removing a codec from the frozenset without editing the migration fails here
# (else already-migrated prod rows keep a wrong transcode_hw the runtime derivation wouldn't catch).
mig = (
Path(__file__).resolve().parent.parent
/ "alembic"
/ "versions"
/ "0077_plex_playback_class.py"
)
m = re.search(r"lower\(codec_video\)\s+IN\s+\(([^)]*)\)", mig.read_text())
assert m, "could not find the codec IN(...) list in migration 0077"
codecs = {c.strip().strip("'\"").lower() for c in m.group(1).split(",")}
assert codecs == {c.lower() for c in sync.HW_DECODABLE_CODECS}
class _FakePlex:
"""Stands in for PlexClient.metadata — returns a canned full-metadata dict per rating_key."""
def __init__(self, by_key):
self.by_key = by_key
def metadata(self, rating_key, markers=False):
return self.by_key.get(rating_key)
def test_probe_media_batch_refines_hw_to_sw_for_10bit(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
hevc8 = PlexItem(
rating_key="h8",
library_id=lib.id,
kind="movie",
title="8bit",
playable="transcode",
codec_video="hevc",
transcode_hw=True,
)
hevc10 = PlexItem(
rating_key="h10",
library_id=lib.id,
kind="movie",
title="10bit",
playable="transcode",
codec_video="hevc",
transcode_hw=True,
)
xvid = PlexItem(
rating_key="xv",
library_id=lib.id,
kind="movie",
title="xvid",
playable="transcode",
codec_video="mpeg4",
transcode_hw=False,
)
db.add_all([hevc8, hevc10, xvid])
db.commit()
plex = _FakePlex(
{
"h8": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]},
"h10": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 10}]}]}]},
}
)
probed = sync._probe_media_batch(db, plex)
db.refresh(hevc8)
db.refresh(hevc10)
db.refresh(xvid)
# Only the two HW-decodable HEVC items are probed; the XviD is SW-by-codec and never fetched.
assert probed == 2
assert hevc8.video_bit_depth == 8 and hevc8.transcode_hw is True and hevc8.media_probed_at
assert hevc10.video_bit_depth == 10 and hevc10.transcode_hw is False # downgraded to software
assert xvid.media_probed_at is None # not in the work-list
# A second run has nothing left to do (watermarks set).
assert sync._probe_media_batch(db, plex) == 0
def test_probe_stamps_even_when_plex_has_no_stream(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
item = PlexItem(
rating_key="ns",
library_id=lib.id,
kind="movie",
title="nostream",
playable="transcode",
codec_video="hevc",
transcode_hw=True,
)
db.add(item)
db.commit()
plex = _FakePlex({"ns": {"Media": [{"Part": [{}]}]}}) # no Stream array
assert sync._probe_media_batch(db, plex) == 1
db.refresh(item)
# Probed (watermark set so it isn't re-fetched forever); bit depth unknown → stays codec-HW.
assert item.media_probed_at is not None and item.video_bit_depth is None
assert item.transcode_hw is True
def test_probe_media_batch_is_bounded_and_oldest_first(db):
# The per-run bound (limit) caps Plex load; oldest-id-first means successive runs cover the whole
# backlog without starving any item. Seed 3 eligible items, probe with limit=2.
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
items = [
PlexItem(
rating_key=f"h{i}",
library_id=lib.id,
kind="movie",
title=f"m{i}",
playable="transcode",
codec_video="hevc",
transcode_hw=True,
)
for i in range(3)
]
db.add_all(items) # inserted in order → ascending ids
db.commit()
plex = _FakePlex(
{
f"h{i}": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]}
for i in range(3)
}
)
assert sync._probe_media_batch(db, plex, limit=2) == 2 # exactly the bound
for it in items:
db.refresh(it)
# The two LOWEST-id rows were probed; the third waits for the next run.
assert items[0].media_probed_at is not None and items[1].media_probed_at is not None
assert items[2].media_probed_at is None
# The next run continues with the remaining item.
assert sync._probe_media_batch(db, plex, limit=2) == 1
db.refresh(items[2])
assert items[2].media_probed_at is not None
class _RaisingPlex:
"""metadata() raises for `bad_key`, returns a canned dict otherwise — to exercise the transient
fetch-failure branch (leave the watermark NULL so the item retries next run)."""
def __init__(self, by_key, bad_key):
self.by_key = by_key
self.bad_key = bad_key
def metadata(self, rating_key, markers=False):
if rating_key == self.bad_key:
raise RuntimeError("plex timeout")
return self.by_key.get(rating_key)
def test_probe_media_batch_leaves_a_transient_failure_unstamped_to_retry(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
good = PlexItem(
rating_key="g",
library_id=lib.id,
kind="movie",
title="good",
playable="transcode",
codec_video="hevc",
transcode_hw=True,
)
bad = PlexItem(
rating_key="b",
library_id=lib.id,
kind="movie",
title="bad",
playable="transcode",
codec_video="hevc",
transcode_hw=True,
)
db.add_all([good, bad])
db.commit()
plex = _RaisingPlex(
{"g": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 10}]}]}]}}, bad_key="b"
)
# The raise on 'b' must not abort the batch: 'g' is still probed and counted; 'b' stays NULL.
assert sync._probe_media_batch(db, plex) == 1
db.refresh(good)
db.refresh(bad)
assert good.media_probed_at is not None and good.transcode_hw is False # 10-bit → SW
assert bad.media_probed_at is None # transient failure → retried next run, not stamped
def _seed_classes(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add_all(
[
PlexItem(
rating_key="d1", library_id=lib.id, kind="movie", title="Direct", playable="direct"
),
PlexItem(
rating_key="r1", library_id=lib.id, kind="movie", title="Remux", playable="remux"
),
PlexItem(
rating_key="thw",
library_id=lib.id,
kind="movie",
title="HW",
playable="transcode",
codec_video="hevc",
transcode_hw=True,
),
PlexItem(
rating_key="tsw",
library_id=lib.id,
kind="movie",
title="SW",
playable="transcode",
codec_video="mpeg4",
transcode_hw=False,
),
]
)
return lib
def _client(db, user, monkeypatch):
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
return TestClient(app)
def test_library_filters_by_playback_class(db, monkeypatch):
_seed_classes(db)
user = User(email="pclass@example.com")
db.add(user)
db.commit()
client = _client(db, user, monkeypatch)
try:
# Software-only transcodes.
sw = client.get("/api/plex/library?scope=movie&playable=transcode_sw")
assert sw.status_code == 200, sw.text
assert {i["id"] for i in sw.json()["items"]} == {"tsw"}
# HW transcode + Direct Play, OR-combined.
both = client.get("/api/plex/library?scope=movie&playable=transcode_hw,direct")
assert {i["id"] for i in both.json()["items"]} == {"thw", "d1"}
# The HW card carries the transcode_hw flag for the 4-way badge.
card = next(i for i in both.json()["items"] if i["id"] == "thw")
assert card["transcode_hw"] is True
# No filter → everything.
allm = client.get("/api/plex/library?scope=movie")
assert {i["id"] for i in allm.json()["items"]} == {"d1", "r1", "thw", "tsw"}
finally:
app.dependency_overrides.clear()
def test_facets_count_playback_classes_in_severity_order(db, monkeypatch):
_seed_classes(db)
user = User(email="pclassfacet@example.com")
db.add(user)
db.commit()
client = _client(db, user, monkeypatch)
try:
resp = client.get("/api/plex/facets?scope=movie")
finally:
app.dependency_overrides.clear()
assert resp.status_code == 200, resp.text
playable = resp.json()["playable"]
# Fixed good→heavy order (not by count), one per seeded class.
assert [p["value"] for p in playable] == ["direct", "remux", "transcode_hw", "transcode_sw"]
assert {p["value"]: p["count"] for p in playable} == {
"direct": 1,
"remux": 1,
"transcode_hw": 1,
"transcode_sw": 1,
}
def test_unprobed_transcode_folds_into_sw(db, monkeypatch):
# transcode_hw NULL is the COMMON initial state (before the bit-depth backfill runs). The query's
# CASE has no explicit NULL arm — it relies on the `playable=='transcode'` else folding NULL in with
# software. Pin that fold for BOTH the filter and the facet, so a reorder can't silently drop the
# item from the feed (an honest-state violation).
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add(
PlexItem(
rating_key="tnull",
library_id=lib.id,
kind="movie",
title="Unprobed",
playable="transcode",
codec_video="hevc",
transcode_hw=None,
)
)
user = User(email="pnull@example.com")
db.add(user)
db.commit()
client = _client(db, user, monkeypatch)
try:
sw = client.get("/api/plex/library?scope=movie&playable=transcode_sw")
assert {i["id"] for i in sw.json()["items"]} == {"tnull"} # NULL → SW bucket
hw = client.get("/api/plex/library?scope=movie&playable=transcode_hw")
assert hw.json()["items"] == [] # never HW
facets = client.get("/api/plex/facets?scope=movie")
counts = {p["value"]: p["count"] for p in facets.json()["playable"]}
assert counts.get("transcode_sw") == 1 and counts.get("transcode_hw", 0) == 0
finally:
app.dependency_overrides.clear()
def test_facet_playable_self_excludes_the_active_selection(db, monkeypatch):
# A selected playable class must NOT narrow the playable facet's OWN counts (facet self-exclusion),
# otherwise picking one class would collapse the list and you couldn't OR-add a second. Selecting
# 'direct' must still return all four classes at their full, unnarrowed counts.
_seed_classes(db)
user = User(email="pself@example.com")
db.add(user)
db.commit()
client = _client(db, user, monkeypatch)
try:
resp = client.get("/api/plex/facets?scope=movie&playable=direct")
finally:
app.dependency_overrides.clear()
assert resp.status_code == 200, resp.text
counts = {p["value"]: p["count"] for p in resp.json()["playable"]}
assert counts == {"direct": 1, "remux": 1, "transcode_hw": 1, "transcode_sw": 1}
def test_active_playable_narrows_the_other_facets(db, monkeypatch):
# The flip side of self-exclusion: an active playable class DOES narrow the OTHER facet groups (so
# e.g. the year range reflects only the selected class). Seed a Direct movie (2000) and a
# software-transcode movie (2020); selecting playable=direct must collapse the year facet to 2000.
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add_all(
[
PlexItem(
rating_key="d2000",
library_id=lib.id,
kind="movie",
title="Direct2000",
playable="direct",
year=2000,
),
PlexItem(
rating_key="s2020",
library_id=lib.id,
kind="movie",
title="SW2020",
playable="transcode",
codec_video="mpeg4",
transcode_hw=False,
year=2020,
),
]
)
user = User(email="pnarrow@example.com")
db.add(user)
db.commit()
client = _client(db, user, monkeypatch)
try:
wide = client.get("/api/plex/facets?scope=movie").json()
narrow = client.get("/api/plex/facets?scope=movie&playable=direct").json()
finally:
app.dependency_overrides.clear()
assert (wide["year_min"], wide["year_max"]) == (2000, 2020) # both classes span the range
assert (narrow["year_min"], narrow["year_max"]) == (2000, 2000) # narrowed to the Direct subset
+107
View File
@@ -0,0 +1,107 @@
"""The playback-session route's mode routing (E6 S1b).
`POST /stream/{rk}/session` is the single switch that decides how an item plays: a browser-native
file is served raw (`mode:"direct"`), everything else (re)starts an HLS session — and a
`playable="transcode"` item (non-h264: HEVC/VP9/AV1/…) must take that HLS path with the video
RE-ENCODED, not stream-copied. That routing replaced a former `HTTP 501`, and the only thing that
tells `start_session` to re-encode is the `transcode=` flag this route derives from `it.playable`.
The command-builder itself is unit-tested in `test_plex_stream.py`; here we pin the ROUTE decision
(no ffmpeg spawned — `start_session` is stubbed to echo the flag it was handed).
DB lane (`db` fixture) — skipped when no Postgres is reachable.
"""
from types import SimpleNamespace
from fastapi.testclient import TestClient
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import PlexItem, PlexLibrary, User
from app.plex import paths as plex_paths
from app.plex import stream as plex_stream
def _item(db, *, playable, rating_key="m1"):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
it = PlexItem(
rating_key=rating_key,
library_id=lib.id,
kind="movie",
title="A Movie",
file_path="/media/a.mkv",
playable=playable,
)
db.add(it)
db.commit()
return it
def _user(db, email="play@example.com"):
u = User(email=email)
db.add(u)
db.flush()
return u
def _post(db, user, monkeypatch, rating_key, *, captured):
"""Drive the endpoint with ffmpeg stubbed out. `start_session` records the `transcode` flag it
was called with into `captured` and returns a stand-in session echoing it back."""
monkeypatch.setattr(state, "_configured_cache", True) # pass the request gate
monkeypatch.setattr(plex_stream, "reap_idle", lambda: None)
monkeypatch.setattr(plex_paths, "local_media_path", lambda db_, fp: "/local/a.mkv")
def fake_start(db_, it, start, audio, aoff, multi, transcode=False):
captured["transcode"] = transcode
return SimpleNamespace(
media_start_s=start, entry="index.m3u8", transcode=transcode, hw=False, tonemap=False
)
monkeypatch.setattr(plex_stream, "start_session", fake_start)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
try:
return TestClient(app).post(f"/api/plex/stream/{rating_key}/session")
finally:
app.dependency_overrides.clear()
def test_transcode_item_routes_to_an_hls_reencode_session(db, monkeypatch):
# A non-h264 item (formerly HTTP 501) now plays: mode=hls, transcode=True threaded to start_session.
it = _item(db, playable="transcode")
user = _user(db)
captured: dict = {}
resp = _post(db, user, monkeypatch, it.rating_key, captured=captured)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["mode"] == "hls"
assert body["transcode"] is True
assert captured["transcode"] is True # start_session was told to re-encode
def test_remux_item_stays_a_stream_copy_session(db, monkeypatch):
# A remux (h264) item takes the SAME HLS path but must NOT re-encode.
it = _item(db, playable="remux", rating_key="m2")
user = _user(db, "remux@example.com")
captured: dict = {}
resp = _post(db, user, monkeypatch, it.rating_key, captured=captured)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["mode"] == "hls"
assert body["transcode"] is False
assert captured["transcode"] is False
def test_direct_item_never_opens_a_session(db, monkeypatch):
# A browser-native file is served raw — start_session is never reached, so no transcode decision.
it = _item(db, playable="direct", rating_key="m3")
user = _user(db, "direct@example.com")
captured: dict = {}
resp = _post(db, user, monkeypatch, it.rating_key, captured=captured)
assert resp.status_code == 200, resp.text
assert resp.json()["mode"] == "direct"
assert "transcode" not in captured # start_session not called
+714
View File
@@ -8,6 +8,7 @@ Since the sweep stopped being awaited at startup it also runs CONCURRENTLY with
second half of this file pins what keeps it off a LIVE session's segments."""
import os
from pathlib import Path
import pytest
@@ -154,3 +155,716 @@ class TestSweepOrphans:
os.utime(f, (f.stat().st_mtime - 3600,) * 2) # old enough that only is_dir() saves it
assert stream.sweep_orphans(older_than=_cutoff(hls_root)) == 0
assert f.exists()
class _FakeProc:
"""Stand-in for the ffmpeg Popen — start_session only stores it (and _kill/reap poll it later)."""
def poll(self):
return None
class _FakeItem:
def __init__(self, rating_key="12345", file_path="/media/movie.mkv", codec_video="h264"):
self.rating_key = rating_key
self.file_path = file_path
self.playable = "remux"
self.codec_video = codec_video
class _CapturingProc:
"""Records the argv + kwargs each Popen was called with, then behaves like the ffmpeg proc."""
calls: list[tuple] = []
def __init__(self, *args, **kwargs):
_CapturingProc.calls.append((args, kwargs))
def poll(self):
return None
def _venc(cmd):
"""The video codec ffmpeg was told to use (the value after the first `-c:v`)."""
return cmd[cmd.index("-c:v") + 1]
class TestTranscodeCommand:
"""The transcode video re-encode (playable='transcode'): the SAME HLS command as remux with the
video encoder swapped in for `-c:v copy`. The HW path can only be exercised on prod (VAAPI needs
Linux + Intel /dev/dri), so its SHAPE is pinned here; the software path is the one UAT'd locally."""
SRC = Path("/media/movie.mkv")
OUT = Path("/out")
def test_remux_still_stream_copies_the_video(self):
cmd = stream._ffmpeg_cmd(self.SRC, 0.0, self.OUT, None)
assert _venc(cmd) == "copy"
assert "libx264" not in cmd and "h264_vaapi" not in cmd
assert "-hwaccel" not in cmd # no GPU decode on the copy path
assert "-force_key_frames" not in cmd # copy leans on the source's own keyframes
def test_software_transcode_uses_libx264_and_mints_keyframes(self):
cmd = stream._ffmpeg_cmd(self.SRC, 0.0, self.OUT, None, transcode=True, hw=False)
assert _venc(cmd) == "libx264"
assert "copy" not in cmd # the video is no longer stream-copied
# 8-bit High (yuv420p), not High10, so any browser decodes a 10-bit HEVC source.
assert cmd[cmd.index("-pix_fmt") + 1] == "yuv420p"
assert "-threads" in cmd # a core reserved for request handling / playback
assert "-hwaccel" not in cmd # software path: CPU decode, no render node needed
# A forced keyframe at every segment boundary so the HLS muxer cuts independent segments.
assert (
cmd[cmd.index("-force_key_frames") + 1] == f"expr:gte(t,n_forced*{stream._SEG_SECONDS})"
)
def test_hardware_transcode_uses_the_full_vaapi_pipeline(self):
cmd = stream._ffmpeg_cmd(self.SRC, 0.0, self.OUT, None, transcode=True, hw=True)
assert _venc(cmd) == "h264_vaapi"
assert "libx264" not in cmd
# GPU decode options must precede the input so the source is decoded on-device.
assert cmd[cmd.index("-hwaccel") + 1] == "vaapi"
assert "-hwaccel_output_format" in cmd
assert cmd[cmd.index("-vaapi_device") + 1] == stream._VAAPI_DEVICE
assert cmd.index("-hwaccel") < cmd.index("-i")
# scale_vaapi=format=nv12 downconverts a 10-bit source to 8-bit nv12 for h264_vaapi.
assert "scale_vaapi=format=nv12" in cmd
assert (
cmd[cmd.index("-force_key_frames") + 1] == f"expr:gte(t,n_forced*{stream._SEG_SECONDS})"
)
def test_multi_audio_transcode_reencodes_the_video_variant(self):
cmd = stream._ffmpeg_cmd(
self.SRC, 0.0, self.OUT, None, audio_langs=["eng", "hun"], transcode=True, hw=False
)
assert _venc(cmd) == "libx264" # video re-encoded…
assert "-var_stream_map" in cmd # …but still multi-rendition (client-side audio switch)
assert "copy" not in cmd
def test_software_preset_and_crf_come_from_the_knobs(self):
cmd = stream._ffmpeg_cmd(
self.SRC, 0.0, self.OUT, None, transcode=True, hw=False, preset="slow", crf=18
)
assert cmd[cmd.index("-preset") + 1] == "slow"
assert cmd[cmd.index("-crf") + 1] == "18"
def test_software_max_height_caps_with_a_downscale_only_filter(self):
cmd = stream._ffmpeg_cmd(
self.SRC, 0.0, self.OUT, None, transcode=True, hw=False, max_height=720
)
# Downscale-only, aspect-preserving, even width; the comma in min() escaped so it's ONE filter.
assert cmd[cmd.index("-vf") + 1] == "scale=-2:min(ih\\,720)"
def test_software_max_height_zero_means_no_scale_filter(self):
cmd = stream._ffmpeg_cmd(
self.SRC, 0.0, self.OUT, None, transcode=True, hw=False, max_height=0
)
assert "-vf" not in cmd # source resolution kept
def test_hardware_ignores_max_height(self):
# The GPU encode is cheap, so it runs at source resolution; max_height is a software-only knob.
cmd = stream._ffmpeg_cmd(
self.SRC, 0.0, self.OUT, None, transcode=True, hw=True, max_height=720
)
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # not a software scale
def test_hdr_tonemap_uses_the_opencl_graph(self):
# The HDR/10-bit HW path: VAAPI decode → OpenCL tonemap → VAAPI encode. Gen 9.5 can't tonemap in
# the VAAPI VPP, so this is the only HW route for a 10-bit/HDR source.
cmd = stream._ffmpeg_cmd(
self.SRC, 0.0, self.OUT, None, transcode=True, hw=True, tonemap=True
)
assert _venc(cmd) == "h264_vaapi"
assert "libx264" not in cmd
# Named devices + an OpenCL device DERIVED from the VAAPI one, all before -i.
assert cmd[cmd.index("-init_hw_device") + 1] == f"vaapi=va:{stream._VAAPI_DEVICE}"
assert "opencl=ocl@va" in cmd
assert cmd[cmd.index("-hwaccel_device") + 1] == "va"
assert cmd.index("-init_hw_device") < cmd.index("-i")
# The filter graph runs on the derived OpenCL device and tone-maps HDR→SDR to 8-bit nv12.
assert cmd[cmd.index("-filter_hw_device") + 1] == "ocl"
vf = cmd[cmd.index("-vf") + 1]
assert "hwmap=derive_device=opencl" in vf
assert "tonemap_opencl=" in vf and "format=nv12" in vf
assert "hwmap=derive_device=vaapi:reverse=1" in vf
assert "scale_vaapi" not in vf # the plain 8-bit path's filter must NOT be used here
assert (
cmd[cmd.index("-force_key_frames") + 1] == f"expr:gte(t,n_forced*{stream._SEG_SECONDS})"
)
class TestWantHw:
"""VAAPI is used only where the render node was passed in AND (in auto) the source is HW-decodable,
with an admin `plex_transcode_hw` policy (auto/on/off) on top. A HW session on a 10-bit/HDR source
tone-maps on the GPU via OpenCL (tonemap=True) when the OpenCL runtime is present, else drops to
software — see the `_select_encode` cases below."""
SRC = Path("/media/movie.mkv")
@staticmethod
def _policy(monkeypatch, mode):
monkeypatch.setattr(stream.sysconfig, "get_str", lambda db, k: mode)
@staticmethod
def _kind(monkeypatch, kind):
# Pin the source classifier so the tests don't shell out to ffprobe. `kind` is one of
# stream._VK_HWSAFE / _VK_HDR / _VK_SDR10.
monkeypatch.setattr(stream, "_probe_video_kind", lambda src: kind)
@staticmethod
def _hwsafe(monkeypatch, safe):
# Back-compat convenience for the hw-gate tests that only care about 8-bit-SDR vs not: True →
# _VK_HWSAFE, False → _VK_SDR10 (a non-HDR non-hwsafe class, so the hw half still resolves).
monkeypatch.setattr(
stream, "_probe_video_kind", lambda src: stream._VK_HWSAFE if safe else stream._VK_SDR10
)
@staticmethod
def _opencl(monkeypatch, present):
# Pin the OpenCL-runtime probe so the routing doesn't depend on the CI box's /etc/OpenCL.
monkeypatch.setattr(stream, "_opencl_present", lambda: present)
def test_no_hw_when_not_transcoding(self, monkeypatch):
self._policy(monkeypatch, "auto")
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert (
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=False, src=self.SRC)
is False
)
def test_no_hw_without_a_render_node(self, monkeypatch):
self._policy(monkeypatch, "auto")
monkeypatch.setattr(stream, "_render_node_present", lambda: False)
assert (
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC)
is False
)
def test_auto_uses_hw_for_a_decodable_8bit_source_with_the_render_node(self, monkeypatch):
self._policy(monkeypatch, "auto")
self._hwsafe(monkeypatch, True)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert (
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC)
is True
)
def test_auto_falls_back_to_software_for_a_non_decodable_source(self, monkeypatch):
# AV1 / MPEG-4 ASP (XviD) aren't decodable on Gen 9.5 → the GPU can't feed its own encoder.
self._policy(monkeypatch, "auto")
self._hwsafe(monkeypatch, True) # decodability gate must fire BEFORE the probe
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert (
stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True, src=self.SRC)
is False
)
assert (
stream._want_hw(None, _FakeItem(codec_video="mpeg4"), transcode=True, src=self.SRC)
is False
)
def test_auto_hdr_source_falls_back_to_software_without_opencl(self, monkeypatch):
# A decodable HEVC that is true-HDR (PQ/HLG) can't use scale_vaapi=format=nv12 (Gen 9.5), and with
# no OpenCL runtime to tonemap on the GPU it drops to software libx264 (BUG 1's behaviour).
self._policy(monkeypatch, "auto")
self._kind(monkeypatch, stream._VK_HDR)
self._opencl(monkeypatch, False)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (False, False)
def test_auto_hdr_source_uses_hw_tonemap_with_opencl(self, monkeypatch):
# With the OpenCL runtime present, a true-HDR HEVC tone-maps on the GPU (VAAPI decode → OpenCL
# tonemap → VAAPI encode) instead of dropping to CPU — the HW HDR path.
self._policy(monkeypatch, "auto")
self._kind(monkeypatch, stream._VK_HDR)
self._opencl(monkeypatch, True)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, True)
def test_auto_10bit_SDR_source_uses_software_not_tonemap(self, monkeypatch):
# F1 regression: a 10-bit *SDR* (bt709) source is NOT HDR, so it must NOT get the hable HDR tone
# curve even when the OpenCL runtime is present — that would visibly alter an SDR picture. It takes
# software libx264 (an exact 10→8-bit downconvert), never (True, True).
self._policy(monkeypatch, "auto")
self._kind(monkeypatch, stream._VK_SDR10)
self._opencl(monkeypatch, True)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (False, False)
def test_auto_8bit_source_uses_plain_hw_not_tonemap(self, monkeypatch):
# An 8-bit SDR source takes the plain full-GPU scale path — never the tonemap graph — even where
# OpenCL is available.
self._policy(monkeypatch, "auto")
self._kind(monkeypatch, stream._VK_HWSAFE)
self._opencl(monkeypatch, True)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, False)
def test_off_forces_software_even_for_a_decodable_source(self, monkeypatch):
self._policy(monkeypatch, "off")
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert (
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC)
is False
)
def test_on_forces_hw_and_skips_both_source_gates(self, monkeypatch):
# The admin override trusts the GPU; skips the decodability AND the bit-depth gate (the runtime
# HW→SW fallback still backstops it). Render node still required — HW is impossible without it.
self._policy(monkeypatch, "on")
self._hwsafe(monkeypatch, False) # would veto in auto; "on" ignores it
self._opencl(
monkeypatch, False
) # pin: tonemap-half is irrelevant here, keep it deterministic
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert (
stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True, src=self.SRC)
is True
)
monkeypatch.setattr(stream, "_render_node_present", lambda: False)
assert (
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC)
is False
)
def test_on_still_picks_the_tonemap_graph_for_an_hdr_source(self, monkeypatch):
# "on" trusts the GPU but must still route a true-HDR source through the tonemap graph (plain
# scale_vaapi would hit the VPP wall) when OpenCL is present.
self._policy(monkeypatch, "on")
self._kind(monkeypatch, stream._VK_HDR)
self._opencl(monkeypatch, True)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, True)
def test_on_10bit_SDR_uses_plain_hw_not_tonemap(self, monkeypatch):
# F1 regression under "on": a 10-bit SDR source is NOT HDR, so even with OpenCL present "on" must
# pick plain HW (True, False) — the HDR tone curve must never touch an SDR picture.
self._policy(monkeypatch, "on")
self._kind(monkeypatch, stream._VK_SDR10)
self._opencl(monkeypatch, True)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, False)
def test_on_without_opencl_is_plain_hw(self, monkeypatch):
# "on" with no OpenCL runtime → plain HW, tonemap=False, regardless of source class (the runtime
# HW→SW fallback backstops a source the plain path can't actually handle). Pins the tonemap half
# of the "on" tuple, which test_on_forces_hw_and_skips_both_source_gates only checks via _want_hw.
self._policy(monkeypatch, "on")
self._kind(monkeypatch, stream._VK_HDR) # even HDR: no runtime → can't tonemap on the GPU
self._opencl(monkeypatch, False)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, False)
class TestColdStartWarmup:
"""The cold-start buffer warm-up: before reporting a session ready, start_session waits for the
first _WARMUP_SEGMENTS video segments so hls.js doesn't drain a one-segment buffer and stall a
few seconds in (the auto-advance-to-next-episode stall). We spy on wait_for instead of running
ffmpeg."""
@pytest.fixture
def fake_env(self, tmp_path, monkeypatch):
monkeypatch.setattr(stream, "_HLS_ROOT", tmp_path)
monkeypatch.setattr(stream, "_sessions", {})
src = tmp_path / "movie.mkv"
src.write_bytes(b"x")
monkeypatch.setattr(stream.paths, "local_media_path", lambda db, fp: src)
monkeypatch.setattr(stream.sysconfig, "get_int", lambda db, k: 2)
monkeypatch.setattr(stream.subprocess, "Popen", lambda *a, **k: _FakeProc())
return tmp_path
def test_warmup_waits_for_the_first_few_segments(self, fake_env, monkeypatch):
waited = []
def spy(path, timeout=20.0, proc=None):
waited.append(path.name)
return True
monkeypatch.setattr(stream, "wait_for", spy)
s = stream.start_session(db=None, item=_FakeItem(), start_s=0.0)
assert s is not None
# seg_0 (unconditional now) + the warm-up run up to _WARMUP_SEGMENTS.
assert waited == [f"seg_{i}.ts" for i in range(stream._WARMUP_SEGMENTS)]
def test_warmup_stops_at_the_first_segment_that_isnt_ready(self, fake_env, monkeypatch):
waited = []
def spy(path, timeout=20.0, proc=None):
waited.append(path.name)
return path.name != "seg_1.ts" # seg_1 never appears → warm-up gives up there
monkeypatch.setattr(stream, "wait_for", spy)
s = stream.start_session(db=None, item=_FakeItem(), start_s=0.0)
assert s is not None
assert waited == ["seg_0.ts", "seg_1.ts"] # never asked for seg_2
def test_no_warmup_wait_when_the_first_segment_never_arrives(self, fake_env, monkeypatch):
# If seg_0 itself times out we don't warm up at all (nothing to build on) — and the session is
# still returned so the client's own segment requests can drive it, as before.
waited = []
def spy(path, timeout=20.0, proc=None):
waited.append(path.name)
return False
monkeypatch.setattr(stream, "wait_for", spy)
s = stream.start_session(db=None, item=_FakeItem(), start_s=0.0)
assert s is not None
assert waited == ["seg_0.ts"]
class TestTranscodeStartSession:
"""The seam TestTranscodeCommand can't reach: start_session reading the three admin knobs
(plex_transcode_preset/crf/max_height) and threading them + transcode/hw into the ffmpeg argv and
the HlsSession. A swapped knob here is a SILENT mis-encode (still plays), so pin the whole wiring
end-to-end with a real (captured) Popen argv — not just _ffmpeg_cmd called with literals."""
@pytest.fixture
def capturing_env(self, tmp_path, monkeypatch):
monkeypatch.setattr(stream, "_HLS_ROOT", tmp_path)
monkeypatch.setattr(stream, "_sessions", {})
src = tmp_path / "movie.mkv"
src.write_bytes(b"x")
monkeypatch.setattr(stream.paths, "local_media_path", lambda db, fp: src)
# Distinct sentinels per knob so a crf<->max_height swap (both ints, both via get_int) fails.
monkeypatch.setattr(
stream.sysconfig,
"get_int",
lambda db, k: {
"plex_transcode_crf": 18,
"plex_transcode_max_height": 480,
"plex_max_transcodes": 4,
}[k],
)
monkeypatch.setattr(
stream.sysconfig,
"get_str",
lambda db, k: {"plex_transcode_preset": "slow"}.get(k, "auto"),
)
monkeypatch.setattr(stream, "wait_for", lambda path, timeout=20.0, proc=None: True)
_CapturingProc.calls = []
monkeypatch.setattr(stream.subprocess, "Popen", _CapturingProc)
return tmp_path
def test_software_transcode_threads_the_knobs_into_the_command(
self, capturing_env, monkeypatch
):
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
)
s = stream.start_session(
db=None, item=_FakeItem(codec_video="mpeg4"), start_s=0.0, transcode=True
)
assert s is not None and s.transcode is True and s.hw is False
(cmd, *_), kwargs = _CapturingProc.calls[0]
assert _venc(cmd) == "libx264"
assert cmd[cmd.index("-preset") + 1] == "slow" # plex_transcode_preset
assert cmd[cmd.index("-crf") + 1] == "18" # plex_transcode_crf, NOT max_height
assert cmd[cmd.index("-vf") + 1] == "scale=-2:min(ih\\,480)" # plex_transcode_max_height
# The CPU-bound child is niced via preexec_fn on POSIX (the API process itself must not be).
if os.name == "posix":
assert kwargs.get("preexec_fn") is stream._sw_child_preexec
def test_hardware_transcode_uses_the_gpu_and_never_nices(self, capturing_env, monkeypatch):
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (True, False)
)
s = stream.start_session(
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
)
assert s is not None and s.transcode is True and s.hw is True and s.tonemap is False
assert len(_CapturingProc.calls) == 1 # first segment arrived — no HW→SW fallback respawn
(cmd, *_), kwargs = _CapturingProc.calls[0]
assert _venc(cmd) == "h264_vaapi"
assert "libx264" not in cmd
# Plain-HW path unconditionally emits the nv12 downconvert (a disjunction on "-vf not in cmd"
# would silently pass if the filter were dropped — assert it is actually present).
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # no SW cap
assert kwargs.get("preexec_fn") is None # GPU encode is cheap — never niced
def test_hdr_tonemap_transcode_threads_the_opencl_graph_and_flags_the_session(
self, capturing_env, monkeypatch
):
# A HW tonemap session (10-bit/HDR + OpenCL) must build the OpenCL graph in the argv and carry
# tonemap=True on the session, still on the GPU (hw=True) and never niced.
monkeypatch.setattr(stream, "_select_encode", lambda db, item, transcode, src: (True, True))
s = stream.start_session(
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
)
assert s is not None and s.transcode is True and s.hw is True and s.tonemap is True
(cmd, *_), kwargs = _CapturingProc.calls[0]
assert _venc(cmd) == "h264_vaapi"
assert "opencl=ocl@va" in cmd and cmd[cmd.index("-filter_hw_device") + 1] == "ocl"
assert "tonemap_opencl=" in cmd[cmd.index("-vf") + 1]
assert kwargs.get("preexec_fn") is None
def test_a_hw_session_that_produces_nothing_falls_back_to_software(
self, capturing_env, monkeypatch
):
# BUG 1 backstop: _want_hw picked HW but the VAAPI session emits no first segment (a VAProfile
# refusal the gates didn't catch). start_session must respawn on libx264 and report hw=False so
# the item still plays and the route/player honestly show CPU.
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (True, False)
)
seen = {"n": 0}
def flaky_wait(path, timeout=20.0, proc=None):
seen["n"] += 1
return seen["n"] > 1 # the first seg-0 wait (HW) fails; every wait after succeeds (SW)
monkeypatch.setattr(stream, "wait_for", flaky_wait)
s = stream.start_session(
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
)
assert (
s is not None and s.transcode is True and s.hw is False
) # honest CPU after the fallback
assert len(_CapturingProc.calls) == 2 # the HW attempt, then the SW respawn
(hw_cmd, *_), _ = _CapturingProc.calls[0]
(sw_cmd, *_), sw_kwargs = _CapturingProc.calls[1]
assert _venc(hw_cmd) == "h264_vaapi"
assert _venc(sw_cmd) == "libx264"
if os.name == "posix":
assert sw_kwargs.get("preexec_fn") is stream._sw_child_preexec # the SW child is niced
def test_a_hw_TONEMAP_session_that_produces_nothing_falls_back_to_software(
self, capturing_env, monkeypatch
):
# The fallback gate is `not ready and s.hw`, shared by plain-HW and tonemap sessions — but only
# the plain-HW variant was covered. Pin the tonemap variant too: a (True, True) session that
# emits nothing (an OpenCL/tonemap init refusal) must respawn on libx264 and end hw=False,
# tonemap=False, so an HDR item still plays on CPU.
monkeypatch.setattr(stream, "_select_encode", lambda db, item, transcode, src: (True, True))
seen = {"n": 0}
def flaky_wait(path, timeout=20.0, proc=None):
seen["n"] += 1
return (
seen["n"] > 1
) # the first seg-0 wait (HW tonemap) fails; every wait after succeeds
monkeypatch.setattr(stream, "wait_for", flaky_wait)
s = stream.start_session(
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
)
assert s is not None and s.transcode is True and s.hw is False and s.tonemap is False
assert len(_CapturingProc.calls) == 2 # the HW tonemap attempt, then the SW respawn
(hw_cmd, *_), _ = _CapturingProc.calls[0]
(sw_cmd, *_), _ = _CapturingProc.calls[1]
assert (
"tonemap_opencl=" in hw_cmd[hw_cmd.index("-vf") + 1]
) # first attempt built the OpenCL graph
assert _venc(sw_cmd) == "libx264" # respawn is plain software, no tonemap
def test_a_superseded_hw_session_does_not_respawn_over_the_newer_one(
self, capturing_env, monkeypatch
):
# F2/race: start_session waits on seg_0 OUTSIDE _lock. If a concurrent same-key start (a rapid
# seek-restart) or _enforce_cap replaces _sessions[key] while we wait, our proc dying must NOT
# trigger the SW fallback — that would tear down the NEWER session and breach the transcode cap.
# The identity guard must detect it's been superseded and yield the currently-registered session.
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (True, False)
)
class _Newer: # stands in for the session a concurrent _spawn registered
hw = True
tonemap = False
transcode = True
last_access = 0.0
newer = _Newer()
def wait_then_supersede(path, timeout=20.0, proc=None):
# Simulate the concurrent replacement happening during our out-of-lock wait, then report the
# (now-killed) HW proc produced nothing.
stream._sessions["12345"] = newer
return False
monkeypatch.setattr(stream, "wait_for", wait_then_supersede)
s = stream.start_session(
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
)
# No SW respawn — only the initial HW _spawn ran; the newer session was left intact.
assert len(_CapturingProc.calls) == 1
assert s is newer # yielded the currently-registered session, did not clobber it
assert stream._sessions["12345"] is newer
def test_a_cap_culled_hw_session_does_not_respawn_and_is_not_reported_missing(
self, capturing_env, monkeypatch
):
# F2/race, the cull sub-case: while we wait on seg_0 out of lock, cross-key cap pressure lets
# _enforce_cap evict THIS session's key entirely (no replacement). The fallback must still not
# respawn (that would re-breach the cap the cull just enforced), and must NOT return None — a None
# is turned by the route into a dishonest "file not found" 404 for a file that exists. It yields
# our own (now-dead) session instead, so the client's segment requests degrade to the retryable
# "not available yet" path.
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (True, False)
)
captured = {}
def wait_then_cull(path, timeout=20.0, proc=None):
captured["s"] = stream._sessions[
"12345"
] # the HW session the initial _spawn just registered
del stream._sessions[
"12345"
] # simulate _enforce_cap dropping the LRU under cap pressure
return False
monkeypatch.setattr(stream, "wait_for", wait_then_cull)
s = stream.start_session(
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
)
assert (
len(_CapturingProc.calls) == 1
) # no SW respawn — the cull is respected, cap not re-breached
assert (
s is captured["s"]
) # our own session yielded, NOT None (which the route would 404 as missing)
assert "12345" not in stream._sessions # the culled key stays gone
def test_remux_reads_no_transcode_knobs_and_stream_copies(self, capturing_env, monkeypatch):
# A remux start must stream-copy the video and spawn no niced child (the transcode knobs and
# the preexec nice are gated behind `transcode=True`).
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
)
s = stream.start_session(db=None, item=_FakeItem(), start_s=0.0) # transcode defaults False
assert s is not None and s.transcode is False
(cmd, *_), kwargs = _CapturingProc.calls[0]
assert _venc(cmd) == "copy"
assert kwargs.get("preexec_fn") is None
class TestProbeVideoKind:
"""The ffprobe-based source classifier that routes the transcode pipeline: 8-bit SDR → plain VAAPI;
true-HDR (PQ/HLG) → the OpenCL tonemap graph; 10-bit *SDR* → software (an exact downconvert, NEVER
the HDR tone curve — the F1 distinction). Exercises the REAL function body (not a monkeypatch)."""
@staticmethod
def _probe(monkeypatch, stdout):
class _R:
def __init__(self):
self.stdout = stdout
monkeypatch.setattr(stream.subprocess, "run", lambda *a, **k: _R())
def test_8bit_sdr_is_hwsafe(self, monkeypatch):
self._probe(monkeypatch, "pix_fmt=yuv420p\nbits_per_raw_sample=8\ncolor_transfer=bt709\n")
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
def test_10bit_sdr_pix_fmt_is_sdr10(self, monkeypatch):
# yuv420p10le with a non-HDR transfer (bt2020-10 is NOT PQ/HLG) = 10-bit SDR → software, not
# tonemap. Signalled by the pix_fmt suffix even when bits_per_raw_sample is absent.
self._probe(monkeypatch, "pix_fmt=yuv420p10le\ncolor_transfer=bt2020-10\n")
assert stream._probe_video_kind(Path("/x")) == stream._VK_SDR10
def test_bits_per_raw_sample_over_8_sdr_is_sdr10(self, monkeypatch):
self._probe(monkeypatch, "pix_fmt=yuv420p\nbits_per_raw_sample=10\ncolor_transfer=bt709\n")
assert stream._probe_video_kind(Path("/x")) == stream._VK_SDR10
def test_pq_transfer_is_hdr(self, monkeypatch):
# PQ (smpte2084) is true HDR — needs a real tonemap, even if only 8-bit-tagged.
self._probe(
monkeypatch, "pix_fmt=yuv420p10le\nbits_per_raw_sample=10\ncolor_transfer=smpte2084\n"
)
assert stream._probe_video_kind(Path("/x")) == stream._VK_HDR
def test_hlg_transfer_is_hdr(self, monkeypatch):
# HLG (arib-std-b67) is the other HDR transfer — previously untested; it must classify as HDR.
self._probe(
monkeypatch,
"pix_fmt=yuv420p10le\nbits_per_raw_sample=10\ncolor_transfer=arib-std-b67\n",
)
assert stream._probe_video_kind(Path("/x")) == stream._VK_HDR
def test_empty_probe_output_is_hwsafe(self, monkeypatch):
# A probe that returns nothing (no streams / stripped metadata) → assume hwsafe; the runtime
# HW→SW fallback backstops a wrong guess.
self._probe(monkeypatch, "")
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
def test_probe_failure_assumes_hwsafe(self, monkeypatch):
# ffprobe missing/erroring → assume hwsafe; the runtime HW→SW fallback is the real backstop, so a
# probe miss costs at most one wasted HW attempt, never a stranded item.
def boom(*a, **k):
raise OSError("no ffprobe")
monkeypatch.setattr(stream.subprocess, "run", boom)
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
class TestOpenclPresent:
"""The OpenCL-runtime probe (_opencl_present) — the precondition for the HW HDR tonemap graph. Drives
the REAL glob against a temp ICD dir instead of monkeypatching the function, so a regression to the
ICD path / glob / OSError guard would be caught (this routes every HDR source on prod)."""
def test_true_when_an_icd_file_is_registered(self, tmp_path, monkeypatch):
monkeypatch.setattr(stream, "_OPENCL_ICD_DIR", tmp_path)
(tmp_path / "intel.icd").write_text("libintelocl.so\n")
assert stream._opencl_present() is True
def test_false_when_the_dir_is_empty(self, tmp_path, monkeypatch):
monkeypatch.setattr(stream, "_OPENCL_ICD_DIR", tmp_path)
assert stream._opencl_present() is False
def test_false_and_no_raise_when_the_dir_is_missing(self, tmp_path, monkeypatch):
# A missing dir: Path.glob yields nothing (no raise) → False.
monkeypatch.setattr(stream, "_OPENCL_ICD_DIR", tmp_path / "nope")
assert stream._opencl_present() is False
def test_false_when_glob_raises_oserror(self, monkeypatch):
# The `except OSError` guard proper: an unreadable ICD dir (permission denied) makes glob RAISE,
# not return empty. That must be swallowed to False, never propagated up through _select_encode
# into start_session (which would crash HDR routing).
class _BoomDir:
def glob(self, pattern):
raise OSError("permission denied")
monkeypatch.setattr(stream, "_OPENCL_ICD_DIR", _BoomDir())
assert stream._opencl_present() is False
class TestWaitForProcFastFail:
"""wait_for(proc=…) must stop as soon as the ffmpeg proc exits (a VAAPI init refusal dies in ~1s
producing nothing) instead of blocking the whole timeout — that fast-fail is what keeps the HW→SW
fallback from turning into a 25s stall. Exercises the REAL wait_for (every other test monkeypatches
it wholesale, so this branch would otherwise never run)."""
class _DeadProc:
def poll(self):
return 1 # already exited non-zero
def test_returns_true_when_the_segment_exists_despite_a_dead_proc(self, tmp_path):
seg = tmp_path / "seg_0.ts"
seg.write_bytes(b"x")
# The proc exited but DID write a first segment → ready. (The final existence re-check.)
assert stream.wait_for(seg, timeout=5.0, proc=self._DeadProc()) is True
def test_returns_false_fast_when_a_dead_proc_wrote_nothing(self, tmp_path):
# Missing segment + dead proc → give up immediately, not after `timeout`. Assert it's fast by
# using a long timeout the test would visibly hang on if the fast-fail branch regressed.
seg = tmp_path / "seg_0.ts"
assert stream.wait_for(seg, timeout=30.0, proc=self._DeadProc()) is False
def test_empty_segment_from_a_dead_proc_is_not_ready(self, tmp_path):
# A zero-byte segment is not a real segment (size > 0 required).
seg = tmp_path / "seg_0.ts"
seg.write_bytes(b"")
assert stream.wait_for(seg, timeout=30.0, proc=self._DeadProc()) is False
+386
View File
@@ -0,0 +1,386 @@
"""S2 watch-sync semantics: Plex is the source of truth.
Two independent pieces:
- `_pull_apply` (DB lane): on a genuine divergence for an item Plex knows about, Plex now WINS —
the old "keep Siftlode when it's newer within a tolerance" tug-of-war is gone. The anti-bounce
short-circuit (`_same_state` + `synced_to_plex`) must still keep a freshly-pushed row from bouncing.
- `_should_push_resume` (pure): the in-memory write-back throttle — a `final` checkpoint always pushes,
a periodic one pushes at most once per _PUSH_THROTTLE_S seconds per (user, item).
"""
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
from sqlalchemy import select
from app.models import PlexItem, PlexLibrary, PlexState, User
from app.plex.watch_sync import _pull_apply, _reconcile_inprogress_movies
from app.routes.plex.playback import (
_PUSH_THROTTLE_S,
_resume_push_superseded,
_should_push_resume,
)
def _seed_item_and_user(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
item = PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="A Movie")
db.add(item)
user = User(email="watch@example.com")
db.add(user)
db.flush()
return item.id, user.id
class _FakePlex:
"""Minimal PlexClient stand-in for the movie-probe tests: metadata(rating_key) → a leaf dict."""
def __init__(self, by_rk: dict[str, dict]):
self._by_rk = by_rk
def metadata(self, rating_key: str, markers: bool = False):
return self._by_rk.get(str(rating_key))
def test_pull_plex_wins_over_diverging_synced_local_resume(db):
"""The core S2 change: a SYNCED local row (its state already reached Plex) that later diverges from
Plex = a real external Plex-side change, so the incremental pull resolves to PLEX's position — Plex
is authoritative, no LWW. (A DIRTY local row is protected instead — see the dirty-guard test.)"""
item_id, user_id = _seed_item_and_user(db)
now = datetime.now(timezone.utc)
db.add(
PlexState(
user_id=user_id,
item_id=item_id,
status="new",
position_seconds=300,
progress_updated_at=now, # local touched just now — "newer" than Plex
synced_to_plex=True, # but already pushed → a later Plex divergence is an external change
)
)
db.flush()
# Plex reports a DIFFERENT (older) resume position for the same item.
changed = _pull_apply(db, user_id, item_id, "new", 120, plex_ts=now - timedelta(seconds=60))
db.flush()
assert changed == "in_progress"
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.one()
)
assert st.position_seconds == 120, "Plex's position must win, even though local was newer"
assert st.synced_to_plex is True
def test_pull_plex_watched_wins_over_synced_local_in_progress(db):
"""Plex marking the item watched propagates over a SYNCED local in-progress row (external change)."""
item_id, user_id = _seed_item_and_user(db)
now = datetime.now(timezone.utc)
db.add(
PlexState(
user_id=user_id,
item_id=item_id,
status="new",
position_seconds=300,
progress_updated_at=now,
synced_to_plex=True,
)
)
db.flush()
changed = _pull_apply(db, user_id, item_id, "watched", 0, plex_ts=now - timedelta(seconds=60))
db.flush()
assert changed == "watched"
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.one()
)
assert st.status == "watched"
assert st.position_seconds == 0
def test_pull_no_bounce_on_matching_row(db):
"""Anti-bounce short-circuit: when the row already matches what Plex reports (a just-pushed
resume), the pull makes NO change and only settles the synced flag — no churn, no re-pull."""
item_id, user_id = _seed_item_and_user(db)
now = datetime.now(timezone.utc)
db.add(
PlexState(
user_id=user_id,
item_id=item_id,
status="new",
position_seconds=100,
progress_updated_at=now,
synced_to_plex=True, # freshly pushed
)
)
db.flush()
# Plex echoes the same position a beat later (well within the _PROGRESS_MIN_S fuzz).
changed = _pull_apply(db, user_id, item_id, "new", 102, plex_ts=now + timedelta(seconds=2))
db.flush()
assert changed is None, "a matching row must not bounce"
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.one()
)
assert st.position_seconds == 100, "no churn to the stored position"
def test_pull_keeps_dirty_local_on_divergence(db):
"""Durability guard: a DIRTY row (a local change whose immediate push failed → synced_to_plex=False)
must NOT be overwritten by a diverging Plex signal, and must stay dirty so `_repush_dirty` can push
the local value up. Overwriting-and-clearing here would strand the local change forever."""
item_id, user_id = _seed_item_and_user(db)
now = datetime.now(timezone.utc)
db.add(
PlexState(
user_id=user_id,
item_id=item_id,
status="watched", # user just marked watched locally; the scrobble to Plex failed
position_seconds=0,
watched_at=now,
progress_updated_at=now,
synced_to_plex=False, # dirty — never reached Plex
)
)
db.flush()
# Plex still lists it in on_deck at an old in-progress offset (it never got the watched scrobble).
changed = _pull_apply(db, user_id, item_id, "new", 240, plex_ts=now - timedelta(seconds=60))
db.flush()
assert changed is None, "a dirty local row must not be clobbered by the pull"
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.one()
)
assert st.status == "watched", "local watched preserved"
assert st.position_seconds == 0
assert st.synced_to_plex is False, "stays dirty so _repush_dirty retries the push"
def test_pull_settles_flag_on_matching_dirty_row(db):
"""Anti-bounce, dirty branch: a row that already MATCHES Plex but is still dirty gets its
synced_to_plex flag settled to True (no churn), so `_repush_dirty` stops re-pushing it forever."""
item_id, user_id = _seed_item_and_user(db)
now = datetime.now(timezone.utc)
db.add(
PlexState(
user_id=user_id,
item_id=item_id,
status="new",
position_seconds=100,
progress_updated_at=now,
synced_to_plex=False, # dirty, but already agrees with Plex
)
)
db.flush()
changed = _pull_apply(db, user_id, item_id, "new", 102, plex_ts=now + timedelta(seconds=2))
db.flush()
assert changed is None, "a matching row makes no change"
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.one()
)
assert st.position_seconds == 100, "no churn to the stored position"
assert st.synced_to_plex is True, "the dirty flag is settled so it isn't re-pushed endlessly"
def test_movie_probe_flips_stale_in_progress_to_watched(db):
"""S1 core: a local in-progress MOVIE that Plex has since finished (viewCount>0) and dropped from
Continue-Watching flips to watched on the incremental pass — without waiting for the daily reconcile.
This is the "movie stuck in progress after scrubbing the credits" bug."""
item_id, user_id = _seed_item_and_user(db)
now = datetime.now(timezone.utc)
db.add(
PlexState(
user_id=user_id,
item_id=item_id,
status="new",
position_seconds=300,
progress_updated_at=now,
synced_to_plex=True, # was created by a prior pull; the user then watched it on Plex
)
)
db.flush()
# Plex now reports the movie watched (viewCount>0, no offset) and it is NOT on the deck.
plex = _FakePlex({"m1": {"viewCount": 1, "lastViewedAt": int(now.timestamp())}})
out = _reconcile_inprogress_movies(
db, plex, SimpleNamespace(user_id=user_id), on_deck_ids=set()
)
db.flush()
assert out == {"watched": 1, "in_progress": 0, "probed": 1}
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.one()
)
assert st.status == "watched"
assert st.position_seconds == 0
def test_movie_probe_skips_ondeck_and_episodes(db):
"""The probe must NOT touch (a) a movie Plex still lists on the deck — it's legitimately in
progress — nor (b) an in-progress EPISODE (episodes ride onDeck/history + rollup). Only stale
in-progress movies are probed, and metadata() is called for those alone."""
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
on_deck_movie = PlexItem(rating_key="mDeck", library_id=lib.id, kind="movie", title="On deck")
episode = PlexItem(rating_key="e1", library_id=lib.id, kind="episode", title="An episode")
db.add_all([on_deck_movie, episode])
user = User(email="probe2@example.com")
db.add(user)
db.flush()
now = datetime.now(timezone.utc)
for iid in (on_deck_movie.id, episode.id):
db.add(
PlexState(
user_id=user.id,
item_id=iid,
status="new",
position_seconds=300,
progress_updated_at=now,
synced_to_plex=True,
)
)
db.flush()
# Plex would report both watched — but neither should be probed at all, so the fake returning
# "watched" must have NO effect.
plex = _FakePlex(
{
"mDeck": {"viewCount": 1, "lastViewedAt": int(now.timestamp())},
"e1": {"viewCount": 1, "lastViewedAt": int(now.timestamp())},
}
)
out = _reconcile_inprogress_movies(
db, plex, SimpleNamespace(user_id=user.id), on_deck_ids={on_deck_movie.id}
)
db.flush()
assert out == {"watched": 0, "in_progress": 0, "probed": 0}, "neither item is probed"
for iid in (on_deck_movie.id, episode.id):
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user.id, PlexState.item_id == iid)
)
.scalars()
.one()
)
assert st.status == "new" and st.position_seconds == 300, "left untouched"
def test_movie_probe_leaves_row_when_plex_has_no_signal(db):
"""If Plex reports no watch signal (viewCount 0, no offset) for a probed movie, the incremental pass
must NOT clear the local in-progress row — un-watch is the daily reconcile's job, and a Siftlode-only
in-progress state must survive."""
item_id, user_id = _seed_item_and_user(db)
now = datetime.now(timezone.utc)
db.add(
PlexState(
user_id=user_id,
item_id=item_id,
status="new",
position_seconds=300,
progress_updated_at=now,
synced_to_plex=True,
)
)
db.flush()
plex = _FakePlex({"m1": {"viewCount": 0}})
out = _reconcile_inprogress_movies(
db, plex, SimpleNamespace(user_id=user_id), on_deck_ids=set()
)
db.flush()
assert out == {"watched": 0, "in_progress": 0, "probed": 1}
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.one()
)
assert st.status == "new" and st.position_seconds == 300, "in-progress row preserved"
def test_should_push_resume_scopes_by_user_and_item():
"""The throttle window is keyed by the FULL (user, item) tuple: pushing for one key must not
suppress a distinct user or a distinct item within the same window."""
now = datetime.now(timezone.utc)
assert _should_push_resume(9101, 9201, now, final=False) is True # (u1, i1) first tick
# Same window, different item for the same user → independent, still pushes.
assert _should_push_resume(9101, 9202, now, final=False) is True
# Same window, different user, same item → independent, still pushes.
assert _should_push_resume(9102, 9201, now, final=False) is True
# And the original key is still throttled within its window (proves the above weren't just "always").
assert _should_push_resume(9101, 9201, now + timedelta(seconds=1), final=False) is False
def test_resume_push_superseded_drops_older_pending_push():
"""The write-ordering guard: an older still-pending resume push is dropped once a newer push for the
same (user, item) has been scheduled, so a reordered stale position can't land at Plex last."""
key = (9301, 9301)
t0 = datetime.now(timezone.utc)
# Schedule an initial push at t0 (records t0 as the last-push time for this key).
assert _should_push_resume(*key, t0, final=False) is True
# Nothing newer yet → the t0 push is current, not superseded.
assert _resume_push_superseded(*key, t0) is False
# A newer push (e.g. a backward seek) is scheduled at t1 > t0.
t1 = t0 + timedelta(seconds=1)
assert _should_push_resume(*key, t1, final=True) is True
# Now the still-pending t0 push is superseded and must drop; the t1 push is current.
assert _resume_push_superseded(*key, t0) is True
assert _resume_push_superseded(*key, t1) is False
def test_should_push_resume_final_always_pushes():
"""A final checkpoint (pause / seek / stop) always pushes, bypassing the throttle."""
now = datetime.now(timezone.utc)
key = (9001, 9001)
assert _should_push_resume(*key, now, final=True) is True
# A second final a fraction of a second later still pushes.
assert _should_push_resume(*key, now + timedelta(seconds=1), final=True) is True
def test_should_push_resume_periodic_is_throttled():
"""Periodic (non-final) checkpoints push at most once per _PUSH_THROTTLE_S per (user, item)."""
now = datetime.now(timezone.utc)
key = (9002, 9002)
assert _should_push_resume(*key, now, final=False) is True # first tick pushes
# A tick within the window is suppressed.
assert (
_should_push_resume(*key, now + timedelta(seconds=_PUSH_THROTTLE_S - 1), final=False)
is False
)
# Once the window elapses, the next tick pushes again.
assert _should_push_resume(*key, now + timedelta(seconds=_PUSH_THROTTLE_S), final=False) is True
+65
View File
@@ -0,0 +1,65 @@
"""The sliding-window limiter behind /auth/register, password-login, password-reset and demo.
Pure logic, no DB. `reset()` in particular is only ever called by a test fixture (conftest clears
the module-level limiters between tests), so without a test here a refactor could turn it into a
no-op and nothing would notice until an unrelated test started failing with a 429.
"""
from app import ratelimit
from app.ratelimit import RateLimiter
def test_allow_trips_at_the_cap_and_stays_tripped():
lim = RateLimiter(max_events=2, window_seconds=300)
assert lim.allow("ip") is True
assert lim.allow("ip") is True
assert lim.allow("ip") is False
assert lim.allow("ip") is False
def test_hammering_while_blocked_does_not_push_the_window_forward(monkeypatch):
# A blocked attempt must not be recorded, otherwise someone who keeps retrying holds their own
# window open forever — an indefinite lockout for anyone sharing that IP. Only observable across
# the window boundary, so drive the clock: two attempts fill the cap, ten blocked retries land
# just before it expires, and the caller must still be let back in on the far side.
clock = {"t": 1000.0}
monkeypatch.setattr(ratelimit.time, "monotonic", lambda: clock["t"])
lim = RateLimiter(max_events=2, window_seconds=60)
assert lim.allow("ip") is True
assert lim.allow("ip") is True
for _ in range(10):
clock["t"] += 5 # still inside the window; every one of these is refused
assert lim.allow("ip") is False
clock["t"] = 1000.0 + 61 # past the window of the two attempts that actually counted
assert lim.allow("ip") is True
def test_keys_are_independent():
lim = RateLimiter(max_events=1, window_seconds=300)
assert lim.allow("a") is True
assert lim.allow("a") is False
assert lim.allow("b") is True
def test_reset_clears_every_key():
lim = RateLimiter(max_events=1, window_seconds=300)
lim.allow("a")
lim.allow("b")
assert lim.allow("a") is False
lim.reset()
assert lim.allow("a") is True
assert lim.allow("b") is True
def test_expired_attempts_fall_out_of_the_window():
# A zero-length window means every previous attempt is already outside it.
lim = RateLimiter(max_events=1, window_seconds=0)
assert lim.allow("ip") is True
assert lim.allow("ip") is True
+286
View File
@@ -0,0 +1,286 @@
"""DB-backed regression (E8·S4): the manual-tag-pattern suggester.
compute_tag_suggestions learns which auto TOPIC tags are *characteristic* of each user tag (from the
channels the user tagged), then proposes that tag for still-untagged subscribed channels whose own
topics cover ≥60% of the profile. The cases below pin the behaviours that would silently ship a
useless suggester if they broke: the min-examples floor, the intersection-over-profile fraction that
keeps a weak partial match out, the "language is not a signal" exclusion (its inclusion once flooded
every channel with a suggestion), and the "already tagged → excluded" scope. Deterministic — no
network, pure DB state.
Uses the DB lane (`db` fixture) — skipped when no Postgres is reachable.
"""
from app.models import Channel, ChannelTag, Subscription, Tag, User
from app.sync.tagsuggest import (
AUTOAPPLY_THRESHOLD,
CHAR_SUPPORT,
MAX_PER_CHANNEL,
SUGGEST_THRESHOLD,
compute_tag_suggestions,
)
def _auto(db, name, category="topic"):
tag = Tag(user_id=None, name=name, category=category)
db.add(tag)
db.flush()
return tag
def _subscribe(db, user, ch_id, *, auto_tag_ids=(), title=None, hidden=False):
"""A subscribed channel carrying the given auto-tags (source='auto', user_id NULL)."""
db.add(Channel(id=ch_id, title=title or ch_id))
db.add(Subscription(user_id=user.id, channel_id=ch_id, hidden=hidden))
for tid in auto_tag_ids:
db.add(ChannelTag(channel_id=ch_id, tag_id=tid, user_id=None, source="auto"))
def test_suggests_a_user_tag_for_a_channel_matching_its_topic_profile(db):
me = User(email="me@example.com")
db.add(me)
db.flush()
edu = _auto(db, "Education")
film = _auto(db, "Film")
eng = _auto(db, "English", category="language") # a language auto-tag → NOT part of the signal
docu = Tag(user_id=me.id, name="docu", category="other")
db.add(docu)
db.flush()
# Three example channels the user tagged "docu", all Education+Film (+English). The topic profile
# is {Education, Film}; the English language tag is excluded from the profile.
for i in range(3):
cid = f"UC_ex{i}"
_subscribe(db, me, cid, auto_tag_ids=[edu.id, film.id, eng.id])
db.add(ChannelTag(channel_id=cid, tag_id=docu.id, user_id=me.id, source="user"))
# An untagged subscribed channel sharing the full topic profile → full-confidence suggestion.
_subscribe(db, me, "UC_match", auto_tag_ids=[edu.id, film.id, eng.id])
# An untagged channel sharing ONLY the language → no topic overlap at all, so no suggestion. This
# is the anti-flood guarantee: a shared language must never carry a suggestion on its own.
_subscribe(db, me, "UC_english_only", auto_tag_ids=[eng.id])
db.commit()
out = compute_tag_suggestions(db, me)
assert "UC_match" in out
assert out["UC_match"][0]["tag_id"] == docu.id
# Both profile topics present → full confidence, i.e. an "accept all confident" hit.
assert out["UC_match"][0]["confidence"] >= AUTOAPPLY_THRESHOLD
assert "UC_english_only" not in out
# Example channels are already tagged → excluded from suggestions.
assert "UC_ex0" not in out
def test_a_tag_below_the_example_floor_produces_no_profile(db):
me = User(email="me2@example.com")
db.add(me)
db.flush()
edu = _auto(db, "Education")
niche = Tag(user_id=me.id, name="niche", category="other")
db.add(niche)
db.flush()
# Only TWO example channels for "niche" (< MIN_EXAMPLES=3) → no profile, so no suggestion even
# for a perfectly-matching untagged channel.
for i in range(2):
cid = f"UC_niche{i}"
_subscribe(db, me, cid, auto_tag_ids=[edu.id])
db.add(ChannelTag(channel_id=cid, tag_id=niche.id, user_id=me.id, source="user"))
_subscribe(db, me, "UC_would_match", auto_tag_ids=[edu.id])
db.commit()
assert compute_tag_suggestions(db, me) == {}
def test_partial_topic_overlap_is_a_fraction_of_the_profile(db):
# Guards the intersection-over-profile shape at the SUGGEST_THRESHOLD boundary: a 3-topic profile
# matched by 1 topic scores 0.33 (dropped); matched by 2 scores 0.67 (kept). Pins the threshold
# so a change to it can't silently widen/narrow what "a suggestion" means.
assert SUGGEST_THRESHOLD == 0.6
me = User(email="me3@example.com")
db.add(me)
db.flush()
a, b, c = _auto(db, "A"), _auto(db, "B"), _auto(db, "C")
tag = Tag(user_id=me.id, name="triad", category="other")
db.add(tag)
db.flush()
for i in range(3):
cid = f"UC_triad{i}"
_subscribe(db, me, cid, auto_tag_ids=[a.id, b.id, c.id])
db.add(ChannelTag(channel_id=cid, tag_id=tag.id, user_id=me.id, source="user"))
_subscribe(db, me, "UC_one_of_three", auto_tag_ids=[a.id]) # 1/3 = 0.33 < 0.6 → dropped
_subscribe(db, me, "UC_two_of_three", auto_tag_ids=[a.id, b.id]) # 2/3 = 0.67 ≥ 0.6 → kept
db.commit()
out = compute_tag_suggestions(db, me)
assert "UC_one_of_three" not in out
assert "UC_two_of_three" in out
assert out["UC_two_of_three"][0]["tag_id"] == tag.id
assert out["UC_two_of_three"][0]["confidence"] == round(2 / 3, 3)
def test_language_alone_is_never_a_profile_even_when_it_is_the_only_shared_tag(db):
# The anti-flood invariant WITH TEETH. Every example channel shares only a language; each carries a
# DIFFERENT topic, so no topic reaches CHAR_SUPPORT → the topic-only profile is empty → no
# suggestion. If language were ever admitted to the signal the profile would collapse to {language}
# and the language-only channel would score 1.0 and be suggested — the exact flood the exclusion
# prevents. (test_suggests_… can't catch a regression here: there the language merely enlarges an
# already topic-driven profile, so removing the exclusion flips no assertion.)
me = User(email="lang@example.com")
db.add(me)
db.flush()
eng = _auto(db, "English", category="language")
topics = [_auto(db, f"Topic{i}") for i in range(3)]
lang = Tag(user_id=me.id, name="english-stuff", category="other")
db.add(lang)
db.flush()
for i in range(3):
cid = f"UC_lang_ex{i}"
_subscribe(db, me, cid, auto_tag_ids=[eng.id, topics[i].id])
db.add(ChannelTag(channel_id=cid, tag_id=lang.id, user_id=me.id, source="user"))
_subscribe(db, me, "UC_lang_only", auto_tag_ids=[eng.id])
db.commit()
# No characteristic topic → no profile → nothing suggested for anyone (language never carries it).
assert compute_tag_suggestions(db, me) == {}
def test_char_support_admits_a_two_thirds_topic_and_excludes_a_one_third_topic(db):
# Pins the characteristic-profile threshold (CHAR_SUPPORT) at its PARTIAL-support boundary: across 3
# example channels a topic in 2/3 (0.67 ≥ 0.6) enters the profile, one in 1/3 (0.33) does not. The
# other fixtures give every topic full 3/3 support, so a regression demanding full support (or
# flipping ≥ to >) would ship green — this is the only case that exercises fractional support.
assert CHAR_SUPPORT == 0.6
me = User(email="chr@example.com")
db.add(me)
db.flush()
p, q, r = _auto(db, "P"), _auto(db, "Q"), _auto(db, "R")
mix = Tag(user_id=me.id, name="mix", category="other")
db.add(mix)
db.flush()
# p in 3/3 (kept), q in 2/3 (kept, 0.67), r in 1/3 (excluded, 0.33) → profile = {p, q}.
example_topics = [[p.id, q.id, r.id], [p.id, q.id], [p.id]]
for i, topics in enumerate(example_topics):
cid = f"UC_mix_ex{i}"
_subscribe(db, me, cid, auto_tag_ids=topics)
db.add(ChannelTag(channel_id=cid, tag_id=mix.id, user_id=me.id, source="user"))
_subscribe(db, me, "UC_pq", auto_tag_ids=[p.id, q.id]) # {p,q}/{p,q} = 1.0 → suggested
_subscribe(db, me, "UC_only_p", auto_tag_ids=[p.id]) # 1/2 = 0.5 < 0.6 → dropped
_subscribe(db, me, "UC_p_and_r", auto_tag_ids=[p.id, r.id]) # {p}/{p,q} = 0.5 → dropped
db.commit()
out = compute_tag_suggestions(db, me)
assert "UC_pq" in out and out["UC_pq"][0]["confidence"] == 1.0
# q (2/3) IS in the profile: p alone covers only half of it. Would be a full 1.0 match if the
# builder wrongly required 3/3 support (profile would collapse to {p}).
assert "UC_only_p" not in out
# r (1/3) is NOT in the profile: {p,r} covers only p. Would score 0.67 (suggested) if r wrongly
# entered the profile (a downward drift of CHAR_SUPPORT).
assert "UC_p_and_r" not in out
def test_suggestions_are_capped_and_ordered_best_first(db):
# A channel matching many user-tag profiles gets at most MAX_PER_CHANNEL suggestions, ordered best
# score first, with the weakest dropped by the cap. No other fixture matches >1 profile, so the sort
# and the cap are both no-ops there — dropping either ships green. Four DISTINCT confidences keep the
# order unambiguous. (The exact-tie tag_id tie-break in the sort key is a determinism detail, not
# asserted here: on a stable sort it isn't catchable without relying on unspecified DB row order.)
me = User(email="cap@example.com")
db.add(me)
db.flush()
def profile_tag(name, n_topics):
"""A user tag whose characteristic profile is exactly n_topics (3 identical example channels)."""
topics = [_auto(db, f"{name}_t{i}") for i in range(n_topics)]
tag = Tag(user_id=me.id, name=name, category="other")
db.add(tag)
db.flush()
for i in range(3):
cid = f"UC_{name}_ex{i}"
_subscribe(db, me, cid, auto_tag_ids=[t.id for t in topics])
db.add(ChannelTag(channel_id=cid, tag_id=tag.id, user_id=me.id, source="user"))
return tag, topics
t0, top0 = profile_tag("p0", 1) # profile size 1
t1, top1 = profile_tag("p1", 5) # profile size 5
t2, top2 = profile_tag("p2", 10) # profile size 10
t3, top3 = profile_tag("p3", 10) # profile size 10
# One untagged channel matching all four at distinct confidences:
# t0: 1/1 = 1.00
# t1: 4/5 = 0.80
# t2: 7/10 = 0.70
# t3: 6/10 = 0.60 → a valid hit (≥ SUGGEST_THRESHOLD) but the weakest, dropped by the cap of 3
matched = (
[top0[0].id]
+ [t.id for t in top1[:4]]
+ [t.id for t in top2[:7]]
+ [t.id for t in top3[:6]]
)
_subscribe(db, me, "UC_many", auto_tag_ids=matched)
db.commit()
hits = compute_tag_suggestions(db, me)["UC_many"]
assert len(hits) == MAX_PER_CHANNEL # 4 matches, capped to 3
assert [h["tag_id"] for h in hits] == [t0.id, t1.id, t2.id] # best score first
assert [h["confidence"] for h in hits] == [1.0, 0.8, 0.7]
assert t3.id not in [
h["tag_id"] for h in hits
] # the weakest match (0.60) is the one the cap drops
def test_unsubscribed_example_channel_does_not_inflate_the_profile_denominator(db):
# Unsubscribe deletes the Subscription but NOT the source='user' ChannelTag, so a tagged-then-
# unsubscribed channel must NOT count as a profile example: it contributes no topic features (those
# are scanned over subscribed channels only) yet would inflate CHAR_SUPPORT's denominator and drop a
# genuinely characteristic topic. Here 'X' is characteristic at 2/3 of the SUBSCRIBED examples
# (threshold 1.8, count 2 → kept), but if the unsubscribed 4th example counted, threshold 2.4 would
# drop it and suppress the suggestion.
me = User(email="unsub@example.com")
db.add(me)
db.flush()
x, y = _auto(db, "X"), _auto(db, "Y")
wood = Tag(user_id=me.id, name="wood", category="other")
db.add(wood)
db.flush()
# 3 subscribed examples: 2 carry topic X, 1 carries only Y → X support 2/3 among subscribed.
for cid, topic in [("UC_w0", x), ("UC_w1", x), ("UC_w2", y)]:
_subscribe(db, me, cid, auto_tag_ids=[topic.id])
db.add(ChannelTag(channel_id=cid, tag_id=wood.id, user_id=me.id, source="user"))
# A 4th channel tagged "wood" but NO LONGER subscribed (channel + user tag, no Subscription row).
db.add(Channel(id="UC_gone", title="UC_gone"))
db.add(ChannelTag(channel_id="UC_gone", tag_id=wood.id, user_id=me.id, source="user"))
_subscribe(
db, me, "UC_cand", auto_tag_ids=[x.id]
) # untagged X match → should be suggested "wood"
db.commit()
out = compute_tag_suggestions(db, me)
assert "UC_cand" in out
assert out["UC_cand"][0]["tag_id"] == wood.id
def test_hidden_subscriptions_are_not_suggestion_targets(db):
# The Channel Manager renders suggestion ghost chips only on non-hidden rows, so a hidden channel
# must not be a suggestion target — otherwise the header count / bulk-accept would span channels with
# no visible chip. (Hidden channels still count as example/feature signal; only targeting is excluded.)
me = User(email="hid@example.com")
db.add(me)
db.flush()
z = _auto(db, "Z")
ztag = Tag(user_id=me.id, name="ztag", category="other")
db.add(ztag)
db.flush()
for i in range(3):
cid = f"UC_z{i}"
_subscribe(db, me, cid, auto_tag_ids=[z.id])
db.add(ChannelTag(channel_id=cid, tag_id=ztag.id, user_id=me.id, source="user"))
# A hidden, untagged, perfectly-matching subscription → NOT a target; a visible one → suggested.
_subscribe(db, me, "UC_hidden", auto_tag_ids=[z.id], hidden=True)
_subscribe(db, me, "UC_visible", auto_tag_ids=[z.id])
db.commit()
out = compute_tag_suggestions(db, me)
assert "UC_visible" in out
assert "UC_hidden" not in out
+503
View File
@@ -0,0 +1,503 @@
"""E7 S5a — local file upload ingest (the non-yt-dlp path into the media library).
Covers the ingest spine: media vs generic-file classification, the download-only serving of a
generic file (force-download so uploaded HTML/SVG can't execute inline), quota accounting with the
size known up front, the per-file cap enforced while streaming, filename sanitization, and the two
guards that keep an upload out of paths that assume a re-fetchable source (edit / redownload).
DB-backed (real Postgres) — skipped when no database is reachable. `settings.download_root` is
pointed at a per-test tmp dir so ingest's on-disk moves are isolated. The ingest/route tests
monkeypatch the shared `probe.probe_media` so they don't shell out to ffprobe; the classifier's OWN
parsing (image → file, cover-art skip, dims, duration) is covered directly by `TestProbeClassifier`
below, which stubs one level down (`run_ffprobe_json`) and exercises the real `probe_media`.
"""
from pathlib import Path
import pytest
from app.config import settings
from app.downloads import mediakind, quota, storage
from app.downloads import probe as probemod
from app.downloads import upload as uploadmod
from app.models import DownloadQuota, MediaAsset
# --- pure: the kind invariant ---------------------------------------------------------------
def test_derive_media_kind():
assert uploadmod.mediakind.derive("h264", "aac") == mediakind.VIDEO
assert uploadmod.mediakind.derive("vp9", None) == mediakind.VIDEO
assert uploadmod.mediakind.derive(None, "aac") == mediakind.AUDIO
assert uploadmod.mediakind.derive(None, None) == mediakind.FILE
# --- ingest (direct) ------------------------------------------------------------------------
@pytest.fixture
def _root(tmp_path, monkeypatch):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
return tmp_path
def _probe(monkeypatch, result):
# upload ingest now calls the shared `probe.probe_media` (same ground-truth ffprobe the download
# worker uses); patch it at that shared home.
monkeypatch.setattr(probemod, "probe_media", lambda _p: result)
def test_ingest_generic_file_is_download_only(db, make_user, _root, monkeypatch):
_probe(monkeypatch, (None, None, None, None, None))
u = make_user()
src = _root / "src.bin"
src.write_bytes(b"%PDF-1.4 hello")
job = uploadmod.ingest_upload(
db, u.id, filename="My Notes.pdf", src_path=src, size=src.stat().st_size
)
asset = db.get(MediaAsset, job.asset_id)
assert asset.media_kind == mediakind.FILE
assert asset.status == "ready" and job.status == "done" and job.progress == 100
assert asset.container == "pdf"
assert asset.poster_path is None # a generic file has no player and no poster
assert asset.title == "My Notes" # display title from the original name
assert job.job_kind == "upload" and job.source_kind == "upload"
# The file was MOVED into the per-user uploads tree, inside the root, and the temp is gone.
assert not src.exists()
assert asset.rel_path.startswith(".uploads/")
assert storage.safe_abs_path(str(_root), asset.rel_path) is not None
def test_ingest_audio_is_media_no_poster(db, make_user, _root, monkeypatch):
_probe(monkeypatch, (None, "aac", None, None, 200))
u = make_user()
src = _root / "song.m4a"
src.write_bytes(b"fake-audio")
job = uploadmod.ingest_upload(db, u.id, filename="song.m4a", src_path=src, size=10)
asset = db.get(MediaAsset, job.asset_id)
assert asset.media_kind == mediakind.AUDIO
assert asset.duration_s == 200
assert asset.poster_path is None # audio has no frame to poster
def test_ingest_video_records_dims_and_poster(db, make_user, _root, monkeypatch):
_probe(monkeypatch, ("h264", "aac", 640, 480, 60))
# Stub the frame-cut so the test doesn't shell out to ffmpeg.
monkeypatch.setattr(uploadmod.editmod, "ensure_poster", lambda _a, _root: "p/x.jpg")
u = make_user()
src = _root / "clip.mp4"
src.write_bytes(b"fake-video")
job = uploadmod.ingest_upload(db, u.id, filename="clip.mp4", src_path=src, size=10)
asset = db.get(MediaAsset, job.asset_id)
assert asset.media_kind == mediakind.VIDEO
assert (asset.width, asset.height) == (640, 480)
assert asset.poster_path == "p/x.jpg"
def test_ingest_filename_is_sanitized_no_traversal(db, make_user, _root, monkeypatch):
_probe(monkeypatch, (None, None, None, None, None))
u = make_user()
src = _root / "src"
src.write_bytes(b"x")
job = uploadmod.ingest_upload(db, u.id, filename="../../etc/passwd", src_path=src, size=1)
asset = db.get(MediaAsset, job.asset_id)
assert ".." not in asset.rel_path
assert asset.rel_path.startswith(f".uploads/{u.id}/")
assert storage.safe_abs_path(str(_root), asset.rel_path) is not None
def test_ingest_rejects_over_footprint_cap_before_writing(db, make_user, _root, monkeypatch):
_probe(monkeypatch, (None, None, None, None, None))
u = make_user()
db.add(
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=50, unlimited=False)
)
db.commit()
src = _root / "big.bin"
src.write_bytes(b"x" * 10)
with pytest.raises(quota.QuotaExceeded) as e:
uploadmod.ingest_upload(db, u.id, filename="big.bin", src_path=src, size=2000)
assert e.value.reason == "max_bytes"
assert src.exists() # rejected before any disk move — nothing landed
assert db.query(MediaAsset).count() == 0 # and no orphan asset row
# --- route ----------------------------------------------------------------------------------
def test_upload_route_generic_forces_download(as_user, make_user, _root, monkeypatch):
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
r = c.post(
"/api/downloads/upload",
files={"file": ("report.html", b"<script>alert(1)</script>", "text/html")},
)
assert r.status_code == 200, r.text
job = r.json()
assert job["asset"]["media_kind"] == "file"
assert job["can_download"] is True
# The served file must be a neutral force-download attachment — never inline text/html.
fr = c.get(f"/api/downloads/{job['id']}/file")
assert fr.status_code == 200
assert fr.headers["content-type"].startswith("application/octet-stream")
assert "attachment" in fr.headers["content-disposition"].lower()
def test_upload_route_media_html_extension_not_served_as_html(
as_user, make_user, _root, monkeypatch
):
"""The XSS guard: a MEDIA-classified upload (real codecs) named with an executable extension must
NOT be served with a guessed text/html type. The served Content-Type comes from the verified
media_kind + a container allow-list ('html' isn't one), so it force-downloads as octet-stream —
on both the authenticated file endpoint AND the login-free public watch endpoint. Without the fix
the public inline branch let a media/HTML polyglot named evil.html run as stored XSS."""
_probe(monkeypatch, ("h264", "aac", 320, 240, 5)) # ffprobe says it's video…
c = as_user(make_user())
r = c.post(
"/api/downloads/upload",
files={"file": ("evil.html", b"<script>alert(1)</script>", "text/html")},
)
assert r.status_code == 200, r.text
job = r.json()
assert job["asset"]["media_kind"] == "video" and job["asset"]["container"] == "html"
# Authenticated endpoint: never text/html.
fr = c.get(f"/api/downloads/{job['id']}/file")
assert fr.status_code == 200
assert fr.headers["content-type"].startswith("application/octet-stream")
assert "attachment" in fr.headers["content-disposition"].lower()
# Public login-free endpoint via a STREAM-ONLY link (the inline branch) — still not text/html.
link = c.post(f"/api/downloads/{job['id']}/links", json={"allow_download": False}).json()
pr = c.get(f"/api/public/watch/{link['token']}/file")
assert pr.status_code == 200
assert not pr.headers["content-type"].startswith("text/html")
assert pr.headers["content-type"].startswith("application/octet-stream")
assert "attachment" in pr.headers["content-disposition"].lower()
def test_public_watch_file_forces_download_for_generic_file(as_user, make_user, _root, monkeypatch):
"""The login-free public serve of a generic upload (media_kind='file') must be a neutral
force-download attachment — the branch that stops uploaded HTML/SVG executing on the public
origin. This is a SEPARATE code path from the authenticated /file route, so it needs its own
route-level assertion (previously only the authenticated branch was tested)."""
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
job = c.post(
"/api/downloads/upload",
files={"file": ("notes.html", b"<script>alert(1)</script>", "text/html")},
).json()
assert job["asset"]["media_kind"] == "file"
link = c.post(f"/api/downloads/{job['id']}/links", json={"allow_download": True}).json()
pr = c.get(f"/api/public/watch/{link['token']}/file")
assert pr.status_code == 200
assert pr.headers["content-type"].startswith("application/octet-stream")
assert "attachment" in pr.headers["content-disposition"].lower()
def test_public_watch_file_serves_media_inline_with_real_type(
as_user, make_user, _root, monkeypatch
):
"""A recognised media container IS still served inline with its real audio/video type (so it
plays in the watch page) — the fix must not break legitimate playback. A stream-only link →
inline; the type is video/mp4 (from the container allow-list), never text/* and never a guess."""
_probe(monkeypatch, ("h264", "aac", 640, 360, 12))
c = as_user(make_user())
job = c.post(
"/api/downloads/upload",
files={"file": ("clip.mp4", b"fake-mp4-bytes", "video/mp4")},
).json()
link = c.post(f"/api/downloads/{job['id']}/links", json={"allow_download": False}).json()
pr = c.get(f"/api/public/watch/{link['token']}/file")
assert pr.status_code == 200
assert pr.headers["content-type"].startswith("video/mp4")
assert "inline" in pr.headers["content-disposition"].lower()
def test_upload_route_enforces_per_file_cap(as_user, make_user, db, _root, monkeypatch):
from app import sysconfig
sysconfig.set_value(db, "download_upload_max_bytes", 5)
db.commit()
c = as_user(make_user())
r = c.post(
"/api/downloads/upload",
files={"file": ("big.bin", b"way over the cap", "application/octet-stream")},
)
assert r.status_code == 422
assert r.json()["detail"]["code"] == "upload_too_large"
def test_upload_cap_cannot_be_disabled_with_zero(db):
"""download_upload_max_bytes floors at 1: a stored 0 would read falsy in the route's
`if cap and size > cap` guard and SILENTLY disable the per-file cap, so set_value must reject it.
Pins the min=1 invariant so a revert to min=0 can't pass the gate."""
from app import sysconfig
with pytest.raises(ValueError):
sysconfig.set_value(db, "download_upload_max_bytes", 0)
def test_upload_route_rejects_empty_file(as_user, make_user, _root, monkeypatch):
c = as_user(make_user())
r = c.post(
"/api/downloads/upload", files={"file": ("empty.bin", b"", "application/octet-stream")}
)
assert r.status_code == 400
assert r.json()["detail"]["code"] == "upload_empty"
def test_redownload_rejects_an_upload(as_user, make_user, _root, monkeypatch):
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
job = c.post(
"/api/downloads/upload", files={"file": ("a.bin", b"data", "application/octet-stream")}
).json()
r = c.post(f"/api/downloads/{job['id']}/redownload")
assert r.status_code == 409
def test_edit_rejects_a_generic_upload_source(as_user, make_user, _root, monkeypatch):
_probe(monkeypatch, (None, None, None, None, None))
c = as_user(make_user())
job = c.post(
"/api/downloads/upload", files={"file": ("a.bin", b"data", "application/octet-stream")}
).json()
r = c.post(
"/api/downloads/edit",
json={"source_job_id": job["id"], "edit_spec": {"trim": {"start_s": 0, "end_s": 1}}},
)
assert r.status_code == 400
assert r.json()["detail"]["reason"] == "not_media"
def test_ingest_rejects_over_max_jobs(db, make_user, _root, monkeypatch):
"""The other check_upload rejection branch (holdings count, not bytes): once the user is at
max_jobs, a further upload is refused before any disk write."""
_probe(monkeypatch, (None, None, None, None, None))
u = make_user()
db.add(
DownloadQuota(user_id=u.id, max_bytes=10**12, max_concurrent=2, max_jobs=1, unlimited=False)
)
db.commit()
src1 = _root / "a.bin"
src1.write_bytes(b"a")
uploadmod.ingest_upload(db, u.id, filename="a.bin", src_path=src1, size=1) # now 1 holding job
src2 = _root / "b.bin"
src2.write_bytes(b"b")
with pytest.raises(quota.QuotaExceeded) as e:
uploadmod.ingest_upload(db, u.id, filename="b.bin", src_path=src2, size=1)
assert e.value.reason == "max_jobs"
assert src2.exists() # rejected before the move — nothing landed
def test_ingest_cleans_up_placed_files_on_commit_failure(db, make_user, _root, monkeypatch):
"""A failure AFTER storage.place_file has moved the file onto disk must roll back AND unlink the
placed file + any (even partial) poster — otherwise the orphan is permanent (the per-user delete
keys off the rolled-back asset.rel_path, and the GC orphan sweep skips the dot-prefixed .uploads/
tree). Exercises the rollback/cleanup branch that IS the leak remediation, incl. the poster case
where ensure_poster left a partial file but returned None."""
_probe(monkeypatch, ("h264", "aac", 320, 240, 5)) # video → the poster path runs
u = make_user()
# Simulate ffmpeg leaving a partial poster on disk, then failing (ensure_poster returns None).
def _partial_poster(asset, root):
storage.abs_path(root, str(Path(asset.rel_path).with_suffix(".jpg"))).write_bytes(
b"partial"
)
return None
monkeypatch.setattr(uploadmod.editmod, "ensure_poster", _partial_poster)
# Force the commit to fail after the file (and partial poster) are on disk.
def _boom():
raise RuntimeError("db down")
monkeypatch.setattr(db, "commit", _boom)
src = _root / "clip.mp4"
src.write_bytes(b"fake-video-bytes")
with pytest.raises(RuntimeError):
uploadmod.ingest_upload(db, u.id, filename="clip.mp4", src_path=src, size=16)
db.rollback() # drop the monkeypatched-commit session state before querying
uploads = _root / ".uploads"
leftover = [p for p in uploads.rglob("*") if p.is_file()] if uploads.exists() else []
assert leftover == [], f"orphaned files left on disk: {leftover}"
assert db.query(MediaAsset).count() == 0 # the flushed asset rolled back — no dangling row
# --- classifier: the REAL probe_media parsing (stubs ffprobe one level down) ------------------
# The ingest/route tests above stub probe_media wholesale; these exercise its OWN logic — image
# detection, the cover-art skip, dimension/duration parsing — by faking only run_ffprobe_json, so a
# regression in the classifier the force-download guarantee leans on is actually caught.
class TestProbeClassifier:
def _probe_with(self, monkeypatch, data):
monkeypatch.setattr(probemod.editmod, "run_ffprobe_json", lambda *a, **k: data)
return probemod.probe_media(Path("x"))
def test_probe_failure_is_a_generic_file(self, monkeypatch):
# run_ffprobe_json returns None (ffprobe failed / not media) → all None → derive() == FILE.
v, a, w, h, d = self._probe_with(monkeypatch, None)
assert (v, a, w, h, d) == (None, None, None, None, None)
assert mediakind.derive(v, a) == mediakind.FILE
@pytest.mark.parametrize("fmt", ["image2", "png_pipe", "jpeg_pipe", "webp_pipe", "gif"])
def test_still_image_classifies_as_file_not_video(self, monkeypatch, fmt):
# A standalone image exposes a real video stream, but its container format is an image demuxer
# — it must NOT become media_kind='video' (broken player + inline serving). Stays a file.
data = {
"format": {"format_name": fmt},
"streams": [{"codec_type": "video", "codec_name": "png", "width": 8, "height": 8}],
}
v, a, w, h, d = self._probe_with(monkeypatch, data)
assert v is None and mediakind.derive(v, a) == mediakind.FILE
def test_cover_art_video_stream_is_skipped_audio_stays_audio(self, monkeypatch):
# Album art embedded in an audio file is a video stream flagged attached_pic — skip it so the
# asset classifies as audio, not video.
data = {
"format": {"format_name": "mp3", "duration": "212.5"},
"streams": [
{
"codec_type": "video",
"codec_name": "mjpeg",
"disposition": {"attached_pic": 1},
"width": 600,
"height": 600,
},
{"codec_type": "audio", "codec_name": "mp3"},
],
}
v, a, w, h, d = self._probe_with(monkeypatch, data)
assert (v, a, d) == (None, "mp3", 212)
assert mediakind.derive(v, a) == mediakind.AUDIO
def test_real_video_records_codecs_dims_duration(self, monkeypatch):
data = {
"format": {"format_name": "mov,mp4,m4a", "duration": "60.0"},
"streams": [
{"codec_type": "video", "codec_name": "h264", "width": 1920, "height": 1080},
{"codec_type": "audio", "codec_name": "aac"},
],
}
v, a, w, h, d = self._probe_with(monkeypatch, data)
assert (v, a, w, h, d) == ("h264", "aac", 1920, 1080, 60)
assert mediakind.derive(v, a) == mediakind.VIDEO
def test_unparseable_duration_is_none(self, monkeypatch):
data = {
"format": {"format_name": "wav", "duration": "N/A"},
"streams": [{"codec_type": "audio", "codec_name": "pcm_s16le"}],
}
v, a, w, h, d = self._probe_with(monkeypatch, data)
assert d is None and mediakind.derive(v, a) == mediakind.AUDIO
# --- resolve_media_facts: probe-of-the-file wins over yt-dlp's info (the FB regression fix) ---------
# The download worker's finalize must classify from the ACTUAL file, not the extractor's `info` dict.
# yt-dlp omits vcodec/acodec/dims for some sources (Facebook), and deriving media_kind from those
# nulls turned a real H.264 video into a download-only 'file' (no in-app play). These pin that the
# probe is authoritative and `info` is only a gap-filler.
class TestResolveMediaFacts:
# These stub run_ffprobe_json (one level below probe_media) so the REAL probe parsing + the
# probe-vs-info precedence are exercised together — the precedence keys off whether ffprobe RAN.
@staticmethod
def _ffprobe(monkeypatch, data):
monkeypatch.setattr(probemod.editmod, "run_ffprobe_json", lambda *a, **k: data)
def test_probe_wins_when_info_lacks_codecs(self, monkeypatch):
# THE original regression: info has no vcodec/acodec/dims (Facebook), but the file probes as
# h264/aac.
self._ffprobe(
monkeypatch,
{
"format": {"format_name": "mov,mp4,m4a"},
"streams": [
{"codec_type": "video", "codec_name": "h264", "width": 720, "height": 1280},
{"codec_type": "audio", "codec_name": "aac"},
],
},
)
v, a, w, h = probemod.resolve_media_facts(Path("x"), {"width": None, "height": None})
assert (v, a, w, h) == ("h264", "aac", 720, 1280)
assert mediakind.derive(v, a) == mediakind.VIDEO # NOT 'file'
def test_probe_wins_over_a_DIFFERING_populated_info(self, monkeypatch):
# Both the probe AND info are populated but DISAGREE (info carries a pre-remux/pre-merge codec).
# The finished file on disk is ground truth, so the probe wins outright — pins the precedence a
# `info or probe` inversion would silently break.
self._ffprobe(
monkeypatch,
{
"format": {"format_name": "mov,mp4,m4a"},
"streams": [
{"codec_type": "video", "codec_name": "h264", "width": 1920, "height": 1080},
{"codec_type": "audio", "codec_name": "aac"},
],
},
)
v, a, w, h = probemod.resolve_media_facts(
Path("x"), {"vcodec": "vp9", "acodec": "opus", "width": 640, "height": 480}
)
assert (v, a, w, h) == ("h264", "aac", 1920, 1080) # probe, not info
def test_cover_art_audio_download_is_not_resurrected_to_video_by_info(self, monkeypatch):
# THE bug this fix targets: an audio-mode download from a progressive source embeds cover art,
# and yt-dlp's info still carries the source's video codec. The probe correctly finds only audio
# (the attached_pic is skipped); info's vcodec must NOT resurrect a video track → media_kind
# stays audio, not a broken in-app 'video'.
self._ffprobe(
monkeypatch,
{
"format": {"format_name": "mp3", "duration": "180.0"},
"streams": [
{
"codec_type": "video",
"codec_name": "mjpeg",
"disposition": {"attached_pic": 1},
},
{"codec_type": "audio", "codec_name": "mp3"},
],
},
)
v, a, w, h = probemod.resolve_media_facts(
Path("x"), {"vcodec": "avc1", "acodec": "mp4a", "width": 1280, "height": 720}
)
assert v is None and a == "mp3" # info's avc1 did NOT win
assert mediakind.derive(v, a) == mediakind.AUDIO # NOT VIDEO
def test_still_image_download_is_not_resurrected_to_video_by_info(self, monkeypatch):
# A downloaded still image: the probe returns no vcodec (image-format guard). info's image
# "vcodec" must not turn it into a broken 'video' → stays a generic file.
self._ffprobe(
monkeypatch,
{
"format": {"format_name": "png_pipe"},
"streams": [{"codec_type": "video", "codec_name": "png", "width": 8, "height": 8}],
},
)
v, a, w, h = probemod.resolve_media_facts(
Path("x"), {"vcodec": "png", "width": 8, "height": 8}
)
assert v is None and mediakind.derive(v, a) == mediakind.FILE
def test_info_fills_in_only_when_ffprobe_could_not_read_the_file(self, monkeypatch):
# ffprobe failed entirely (None) → fall back to info so we never regress the working path.
self._ffprobe(monkeypatch, None)
v, a, w, h = probemod.resolve_media_facts(
Path("x"), {"vcodec": "vp9", "acodec": "opus", "width": 1920, "height": 1080}
)
assert (v, a, w, h) == ("vp9", "opus", 1920, 1080)
def test_info_none_codec_string_is_treated_as_absent(self, monkeypatch):
# yt-dlp writes the literal "none" for a missing stream — it must not become a codec name.
self._ffprobe(monkeypatch, None) # probe failed → info path, where the "none" scrub applies
v, a, w, h = probemod.resolve_media_facts(Path("x"), {"vcodec": "none", "acodec": "aac"})
assert (v, a) == (None, "aac")
assert mediakind.derive(v, a) == mediakind.AUDIO
+106 -4
View File
@@ -3,10 +3,11 @@ import { qk } from "../lib/queryKeys";
import { useTranslation } from "react-i18next";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Ban, Check, RotateCcw, Shield, Trash2, UserPlus, X } from "lucide-react";
import { api, type AdminUserRow, type Invite, type Me } from "../lib/api";
import { api, HttpError, type AdminUserRow, type Invite, type Me } from "../lib/api";
import { notify } from "../lib/notifications";
import { ADMIN_USERS_TAB_KEY } from "../lib/adminUsersTab";
import Tooltip from "./Tooltip";
import InlineNotice from "./InlineNotice";
import { Section } from "./ui/form";
import { LoadingState, StateMessage } from "./QueryState";
import { useConfirm } from "./ConfirmProvider";
@@ -266,6 +267,7 @@ function AdminInvites() {
const { t } = useTranslation();
const qc = useQueryClient();
const confirm = useConfirm();
const me = useMe();
const invites = useQuery({ queryKey: qk.adminInvites(), queryFn: () => api.adminInvites() });
const [newEmail, setNewEmail] = useState("");
const refresh = () => {
@@ -297,6 +299,45 @@ function AdminInvites() {
});
if (ok) deny.mutate(i.id);
};
const remove = useMutation({
mutationFn: (id: number) => api.deleteInvite(id),
onSuccess: (d) => {
refresh();
// Removing the row is not always a revoke: is_allowed falls back to the env allow-lists, so
// for an address on those the person keeps access. Saying "removed" alone would mislead.
if (d.still_allowed_by_env)
notify({ level: "warning", message: t("settings.invites.deletedStillAllowed") });
else notify({ level: "success", message: t("settings.invites.deleted") });
},
onError: (e) => {
// A 409 means the list was stale — someone registered on that address since it loaded. Say
// what to do instead of a generic failure, and refetch so the row re-renders as read-only.
const conflict = e instanceof HttpError && e.status === 409;
notify({
level: "error",
message: t(
conflict ? "settings.invites.deleteHasAccount" : "settings.invites.deleteFailed",
),
});
if (conflict) refresh();
},
});
const onDelete = async (i: Invite) => {
const ok = await confirm({
title: t("settings.invites.deleteTitle"),
// Removing the row is not a revoke when the address is also in the server's env allow-lists —
// say so while the admin can still change their mind, not in a toast after the fact.
message: t(
i.still_allowed_by_env
? "settings.invites.deleteConfirmStillAllowed"
: "settings.invites.deleteConfirm",
{ email: i.email },
),
confirmLabel: t("settings.invites.delete"),
danger: true,
});
if (ok) remove.mutate(i.id);
};
const add = useMutation({
mutationFn: (email: string) => api.addInvite(email),
onSuccess: (_d, email) => {
@@ -313,6 +354,9 @@ function AdminInvites() {
return (
<Section card title={t("settings.invites.title")}>
{!me.email_configured && (
<InlineNotice className="mb-3">{t("settings.invites.emailOff")}</InlineNotice>
)}
{invites.isError && !invites.data ? (
<StateMessage tone="error" onRetry={() => invites.refetch()}>
{t("common.loadError")}
@@ -329,7 +373,8 @@ function AdminInvites() {
inv={i}
onApprove={() => approve.mutate({ id: i.id, email: i.email })}
onDeny={() => onDeny(i)}
busy={approve.isPending || deny.isPending}
onDelete={() => onDelete(i)}
busy={approve.isPending || deny.isPending || remove.isPending}
/>
))}
</div>
@@ -364,11 +409,38 @@ function AdminInvites() {
</summary>
<div className="mt-2 space-y-1 text-xs text-muted">
{decided.map((i) => (
<div key={i.id} className="flex items-center justify-between gap-2">
<span className="truncate">{i.email}</span>
<div key={i.id} className="flex items-center gap-2">
<span className="truncate flex-1">{i.email}</span>
<span className={i.status === "approved" ? "text-accent" : "text-danger"}>
{i.status}
</span>
{/* An entry with an account behind it stays read-only: deleting it would revoke that
user's access, since is_allowed reads this row. Delete the user instead. */}
{i.has_account ? (
// `disabled` would drop it from the tab order AND suppress the title, so the
// reason reached nobody — same call as the Playlists drag handle.
<button
type="button"
aria-disabled
onClick={(e) => e.preventDefault()}
title={t("settings.invites.deleteHasAccount")}
aria-label={t("settings.invites.deleteHasAccount")}
className="shrink-0 p-1 text-muted/50 cursor-not-allowed"
>
<Trash2 className="w-3.5 h-3.5" />
</button>
) : (
<Tooltip hint={t("settings.invites.deleteHint")}>
<button
onClick={() => onDelete(i)}
disabled={remove.isPending}
className="shrink-0 p-1 rounded-md text-muted hover:text-danger disabled:opacity-50 transition"
aria-label={t("settings.invites.deleteLabel", { email: i.email })}
>
<Trash2 className="w-3.5 h-3.5" />
</button>
</Tooltip>
)}
</div>
))}
</div>
@@ -506,11 +578,13 @@ function InviteRow({
inv,
onApprove,
onDeny,
onDelete,
busy,
}: {
inv: Invite;
onApprove: () => void;
onDeny: () => void;
onDelete: () => void;
busy: boolean;
}) {
const { t } = useTranslation();
@@ -546,6 +620,34 @@ function InviteRow({
<X className="w-4 h-4" />
</button>
</Tooltip>
{/* Deny records a decision and keeps the row; delete forgets the address. Once an account
exists for it the row is that account's approval gate, so it is managed from Users & roles
instead — shown as a disabled control rather than omitted, or the admin sees a row that is
silently missing a button their other rows have. */}
{inv.has_account ? (
// aria-disabled, not `disabled`: the reason has to stay reachable by keyboard.
<button
type="button"
aria-disabled
onClick={(e) => e.preventDefault()}
title={t("settings.invites.deleteHasAccount")}
aria-label={t("settings.invites.deleteHasAccount")}
className="shrink-0 p-1.5 rounded-lg border border-border/50 text-muted/50 cursor-not-allowed"
>
<Trash2 className="w-4 h-4" />
</button>
) : (
<Tooltip hint={t("settings.invites.deleteHint")}>
<button
onClick={onDelete}
disabled={busy}
className="shrink-0 p-1.5 rounded-lg border border-border text-muted hover:text-danger disabled:opacity-50 transition"
aria-label={t("settings.invites.deleteLabel", { email: inv.email })}
>
<Trash2 className="w-4 h-4" />
</button>
</Tooltip>
)}
</div>
);
}
+46 -16
View File
@@ -12,6 +12,8 @@ import {
Ban,
Check,
ExternalLink,
Eye,
EyeOff,
History,
Loader2,
Plus,
@@ -32,6 +34,7 @@ import type { FeedView } from "../lib/feedView";
import { channelYouTubeUrl, formatViews } from "../lib/format";
import { ChannelTagsControl } from "./ChannelTags";
import { useChannelTagActions } from "../lib/useChannelTagActions";
import { Button } from "./ui/Button";
// A dedicated channel page: an "About"-style header (banner, avatar, stats, subscribe) over the
// channel's videos (the catalog filtered to this channel). For an un-subscribed channel it
@@ -137,6 +140,17 @@ export default function ChannelPage({
// a caller toast here would just double it.
});
// "Hide from feed": the same per-subscription flag the manager toggles (updateChannel {hidden}).
// Local DB op — keeps the subscription, only drops the channel's videos from the feed/explore.
const toggleHidden = useMutation({
mutationFn: () => api.updateChannel(channelId, { hidden: !ch?.hidden }),
onSuccess: () => {
qc.invalidateQueries({ queryKey: qk.channel(channelId) });
qc.invalidateQueries({ queryKey: qk.feed() });
qc.invalidateQueries({ queryKey: qk.channels() });
},
});
const subscribe = useMutation({
mutationFn: () => api.subscribeChannel(channelId),
onSuccess: () => {
@@ -418,6 +432,7 @@ export default function ChannelPage({
priority={ch.priority}
onPriority={(d) => bumpPriority(channelId, d)}
onSet={(v) => setPriority(channelId, v)}
size="md"
/>
</div>
)}
@@ -427,21 +442,18 @@ export default function ChannelPage({
it) so a large channel's paced tail reads as visible progress rather than a stuck
spinner. */}
{ch?.subscribed && !ch.backfill_done && !me.is_demo && (
<button
<Button
variant="outline"
onClick={() => !ch.deep_requested && deepBackfill.mutate()}
disabled={ch.deep_requested || deepBackfill.isPending}
loading={ch.deep_requested}
icon={<History className="w-4 h-4" />}
title={
ch.deep_requested
? t("channel.fullHistoryLoadingHint")
: t("channel.fullHistory")
}
className="inline-flex items-center gap-1.5 px-3 py-2 rounded-lg text-sm border border-border text-muted hover:text-accent hover:border-accent disabled:opacity-60 disabled:hover:text-muted disabled:hover:border-border transition"
>
{ch.deep_requested ? (
<Loader2 className="w-4 h-4 animate-spin" />
) : (
<History className="w-4 h-4" />
)}
<span className="hidden sm:inline whitespace-nowrap">
{!ch.deep_requested
? t("channel.fullHistory")
@@ -452,7 +464,7 @@ export default function ChannelPage({
})
: t("channel.fullHistoryLoading")}
</span>
</button>
</Button>
)}
{!me.is_demo && (
<button
@@ -469,27 +481,45 @@ export default function ChannelPage({
<Ban className="w-4 h-4" />
</button>
)}
{/* Hide-from-feed toggle — subscription-scoped, so offered only on a followed channel
(mirrors the manager's Eye/EyeOff). Keeps the subscription; only drops its videos
from the feed. EyeOff = currently hidden (click to unhide). */}
{ch?.subscribed && !me.is_demo && (
<button
onClick={() => toggleHidden.mutate()}
disabled={toggleHidden.isPending}
title={ch.hidden ? t("channel.hiddenHint") : t("channel.hideHint")}
aria-label={ch.hidden ? t("channel.unhide") : t("channel.hide")}
className={`inline-flex items-center justify-center w-9 h-9 rounded-lg border disabled:opacity-50 transition ${
ch.hidden
? "border-accent text-accent"
: "border-border text-muted hover:text-fg hover:border-fg/40"
}`}
>
{ch.hidden ? <EyeOff className="w-4 h-4" /> : <Eye className="w-4 h-4" />}
</button>
)}
{!me.is_demo &&
!ch?.blocked &&
(ch?.subscribed ? (
<button
<Button
variant="outline"
onClick={onUnsubscribe}
disabled={unsubscribe.isPending}
title={t("channel.unsubscribe")}
className="inline-flex items-center gap-1.5 px-3 py-2 rounded-lg text-sm border border-border text-muted hover:text-fg disabled:opacity-50"
icon={<Check className="w-4 h-4" />}
>
<Check className="w-4 h-4" />
{t("channel.subscribedState")}
</button>
</Button>
) : (
<button
<Button
variant="outline"
onClick={() => subscribe.mutate()}
disabled={subscribe.isPending}
className="inline-flex items-center gap-1.5 px-3 py-2 rounded-lg text-sm btn-primary disabled:opacity-50"
icon={<Plus className="w-4 h-4" />}
>
<Plus className="w-4 h-4" />
{t("channel.subscribe")}
</button>
</Button>
))}
{/* Tagging is a subscription-scoped action (attach_tag → _user_subscription 404s
otherwise), and personal tags are for your subscriptions — so only offer it on a
@@ -0,0 +1,105 @@
// @vitest-environment jsdom
//
// Contract of the suggested-tag ghost chips (E8·S4): on an UNTAGGED channel, ChannelTagsControl renders
// one dashed "ghost" chip per suggestion; clicking the chip body accepts it (onAcceptSuggestion with the
// tag id), the trailing × dismisses it (onDismissSuggestion), and a suggestion whose tag is already
// attached is never shown. i18n is key-passthrough; the control rides the floating-ui popover platform
// and a ResizeObserver-driven fit measurer, so we stub the observer and wrap in a FloatingTree.
import { describe, it, expect, afterEach, beforeAll, vi } from "vitest";
import { render, screen, fireEvent, cleanup } from "@testing-library/react";
import { FloatingTree } from "@floating-ui/react";
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
// Interpolate the `name` option so per-tag accept labels are distinguishable ("…acceptSuggestion:docu").
return {
...actual,
useTranslation: () => ({
t: (key: string, opts?: { name?: string }) => (opts?.name ? `${key}:${opts.name}` : key),
i18n: { language: "en" },
}),
};
});
import { ChannelTagsControl, type SuggestedTag } from "./ChannelTags";
import type { Tag } from "../lib/api";
beforeAll(() => {
class Stub {
observe() {}
unobserve() {}
disconnect() {}
takeRecords() {
return [];
}
}
globalThis.ResizeObserver ??= Stub as unknown as typeof ResizeObserver;
});
afterEach(cleanup);
const tag = (id: number, name: string): Tag => ({
id,
name,
color: null,
category: "other",
system: false,
channel_count: 0,
});
const docu = tag(1, "docu");
const history = tag(2, "history");
const suggestions: SuggestedTag[] = [
{ tag: docu, confidence: 0.95 },
{ tag: history, confidence: 0.6 },
];
function renderControl(props: Partial<Parameters<typeof ChannelTagsControl>[0]> = {}) {
const onAccept = vi.fn();
const onDismiss = vi.fn();
render(
<FloatingTree>
<ChannelTagsControl
userTags={[docu, history]}
attachedIds={[]}
onToggle={() => {}}
onCreate={() => {}}
suggestions={suggestions}
onAcceptSuggestion={onAccept}
onDismissSuggestion={onDismiss}
{...props}
/>
</FloatingTree>,
);
return { onAccept, onDismiss };
}
describe("ChannelTagsControl suggested ghost chips", () => {
it("renders a ghost chip per suggestion and accepts on click", () => {
const { onAccept } = renderControl();
// Role query excludes the aria-hidden fit-measurer copy; the name carries the interpolated tag.
fireEvent.click(screen.getByRole("button", { name: "channels.tags.acceptSuggestion:docu" }));
expect(onAccept).toHaveBeenCalledWith(docu.id);
});
it("dismisses via the × without accepting", () => {
const { onAccept, onDismiss } = renderControl();
const dismissButtons = screen.getAllByRole("button", {
name: "channels.tags.dismissSuggestion",
});
// Two visible ghost chips (the measurer's copies are aria-hidden → excluded); order matches the
// suggestions array, so the second × is the "history" suggestion.
expect(dismissButtons).toHaveLength(2);
fireEvent.click(dismissButtons[1]!);
expect(onDismiss).toHaveBeenCalledWith(history.id);
expect(onAccept).not.toHaveBeenCalled();
});
it("shows the attached lane (no ghost chips) once the channel carries a user tag", () => {
renderControl({ attachedIds: [docu.id] });
// A tagged channel is not "untagged" → the ghost lane is not rendered at all (the server excludes
// tagged channels from suggestions; the control mirrors that with its attached-XOR-ghost branch).
expect(
screen.queryAllByRole("button", { name: "channels.tags.dismissSuggestion" }),
).toHaveLength(0);
});
});
+239 -52
View File
@@ -1,5 +1,5 @@
import { useEffect, useRef, useState } from "react";
import { Check, Plus } from "lucide-react";
import { Check, Plus, Sparkles, X } from "lucide-react";
import { useTranslation } from "react-i18next";
import { type Tag } from "../lib/api";
import { useAnchoredPopover, PopoverSurface, PopoverItem } from "./AnchoredPopover";
@@ -9,8 +9,8 @@ import { useChipOverflowFit } from "../lib/useChipOverflowFit";
// The channel tag surface, shared by the channel manager (table + cards) and the channel detail
// page. Built on the R8 popover platform. Two design goals it fixes over the old inline TagsCell:
// 1. Stable edit anchor — the "+" edit trigger sits in a FIXED slot at the end, so toggling tags
// (which grows the chip row) never moves it, and the editor popover doesn't drift.
// 1. Stable edit anchor — the "+" edit trigger LEADS the row (fixed left slot), so toggling tags
// (which grows the chip row to its right) never moves it, and the editor popover doesn't drift.
// 2. Fixed footprint — the chip row lives in a width-bounded, single-line, overflow-hidden box and
// collapses the excess into a "+N" pill (hover reveals the full set). The cell therefore never
// wraps to a 2nd line, so it can't widen its table column and reflow every other row.
@@ -40,6 +40,157 @@ function TagChip({ tag, onClick, title }: TagChipProps) {
);
}
// A suggested user tag (E8·S4): a dashed "ghost" chip the user hasn't applied yet, learned from
// their own tagging pattern. The chip body accepts it (one tap → attaches the real tag); the trailing
// × dismisses it for the session. Rendered only on untagged channels, so it never competes with the
// measured attached-chip row. (Two sibling buttons, not a button-in-button — invalid nesting.)
export interface SuggestedTag {
tag: Tag;
confidence: number;
}
function SuggestedChip({
suggestion,
onAccept,
onDismiss,
}: {
suggestion: SuggestedTag;
onAccept: () => void;
onDismiss: () => void;
}) {
const { t } = useTranslation();
const pct = Math.round(suggestion.confidence * 100);
return (
<span className="shrink-0 inline-flex items-center rounded-full border border-dashed border-accent/50 text-accent/90 text-2xs font-medium pl-1 pr-0.5 py-0.5 max-w-36">
<button
type="button"
onClick={onAccept}
title={t("channels.tags.suggestedHint", { pct })}
aria-label={t("channels.tags.acceptSuggestion", { name: suggestion.tag.name })}
className="inline-flex items-center gap-1 min-w-0 hover:text-accent transition"
>
<Sparkles className="w-2.5 h-2.5 shrink-0" />
<span className="truncate">{suggestion.tag.name}</span>
</button>
<button
type="button"
onClick={onDismiss}
title={t("channels.tags.dismissSuggestion")}
aria-label={t("channels.tags.dismissSuggestion")}
className="shrink-0 ml-0.5 rounded-full text-accent/60 hover:text-danger transition"
>
<X className="w-2.5 h-2.5" />
</button>
</span>
);
}
const GHOST_PILL =
"shrink-0 text-3xs px-1.5 py-0.5 rounded-full border border-dashed border-accent/50 text-accent/90";
/** A single-line lane of suggested ghost chips that collapses its overflow into a "+N" pill (hover
* reveals the rest, still accept/dismiss-able), reusing the SAME fit machinery as the attached-chip
* lane. Rendered in the untagged-channel case only, so it owns the full tags-cell width — this is what
* stops a 2nd suggestion from being clipped mid-word right next to the edit "+". */
function SuggestedChipsLane({
suggestions,
onAccept,
onDismiss,
}: {
suggestions: SuggestedTag[];
onAccept: (tagId: number) => void;
onDismiss: (tagId: number) => void;
}) {
const { t } = useTranslation();
const rowRef = useRef<HTMLDivElement | null>(null);
const chipRefs = useRef<Array<HTMLElement | null>>([]);
const pillRef = useRef<HTMLSpanElement | null>(null);
const key = suggestions.map((s) => `${s.tag.id}:${s.tag.name}`).join("|");
const visible = useChipOverflowFit<number>({
rowRef,
chipsRef: chipRefs,
pillRef,
deps: key,
initial: suggestions.length,
decide: (g) =>
fitChipsWithPill({ chipRights: g.chipRights, pillW: g.pillW, avail: g.avail, gap: g.gap }),
});
const hidden = suggestions.length - visible;
const [overflowOpen, setOverflowOpen] = useState(false);
const overflowPop = useAnchoredPopover({
open: overflowOpen,
onOpenChange: setOverflowOpen,
mode: "hover",
placement: "bottom",
});
useEffect(() => {
if (hidden === 0 && overflowOpen) setOverflowOpen(false);
}, [hidden, overflowOpen]);
return (
<div className="relative flex items-center gap-1 min-w-0 flex-1">
{/* Hidden measurer — real ghost chips at natural width so the fit hook reads their right edges. */}
<div
aria-hidden
className="pointer-events-none invisible absolute left-0 top-0 flex items-center gap-1 whitespace-nowrap"
>
{suggestions.map((s, i) => (
<span
key={s.tag.id}
ref={(el) => {
chipRefs.current[i] = el;
}}
className="inline-flex"
>
<SuggestedChip suggestion={s} onAccept={() => {}} onDismiss={() => {}} />
</span>
))}
<span ref={pillRef} className={GHOST_PILL}>
+{suggestions.length}
</span>
</div>
<div ref={rowRef} className="flex items-center gap-1 min-w-0 flex-1 overflow-hidden">
{suggestions.slice(0, visible).map((s) => (
<SuggestedChip
key={s.tag.id}
suggestion={s}
onAccept={() => onAccept(s.tag.id)}
onDismiss={() => onDismiss(s.tag.id)}
/>
))}
{hidden > 0 && (
<button
ref={overflowPop.refs.setReference}
type="button"
aria-label={t("channels.tags.moreCount", { count: hidden })}
className={`${GHOST_PILL} hover:text-accent transition`}
{...overflowPop.getReferenceProps()}
>
+{hidden}
</button>
)}
</div>
{/* Hover: the overflowed suggestions, each still one-tap accept / × dismiss. */}
<PopoverSurface
popover={overflowPop}
className="glass-menu max-w-[18rem] rounded-xl p-1.5 flex flex-wrap items-center gap-1 animate-[popIn_0.16s_ease]"
>
{suggestions.slice(visible).map((s) => (
<SuggestedChip
key={s.tag.id}
suggestion={s}
onAccept={() => onAccept(s.tag.id)}
onDismiss={() => onDismiss(s.tag.id)}
/>
))}
</PopoverSurface>
</div>
);
}
/** The editor body: toggle the user's existing tags on/off for this channel, plus an inline
* "create + attach" input. Rendered inside an anchored `menu` popover. (Exported for the channel
* detail page in Part 2; internal for now.) */
@@ -130,15 +281,18 @@ function TagEditorMenu({
);
}
/** The full tag control for a channel: a width-bounded chip row with "+N" overflow (hover reveals
* all), followed by a stable "+" trigger that opens the shared editor. Fills its container width,
* so a table cell bounds it via the column width and a card bounds it via the card. */
/** The full tag control for a channel: a stable leading "+" trigger (opens the shared editor),
* followed by a width-bounded chip row with "+N" overflow (hover reveals all). Fills its container
* width, so a table cell bounds it via the column width and a card bounds it via the card. */
export function ChannelTagsControl({
userTags,
attachedIds,
onToggle,
onCreate,
onChipClick,
suggestions,
onAcceptSuggestion,
onDismissSuggestion,
bounded = true,
}: {
userTags: Tag[];
@@ -147,6 +301,11 @@ export function ChannelTagsControl({
onCreate: (name: string) => void;
/** Manager only: clicking a chip filters the feed by that tag. Omit on the detail page. */
onChipClick?: (tagId: number, name: string) => void;
/** Manager only (E8·S4): suggested user tags for an UNTAGGED channel, shown as accept/dismiss ghost
* chips after the (empty) attached row. Omit on the detail page. */
suggestions?: SuggestedTag[];
onAcceptSuggestion?: (tagId: number) => void;
onDismissSuggestion?: (tagId: number) => void;
/** Cap the chip row at a fixed 12rem so it can't widen a `table-layout:auto` column and reflow the
* table (the manager). Set false where there is free width (the channel detail header), so tags
* fit inline instead of needlessly collapsing into "+N". */
@@ -154,6 +313,9 @@ export function ChannelTagsControl({
}) {
const { t } = useTranslation();
const attached = userTags.filter((tg) => attachedIds.includes(tg.id));
// Suggestions only surface for channels with no user tag yet, but guard anyway: never suggest a tag
// the channel already carries (an optimistic accept adds it before the query refetches).
const shownSuggestions = (suggestions ?? []).filter((s) => !attachedIds.includes(s.tag.id));
// Deterministic single-line fit: a hidden measurer lays out ALL chips + a sample "+N" pill at their
// natural widths; the shared measure core (useChipOverflowFit) reads each chip's right edge, the rem
@@ -214,61 +376,86 @@ export function ChannelTagsControl({
<div
className={`relative flex items-center gap-1 min-w-0 ${bounded ? "w-48 max-w-full" : "w-full"}`}
>
{/* Hidden measurer: all chips + a sample pill, laid out unbounded to read their true widths. Each
chip is wrapped in a ref span (index-aligned) so the shared fit hook can read its right edge;
the pill is the sample the hook reserves room for. */}
<div
aria-hidden
className="pointer-events-none invisible absolute left-0 top-0 flex items-center gap-1 whitespace-nowrap"
>
{attached.map((tg, i) => (
<span
key={tg.id}
ref={(el) => {
chipRefs.current[i] = el;
}}
className="inline-flex"
>
<TagChip tag={tg} />
</span>
))}
<span ref={pillRef} className={pillClass}>
+{attached.length}
</span>
</div>
<div ref={rowRef} className="flex items-center gap-1 min-w-0 flex-1 overflow-hidden">
{attached.slice(0, visible).map((tg) => (
<TagChip
key={tg.id}
tag={tg}
onClick={onChipClick ? () => onChipClick(tg.id, tg.name) : undefined}
title={onChipClick ? t("channels.row.filterFeedByTag", { name: tg.name }) : undefined}
/>
))}
{hidden > 0 && (
<button
ref={overflowPop.refs.setReference}
type="button"
className={`${pillClass} hover:text-fg transition`}
{...overflowPop.getReferenceProps()}
>
+{hidden}
</button>
)}
</div>
{/* The add trigger LEADS the row — a stable left anchor that never moves as chips are added
(they grow to its right), so its editor popover can't drift. Compact icon in the width-
bounded manager cell; a labelled "+ Tag" pill where there's free width (the detail header). */}
<button
ref={editPop.refs.setReference}
type="button"
title={t("channels.row.editTags")}
aria-label={t("channels.row.editTags")}
className="shrink-0 w-5 h-5 inline-flex items-center justify-center rounded-full border border-dashed border-border text-muted hover:text-accent hover:border-accent transition"
className={
bounded
? "shrink-0 w-5 h-5 inline-flex items-center justify-center rounded-full border border-dashed border-border text-muted hover:text-accent hover:border-accent transition"
: "shrink-0 inline-flex items-center gap-1 h-6 pl-1.5 pr-2 rounded-full border border-dashed border-border text-2xs font-medium text-muted hover:text-accent hover:border-accent transition"
}
{...editPop.getReferenceProps()}
>
<Plus className="w-3 h-3" />
<Plus className="w-3 h-3 shrink-0" />
{!bounded && <span className="whitespace-nowrap">{t("channels.tags.add")}</span>}
</button>
{/* Untagged + has suggestions → the ghost-chip lane (its own overflow "+N"); otherwise the
attached-chip lane. The two are mutually exclusive: suggestions are computed for untagged
channels only, and accepting one removes the channel from the suggestion set — so a channel
never shows both, and each lane gets the full cell width instead of clipping the other. */}
{attached.length === 0 && shownSuggestions.length > 0 ? (
<SuggestedChipsLane
suggestions={shownSuggestions}
onAccept={(tagId) => onAcceptSuggestion?.(tagId)}
onDismiss={(tagId) => onDismissSuggestion?.(tagId)}
/>
) : (
<>
{/* Hidden measurer: all chips + a sample pill, laid out unbounded to read their true widths.
Each chip is wrapped in a ref span (index-aligned) so the shared fit hook can read its
right edge; the pill is the sample the hook reserves room for. */}
<div
aria-hidden
className="pointer-events-none invisible absolute left-0 top-0 flex items-center gap-1 whitespace-nowrap"
>
{attached.map((tg, i) => (
<span
key={tg.id}
ref={(el) => {
chipRefs.current[i] = el;
}}
className="inline-flex"
>
<TagChip tag={tg} />
</span>
))}
<span ref={pillRef} className={pillClass}>
+{attached.length}
</span>
</div>
<div ref={rowRef} className="flex items-center gap-1 min-w-0 flex-1 overflow-hidden">
{attached.slice(0, visible).map((tg) => (
<TagChip
key={tg.id}
tag={tg}
onClick={onChipClick ? () => onChipClick(tg.id, tg.name) : undefined}
title={
onChipClick ? t("channels.row.filterFeedByTag", { name: tg.name }) : undefined
}
/>
))}
{hidden > 0 && (
<button
ref={overflowPop.refs.setReference}
type="button"
aria-label={t("channels.tags.moreCount", { count: hidden })}
className={`${pillClass} hover:text-fg transition`}
{...overflowPop.getReferenceProps()}
>
+{hidden}
</button>
)}
</div>
</>
)}
{/* Hover: reveal every attached tag when some are collapsed into "+N". */}
<PopoverSurface
popover={overflowPop}
+108 -41
View File
@@ -4,7 +4,8 @@ import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { qk } from "../lib/queryKeys";
import { Ban, Eye, EyeOff, History, LayoutGrid, Pencil, RefreshCw, Table, X } from "lucide-react";
import ExcludedChannelList from "./ExcludedChannelList";
import { api, type ManagedChannel } from "../lib/api";
import { api, NO_TAG_SENTINEL, type ManagedChannel } from "../lib/api";
import { Sparkles } from "lucide-react";
import { notifyYouTubeActionError } from "../lib/youtubeErrors";
import { accountKey, LS } from "../lib/storage";
import { buildChannelColumns, applySyncFilter } from "./channelColumns";
@@ -15,10 +16,12 @@ import Tabs, { type TabDef } from "./Tabs";
import ChannelLayoutGrid from "./ChannelLayoutGrid";
import { LoadingState, StateMessage } from "./QueryState";
import { useChannelTagActions } from "../lib/useChannelTagActions";
import { useTagSuggestions } from "./useTagSuggestions";
import { useChannelPriority } from "../lib/useChannelPriority";
import ViewSwitcher, { type ViewOption } from "./ViewSwitcher";
import ColumnSortControl from "./ColumnSortControl";
import FilterChipGroup, { filterChipClass } from "./FilterChipGroup";
import FilterChipGroup from "./FilterChipGroup";
import TagFilterChips, { type FilterChip } from "./TagFilterChips";
import ActionMenu, { type ActionMenuItem } from "./ActionMenu";
import ChannelStatsPopover from "./ChannelStatsPopover";
import Pager from "./Pager";
@@ -164,10 +167,37 @@ export default function Channels() {
),
),
// Keep the channel DETAIL cache in step with the manager list (the detail page reads its tags
// from qk.channel(id)); mirrors the detail page invalidating the manager list on its side.
onSuccess: (id) => qc.invalidateQueries({ queryKey: qk.channel(id) }),
onError: () => qc.invalidateQueries({ queryKey: qk.channels() }),
// from qk.channel(id)); mirrors the detail page invalidating the manager list on its side. Also
// recompute suggestions: a channel that just gained/lost a user tag changes the untagged set and
// the learned pattern (E8·S4).
onSuccess: (id) => {
qc.invalidateQueries({ queryKey: qk.channel(id) });
qc.invalidateQueries({ queryKey: qk.tagSuggestions() });
},
// A failed attach/detach reverts the optimistic change by refetching the list. Also refetch
// suggestions: acceptSuggestion optimistically dropped the channel from that cache, and on
// failure the tag never actually attached — so the server still suggests it and the ghost chips
// must come back rather than leaving an empty cell (honest states). Surface the write failure too.
onError: () => {
qc.invalidateQueries({ queryKey: qk.channels() });
qc.invalidateQueries({ queryKey: qk.tagSuggestions() });
notify({ level: "error", message: t("channels.tags.updateFailed") });
},
});
// --- Suggested tags (E8·S4): learned from the user's own manual-tag pattern, shown as accept/dismiss
// ghost chips on untagged channels. Dismissal is session-only (an accepted suggestion self-clears —
// the channel becomes tagged and the server drops it; a dismissed one just hides until reload). The
// /tags/suggestions query, the session dismiss set, and the optimistic accept/dismiss/bulk logic all
// live in useTagSuggestions so they can be unit-tested apart from this component.
const {
suggestionsFor,
dismissSuggestion,
acceptSuggestion,
visibleSuggestions,
highConfidence,
acceptAllHighConfidence,
} = useTagSuggestions({ tags: tagsQuery.data ?? [], toggleTag });
const syncSubs = useMutation({
mutationFn: () => api.syncSubscriptions(),
onSuccess: (r: { subscriptions?: number }) => {
@@ -217,11 +247,34 @@ export default function Channels() {
// funnel share ONE state and stay in sync (bidirectional). Toggling a chip rewrites `filters.tags`;
// filterRows / DataTable then applies it. Channel-name search stays separate (the shared top box).
const tagFilter = Array.isArray(filters.tags) ? filters.tags : [];
// Picking a real tag drops the "No tag" sentinel (item B) — they're mutually exclusive, as in the feed.
const toggleTagFilter = (tagId: number) => {
const id = String(tagId);
const next = tagFilter.includes(id) ? tagFilter.filter((x) => x !== id) : [...tagFilter, id];
const base = tagFilter.filter((x) => x !== NO_TAG_SENTINEL);
const next = base.includes(id) ? base.filter((x) => x !== id) : [...base, id];
setFilters({ ...filters, tags: next });
};
// The "No tag" chip: filter the table to channels you haven't personally tagged yet. Exclusive with
// real tag chips (clicking it drops them); the Tags column's test reads the sentinel (channelColumns).
const noTagOn = tagFilter.includes(NO_TAG_SENTINEL);
const toggleNoTagFilter = () =>
setFilters({ ...filters, tags: noTagOn ? [] : [NO_TAG_SENTINEL] });
// The tag-filter chips (user tags + the "No tag" sentinel), fed to TagFilterChips which folds any
// overflow into a "+N" pill so the row stays one line and the trailing "…" actions can't wrap.
const tagFilterChips: FilterChip[] = [
...userTags.map((tg) => ({
key: `t${tg.id}`,
label: tg.name,
active: tagFilter.includes(String(tg.id)),
onToggle: () => toggleTagFilter(tg.id),
})),
{
key: NO_TAG_SENTINEL,
label: t("channels.tags.noTag"),
active: noTagOn,
onToggle: toggleNoTagFilter,
},
];
// A focus-channel intent (header "without full history" / tag manager) seeds the search box.
useEffect(() => {
if (focusChannelName) setSearch(focusChannelName);
@@ -295,6 +348,9 @@ export default function Channels() {
toggleTag,
createAndAttachTag,
onFilterByTag,
suggestionsFor,
onAcceptSuggestion: acceptSuggestion,
onDismissSuggestion: dismissSuggestion,
});
// Status chips sit in the table's controls row (next to paging) — compact and close to
@@ -426,41 +482,52 @@ export default function Channels() {
{tabs}
{view === "subscribed" && (
<div className="px-4 pt-2 pb-2 max-w-384 mx-auto">
{/* Your tags (left) share one row with the two catalog-wide actions (right); the channel-name
search lives in the shared top SearchBar, tag create/delete in the manager. */}
<div className="flex items-center justify-between gap-4 flex-wrap">
<div className="flex flex-wrap items-center gap-1.5">
{/* The "TAGS" label IS the tags-editor trigger (pencil), replacing the separate
"Manage tags" button; the tooltip explains what tags are + that it's clickable. */}
<Tooltip hint={t("channels.tags.yourTagsHint")}>
<button
onClick={() => setTagManagerOpen(true)}
className="mr-1 inline-flex items-center gap-1.5 rounded-lg border border-border px-2 py-1 text-xs uppercase tracking-wide text-muted transition hover:border-accent hover:text-accent"
>
<Pencil className="h-3 w-3" />
{t("channels.tags.label")}
</button>
</Tooltip>
{userTags.map((tg) => (
<button
key={tg.id}
onClick={() => toggleTagFilter(tg.id)}
aria-pressed={tagFilter.includes(String(tg.id))}
className={filterChipClass}
>
{tg.name}
</button>
))}
{tagFilter.length > 0 && (
<button
onClick={() => setFilters({ ...filters, tags: [] })}
className="text-xs px-2 py-1 text-muted hover:text-accent transition"
>
{t("datatable.clear")}
</button>
)}
</div>
{actionButtons}
{/* One non-wrapping row: the TAGS editor trigger (left) + the tag-filter chip lane (folds
overflow into "+N") + Clear, then the catalog "…" actions pinned right. It never wraps, so the
"…" menu always stays on the first line. Channel-name search lives in the shared top SearchBar. */}
<div className="flex items-center gap-2 min-w-0">
{/* The "TAGS" label IS the tags-editor trigger (pencil), replacing the separate
"Manage tags" button; the tooltip explains what tags are + that it's clickable. */}
<Tooltip hint={t("channels.tags.yourTagsHint")}>
<button
onClick={() => setTagManagerOpen(true)}
className="shrink-0 inline-flex items-center gap-1.5 rounded-lg border border-border px-2 py-1 text-xs uppercase tracking-wide text-muted transition hover:border-accent hover:text-accent"
>
<Pencil className="h-3 w-3" />
{t("channels.tags.label")}
</button>
</Tooltip>
<TagFilterChips chips={tagFilterChips} className="flex-1" />
{tagFilter.length > 0 && (
<button
onClick={() => setFilters({ ...filters, tags: [] })}
className="shrink-0 text-xs px-2 py-1 text-muted hover:text-accent transition"
>
{t("datatable.clear")}
</button>
)}
{/* Suggested-tags affordance (E8·S4): a sparkle count of the visible suggestions, with a
one-click bulk-accept for the high-confidence ones. The per-row ghost chips handle the
rest. Hidden when there's nothing to suggest. */}
{visibleSuggestions.length > 0 && (
<div className="shrink-0 flex items-center gap-1.5">
<Tooltip hint={t("channels.tags.suggestionsHint")}>
<span className="inline-flex items-center gap-1 rounded-lg border border-dashed border-accent/50 px-2 py-1 text-xs text-accent/90">
<Sparkles className="h-3 w-3" />
{t("channels.tags.suggestionsCount", { count: visibleSuggestions.length })}
</span>
</Tooltip>
{highConfidence.length > 0 && (
<button
onClick={acceptAllHighConfidence}
className="text-xs px-2 py-1 rounded-lg border border-border text-muted hover:border-accent hover:text-accent transition"
>
{t("channels.tags.acceptAllHigh", { count: highConfidence.length })}
</button>
)}
</div>
)}
<div className="shrink-0">{actionButtons}</div>
</div>
</div>
)}
@@ -17,6 +17,7 @@ vi.mock("./NavigationProvider", () => ({
}));
import { ChannelsProvider, useChannels, useChannelsActions } from "./ChannelsProvider";
import { setActiveAccount } from "../lib/api/core";
const wrapper = ({ children }: { children: ReactNode }) => (
<ChannelsProvider>{children}</ChannelsProvider>
@@ -27,7 +28,10 @@ const VIEW_KEY = "siftlode.channelsView.1"; // account-scoped (accountKey suffix
beforeEach(() => {
localStorage.clear();
sessionStorage.setItem("siftlode.activeAccount", "1"); // pin the tab's account
window.history.replaceState({ sfPage: "channels" }, ""); // setView only owns an entry on this page
// Reset BOTH the history state and the query string: setView only owns an entry on the channels page,
// and the S8 URL mirror writes ctab/csort/cf — a stale key left by a prior test would be read back by
// this provider's mount-time URL seed and change the starting tab.
window.history.replaceState({ sfPage: "channels" }, "", "/");
});
describe("ChannelsProvider tab history", () => {
@@ -71,3 +75,58 @@ describe("ChannelsProvider tab history", () => {
expect(result.current.view).toBe("hidden"); // unchanged
});
});
describe("ChannelsProvider — S8 shareable URL", () => {
it("seeds tab + sort + filters from a shared manager link at mount (URL wins over stored)", () => {
// Stored view says "blocked"; the shared link says "hidden" + a sort + a filter → the URL wins.
localStorage.setItem(VIEW_KEY, "blocked");
const cf = encodeURIComponent(JSON.stringify({ sync: "needsFull" }));
window.history.replaceState(
{ sfPage: "channels" },
"",
`/?ctab=hidden&csort=priority:desc&cf=${cf}`,
);
const { result } = render();
expect(result.current.view).toBe("hidden");
expect(result.current.sort).toEqual({ key: "priority", dir: "desc" });
expect(result.current.filters).toEqual({ sync: "needsFull" });
});
it("mirrors the tab into the address bar and clears it back to the default", () => {
const { result } = render();
act(() => result.current.setView("blocked"));
expect(new URLSearchParams(window.location.search).get("ctab")).toBe("blocked");
// Back to the default tab clears the key (only non-defaults are emitted).
act(() => result.current.setView("subscribed"));
expect(new URLSearchParams(window.location.search).has("ctab")).toBe(false);
});
it("a PARTIAL shared link (fresh tab) does not clobber the account's stored sort/filters", () => {
// Regression: on a fresh tab the account pins AFTER mount, so the seed's base for the OMITTED
// dimensions was read as defaults. Persisting the whole seed used to overwrite the account's real
// stored sort + filter map. The fix persists ONLY the URL-present dimensions.
localStorage.setItem(
"siftlode.channelSort.1",
JSON.stringify({ key: "subscribers", dir: "desc" }),
);
localStorage.setItem("siftlode.channelsFilterMap.1", JSON.stringify({ sync: "needsFull" }));
sessionStorage.removeItem("siftlode.activeAccount"); // account not pinned yet at mount
window.history.replaceState({ sfPage: "channels" }, "", "/?ctab=hidden"); // only the tab in the URL
const { result } = render();
act(() => setActiveAccount(1)); // App pins the account after /api/me
// The present dim (tab) is adopted from the link...
expect(result.current.view).toBe("hidden");
// ...but the omitted sort + filters keep the account's real stored values, in state AND storage.
expect(result.current.sort).toEqual({ key: "subscribers", dir: "desc" });
expect(result.current.filters).toEqual({ sync: "needsFull" });
expect(JSON.parse(localStorage.getItem("siftlode.channelSort.1")!)).toEqual({
key: "subscribers",
dir: "desc",
});
expect(JSON.parse(localStorage.getItem("siftlode.channelsFilterMap.1")!)).toEqual({
sync: "needsFull",
});
});
});
+97 -2
View File
@@ -3,16 +3,25 @@ import {
useCallback,
useContext,
useEffect,
useLayoutEffect,
useMemo,
useRef,
useState,
type ReactNode,
} from "react";
import { getAccountRaw, LS } from "../lib/storage";
import { getAccountRaw, setAccountRaw, LS } from "../lib/storage";
import { parseChannelLayout, type ChannelLayout } from "../lib/channelLayout";
import { type SortState } from "../lib/columnSort";
import { hydrateSort, type SortState } from "../lib/columnSort";
import { useSortState } from "../lib/useSortState";
import { useFilterState } from "../lib/useFilterState";
import { useActiveAccountId } from "../lib/activeAccount";
import { useUrlMirror } from "../lib/useUrlMirror";
import {
channelsParamsToState,
channelsParamUpdates,
hasChannelsParams,
type ChannelsUrlState,
} from "../lib/channelsUrlState";
import { usePersistedTab } from "./Tabs";
import type { FilterMap } from "./tableHeader";
import { useNavigation, useNavigationActions } from "./NavigationProvider";
@@ -44,6 +53,47 @@ function migrateSyncChip(): FilterMap | null {
return null;
}
// The manager's stored column-filter map, healed to a plain string/string[] object (same discipline as
// useFilterState's hydrate). Used only to build the seed's base — the URL keeps whatever the stored map
// has for any dimension a partial shared link omits.
function readStoredFilters(): FilterMap {
const raw = getAccountRaw(LS.channelsFilterMap);
if (raw) {
try {
const v = JSON.parse(raw);
if (v && typeof v === "object" && !Array.isArray(v)) return v as FilterMap;
} catch {
/* fall through to the migrate seed */
}
}
return migrateSyncChip() ?? {};
}
// S8: a shared manager link carries the content state in namespaced URL keys (ctab/csort/cf). When
// present, the URL wins over this account's stored view — read once at mount and applied over the slice
// hooks (see the provider). The keys are channels-unique, so their mere presence identifies a shared
// manager view; no ?page check is needed (the mirror below is page-gated so it never pollutes others).
// The seed = the URL-merged state PLUS which dimensions the URL actually carried. `present` matters
// because the base below is read at MOUNT, before the account is pinned (getAccountRaw returns null),
// so an OMITTED dimension's base is a default, not this account's stored value. The provider seeds and
// persists only the PRESENT dimensions and lets the absent ones re-hydrate from storage normally —
// otherwise a partial shared link silently resets the omitted stored sort / filter map.
type ChannelsSeedPresent = { view: boolean; sort: boolean; filters: boolean };
type ChannelsUrlSeed = { state: ChannelsUrlState; present: ChannelsSeedPresent };
function readChannelsUrlSeed(): ChannelsUrlSeed | null {
const params = new URLSearchParams(window.location.search);
if (!hasChannelsParams(params)) return null;
const state = channelsParamsToState(params, {
view: clampView(getAccountRaw(LS.channelsView)),
sort: hydrateSort(getAccountRaw(LS.channelSort), [], null),
filters: readStoredFilters(),
});
return {
state,
present: { view: params.has("ctab"), sort: params.has("csort"), filters: params.has("cf") },
};
}
type ChannelsState = {
search: string;
// The subscribed table's column filters, lifted out of DataTable so the top status chips, the Sync
@@ -132,6 +182,45 @@ export function ChannelsProvider({ children }: { children: ReactNode }) {
const [focusName, setFocusName] = useState<string | null>(null);
const [focusToken, setFocusToken] = useState(0);
// S8 — shareable manager view. Captured once at mount (or null when no manager URL keys are present).
const [urlSeed] = useState(readChannelsUrlSeed);
const accountId = useActiveAccountId();
const seededRef = useRef(false);
const persistedRef = useRef(false);
// Apply the shared-link seed once, BEFORE paint, through the slice setters. Going through the setters
// (not raw state) is deliberate: each one calls its hook's markDirty, so the account-pin re-hydrate
// (which fires later on a fresh tab) sees the slice as user-edited and won't reload the stored view
// over the shared one. setViewRaw (not setView) restores the tab without pushing a phantom history
// entry. A normal load has urlSeed === null and this is a no-op.
useLayoutEffect(() => {
if (!urlSeed || seededRef.current) return;
seededRef.current = true;
// Only apply the dimensions the URL actually carried. An omitted dimension must NOT go through its
// setter (which would markDirty and block the account-pin re-hydrate from loading the stored value)
// — leave it to hydrate from storage normally. Fixes a partial link wiping the stored sort/filters.
const { state, present } = urlSeed;
if (present.view && state.view !== view) setViewRaw(state.view);
if (present.filters) setFilters(state.filters);
if (present.sort) setSort(state.sort);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
// On a fresh tab the account pins AFTER mount, so the seed's setter writes above no-op'd (no account
// yet). Persist the seed to the account once it's known, so the shared view also survives a later
// PLAIN reload (no URL) — mirrors PlexProvider's rehydrate-persist. The slice hooks' own re-hydrate
// already skips (marked dirty above), so this only touches storage, not state.
useEffect(() => {
if (!urlSeed || accountId == null || persistedRef.current) return;
persistedRef.current = true;
// Persist ONLY the URL-present dimensions — the omitted ones were seeded from mount-time defaults
// (no account yet), so writing them here would overwrite the account's real stored sort/filter map.
const { state, present } = urlSeed;
if (present.view) setAccountRaw(LS.channelsView, state.view);
if (present.filters) setAccountRaw(LS.channelsFilterMap, JSON.stringify(state.filters));
if (present.sort) setAccountRaw(LS.channelSort, JSON.stringify(state.sort));
}, [urlSeed, accountId]);
// Latest-value ref so the STABLE setView can compare against the current tab without depending on
// it (a dep would recreate the action on every switch and re-render every actions consumer).
// Written during render — the sanctioned pattern NavigationProvider uses for `pageRef`.
@@ -218,6 +307,12 @@ export function ChannelsProvider({ children }: { children: ReactNode }) {
[setSearch, setFilters, setView, setLayout, setSort, focusChannel, goToFullHistory],
);
// S8: mirror the content state (tab + sort + column filters) into the address bar under the namespaced
// keys so a manager view is shareable. Gated on page===channels: off the manager the keys are cleared
// (localStorage still holds the state; it re-mirrors on return). The name `search` stays OUT — it's a
// deliberately transient rail-search, like the feed's and plex's.
useUrlMirror(channelsParamUpdates(page === "channels" ? { view, sort, filters } : null));
return (
<ActionsContext.Provider value={actions}>
<LayoutContext.Provider value={layout}>
+12 -1
View File
@@ -6,6 +6,8 @@ import { Plug, RotateCcw, Send } from "lucide-react";
import { api, type ConfigItem, type PlexTestResult } from "../lib/api";
import { notify } from "../lib/notifications";
import Tooltip from "./Tooltip";
import InlineNotice from "./InlineNotice";
import { useMe } from "../lib/useMe";
import Tabs, { usePersistedTab } from "./Tabs";
import { LoadingState, StateMessage } from "./QueryState";
import { PageToolbar } from "./PageShell";
@@ -35,6 +37,7 @@ const GROUP_ORDER = [
export default function ConfigPanel() {
const { t } = useTranslation();
const qc = useQueryClient();
const me = useMe();
const q = useQuery({ queryKey: qk.adminConfig(), queryFn: api.adminConfig });
const data = q.data;
@@ -100,6 +103,10 @@ export default function ConfigPanel() {
}
}
await qc.invalidateQueries({ queryKey: qk.adminConfig() });
// The email-not-configured notice reads me.email_configured, which is otherwise only
// refetched on its own 60s interval — without this it keeps warning about fields the admin
// has just filled in, which reads as the save not having taken.
await qc.invalidateQueries({ queryKey: qk.me() });
setReseedToken((n) => n + 1); // re-seed the draft from the saved server state
setSaveState("saved");
setTimeout(() => setSaveState((s) => (s === "saved" ? "idle" : s)), 2500);
@@ -204,7 +211,11 @@ export default function ConfigPanel() {
</div>
{activeGroup === "email" && (
<div className="mt-3 pt-3 border-t border-border/60">
<div className="mt-3 pt-3 border-t border-border/60 space-y-3">
{/* The one place an operator can see that mail is off. Sending is fail-soft, so a
half-filled config costs verification links, resets and access-request notices
with no error anywhere — say so next to the fields that fix it. */}
{!me.email_configured && <InlineNotice>{t("config.emailOff")}</InlineNotice>}
<Tooltip hint={t("config.testEmailHint")}>
<button
onClick={() => testEmail.mutate()}
+10 -10
View File
@@ -1,6 +1,7 @@
import { createContext, useCallback, useContext, useState, type ReactNode } from "react";
import { useTranslation } from "react-i18next";
import Modal from "./Modal";
import { Button } from "./ui/Button";
// App-styled, promise-based replacement for window.confirm. Wrap the tree in
// <ConfirmProvider> once, then anywhere: const confirm = useConfirm();
@@ -100,26 +101,25 @@ export function ConfirmProvider({ children }: { children: ReactNode }) {
<div
className={multi ? "flex flex-col gap-2 mt-5" : "flex justify-end gap-2 mt-5 flex-wrap"}
>
<button
<Button
variant="outline"
autoFocus={anyDanger}
onClick={() => close(null)}
className={`px-4 py-2 rounded-lg text-sm border border-border text-muted hover:text-fg hover:bg-surface transition ${
multi ? "w-full order-last" : ""
}`}
fullWidth={multi}
className={multi ? "order-last" : ""}
>
{state.opts.cancelLabel ?? t("common.cancel")}
</button>
</Button>
{choices.map((c, i) => (
<button
<Button
key={c.id}
variant={c.danger ? "danger" : "primary"}
autoFocus={!anyDanger && i === choices.length - 1}
onClick={() => close(c.id)}
className={`px-4 py-2 rounded-lg text-sm font-semibold transition ${
multi ? "w-full" : ""
} ${c.danger ? "btn-danger" : "btn-primary"}`}
fullWidth={multi}
>
{c.label || t("common.confirm")}
</button>
</Button>
))}
</div>
</Modal>
@@ -151,6 +151,7 @@ export default function DateFilterSelect({
...filters,
dateFrom: e.target.value || undefined,
maxAgeDays: undefined,
year: undefined,
})
}
className={dateInputCls}
@@ -169,6 +170,7 @@ export default function DateFilterSelect({
...filters,
dateTo: e.target.value || undefined,
maxAgeDays: undefined,
year: undefined,
})
}
className={dateInputCls}
+164 -83
View File
@@ -6,8 +6,10 @@ import {
Ban,
ClipboardPaste,
Download,
Link2,
Pause,
Play,
PlayCircle,
Plus,
RotateCcw,
Scissors,
@@ -15,6 +17,7 @@ import {
Share2,
Pencil,
Trash2,
UploadCloud,
} from "lucide-react";
import Tabs from "./Tabs";
import { PageToolbar } from "./PageShell";
@@ -22,15 +25,19 @@ import { LoadingState, StateMessage } from "./QueryState";
// Lazy: these dialogs/editors open on demand, so they stay out of the Downloads page chunk until
// used (the video editor in particular is heavy — filmstrip + scrubber).
const DownloadDialog = lazy(() => import("./DownloadDialog"));
const UploadDialog = lazy(() => import("./UploadDialog"));
const ProfileEditor = lazy(() => import("./ProfileEditor"));
const VideoEditor = lazy(() => import("./VideoEditor"));
const ShareDialog = lazy(() => import("./ShareDialog"));
const LibraryPlayer = lazy(() => import("./LibraryPlayer"));
import { useConfirm } from "./ConfirmProvider";
import { useLiveQuery } from "../lib/useLiveQuery";
import { useMe } from "../lib/useMe";
import { api, type DownloadJob } from "../lib/api";
import { invalidateDownloads } from "../lib/downloads";
import { invalidateDownloads, quickShareLink } from "../lib/downloads";
import { libraryRowActions, sharedRowActions } from "../lib/downloadActions";
import { inputCls } from "./ui/form";
import { Button } from "./ui/Button";
import { DownloadRow, IconBtn, UsageBar } from "./downloadRow";
import { MetaEditModal, ErrorModal } from "./downloadModals";
import { AdminSystem } from "./downloadAdmin";
@@ -45,13 +52,21 @@ export default function DownloadCenter() {
const [tab, setTab] = useState("queue");
const [addUrl, setAddUrl] = useState("");
const [addSource, setAddSource] = useState<string | null>(null);
const [upload, setUpload] = useState(false);
const [manage, setManage] = useState(false);
const [metaJob, setMetaJob] = useState<DownloadJob | null>(null);
const [shareJob, setShareJob] = useState<DownloadJob | null>(null);
const [playJob, setPlayJob] = useState<DownloadJob | null>(null);
const [editJob, setEditJob] = useState<DownloadJob | null>(null);
const [errorJob, setErrorJob] = useState<DownloadJob | null>(null);
const jobsQ = useLiveQuery(qk.downloads(), api.downloads, { intervalMs: 2000 });
// Poll fast (2s) while work is actually in flight; back off to 15s once the queue is settled
// (only done/error/paused rows left, which don't advance on their own) so an idle Download Center
// isn't hitting the API — and adding to the shared box's load — every 2s for nothing.
const jobsQ = useLiveQuery(qk.downloads(), api.downloads, {
intervalMs: (data) =>
data?.some((j) => j.status === "queued" || j.status === "running") ? 2000 : 15000,
});
const sharedQ = useQuery({
queryKey: qk.downloadsShared(),
queryFn: api.downloadsShared,
@@ -94,6 +109,21 @@ export default function DownloadCenter() {
if (await confirm({ title, message: body, confirmLabel: title, danger: true })) fn();
};
// One-click share (E7 S3 · D2-A): ensure a downloadable link, copy it, toast with "open in new tab"
// (see quickShareLink). Re-clicking reuses the existing link; a per-job in-flight guard also stops a
// double-click from racing two POSTs into two links before the cache is primed. Advanced controls
// (password, expiry, revoke, share-with-a-user) live in the Share dialog, via the manage-share button.
const [sharingId, setSharingId] = useState<number | null>(null);
const quickShare = async (job: DownloadJob) => {
if (sharingId !== null) return;
setSharingId(job.id);
try {
await quickShareLink(qc, job, t);
} finally {
setSharingId(null);
}
};
// Pull a link off the clipboard into the add-URL box. Clipboard reads need a secure context +
// a user gesture/permission (iOS Safari/Brave are strict) — so this is best-effort: the paste
// button is an explicit gesture (reliable), and the input's onFocus tries silently (ignored if
@@ -163,20 +193,34 @@ export default function DownloadCenter() {
placeholder={t("downloads.page.addUrl")}
className={inputCls}
/>
<button
<Button
variant="ghost"
size="sm"
iconOnly
icon={<ClipboardPaste className="w-4 h-4" />}
label={t("downloads.page.paste")}
onClick={() => pasteFromClipboard(true)}
title={t("downloads.page.paste")}
aria-label={t("downloads.page.paste")}
className="inline-flex items-center justify-center px-2.5 py-1.5 rounded-lg text-muted hover:text-fg hover:bg-surface transition shrink-0"
>
<ClipboardPaste className="w-4 h-4" />
</button>
<button
className="shrink-0"
/>
<Button
variant="primary"
size="sm"
icon={<Plus className="w-4 h-4" />}
onClick={() => addUrl.trim() && setAddSource(addUrl.trim())}
className="inline-flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-sm font-medium btn-primary transition shrink-0"
className="shrink-0"
>
<Plus className="w-4 h-4" /> {t("downloads.page.add")}
</button>
{t("downloads.page.add")}
</Button>
{/* Local upload is a peer to "add a URL": a general file → the same library (E7 S5c). */}
<Button
variant="outline"
size="sm"
icon={<UploadCloud className="w-4 h-4" />}
onClick={() => setUpload(true)}
className="shrink-0"
>
{t("downloads.page.upload")}
</Button>
</div>
<div className="space-y-2">
{queue.map((job) => (
@@ -236,50 +280,69 @@ export default function DownloadCenter() {
<>
<UsageBar />
<div className="space-y-2">
{library.map((job) => (
<DownloadRow key={job.id} job={job}>
{saveBtn(job)}
{job.can_download && (job.asset?.duration_s ?? 0) > 0 && (
<IconBtn onClick={() => setEditJob(job)} title={t("downloads.actions.edit")}>
<Scissors className="w-4 h-4" />
{library.map((job) => {
// Row actions gated on the stored media_kind + job_kind (see downloadActions).
const a = libraryRowActions(job);
return (
<DownloadRow key={job.id} job={job}>
{a.play && (
<IconBtn onClick={() => setPlayJob(job)} title={t("downloads.actions.play")}>
<PlayCircle className="w-4 h-4" />
</IconBtn>
)}
{saveBtn(job)}
{a.edit && (
<IconBtn onClick={() => setEditJob(job)} title={t("downloads.actions.edit")}>
<Scissors className="w-4 h-4" />
</IconBtn>
)}
{a.editDetails && (
<IconBtn
onClick={() => setMetaJob(job)}
title={t("downloads.actions.editDetails")}
>
<Pencil className="w-4 h-4" />
</IconBtn>
)}
<IconBtn onClick={() => quickShare(job)} title={t("downloads.actions.share")}>
<Share2 className="w-4 h-4" />
</IconBtn>
)}
<IconBtn
onClick={() => setMetaJob(job)}
title={t("downloads.actions.editDetails")}
>
<Pencil className="w-4 h-4" />
</IconBtn>
<IconBtn onClick={() => setShareJob(job)} title={t("downloads.actions.share")}>
<Share2 className="w-4 h-4" />
</IconBtn>
<IconBtn
onClick={() =>
confirmThen(
t("downloads.confirm.redownloadTitle"),
t("downloads.confirm.redownloadBody"),
() => act.mutate({ id: job.id, op: "redownload" }),
)
}
title={t("downloads.actions.redownload")}
>
<RotateCcw className="w-4 h-4" />
</IconBtn>
<IconBtn
onClick={() =>
confirmThen(
t("downloads.confirm.deleteTitle"),
t("downloads.confirm.deleteBody"),
() => act.mutate({ id: job.id, op: "delete" }),
)
}
title={t("downloads.actions.delete")}
danger
>
<Trash2 className="w-4 h-4" />
</IconBtn>
</DownloadRow>
))}
<IconBtn
onClick={() => setShareJob(job)}
title={t("downloads.actions.shareManage")}
>
<Link2 className="w-4 h-4" />
</IconBtn>
{a.redownload && (
<IconBtn
onClick={() =>
confirmThen(
t("downloads.confirm.redownloadTitle"),
t("downloads.confirm.redownloadBody"),
() => act.mutate({ id: job.id, op: "redownload" }),
)
}
title={t("downloads.actions.redownload")}
>
<RotateCcw className="w-4 h-4" />
</IconBtn>
)}
<IconBtn
onClick={() =>
confirmThen(
t("downloads.confirm.deleteTitle"),
t("downloads.confirm.deleteBody"),
() => act.mutate({ id: job.id, op: "delete" }),
)
}
title={t("downloads.actions.delete")}
danger
>
<Trash2 className="w-4 h-4" />
</IconBtn>
</DownloadRow>
);
})}
{library.length === 0 &&
(jobsQ.isError && !jobsQ.data ? (
<StateMessage tone="error" onRetry={() => jobsQ.refetch()}>
@@ -298,34 +361,50 @@ export default function DownloadCenter() {
{tab === "shared" && (
<div className="space-y-2">
{(sharedQ.data ?? []).map((job) => (
<DownloadRow key={job.id} job={job}>
{job.can_download && saveBtn(job)}
{job.can_download && (job.asset?.duration_s ?? 0) > 0 && (
<IconBtn onClick={() => setEditJob(job)} title={t("downloads.actions.edit")}>
<Scissors className="w-4 h-4" />
{(sharedQ.data ?? []).map((job) => {
// Same media_kind + can_download gate as the library, via the shared helper (S5b).
const a = sharedRowActions(job);
return (
<DownloadRow key={job.id} job={job}>
{a.play && (
<IconBtn onClick={() => setPlayJob(job)} title={t("downloads.actions.play")}>
<PlayCircle className="w-4 h-4" />
</IconBtn>
)}
{a.save && saveBtn(job)}
{a.edit && (
<IconBtn onClick={() => setEditJob(job)} title={t("downloads.actions.edit")}>
<Scissors className="w-4 h-4" />
</IconBtn>
)}
<IconBtn
onClick={() =>
confirmThen(
t("downloads.confirm.removeSharedTitle"),
t("downloads.confirm.removeSharedBody"),
() => removeShared.mutate(job.id),
)
}
title={t("downloads.actions.removeShared")}
danger
>
<Trash2 className="w-4 h-4" />
</IconBtn>
)}
<IconBtn
onClick={() =>
confirmThen(
t("downloads.confirm.removeSharedTitle"),
t("downloads.confirm.removeSharedBody"),
() => removeShared.mutate(job.id),
)
}
title={t("downloads.actions.removeShared")}
danger
>
<Trash2 className="w-4 h-4" />
</IconBtn>
</DownloadRow>
))}
{sharedQ.data && sharedQ.data.length === 0 && (
<div className="text-sm text-muted py-10 text-center">
{t("downloads.empty.shared")}
</div>
)}
</DownloadRow>
);
})}
{(sharedQ.data ?? []).length === 0 &&
(sharedQ.isError && !sharedQ.data ? (
<StateMessage tone="error" onRetry={() => sharedQ.refetch()}>
{t("common.loadError")}
</StateMessage>
) : sharedQ.isLoading ? (
<LoadingState />
) : (
<div className="text-sm text-muted py-10 text-center">
{t("downloads.empty.shared")}
</div>
))}
</div>
)}
@@ -341,9 +420,11 @@ export default function DownloadCenter() {
}}
/>
)}
{upload && <UploadDialog onClose={() => setUpload(false)} />}
{manage && <ProfileEditor onClose={() => setManage(false)} />}
{metaJob && <MetaEditModal job={metaJob} onClose={() => setMetaJob(null)} />}
{shareJob && <ShareDialog job={shareJob} onClose={() => setShareJob(null)} />}
{playJob && <LibraryPlayer job={playJob} onClose={() => setPlayJob(null)} />}
{editJob && <VideoEditor job={editJob} onClose={() => setEditJob(null)} />}
{errorJob && (
<ErrorModal
+5 -11
View File
@@ -3,6 +3,7 @@ import { qk } from "../lib/queryKeys";
import { useTranslation } from "react-i18next";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import Modal from "./Modal";
import { Button } from "./ui/Button";
import { api } from "../lib/api";
import { invalidateDownloads } from "../lib/downloads";
@@ -89,19 +90,12 @@ export default function DownloadDialog({
/>
<div className="flex justify-end gap-2">
<button
onClick={onClose}
className="px-3 py-1.5 rounded-lg text-sm text-muted hover:text-fg hover:bg-surface transition"
>
<Button variant="ghost" size="sm" onClick={onClose}>
{t("downloads.actions.cancel")}
</button>
<button
onClick={submit}
disabled={busy || !chosen}
className="px-3 py-1.5 rounded-lg text-sm font-medium btn-primary disabled:opacity-50 transition"
>
</Button>
<Button variant="primary" size="sm" onClick={submit} disabled={busy || !chosen}>
{t("downloads.dialog.submit")}
</button>
</Button>
</div>
{manage && (
+2 -6
View File
@@ -2,6 +2,7 @@ import { useSyncExternalStore } from "react";
import { useTranslation } from "react-i18next";
import { dismissError, getError, subscribeError } from "../lib/errorDialog";
import Modal from "./Modal";
import { Button } from "./ui/Button";
// Renders the current app error (if any) as a modal the user must acknowledge.
export default function ErrorDialog() {
@@ -12,12 +13,7 @@ export default function ErrorDialog() {
<Modal title={err.title} onClose={dismissError} maxWidth="max-w-md">
<p className="text-sm leading-relaxed text-fg/90">{err.message}</p>
<div className="flex justify-end mt-5">
<button
onClick={dismissError}
className="px-4 py-2 rounded-lg btn-primary text-sm font-medium transition"
>
{t("errors.ok")}
</button>
<Button onClick={dismissError}>{t("errors.ok")}</Button>
</div>
</Modal>
);
@@ -13,6 +13,7 @@ import {
api,
getActiveAccount,
coerceTagModes,
coerceUntagged,
DEFAULT_TAG_MODES,
type FeedFilters,
} from "../lib/api";
@@ -39,6 +40,7 @@ import { useNavigation } from "./NavigationProvider";
const DEFAULT_FILTERS: FeedFilters = {
tags: [],
tagModes: DEFAULT_TAG_MODES,
untagged: false,
channelIds: [],
q: "",
sort: "newest",
@@ -74,7 +76,12 @@ export function loadAccountFilters(id: number | null): FeedFilters {
// (all "or"), which coerceTagModes prefers over the legacy scalar and would mask it.
if (hasStored && fKey) {
const raw = readJSON<Partial<FeedFilters>>(fKey, {});
return { ...DEFAULT_FILTERS, ...raw, tagModes: coerceTagModes(raw) };
return {
...DEFAULT_FILTERS,
...raw,
tagModes: coerceTagModes(raw),
untagged: coerceUntagged(raw),
};
}
return { ...DEFAULT_FILTERS };
}
+26
View File
@@ -0,0 +1,26 @@
import type { ReactNode } from "react";
import type { LucideIcon } from "lucide-react";
import { AlertTriangle } from "lucide-react";
// An in-flow warning that sits INSIDE a panel, next to the thing it is about — as opposed to
// Banner, which is the full-bleed top-of-app bar (border-b, app chrome) and looks wrong nested in a
// card. Same warn tokens as Banner's warning tone so the two read as one system.
export default function InlineNotice({
children,
icon: Icon = AlertTriangle,
className = "",
}: {
children: ReactNode;
icon?: LucideIcon;
className?: string;
}) {
return (
<div
role="status"
className={`flex items-start gap-2 rounded-xl border border-warn/30 bg-warn/15 px-3 py-2 text-sm text-fg ${className}`}
>
<Icon className="w-4 h-4 shrink-0 mt-0.5 text-warn" aria-hidden />
<span className="min-w-0">{children}</span>
</div>
);
}
@@ -0,0 +1,125 @@
// @vitest-environment jsdom
//
// E7·S4b — the in-app library player streams the completed item's LOCAL file into a native media
// element. video/audio is chosen from the stored media_kind (the single source), NOT a container
// list — so an audio upload with an uncommon/absent container still renders <audio src=/file>. i18n
// is key-passthrough here.
import { describe, it, expect, afterEach, vi } from "vitest";
import { render, cleanup } from "@testing-library/react";
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
return { ...actual, useTranslation: () => ({ t: (key: string) => key }) };
});
import LibraryPlayer from "./LibraryPlayer";
import type { DownloadJob } from "../lib/api";
const baseJob: DownloadJob = {
id: 7,
status: "done",
progress: 100,
speed_bps: null,
eta_s: null,
phase: null,
error: null,
error_code: null,
queue_pos: 0,
created_at: null,
source_kind: "youtube",
source_ref: "abc",
source_url: null,
poster_url: null,
profile_id: null,
spec: {
mode: "av",
max_height: null,
container: null,
audio_format: null,
vcodec: null,
embed_subs: false,
embed_chapters: false,
embed_thumbnail: false,
sponsorblock: false,
},
display_name: "Clip",
display_uploader: null,
display_uploader_url: null,
extra_links: [],
can_download: true,
expired: false,
asset: null,
};
const withAsset = (over: Partial<NonNullable<DownloadJob["asset"]>>): DownloadJob => ({
...baseJob,
asset: {
id: 10,
status: "ready",
title: "Clip",
uploader: null,
uploader_url: null,
upload_date: null,
duration_s: 12,
size_bytes: 1000,
width: null,
height: null,
container: null,
media_kind: "video",
thumbnail_url: null,
expires_at: null,
...over,
},
});
afterEach(cleanup);
describe("LibraryPlayer", () => {
it("plays a video asset in a <video> pointed at the range-aware /file endpoint", () => {
render(
<LibraryPlayer
job={withAsset({ width: 1280, height: 720, container: "mp4" })}
onClose={() => {}}
/>,
);
const video = document.body.querySelector("video");
expect(video).not.toBeNull();
expect(video!.getAttribute("src")).toBe("/api/downloads/7/file");
// No <audio> element for a video asset.
expect(document.body.querySelector("audio")).toBeNull();
});
it("plays an audio-only asset (media_kind='audio') in an <audio> element", () => {
render(
<LibraryPlayer
job={withAsset({ media_kind: "audio", container: "m4a" })}
onClose={() => {}}
/>,
);
const audio = document.body.querySelector("audio");
expect(audio).not.toBeNull();
expect(audio!.getAttribute("src")).toBe("/api/downloads/7/file");
expect(document.body.querySelector("video")).toBeNull();
});
it("uses media_kind — an audio upload with an uncommon container still renders <audio>", () => {
// The exact bug the fix targets: the old container whitelist (m4a/mp3/…) would drop an .m4b/.weba
// audio upload into a <video> box. media_kind='audio' is authoritative regardless of container.
render(
<LibraryPlayer
job={withAsset({ media_kind: "audio", container: "m4b" })}
onClose={() => {}}
/>,
);
expect(document.body.querySelector("audio")).not.toBeNull();
expect(document.body.querySelector("video")).toBeNull();
});
it("falls back to the container heuristic for a legacy null media_kind", () => {
// A pre-discriminator asset (media_kind null) still resolves audio vs video from dims+container.
render(
<LibraryPlayer job={withAsset({ media_kind: null, container: "mp3" })} onClose={() => {}} />,
);
expect(document.body.querySelector("audio")).not.toBeNull();
});
});
+62
View File
@@ -0,0 +1,62 @@
import { useMemo } from "react";
import { useTranslation } from "react-i18next";
import Modal from "./Modal";
import { api, type DownloadJob } from "../lib/api";
import { jobTitle } from "./downloadRow";
// E7 S4b — in-app player for a completed library item. We do NOT reuse PlayerModal (a cross-origin
// YouTube IFrame shell) or PlexPlayer (a Plex HLS/session shell): both are bound to their own domain
// APIs and can't point at a local file. Instead we reuse the shared a11y shell (`Modal`: focus-trap,
// ESC/back-to-close, scroll-lock, layer-stack, portal) and stream the local file into a NATIVE
// <video>/<audio controls> — the same element VideoEditor already plays this URL in. Downloads are a
// browser-native container (mp4/H.264/AAC + m4a/mp3), so native playback (seek via the range-aware
// /file endpoint, volume, fullscreen, speed) works with zero transcode; an uploaded media item may be
// any codec (best-effort playback).
export default function LibraryPlayer({ job, onClose }: { job: DownloadJob; onClose: () => void }) {
const { t } = useTranslation();
const fileUrl = useMemo(() => api.downloadFileUrl(job.id), [job.id]);
// Video vs audio-only comes from the stored media_kind (the single source every sibling reads —
// WatchPage/downloadRow/DownloadCenter) — NOT a re-derived container list, which would misclassify
// an uploaded audio file whose container is outside a hardcoded set (uploads accept any container).
// Only fall back to the old dimension+container heuristic for a legacy asset with a null media_kind.
const asset = job.asset;
const isAudio =
asset?.media_kind === "audio" ||
(asset?.media_kind == null &&
!(asset?.width && asset?.height) &&
!!asset?.container &&
["m4a", "mp3", "opus", "ogg", "wav", "flac", "aac"].includes(asset.container.toLowerCase()));
// Our self-hosted poster (always same-origin, never expires) is the primary; the source thumbnail is
// the fallback. Same precedence as the library-row thumbnail.
const poster = job.poster_url ? api.posterImageUrl(job.id) : (asset?.thumbnail_url ?? undefined);
return (
<Modal title={jobTitle(job)} onClose={onClose} maxWidth="max-w-4xl">
{isAudio ? (
<div className="flex flex-col items-center gap-4 py-2">
{poster ? (
<img
src={poster}
alt=""
className="w-40 h-40 rounded-xl object-cover border border-border"
/>
) : null}
<audio src={fileUrl} controls autoPlay className="w-full">
{t("downloads.player.unsupported")}
</audio>
</div>
) : (
<video
src={fileUrl}
poster={poster}
controls
autoPlay
playsInline
className="w-full max-h-[70vh] rounded-lg bg-black object-contain"
>
{t("downloads.player.unsupported")}
</video>
)}
</Modal>
);
}
+16 -3
View File
@@ -126,12 +126,25 @@ export function NavigationProvider({ children }: { children: ReactNode }) {
// Push an in-app history entry so browser/mouse Back steps through pages instead of leaving the
// app. A fresh { sfPage } (no carried-over _yt/_chan markers) so each page starts at its root, and
// the address bar reflects the page (feed is implicit) while clearing the view-specific params
// (chan/v/pl) — a page switch resets to that module's root. Feed filters (owned by
// FeedFiltersProvider) are left intact.
// (chan/v/pl and the plex content keys pscope/psort/pshow/pf) — a page switch resets to that
// module's root. Feed filters (owned by FeedFiltersProvider) are left intact; the plex/channels
// keys re-mirror from PlexProvider / ChannelsProvider on return to their page.
window.history.pushState(
{ sfPage: next, sfScroll: nextNavId() },
"",
urlWithParams({ page: next === "feed" ? null : next, chan: null, v: null, pl: null }),
urlWithParams({
page: next === "feed" ? null : next,
chan: null,
v: null,
pl: null,
pscope: null,
psort: null,
pshow: null,
pf: null,
ctab: null,
csort: null,
cf: null,
}),
);
notifyNav();
},
@@ -243,6 +243,7 @@ function NotificationRow({
const isMaintenance = n.type === "maintenance" && typeof n.data?.count === "number";
const isScheduler = n.type === "scheduler" && typeof n.data?.job_id === "string";
const isDownloadDone = n.type === "download_done";
const isEmailUndelivered = n.type === "email_undelivered" && typeof n.data?.reason === "string";
let title = n.title;
let body = n.body;
if (isMaintenance) {
@@ -258,6 +259,17 @@ function NotificationRow({
job,
});
body = d.summary || n.body;
} else if (isEmailUndelivered) {
// Email is fail-soft: a drop leaves no error anywhere else, so this notice IS the trace. Name
// the message that was lost, since that is what tells the operator what their users didn't get.
const d = n.data as { reason: string; subject?: string };
title = t("inbox.emailUndelivered.title");
body = t(
d.reason === "not_configured"
? "inbox.emailUndelivered.notConfigured"
: "inbox.emailUndelivered.sendFailed",
{ subject: d.subject || n.body || "" },
);
} else if (isDownloadDone) {
// "Download ready" / "Edit ready" per the producer's kind, with the video's title as the body
// (shipped in data.title). Same notification type; the copy differs so an edit isn't mislabelled.
@@ -0,0 +1,186 @@
// @vitest-environment jsdom
//
// S7 — the Personal-views manager gains an EDITABLE detail pane: captured filters are now removable
// chips that write the pruned set back to the stored view (undoable), an explicit Save-current button
// stores the live filters into the view, and rename has a visible commit affordance. This drives the
// real modal against a mocked SavedViewsProvider so we can assert exactly which mutation fires; the
// captured-filter row uses the REAL useActiveFilters (fed a tag from a seeded react-query cache), so
// the prune snapshots are the genuine ones the live chip row computes.
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import { render, screen, waitFor, cleanup, fireEvent } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
const h = vi.hoisted(() => ({
updateFilters: vi.fn(async (_id: number, _filters: Record<string, unknown>) => true),
saveCurrent: vi.fn(async (_id: number) => ({ saved: true }) as Record<string, unknown> | null),
rename: vi.fn(async (_id: number, _name: string) => true),
activeViewId: null as number | null,
}));
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
return { ...actual, useTranslation: () => ({ t: (key: string) => key }) };
});
// A full feed filter set that differs from the sidebar default ONLY in `tags`, so useActiveFilters
// surfaces exactly one chip (the tag) — a partial object would read missing show/sort/content as
// "active" and spray spurious chips.
const VIEW_FILTERS = {
tags: [1],
tagModes: { language: "or", topic: "or", other: "or" },
untagged: false,
channelIds: [],
sort: "newest",
includeNormal: true,
includeShorts: false,
includeLive: false,
show: "unwatched",
scope: "my",
q: "",
};
const CTX = {
source: "feed" as const,
views: [{ id: 5, name: "Docs", filters: VIEW_FILTERS, position: 0, is_default: false }],
loading: false,
displayView: null,
isDirty: false,
modalOpen: true,
openModal: vi.fn(),
closeModal: vi.fn(),
apply: vi.fn(),
clearActive: vi.fn(),
create: vi.fn(async () => true),
reorder: vi.fn(async () => {}),
remove: vi.fn(async () => true),
toggleDefault: vi.fn(async () => {}),
};
vi.mock("./SavedViewsProvider", () => ({
useSavedViews: () => ({
...CTX,
activeViewId: h.activeViewId,
updateFilters: h.updateFilters,
saveCurrent: h.saveCurrent,
rename: h.rename,
}),
// Passthrough coercion — the modal only needs tags/channelIds to be arrays before the chip builder.
asFilters: (raw: Record<string, unknown>) => ({
...raw,
tags: (raw.tags as unknown[]) ?? [],
channelIds: (raw.channelIds as unknown[]) ?? [],
}),
}));
import PersonalViewsModal from "./PersonalViewsModal";
import { api } from "../lib/api";
import { qk } from "../lib/queryKeys";
function mount() {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
qc.setQueryData(qk.tags(), [{ id: 1, name: "Docu" }]);
vi.spyOn(api, "tags").mockResolvedValue([{ id: 1, name: "Docu" }] as never);
const utils = render(
<QueryClientProvider client={qc}>
<PersonalViewsModal />
</QueryClientProvider>,
);
return { ...utils, qc };
}
beforeEach(() => {
h.updateFilters.mockClear();
h.saveCurrent.mockClear();
h.rename.mockClear();
h.activeViewId = null;
// Fresh baseline each test — one test mutates CTX.views to simulate a refetch.
CTX.views = [{ id: 5, name: "Docs", filters: VIEW_FILTERS, position: 0, is_default: false }];
});
afterEach(() => {
cleanup();
vi.restoreAllMocks();
});
describe("PersonalViewsModal — S7 editable detail pane", () => {
it("removing a captured-filter chip writes the pruned filters back to the stored view", async () => {
mount();
const removeBtn = await screen.findByLabelText("feed.chips.remove");
fireEvent.click(removeBtn);
await waitFor(() => expect(h.updateFilters).toHaveBeenCalledTimes(1));
expect(h.updateFilters.mock.calls[0]![0]).toBe(5);
expect(h.updateFilters.mock.calls[0]![1]).toMatchObject({ tags: [] });
});
it("the prune is undoable — Undo re-persists the original filters", async () => {
mount();
fireEvent.click(await screen.findByLabelText("feed.chips.remove"));
await waitFor(() => expect(h.updateFilters).toHaveBeenCalledTimes(1));
// The undo control only appears once there is history.
const undo = await screen.findByLabelText("common.undo");
fireEvent.click(undo);
await waitFor(() => expect(h.updateFilters).toHaveBeenCalledTimes(2));
expect(h.updateFilters.mock.calls[1]![1]).toMatchObject({ tags: [1] });
});
it("a same-view refetch (new-identity filters) does not wipe the undo history", async () => {
// Guards the [selectedId]-only re-seed effect (its exhaustive-deps disable): after a chip prune the
// real provider refetches and hands back the SAME view id with a fresh-identity filters object. The
// draft must NOT re-seed on that (it would destroy the undo history and the Undo control). A deps
// regression that added selected.filters would make this fail.
const { rerender, qc } = mount();
fireEvent.click(await screen.findByLabelText("feed.chips.remove"));
await waitFor(() => expect(h.updateFilters).toHaveBeenCalledTimes(1));
expect(await screen.findByLabelText("common.undo")).toBeTruthy();
// Simulate the post-write refetch: same id 5, new object identity, carrying the pruned value.
CTX.views = [
{
id: 5,
name: "Docs",
filters: { ...VIEW_FILTERS, tags: [] },
position: 0,
is_default: false,
},
];
rerender(
<QueryClientProvider client={qc}>
<PersonalViewsModal />
</QueryClientProvider>,
);
// Undo history survived the refetch — the control is still there and still re-persists on click.
const undo = await screen.findByLabelText("common.undo");
fireEvent.click(undo);
await waitFor(() => expect(h.updateFilters).toHaveBeenCalledTimes(2));
expect(h.updateFilters.mock.calls[1]![1]).toMatchObject({ tags: [1] });
});
it("Save-current stores the live filters into the view when it isn't already the active one", async () => {
mount();
fireEvent.click(await screen.findByText("views.saveCurrent"));
await waitFor(() => expect(h.saveCurrent).toHaveBeenCalledWith(5));
});
it("Save-current is disabled when the selected view is already the active one", async () => {
h.activeViewId = 5;
mount();
const btn = (await screen.findByText("views.saveCurrent")).closest(
"button",
) as HTMLButtonElement;
expect(btn.disabled).toBe(true);
fireEvent.click(btn);
expect(h.saveCurrent).not.toHaveBeenCalled();
});
it("rename commits via the visible check button once the name changes", async () => {
mount();
const input = await screen.findByDisplayValue("Docs");
// No change yet → the commit button is disabled.
const commit = screen.getByLabelText("views.rename").closest("button") as HTMLButtonElement;
expect(commit.disabled).toBe(true);
fireEvent.change(input, { target: { value: "Documentaries" } });
expect(commit.disabled).toBe(false);
fireEvent.click(commit);
await waitFor(() => expect(h.rename).toHaveBeenCalledWith(5, "Documentaries"));
});
});
+134 -39
View File
@@ -1,6 +1,6 @@
import { useState } from "react";
import { useEffect, useState } from "react";
import { useTranslation } from "react-i18next";
import { BookmarkPlus, GripVertical, Play, Star, Trash2 } from "lucide-react";
import { BookmarkPlus, Check, GripVertical, Play, Save, Star, Trash2, X } from "lucide-react";
import {
closestCenter,
DndContext,
@@ -16,15 +16,19 @@ import {
verticalListSortingStrategy,
} from "@dnd-kit/sortable";
import { CSS } from "@dnd-kit/utilities";
import { coercePlexFilters, type FeedFilters, type SavedView } from "../lib/api";
import { coercePlexFilters, type FeedFilters, type PlexFilters, type SavedView } from "../lib/api";
import Modal from "./Modal";
import { Button } from "./ui/Button";
import { useSavedViews, asFilters } from "./SavedViewsProvider";
import { useActiveFilters } from "../lib/useActiveFilters";
import { buildPlexActiveFilters } from "./usePlexActiveFilters";
import { useUndoable } from "../lib/useUndoable";
import UndoToolbar from "./UndoToolbar";
// Master–detail manager for saved ("personal") views: the left column lists the views (drag to
// reorder) plus a "save the current filters" input; the right column shows the selected view's name,
// what filters it captures, its default flag, apply, and delete. Opened from the filter panel's title
// reorder) plus a "save the current filters" input; the right column shows the selected view's name
// (renamable), its captured filters as REMOVABLE chips (pruning writes back to the stored view,
// undoable), its default flag, apply, save-current, and delete. Opened from the filter panel's title
// or the saved-views popover; all mutations flow through SavedViewsProvider.
function ViewRow({
view,
@@ -80,6 +84,8 @@ export default function PersonalViewsModal() {
apply,
create,
rename,
updateFilters,
saveCurrent,
toggleDefault,
remove,
reorder,
@@ -89,21 +95,55 @@ export default function PersonalViewsModal() {
const [selectedId, setSelectedId] = useState<number | null>(activeViewId ?? views[0]?.id ?? null);
const selected = views.find((v) => v.id === selectedId) ?? null;
const [draftName, setDraftName] = useState("");
// Two independent name fields: the left one names a NEW view; the right one renames the selected one.
const [createName, setCreateName] = useState("");
const [nameDraft, setNameDraft] = useState("");
// The selected view's captured filters, as the same human labels its module's chip row shows. The
// feed hook runs unconditionally (fed empty filters off-feed); Plex uses the pure builder (no-op
// removes — this is display-only). A plex view captures its rail filters, not the watch-state.
// A local, undoable draft of the SELECTED view's captured filters. Removing a chip prunes the draft
// and — via onApply — writes the pruned set straight back to the stored view; Undo/Redo re-persist
// the same way. The draft is the single source of truth for the chip row, so a chip-prune refetch
// (which returns the value we just wrote) never clobbers the undo history. We only re-seed it on an
// external change of the visible filters: switching the selection, or Save-current below.
const filtersDraft = useUndoable<Record<string, unknown>>(selected?.filters ?? {}, {
onApply: (f) => {
if (selected) void updateFilters(selected.id, f);
},
});
const resetDraft = filtersDraft.reset;
useEffect(() => {
resetDraft(selected?.filters ?? {});
setNameDraft(selected?.name ?? "");
// Re-seed only when the selection changes (not on every same-view refetch), so pruning keeps its
// undo history. Save-current re-seeds explicitly in its own handler.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [selectedId]);
// The selected view's captured filters, as the same human labels its module's chip row shows, read
// from the undoable draft. The feed hook runs unconditionally (fed empty filters off-feed); Plex
// uses the pure builder. A plex view captures its rail filters, not the watch-state (show="all").
const noop = () => {};
const setDraftFeed = (f: FeedFilters) =>
filtersDraft.set(f as unknown as Record<string, unknown>);
const feedCaptured = useActiveFilters(
source === "feed" && selected
? asFilters(selected.filters as unknown as FeedFilters)
? asFilters(filtersDraft.value as unknown as FeedFilters)
: ({ tags: [], channelIds: [] } as never),
noop,
source === "feed" && selected ? setDraftFeed : noop,
);
const patchDraftPlex = (p: Partial<PlexFilters>) =>
filtersDraft.set({
...coercePlexFilters(filtersDraft.value),
...p,
} as unknown as Record<string, unknown>);
const captured =
source === "plex" && selected
? buildPlexActiveFilters(coercePlexFilters(selected.filters), "all", noop, noop, t)
? buildPlexActiveFilters(
coercePlexFilters(filtersDraft.value),
"all",
patchDraftPlex,
noop,
t,
)
: feedCaptured;
async function onDragEnd(e: DragEndEvent) {
@@ -115,11 +155,20 @@ export default function PersonalViewsModal() {
}
async function onCreate() {
const name = draftName.trim();
const name = createName.trim();
if (!name) return;
if (await create(name)) setDraftName("");
if (await create(name)) setCreateName("");
}
const nameChanged = !!selected && nameDraft.trim() !== "" && nameDraft.trim() !== selected.name;
async function onRename() {
if (!selected || !nameChanged) return;
await rename(selected.id, nameDraft.trim());
}
// Save-current is a no-op when the live filters already equal this exact view (it's the active one).
const saveCurrentDisabled = selected != null && selected.id === activeViewId;
return (
<Modal title={t("views.manageTitle")} onClose={closeModal} maxWidth="max-w-2xl">
<div className="grid min-h-[20rem] grid-cols-[minmax(0,14rem)_1fr] gap-4">
@@ -148,38 +197,59 @@ export default function PersonalViewsModal() {
</div>
<div className="mt-2 flex items-center gap-1.5 border-t border-border/60 pt-2">
<input
value={draftName}
onChange={(e) => setDraftName(e.target.value)}
value={createName}
onChange={(e) => setCreateName(e.target.value)}
onKeyDown={(e) => e.key === "Enter" && onCreate()}
placeholder={t("views.namePlaceholder")}
className="min-w-0 flex-1 rounded-lg border border-border bg-surface px-2 py-1.5 text-sm focus:border-accent focus:outline-hidden focus:ring-2 focus:ring-accent/30"
/>
<button
<Button
variant="primary"
size="sm"
iconOnly
icon={<BookmarkPlus className="h-4 w-4" />}
label={t("views.save")}
onClick={onCreate}
disabled={!draftName.trim()}
title={t("views.save")}
className="inline-flex shrink-0 items-center gap-1 rounded-lg btn-primary px-2.5 py-1.5 text-sm font-semibold disabled:opacity-40"
>
<BookmarkPlus className="h-4 w-4" />
</button>
disabled={!createName.trim()}
className="shrink-0"
/>
</div>
</div>
{/* Right: the selected view's detail */}
{selected ? (
<div className="flex flex-col gap-3">
<input
key={selected.id}
defaultValue={selected.name}
onBlur={(e) => {
const next = e.target.value.trim();
if (next && next !== selected.name) void rename(selected.id, next);
}}
className="rounded-lg border border-border bg-surface px-3 py-2 text-sm font-medium focus:border-accent focus:outline-hidden focus:ring-2 focus:ring-accent/30"
/>
<div className="flex items-center gap-1.5">
<input
value={nameDraft}
onChange={(e) => setNameDraft(e.target.value)}
onKeyDown={(e) => e.key === "Enter" && void onRename()}
onBlur={() => void onRename()}
className="min-w-0 flex-1 rounded-lg border border-border bg-surface px-3 py-2 text-sm font-medium focus:border-accent focus:outline-hidden focus:ring-2 focus:ring-accent/30"
/>
<button
onClick={() => void onRename()}
disabled={!nameChanged}
title={t("views.rename")}
aria-label={t("views.rename")}
className="inline-flex shrink-0 items-center rounded-lg border border-border p-2 text-muted transition enabled:hover:border-accent enabled:hover:text-accent disabled:opacity-30"
>
<Check className="h-4 w-4" />
</button>
</div>
<div>
<div className="mb-1.5 text-2xs font-semibold uppercase tracking-wider text-muted">
{t("views.capturedFilters")}
<div className="mb-1.5 flex items-center justify-between gap-2">
<span className="text-2xs font-semibold uppercase tracking-wider text-muted">
{t("views.capturedFilters")}
</span>
{(filtersDraft.canUndo || filtersDraft.canRedo) && (
<UndoToolbar
canUndo={filtersDraft.canUndo}
canRedo={filtersDraft.canRedo}
onUndo={filtersDraft.undo}
onRedo={filtersDraft.redo}
/>
)}
</div>
{captured.length === 0 ? (
<p className="text-xs text-muted">{t("views.capturedNone")}</p>
@@ -188,21 +258,46 @@ export default function PersonalViewsModal() {
{captured.map((f) => (
<span
key={f.key}
className="rounded-full bg-accent/10 px-2 py-0.5 text-2xs text-accent"
className="group inline-flex items-center gap-1 rounded-full bg-accent/10 py-0.5 pl-2 pr-1 text-2xs text-accent"
>
{f.label}
<button
onClick={f.remove}
title={t("feed.chips.remove", { name: f.label })}
aria-label={t("feed.chips.remove", { name: f.label })}
className="rounded-full p-0.5 opacity-60 transition hover:bg-accent/20 hover:opacity-100"
>
<X className="h-3 w-3" />
</button>
</span>
))}
</div>
)}
</div>
<div className="mt-auto flex items-center gap-2 border-t border-border/60 pt-3">
<button
<div className="mt-auto flex flex-wrap items-center gap-2 border-t border-border/60 pt-3">
<Button
variant="primary"
size="sm"
icon={<Play className="h-3.5 w-3.5" />}
onClick={() => apply(selected)}
className="inline-flex items-center gap-1.5 rounded-lg btn-primary px-3 py-1.5 text-sm font-semibold"
>
<Play className="h-3.5 w-3.5" />
{t("views.apply")}
</Button>
<button
onClick={async () => {
const saved = await saveCurrent(selected.id);
// Re-seed the draft to what we just stored (the live filters) so the chip row
// reflects the save; clears the now-stale undo history.
if (saved) resetDraft(saved);
}}
disabled={saveCurrentDisabled}
title={
saveCurrentDisabled ? t("views.saveCurrentUpToDate") : t("views.saveCurrentHint")
}
className="inline-flex items-center gap-1.5 rounded-lg border border-border px-3 py-1.5 text-sm text-muted transition enabled:hover:border-accent enabled:hover:text-accent disabled:opacity-40"
>
<Save className="h-3.5 w-3.5" />
{t("views.saveCurrent")}
</button>
<button
onClick={() => void toggleDefault(selected)}
+214 -166
View File
@@ -39,6 +39,7 @@ import { notifyYouTubeActionError } from "../lib/youtubeErrors";
import { playlistName } from "../lib/playlistName";
import { useUndoable } from "../lib/useUndoable";
import { cycleSort, type SortState } from "../lib/columnSort";
import { formatRuntime } from "../lib/format";
import { applyView, sameOrder } from "../lib/playlistView";
const PlayerModal = lazy(() => import("./PlayerModal"));
import UndoToolbar from "./UndoToolbar";
@@ -259,6 +260,13 @@ export default function Playlists() {
return (v: Video) => m.get(v.id) ?? 0;
}, [items]);
// Total playtime of the whole playlist (sums every item, matching the itemCount beside it — not
// the filtered `shown` set). Items with an unknown duration count as 0; hidden when nothing sums.
const totalSeconds = useMemo(
() => items.reduce((acc, v) => acc + (v.duration_seconds ?? 0), 0),
[items],
);
const columns = useMemo(
() => playlistColumns({ t, indexOf: storedIndex, onPlay, onRemove, canEdit }),
[t, storedIndex, onPlay, onRemove, canEdit],
@@ -362,7 +370,15 @@ export default function Playlists() {
setPushing(true);
try {
const plan = await api.playlistPushPlan(selectedId);
if (plan.action === "update" && !plan.to_insert && !plan.to_delete && !plan.to_reorder) {
const nothingToDo =
plan.action === "update" &&
!plan.add_yt &&
!plan.add_local &&
!plan.remove_yt &&
!plan.remove_local &&
!plan.to_reorder &&
!plan.rename;
if (nothingToDo) {
notify({ level: "info", message: t("playlists.pushUpToDate", { name: plName(detail) }) });
return;
}
@@ -376,21 +392,37 @@ export default function Playlists() {
});
return;
}
let message =
plan.action === "create"
? t("playlists.pushPlanCreate", {
count: plan.to_insert,
units: plan.units_estimate,
left: plan.remaining_today,
})
: t("playlists.pushPlanUpdate", {
insert: plan.to_insert,
del: plan.to_delete,
reorder: plan.to_reorder,
units: plan.units_estimate,
left: plan.remaining_today,
});
if (plan.yt_extra > 0) message += " " + t("playlists.pushDiverged", { count: plan.yt_extra });
let message: string;
if (plan.action === "create") {
message = t("playlists.pushPlanCreate", {
count: plan.add_yt,
units: plan.units_estimate,
left: plan.remaining_today,
});
} else {
// One sentence per side that actually changes; the untouched side gets a short "stays
// unchanged" reassurance instead of a "+0 / −0" the reader has to decode.
const here: string[] = [];
if (plan.add_local) here.push(t("playlists.mergeAddLocal", { count: plan.add_local }));
if (plan.remove_local)
here.push(t("playlists.mergeRemoveLocal", { count: plan.remove_local }));
const yt: string[] = [];
if (plan.add_yt) yt.push(t("playlists.mergeAddYt", { count: plan.add_yt }));
if (plan.remove_yt) yt.push(t("playlists.mergeRemoveYt", { count: plan.remove_yt }));
if (plan.to_reorder) yt.push(t("playlists.mergeReorder", { count: plan.to_reorder }));
if (plan.rename) yt.push(t("playlists.mergeRename"));
const sentences = [...here, ...yt];
if (here.length && !yt.length) sentences.push(t("playlists.mergeYtUnchanged"));
if (yt.length && !here.length) sentences.push(t("playlists.mergeLocalUnchanged"));
const quota =
plan.units_estimate > 0
? t("playlists.mergeQuota", {
units: plan.units_estimate,
left: plan.remaining_today,
})
: t("playlists.mergeQuotaFree");
message = sentences.join(" ") + quota;
}
const ok = await confirm({
title: t("playlists.pushTitle"),
message,
@@ -567,176 +599,192 @@ export default function Playlists() {
}
return (
<div className="px-4 pb-4 pt-3">
// No top padding: the sticky column header must sit flush at the scroller top so it doesn't
// shift up when `position: sticky` engages on the first scroll (the fixed toolbar band above
// gives the breathing room). Same reason as the channel table — see Channels.tsx.
<div className="px-4 pb-4">
{/* Header (title, actions, view controls) is fixed chrome — it portals into the shell's
fixed band, so only the video list scrolls. */}
<PageToolbar>
<div className="px-4 pt-2 pb-2 max-w-5xl mx-auto">
<div className="flex gap-3 items-start mb-4">
<div className="w-24 h-13.5 rounded-lg bg-card overflow-hidden shrink-0">
{/* Two tight rows beside a small cover: title + actions on the first, the meta line and the
view controls sharing the second. This band is FIXED chrome (it never scrolls), so every
row it sheds is list height won back — the old layout stacked three rows next to the
cover plus a fourth block under it, for roughly twice this height. */}
<div className="px-4 py-2 max-w-5xl mx-auto">
<div className="flex gap-3 items-center">
<div className="w-20 h-11.25 rounded-lg bg-card overflow-hidden shrink-0">
{items[0]?.thumbnail_url && (
<img src={items[0].thumbnail_url} alt="" className="w-full h-full object-cover" />
)}
</div>
<div className="min-w-0 flex-1">
{renaming ? (
<input
autoFocus
value={renameValue}
onChange={(e) => setRenameValue(e.target.value)}
onBlur={saveRename}
onKeyDown={(e) => {
if (e.key === "Enter") saveRename();
if (e.key === "Escape") setRenaming(false);
}}
aria-label={t("playlists.rename")}
className="text-lg font-semibold bg-surface border border-border rounded-md px-2 py-0.5 focus:outline-hidden focus:ring-2 focus:ring-accent/40"
/>
) : (
<div className="flex items-center gap-2 min-w-0">
<h2 className="text-lg font-semibold truncate">{plName(detail)}</h2>
{editable && (
<button
onClick={() => {
setRenameValue(detail.name);
setRenaming(true);
}}
title={t("playlists.rename")}
aria-label={t("playlists.rename")}
className="shrink-0 text-muted hover:text-fg"
<div className="flex items-center gap-2 min-w-0">
{renaming ? (
<input
autoFocus
value={renameValue}
onChange={(e) => setRenameValue(e.target.value)}
onBlur={saveRename}
onKeyDown={(e) => {
if (e.key === "Enter") saveRename();
if (e.key === "Escape") setRenaming(false);
}}
aria-label={t("playlists.rename")}
className="min-w-0 flex-1 text-lg font-semibold bg-surface border border-border rounded-md px-2 py-0.5 focus:outline-hidden focus:ring-2 focus:ring-accent/40"
/>
) : (
<>
<h2 className="text-lg font-semibold truncate">{plName(detail)}</h2>
{editable && (
<button
onClick={() => {
setRenameValue(detail.name);
setRenaming(true);
}}
title={t("playlists.rename")}
aria-label={t("playlists.rename")}
className="shrink-0 text-muted hover:text-fg"
>
<Pencil className="w-3.5 h-3.5" />
</button>
)}
{detail.yt_playlist_id && (
<a
href={`https://www.youtube.com/playlist?list=${detail.yt_playlist_id}`}
target="_blank"
rel="noreferrer noopener"
title={t("playlists.openOnYoutube")}
aria-label={t("playlists.openOnYoutube")}
className="shrink-0 text-muted hover:text-accent"
>
<ExternalLink className="w-3.5 h-3.5" />
</a>
)}
</>
)}
<div className="ml-auto flex gap-1.5 items-center shrink-0">
{linked && detail.dirty && (
<span
title={t("playlists.unsyncedHint")}
className="inline-flex items-center gap-1.5 text-2xs text-accent mr-1"
>
<Pencil className="w-3.5 h-3.5" />
<span className="w-1.5 h-1.5 rounded-full bg-accent" />
{t("playlists.unsynced")}
</span>
)}
<button
onClick={() => shown.length && setPlayingId(shown[0]!.id)}
disabled={!shown.length}
className="inline-flex items-center gap-1.5 text-xs font-semibold px-3 py-1.5 rounded-lg btn-primary disabled:opacity-40 transition"
>
<Play className="w-3.5 h-3.5 fill-current" /> {t("playlists.playAll")}
</button>
{editable && canWrite && (
<button
onClick={pushToYoutube}
disabled={pushing}
title={linked ? t("playlists.pushToYoutube") : t("playlists.exportToYoutube")}
aria-label={
linked ? t("playlists.pushToYoutube") : t("playlists.exportToYoutube")
}
className={`inline-flex items-center justify-center p-2 rounded-lg border transition disabled:opacity-40 ${
detail.dirty
? "border-accent text-accent hover:bg-accent/10"
: "border-border text-muted hover:text-fg hover:border-accent"
}`}
>
{pushing ? (
<RefreshCw className="w-3.5 h-3.5 animate-spin" />
) : linked ? (
<Youtube className="w-3.5 h-3.5" />
) : (
<Upload className="w-3.5 h-3.5" />
)}
</button>
)}
{detail.yt_playlist_id && (
<a
href={`https://www.youtube.com/playlist?list=${detail.yt_playlist_id}`}
target="_blank"
rel="noreferrer noopener"
title={t("playlists.openOnYoutube")}
aria-label={t("playlists.openOnYoutube")}
className="shrink-0 text-muted hover:text-accent"
{linked && detail.dirty && (
<button
onClick={revertToYoutube}
disabled={reverting}
title={t("playlists.revert")}
aria-label={t("playlists.revert")}
className="inline-flex items-center justify-center p-2 rounded-lg border border-border text-muted hover:text-fg hover:border-accent transition disabled:opacity-40"
>
<ExternalLink className="w-3.5 h-3.5" />
</a>
<RotateCcw className={`w-3.5 h-3.5 ${reverting ? "animate-spin" : ""}`} />
</button>
)}
{editable && (
<button
onClick={deletePlaylist}
title={t("playlists.delete")}
aria-label={t("playlists.delete")}
className="inline-flex items-center justify-center p-2 rounded-lg border border-border text-muted hover:text-danger hover:border-danger/50 transition"
>
<Trash2 className="w-3.5 h-3.5" />
</button>
)}
</div>
)}
<div className="text-xs text-muted mt-0.5">
{t("playlists.itemCount", { count: items.length })}
{filterActive && ` · ${t("playlists.filteredCount", { count: shown.length })}`}
</div>
<div className="flex gap-1.5 mt-3 flex-wrap items-center">
<button
onClick={() => shown.length && setPlayingId(shown[0]!.id)}
disabled={!shown.length}
className="inline-flex items-center gap-1.5 text-xs font-semibold px-3 py-1.5 rounded-lg btn-primary disabled:opacity-40 transition"
>
<Play className="w-3.5 h-3.5 fill-current" /> {t("playlists.playAll")}
</button>
{editable && canWrite && (
<button
onClick={pushToYoutube}
disabled={pushing}
title={linked ? t("playlists.pushToYoutube") : t("playlists.exportToYoutube")}
aria-label={
linked ? t("playlists.pushToYoutube") : t("playlists.exportToYoutube")
}
className={`inline-flex items-center justify-center p-2 rounded-lg border transition disabled:opacity-40 ${
detail.dirty
? "border-accent text-accent hover:bg-accent/10"
: "border-border text-muted hover:text-fg hover:border-accent"
}`}
>
{pushing ? (
<RefreshCw className="w-3.5 h-3.5 animate-spin" />
) : linked ? (
<Youtube className="w-3.5 h-3.5" />
) : (
<Upload className="w-3.5 h-3.5" />
<div className="flex items-center gap-2 mt-1 flex-wrap">
<span className="text-xs text-muted">
{t("playlists.itemCount", { count: items.length })}
{totalSeconds > 0 && ` · ${formatRuntime(totalSeconds)}`}
{filterActive && ` · ${t("playlists.filteredCount", { count: shown.length })}`}
</span>
{(items.length > 1 || viewActive) && (
<>
<span aria-hidden="true" className="text-xs text-muted/40">
·
</span>
<label className="inline-flex items-center gap-1.5 text-xs text-muted cursor-pointer select-none">
<input
type="checkbox"
checked={groupBy}
onChange={(e) => setGroupBy(e.target.checked)}
className="accent-accent"
/>
{t("playlists.groupByChannel")}
</label>
{viewActive && (
<button
onClick={clearView}
className="text-xs text-muted hover:text-accent underline underline-offset-2 transition"
>
{t("playlists.clearView")}
</button>
)}
</button>
)}
{linked && detail.dirty && (
<button
onClick={revertToYoutube}
disabled={reverting}
title={t("playlists.revert")}
aria-label={t("playlists.revert")}
className="inline-flex items-center justify-center p-2 rounded-lg border border-border text-muted hover:text-fg hover:border-accent transition disabled:opacity-40"
>
<RotateCcw className={`w-3.5 h-3.5 ${reverting ? "animate-spin" : ""}`} />
</button>
)}
{editable && (
<button
onClick={deletePlaylist}
title={t("playlists.delete")}
aria-label={t("playlists.delete")}
className="inline-flex items-center justify-center p-2 rounded-lg border border-border text-muted hover:text-danger hover:border-danger/50 transition"
>
<Trash2 className="w-3.5 h-3.5" />
</button>
)}
{linked && detail.dirty && (
<span
title={t("playlists.unsyncedHint")}
className="inline-flex items-center gap-1.5 text-2xs text-accent ml-1"
>
<span className="w-1.5 h-1.5 rounded-full bg-accent" />
{t("playlists.unsynced")}
</span>
{canSaveOrder && (
<button
onClick={saveOrder}
className="inline-flex items-center gap-1.5 text-xs font-semibold px-2.5 py-1 rounded-lg border border-accent text-accent hover:bg-accent/10 transition"
>
<Check className="w-3.5 h-3.5" /> {t("playlists.saveOrder")}
</button>
)}
{/* ONE sentence that names the state. Three loose texts used to pile up here
(reset link + save hint + "reordering is off"), and none of them answered the
question a user actually has — "did I just change my playlist?". Worse,
sorting a list whose stored order ALREADY matches the sort produced a
warning, no Save button and no visible change, which reads as a malfunction.
Now the line says which of the three situations you are in; the disabled grip
still explains itself on hover/focus. */}
{viewActive && <span className="text-2xs text-muted/80">{viewStatus}</span>}
{canEdit && (
<div className="ml-auto">
<UndoToolbar
canUndo={order.canUndo}
canRedo={order.canRedo}
onUndo={order.undo}
onRedo={order.redo}
/>
</div>
)}
</>
)}
</div>
</div>
</div>
{(items.length > 1 || viewActive) && (
<div className="flex items-center gap-2 mb-3 flex-wrap max-w-5xl">
<label className="inline-flex items-center gap-1.5 text-xs text-muted cursor-pointer select-none">
<input
type="checkbox"
checked={groupBy}
onChange={(e) => setGroupBy(e.target.checked)}
className="accent-accent"
/>
{t("playlists.groupByChannel")}
</label>
{viewActive && (
<button
onClick={clearView}
className="text-xs text-muted hover:text-accent underline underline-offset-2 transition"
>
{t("playlists.clearView")}
</button>
)}
{canSaveOrder && (
<button
onClick={saveOrder}
className="inline-flex items-center gap-1.5 text-xs font-semibold px-2.5 py-1 rounded-lg border border-accent text-accent hover:bg-accent/10 transition"
>
<Check className="w-3.5 h-3.5" /> {t("playlists.saveOrder")}
</button>
)}
{/* ONE sentence that names the state. Three loose texts used to pile up here (reset
link + save hint + "reordering is off"), and none of them answered the question a
user actually has — "did I just change my playlist?". Worse, sorting a list whose
stored order ALREADY matches the sort produced a warning, no Save button and no
visible change, which reads as a malfunction. Now the line says which of the three
situations you are in; the disabled grip still explains itself on hover/focus. */}
{viewActive && <span className="text-2xs text-muted/80">{viewStatus}</span>}
{canEdit && (
<div className="ml-auto">
<UndoToolbar
canUndo={order.canUndo}
canRedo={order.canRedo}
onUndo={order.undo}
onRedo={order.redo}
/>
</div>
)}
</div>
)}
</div>
</PageToolbar>
+2 -2
View File
@@ -4,7 +4,7 @@ import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { qk } from "../lib/queryKeys";
import { ListVideo, Pin, Plus, RefreshCw, Youtube } from "lucide-react";
import { api, type Playlist } from "../lib/api";
import { formatDuration } from "../lib/format";
import { formatRuntime } from "../lib/format";
import { notify } from "../lib/notifications";
import { notifyYouTubeActionError } from "../lib/youtubeErrors";
import { LS, readAccountMerged, writeAccount } from "../lib/storage";
@@ -247,7 +247,7 @@ export default function PlaylistsRail({
<div className="text-2xs text-muted">
{t("playlists.itemCount", { count: pl.item_count })}
{pl.total_duration_seconds > 0 &&
` · ${formatDuration(pl.total_duration_seconds)}`}
` · ${formatRuntime(pl.total_duration_seconds)}`}
</div>
</div>
</button>
+27
View File
@@ -11,6 +11,7 @@ import { User, X } from "lucide-react";
import { qk } from "../lib/queryKeys";
import { healPlexField } from "../lib/plexSearch";
import { healPlexSort, plexSortOptions, PLEX_SORT_FALLBACK } from "../lib/plexSort";
import type { SortState } from "../lib/columnSort";
import {
api,
EMPTY_PLEX_FILTERS,
@@ -52,6 +53,13 @@ const PlexCollectionView = lazy(() => import("./PlexCollectionView"));
const PAGE = 40;
// D3: the "last watched" sort auto-applied when you enter the in-progress watch-state view (most
// recently watched first). Frozen for the same reason as PLEX_SORT_FALLBACK (handed out by reference).
const LASTWATCHED_SORT: NonNullable<SortState> = Object.freeze({
key: "lastwatched",
dir: "desc" as const,
});
// Watch-state options for the toolbar pill (moved out of the filter rail — it's a top-level "what am
// I looking at" control now, like the feed's Watch-state pill, not a rail widget).
const SHOW_OPTS = ["all", "unwatched", "in_progress", "watched"] as const;
@@ -185,6 +193,25 @@ export default function PlexBrowse() {
return () => window.removeEventListener("keydown", onKey);
}, [sub]);
// D3: entering the in-progress watch-state view auto-applies the "last watched" sort (most recently
// watched first); leaving restores the sort that was active before entering. Only the ENTER edge
// forces the sort, so a manual sort change while in the in-progress view still wins — it's just not
// carried back out (exit returns to the pre-entry preference). We stash the raw provider `sort`, not
// the scope-healed one, so the user's real preference round-trips.
const prevShow = useRef(show);
const preInProgressSort = useRef<SortState | null>(null);
useEffect(() => {
const was = prevShow.current;
if (was !== "in_progress" && show === "in_progress") {
preInProgressSort.current = sort;
setSort(LASTWATCHED_SORT);
} else if (was === "in_progress" && show !== "in_progress") {
if (preInProgressSort.current) setSort(preInProgressSort.current);
preInProgressSort.current = null;
}
prevShow.current = show;
}, [show, sort, setSort]);
const browseQ = useInfiniteQuery({
queryKey: qk.plexLibrary(scope, dq, sortState, show, filters, fieldState, lang),
// Gate on committedLang so a language switch fires ONE fetch (after the genre-clear effect
+30 -1
View File
@@ -19,6 +19,7 @@ import i18n from "../i18n";
import { Carousel } from "./Carousel";
import PlexCollectionEditor from "./PlexCollectionEditor";
import PlexPlaylistAdd from "./PlexPlaylistAdd";
import { PLAYABLE_TINT, playableClass } from "./plexCards";
// The rich "media info" surface for a Plex item — poster/art, IMDb score + link, content rating,
// genres, director, cast (with photos), summary. Reused in TWO places: a full-page view opened from
@@ -253,6 +254,33 @@ function CrewPanel({
);
}
// The playback-class chip (dot + Plex-native label), with a muted codec · bit-depth subtitle when
// known. Class + colour are derived through the SAME shared helper the browse-card corner dot uses,
// so the two surfaces can't drift. `overlay` tints the label for the dark player background.
function PlaybackChip({ detail, overlay }: { detail: PlexItemDetail; overlay: boolean }) {
const { t } = useTranslation();
if (!detail.playable) return null;
const cls = playableClass(detail.playable, detail.transcode_hw);
const sub = [
detail.codec_video ? detail.codec_video.toUpperCase() : null,
detail.video_bit_depth != null ? `${detail.video_bit_depth}-bit` : null,
]
.filter(Boolean)
.join(" · ");
return (
<span
className={`inline-flex items-center gap-1.5 rounded-full px-2 py-0.5 text-xs ${
overlay ? "bg-white/10 text-white/90" : "bg-surface text-muted"
}`}
title={t(`plex.playable.${cls}`)}
>
<span className={`h-2 w-2 shrink-0 rounded-full ${PLAYABLE_TINT[cls] ?? "bg-gray-400"}`} />
<span className="font-medium">{t(`plex.filter.playableOpt.${cls}`)}</span>
{sub && <span className="opacity-70">{sub}</span>}
</span>
);
}
type Props = {
detail: PlexItemDetail;
variant: "page" | "overlay";
@@ -425,7 +453,7 @@ export default function PlexInfo({
<button
onClick={onClose}
className="shrink-0 p-1.5 rounded-lg text-white/80 hover:bg-white/15"
aria-label={t("plex.player.back")}
aria-label={t("plex.player.close")}
>
<X className="w-5 h-5" />
</button>
@@ -479,6 +507,7 @@ export default function PlexInfo({
{detail.studio}
</Filterable>
)}
<PlaybackChip detail={detail} overlay={overlay} />
</div>
{/* Multi-source ratings row (E5 S5): IMDb / TMDb / audience, whichever are present. */}
+161 -11
View File
@@ -9,9 +9,11 @@ import {
ArrowLeft,
Clock,
Download,
Gauge,
Info,
Keyboard,
Languages,
Loader2,
Maximize,
Minimize,
Pause,
@@ -37,8 +39,10 @@ import {
subShadowCss,
type PlexPlayerPrefs,
} from "../lib/plexPlayerPrefs";
import { Ctrl, PanelHead } from "./plexPlayerControls";
import { Ctrl, PanelHead, Skip10 } from "./plexPlayerControls";
import { PlexPlayerSettings } from "./PlexPlayerSettings";
import { PlexStatsOverlay } from "./PlexStatsOverlay";
import { UpNextCard } from "./UpNextCard";
import {
useAutoSkip,
useControlsVisibility,
@@ -94,6 +98,12 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
const aliveRef = useRef(true);
const wrapRef = useRef<HTMLDivElement>(null);
const sessionStartRef = useRef(0); // absolute offset the current HLS/direct session begins at
// Which encoder the current transcode session ran: true=VAAPI hardware, false=libx264 software,
// null=not a transcode. Feeds the stats overlay's mode row ("transcode · GPU" / "· CPU").
const hwRef = useRef<boolean | null>(null);
// Whether the current HW transcode tone-mapped a 10-bit/HDR source on the GPU (OpenCL). Feeds the
// stats mode row's "· HDR→SDR" tag; null/false for an 8-bit HW, software, or non-transcode session.
const tonemapRef = useRef<boolean | null>(null);
const durationRef = useRef(0);
// ≥2 audio tracks → multi-rendition HLS: all tracks ship in one session and hls.js switches audio
// client-side (no restart). Ref so the stable loadSession/changeAudio callbacks read it fresh.
@@ -108,6 +118,15 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
const [seekableEnd, setSeekableEnd] = useState(0); // absolute end of the loaded region
const { fs, toggleFs } = useFullscreen(wrapRef);
const [ready, setReady] = useState(false);
// True while playback is stalled waiting for data (mid-stream rebuffer) or a seek is in flight — drives
// a spinner so a stall reads as "loading, hang on" instead of a frozen frame. Distinct from `ready`,
// which only covers the INITIAL session load; this covers stalls AFTER playback has started (e.g. the
// brief cold-ffmpeg rebuffer on auto-advance, now largely prevented by the server-side segment warm-up).
const [buffering, setBuffering] = useState(false);
// The current session re-encodes the video (playable="transcode"). Its cold-start (the first transcode
// segment) is genuinely slower than a remux stall, so the initial-load overlay says "Preparing stream…"
// instead of the generic "Loading…" — honest, and it stops a warm-up reading as a hang.
const [sessionIsTranscode, setSessionIsTranscode] = useState(false);
const [loadError, setLoadError] = useState<string | null>(null);
// The current session's real media-start K (reactive mirror of sessionStartRef). Subtitles are
// absolute-time VTT shifted by K to the session's zero-based clock, so the <track> src/key keys off
@@ -165,6 +184,16 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
const [infoOpen, setInfoOpen] = useState(false);
const infoOpenRef = useRef(false);
infoOpenRef.current = infoOpen;
// "D" toggles the stats-for-nerds debug overlay (active mode, codec, dropped frames, buffer…).
const [statsOpen, setStatsOpen] = useState(false);
const statsOpenRef = useRef(false);
statsOpenRef.current = statsOpen;
// Up-next end-card: shown at episode end (with a next item) instead of an instant jump (S4).
const [upNextOpen, setUpNextOpen] = useState(false);
const upNextOpenRef = useRef(false);
upNextOpenRef.current = upNextOpen;
const upNextPrefRef = useRef(prefs.upNext);
upNextPrefRef.current = prefs.upNext;
const now = useWallClock();
durationRef.current = duration;
@@ -178,7 +207,12 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
const video = videoRef.current;
if (!video) return;
setReady(false);
setBuffering(false);
setLoadError(null);
// Clear the prior item's transcode flag up front so the cold-start overlay can't show a stale
// "Preparing stream…" for a not-yet-known session while the plexSession request is in flight —
// it's re-set honestly from the response below.
setSessionIsTranscode(false);
let sess;
try {
// Multi-audio → request the multi-rendition session (no single `audio` ordinal); the track is
@@ -199,6 +233,12 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
);
return;
}
// Surface the active mode for the stats overlay + the honest cold-start label: a transcode session
// re-encodes the video (hw = which encoder), and its first segment is genuinely slower than remux.
const isTranscode = sess.mode === "hls" && !!sess.transcode;
hwRef.current = isTranscode ? (sess.hw ?? false) : null;
tonemapRef.current = isTranscode ? (sess.tonemap ?? false) : null;
setSessionIsTranscode(isTranscode);
// hls.js plays our non-zero-start copyts stream ~1s BEHIND the media clock (see HLS_SUB_LAG);
// fold that into the session start so the seekbar clock, seeking AND the subtitle shift are all
// content-accurate. Direct-play (start_s 0) uses the file's real clock — no compensation.
@@ -221,7 +261,15 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
hlsRef.current = null;
}
if (sess.mode === "hls" && Hls.isSupported()) {
const hls = new Hls({ maxBufferLength: 30, enableWorker: true });
// startPosition: 0 — our HLS is a growing EVENT playlist with no ENDLIST (see _HLS_TAIL in
// stream.py), which hls.js classifies as LIVE. Its default (startPosition: -1) then starts
// playback at the live edge and hugs it, so it never builds a forward buffer (rebuffer
// spinners), and on a resume/seek session the segments' absolute PTS (-copyts, offset by the
// server's -ss) don't line up with the live-edge timeline → 0 fragments load → permanent
// stall. But the seek-restart model means fragment 0 IS the requested content position
// (`?t=` positioned the server ffmpeg there), so we always want to start VOD-style at 0 and
// buffer forward up to maxBufferLength. Applies to remux + transcode alike (both use copyts).
const hls = new Hls({ maxBufferLength: 30, enableWorker: true, startPosition: 0 });
hlsRef.current = hls;
hls.loadSource(`${sess.url}?t=${Math.floor(startAt)}`);
hls.attachMedia(video);
@@ -362,12 +410,24 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
// button lie (⏸ shown while actually paused). Resync the flag from reality on the settling events
// so the icon can never contradict the video.
const syncPlaying = () => setPlaying(!video.paused);
// Buffering spinner: `waiting`/`stalled` = playback stopped for lack of data; `seeking` = a seek is
// in flight. Cleared once data flows again (`playing`/`canplay`/`seeked`) or the user pauses (a
// deliberate pause isn't a stall, and would otherwise leave the spinner stuck).
const onWaiting = () => setBuffering(true);
const onResumed = () => setBuffering(false);
video.addEventListener("timeupdate", onTime);
video.addEventListener("play", onPlay);
video.addEventListener("pause", onPause);
video.addEventListener("emptied", syncPlaying);
video.addEventListener("canplay", syncPlaying);
video.addEventListener("seeked", syncPlaying);
video.addEventListener("waiting", onWaiting);
video.addEventListener("stalled", onWaiting);
video.addEventListener("seeking", onWaiting);
video.addEventListener("playing", onResumed);
video.addEventListener("canplay", onResumed);
video.addEventListener("seeked", onResumed);
video.addEventListener("pause", onResumed);
return () => {
video.removeEventListener("timeupdate", onTime);
video.removeEventListener("play", onPlay);
@@ -375,6 +435,13 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
video.removeEventListener("emptied", syncPlaying);
video.removeEventListener("canplay", syncPlaying);
video.removeEventListener("seeked", syncPlaying);
video.removeEventListener("waiting", onWaiting);
video.removeEventListener("stalled", onWaiting);
video.removeEventListener("seeking", onWaiting);
video.removeEventListener("playing", onResumed);
video.removeEventListener("canplay", onResumed);
video.removeEventListener("seeked", onResumed);
video.removeEventListener("pause", onResumed);
};
}, [id]);
@@ -488,7 +555,11 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
// (absRef 0) — its stray 'ended' must not mark anything watched or cascade another advance.
if (absRef.current <= 0) return;
api.plexSetState(id, "watched").catch(() => {});
if (nextId) go(nextId);
// With a next item and the up-next pref on, surface the end-card (poster + countdown + Cancel)
// and let IT advance — binge stays in the viewer's control (S4/D6-A). The card only appears here,
// on play-to-the-end; credits-auto-skip advances earlier and this handler never fires for it.
if (nextId && upNextPrefRef.current) setUpNextOpen(true);
else if (nextId) go(nextId);
// Nothing queued next → dismiss the player back to the previous page (same as Stop/Escape),
// instead of stranding a black, ended <video> on screen.
else onClose();
@@ -497,6 +568,14 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
return () => video.removeEventListener("ended", onEnded);
}, [id, nextId, go, onClose]);
// Up-next card actions: advance = go to the next item (it was already marked watched on 'ended');
// cancel = dismiss and stay on the finished frame. Loading a new item closes any lingering card.
const advanceUpNext = useCallback(() => {
setUpNextOpen(false);
if (nextId) go(nextId);
}, [nextId, go]);
useEffect(() => setUpNextOpen(false), [id]);
// "Back" cascade shared by the on-screen Back arrow AND the keyboard/remote Backspace·Escape: peel
// off whatever is layered on top (a tuning/tracks menu, then the shortcuts sheet, then the info
// overlay) before leaving the player. The arrow used to call onClose() unconditionally, so Back
@@ -507,6 +586,8 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
setTracksOpen(false);
} else if (helpOpenRef.current) setHelpOpen(false);
else if (infoOpenRef.current) setInfoOpen(false);
else if (statsOpenRef.current) setStatsOpen(false);
else if (upNextOpenRef.current) setUpNextOpen(false);
else onClose();
}, [onClose]);
@@ -592,6 +673,10 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
case "I":
setInfoOpen((v) => !v);
break;
case "d":
case "D":
setStatsOpen((v) => !v);
break;
case "Backspace":
// HTPC-style "stop & back to the feed" — mirrors the mouse Back button (shared cascade).
e.preventDefault();
@@ -613,6 +698,8 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
setTracksOpen(false);
} else if (helpOpenRef.current) setHelpOpen(false);
else if (infoOpenRef.current) setInfoOpen(false);
else if (statsOpenRef.current) setStatsOpen(false);
else if (upNextOpenRef.current) setUpNextOpen(false);
else if (!fullscreenElement()) onClose();
break;
}
@@ -730,21 +817,44 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
)}
</video>
{!ready && (
<div className="absolute inset-0 grid place-items-center px-6 text-center text-sm pointer-events-none">
{/* Loading / buffering / error overlay. `!ready` = initial session load; `buffering` = a
mid-playback stall or in-flight seek (see the state); `loadError` surfaces a fatal failure so
it can't hide behind a running video. Spinner on load/stall, error text otherwise. */}
{(!ready || buffering || loadError) && (
<div
role="status"
aria-live="polite"
className="absolute inset-0 grid place-items-center px-6 text-center text-sm pointer-events-none"
>
{loadError ? (
<span className="max-w-md text-danger">{loadError}</span>
) : (
<span className="text-white/70">{t("plex.player.loading")}</span>
<div className="flex flex-col items-center gap-3 text-white/80">
<Loader2 className="w-10 h-10 animate-spin" aria-hidden />
{/* Announce the stall to assistive tech. Initial load shows visible text; a mid-playback
rebuffer/seek (ready, no error) has no visible label, so give it an sr-only one via the
aria-live region so a screen reader distinguishes "buffering" from a frozen frame. */}
{!ready ? (
<span>
{t(sessionIsTranscode ? "plex.player.preparing" : "plex.player.loading")}
</span>
) : buffering ? (
<span className="sr-only">{t("plex.player.buffering")}</span>
) : null}
</div>
)}
</div>
)}
{/* While a panel/menu is open or an auto-skip countdown is running, take over the browser/mouse
Back button (history popstate) so it closes/cancels that instead of dropping to the grid. */}
{(menuOpen || tracksOpen || helpOpen || infoOpen || skipProgress != null) && (
<BackClose onBack={backIntent} />
)}
{(menuOpen ||
tracksOpen ||
helpOpen ||
infoOpen ||
statsOpen ||
upNextOpen ||
skipProgress != null) && <BackClose onBack={backIntent} />}
{/* Top bar */}
<div
@@ -893,9 +1003,19 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
</div>
<div className="flex items-center gap-3 text-white">
<Ctrl label={t("plex.player.playPause")} onClick={togglePlay} align="start">
<Ctrl
label={t("plex.player.rewind10")}
onClick={() => seekTo(absRef.current - 10)}
align="start"
>
<Skip10 dir="back" />
</Ctrl>
<Ctrl label={t("plex.player.playPause")} onClick={togglePlay}>
{playing ? <Pause className="w-6 h-6" /> : <Play className="w-6 h-6" />}
</Ctrl>
<Ctrl label={t("plex.player.forward10")} onClick={() => seekTo(absRef.current + 10)}>
<Skip10 dir="forward" />
</Ctrl>
<Ctrl label={t("plex.player.restart")} onClick={() => seekTo(0)}>
<RotateCcw className="w-5 h-5" />
</Ctrl>
@@ -1071,6 +1191,9 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
)}
</div>
)}
<Ctrl label={t("plex.player.stats.title")} onClick={() => setStatsOpen((v) => !v)}>
<Gauge className="w-5 h-5" />
</Ctrl>
<Ctrl label={t("plex.player.infoBtn")} onClick={() => setInfoOpen((v) => !v)}>
<Info className="w-5 h-5" />
</Ctrl>
@@ -1087,6 +1210,32 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
</div>
</div>
{/* Stats-for-nerds debug overlay (toggle with "d" or the gauge button). Non-modal corner panel —
the video keeps playing and stays interactive behind it. */}
{statsOpen && (
<PlexStatsOverlay
videoRef={videoRef}
hlsRef={hlsRef}
playable={detail?.playable ?? ""}
hwRef={hwRef}
tonemapRef={tonemapRef}
sessionStartRef={sessionStartRef}
absRef={absRef}
durationRef={durationRef}
onClose={() => setStatsOpen(false)}
/>
)}
{/* Up-next end-card (episode end, next item queued, pref on). Non-modal — the finished frame
stays behind it; the card's own countdown advances unless the viewer cancels. */}
{upNextOpen && nextId && (
<UpNextCard
nextId={nextId}
onAdvance={advanceUpNext}
onCancel={() => setUpNextOpen(false)}
/>
)}
{/* Keyboard-shortcut cheat sheet (toggle with "h" or the keyboard button). */}
{helpOpen && (
<div
@@ -1107,7 +1256,7 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
<button
onClick={() => setHelpOpen(false)}
className="p-1 rounded-lg hover:bg-white/10"
aria-label={t("plex.player.back")}
aria-label={t("plex.player.close")}
>
<X className="w-5 h-5" />
</button>
@@ -1127,6 +1276,7 @@ export default function PlexPlayer({ itemId, onClose, queue }: Props) {
[["S"], "plex.player.keys.subtitle"],
[["F"], "plex.player.keys.fullscreen"],
[["I"], "plex.player.keys.info"],
[["D"], "plex.player.keys.stats"],
[["⌫", "Esc"], "plex.player.keys.back"],
[["H"], "plex.player.keys.help"],
] as [string[], string][]

Some files were not shown because too many files have changed in this diff Show More