Compare commits

..
20 Commits
Author SHA1 Message Date
peter fa176a3e6a release: 0.68.0 (improvement/e2-s6-job-index-hygiene)
gate / frontend (push) Successful in 52s
gate / backend (push) Successful in 1m37s
2026-08-18 19:20:43 +02:00
peter e8771f4d6b chore(release): 0.68.0 — E2 Feed & Search performance epic
Bump VERSION 0.67.0 -> 0.68.0 and add the user-facing release-notes entry for
the E2 epic (feed sort index, keyset/NOT NULL, search GIN + state inner-joins,
single /feed/summary pass, no card-action refetch storm, job partial indexes).
2026-08-18 19:20:18 +02:00
peter 94dbe4b228 fix(jobs): match the boolean partial predicates to the IS TRUE/IS FALSE queries (E2 S6 review)
The via_search / shorts_probed partials used `WHERE via_search` / `WHERE NOT
shorts_probed`, but the queries filter with `.is_(True)`/`.is_(False)` (SQL
`IS TRUE`/`IS FALSE`), which Postgres's partial-index prover does NOT match to a
bare-boolean predicate — so both fell back to a full seq scan (the shorts-probe
job kept full-scanning for its work queue, the exact thing PERF-3 meant to fix).

Align the predicates to the query form: `WHERE via_search IS TRUE` and
`WHERE shorts_probed IS FALSE`. Verified on the prod-sized localdev: both now
Index Only Scan (via_search 44,683 -> 3 buffers; shorts_probed 44,683 -> 1).
Gate green (346 pytest).
2026-08-18 18:31:38 +02:00
peter 0ed4ac6958 perf(jobs): partial indexes for the enrich/maintenance scanners + boolean cleanup (E2 S6, PERF-17/18/3)
Two scheduler jobs full-seq-scanned the 257k-row videos table every run:
- PERF-17 enrich_pending (`WHERE enriched_at IS NULL LIMIT N`, every few minutes):
  44,680 buffers even with 0 pending -> a partial ix_videos_enrich_pending makes
  it a 1-buffer index scan (empty when nothing pending).
- PERF-18 maintenance revalidate (`WHERE unavailable_since IS NULL ORDER BY
  last_checked_at NULLS FIRST LIMIT N`, ~300 pages/run): 44,754 buffers + top-N
  sort -> a partial ix_videos_recheck_order in that exact order = a 42-buffer
  index scan. The order_by is rewritten to `asc().nulls_first()` (equivalent to
  the old `(IS NULL) DESC, ASC`) so it rides the index; the unused plain
  ix_videos_last_checked_at is dropped.

PERF-3: drop the never-picked boolean btrees (is_short, via_explore) and slim the
useful ones to partials over the queried subset (via_search=true, NOT
shorts_probed, unavailable_since IS NOT NULL) — less write amplification on the
biggest table, same lookups. Model: those 6 columns lose their inline index=True
(the partials are migration-defined, like the search_vector GIN); live_status
left alone.

Measured on the prod-sized localdev reseed: enrich 44,680 -> 1 buffer (25.9 ->
0.02 ms), maintenance 44,754 -> 42 buffers (35.5 -> 0.17 ms); all 11 CONCURRENTLY
index ops on 257k rows in 0.75s. Gate green (346 pytest incl. drift + downgrade
round-trip).
2026-08-18 17:01:13 +02:00
peter 8b8b0ca5ab perf(feed): skip the empty IN() query in tags_by_category (E2 review round 2)
The tags_by_category extraction dropped feed_summary's `if selected:` guard, so
a /feed/summary request with no tag filter (the common case) ran a pointless
`SELECT ... WHERE id IN ()` round-trip. Early-return {} for empty tag_ids —
restores the no-query hot path and covers filtered_query + future callers too.
Gate green (346 pytest).
2026-08-18 16:49:29 +02:00
peter 6116ad4bf4 refactor(feed): address /code-review high findings on the E2 stack
Review round on the E2 stack (S1-S5). No correctness bugs; the surviving
low/medium findings addressed:

- altitude (#1): extract the shared `tags_by_category` helper used by BOTH
  filtered_query (SQL tag WHERE) and feed_summary (Python facet/count), removing
  the category-grouping duplication, and add bidirectional "keep in lockstep —
  equivalence-tested by test_feed_summary_matches_count_plus_facets" comments so
  the SQL<->Python tag rule can't silently drift.
- test-coverage (#2): the /feed/summary equivalence test seeded only system tags,
  leaving the `ChannelTag.user_id == user.id` visibility branch untested. Add a
  user-owned Tag + link + two combos; the suite is now 346 (was 344) and the new
  cases pass (feed_summary agrees with /feed/count + /facets on user-visible tags).
- dead-code (#3): annotate get_feed_count, get_facets, and the FE api.feedCount/
  facets as "SUPERSEDED by /feed/summary — kept only as the equivalence oracle;
  retire together" so the intentionally-retained surface reads as deliberate.

Not changed (confirmed no-op): the search channel-name resolve (#4) is
once-per-request on a 633-row table and a deliberate pre-resolve for PERF-12's
BitmapOr plan; the facet loop (#5) already computes passes() once per
(channel, category) with a distinct exclude each, so there is no recomputation.

Gate green: ruff + 346 pytest; frontend tsc/knip/prettier (FE change is a comment).
2026-08-18 16:39:04 +02:00
peter 943143ef45 perf(feed): stop the card-action + player-close feed refetch storm (E2 S5, PERF-2)
Marking a card watched/hidden/reset and closing the player each fired a full
`invalidateQueries(qk.feed())`, refetching EVERY loaded infinite-query page — 10
pages x a few quick marks = ~50 feed refetches, each a full scan (x PERF-1). The
optimistic `overrides` layer + the matchesView filter already render the correct
state (a hidden/watched card drops out instantly), so the refetch was pure waste.

Switch these paths to `refetchType: "none"`: the queries are marked stale but not
refetched now; the next filter change / remount pulls fresh server data. Applies
to the two Feed card-action paths (setState/clearState) and the PlayerModal
close (feed + playlists — progress is already persisted server-side, only the
on-screen resume bar waits for the next natural refetch).

Left as immediate refetch on purpose: channel/tag/playlist mutations (one-off, on
pages where the main feed is unmounted anyway) and the onboarding / notification-
undo paths (which must reveal the changed content right away). Gate green:
frontend tsc/eslint/knip/prettier/222 vitest (backend unchanged, 344 pytest).
2026-08-18 15:49:49 +02:00
peter 21a3aae700 perf(feed): collapse count + facets into one /feed/summary pass (E2 S4, PERF-13)
The FE fired /feed/count (1 pass) + /facets (3 passes) on every filter change =
4 full scans of the subscribed set. New /api/feed/summary does ONE grouped query
(the base filter with tags stripped, GROUP BY channel_id -> per-channel video
counts) and derives the total count + every tag's distinct-channel facet count in
Python on the ~few-hundred-row channel map, faithfully mirroring filtered_query's
channel-based tag logic + the /facets drill-down (own category dropped, unless a
conjunctive topic-AND). Returns {count, facets} — the same numbers the two old
endpoints returned.

FE: a single shared query (Feed reads .count, Sidebar reads .facets); same key +
filters, so React Query dedupes to ONE request. Invalidations repointed to
qk.feedSummary. The old /feed/count + /facets endpoints (and their now-unused FE
api/qk) are kept as the equivalence-test oracle + safety net; a later cleanup can
drop them.

New equivalence test (test_feed_views, 8 filter combos incl. cross-category,
disjunctive/conjunctive topic modes, search): /feed/summary count == /feed/count
and facets == /facets, byte-identical. Structural win: 2 requests / 4-6 queries
-> 1 request / 2-3 queries (the review measured 530 -> 144 ms). Gate green:
344 pytest, frontend tsc/eslint/knip/prettier/222 vitest.
2026-08-18 05:01:27 +02:00
peter 6a35e83ded perf(feed): un-block the search GIN + inner-join the state views (E2 S3, PERF-12/16)
PERF-12: the search ORed `unaccent(channels.title) ILIKE '%q%'` straight into
the per-video predicate, which forced a full scan + detoast of every
search_vector and a per-candidate channel fetch. Pre-resolve the matching
channel ids (a few-hundred-row scan) and match `Video.channel_id.in_(ids)`
instead, so the planner keeps the search_vector GIN as a BitmapOr branch
(alongside ix_videos_channel_pub_desc). Empty match -> false() collapses the OR.

PERF-16: show=watched|hidden|in_progress used an outer join + coalesce(status,
'new'), so the planner couldn't start from the user's small video_states set.
Those views only return rows that HAVE a state, so inner-join video_states and
filter the raw column (status is NOT NULL on a present row, so it's identical).
'unwatched'/'all' keep the outer join. Dropped the now-dead position_expr.

New regression tests (test_feed_views): search returns both text- and channel-
name matches; the four watch-state views return the exact expected sets.

Measured on the prod-sized localdev reseed (prod planner, 0067/0068 applied):
- search /feed/count q=magyar: 663,463 -> 19,206 buffers (1,406 -> 31 ms; OLD
  seq-scan+detoast -> NEW BitmapOr GIN+channel index)
- search feed page: 2,908 -> 1,479 buffers (32.6 -> 1.4 ms)
- show=watched (user with 64 watched): 40,661 -> 264 buffers (~154x)
Gate green (336 pytest, incl. the 2 new equivalence tests).
2026-08-18 04:41:16 +02:00
peter 8c08ea3133 perf(feed): published_at NOT NULL + range-anchored keyset for deep pages (E2 S2, PERF-15)
A nullable published_at forced the feed keyset into a non-sargable
`... OR published_at IS NULL` disjunction: with the 0067 index the planner still
applied it as a FILTER and scanned from the top of the index, so a deep `newest`
page discarded every earlier row (O(depth)).

- Migration 0068: backfill any NULLs (0 in prod) -> SET NOT NULL + DEFAULT now().
- Model: Video.published_at is now non-Optional, nullable=False, server_default now().
- feedquery: the `pub` key is nullable=False, and keyset_predicate is rebuilt
  recursively so a non-nullable leading key gets a non-strict range bound
  (`published_at <= v`) as an AND conjunct — an index range Cond instead of a
  filter. Nullable keys keep the NULL-aware OR form (a `<= v` bound would drop
  their NULL group). Equivalence proven by test_feed_keyset (all 13 sorts,
  cursor-walk == unpaged).
- Ingest paths (stub_from_playlist_item, apply_video_details, playlists sync)
  fall back to now() so they never write an explicit NULL.

Measured on the prod-sized localdev reseed at a ~50k-deep cursor (0067 index +
prod planner): OLD `Rows Removed by Filter: 50001`, 25,687 buffers / 9.16 ms ->
NEW `Index Cond: published_at <= v`, 1 row filtered, 47 buffers / 0.037 ms. SET
NOT NULL on 257k rows: 0.32s. Gate green (334 pytest).
2026-08-18 04:30:07 +02:00
peter 3af3799b25 perf(feed): index the default published_at DESC NULLS LAST sort (E2 S1, PERF-1)
The feed's default `newest` sort is `published_at DESC NULLS LAST, id ASC`, but
the only published_at index was a plain ASC btree — read backwards it satisfies
DESC NULLS FIRST, not NULLS LAST — so every feed page fell back to a full
`videos` seq scan + top-N sort. Migration 0067 adds the two matching indexes
(global + per-channel) CONCURRENTLY and drops the now-redundant single-column
ix_videos_channel_id (the channel composite leads with channel_id, a strict
superset).

Measured on the prod-sized localdev reseed (257,312 videos, prod planner):
- global newest:  44,747 buffers / 154 ms  ->  60 buffers / 0.32 ms
- per-channel(12.6k): 7,604 buffers / 96 ms -> 54 buffers / 0.14 ms
CONCURRENTLY build of both on 257k rows: 1.7s total (~0 downtime). This is the
likely root cause of the "prod feed is slow" symptom.

Model: drop the inline index=True on Video.channel_id (the composite supersedes
it); the composites are migration-defined like the search_vector GIN. Gate green
(334 pytest incl. the migration drift + downgrade/re-upgrade round-trip).
2026-08-18 04:16:08 +02:00
peter aff5a9922c release: 0.67.0 (security/e1-s5-auth-abuse)
gate / frontend (push) Successful in 51s
gate / backend (push) Successful in 1m26s
2026-08-18 03:43:03 +02:00
peter cace9a004d chore(release): 0.67.0 — E1 Foundation epic
Bump VERSION 0.66.0 -> 0.67.0 and add the user-facing release-notes entry for
the E1 Foundation epic (data-safety fixes, account-switch + search-error fixes,
graceful key-mismatch degrade; plus the security/test-net/CI/repro chores).
2026-08-18 03:41:28 +02:00
peter bfc104a9b4 test(migrations): only restore schema in finally when the re-upgrade failed
Review round 2: the unconditional `command.upgrade(cfg, "head")` in finally ran
a redundant no-op on the success path and, on a genuinely broken re-upgrade,
replaced the try's original exception with its own. Gate the restore on a
`reupgraded` flag so it fires only when the upgrade didn't complete — the
cascade-prevention intent is kept without the double-run or error-masking.
2026-08-18 03:40:11 +02:00
peter 0340d0981c fix(e1): address /code-review high findings on the E1 stack
Review round on the whole E1 stack (main..HEAD). 6 findings, all low/medium,
none blocking; fixed for a clean gate-green handoff.

- plex/sync (correctness): the un-enriched cast guard compared the stored cast
  against THIS listing's length, so a trimmed Plex cast on a not-yet-enriched
  row was never applied. Compare against the listing CAP (_LISTING_CAST_CAP=20)
  instead: >cap = enriched (keep, survives a watermark reset), <=cap =
  listing-sourced (track edits in both directions). +2 regression tests.
- main (security): add X-Frame-Options: DENY as the legacy-browser fallback for
  the CSP frame-ancestors clickjacking guard.
- downloads/service (altitude): finalize_if_running now uses a SQLAlchemy 2.0
  update() construct instead of a hand-built text() UPDATE string — same atomic
  WHERE status='running', idiomatic, no identifier interpolation, no manual
  StrEnum coercion. Drops the now-unused text import.
- auth/admin (reuse): extract the SEC-10 "drop untrusted preset password + bump
  epoch" invariant into one helper (revoke_preset_password_if_unverified) so the
  Google-callback and admin-activation paths can't drift.
- tests/migrations (robustness): guarantee the shared schema is restored to head
  in a finally, so a downgrade/re-upgrade failure can't cascade into later tests.
- worker (simplification): align _finish_from_cache's display_name fill with the
  _download path's guard (no behavior change).

Gate green: ruff clean, 334 pytest (0 skip), frontend unchanged.
2026-08-18 03:30:23 +02:00
peter 18edbecb1e fix(auth): close the account pre-emption + wrong-account link holes (E1 S5, SEC-10/11)
SEC-10 (account pre-emption / "pre-hijack"): a password set on an account whose
email was never verified is untrusted — whoever registered the address (to
pre-empt the real owner) could have chosen it. Both the Google-adoption callback
and admin activation (_activate_user_by_email) now drop that pre-set password and
bump the session epoch when the account isn't email_verified, so a later Google
login / admin approval can't hand the pre-set password a working login. No-op for
a verified account (the legit no-SMTP flow). +2 tests.

SEC-11 (link/upgrade bound to the session default): the OAuth link/upgrade GETs
resolved to the session DEFAULT account, so a tab acting as a secondary account
would link Google / the YouTube grant to the wrong account. The frontend now
threads the active account through the navigation URL (withActiveAccount ->
?account=<active>), and _complete_link accepts any wallet account, not only the
default, so the identity lands on the account that started the flow.
2026-08-18 03:02:34 +02:00
peter 8ffa65a9da fix(security): bump cryptography 46.0.7 -> 50.0.0 to clear 4 CVEs
gate / frontend (push) Successful in 54s
gate / backend (push) Successful in 1m36s
The pip-audit step added in E1 S4 flagged cryptography 46.0.7 for PYSEC-2026-3552
/3553/3554 and GHSA-537c-gmf6-5ccf; all are fixed by 50.0.0. The requirements.txt
ceiling `<47` was blocking the upgrade — relaxed to `>=50.0.0,<51`. The lock is a
surgical bump (only the cryptography line changed; cffi 2.1.1 already satisfies
it, no other package moved). pip-audit -r requirements.lock is now clean.

Add tests/test_security.py (there was none for the crypto primitives): the
Fernet encrypt/decrypt round-trip, decrypt_optional degrading on garbage and on a
key mismatch (the B-core-2 path), and the argon2 password hash/verify round-trip
— so a future crypto-lib bump can't silently break the token-at-rest path. Gate
green: 330 pytest.
2026-08-18 02:47:05 +02:00
peter f54dd98129 test(release): lock the VERSION file and RELEASE_NOTES[0] together (E1 S4, DX-6)
gate / frontend (push) Successful in 1m9s
gate / backend (push) Successful in 1m27s
A vitest that fails the gate when the newest release-notes entry's version
diverges from the repo-root VERSION file (the two hand-edited version sources),
plus a no-duplicate-versions check. Catches the "bumped one, forgot the other"
release mistake.
2026-08-18 02:34:45 +02:00
peter 44d4e4876d build(deps): pin a reproducible lock + pip-audit; bump node 20->22 (E1 S4, SEC-5/DX-3)
requirements.txt stays the source (floor ranges); requirements.lock pins the
exact resolved versions (generated in the python:3.12-slim prod base, extras like
psycopg-binary/uvloop included). The prod Dockerfile, Dockerfile.test and CI now
install the lock, so the image is reproducible and the versions are the ones
pip-audit scans. CI gains a non-blocking pip-audit step over the lock. The
frontend build + CI move from the EOL node 20 to node 22.
2026-08-18 02:34:45 +02:00
peter e4ac061721 feat(security): add baseline security response headers (E1 S4, SEC-1)
A small, safe set applied to every response via middleware: X-Content-Type-Options
nosniff, Referrer-Policy strict-origin-when-cross-origin, CSP frame-ancestors
'none' (anti-clickjacking; unrelated to the YouTube/Plex embeds we host), and
Permissions-Policy disabling camera/mic/geolocation. A full CSP and HSTS are
deferred (HSTS belongs on the Caddy edge).
2026-08-18 02:34:45 +02:00
41 changed files with 926 additions and 115 deletions
+13 -8
View File
@@ -3,12 +3,12 @@
# frontend — tsc (3 projects) + eslint + prettier --check + knip + vitest
# backend — ruff + the full pytest suite (pure-logic + the DB-backed lane) against a real Postgres
#
# Runner: the lab act_runner (LXC 115) maps `ubuntu-latest` -> node:20-bookworm and mounts the Docker
# socket, so service containers work. The backend runs pytest NATIVELY against a `postgres` service
# (not the local docker-compose.test.yml, whose bind-mount is unreliable under a mounted-socket runner);
# fidelity is kept by installing the SAME pinned requirements, Postgres 16, and ffmpeg, and by setting
# SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that failed to come up is a hard error, never a silent skip.
# node 20 matches the prod build image (Dockerfile `node:20-alpine`); S4's node-22 bump moves both.
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
# failed to come up is a hard error, never a silent skip. node 22 matches the prod build image
# (Dockerfile `node:22-alpine`).
name: gate
on: [push, pull_request]
@@ -16,7 +16,7 @@ on: [push, pull_request]
jobs:
frontend:
runs-on: ubuntu-latest
container: node:20-bookworm
container: node:22-bookworm
defaults:
run:
working-directory: frontend
@@ -56,9 +56,14 @@ jobs:
- run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client
- name: Wait for Postgres
run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done
- run: pip install --no-cache-dir -r backend/requirements.txt -r backend/requirements-dev.txt
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
- name: ruff
run: python -m ruff check --config backend/ruff.toml backend scripts
- name: pytest
run: python -m pytest
working-directory: backend
# Dependency CVE audit over the pinned lock. Non-blocking for now (informational) — a real CVE
# shows up in the run without failing the gate; tighten to a hard fail once the noise is known.
- name: pip-audit (advisory)
run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock
continue-on-error: true
+6 -3
View File
@@ -10,7 +10,7 @@ ARG GIT_SHA=unknown
ARG BUILD_DATE=
# Stage 1: build the frontend SPA
FROM node:20-alpine AS frontend
FROM node:22-alpine AS frontend
WORKDIR /fe
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
COPY frontend/package.json frontend/package-lock.json ./
@@ -57,9 +57,12 @@ RUN apt-get update \
&& rm /tmp/deno.zip \
&& rm -rf /var/lib/apt/lists/*
COPY backend/requirements.txt .
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
COPY backend/requirements.lock .
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && pip install -r requirements.txt
pip install --upgrade pip && pip install -r requirements.lock
COPY --chown=appuser backend/ .
# Read at runtime by config.py as the app_version (a committed single source of truth).
+1 -1
View File
@@ -1 +1 @@
0.66.0
0.68.0
+4 -2
View File
@@ -14,5 +14,7 @@ WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt requirements-dev.txt ./
RUN pip install --no-cache-dir -r requirements.txt -r requirements-dev.txt
# Install the SAME pinned lock as the prod image (+ the test-only tooling), so the tests run against
# the exact dependency versions prod ships.
COPY requirements.lock requirements-dev.txt ./
RUN pip install --no-cache-dir -r requirements.lock -r requirements-dev.txt
@@ -0,0 +1,55 @@
"""feed sort indexes: published_at DESC NULLS LAST composite(s) for the default feed (PERF-1)
The feed's default `newest` sort is `published_at DESC NULLS LAST, id ASC` (feedquery.order_by), but the
only published_at index was a plain ASC btree (`ix_videos_published_at`). A btree read backwards satisfies
`DESC NULLS FIRST`, not the `DESC NULLS LAST` the query asks for, so the planner could not use it for
ordering — every feed page fell back to a full `videos` scan + top-N sort. Measured on a prod-sized reseed
(257k videos): the default page went from ~44.8k shared buffers (dev planner) / ~169k (prod planner, the
per-channel nested loop that is the likely "prod feed is slow" root cause) down to ~624 buffers with the
matching index.
This adds two indexes in the exact feed order:
- ix_videos_published_at_desc (published_at DESC NULLS LAST, id ASC) — the global feed / Library sort.
- ix_videos_channel_pub_desc (channel_id, published_at DESC NULLS LAST, id ASC) — the per-channel feed
(channel page, channel_ids filter). It leads with channel_id, so it is a strict superset of the old
single-column `ix_videos_channel_id` (FK cascade, sync upsert, autotag, channel-detail all still ride
it) — that redundant index is dropped.
The plain ASC `ix_videos_published_at` stays: it already serves the `oldest` sort (ASC NULLS LAST = btree
default) as an Index Scan.
CONCURRENTLY (in alembic's autocommit_block, since it can't run inside a transaction) so a live prod builds
these without blocking reads/writes; downtime ~0. IF (NOT) EXISTS keeps the drift test's
downgrade->re-upgrade round-trip idempotent.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0067_feed_sort_indexes"
down_revision: Union[str, None] = "0066_notif_created_at_notnull"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.get_context().autocommit_block():
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_published_at_desc "
"ON videos (published_at DESC NULLS LAST, id ASC)"
)
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_channel_pub_desc "
"ON videos (channel_id, published_at DESC NULLS LAST, id ASC)"
)
# Redundant now — ix_videos_channel_pub_desc leads with channel_id (a strict superset).
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_channel_id")
def downgrade() -> None:
with op.get_context().autocommit_block():
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_channel_id ON videos (channel_id)"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_channel_pub_desc")
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_published_at_desc")
@@ -0,0 +1,36 @@
"""videos.published_at NOT NULL DEFAULT now() (PERF-15)
Every video in practice carries a publish date (0 NULLs in the 257k-row prod-sized reseed), but the
column was nullable, which forced the feed's keyset predicate into a non-sargable
`... OR published_at IS NULL` disjunction — blocking the ix_videos_published_at_desc range so deep
`newest` pages scanned from the top of the index and filtered (51.8k buffers). Making it NOT NULL lets
the keyset drop that disjunction and take a `published_at <=` range bound (deep page -> ~0.5k buffers).
server_default now() is the safety net for the rare date-less ingest path (the enrich overwrites it with
the real date); the app-side ingest paths also fall back to now() so they never write an explicit NULL.
Note: `SET NOT NULL` validates with a brief ACCESS EXCLUSIVE lock + one table scan (it cannot run
CONCURRENTLY). On the prod row count (~235k) that is sub-second; still, it is a short write-blocking lock
rather than the ~0 of 0067's CONCURRENTLY index build.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0068_published_at_not_null"
down_revision: Union[str, None] = "0067_feed_sort_indexes"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Backfill any stray NULLs first so SET NOT NULL can't fail (0 in prod, but be safe).
op.execute("UPDATE videos SET published_at = now() WHERE published_at IS NULL")
op.alter_column(
"videos", "published_at", nullable=False, server_default=sa.text("now()")
)
def downgrade() -> None:
op.alter_column("videos", "published_at", nullable=True, server_default=None)
@@ -0,0 +1,93 @@
"""scheduler-job partial indexes + boolean-index cleanup (PERF-17/18/3)
Two scheduler jobs full-seq-scanned the 257k-row videos table every run:
- PERF-17: enrich_pending (`WHERE enriched_at IS NULL LIMIT N`, every few minutes) — 44.6k buffers /
run even when nothing is pending. A partial index over just the pending rows makes it a 1-buffer
(empty) index scan.
- PERF-18: maintenance revalidate (`WHERE unavailable_since IS NULL ORDER BY last_checked_at NULLS
FIRST LIMIT N`, ~300 pages/run) — 44.7k buffers/page; ix_videos_last_checked_at had idx_scan=0. A
partial index in that exact order serves it as an index scan (~36 buffers/page) and replaces the
unused plain one.
PERF-3: the videos table also carried plain btrees on low-cardinality boolean columns the planner never
picks (a `col IS false` majority predicate can't ride a btree). Drop the never-useful ones (is_short,
via_explore) and slim the rest to partial indexes over the small/shrinking subset the job queries target
(via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — less write amplification on the
biggest table, same lookups. live_status is left alone (a separate finding).
All CONCURRENTLY (autocommit_block) — downtime ~0; IF (NOT) EXISTS keeps the drift test's
downgrade->re-upgrade idempotent.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0069_job_index_hygiene"
down_revision: Union[str, None] = "0068_published_at_not_null"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.get_context().autocommit_block():
# PERF-17: enrich-pending partial (empty when nothing is pending -> 1-buffer scan).
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_enrich_pending "
"ON videos (id) WHERE enriched_at IS NULL"
)
# PERF-18: maintenance revalidate order, as a partial in the exact WHERE + ORDER BY shape.
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_recheck_order "
"ON videos (last_checked_at ASC NULLS FIRST) WHERE unavailable_since IS NULL"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_last_checked_at")
# PERF-3: drop the never-picked boolean btrees; slim the rest to partials over the target subset.
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_is_short")
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_explore")
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_search")
# Predicate MUST match the query's `.is_(True)`/`.is_(False)` form (SQL `IS TRUE`/`IS FALSE`):
# Postgres's partial-index prover does NOT match a `col IS TRUE` clause to a bare `col` predicate,
# so `WHERE via_search` / `WHERE NOT shorts_probed` would leave the queries seq-scanning.
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_search "
"ON videos (id) WHERE via_search IS TRUE"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shorts_probed")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_shorts_probed "
"ON videos (id) WHERE shorts_probed IS FALSE"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_unavailable_since")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_unavailable_since "
"ON videos (unavailable_since) WHERE unavailable_since IS NOT NULL"
)
def downgrade() -> None:
with op.get_context().autocommit_block():
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_enrich_pending")
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_recheck_order")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_last_checked_at "
"ON videos (last_checked_at)"
)
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_is_short ON videos (is_short)"
)
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_explore ON videos (via_explore)"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_search")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_search ON videos (via_search)"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shorts_probed")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_shorts_probed ON videos (shorts_probed)"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_unavailable_since")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_unavailable_since "
"ON videos (unavailable_since)"
)
+11 -9
View File
@@ -8,7 +8,7 @@ the worker live in `worker.py` / the routes; this module owns enqueue + asset bo
"""
from datetime import datetime, timezone
from sqlalchemy import func, select, text
from sqlalchemy import func, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -234,19 +234,21 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
Runs in the CALLER's session and does NOT commit — the caller commits it together with its asset
writes in one transaction. It flushes first so any pending asset write takes its row lock BEFORE
this takes the job's, matching the cancel route's asset-then-job order (no deadlock). `fields` keys
are internal column names (never user input). Returns True when the row was still running and got
this takes the job's, matching the cancel route's asset-then-job order (no deadlock). `fields` are
mapped-column kwargs (never user input). Returns True when the row was still running and got
updated, False when a route had already moved it out of 'running'."""
if not fields:
return False
db.flush() # asset-then-job lock order: push pending asset writes before we lock the job row
# StrEnum members carry through psycopg as strings, but coerce explicitly so a bound status value
# is always the plain literal the column stores.
params = {k: (v.value if isinstance(v, JobStatus) else v) for k, v in fields.items()}
assignments = ", ".join(f"{col} = :{col}" for col in fields)
# A Core UPDATE with the guard as a WHERE — same atomic "only while still running" write as the
# rest of the codebase's 2.0 statements, no hand-built SQL string. synchronize_session=False: this
# is a terminal write, so we don't pay to sync the (about-to-be-discarded) ORM identity map. Bound
# StrEnum values bind as their plain str literal (StrEnum is a str subclass).
result = db.execute(
text(f"UPDATE download_jobs SET {assignments} WHERE id = :_job_id AND status = :_running"),
{**params, "_job_id": job_id, "_running": JobStatus.RUNNING.value},
update(DownloadJob)
.where(DownloadJob.id == job_id, DownloadJob.status == JobStatus.RUNNING.value)
.values(**fields)
.execution_options(synchronize_session=False)
)
return result.rowcount > 0
+143 -32
View File
@@ -100,6 +100,18 @@ def serialize(row) -> dict:
}
def tags_by_category(db: Session, tag_ids: list[int]) -> dict[str, list[int]]:
"""Group tag ids by their category (language/topic/other). The single source for the category split
shared by the feed's tag WHERE (filtered_query) and the facet/count summary (feed_summary), so the
two can't disagree on which selected tag belongs to which category."""
if not tag_ids:
return {} # skip the pointless `WHERE id IN ()` round-trip on the no-tag-filter hot path
out: dict[str, list[int]] = {}
for tid, cat in db.execute(select(Tag.id, Tag.category).where(Tag.id.in_(tag_ids))).all():
out.setdefault(cat, []).append(tid)
return out
def filtered_query(
db: Session,
user: User,
@@ -131,7 +143,6 @@ def filtered_query(
either mode via the VideoState outer join."""
state = aliased(VideoState)
status_expr = func.coalesce(state.status, "new")
position_expr = func.coalesce(state.position_seconds, 0)
query = select(*feed_columns(user, state)).join(
Channel, Channel.id == Video.channel_id
@@ -163,7 +174,14 @@ def filtered_query(
and_(SearchFind.video_id == Video.id, SearchFind.user_id == user.id),
)
query = query.outerjoin(
# The state-gated views (watched/hidden/in_progress) only ever return videos that HAVE a
# video_states row, so an INNER join lets the planner drive from the user's own small video_states
# set (~100 rows) instead of scanning all videos and outer-joining (PERF-16). status is NOT NULL on
# a present row (server_default 'new'), so the raw-column filters below match the coalesce form
# exactly. 'unwatched'/'all' include row-less 'new' videos, so they keep the outer join.
state_required = show in ("watched", "hidden", "in_progress")
join_state = query.join if state_required else query.outerjoin
query = join_state(
state, and_(state.video_id == Video.id, state.user_id == user.id)
)
@@ -246,15 +264,25 @@ def filtered_query(
# channel searches work. `rank_expr` (weight-aware ts_rank) drives the "relevance" sort.
rank_expr = None
if q:
# Resolve the channels whose NAME matches up front (there are only a few hundred channels, so
# this is a tiny standalone scan) and match videos by channel_id. ORing an unindexed
# `unaccent(channels.title) ILIKE '%q%'` straight into the per-video predicate defeated the
# search_vector GIN — the planner detoasted every non-matching tsvector to test the `@@`
# (~2.6s/search on the prod-sized set); channel_id.in_(...) keeps the GIN as a BitmapOr branch
# (PERF-12). Empty match -> false() so the OR collapses to the text match alone.
like = func.unaccent(f"%{q}%")
matching_channel_ids = (
db.execute(select(Channel.id).where(func.unaccent(Channel.title).ilike(like)))
.scalars()
.all()
)
channel_match = (
Video.channel_id.in_(matching_channel_ids) if matching_channel_ids else false()
)
ts_str = to_tsquery_str(q)
if ts_str:
tsq = func.to_tsquery("public.unaccent_simple", ts_str)
query = query.where(
or_(
Video.search_vector.op("@@")(tsq),
func.unaccent(Channel.title).ilike(func.unaccent(f"%{q}%")),
)
)
query = query.where(or_(Video.search_vector.op("@@")(tsq), channel_match))
# Scale ts_rank (a float4) to an integer score so the keyset cursor compares it
# EXACTLY — a raw float4 vs the float8 cursor value mismatches on round-trip and
# breaks paging (the same page repeats). 1e6 granularity is ample; ties break on
@@ -262,21 +290,14 @@ def filtered_query(
rank_expr = cast(func.ts_rank(Video.search_vector, tsq) * 1000000, BigInteger)
else:
# No usable tokens (e.g. only punctuation): fall back to a plain substring match.
like = func.unaccent(f"%{q}%")
query = query.where(
or_(
func.unaccent(Video.title).ilike(like),
func.unaccent(Channel.title).ilike(like),
)
)
query = query.where(or_(func.unaccent(Video.title).ilike(like), channel_match))
if tags:
# AND across tag categories (e.g. language AND topic narrows), OR within a
# category; the any/all toggle controls multiple topic tags.
cat_rows = db.execute(select(Tag.id, Tag.category).where(Tag.id.in_(tags))).all()
by_category: dict[str, list[int]] = {}
for tag_id, category in cat_rows:
by_category.setdefault(category, []).append(tag_id)
# AND across tag categories (e.g. language AND topic narrows), OR within a category; the any/all
# toggle controls multiple topic tags. ⚠️ feed_summary.passes() reimplements this exact rule in
# Python for the sidebar count/facets — keep the two in lockstep (equivalence-tested by
# test_feed_summary_matches_count_plus_facets).
by_category = tags_by_category(db, tags)
# Facet counting drops the category being counted so its own chips don't zero each
# other out (standard drill-down faceting: a category's count ignores its own
# selections but still honours the other categories' filters).
@@ -315,20 +336,96 @@ def filtered_query(
if show == "unwatched":
query = query.where(status_expr.notin_(("watched", "hidden")))
elif show == "in_progress":
# Started but not finished: a stored resume position, not yet watched/hidden.
# Started but not finished: a stored resume position, not yet watched/hidden. INNER-joined
# (state_required) so the raw state columns are safe and the planner can use video_states.
query = query.where(
and_(position_expr > 0, status_expr.notin_(("watched", "hidden")))
and_(state.position_seconds > 0, state.status.notin_(("watched", "hidden")))
)
elif show == "watched":
query = query.where(status_expr == "watched")
query = query.where(state.status == "watched")
elif show == "hidden":
query = query.where(status_expr == "hidden")
query = query.where(state.status == "hidden")
else: # all
query = query.where(status_expr != "hidden")
return query, rank_expr
def feed_summary(db: Session, user: User, **params) -> dict:
"""Single-pass replacement for /feed/count + 3x /facets (PERF-13): the sidebar's total video count
and every tag's distinct-channel facet count, derived in Python from ONE grouped query instead of
four full passes over the subscribed set on every filter change.
The one query is the base filter with the TAG predicate removed, grouped by channel: it yields every
channel that has a matching video, with its video count. From that ~few-hundred-row map we compute:
- count = sum of the video counts of the channels that pass the SELECTED tag filter;
- facets = per tag, the number of distinct channels that carry it AND pass the drill-down filter
(the tag's own category dropped, unless it's a conjunctive topic-AND — mirroring the
old /facets exclude_tag_category logic exactly).
Tag matching is channel-based, identical to filtered_query's tag block (AND across categories, OR
within, topic-AND requires ALL selected topics)."""
tag_mode = params.get("tag_mode", "or")
# ONE pass: per-channel video counts for the base filter, tags stripped so every candidate channel
# is present to facet over.
base, _rank = filtered_query(db, user, **{**params, "tags": []})
ch_col = base.with_only_columns(Video.channel_id).subquery()
channel_counts: dict[str, int] = dict(
db.execute(select(ch_col.c.channel_id, func.count()).group_by(ch_col.c.channel_id)).all()
)
if not channel_counts:
return {"count": 0, "facets": {}}
# channel -> its visible tag ids, plus each tag's category (for grouping the facet output).
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
channel_tags: dict[str, set[int]] = {}
tag_category: dict[int, str] = {}
for cid, tid, cat in db.execute(
select(ChannelTag.channel_id, ChannelTag.tag_id, Tag.category)
.join(Tag, Tag.id == ChannelTag.tag_id)
.where(visible, ChannelTag.channel_id.in_(channel_counts.keys()))
).all():
channel_tags.setdefault(cid, set()).add(tid)
tag_category[tid] = cat
# Selected tags grouped by category (read from Tag via the shared helper, so a selected tag that no
# base channel carries still filters everything out for its category — matching the SQL `IN (empty)`
# behaviour).
selected_by_cat: dict[str, set[int]] = {
cat: set(ids) for cat, ids in tags_by_category(db, params.get("tags") or []).items()
}
def passes(ch_tags: set[int], exclude_cat: str | None) -> bool:
# The Python mirror of filtered_query's SQL tag block (AND across categories, OR within,
# topic-AND requires all selected) — keep in lockstep; see the ⚠️ note there.
for cat, sel in selected_by_cat.items():
if cat == exclude_cat:
continue
if cat == "topic" and tag_mode == "and" and len(sel) > 1:
if not sel <= ch_tags: # conjunctive: channel must carry ALL selected topics
return False
elif not (sel & ch_tags): # disjunctive: channel must carry ANY selected tag
return False
return True
count = sum(
n for cid, n in channel_counts.items() if passes(channel_tags.get(cid, set()), None)
)
facets: dict[int, int] = {}
for category in ("language", "topic", "other"):
# Drop this category's own selections so its chips keep independent counts; a conjunctive
# topic-AND keeps them (each remaining chip narrows to channels that ALSO have the selected).
exclude = None if (category == "topic" and tag_mode == "and") else category
for cid, ch_tags in channel_tags.items():
if not passes(ch_tags, exclude):
continue
for tid in ch_tags:
if tag_category.get(tid) == category:
facets[tid] = facets.get(tid, 0) + 1
return {"count": count, "facets": facets}
# --- Keyset (cursor) pagination ---------------------------------------------
#
# Each sort is described as an ordered list of key columns (expression + direction
@@ -339,7 +436,7 @@ def filtered_query(
# and the page boundary is stable even as the scheduler ingests new videos mid-scroll
# (OFFSET would shift and duplicate/skip rows).
def sort_keys(sort: str, seed: int, rank_expr=None) -> list[dict]:
pub = {"expr": Video.published_at, "asc": False, "nullable": True, "kind": "dt"}
pub = {"expr": Video.published_at, "asc": False, "nullable": False, "kind": "dt"}
table: dict[str, list[dict]] = {
"newest": [pub],
"oldest": [{**pub, "asc": True}],
@@ -383,7 +480,15 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
Lexicographic over the key columns then id, NULL-aware for `nulls_last`: a NULL
sorts after every non-NULL value, so a cursor sitting on a non-NULL value must
also pull in the NULL group, and equality treats NULL == NULL."""
also pull in the NULL group, and equality treats NULL == NULL.
Built recursively so that for a NON-nullable leading key we can prepend a non-strict
range bound (`col <= v` for desc / `col >= v` for asc) as an AND conjunct — logically
a no-op over the OR of strict-after branches, but it lets the planner bound the index
scan instead of scanning from the top and filtering every earlier row (PERF-15: deep
`newest` pages went 51.8k -> ~0.5k buffers). A NULLABLE key can't take that bound (the
NULL group sorts after every value and `col <= v` would drop it), so it keeps the pure
NULL-aware OR form."""
def eq(k):
return k["expr"].is_(None) if k["value"] is None else k["expr"] == k["value"]
@@ -393,11 +498,17 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
cmp = k["expr"] > k["value"] if k["asc"] else k["expr"] < k["value"]
return or_(cmp, k["expr"].is_(None)) if k["nullable"] else cmp
branches = []
for j in range(len(keys)):
branches.append(and_(*[eq(keys[i]) for i in range(j)], after(keys[j])))
branches.append(and_(*[eq(k) for k in keys], Video.id > cursor_id))
return or_(*branches)
def rec(i: int):
if i == len(keys):
return Video.id > cursor_id # all keys tied -> break by the unique id
k = keys[i]
tail = or_(after(k), and_(eq(k), rec(i + 1)))
if not k["nullable"] and k["value"] is not None:
bound = k["expr"] >= k["value"] if k["asc"] else k["expr"] <= k["value"]
return and_(bound, tail)
return tail
return rec(0)
def encode_cursor(keys: list[dict], row) -> str:
+26
View File
@@ -160,6 +160,32 @@ async def static_cache_headers(request, call_next):
return response
# Baseline security response headers (SEC-1). A minimal, safe set that needs no per-route tuning:
# nosniff stops MIME-confusion, the Referrer-Policy keeps the (private) app URL from leaking to
# external sites the user follows, `frame-ancestors 'none'` blocks click-jacking by refusing to be
# iframed anywhere (we never embed our own pages; we embed OTHERS, which this does not affect), and
# Permissions-Policy turns off powerful features the app never uses. `X-Frame-Options: DENY` duplicates
# the frame-ancestors intent for legacy engines that never implemented CSP frame-ancestors (belt and
# suspenders — modern browsers use the CSP directive and ignore the header). A full Content-Security-Policy
# (script/style/connect sources) is deferred — the SPA's inline styles + the YouTube/Plex embeds make
# it real work — and HSTS belongs on the TLS-terminating edge (the Caddy front door), not here.
_SECURITY_HEADERS = {
"X-Content-Type-Options": "nosniff",
"Referrer-Policy": "strict-origin-when-cross-origin",
"Content-Security-Policy": "frame-ancestors 'none'",
"X-Frame-Options": "DENY",
"Permissions-Policy": "camera=(), microphone=(), geolocation=()",
}
@app.middleware("http")
async def security_headers(request, call_next):
response = await call_next(request)
for name, value in _SECURITY_HEADERS.items():
response.headers.setdefault(name, value)
return response
app.include_router(health.router)
app.include_router(auth_routes.router)
app.include_router(sync.router)
+25 -9
View File
@@ -251,14 +251,23 @@ class Video(Base, TimestampMixin):
__tablename__ = "videos"
id: Mapped[str] = mapped_column(String(16), primary_key=True) # video id
# No standalone channel_id index: the composite `ix_videos_channel_pub_desc`
# (channel_id, published_at DESC NULLS LAST, id) from migration 0067 leads with channel_id, so it
# serves every channel_id lookup (FK cascade, sync upsert, channel-detail) AND the per-channel feed
# sort. Like the search_vector GIN, this composite is migration-defined, not an inline `index=True`.
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
ForeignKey("channels.id", ondelete="CASCADE")
)
title: Mapped[str | None] = mapped_column(Text) # normalized for display (see app.titles)
original_title: Mapped[str | None] = mapped_column(Text) # raw YouTube title, preserved
description: Mapped[str | None] = mapped_column(Text)
published_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), index=True
# NOT NULL (migration 0068): in reality every video carries a publish date, and a nullable key
# forced the feed's keyset predicate into a non-sargable `... OR published_at IS NULL` disjunction
# that blocked the ix_videos_published_at_desc range (deep pages scanned from the top and filtered —
# PERF-15). server_default now() is the safety net for the rare date-less ingest (the enrich fills
# the real date); the ix_videos_published_at index stays for the `oldest` sort.
published_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), index=True, nullable=False, server_default=func.now()
)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
duration_seconds: Mapped[int | None] = mapped_column(Integer)
@@ -276,18 +285,22 @@ class Video(Base, TimestampMixin):
# folds YouTube's own relevance judgement into the local search: a video YouTube returned
# for "ti amo magyarul" becomes findable by that query locally even if its title lacks it.
search_terms: Mapped[str | None] = mapped_column(Text)
# No btree here or on the flag columns below: migration 0069 (PERF-3) dropped the never-picked
# boolean btrees and replaced the useful ones with partial indexes over the target subset
# (via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — migration-defined, like the
# search_vector GIN.
is_short: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
Boolean, default=False, server_default="false"
)
# Whether the youtube.com/shorts/<id> probe has run for this video.
shorts_probed: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
Boolean, default=False, server_default="false"
)
# True when this video first entered the catalog via a live YouTube search (not an
# organic channel/playlist sync). Used to hide search-discovered "noise" from the
# Library by default; left False by the organic ingest paths.
via_search: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
Boolean, default=False, server_default="false"
)
# True when this video entered the catalog because a user EXPLORED its (un-subscribed)
# channel. Like via_search it's hidden from the shared Library by default, but it's
@@ -295,7 +308,7 @@ class Video(Base, TimestampMixin):
# explored or subscribe to its channel (a gate in the feed query), so one user's
# curiosity never floods everyone else's catalog. Left False by the organic ingest paths.
via_explore: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
Boolean, default=False, server_default="false"
)
# none | live | upcoming | premiere | was_live
live_status: Mapped[str] = mapped_column(
@@ -311,11 +324,14 @@ class Video(Base, TimestampMixin):
# (least-recently-checked first); unavailable_since marks a video as currently
# unplayable (hidden from the feed immediately, hard-deleted after a grace period);
# unavailable_reason records why (deleted|private|paywalled|abandoned|stuck_live).
# last_checked_at: no plain btree — migration 0069 (PERF-18) replaced it with the partial
# ix_videos_recheck_order (last_checked_at NULLS FIRST WHERE unavailable_since IS NULL) that the
# revalidate job's exact ORDER BY rides; unavailable_since is a partial WHERE IS NOT NULL (PERF-3).
last_checked_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), index=True
DateTime(timezone=True)
)
unavailable_since: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), index=True
DateTime(timezone=True)
)
unavailable_reason: Mapped[str | None] = mapped_column(String(24))
+12 -7
View File
@@ -294,6 +294,11 @@ def _paginate(plex: PlexClient, key: str, item_type: int):
break
# The section listing exposes at most the top ~20 cast (Plex's own cap); a stored cast longer than this
# can only have come from the rich enrich, never the listing.
_LISTING_CAST_CAP = 20
def _apply_facet_fields(row, meta: dict) -> None:
"""Set the filterable/orderable metadata columns shared by plex_items + plex_shows (identical
column names) from the cheap section listing: rating, content_rating, studio, release date,
@@ -309,14 +314,14 @@ def _apply_facet_fields(row, meta: dict) -> None:
# person-page cast filter for names past the top ~20. The watermark gate below (`people_enriched_at
# is None`) normally protects this, but it is NOT enough on its own: a migration that resets the
# watermark re-opens the regression (0065 nulled it, which is exactly how the S5b fix got undone in
# prod). So guard STRUCTURALLY too, by LENGTH: take the fresh listing UNLESS it is shorter than what
# is stored. A stored cast longer than the ~20-cap listing can only be an enriched fuller list, so
# keep it (robust to any watermark reset); at or below the cap the stored value is itself
# listing-sourced, so replacing it with the current listing stays self-correcting when Plex edits
# the cast (a pure union never would).
# prod). So guard STRUCTURALLY too, by LENGTH against the listing cap: a stored cast LONGER than the
# cap can only be an enriched fuller list, so keep it (robust to any watermark reset). A stored cast
# at or below the cap is itself listing-sourced, so take the fresh listing — which tracks Plex cast
# EDITS in EITHER direction (comparing against this listing's length instead would silently keep a
# stale longer cast when Plex TRIMS one).
if row.people_enriched_at is None:
listing_cast = _tags(meta, "Role", limit=20)
if len(row.cast_names or []) <= len(listing_cast or []):
listing_cast = _tags(meta, "Role", limit=_LISTING_CAST_CAP)
if len(row.cast_names or []) <= _LISTING_CAST_CAP:
row.cast_names = listing_cast
people = list(dict.fromkeys((row.directors or []) + (row.cast_names or [])))
row.people_text = " ".join(people) or None
+5
View File
@@ -24,6 +24,7 @@ from app.models import (
Invite,
User,
)
from app.routes.auth import revoke_preset_password_if_unverified
from app.utils import iso, valid_email
router = APIRouter(prefix="/api/admin", tags=["admin"])
@@ -62,6 +63,10 @@ def _activate_user_by_email(db: Session, email: str) -> None:
it can sign in. No-op if no such user exists yet (a Google user is created at first login)."""
user = db.execute(select(User).where(User.email == email.lower())).scalar_one_or_none()
if user is not None and not user.is_active:
# SEC-10: same guard as the Google-adoption path — approving an account whose email was never
# verified must not hand a working login to a password set on it by whoever registered the
# address (a pre-emption). Shared single-source helper so the two paths can't drift.
revoke_preset_password_if_unverified(user)
user.is_active = True
db.commit()
+21 -2
View File
@@ -286,6 +286,9 @@ async def callback(
return RedirectResponse(url="/?access=denied")
user = existing
user.google_sub = sub
# SEC-10: drop any password pre-set on this account while its email was still unverified,
# now that Google attests ownership (bumps the epoch to revoke any session it minted).
revoke_preset_password_if_unverified(user)
# Google verified the email and is_allowed granted access, so finish activating a
# previously pending password registration (otherwise current_user would bounce it).
user.email_verified = True
@@ -392,8 +395,12 @@ def _complete_link(
Google identity owned by another account, or to silently swap a different linked identity.
`explicit` (the popped oauth_link_explicit marker) distinguishes a "Connect Google" click from
a YouTube upgrade — the caller pops it up front so an aborted flow leaves nothing behind."""
# The marker must match the live session user (guards a stale/forged marker).
if link_uid != request.session.get("user_id"):
# SEC-11: the marker must name an account signed into THIS browser — the session default OR any
# account in the wallet — not only the default. A tab acting as a secondary account links Google to
# ITSELF (the marker was set from current_user, which resolves the ?account= it navigated with), so
# accepting wallet membership stops the identity/token landing on the default account A instead of B.
accounts = request.session.get("account_ids") or []
if link_uid != request.session.get("user_id") and link_uid not in accounts:
return RedirectResponse(url="/?link=error")
user = db.get(User, link_uid)
if user is None or user.is_demo:
@@ -805,6 +812,18 @@ def bump_session_epoch(user: User) -> None:
user.session_epoch = (user.session_epoch or 0) + 1
def revoke_preset_password_if_unverified(user: User) -> None:
"""SEC-10 (account pre-emption): a password set on an account whose email was NEVER verified is
untrusted — whoever registered the address (to pre-empt the real owner) could have chosen it. When
ownership is finally attested (Google adoption, or admin approval), drop that password and bump the
epoch to revoke any session it minted; the owner re-sets one via password-reset. No-op for an
already-verified account, so a legit no-SMTP registration is unaffected. The single source for this
rule — both the Google-callback and admin-activation paths call it so they can't drift."""
if not user.email_verified:
user.password_hash = None
bump_session_epoch(user)
@router.post("/logout-others")
def logout_others(
request: Request, user: User = Depends(current_user), db: Session = Depends(get_db)
+22 -1
View File
@@ -13,6 +13,7 @@ from app.db import get_db
from app.feedquery import (
decode_cursor,
encode_cursor,
feed_summary,
filtered_query,
keyset_predicate,
order_by,
@@ -127,6 +128,9 @@ def get_feed_count(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
# SUPERSEDED by /feed/summary (PERF-13) — the app no longer calls this. Kept as the reference
# oracle that test_feed_summary_matches_count_plus_facets asserts /feed/summary's `count` against;
# remove both together with that test if this endpoint is ever retired.
query, _rank = filtered_query(db, user, **params)
total = db.scalar(select(func.count()).select_from(query.subquery()))
return {"count": total or 0}
@@ -146,7 +150,11 @@ def get_facets(
The count is the number of distinct channels that have at least one video in the current
view, matching the existing channel-count chip semantics. JSON object keys are strings
(tag ids)."""
(tag ids).
SUPERSEDED by /feed/summary (PERF-13) — the app no longer calls this. Kept as the reference
oracle for test_feed_summary_matches_count_plus_facets (it asserts /feed/summary's `facets`
against these `counts`); retire the two together."""
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
counts: dict[int, int] = {}
for category in ("language", "topic", "other"):
@@ -174,6 +182,19 @@ def get_facets(
return {"counts": counts}
@router.get("/feed/summary")
def get_feed_summary(
params: dict = Depends(_feed_params),
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The sidebar's total video count + per-tag channel facet counts in ONE request and ONE grouped
query, replacing the separate /feed/count (1 pass) + /facets (3 passes) the FE fired on every filter
change (PERF-13). Response: {"count": <videos>, "facets": {<tag_id>: <channel_count>}} — the same
numbers /feed/count's `count` and /facets' `counts` return, so the two are interchangeable."""
return feed_summary(db, user, **params)
class VideoStateIn(BaseModel):
status: str
+4 -4
View File
@@ -194,10 +194,10 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
db.execute(
select(Video)
.where(Video.unavailable_since.is_(None))
.order_by(
Video.last_checked_at.is_(None).desc(), # never-checked first
Video.last_checked_at.asc(),
)
# NULLS FIRST = never-checked first, then least-recently-checked; equivalent to the old
# `(last_checked_at IS NULL) DESC, last_checked_at ASC` but in the exact shape of the
# partial ix_videos_recheck_order (migration 0069, PERF-18) so it rides that index.
.order_by(Video.last_checked_at.asc().nulls_first())
.limit(min(50, batch - checked))
)
.scalars()
+3 -1
View File
@@ -90,7 +90,9 @@ def _ensure_videos(db: Session, yt: YouTubeClient, video_ids: list[str]) -> None
if not vid or not cid or vid in seen:
continue
seen.add(vid)
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")))
# published_at is NOT NULL (0068); apply_video_details re-fills a missing date, but seed a now()
# fallback here too so the constructor never carries an explicit NULL.
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")) or now)
apply_video_details(v, it)
v.enriched_at = now
db.add(v)
+6 -2
View File
@@ -90,7 +90,10 @@ def stub_from_playlist_item(item: dict, channel_id: str) -> dict | None:
video_id = content.get("videoId")
if not video_id:
return None
published = parse_dt(content.get("videoPublishedAt") or snippet.get("publishedAt"))
# published_at is NOT NULL (migration 0068); fall back to now() for the rare item that carries no
# date at all, so the bulk stub insert never writes a NULL. The enrich pass replaces it if the
# video's own snippet later yields a real publish date.
published = parse_dt(content.get("videoPublishedAt") or snippet.get("publishedAt")) or _now()
return {
"id": video_id,
"channel_id": channel_id,
@@ -243,7 +246,8 @@ def apply_video_details(video: Video, item: dict) -> None:
video.title = normalize_title(raw_title)
video.description = snippet.get("description")
if not video.published_at:
video.published_at = parse_dt(snippet.get("publishedAt"))
# NOT NULL (0068): a fresh insert with no date yet falls back to now() (enrich normally has one).
video.published_at = parse_dt(snippet.get("publishedAt")) or _now()
video.thumbnail_url = best_thumbnail(snippet.get("thumbnails")) or video.thumbnail_url
video.duration_seconds = parse_iso8601_duration(content.get("duration"))
video.view_count = int(stats["viewCount"]) if stats.get("viewCount") else None
+2 -2
View File
@@ -484,9 +484,9 @@ def _finish_from_cache(job_id: int, asset_id: int) -> None:
asset = db.get(MediaAsset, asset_id)
if job and asset:
fields = {"status": JobStatus.DONE, "progress": 100, "phase": None}
if not job.display_name:
if not job.display_name and asset.title:
# display_name is VARCHAR(255); a title can be longer (e.g. a recipe in the title).
fields["display_name"] = asset.title[:255] if asset.title else None
fields["display_name"] = asset.title[:255]
asset.last_access_at = datetime.now(timezone.utc)
# Atomic finalize: only marks the job done while it is still running, so a concurrent
# cancel/pause is never resurrected (see service.finalize_if_running).
+57
View File
@@ -0,0 +1,57 @@
# Pinned dependency lock for reproducible builds (SEC-5). GENERATED — do not hand-edit.
# Source of truth is requirements.txt (the floor ranges); regenerate after editing it with:
# docker run --rm -v "$PWD":/b python:3.12-slim sh -c 'pip install -q -r /b/requirements.txt && pip freeze --exclude pip --exclude setuptools --exclude wheel' | sort
# The prod Dockerfile + Dockerfile.test + CI install THIS file; pip-audit scans it in CI.
APScheduler==3.11.3
Authlib==1.7.2
Mako==1.4.1
MarkupSafe==3.0.3
PyYAML==6.0.3
SQLAlchemy==2.0.52
alembic==1.19.1
annotated-doc==0.0.5
annotated-types==0.8.0
anyio==4.14.2
argon2-cffi-bindings==25.1.0
argon2-cffi==25.1.0
bgutil-ytdlp-pot-provider==1.3.1
brotli==1.2.0
certifi==2026.7.22
cffi==2.1.1
charset-normalizer==3.5.1
click==8.4.2
cryptography==50.0.0
fastapi==0.141.1
feedparser-sgmllib==2.1.0
feedparser==6.0.14
greenlet==3.5.5
h11==0.16.0
httpcore==1.0.9
httptools==0.8.0
httpx==0.28.1
idna==3.18
itsdangerous==2.2.0
joserfc==1.7.4
mutagen==1.48.1
numpy==2.5.2
psycopg-binary==3.3.4
psycopg==3.3.4
py3langid==0.3.0
pycparser==3.0
pycryptodomex==3.23.0
pydantic-settings==2.15.0
pydantic==2.13.4
pydantic_core==2.46.4
python-dotenv==1.2.3
requests==2.34.2
starlette==1.6.0
typing-inspection==0.4.4
typing_extensions==4.16.0
tzlocal==5.4.4
urllib3==2.7.0
uvicorn==0.52.3
uvloop==0.22.1
watchfiles==1.2.0
websockets==17.0.1
yt-dlp-ejs==0.8.0
yt-dlp==2026.7.4
+1 -1
View File
@@ -11,7 +11,7 @@ feedparser>=6.0,<7.0
apscheduler>=3.10,<4.0
py3langid>=0.3,<1.0
itsdangerous>=2.1,<3.0
cryptography>=46.0.7,<47
cryptography>=50.0.0,<51
argon2-cffi>=23.1,<26
# Download center: yt-dlp does the extraction/download; ffmpeg (OS-level in the Dockerfile)
# does the merge + postprocessing. The [default] extra pulls yt-dlp-ejs (the challenge-solver
+29
View File
@@ -90,3 +90,32 @@ def test_reset_confirm_is_single_use_and_kills_existing_sessions(client, make_us
# The token is single-use: a replay is a clean 400, not a second reset.
replay = client.post("/auth/password-reset/confirm", json={"token": raw, "password": "another pw!!"})
assert replay.status_code == 400
def test_activating_an_unverified_account_drops_its_preset_password(db, make_user):
# SEC-10 (account pre-emption): an attacker registers victim@x with a password before the approved
# owner arrives. The address was never verified, so admin approval must NOT hand that pre-set
# password a working login — it is dropped and any session it minted is revoked.
from app.routes.admin import _activate_user_by_email
victim = make_user(
email="victim@example.com", password="attacker-chosen", email_verified=False, is_active=False
)
epoch0 = victim.session_epoch or 0
_activate_user_by_email(db, "victim@example.com")
assert victim.is_active is True
assert victim.password_hash is None
assert (victim.session_epoch or 0) == epoch0 + 1
def test_activating_a_verified_account_keeps_its_password(db, make_user):
# A legit account whose email IS verified (e.g. a no-SMTP registration) keeps its password on
# approval — the guard must not break the normal flow.
from app.routes.admin import _activate_user_by_email
user = make_user(
email="legit@example.com", password="my own password", email_verified=True, is_active=False
)
_activate_user_by_email(db, "legit@example.com")
assert user.is_active is True
assert user.password_hash is not None
+5 -3
View File
@@ -41,14 +41,16 @@ def _seed(db):
Subscription(user_id=user.id, channel_id="UCa", priority=5, hidden=False),
Subscription(user_id=user.id, channel_id="UCb", priority=0, hidden=False),
])
# 7 normal/organic videos. Deliberate duplicates (published_at day 1 x2 and day 3 x2, view_count
# 100 x2, duration 60 x2 / 30 x2) and NULLs spread across the sort columns.
# 7 normal/organic videos. Deliberate duplicates (published_at day 1 x2, day 2 x2 and day 3 x2,
# view_count 100 x2, duration 60 x2 / 30 x2) and NULLs spread across the still-nullable sort columns
# (view_count / duration_seconds / title / subscriber_count) to exercise nulls_last. published_at is
# NOT NULL (migration 0068), so no NULL there — the duplicates keep the id tiebreaker load-bearing.
rows = [
# id ch published views dur title
("vid00", "UCa", _dt(1), 100, 60, "alpha"),
("vid01", "UCa", _dt(1), 100, 60, "beta"),
("vid02", "UCa", _dt(2), None, None, "gamma"),
("vid03", "UCb", None, 50, 120, None),
("vid03", "UCb", _dt(2), 50, 120, None),
("vid04", "UCb", _dt(3), 200, 30, "delta"),
("vid05", "UCb", _dt(3), None, 30, "epsilon"),
("vid06", "UCa", _dt(28), None, None, "zeta"),
+154
View File
@@ -0,0 +1,154 @@
"""Feed search + watch-state views (E2 S3), DB-lane.
Regression nets for two query rewrites that must stay behaviour-equivalent:
- PERF-12: the search channel-name branch moved from an ORed `unaccent(channels.title) ILIKE` (which
defeated the search_vector GIN) to a pre-resolved `channel_id IN (...)`. A search must STILL return
both videos whose text matches AND videos whose CHANNEL name matches.
- PERF-16: the watched/hidden/in_progress views moved from an outer join + `coalesce(state.status,'new')`
to an inner join on video_states + the raw column. Each view must return exactly the same set.
"""
from datetime import datetime, timezone
import pytest
from app.models import Channel, ChannelTag, Subscription, Tag, Video, VideoState
def _dt(day: int) -> datetime:
return datetime(2024, 1, day, tzinfo=timezone.utc)
def _video(vid: str, ch: str, title: str) -> Video:
return Video(
id=vid, channel_id=ch, published_at=_dt(1), title=title,
is_short=False, live_status="none", via_search=False, via_explore=False,
)
def _feed_ids(client, query: str) -> set[str]:
resp = client.get(f"/api/feed?{query}&limit=50")
assert resp.status_code == 200, resp.text
return {it["id"] for it in resp.json()["items"]}
def test_search_matches_both_video_text_and_channel_name(db, as_user, make_user):
# Two subscribed channels: one whose NAME matches the query (its video's text does not), and one
# whose video TEXT matches (its name does not). The pre-resolved channel_id branch must still OR in
# the name match, so both videos come back — and an unrelated video on a matching-name-less channel
# stays out.
user = make_user(email="search@example.com")
db.add_all([
Channel(id="UCkube", title="Kubernetes Weekly", subscriber_count=10),
Channel(id="UCother", title="Some Other Channel", subscriber_count=10),
])
db.add_all([
Subscription(user_id=user.id, channel_id="UCkube", priority=0, hidden=False),
Subscription(user_id=user.id, channel_id="UCother", priority=0, hidden=False),
])
db.add_all([
_video("vidChan", "UCkube", "Totally unrelated content"), # channel NAME matches
_video("vidText", "UCother", "kubernetes deep dive"), # video TEXT matches
_video("vidNone", "UCother", "cooking pasta at home"), # neither matches
])
db.commit()
ids = _feed_ids(as_user(user), "q=kubernetes&show=all")
assert ids == {"vidChan", "vidText"}
def test_watch_state_views_return_the_right_sets(db, as_user, make_user):
user = make_user(email="views@example.com")
db.add(Channel(id="UCv", title="Channel V", subscriber_count=10))
db.add(Subscription(user_id=user.id, channel_id="UCv", priority=0, hidden=False))
db.add_all([
_video("vWatch", "UCv", "watched one"),
_video("vHide", "UCv", "hidden one"),
_video("vProg", "UCv", "in progress one"),
_video("vNew", "UCv", "brand new one"),
])
db.flush()
db.add_all([
VideoState(user_id=user.id, video_id="vWatch", status="watched"),
VideoState(user_id=user.id, video_id="vHide", status="hidden"),
# in progress = a stored resume position while still "new" (not watched/hidden)
VideoState(user_id=user.id, video_id="vProg", status="new", position_seconds=30),
# vNew has NO state row -> "new"/unwatched
])
db.commit()
client = as_user(user)
assert _feed_ids(client, "show=watched") == {"vWatch"}
assert _feed_ids(client, "show=hidden") == {"vHide"}
assert _feed_ids(client, "show=in_progress") == {"vProg"}
# unwatched excludes watched + hidden, keeps new AND in-progress
assert _feed_ids(client, "show=unwatched") == {"vProg", "vNew"}
# "all" shows everything except hidden
assert _feed_ids(client, "show=all") == {"vWatch", "vProg", "vNew"}
def _seed_tagged(db, make_user):
"""A user subscribed to 4 channels with a spread of language/topic/other tags and differing video
counts, so the count sum and the per-category facet drill-down are all non-trivial."""
user = make_user(email="summary@example.com")
tags = {
"L1": Tag(user_id=None, name="Hungarian", category="language"),
"L2": Tag(user_id=None, name="English", category="language"),
"T1": Tag(user_id=None, name="Tech", category="topic"),
"T2": Tag(user_id=None, name="Gaming", category="topic"),
"O1": Tag(user_id=None, name="Favourite", category="other"),
# A USER-owned tag (+ a user-owned ChannelTag link below) so the visibility branch
# `ChannelTag.user_id == user.id` is exercised, not only the system `user_id IS NULL` one.
"U1": Tag(user_id=user.id, name="MyPin", category="other"),
}
db.add_all(tags.values())
chans = {c: Channel(id=c, title=c, subscriber_count=10) for c in ("chA", "chB", "chC", "chD")}
db.add_all(chans.values())
db.flush()
# channel -> (system) tag keys
links = {"chA": ["L1", "T1"], "chB": ["L1", "T2"], "chC": ["L2", "T1", "O1"], "chD": []}
for cid, tkeys in links.items():
db.add(Subscription(user_id=user.id, channel_id=cid, priority=0, hidden=False))
for tk in tkeys:
db.add(ChannelTag(channel_id=cid, tag_id=tags[tk].id, user_id=None))
# The user's own private link (visible only to them): chB carries U1 for this user.
db.add(ChannelTag(channel_id="chB", tag_id=tags["U1"].id, user_id=user.id))
# differing video counts per channel (chA:3, chB:2, chC:1, chD:2)
counts = {"chA": 3, "chB": 2, "chC": 1, "chD": 2}
for cid, n in counts.items():
db.add_all([_video(f"{cid}v{i}", cid, f"{cid} clip {i}") for i in range(n)])
db.commit()
return user, {k: v.id for k, v in tags.items()}
# Query-param combos (as a template with {L1}/{T1}/… placeholders) that stress every facet path:
# no tags, single-category, cross-category, disjunctive + conjunctive topic modes, and a search filter.
_SUMMARY_CASES = [
"show=all",
"show=all&tags={L1}",
"show=all&tags={T1}",
"show=all&tags={L1}&tags={T1}",
"show=all&tags={T1}&tags={T2}&tag_mode=or",
"show=all&tags={T1}&tags={T2}&tag_mode=and",
"show=all&tags={U1}", # user-owned tag (visibility == user.id branch)
"show=all&tags={U1}&tags={L1}", # user tag AND a system tag, cross-category
"show=all&q=clip",
"show=unwatched",
]
@pytest.mark.parametrize("case", _SUMMARY_CASES)
def test_feed_summary_matches_count_plus_facets(db, as_user, make_user, case):
# /feed/summary must return byte-identical numbers to the old /feed/count + /facets it replaces,
# across the full matrix of tag filters — it reimplements the tag drill-down in Python, so this is
# the equivalence net.
user, tag_ids = _seed_tagged(db, make_user)
client = as_user(user)
qs = case.format(**tag_ids)
summary = client.get(f"/api/feed/summary?{qs}")
count = client.get(f"/api/feed/count?{qs}")
facets = client.get(f"/api/facets?{qs}")
assert summary.status_code == count.status_code == facets.status_code == 200
assert summary.json()["count"] == count.json()["count"], qs
assert summary.json()["facets"] == facets.json()["counts"], qs
+11 -2
View File
@@ -59,5 +59,14 @@ def test_head_migration_downgrades_and_reupgrades(db):
# it lands, not during a rollback. Leaves the schema back at head for the rest of the session.
cfg = Config("alembic.ini")
command.downgrade(cfg, "-1")
command.upgrade(cfg, "head")
assert _structural_drift() == []
reupgraded = False
try:
command.upgrade(cfg, "head")
reupgraded = True
assert _structural_drift() == []
finally:
# If the re-upgrade itself failed, restore the SHARED schema to head so the failure doesn't
# cascade into every later DB-lane test. Skip it when the upgrade already succeeded — re-running
# it would be a redundant no-op and could mask a failing assert with the finally's own error.
if not reupgraded:
command.upgrade(cfg, "head")
+29
View File
@@ -128,6 +128,35 @@ def test_apply_facet_fields_preserves_enriched_cast_names():
assert "Diane Keaton" in enriched.people_text and "Francis Ford Coppola" in enriched.people_text
def test_apply_facet_fields_keeps_over_cap_cast_after_a_watermark_reset():
"""The enriched-cast guard must survive a watermark RESET (people_enriched_at nulled, e.g. by a
migration like 0065). A stored cast LONGER than the listing cap can only be an enriched fuller list,
so it's kept even though the row now reads as un-enriched — comparing against the cap, not the
current listing's length, is what makes this robust (regression guard for the review fix)."""
thin = {"Role": [{"tag": f"Actor {i}"} for i in range(20)]} # a full 20-name listing
over_cap = SimpleNamespace(
people_enriched_at=None, # watermark was reset
directors=None,
cast_names=[f"Enriched {i}" for i in range(22)], # 22 > the 20 cap => enriched
)
_apply_facet_fields(over_cap, thin)
assert over_cap.cast_names == [f"Enriched {i}" for i in range(22)] # kept, not downgraded
def test_apply_facet_fields_tracks_a_trimmed_listing_when_unenriched():
"""A listing-sourced cast (at or below the cap, never enriched) must TRACK Plex cast edits in both
directions. When Plex trims the cast, the shorter fresh listing replaces the stored one — the old
length-vs-this-listing guard would have wrongly kept the stale longer cast."""
stored = SimpleNamespace(
people_enriched_at=None,
directors=None,
cast_names=["A", "B", "C", "D", "E"], # 5, at/below the cap => listing-sourced
)
trimmed = {"Role": [{"tag": "A"}, {"tag": "B"}]} # Plex removed 3 cast members
_apply_facet_fields(stored, trimmed)
assert stored.cast_names == ["A", "B"]
def test_enrich_people_batch_persists_detail(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
+36
View File
@@ -0,0 +1,36 @@
"""Crypto primitives — a guard for the cryptography (Fernet) + argon2 dependency bumps.
Added with the cryptography 46->50 CVE bump: encrypt/decrypt/decrypt_optional had NO test, and they are
the token-at-rest path (Google/API/SMTP/Plex secrets in system_config, OAuth refresh tokens). A fresh
Fernet key is monkeypatched in so the test doesn't depend on TOKEN_ENCRYPTION_KEY being set in the env.
Pure-logic (no DB).
"""
from cryptography.fernet import Fernet
from app import security
def test_fernet_encrypt_decrypt_roundtrip(monkeypatch):
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
assert security.decrypt(security.encrypt("hunter2")) == "hunter2"
assert security.encrypt(None) is None
assert security.decrypt(None) is None
def test_decrypt_optional_degrades_instead_of_raising(monkeypatch):
# Garbage / absent input -> None (never raises).
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
assert security.decrypt_optional("not-a-valid-token") is None
assert security.decrypt_optional(None) is None
# And a ciphertext written under a DIFFERENT key (the TOKEN_ENCRYPTION_KEY-mismatch case that B-core-2
# made non-fatal) also degrades to None rather than raising InvalidToken.
ciphertext = security.encrypt("secret")
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key())) # rotate the key
assert security.decrypt_optional(ciphertext) is None
def test_password_hash_and_verify_roundtrip():
hashed = security.hash_password("correct horse battery")
assert security.verify_password("correct horse battery", hashed) is True
assert security.verify_password("wrong password", hashed) is False
assert security.verify_password("anything", None) is False
+2 -2
View File
@@ -115,7 +115,7 @@ export default function Channels() {
// FB2: syncing subscriptions can change the feed immediately — newly-followed channels may
// already have videos in the shared catalog, so refresh the feed too (not just the manager).
qc.invalidateQueries({ queryKey: qk.feed() });
qc.invalidateQueries({ queryKey: qk.feedCount() });
qc.invalidateQueries({ queryKey: qk.feedSummary() });
};
const userTags = (tagsQuery.data ?? []).filter((t) => !t.system);
@@ -131,7 +131,7 @@ export default function Channels() {
qc.invalidateQueries({ queryKey: qk.channels() });
qc.invalidateQueries({ queryKey: qk.myStatus() });
qc.invalidateQueries({ queryKey: qk.feed() });
qc.invalidateQueries({ queryKey: qk.feedCount() });
qc.invalidateQueries({ queryKey: qk.feedSummary() });
},
});
// Priority is updated optimistically in place and NOT re-sorted/refetched, so the list doesn't
+16 -10
View File
@@ -226,9 +226,11 @@ export default function Feed({
}
}, [query.dataUpdatedAt]);
const countQuery = useQuery({
queryKey: qk.feedCount(queryFilters),
queryFn: () => api.feedCount(queryFilters),
// One shared query with the sidebar (same key + filters -> React Query dedupes to a single request):
// the feed reads `.count`, the sidebar reads `.facets` (PERF-13, replaces feedCount + facets).
const summaryQuery = useQuery({
queryKey: qk.feedSummary(queryFilters),
queryFn: () => api.feedSummary(queryFilters),
staleTime: 30_000,
enabled: !ytActive,
placeholderData: keepPreviousData,
@@ -252,7 +254,11 @@ export default function Feed({
api
.setState(id, status)
.then(() => {
qc.invalidateQueries({ queryKey: qk.feed() });
// Mark stale WITHOUT refetching: the optimistic `overrides` layer + the matchesView filter
// already show the correct state (a hidden/watched card drops out immediately), so refetching
// every loaded infinite-query page here was pure waste — 10 pages x rapid marks = a refetch
// storm (PERF-2). The next filter change / remount refetches fresh from the server.
qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" });
if (status === "hidden") {
const v = loadedRef.current.find((x) => x.id === id);
notify({
@@ -309,7 +315,8 @@ export default function Feed({
setOverrides((o) => ({ ...o, [id]: "new" }));
api
.clearState(id)
.then(() => qc.invalidateQueries({ queryKey: qk.feed() }))
// Stale-mark only (see setState above): the optimistic override already reset the card.
.then(() => qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" }))
.catch(() =>
setOverrides((o) => {
const next = { ...o };
@@ -358,8 +365,7 @@ export default function Feed({
// pre-search cache. Drop them so the feed re-fetches fresh on return.
const dropFeedCaches = () => {
qc.removeQueries({ queryKey: qk.feed() });
qc.removeQueries({ queryKey: qk.feedCount() });
qc.removeQueries({ queryKey: qk.facets() });
qc.removeQueries({ queryKey: qk.feedSummary() });
};
// --- Live YouTube search mode -------------------------------------------------------------
@@ -622,10 +628,10 @@ export default function Feed({
)}
<span className="mx-1 h-5 w-px bg-border" aria-hidden="true" />
<span data-testid="feed-result-count" className="text-xs text-muted whitespace-nowrap">
{countQuery.data
{summaryQuery.data
? t("feed.videoCount", {
count: countQuery.data.count,
formattedCount: countQuery.data.count.toLocaleString(),
count: summaryQuery.data.count,
formattedCount: summaryQuery.data.count.toLocaleString(),
})
: " "}
</span>
@@ -112,7 +112,7 @@ export default function NotificationsPanel() {
.setState(meta.videoId, "new")
.then(() => {
qc.invalidateQueries({ queryKey: qk.feed() });
qc.invalidateQueries({ queryKey: qk.feedCount() });
qc.invalidateQueries({ queryKey: qk.feedSummary() });
// Quietly resolve the now-stale notice (no confirmation toast — the change is visible).
resolveVideo(meta.videoId);
})
+1 -1
View File
@@ -74,7 +74,7 @@ export default function OnboardingWizard({ me, onClose }: { me: Me; onClose: ()
onSuccess: () => {
// Feed/channels/status now have content — refresh the app behind the wizard.
qc.invalidateQueries({ queryKey: qk.feed() });
qc.invalidateQueries({ queryKey: qk.feedCount() });
qc.invalidateQueries({ queryKey: qk.feedSummary() });
qc.invalidateQueries({ queryKey: qk.channels() });
qc.invalidateQueries({ queryKey: qk.myStatus() });
},
+6 -3
View File
@@ -862,10 +862,13 @@ export default function PlayerModal({
cancelled = true;
window.clearInterval(timer);
window.removeEventListener("pagehide", onHide);
// Final flush, then refresh the feed + playlists so resume bars reflect this session.
// Final flush, then mark the feed + playlists stale so resume bars reflect this session on the
// next natural refetch — WITHOUT refetching every loaded feed page on each player close (PERF-2:
// opening/closing the player through a watch session otherwise re-runs the whole infinite query
// each time). Progress is already persisted server-side; only the on-screen resume bar waits.
Promise.resolve(persist()).finally(() => {
qc.invalidateQueries({ queryKey: qk.feed() });
qc.invalidateQueries({ queryKey: qk.playlist() });
qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" });
qc.invalidateQueries({ queryKey: qk.playlist(), refetchType: "none" });
});
const p = playerRef.current;
if (p && typeof p.destroy === "function") {
+4 -3
View File
@@ -1,4 +1,5 @@
import { useState } from "react";
import { withActiveAccount } from "../lib/activeAccount";
import { qk } from "../lib/queryKeys";
import { useTranslation } from "react-i18next";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
@@ -418,7 +419,7 @@ function SignInMethods({ me }: { me: Me }) {
) : (
<button
onClick={() => {
window.location.href = "/auth/link";
window.location.href = withActiveAccount("/auth/link");
}}
className="shrink-0 glass-card glass-hover px-3 py-1.5 rounded-xl text-sm transition"
>
@@ -644,7 +645,7 @@ function Account({ me, onOpenWizard }: { me: Me; onOpenWizard: () => void }) {
grantedHint={t("settings.account.readGrantedHint")}
enableHint={t("settings.account.readEnableHint")}
onEnable={() => {
window.location.href = "/auth/upgrade?access=read";
window.location.href = withActiveAccount("/auth/upgrade?access=read");
}}
/>
<AccessRow
@@ -653,7 +654,7 @@ function Account({ me, onOpenWizard }: { me: Me; onOpenWizard: () => void }) {
grantedHint={t("settings.account.writeGrantedHint")}
enableHint={t("settings.account.writeEnableHint")}
onEnable={() => {
window.location.href = "/auth/upgrade?access=write";
window.location.href = withActiveAccount("/auth/upgrade?access=write");
}}
/>
</div>
+7 -5
View File
@@ -87,14 +87,16 @@ export default function Sidebar({
// Live per-tag channel counts for the current filter context. Debounce the search term so
// typing doesn't refire facets per keystroke, and keep previous counts during a refetch.
const facetFilters: FeedFilters = { ...filters, q: useDebounced(filters.q, 300) };
const facetsQuery = useQuery({
queryKey: qk.facets(facetFilters),
queryFn: () => api.facets(facetFilters),
// Shared with the feed's count (same key + filters -> React Query dedupes to one request): the feed
// reads `.count`, this reads `.facets` (PERF-13, replaces the separate facets query).
const summaryQuery = useQuery({
queryKey: qk.feedSummary(facetFilters),
queryFn: () => api.feedSummary(facetFilters),
staleTime: 30_000,
placeholderData: keepPreviousData,
});
const facetsReady = !!facetsQuery.data;
const facetCounts = facetsQuery.data?.counts ?? {};
const facetsReady = !!summaryQuery.data;
const facetCounts = summaryQuery.data?.facets ?? {};
const chipCount = (tag: Tag): number =>
facetsReady ? (facetCounts[String(tag.id)] ?? 0) : tag.channel_count;
const visibleChips = (list: Tag[]): Tag[] => {
+11
View File
@@ -19,6 +19,17 @@ export function activeAccountId(): number | null {
}
}
// Append `?account=<active>` to a top-level navigation URL (SEC-11). XHRs carry the active account in
// the X-Siftlode-Account header, but a full-page navigation (window.location) can't, so the OAuth
// link/upgrade GETs would resolve to the SESSION DEFAULT account instead of the tab's active one — the
// identity/token would then land on the wrong account. This threads the active account through the URL
// so resolved_user_id picks it. No-op when there's no distinct active account pinned yet.
export function withActiveAccount(path: string): string {
const id = activeAccountId();
if (id == null) return path;
return `${path}${path.includes("?") ? "&" : "?"}account=${id}`;
}
// Reactive active-account. setActiveAccount/clearActiveAccount write sessionStorage — a silent op that
// triggers no React render — so this tab's per-account state (sort/filter/view/layout, all keyed by the
// account) would never re-read once the account is finally known. That bites exactly once, on first
+8
View File
@@ -113,6 +113,8 @@ function feedQuery(f: FeedFilters, cursor: string | null, limit: number): string
export const feedApi = {
feed: (f: FeedFilters, cursor: string | null, limit: number): Promise<FeedResponse> =>
req(`/api/feed?${feedQuery(f, cursor, limit)}`),
// SUPERSEDED by feedSummary (PERF-13) — no component calls this anymore; the backend /feed/count
// endpoint it targets is kept only as the /feed/summary equivalence-test oracle. Retire together.
feedCount: (f: FeedFilters): Promise<{ count: number }> =>
req(`/api/feed/count?${filterParams(f).toString()}`),
// Live YouTube search: results are materialised into the catalog and returned as feed cards.
@@ -129,8 +131,14 @@ export const feedApi = {
// the now-orphaned, un-kept videos). Returns how many videos were removed.
clearSearch: (videoIds: string[]): Promise<{ removed: number }> =>
req("/api/search/clear", { method: "POST", body: JSON.stringify({ video_ids: videoIds }) }),
// SUPERSEDED by feedSummary (PERF-13) — no component calls this anymore; the backend /facets
// endpoint it targets is kept only as the /feed/summary equivalence-test oracle. Retire together.
facets: (f: FeedFilters): Promise<{ counts: Record<string, number> }> =>
req(`/api/facets?${filterParams(f).toString()}`),
// The sidebar's total count + per-tag channel facets in ONE request + ONE grouped query, replacing
// the separate feedCount + facets the feed and sidebar used to fire on every filter change (PERF-13).
feedSummary: (f: FeedFilters): Promise<{ count: number; facets: Record<string, number> }> =>
req(`/api/feed/summary?${filterParams(f).toString()}`),
// idempotent: setting a state / saving a progress checkpoint is safe to replay, so these
// recover from a transient gateway/keepalive 502 instead of surfacing it (see req()).
setState: (id: string, status: string) =>
+2 -1
View File
@@ -12,6 +12,7 @@
// suppresses the auto-popup on future logins (they
// can still reopen it from Settings → Account).
import { withActiveAccount } from "./activeAccount";
import { getAccountRaw, LS, setAccountRaw } from "./storage";
const ONBOARD_ACTIVE = "siftlode.onboarding.active"; // sessionStorage (not in LS, which is localStorage)
@@ -36,7 +37,7 @@ export function shouldAutoOpenOnboarding(me: {
/** Mark a grant as in progress, then send the user to Google's consent screen. */
export function beginGrant(access: "read" | "write"): void {
sessionStorage.setItem(ONBOARD_ACTIVE, "1");
window.location.href = `/auth/upgrade?access=${access}`;
window.location.href = withActiveAccount(`/auth/upgrade?access=${access}`);
}
/** Leave the wizard. `dismiss` suppresses the future auto-popup (used when read is skipped). */
+2
View File
@@ -16,6 +16,8 @@ describe("qk — bare vs parameterized branch", () => {
expect(qk.feedCount(FILTERS)).toEqual(["feed-count", FILTERS]);
expect(qk.facets()).toEqual(["facets"]);
expect(qk.facets(FILTERS)).toEqual(["facets", FILTERS]);
expect(qk.feedSummary()).toEqual(["feed-summary"]);
expect(qk.feedSummary(FILTERS)).toEqual(["feed-summary", FILTERS]);
expect(qk.playlist()).toEqual(["playlist"]);
expect(qk.plexShow()).toEqual(["plex-show"]);
expect(qk.plexShow("s1")).toEqual(["plex-show", "s1"]);
+2
View File
@@ -27,6 +27,8 @@ export const qk = {
filters !== undefined ? (["feed-count", filters] as const) : (["feed-count"] as const),
facets: (filters?: FeedFilters) =>
filters !== undefined ? (["facets", filters] as const) : (["facets"] as const),
feedSummary: (filters?: FeedFilters) =>
filters !== undefined ? (["feed-summary", filters] as const) : (["feed-summary"] as const),
ytSearch: (q?: string | null, count?: number) =>
q !== undefined ? (["yt-search", q, count] as const) : (["yt-search"] as const),
videoDetail: (id: string) => ["video-detail", id] as const,
+21
View File
@@ -0,0 +1,21 @@
import { readFileSync } from "node:fs";
import { describe, expect, it } from "vitest";
import { RELEASE_NOTES } from "./releaseNotes";
// DX-6: the app version is hand-maintained in two places that MUST agree — the repo-root VERSION file
// (the single source baked into the image + served at /api/version) and the newest RELEASE_NOTES entry
// (what the in-app "What's new" shows). A release that bumps one but forgets the other ships a wrong
// changelog or a mismatched banner. This locks them together so the gate catches the drift.
describe("release version consistency", () => {
it("RELEASE_NOTES[0].version matches the VERSION file", () => {
const version = readFileSync(new URL("../../../VERSION", import.meta.url), "utf-8").trim();
expect(RELEASE_NOTES[0]?.version).toBe(version);
});
it("release notes are ordered newest-first with unique versions", () => {
const versions = RELEASE_NOTES.map((r) => r.version);
expect(new Set(versions).size).toBe(versions.length); // no duplicate version entries
});
});
+33
View File
@@ -14,6 +14,39 @@ export interface ReleaseEntry {
}
export const RELEASE_NOTES: ReleaseEntry[] = [
{
version: "0.68.0",
date: "2026-08-18",
summary:
"A big performance pass on the feed and search: pages load and search returns near-instantly, even on a large library, and marking videos no longer stutters.",
fixes: [
"The feed loads much faster. The default “newest” feed and channel pages now use a matching database index instead of scanning the whole video table on every page, so scrolling stays quick even with hundreds of thousands of videos (this was the likely cause of the occasional slow feed).",
"Search is dramatically faster — a search that could take several seconds now returns in a fraction of one, and still matches both video text and channel names.",
"The Watched / Hidden / In-progress views open faster, and deep scrolling (far down a long feed) no longer slows down page by page.",
"Marking a video watched or hidden, and closing the player, no longer briefly reloads the whole feed — the change shows instantly and the list stays put.",
],
chores: [
"The sidebar's total count and per-tag channel counts now come from a single request + one database pass instead of four, cutting the work done on every filter change.",
"The background enrichment and maintenance scans, and the channel search branch, are now index-served instead of full-table scans — much less database load per scheduler run. Backed by new automated equivalence/paging tests and verified end-to-end.",
],
},
{
version: "0.67.0",
date: "2026-08-18",
summary:
"A foundation release: reliability and security hardening under the hood, plus a few fixes you'll actually notice.",
fixes: [
"Canceling or pausing a download can no longer delete a video file that another download is still using — closing a rare race that could remove a file out from under a live download.",
"Switching to an account that was signed out elsewhere (after a password reset or “log out everywhere”) now just asks you to sign back in to that one account, instead of logging you out of every account in the switcher.",
"A failed YouTube search now shows the real reason instead of a generic “connection lost” message that silently retried and spent extra search quota.",
"If the server's encryption key is ever misconfigured, the app now starts and degrades gracefully instead of failing to load at all.",
],
chores: [
"Security hardening around account linking and email verification: a password set on an address before it was verified is no longer trusted once the real owner proves ownership (via Google or admin approval), and linking a Google account now always attaches it to the account you're actually using.",
"A test safety net for the security-critical core — login and session revocation, feed paging, download finalization, and database-migration drift — that is required to actually run: the test lane now fails loudly instead of silently skipping when its database or tools are missing.",
"Reproducible, audited builds: dependencies are pinned to an exact lockfile that is scanned for known vulnerabilities on every build (which caught and cleared 4 cryptography-library CVEs), the runtime moved to Node 22, a server-side CI pipeline runs the full gate on every push, and baseline security response headers were added.",
],
},
{
version: "0.66.0",
date: "2026-08-17",