Compare commits
20
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fa176a3e6a | ||
|
|
e8771f4d6b | ||
|
|
94dbe4b228 | ||
|
|
0ed4ac6958 | ||
|
|
8b8b0ca5ab | ||
|
|
6116ad4bf4 | ||
|
|
943143ef45 | ||
|
|
21a3aae700 | ||
|
|
6a35e83ded | ||
|
|
8c08ea3133 | ||
|
|
3af3799b25 | ||
|
|
aff5a9922c | ||
|
|
cace9a004d | ||
|
|
bfc104a9b4 | ||
|
|
0340d0981c | ||
|
|
18edbecb1e | ||
|
|
8ffa65a9da | ||
|
|
f54dd98129 | ||
|
|
44d4e4876d | ||
|
|
e4ac061721 |
@@ -3,12 +3,12 @@
|
||||
# frontend — tsc (3 projects) + eslint + prettier --check + knip + vitest
|
||||
# backend — ruff + the full pytest suite (pure-logic + the DB-backed lane) against a real Postgres
|
||||
#
|
||||
# Runner: the lab act_runner (LXC 115) maps `ubuntu-latest` -> node:20-bookworm and mounts the Docker
|
||||
# socket, so service containers work. The backend runs pytest NATIVELY against a `postgres` service
|
||||
# (not the local docker-compose.test.yml, whose bind-mount is unreliable under a mounted-socket runner);
|
||||
# fidelity is kept by installing the SAME pinned requirements, Postgres 16, and ffmpeg, and by setting
|
||||
# SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that failed to come up is a hard error, never a silent skip.
|
||||
# node 20 matches the prod build image (Dockerfile `node:20-alpine`); S4's node-22 bump moves both.
|
||||
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
|
||||
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
|
||||
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
|
||||
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
|
||||
# failed to come up is a hard error, never a silent skip. node 22 matches the prod build image
|
||||
# (Dockerfile `node:22-alpine`).
|
||||
name: gate
|
||||
|
||||
on: [push, pull_request]
|
||||
@@ -16,7 +16,7 @@ on: [push, pull_request]
|
||||
jobs:
|
||||
frontend:
|
||||
runs-on: ubuntu-latest
|
||||
container: node:20-bookworm
|
||||
container: node:22-bookworm
|
||||
defaults:
|
||||
run:
|
||||
working-directory: frontend
|
||||
@@ -56,9 +56,14 @@ jobs:
|
||||
- run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client
|
||||
- name: Wait for Postgres
|
||||
run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done
|
||||
- run: pip install --no-cache-dir -r backend/requirements.txt -r backend/requirements-dev.txt
|
||||
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
|
||||
- name: ruff
|
||||
run: python -m ruff check --config backend/ruff.toml backend scripts
|
||||
- name: pytest
|
||||
run: python -m pytest
|
||||
working-directory: backend
|
||||
# Dependency CVE audit over the pinned lock. Non-blocking for now (informational) — a real CVE
|
||||
# shows up in the run without failing the gate; tighten to a hard fail once the noise is known.
|
||||
- name: pip-audit (advisory)
|
||||
run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock
|
||||
continue-on-error: true
|
||||
|
||||
+6
-3
@@ -10,7 +10,7 @@ ARG GIT_SHA=unknown
|
||||
ARG BUILD_DATE=
|
||||
|
||||
# Stage 1: build the frontend SPA
|
||||
FROM node:20-alpine AS frontend
|
||||
FROM node:22-alpine AS frontend
|
||||
WORKDIR /fe
|
||||
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
|
||||
COPY frontend/package.json frontend/package-lock.json ./
|
||||
@@ -57,9 +57,12 @@ RUN apt-get update \
|
||||
&& rm /tmp/deno.zip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/requirements.txt .
|
||||
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
|
||||
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
|
||||
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
|
||||
COPY backend/requirements.lock .
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install --upgrade pip && pip install -r requirements.txt
|
||||
pip install --upgrade pip && pip install -r requirements.lock
|
||||
|
||||
COPY --chown=appuser backend/ .
|
||||
# Read at runtime by config.py as the app_version (a committed single source of truth).
|
||||
|
||||
@@ -14,5 +14,7 @@ WORKDIR /app
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
COPY requirements.txt requirements-dev.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt -r requirements-dev.txt
|
||||
# Install the SAME pinned lock as the prod image (+ the test-only tooling), so the tests run against
|
||||
# the exact dependency versions prod ships.
|
||||
COPY requirements.lock requirements-dev.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.lock -r requirements-dev.txt
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
"""feed sort indexes: published_at DESC NULLS LAST composite(s) for the default feed (PERF-1)
|
||||
|
||||
The feed's default `newest` sort is `published_at DESC NULLS LAST, id ASC` (feedquery.order_by), but the
|
||||
only published_at index was a plain ASC btree (`ix_videos_published_at`). A btree read backwards satisfies
|
||||
`DESC NULLS FIRST`, not the `DESC NULLS LAST` the query asks for, so the planner could not use it for
|
||||
ordering — every feed page fell back to a full `videos` scan + top-N sort. Measured on a prod-sized reseed
|
||||
(257k videos): the default page went from ~44.8k shared buffers (dev planner) / ~169k (prod planner, the
|
||||
per-channel nested loop that is the likely "prod feed is slow" root cause) down to ~624 buffers with the
|
||||
matching index.
|
||||
|
||||
This adds two indexes in the exact feed order:
|
||||
- ix_videos_published_at_desc (published_at DESC NULLS LAST, id ASC) — the global feed / Library sort.
|
||||
- ix_videos_channel_pub_desc (channel_id, published_at DESC NULLS LAST, id ASC) — the per-channel feed
|
||||
(channel page, channel_ids filter). It leads with channel_id, so it is a strict superset of the old
|
||||
single-column `ix_videos_channel_id` (FK cascade, sync upsert, autotag, channel-detail all still ride
|
||||
it) — that redundant index is dropped.
|
||||
|
||||
The plain ASC `ix_videos_published_at` stays: it already serves the `oldest` sort (ASC NULLS LAST = btree
|
||||
default) as an Index Scan.
|
||||
|
||||
CONCURRENTLY (in alembic's autocommit_block, since it can't run inside a transaction) so a live prod builds
|
||||
these without blocking reads/writes; downtime ~0. IF (NOT) EXISTS keeps the drift test's
|
||||
downgrade->re-upgrade round-trip idempotent.
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0067_feed_sort_indexes"
|
||||
down_revision: Union[str, None] = "0066_notif_created_at_notnull"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_published_at_desc "
|
||||
"ON videos (published_at DESC NULLS LAST, id ASC)"
|
||||
)
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_channel_pub_desc "
|
||||
"ON videos (channel_id, published_at DESC NULLS LAST, id ASC)"
|
||||
)
|
||||
# Redundant now — ix_videos_channel_pub_desc leads with channel_id (a strict superset).
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_channel_id")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_channel_id ON videos (channel_id)"
|
||||
)
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_channel_pub_desc")
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_published_at_desc")
|
||||
@@ -0,0 +1,36 @@
|
||||
"""videos.published_at NOT NULL DEFAULT now() (PERF-15)
|
||||
|
||||
Every video in practice carries a publish date (0 NULLs in the 257k-row prod-sized reseed), but the
|
||||
column was nullable, which forced the feed's keyset predicate into a non-sargable
|
||||
`... OR published_at IS NULL` disjunction — blocking the ix_videos_published_at_desc range so deep
|
||||
`newest` pages scanned from the top of the index and filtered (51.8k buffers). Making it NOT NULL lets
|
||||
the keyset drop that disjunction and take a `published_at <=` range bound (deep page -> ~0.5k buffers).
|
||||
|
||||
server_default now() is the safety net for the rare date-less ingest path (the enrich overwrites it with
|
||||
the real date); the app-side ingest paths also fall back to now() so they never write an explicit NULL.
|
||||
|
||||
Note: `SET NOT NULL` validates with a brief ACCESS EXCLUSIVE lock + one table scan (it cannot run
|
||||
CONCURRENTLY). On the prod row count (~235k) that is sub-second; still, it is a short write-blocking lock
|
||||
rather than the ~0 of 0067's CONCURRENTLY index build.
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0068_published_at_not_null"
|
||||
down_revision: Union[str, None] = "0067_feed_sort_indexes"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Backfill any stray NULLs first so SET NOT NULL can't fail (0 in prod, but be safe).
|
||||
op.execute("UPDATE videos SET published_at = now() WHERE published_at IS NULL")
|
||||
op.alter_column(
|
||||
"videos", "published_at", nullable=False, server_default=sa.text("now()")
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.alter_column("videos", "published_at", nullable=True, server_default=None)
|
||||
@@ -0,0 +1,93 @@
|
||||
"""scheduler-job partial indexes + boolean-index cleanup (PERF-17/18/3)
|
||||
|
||||
Two scheduler jobs full-seq-scanned the 257k-row videos table every run:
|
||||
- PERF-17: enrich_pending (`WHERE enriched_at IS NULL LIMIT N`, every few minutes) — 44.6k buffers /
|
||||
run even when nothing is pending. A partial index over just the pending rows makes it a 1-buffer
|
||||
(empty) index scan.
|
||||
- PERF-18: maintenance revalidate (`WHERE unavailable_since IS NULL ORDER BY last_checked_at NULLS
|
||||
FIRST LIMIT N`, ~300 pages/run) — 44.7k buffers/page; ix_videos_last_checked_at had idx_scan=0. A
|
||||
partial index in that exact order serves it as an index scan (~36 buffers/page) and replaces the
|
||||
unused plain one.
|
||||
|
||||
PERF-3: the videos table also carried plain btrees on low-cardinality boolean columns the planner never
|
||||
picks (a `col IS false` majority predicate can't ride a btree). Drop the never-useful ones (is_short,
|
||||
via_explore) and slim the rest to partial indexes over the small/shrinking subset the job queries target
|
||||
(via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — less write amplification on the
|
||||
biggest table, same lookups. live_status is left alone (a separate finding).
|
||||
|
||||
All CONCURRENTLY (autocommit_block) — downtime ~0; IF (NOT) EXISTS keeps the drift test's
|
||||
downgrade->re-upgrade idempotent.
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0069_job_index_hygiene"
|
||||
down_revision: Union[str, None] = "0068_published_at_not_null"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
# PERF-17: enrich-pending partial (empty when nothing is pending -> 1-buffer scan).
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_enrich_pending "
|
||||
"ON videos (id) WHERE enriched_at IS NULL"
|
||||
)
|
||||
# PERF-18: maintenance revalidate order, as a partial in the exact WHERE + ORDER BY shape.
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_recheck_order "
|
||||
"ON videos (last_checked_at ASC NULLS FIRST) WHERE unavailable_since IS NULL"
|
||||
)
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_last_checked_at")
|
||||
# PERF-3: drop the never-picked boolean btrees; slim the rest to partials over the target subset.
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_is_short")
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_explore")
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_search")
|
||||
# Predicate MUST match the query's `.is_(True)`/`.is_(False)` form (SQL `IS TRUE`/`IS FALSE`):
|
||||
# Postgres's partial-index prover does NOT match a `col IS TRUE` clause to a bare `col` predicate,
|
||||
# so `WHERE via_search` / `WHERE NOT shorts_probed` would leave the queries seq-scanning.
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_search "
|
||||
"ON videos (id) WHERE via_search IS TRUE"
|
||||
)
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shorts_probed")
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_shorts_probed "
|
||||
"ON videos (id) WHERE shorts_probed IS FALSE"
|
||||
)
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_unavailable_since")
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_unavailable_since "
|
||||
"ON videos (unavailable_since) WHERE unavailable_since IS NOT NULL"
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_enrich_pending")
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_recheck_order")
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_last_checked_at "
|
||||
"ON videos (last_checked_at)"
|
||||
)
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_is_short ON videos (is_short)"
|
||||
)
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_explore ON videos (via_explore)"
|
||||
)
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_search")
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_search ON videos (via_search)"
|
||||
)
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shorts_probed")
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_shorts_probed ON videos (shorts_probed)"
|
||||
)
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_unavailable_since")
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_unavailable_since "
|
||||
"ON videos (unavailable_since)"
|
||||
)
|
||||
@@ -8,7 +8,7 @@ the worker live in `worker.py` / the routes; this module owns enqueue + asset bo
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy import func, select, update
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -234,19 +234,21 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
|
||||
|
||||
Runs in the CALLER's session and does NOT commit — the caller commits it together with its asset
|
||||
writes in one transaction. It flushes first so any pending asset write takes its row lock BEFORE
|
||||
this takes the job's, matching the cancel route's asset-then-job order (no deadlock). `fields` keys
|
||||
are internal column names (never user input). Returns True when the row was still running and got
|
||||
this takes the job's, matching the cancel route's asset-then-job order (no deadlock). `fields` are
|
||||
mapped-column kwargs (never user input). Returns True when the row was still running and got
|
||||
updated, False when a route had already moved it out of 'running'."""
|
||||
if not fields:
|
||||
return False
|
||||
db.flush() # asset-then-job lock order: push pending asset writes before we lock the job row
|
||||
# StrEnum members carry through psycopg as strings, but coerce explicitly so a bound status value
|
||||
# is always the plain literal the column stores.
|
||||
params = {k: (v.value if isinstance(v, JobStatus) else v) for k, v in fields.items()}
|
||||
assignments = ", ".join(f"{col} = :{col}" for col in fields)
|
||||
# A Core UPDATE with the guard as a WHERE — same atomic "only while still running" write as the
|
||||
# rest of the codebase's 2.0 statements, no hand-built SQL string. synchronize_session=False: this
|
||||
# is a terminal write, so we don't pay to sync the (about-to-be-discarded) ORM identity map. Bound
|
||||
# StrEnum values bind as their plain str literal (StrEnum is a str subclass).
|
||||
result = db.execute(
|
||||
text(f"UPDATE download_jobs SET {assignments} WHERE id = :_job_id AND status = :_running"),
|
||||
{**params, "_job_id": job_id, "_running": JobStatus.RUNNING.value},
|
||||
update(DownloadJob)
|
||||
.where(DownloadJob.id == job_id, DownloadJob.status == JobStatus.RUNNING.value)
|
||||
.values(**fields)
|
||||
.execution_options(synchronize_session=False)
|
||||
)
|
||||
return result.rowcount > 0
|
||||
|
||||
|
||||
+143
-32
@@ -100,6 +100,18 @@ def serialize(row) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def tags_by_category(db: Session, tag_ids: list[int]) -> dict[str, list[int]]:
|
||||
"""Group tag ids by their category (language/topic/other). The single source for the category split
|
||||
shared by the feed's tag WHERE (filtered_query) and the facet/count summary (feed_summary), so the
|
||||
two can't disagree on which selected tag belongs to which category."""
|
||||
if not tag_ids:
|
||||
return {} # skip the pointless `WHERE id IN ()` round-trip on the no-tag-filter hot path
|
||||
out: dict[str, list[int]] = {}
|
||||
for tid, cat in db.execute(select(Tag.id, Tag.category).where(Tag.id.in_(tag_ids))).all():
|
||||
out.setdefault(cat, []).append(tid)
|
||||
return out
|
||||
|
||||
|
||||
def filtered_query(
|
||||
db: Session,
|
||||
user: User,
|
||||
@@ -131,7 +143,6 @@ def filtered_query(
|
||||
either mode via the VideoState outer join."""
|
||||
state = aliased(VideoState)
|
||||
status_expr = func.coalesce(state.status, "new")
|
||||
position_expr = func.coalesce(state.position_seconds, 0)
|
||||
|
||||
query = select(*feed_columns(user, state)).join(
|
||||
Channel, Channel.id == Video.channel_id
|
||||
@@ -163,7 +174,14 @@ def filtered_query(
|
||||
and_(SearchFind.video_id == Video.id, SearchFind.user_id == user.id),
|
||||
)
|
||||
|
||||
query = query.outerjoin(
|
||||
# The state-gated views (watched/hidden/in_progress) only ever return videos that HAVE a
|
||||
# video_states row, so an INNER join lets the planner drive from the user's own small video_states
|
||||
# set (~100 rows) instead of scanning all videos and outer-joining (PERF-16). status is NOT NULL on
|
||||
# a present row (server_default 'new'), so the raw-column filters below match the coalesce form
|
||||
# exactly. 'unwatched'/'all' include row-less 'new' videos, so they keep the outer join.
|
||||
state_required = show in ("watched", "hidden", "in_progress")
|
||||
join_state = query.join if state_required else query.outerjoin
|
||||
query = join_state(
|
||||
state, and_(state.video_id == Video.id, state.user_id == user.id)
|
||||
)
|
||||
|
||||
@@ -246,15 +264,25 @@ def filtered_query(
|
||||
# channel searches work. `rank_expr` (weight-aware ts_rank) drives the "relevance" sort.
|
||||
rank_expr = None
|
||||
if q:
|
||||
# Resolve the channels whose NAME matches up front (there are only a few hundred channels, so
|
||||
# this is a tiny standalone scan) and match videos by channel_id. ORing an unindexed
|
||||
# `unaccent(channels.title) ILIKE '%q%'` straight into the per-video predicate defeated the
|
||||
# search_vector GIN — the planner detoasted every non-matching tsvector to test the `@@`
|
||||
# (~2.6s/search on the prod-sized set); channel_id.in_(...) keeps the GIN as a BitmapOr branch
|
||||
# (PERF-12). Empty match -> false() so the OR collapses to the text match alone.
|
||||
like = func.unaccent(f"%{q}%")
|
||||
matching_channel_ids = (
|
||||
db.execute(select(Channel.id).where(func.unaccent(Channel.title).ilike(like)))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
channel_match = (
|
||||
Video.channel_id.in_(matching_channel_ids) if matching_channel_ids else false()
|
||||
)
|
||||
ts_str = to_tsquery_str(q)
|
||||
if ts_str:
|
||||
tsq = func.to_tsquery("public.unaccent_simple", ts_str)
|
||||
query = query.where(
|
||||
or_(
|
||||
Video.search_vector.op("@@")(tsq),
|
||||
func.unaccent(Channel.title).ilike(func.unaccent(f"%{q}%")),
|
||||
)
|
||||
)
|
||||
query = query.where(or_(Video.search_vector.op("@@")(tsq), channel_match))
|
||||
# Scale ts_rank (a float4) to an integer score so the keyset cursor compares it
|
||||
# EXACTLY — a raw float4 vs the float8 cursor value mismatches on round-trip and
|
||||
# breaks paging (the same page repeats). 1e6 granularity is ample; ties break on
|
||||
@@ -262,21 +290,14 @@ def filtered_query(
|
||||
rank_expr = cast(func.ts_rank(Video.search_vector, tsq) * 1000000, BigInteger)
|
||||
else:
|
||||
# No usable tokens (e.g. only punctuation): fall back to a plain substring match.
|
||||
like = func.unaccent(f"%{q}%")
|
||||
query = query.where(
|
||||
or_(
|
||||
func.unaccent(Video.title).ilike(like),
|
||||
func.unaccent(Channel.title).ilike(like),
|
||||
)
|
||||
)
|
||||
query = query.where(or_(func.unaccent(Video.title).ilike(like), channel_match))
|
||||
|
||||
if tags:
|
||||
# AND across tag categories (e.g. language AND topic narrows), OR within a
|
||||
# category; the any/all toggle controls multiple topic tags.
|
||||
cat_rows = db.execute(select(Tag.id, Tag.category).where(Tag.id.in_(tags))).all()
|
||||
by_category: dict[str, list[int]] = {}
|
||||
for tag_id, category in cat_rows:
|
||||
by_category.setdefault(category, []).append(tag_id)
|
||||
# AND across tag categories (e.g. language AND topic narrows), OR within a category; the any/all
|
||||
# toggle controls multiple topic tags. ⚠️ feed_summary.passes() reimplements this exact rule in
|
||||
# Python for the sidebar count/facets — keep the two in lockstep (equivalence-tested by
|
||||
# test_feed_summary_matches_count_plus_facets).
|
||||
by_category = tags_by_category(db, tags)
|
||||
# Facet counting drops the category being counted so its own chips don't zero each
|
||||
# other out (standard drill-down faceting: a category's count ignores its own
|
||||
# selections but still honours the other categories' filters).
|
||||
@@ -315,20 +336,96 @@ def filtered_query(
|
||||
if show == "unwatched":
|
||||
query = query.where(status_expr.notin_(("watched", "hidden")))
|
||||
elif show == "in_progress":
|
||||
# Started but not finished: a stored resume position, not yet watched/hidden.
|
||||
# Started but not finished: a stored resume position, not yet watched/hidden. INNER-joined
|
||||
# (state_required) so the raw state columns are safe and the planner can use video_states.
|
||||
query = query.where(
|
||||
and_(position_expr > 0, status_expr.notin_(("watched", "hidden")))
|
||||
and_(state.position_seconds > 0, state.status.notin_(("watched", "hidden")))
|
||||
)
|
||||
elif show == "watched":
|
||||
query = query.where(status_expr == "watched")
|
||||
query = query.where(state.status == "watched")
|
||||
elif show == "hidden":
|
||||
query = query.where(status_expr == "hidden")
|
||||
query = query.where(state.status == "hidden")
|
||||
else: # all
|
||||
query = query.where(status_expr != "hidden")
|
||||
|
||||
return query, rank_expr
|
||||
|
||||
|
||||
def feed_summary(db: Session, user: User, **params) -> dict:
|
||||
"""Single-pass replacement for /feed/count + 3x /facets (PERF-13): the sidebar's total video count
|
||||
and every tag's distinct-channel facet count, derived in Python from ONE grouped query instead of
|
||||
four full passes over the subscribed set on every filter change.
|
||||
|
||||
The one query is the base filter with the TAG predicate removed, grouped by channel: it yields every
|
||||
channel that has a matching video, with its video count. From that ~few-hundred-row map we compute:
|
||||
- count = sum of the video counts of the channels that pass the SELECTED tag filter;
|
||||
- facets = per tag, the number of distinct channels that carry it AND pass the drill-down filter
|
||||
(the tag's own category dropped, unless it's a conjunctive topic-AND — mirroring the
|
||||
old /facets exclude_tag_category logic exactly).
|
||||
Tag matching is channel-based, identical to filtered_query's tag block (AND across categories, OR
|
||||
within, topic-AND requires ALL selected topics)."""
|
||||
tag_mode = params.get("tag_mode", "or")
|
||||
|
||||
# ONE pass: per-channel video counts for the base filter, tags stripped so every candidate channel
|
||||
# is present to facet over.
|
||||
base, _rank = filtered_query(db, user, **{**params, "tags": []})
|
||||
ch_col = base.with_only_columns(Video.channel_id).subquery()
|
||||
channel_counts: dict[str, int] = dict(
|
||||
db.execute(select(ch_col.c.channel_id, func.count()).group_by(ch_col.c.channel_id)).all()
|
||||
)
|
||||
if not channel_counts:
|
||||
return {"count": 0, "facets": {}}
|
||||
|
||||
# channel -> its visible tag ids, plus each tag's category (for grouping the facet output).
|
||||
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
|
||||
channel_tags: dict[str, set[int]] = {}
|
||||
tag_category: dict[int, str] = {}
|
||||
for cid, tid, cat in db.execute(
|
||||
select(ChannelTag.channel_id, ChannelTag.tag_id, Tag.category)
|
||||
.join(Tag, Tag.id == ChannelTag.tag_id)
|
||||
.where(visible, ChannelTag.channel_id.in_(channel_counts.keys()))
|
||||
).all():
|
||||
channel_tags.setdefault(cid, set()).add(tid)
|
||||
tag_category[tid] = cat
|
||||
|
||||
# Selected tags grouped by category (read from Tag via the shared helper, so a selected tag that no
|
||||
# base channel carries still filters everything out for its category — matching the SQL `IN (empty)`
|
||||
# behaviour).
|
||||
selected_by_cat: dict[str, set[int]] = {
|
||||
cat: set(ids) for cat, ids in tags_by_category(db, params.get("tags") or []).items()
|
||||
}
|
||||
|
||||
def passes(ch_tags: set[int], exclude_cat: str | None) -> bool:
|
||||
# The Python mirror of filtered_query's SQL tag block (AND across categories, OR within,
|
||||
# topic-AND requires all selected) — keep in lockstep; see the ⚠️ note there.
|
||||
for cat, sel in selected_by_cat.items():
|
||||
if cat == exclude_cat:
|
||||
continue
|
||||
if cat == "topic" and tag_mode == "and" and len(sel) > 1:
|
||||
if not sel <= ch_tags: # conjunctive: channel must carry ALL selected topics
|
||||
return False
|
||||
elif not (sel & ch_tags): # disjunctive: channel must carry ANY selected tag
|
||||
return False
|
||||
return True
|
||||
|
||||
count = sum(
|
||||
n for cid, n in channel_counts.items() if passes(channel_tags.get(cid, set()), None)
|
||||
)
|
||||
|
||||
facets: dict[int, int] = {}
|
||||
for category in ("language", "topic", "other"):
|
||||
# Drop this category's own selections so its chips keep independent counts; a conjunctive
|
||||
# topic-AND keeps them (each remaining chip narrows to channels that ALSO have the selected).
|
||||
exclude = None if (category == "topic" and tag_mode == "and") else category
|
||||
for cid, ch_tags in channel_tags.items():
|
||||
if not passes(ch_tags, exclude):
|
||||
continue
|
||||
for tid in ch_tags:
|
||||
if tag_category.get(tid) == category:
|
||||
facets[tid] = facets.get(tid, 0) + 1
|
||||
return {"count": count, "facets": facets}
|
||||
|
||||
|
||||
# --- Keyset (cursor) pagination ---------------------------------------------
|
||||
#
|
||||
# Each sort is described as an ordered list of key columns (expression + direction
|
||||
@@ -339,7 +436,7 @@ def filtered_query(
|
||||
# and the page boundary is stable even as the scheduler ingests new videos mid-scroll
|
||||
# (OFFSET would shift and duplicate/skip rows).
|
||||
def sort_keys(sort: str, seed: int, rank_expr=None) -> list[dict]:
|
||||
pub = {"expr": Video.published_at, "asc": False, "nullable": True, "kind": "dt"}
|
||||
pub = {"expr": Video.published_at, "asc": False, "nullable": False, "kind": "dt"}
|
||||
table: dict[str, list[dict]] = {
|
||||
"newest": [pub],
|
||||
"oldest": [{**pub, "asc": True}],
|
||||
@@ -383,7 +480,15 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
|
||||
|
||||
Lexicographic over the key columns then id, NULL-aware for `nulls_last`: a NULL
|
||||
sorts after every non-NULL value, so a cursor sitting on a non-NULL value must
|
||||
also pull in the NULL group, and equality treats NULL == NULL."""
|
||||
also pull in the NULL group, and equality treats NULL == NULL.
|
||||
|
||||
Built recursively so that for a NON-nullable leading key we can prepend a non-strict
|
||||
range bound (`col <= v` for desc / `col >= v` for asc) as an AND conjunct — logically
|
||||
a no-op over the OR of strict-after branches, but it lets the planner bound the index
|
||||
scan instead of scanning from the top and filtering every earlier row (PERF-15: deep
|
||||
`newest` pages went 51.8k -> ~0.5k buffers). A NULLABLE key can't take that bound (the
|
||||
NULL group sorts after every value and `col <= v` would drop it), so it keeps the pure
|
||||
NULL-aware OR form."""
|
||||
def eq(k):
|
||||
return k["expr"].is_(None) if k["value"] is None else k["expr"] == k["value"]
|
||||
|
||||
@@ -393,11 +498,17 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
|
||||
cmp = k["expr"] > k["value"] if k["asc"] else k["expr"] < k["value"]
|
||||
return or_(cmp, k["expr"].is_(None)) if k["nullable"] else cmp
|
||||
|
||||
branches = []
|
||||
for j in range(len(keys)):
|
||||
branches.append(and_(*[eq(keys[i]) for i in range(j)], after(keys[j])))
|
||||
branches.append(and_(*[eq(k) for k in keys], Video.id > cursor_id))
|
||||
return or_(*branches)
|
||||
def rec(i: int):
|
||||
if i == len(keys):
|
||||
return Video.id > cursor_id # all keys tied -> break by the unique id
|
||||
k = keys[i]
|
||||
tail = or_(after(k), and_(eq(k), rec(i + 1)))
|
||||
if not k["nullable"] and k["value"] is not None:
|
||||
bound = k["expr"] >= k["value"] if k["asc"] else k["expr"] <= k["value"]
|
||||
return and_(bound, tail)
|
||||
return tail
|
||||
|
||||
return rec(0)
|
||||
|
||||
|
||||
def encode_cursor(keys: list[dict], row) -> str:
|
||||
|
||||
@@ -160,6 +160,32 @@ async def static_cache_headers(request, call_next):
|
||||
return response
|
||||
|
||||
|
||||
# Baseline security response headers (SEC-1). A minimal, safe set that needs no per-route tuning:
|
||||
# nosniff stops MIME-confusion, the Referrer-Policy keeps the (private) app URL from leaking to
|
||||
# external sites the user follows, `frame-ancestors 'none'` blocks click-jacking by refusing to be
|
||||
# iframed anywhere (we never embed our own pages; we embed OTHERS, which this does not affect), and
|
||||
# Permissions-Policy turns off powerful features the app never uses. `X-Frame-Options: DENY` duplicates
|
||||
# the frame-ancestors intent for legacy engines that never implemented CSP frame-ancestors (belt and
|
||||
# suspenders — modern browsers use the CSP directive and ignore the header). A full Content-Security-Policy
|
||||
# (script/style/connect sources) is deferred — the SPA's inline styles + the YouTube/Plex embeds make
|
||||
# it real work — and HSTS belongs on the TLS-terminating edge (the Caddy front door), not here.
|
||||
_SECURITY_HEADERS = {
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
"Referrer-Policy": "strict-origin-when-cross-origin",
|
||||
"Content-Security-Policy": "frame-ancestors 'none'",
|
||||
"X-Frame-Options": "DENY",
|
||||
"Permissions-Policy": "camera=(), microphone=(), geolocation=()",
|
||||
}
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def security_headers(request, call_next):
|
||||
response = await call_next(request)
|
||||
for name, value in _SECURITY_HEADERS.items():
|
||||
response.headers.setdefault(name, value)
|
||||
return response
|
||||
|
||||
|
||||
app.include_router(health.router)
|
||||
app.include_router(auth_routes.router)
|
||||
app.include_router(sync.router)
|
||||
|
||||
+25
-9
@@ -251,14 +251,23 @@ class Video(Base, TimestampMixin):
|
||||
__tablename__ = "videos"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(16), primary_key=True) # video id
|
||||
# No standalone channel_id index: the composite `ix_videos_channel_pub_desc`
|
||||
# (channel_id, published_at DESC NULLS LAST, id) from migration 0067 leads with channel_id, so it
|
||||
# serves every channel_id lookup (FK cascade, sync upsert, channel-detail) AND the per-channel feed
|
||||
# sort. Like the search_vector GIN, this composite is migration-defined, not an inline `index=True`.
|
||||
channel_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("channels.id", ondelete="CASCADE"), index=True
|
||||
ForeignKey("channels.id", ondelete="CASCADE")
|
||||
)
|
||||
title: Mapped[str | None] = mapped_column(Text) # normalized for display (see app.titles)
|
||||
original_title: Mapped[str | None] = mapped_column(Text) # raw YouTube title, preserved
|
||||
description: Mapped[str | None] = mapped_column(Text)
|
||||
published_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), index=True
|
||||
# NOT NULL (migration 0068): in reality every video carries a publish date, and a nullable key
|
||||
# forced the feed's keyset predicate into a non-sargable `... OR published_at IS NULL` disjunction
|
||||
# that blocked the ix_videos_published_at_desc range (deep pages scanned from the top and filtered —
|
||||
# PERF-15). server_default now() is the safety net for the rare date-less ingest (the enrich fills
|
||||
# the real date); the ix_videos_published_at index stays for the `oldest` sort.
|
||||
published_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), index=True, nullable=False, server_default=func.now()
|
||||
)
|
||||
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
|
||||
duration_seconds: Mapped[int | None] = mapped_column(Integer)
|
||||
@@ -276,18 +285,22 @@ class Video(Base, TimestampMixin):
|
||||
# folds YouTube's own relevance judgement into the local search: a video YouTube returned
|
||||
# for "ti amo magyarul" becomes findable by that query locally even if its title lacks it.
|
||||
search_terms: Mapped[str | None] = mapped_column(Text)
|
||||
# No btree here or on the flag columns below: migration 0069 (PERF-3) dropped the never-picked
|
||||
# boolean btrees and replaced the useful ones with partial indexes over the target subset
|
||||
# (via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — migration-defined, like the
|
||||
# search_vector GIN.
|
||||
is_short: Mapped[bool] = mapped_column(
|
||||
Boolean, default=False, server_default="false", index=True
|
||||
Boolean, default=False, server_default="false"
|
||||
)
|
||||
# Whether the youtube.com/shorts/<id> probe has run for this video.
|
||||
shorts_probed: Mapped[bool] = mapped_column(
|
||||
Boolean, default=False, server_default="false", index=True
|
||||
Boolean, default=False, server_default="false"
|
||||
)
|
||||
# True when this video first entered the catalog via a live YouTube search (not an
|
||||
# organic channel/playlist sync). Used to hide search-discovered "noise" from the
|
||||
# Library by default; left False by the organic ingest paths.
|
||||
via_search: Mapped[bool] = mapped_column(
|
||||
Boolean, default=False, server_default="false", index=True
|
||||
Boolean, default=False, server_default="false"
|
||||
)
|
||||
# True when this video entered the catalog because a user EXPLORED its (un-subscribed)
|
||||
# channel. Like via_search it's hidden from the shared Library by default, but it's
|
||||
@@ -295,7 +308,7 @@ class Video(Base, TimestampMixin):
|
||||
# explored or subscribe to its channel (a gate in the feed query), so one user's
|
||||
# curiosity never floods everyone else's catalog. Left False by the organic ingest paths.
|
||||
via_explore: Mapped[bool] = mapped_column(
|
||||
Boolean, default=False, server_default="false", index=True
|
||||
Boolean, default=False, server_default="false"
|
||||
)
|
||||
# none | live | upcoming | premiere | was_live
|
||||
live_status: Mapped[str] = mapped_column(
|
||||
@@ -311,11 +324,14 @@ class Video(Base, TimestampMixin):
|
||||
# (least-recently-checked first); unavailable_since marks a video as currently
|
||||
# unplayable (hidden from the feed immediately, hard-deleted after a grace period);
|
||||
# unavailable_reason records why (deleted|private|paywalled|abandoned|stuck_live).
|
||||
# last_checked_at: no plain btree — migration 0069 (PERF-18) replaced it with the partial
|
||||
# ix_videos_recheck_order (last_checked_at NULLS FIRST WHERE unavailable_since IS NULL) that the
|
||||
# revalidate job's exact ORDER BY rides; unavailable_since is a partial WHERE IS NOT NULL (PERF-3).
|
||||
last_checked_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), index=True
|
||||
DateTime(timezone=True)
|
||||
)
|
||||
unavailable_since: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), index=True
|
||||
DateTime(timezone=True)
|
||||
)
|
||||
unavailable_reason: Mapped[str | None] = mapped_column(String(24))
|
||||
|
||||
|
||||
@@ -294,6 +294,11 @@ def _paginate(plex: PlexClient, key: str, item_type: int):
|
||||
break
|
||||
|
||||
|
||||
# The section listing exposes at most the top ~20 cast (Plex's own cap); a stored cast longer than this
|
||||
# can only have come from the rich enrich, never the listing.
|
||||
_LISTING_CAST_CAP = 20
|
||||
|
||||
|
||||
def _apply_facet_fields(row, meta: dict) -> None:
|
||||
"""Set the filterable/orderable metadata columns shared by plex_items + plex_shows (identical
|
||||
column names) from the cheap section listing: rating, content_rating, studio, release date,
|
||||
@@ -309,14 +314,14 @@ def _apply_facet_fields(row, meta: dict) -> None:
|
||||
# person-page cast filter for names past the top ~20. The watermark gate below (`people_enriched_at
|
||||
# is None`) normally protects this, but it is NOT enough on its own: a migration that resets the
|
||||
# watermark re-opens the regression (0065 nulled it, which is exactly how the S5b fix got undone in
|
||||
# prod). So guard STRUCTURALLY too, by LENGTH: take the fresh listing UNLESS it is shorter than what
|
||||
# is stored. A stored cast longer than the ~20-cap listing can only be an enriched fuller list, so
|
||||
# keep it (robust to any watermark reset); at or below the cap the stored value is itself
|
||||
# listing-sourced, so replacing it with the current listing stays self-correcting when Plex edits
|
||||
# the cast (a pure union never would).
|
||||
# prod). So guard STRUCTURALLY too, by LENGTH against the listing cap: a stored cast LONGER than the
|
||||
# cap can only be an enriched fuller list, so keep it (robust to any watermark reset). A stored cast
|
||||
# at or below the cap is itself listing-sourced, so take the fresh listing — which tracks Plex cast
|
||||
# EDITS in EITHER direction (comparing against this listing's length instead would silently keep a
|
||||
# stale longer cast when Plex TRIMS one).
|
||||
if row.people_enriched_at is None:
|
||||
listing_cast = _tags(meta, "Role", limit=20)
|
||||
if len(row.cast_names or []) <= len(listing_cast or []):
|
||||
listing_cast = _tags(meta, "Role", limit=_LISTING_CAST_CAP)
|
||||
if len(row.cast_names or []) <= _LISTING_CAST_CAP:
|
||||
row.cast_names = listing_cast
|
||||
people = list(dict.fromkeys((row.directors or []) + (row.cast_names or [])))
|
||||
row.people_text = " ".join(people) or None
|
||||
|
||||
@@ -24,6 +24,7 @@ from app.models import (
|
||||
Invite,
|
||||
User,
|
||||
)
|
||||
from app.routes.auth import revoke_preset_password_if_unverified
|
||||
from app.utils import iso, valid_email
|
||||
|
||||
router = APIRouter(prefix="/api/admin", tags=["admin"])
|
||||
@@ -62,6 +63,10 @@ def _activate_user_by_email(db: Session, email: str) -> None:
|
||||
it can sign in. No-op if no such user exists yet (a Google user is created at first login)."""
|
||||
user = db.execute(select(User).where(User.email == email.lower())).scalar_one_or_none()
|
||||
if user is not None and not user.is_active:
|
||||
# SEC-10: same guard as the Google-adoption path — approving an account whose email was never
|
||||
# verified must not hand a working login to a password set on it by whoever registered the
|
||||
# address (a pre-emption). Shared single-source helper so the two paths can't drift.
|
||||
revoke_preset_password_if_unverified(user)
|
||||
user.is_active = True
|
||||
db.commit()
|
||||
|
||||
|
||||
@@ -286,6 +286,9 @@ async def callback(
|
||||
return RedirectResponse(url="/?access=denied")
|
||||
user = existing
|
||||
user.google_sub = sub
|
||||
# SEC-10: drop any password pre-set on this account while its email was still unverified,
|
||||
# now that Google attests ownership (bumps the epoch to revoke any session it minted).
|
||||
revoke_preset_password_if_unverified(user)
|
||||
# Google verified the email and is_allowed granted access, so finish activating a
|
||||
# previously pending password registration (otherwise current_user would bounce it).
|
||||
user.email_verified = True
|
||||
@@ -392,8 +395,12 @@ def _complete_link(
|
||||
Google identity owned by another account, or to silently swap a different linked identity.
|
||||
`explicit` (the popped oauth_link_explicit marker) distinguishes a "Connect Google" click from
|
||||
a YouTube upgrade — the caller pops it up front so an aborted flow leaves nothing behind."""
|
||||
# The marker must match the live session user (guards a stale/forged marker).
|
||||
if link_uid != request.session.get("user_id"):
|
||||
# SEC-11: the marker must name an account signed into THIS browser — the session default OR any
|
||||
# account in the wallet — not only the default. A tab acting as a secondary account links Google to
|
||||
# ITSELF (the marker was set from current_user, which resolves the ?account= it navigated with), so
|
||||
# accepting wallet membership stops the identity/token landing on the default account A instead of B.
|
||||
accounts = request.session.get("account_ids") or []
|
||||
if link_uid != request.session.get("user_id") and link_uid not in accounts:
|
||||
return RedirectResponse(url="/?link=error")
|
||||
user = db.get(User, link_uid)
|
||||
if user is None or user.is_demo:
|
||||
@@ -805,6 +812,18 @@ def bump_session_epoch(user: User) -> None:
|
||||
user.session_epoch = (user.session_epoch or 0) + 1
|
||||
|
||||
|
||||
def revoke_preset_password_if_unverified(user: User) -> None:
|
||||
"""SEC-10 (account pre-emption): a password set on an account whose email was NEVER verified is
|
||||
untrusted — whoever registered the address (to pre-empt the real owner) could have chosen it. When
|
||||
ownership is finally attested (Google adoption, or admin approval), drop that password and bump the
|
||||
epoch to revoke any session it minted; the owner re-sets one via password-reset. No-op for an
|
||||
already-verified account, so a legit no-SMTP registration is unaffected. The single source for this
|
||||
rule — both the Google-callback and admin-activation paths call it so they can't drift."""
|
||||
if not user.email_verified:
|
||||
user.password_hash = None
|
||||
bump_session_epoch(user)
|
||||
|
||||
|
||||
@router.post("/logout-others")
|
||||
def logout_others(
|
||||
request: Request, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
|
||||
@@ -13,6 +13,7 @@ from app.db import get_db
|
||||
from app.feedquery import (
|
||||
decode_cursor,
|
||||
encode_cursor,
|
||||
feed_summary,
|
||||
filtered_query,
|
||||
keyset_predicate,
|
||||
order_by,
|
||||
@@ -127,6 +128,9 @@ def get_feed_count(
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
# SUPERSEDED by /feed/summary (PERF-13) — the app no longer calls this. Kept as the reference
|
||||
# oracle that test_feed_summary_matches_count_plus_facets asserts /feed/summary's `count` against;
|
||||
# remove both together with that test if this endpoint is ever retired.
|
||||
query, _rank = filtered_query(db, user, **params)
|
||||
total = db.scalar(select(func.count()).select_from(query.subquery()))
|
||||
return {"count": total or 0}
|
||||
@@ -146,7 +150,11 @@ def get_facets(
|
||||
|
||||
The count is the number of distinct channels that have at least one video in the current
|
||||
view, matching the existing channel-count chip semantics. JSON object keys are strings
|
||||
(tag ids)."""
|
||||
(tag ids).
|
||||
|
||||
SUPERSEDED by /feed/summary (PERF-13) — the app no longer calls this. Kept as the reference
|
||||
oracle for test_feed_summary_matches_count_plus_facets (it asserts /feed/summary's `facets`
|
||||
against these `counts`); retire the two together."""
|
||||
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
|
||||
counts: dict[int, int] = {}
|
||||
for category in ("language", "topic", "other"):
|
||||
@@ -174,6 +182,19 @@ def get_facets(
|
||||
return {"counts": counts}
|
||||
|
||||
|
||||
@router.get("/feed/summary")
|
||||
def get_feed_summary(
|
||||
params: dict = Depends(_feed_params),
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""The sidebar's total video count + per-tag channel facet counts in ONE request and ONE grouped
|
||||
query, replacing the separate /feed/count (1 pass) + /facets (3 passes) the FE fired on every filter
|
||||
change (PERF-13). Response: {"count": <videos>, "facets": {<tag_id>: <channel_count>}} — the same
|
||||
numbers /feed/count's `count` and /facets' `counts` return, so the two are interchangeable."""
|
||||
return feed_summary(db, user, **params)
|
||||
|
||||
|
||||
class VideoStateIn(BaseModel):
|
||||
status: str
|
||||
|
||||
|
||||
@@ -194,10 +194,10 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
|
||||
db.execute(
|
||||
select(Video)
|
||||
.where(Video.unavailable_since.is_(None))
|
||||
.order_by(
|
||||
Video.last_checked_at.is_(None).desc(), # never-checked first
|
||||
Video.last_checked_at.asc(),
|
||||
)
|
||||
# NULLS FIRST = never-checked first, then least-recently-checked; equivalent to the old
|
||||
# `(last_checked_at IS NULL) DESC, last_checked_at ASC` but in the exact shape of the
|
||||
# partial ix_videos_recheck_order (migration 0069, PERF-18) so it rides that index.
|
||||
.order_by(Video.last_checked_at.asc().nulls_first())
|
||||
.limit(min(50, batch - checked))
|
||||
)
|
||||
.scalars()
|
||||
|
||||
@@ -90,7 +90,9 @@ def _ensure_videos(db: Session, yt: YouTubeClient, video_ids: list[str]) -> None
|
||||
if not vid or not cid or vid in seen:
|
||||
continue
|
||||
seen.add(vid)
|
||||
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")))
|
||||
# published_at is NOT NULL (0068); apply_video_details re-fills a missing date, but seed a now()
|
||||
# fallback here too so the constructor never carries an explicit NULL.
|
||||
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")) or now)
|
||||
apply_video_details(v, it)
|
||||
v.enriched_at = now
|
||||
db.add(v)
|
||||
|
||||
@@ -90,7 +90,10 @@ def stub_from_playlist_item(item: dict, channel_id: str) -> dict | None:
|
||||
video_id = content.get("videoId")
|
||||
if not video_id:
|
||||
return None
|
||||
published = parse_dt(content.get("videoPublishedAt") or snippet.get("publishedAt"))
|
||||
# published_at is NOT NULL (migration 0068); fall back to now() for the rare item that carries no
|
||||
# date at all, so the bulk stub insert never writes a NULL. The enrich pass replaces it if the
|
||||
# video's own snippet later yields a real publish date.
|
||||
published = parse_dt(content.get("videoPublishedAt") or snippet.get("publishedAt")) or _now()
|
||||
return {
|
||||
"id": video_id,
|
||||
"channel_id": channel_id,
|
||||
@@ -243,7 +246,8 @@ def apply_video_details(video: Video, item: dict) -> None:
|
||||
video.title = normalize_title(raw_title)
|
||||
video.description = snippet.get("description")
|
||||
if not video.published_at:
|
||||
video.published_at = parse_dt(snippet.get("publishedAt"))
|
||||
# NOT NULL (0068): a fresh insert with no date yet falls back to now() (enrich normally has one).
|
||||
video.published_at = parse_dt(snippet.get("publishedAt")) or _now()
|
||||
video.thumbnail_url = best_thumbnail(snippet.get("thumbnails")) or video.thumbnail_url
|
||||
video.duration_seconds = parse_iso8601_duration(content.get("duration"))
|
||||
video.view_count = int(stats["viewCount"]) if stats.get("viewCount") else None
|
||||
|
||||
@@ -484,9 +484,9 @@ def _finish_from_cache(job_id: int, asset_id: int) -> None:
|
||||
asset = db.get(MediaAsset, asset_id)
|
||||
if job and asset:
|
||||
fields = {"status": JobStatus.DONE, "progress": 100, "phase": None}
|
||||
if not job.display_name:
|
||||
if not job.display_name and asset.title:
|
||||
# display_name is VARCHAR(255); a title can be longer (e.g. a recipe in the title).
|
||||
fields["display_name"] = asset.title[:255] if asset.title else None
|
||||
fields["display_name"] = asset.title[:255]
|
||||
asset.last_access_at = datetime.now(timezone.utc)
|
||||
# Atomic finalize: only marks the job done while it is still running, so a concurrent
|
||||
# cancel/pause is never resurrected (see service.finalize_if_running).
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# Pinned dependency lock for reproducible builds (SEC-5). GENERATED — do not hand-edit.
|
||||
# Source of truth is requirements.txt (the floor ranges); regenerate after editing it with:
|
||||
# docker run --rm -v "$PWD":/b python:3.12-slim sh -c 'pip install -q -r /b/requirements.txt && pip freeze --exclude pip --exclude setuptools --exclude wheel' | sort
|
||||
# The prod Dockerfile + Dockerfile.test + CI install THIS file; pip-audit scans it in CI.
|
||||
APScheduler==3.11.3
|
||||
Authlib==1.7.2
|
||||
Mako==1.4.1
|
||||
MarkupSafe==3.0.3
|
||||
PyYAML==6.0.3
|
||||
SQLAlchemy==2.0.52
|
||||
alembic==1.19.1
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
anyio==4.14.2
|
||||
argon2-cffi-bindings==25.1.0
|
||||
argon2-cffi==25.1.0
|
||||
bgutil-ytdlp-pot-provider==1.3.1
|
||||
brotli==1.2.0
|
||||
certifi==2026.7.22
|
||||
cffi==2.1.1
|
||||
charset-normalizer==3.5.1
|
||||
click==8.4.2
|
||||
cryptography==50.0.0
|
||||
fastapi==0.141.1
|
||||
feedparser-sgmllib==2.1.0
|
||||
feedparser==6.0.14
|
||||
greenlet==3.5.5
|
||||
h11==0.16.0
|
||||
httpcore==1.0.9
|
||||
httptools==0.8.0
|
||||
httpx==0.28.1
|
||||
idna==3.18
|
||||
itsdangerous==2.2.0
|
||||
joserfc==1.7.4
|
||||
mutagen==1.48.1
|
||||
numpy==2.5.2
|
||||
psycopg-binary==3.3.4
|
||||
psycopg==3.3.4
|
||||
py3langid==0.3.0
|
||||
pycparser==3.0
|
||||
pycryptodomex==3.23.0
|
||||
pydantic-settings==2.15.0
|
||||
pydantic==2.13.4
|
||||
pydantic_core==2.46.4
|
||||
python-dotenv==1.2.3
|
||||
requests==2.34.2
|
||||
starlette==1.6.0
|
||||
typing-inspection==0.4.4
|
||||
typing_extensions==4.16.0
|
||||
tzlocal==5.4.4
|
||||
urllib3==2.7.0
|
||||
uvicorn==0.52.3
|
||||
uvloop==0.22.1
|
||||
watchfiles==1.2.0
|
||||
websockets==17.0.1
|
||||
yt-dlp-ejs==0.8.0
|
||||
yt-dlp==2026.7.4
|
||||
@@ -11,7 +11,7 @@ feedparser>=6.0,<7.0
|
||||
apscheduler>=3.10,<4.0
|
||||
py3langid>=0.3,<1.0
|
||||
itsdangerous>=2.1,<3.0
|
||||
cryptography>=46.0.7,<47
|
||||
cryptography>=50.0.0,<51
|
||||
argon2-cffi>=23.1,<26
|
||||
# Download center: yt-dlp does the extraction/download; ffmpeg (OS-level in the Dockerfile)
|
||||
# does the merge + postprocessing. The [default] extra pulls yt-dlp-ejs (the challenge-solver
|
||||
|
||||
@@ -90,3 +90,32 @@ def test_reset_confirm_is_single_use_and_kills_existing_sessions(client, make_us
|
||||
# The token is single-use: a replay is a clean 400, not a second reset.
|
||||
replay = client.post("/auth/password-reset/confirm", json={"token": raw, "password": "another pw!!"})
|
||||
assert replay.status_code == 400
|
||||
|
||||
|
||||
def test_activating_an_unverified_account_drops_its_preset_password(db, make_user):
|
||||
# SEC-10 (account pre-emption): an attacker registers victim@x with a password before the approved
|
||||
# owner arrives. The address was never verified, so admin approval must NOT hand that pre-set
|
||||
# password a working login — it is dropped and any session it minted is revoked.
|
||||
from app.routes.admin import _activate_user_by_email
|
||||
|
||||
victim = make_user(
|
||||
email="victim@example.com", password="attacker-chosen", email_verified=False, is_active=False
|
||||
)
|
||||
epoch0 = victim.session_epoch or 0
|
||||
_activate_user_by_email(db, "victim@example.com")
|
||||
assert victim.is_active is True
|
||||
assert victim.password_hash is None
|
||||
assert (victim.session_epoch or 0) == epoch0 + 1
|
||||
|
||||
|
||||
def test_activating_a_verified_account_keeps_its_password(db, make_user):
|
||||
# A legit account whose email IS verified (e.g. a no-SMTP registration) keeps its password on
|
||||
# approval — the guard must not break the normal flow.
|
||||
from app.routes.admin import _activate_user_by_email
|
||||
|
||||
user = make_user(
|
||||
email="legit@example.com", password="my own password", email_verified=True, is_active=False
|
||||
)
|
||||
_activate_user_by_email(db, "legit@example.com")
|
||||
assert user.is_active is True
|
||||
assert user.password_hash is not None
|
||||
|
||||
@@ -41,14 +41,16 @@ def _seed(db):
|
||||
Subscription(user_id=user.id, channel_id="UCa", priority=5, hidden=False),
|
||||
Subscription(user_id=user.id, channel_id="UCb", priority=0, hidden=False),
|
||||
])
|
||||
# 7 normal/organic videos. Deliberate duplicates (published_at day 1 x2 and day 3 x2, view_count
|
||||
# 100 x2, duration 60 x2 / 30 x2) and NULLs spread across the sort columns.
|
||||
# 7 normal/organic videos. Deliberate duplicates (published_at day 1 x2, day 2 x2 and day 3 x2,
|
||||
# view_count 100 x2, duration 60 x2 / 30 x2) and NULLs spread across the still-nullable sort columns
|
||||
# (view_count / duration_seconds / title / subscriber_count) to exercise nulls_last. published_at is
|
||||
# NOT NULL (migration 0068), so no NULL there — the duplicates keep the id tiebreaker load-bearing.
|
||||
rows = [
|
||||
# id ch published views dur title
|
||||
("vid00", "UCa", _dt(1), 100, 60, "alpha"),
|
||||
("vid01", "UCa", _dt(1), 100, 60, "beta"),
|
||||
("vid02", "UCa", _dt(2), None, None, "gamma"),
|
||||
("vid03", "UCb", None, 50, 120, None),
|
||||
("vid03", "UCb", _dt(2), 50, 120, None),
|
||||
("vid04", "UCb", _dt(3), 200, 30, "delta"),
|
||||
("vid05", "UCb", _dt(3), None, 30, "epsilon"),
|
||||
("vid06", "UCa", _dt(28), None, None, "zeta"),
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Feed search + watch-state views (E2 S3), DB-lane.
|
||||
|
||||
Regression nets for two query rewrites that must stay behaviour-equivalent:
|
||||
- PERF-12: the search channel-name branch moved from an ORed `unaccent(channels.title) ILIKE` (which
|
||||
defeated the search_vector GIN) to a pre-resolved `channel_id IN (...)`. A search must STILL return
|
||||
both videos whose text matches AND videos whose CHANNEL name matches.
|
||||
- PERF-16: the watched/hidden/in_progress views moved from an outer join + `coalesce(state.status,'new')`
|
||||
to an inner join on video_states + the raw column. Each view must return exactly the same set.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from app.models import Channel, ChannelTag, Subscription, Tag, Video, VideoState
|
||||
|
||||
|
||||
def _dt(day: int) -> datetime:
|
||||
return datetime(2024, 1, day, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _video(vid: str, ch: str, title: str) -> Video:
|
||||
return Video(
|
||||
id=vid, channel_id=ch, published_at=_dt(1), title=title,
|
||||
is_short=False, live_status="none", via_search=False, via_explore=False,
|
||||
)
|
||||
|
||||
|
||||
def _feed_ids(client, query: str) -> set[str]:
|
||||
resp = client.get(f"/api/feed?{query}&limit=50")
|
||||
assert resp.status_code == 200, resp.text
|
||||
return {it["id"] for it in resp.json()["items"]}
|
||||
|
||||
|
||||
def test_search_matches_both_video_text_and_channel_name(db, as_user, make_user):
|
||||
# Two subscribed channels: one whose NAME matches the query (its video's text does not), and one
|
||||
# whose video TEXT matches (its name does not). The pre-resolved channel_id branch must still OR in
|
||||
# the name match, so both videos come back — and an unrelated video on a matching-name-less channel
|
||||
# stays out.
|
||||
user = make_user(email="search@example.com")
|
||||
db.add_all([
|
||||
Channel(id="UCkube", title="Kubernetes Weekly", subscriber_count=10),
|
||||
Channel(id="UCother", title="Some Other Channel", subscriber_count=10),
|
||||
])
|
||||
db.add_all([
|
||||
Subscription(user_id=user.id, channel_id="UCkube", priority=0, hidden=False),
|
||||
Subscription(user_id=user.id, channel_id="UCother", priority=0, hidden=False),
|
||||
])
|
||||
db.add_all([
|
||||
_video("vidChan", "UCkube", "Totally unrelated content"), # channel NAME matches
|
||||
_video("vidText", "UCother", "kubernetes deep dive"), # video TEXT matches
|
||||
_video("vidNone", "UCother", "cooking pasta at home"), # neither matches
|
||||
])
|
||||
db.commit()
|
||||
|
||||
ids = _feed_ids(as_user(user), "q=kubernetes&show=all")
|
||||
assert ids == {"vidChan", "vidText"}
|
||||
|
||||
|
||||
def test_watch_state_views_return_the_right_sets(db, as_user, make_user):
|
||||
user = make_user(email="views@example.com")
|
||||
db.add(Channel(id="UCv", title="Channel V", subscriber_count=10))
|
||||
db.add(Subscription(user_id=user.id, channel_id="UCv", priority=0, hidden=False))
|
||||
db.add_all([
|
||||
_video("vWatch", "UCv", "watched one"),
|
||||
_video("vHide", "UCv", "hidden one"),
|
||||
_video("vProg", "UCv", "in progress one"),
|
||||
_video("vNew", "UCv", "brand new one"),
|
||||
])
|
||||
db.flush()
|
||||
db.add_all([
|
||||
VideoState(user_id=user.id, video_id="vWatch", status="watched"),
|
||||
VideoState(user_id=user.id, video_id="vHide", status="hidden"),
|
||||
# in progress = a stored resume position while still "new" (not watched/hidden)
|
||||
VideoState(user_id=user.id, video_id="vProg", status="new", position_seconds=30),
|
||||
# vNew has NO state row -> "new"/unwatched
|
||||
])
|
||||
db.commit()
|
||||
|
||||
client = as_user(user)
|
||||
assert _feed_ids(client, "show=watched") == {"vWatch"}
|
||||
assert _feed_ids(client, "show=hidden") == {"vHide"}
|
||||
assert _feed_ids(client, "show=in_progress") == {"vProg"}
|
||||
# unwatched excludes watched + hidden, keeps new AND in-progress
|
||||
assert _feed_ids(client, "show=unwatched") == {"vProg", "vNew"}
|
||||
# "all" shows everything except hidden
|
||||
assert _feed_ids(client, "show=all") == {"vWatch", "vProg", "vNew"}
|
||||
|
||||
|
||||
def _seed_tagged(db, make_user):
|
||||
"""A user subscribed to 4 channels with a spread of language/topic/other tags and differing video
|
||||
counts, so the count sum and the per-category facet drill-down are all non-trivial."""
|
||||
user = make_user(email="summary@example.com")
|
||||
tags = {
|
||||
"L1": Tag(user_id=None, name="Hungarian", category="language"),
|
||||
"L2": Tag(user_id=None, name="English", category="language"),
|
||||
"T1": Tag(user_id=None, name="Tech", category="topic"),
|
||||
"T2": Tag(user_id=None, name="Gaming", category="topic"),
|
||||
"O1": Tag(user_id=None, name="Favourite", category="other"),
|
||||
# A USER-owned tag (+ a user-owned ChannelTag link below) so the visibility branch
|
||||
# `ChannelTag.user_id == user.id` is exercised, not only the system `user_id IS NULL` one.
|
||||
"U1": Tag(user_id=user.id, name="MyPin", category="other"),
|
||||
}
|
||||
db.add_all(tags.values())
|
||||
chans = {c: Channel(id=c, title=c, subscriber_count=10) for c in ("chA", "chB", "chC", "chD")}
|
||||
db.add_all(chans.values())
|
||||
db.flush()
|
||||
# channel -> (system) tag keys
|
||||
links = {"chA": ["L1", "T1"], "chB": ["L1", "T2"], "chC": ["L2", "T1", "O1"], "chD": []}
|
||||
for cid, tkeys in links.items():
|
||||
db.add(Subscription(user_id=user.id, channel_id=cid, priority=0, hidden=False))
|
||||
for tk in tkeys:
|
||||
db.add(ChannelTag(channel_id=cid, tag_id=tags[tk].id, user_id=None))
|
||||
# The user's own private link (visible only to them): chB carries U1 for this user.
|
||||
db.add(ChannelTag(channel_id="chB", tag_id=tags["U1"].id, user_id=user.id))
|
||||
# differing video counts per channel (chA:3, chB:2, chC:1, chD:2)
|
||||
counts = {"chA": 3, "chB": 2, "chC": 1, "chD": 2}
|
||||
for cid, n in counts.items():
|
||||
db.add_all([_video(f"{cid}v{i}", cid, f"{cid} clip {i}") for i in range(n)])
|
||||
db.commit()
|
||||
return user, {k: v.id for k, v in tags.items()}
|
||||
|
||||
|
||||
# Query-param combos (as a template with {L1}/{T1}/… placeholders) that stress every facet path:
|
||||
# no tags, single-category, cross-category, disjunctive + conjunctive topic modes, and a search filter.
|
||||
_SUMMARY_CASES = [
|
||||
"show=all",
|
||||
"show=all&tags={L1}",
|
||||
"show=all&tags={T1}",
|
||||
"show=all&tags={L1}&tags={T1}",
|
||||
"show=all&tags={T1}&tags={T2}&tag_mode=or",
|
||||
"show=all&tags={T1}&tags={T2}&tag_mode=and",
|
||||
"show=all&tags={U1}", # user-owned tag (visibility == user.id branch)
|
||||
"show=all&tags={U1}&tags={L1}", # user tag AND a system tag, cross-category
|
||||
"show=all&q=clip",
|
||||
"show=unwatched",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case", _SUMMARY_CASES)
|
||||
def test_feed_summary_matches_count_plus_facets(db, as_user, make_user, case):
|
||||
# /feed/summary must return byte-identical numbers to the old /feed/count + /facets it replaces,
|
||||
# across the full matrix of tag filters — it reimplements the tag drill-down in Python, so this is
|
||||
# the equivalence net.
|
||||
user, tag_ids = _seed_tagged(db, make_user)
|
||||
client = as_user(user)
|
||||
qs = case.format(**tag_ids)
|
||||
|
||||
summary = client.get(f"/api/feed/summary?{qs}")
|
||||
count = client.get(f"/api/feed/count?{qs}")
|
||||
facets = client.get(f"/api/facets?{qs}")
|
||||
assert summary.status_code == count.status_code == facets.status_code == 200
|
||||
|
||||
assert summary.json()["count"] == count.json()["count"], qs
|
||||
assert summary.json()["facets"] == facets.json()["counts"], qs
|
||||
@@ -59,5 +59,14 @@ def test_head_migration_downgrades_and_reupgrades(db):
|
||||
# it lands, not during a rollback. Leaves the schema back at head for the rest of the session.
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "-1")
|
||||
command.upgrade(cfg, "head")
|
||||
assert _structural_drift() == []
|
||||
reupgraded = False
|
||||
try:
|
||||
command.upgrade(cfg, "head")
|
||||
reupgraded = True
|
||||
assert _structural_drift() == []
|
||||
finally:
|
||||
# If the re-upgrade itself failed, restore the SHARED schema to head so the failure doesn't
|
||||
# cascade into every later DB-lane test. Skip it when the upgrade already succeeded — re-running
|
||||
# it would be a redundant no-op and could mask a failing assert with the finally's own error.
|
||||
if not reupgraded:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
@@ -128,6 +128,35 @@ def test_apply_facet_fields_preserves_enriched_cast_names():
|
||||
assert "Diane Keaton" in enriched.people_text and "Francis Ford Coppola" in enriched.people_text
|
||||
|
||||
|
||||
def test_apply_facet_fields_keeps_over_cap_cast_after_a_watermark_reset():
|
||||
"""The enriched-cast guard must survive a watermark RESET (people_enriched_at nulled, e.g. by a
|
||||
migration like 0065). A stored cast LONGER than the listing cap can only be an enriched fuller list,
|
||||
so it's kept even though the row now reads as un-enriched — comparing against the cap, not the
|
||||
current listing's length, is what makes this robust (regression guard for the review fix)."""
|
||||
thin = {"Role": [{"tag": f"Actor {i}"} for i in range(20)]} # a full 20-name listing
|
||||
over_cap = SimpleNamespace(
|
||||
people_enriched_at=None, # watermark was reset
|
||||
directors=None,
|
||||
cast_names=[f"Enriched {i}" for i in range(22)], # 22 > the 20 cap => enriched
|
||||
)
|
||||
_apply_facet_fields(over_cap, thin)
|
||||
assert over_cap.cast_names == [f"Enriched {i}" for i in range(22)] # kept, not downgraded
|
||||
|
||||
|
||||
def test_apply_facet_fields_tracks_a_trimmed_listing_when_unenriched():
|
||||
"""A listing-sourced cast (at or below the cap, never enriched) must TRACK Plex cast edits in both
|
||||
directions. When Plex trims the cast, the shorter fresh listing replaces the stored one — the old
|
||||
length-vs-this-listing guard would have wrongly kept the stale longer cast."""
|
||||
stored = SimpleNamespace(
|
||||
people_enriched_at=None,
|
||||
directors=None,
|
||||
cast_names=["A", "B", "C", "D", "E"], # 5, at/below the cap => listing-sourced
|
||||
)
|
||||
trimmed = {"Role": [{"tag": "A"}, {"tag": "B"}]} # Plex removed 3 cast members
|
||||
_apply_facet_fields(stored, trimmed)
|
||||
assert stored.cast_names == ["A", "B"]
|
||||
|
||||
|
||||
def test_enrich_people_batch_persists_detail(db):
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Crypto primitives — a guard for the cryptography (Fernet) + argon2 dependency bumps.
|
||||
|
||||
Added with the cryptography 46->50 CVE bump: encrypt/decrypt/decrypt_optional had NO test, and they are
|
||||
the token-at-rest path (Google/API/SMTP/Plex secrets in system_config, OAuth refresh tokens). A fresh
|
||||
Fernet key is monkeypatched in so the test doesn't depend on TOKEN_ENCRYPTION_KEY being set in the env.
|
||||
Pure-logic (no DB).
|
||||
"""
|
||||
from cryptography.fernet import Fernet
|
||||
|
||||
from app import security
|
||||
|
||||
|
||||
def test_fernet_encrypt_decrypt_roundtrip(monkeypatch):
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
|
||||
assert security.decrypt(security.encrypt("hunter2")) == "hunter2"
|
||||
assert security.encrypt(None) is None
|
||||
assert security.decrypt(None) is None
|
||||
|
||||
|
||||
def test_decrypt_optional_degrades_instead_of_raising(monkeypatch):
|
||||
# Garbage / absent input -> None (never raises).
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
|
||||
assert security.decrypt_optional("not-a-valid-token") is None
|
||||
assert security.decrypt_optional(None) is None
|
||||
# And a ciphertext written under a DIFFERENT key (the TOKEN_ENCRYPTION_KEY-mismatch case that B-core-2
|
||||
# made non-fatal) also degrades to None rather than raising InvalidToken.
|
||||
ciphertext = security.encrypt("secret")
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key())) # rotate the key
|
||||
assert security.decrypt_optional(ciphertext) is None
|
||||
|
||||
|
||||
def test_password_hash_and_verify_roundtrip():
|
||||
hashed = security.hash_password("correct horse battery")
|
||||
assert security.verify_password("correct horse battery", hashed) is True
|
||||
assert security.verify_password("wrong password", hashed) is False
|
||||
assert security.verify_password("anything", None) is False
|
||||
@@ -115,7 +115,7 @@ export default function Channels() {
|
||||
// FB2: syncing subscriptions can change the feed immediately — newly-followed channels may
|
||||
// already have videos in the shared catalog, so refresh the feed too (not just the manager).
|
||||
qc.invalidateQueries({ queryKey: qk.feed() });
|
||||
qc.invalidateQueries({ queryKey: qk.feedCount() });
|
||||
qc.invalidateQueries({ queryKey: qk.feedSummary() });
|
||||
};
|
||||
|
||||
const userTags = (tagsQuery.data ?? []).filter((t) => !t.system);
|
||||
@@ -131,7 +131,7 @@ export default function Channels() {
|
||||
qc.invalidateQueries({ queryKey: qk.channels() });
|
||||
qc.invalidateQueries({ queryKey: qk.myStatus() });
|
||||
qc.invalidateQueries({ queryKey: qk.feed() });
|
||||
qc.invalidateQueries({ queryKey: qk.feedCount() });
|
||||
qc.invalidateQueries({ queryKey: qk.feedSummary() });
|
||||
},
|
||||
});
|
||||
// Priority is updated optimistically in place and NOT re-sorted/refetched, so the list doesn't
|
||||
|
||||
@@ -226,9 +226,11 @@ export default function Feed({
|
||||
}
|
||||
}, [query.dataUpdatedAt]);
|
||||
|
||||
const countQuery = useQuery({
|
||||
queryKey: qk.feedCount(queryFilters),
|
||||
queryFn: () => api.feedCount(queryFilters),
|
||||
// One shared query with the sidebar (same key + filters -> React Query dedupes to a single request):
|
||||
// the feed reads `.count`, the sidebar reads `.facets` (PERF-13, replaces feedCount + facets).
|
||||
const summaryQuery = useQuery({
|
||||
queryKey: qk.feedSummary(queryFilters),
|
||||
queryFn: () => api.feedSummary(queryFilters),
|
||||
staleTime: 30_000,
|
||||
enabled: !ytActive,
|
||||
placeholderData: keepPreviousData,
|
||||
@@ -252,7 +254,11 @@ export default function Feed({
|
||||
api
|
||||
.setState(id, status)
|
||||
.then(() => {
|
||||
qc.invalidateQueries({ queryKey: qk.feed() });
|
||||
// Mark stale WITHOUT refetching: the optimistic `overrides` layer + the matchesView filter
|
||||
// already show the correct state (a hidden/watched card drops out immediately), so refetching
|
||||
// every loaded infinite-query page here was pure waste — 10 pages x rapid marks = a refetch
|
||||
// storm (PERF-2). The next filter change / remount refetches fresh from the server.
|
||||
qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" });
|
||||
if (status === "hidden") {
|
||||
const v = loadedRef.current.find((x) => x.id === id);
|
||||
notify({
|
||||
@@ -309,7 +315,8 @@ export default function Feed({
|
||||
setOverrides((o) => ({ ...o, [id]: "new" }));
|
||||
api
|
||||
.clearState(id)
|
||||
.then(() => qc.invalidateQueries({ queryKey: qk.feed() }))
|
||||
// Stale-mark only (see setState above): the optimistic override already reset the card.
|
||||
.then(() => qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" }))
|
||||
.catch(() =>
|
||||
setOverrides((o) => {
|
||||
const next = { ...o };
|
||||
@@ -358,8 +365,7 @@ export default function Feed({
|
||||
// pre-search cache. Drop them so the feed re-fetches fresh on return.
|
||||
const dropFeedCaches = () => {
|
||||
qc.removeQueries({ queryKey: qk.feed() });
|
||||
qc.removeQueries({ queryKey: qk.feedCount() });
|
||||
qc.removeQueries({ queryKey: qk.facets() });
|
||||
qc.removeQueries({ queryKey: qk.feedSummary() });
|
||||
};
|
||||
|
||||
// --- Live YouTube search mode -------------------------------------------------------------
|
||||
@@ -622,10 +628,10 @@ export default function Feed({
|
||||
)}
|
||||
<span className="mx-1 h-5 w-px bg-border" aria-hidden="true" />
|
||||
<span data-testid="feed-result-count" className="text-xs text-muted whitespace-nowrap">
|
||||
{countQuery.data
|
||||
{summaryQuery.data
|
||||
? t("feed.videoCount", {
|
||||
count: countQuery.data.count,
|
||||
formattedCount: countQuery.data.count.toLocaleString(),
|
||||
count: summaryQuery.data.count,
|
||||
formattedCount: summaryQuery.data.count.toLocaleString(),
|
||||
})
|
||||
: " "}
|
||||
</span>
|
||||
|
||||
@@ -112,7 +112,7 @@ export default function NotificationsPanel() {
|
||||
.setState(meta.videoId, "new")
|
||||
.then(() => {
|
||||
qc.invalidateQueries({ queryKey: qk.feed() });
|
||||
qc.invalidateQueries({ queryKey: qk.feedCount() });
|
||||
qc.invalidateQueries({ queryKey: qk.feedSummary() });
|
||||
// Quietly resolve the now-stale notice (no confirmation toast — the change is visible).
|
||||
resolveVideo(meta.videoId);
|
||||
})
|
||||
|
||||
@@ -74,7 +74,7 @@ export default function OnboardingWizard({ me, onClose }: { me: Me; onClose: ()
|
||||
onSuccess: () => {
|
||||
// Feed/channels/status now have content — refresh the app behind the wizard.
|
||||
qc.invalidateQueries({ queryKey: qk.feed() });
|
||||
qc.invalidateQueries({ queryKey: qk.feedCount() });
|
||||
qc.invalidateQueries({ queryKey: qk.feedSummary() });
|
||||
qc.invalidateQueries({ queryKey: qk.channels() });
|
||||
qc.invalidateQueries({ queryKey: qk.myStatus() });
|
||||
},
|
||||
|
||||
@@ -862,10 +862,13 @@ export default function PlayerModal({
|
||||
cancelled = true;
|
||||
window.clearInterval(timer);
|
||||
window.removeEventListener("pagehide", onHide);
|
||||
// Final flush, then refresh the feed + playlists so resume bars reflect this session.
|
||||
// Final flush, then mark the feed + playlists stale so resume bars reflect this session on the
|
||||
// next natural refetch — WITHOUT refetching every loaded feed page on each player close (PERF-2:
|
||||
// opening/closing the player through a watch session otherwise re-runs the whole infinite query
|
||||
// each time). Progress is already persisted server-side; only the on-screen resume bar waits.
|
||||
Promise.resolve(persist()).finally(() => {
|
||||
qc.invalidateQueries({ queryKey: qk.feed() });
|
||||
qc.invalidateQueries({ queryKey: qk.playlist() });
|
||||
qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" });
|
||||
qc.invalidateQueries({ queryKey: qk.playlist(), refetchType: "none" });
|
||||
});
|
||||
const p = playerRef.current;
|
||||
if (p && typeof p.destroy === "function") {
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { useState } from "react";
|
||||
import { withActiveAccount } from "../lib/activeAccount";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
@@ -418,7 +419,7 @@ function SignInMethods({ me }: { me: Me }) {
|
||||
) : (
|
||||
<button
|
||||
onClick={() => {
|
||||
window.location.href = "/auth/link";
|
||||
window.location.href = withActiveAccount("/auth/link");
|
||||
}}
|
||||
className="shrink-0 glass-card glass-hover px-3 py-1.5 rounded-xl text-sm transition"
|
||||
>
|
||||
@@ -644,7 +645,7 @@ function Account({ me, onOpenWizard }: { me: Me; onOpenWizard: () => void }) {
|
||||
grantedHint={t("settings.account.readGrantedHint")}
|
||||
enableHint={t("settings.account.readEnableHint")}
|
||||
onEnable={() => {
|
||||
window.location.href = "/auth/upgrade?access=read";
|
||||
window.location.href = withActiveAccount("/auth/upgrade?access=read");
|
||||
}}
|
||||
/>
|
||||
<AccessRow
|
||||
@@ -653,7 +654,7 @@ function Account({ me, onOpenWizard }: { me: Me; onOpenWizard: () => void }) {
|
||||
grantedHint={t("settings.account.writeGrantedHint")}
|
||||
enableHint={t("settings.account.writeEnableHint")}
|
||||
onEnable={() => {
|
||||
window.location.href = "/auth/upgrade?access=write";
|
||||
window.location.href = withActiveAccount("/auth/upgrade?access=write");
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
|
||||
@@ -87,14 +87,16 @@ export default function Sidebar({
|
||||
// Live per-tag channel counts for the current filter context. Debounce the search term so
|
||||
// typing doesn't refire facets per keystroke, and keep previous counts during a refetch.
|
||||
const facetFilters: FeedFilters = { ...filters, q: useDebounced(filters.q, 300) };
|
||||
const facetsQuery = useQuery({
|
||||
queryKey: qk.facets(facetFilters),
|
||||
queryFn: () => api.facets(facetFilters),
|
||||
// Shared with the feed's count (same key + filters -> React Query dedupes to one request): the feed
|
||||
// reads `.count`, this reads `.facets` (PERF-13, replaces the separate facets query).
|
||||
const summaryQuery = useQuery({
|
||||
queryKey: qk.feedSummary(facetFilters),
|
||||
queryFn: () => api.feedSummary(facetFilters),
|
||||
staleTime: 30_000,
|
||||
placeholderData: keepPreviousData,
|
||||
});
|
||||
const facetsReady = !!facetsQuery.data;
|
||||
const facetCounts = facetsQuery.data?.counts ?? {};
|
||||
const facetsReady = !!summaryQuery.data;
|
||||
const facetCounts = summaryQuery.data?.facets ?? {};
|
||||
const chipCount = (tag: Tag): number =>
|
||||
facetsReady ? (facetCounts[String(tag.id)] ?? 0) : tag.channel_count;
|
||||
const visibleChips = (list: Tag[]): Tag[] => {
|
||||
|
||||
@@ -19,6 +19,17 @@ export function activeAccountId(): number | null {
|
||||
}
|
||||
}
|
||||
|
||||
// Append `?account=<active>` to a top-level navigation URL (SEC-11). XHRs carry the active account in
|
||||
// the X-Siftlode-Account header, but a full-page navigation (window.location) can't, so the OAuth
|
||||
// link/upgrade GETs would resolve to the SESSION DEFAULT account instead of the tab's active one — the
|
||||
// identity/token would then land on the wrong account. This threads the active account through the URL
|
||||
// so resolved_user_id picks it. No-op when there's no distinct active account pinned yet.
|
||||
export function withActiveAccount(path: string): string {
|
||||
const id = activeAccountId();
|
||||
if (id == null) return path;
|
||||
return `${path}${path.includes("?") ? "&" : "?"}account=${id}`;
|
||||
}
|
||||
|
||||
// Reactive active-account. setActiveAccount/clearActiveAccount write sessionStorage — a silent op that
|
||||
// triggers no React render — so this tab's per-account state (sort/filter/view/layout, all keyed by the
|
||||
// account) would never re-read once the account is finally known. That bites exactly once, on first
|
||||
|
||||
@@ -113,6 +113,8 @@ function feedQuery(f: FeedFilters, cursor: string | null, limit: number): string
|
||||
export const feedApi = {
|
||||
feed: (f: FeedFilters, cursor: string | null, limit: number): Promise<FeedResponse> =>
|
||||
req(`/api/feed?${feedQuery(f, cursor, limit)}`),
|
||||
// SUPERSEDED by feedSummary (PERF-13) — no component calls this anymore; the backend /feed/count
|
||||
// endpoint it targets is kept only as the /feed/summary equivalence-test oracle. Retire together.
|
||||
feedCount: (f: FeedFilters): Promise<{ count: number }> =>
|
||||
req(`/api/feed/count?${filterParams(f).toString()}`),
|
||||
// Live YouTube search: results are materialised into the catalog and returned as feed cards.
|
||||
@@ -129,8 +131,14 @@ export const feedApi = {
|
||||
// the now-orphaned, un-kept videos). Returns how many videos were removed.
|
||||
clearSearch: (videoIds: string[]): Promise<{ removed: number }> =>
|
||||
req("/api/search/clear", { method: "POST", body: JSON.stringify({ video_ids: videoIds }) }),
|
||||
// SUPERSEDED by feedSummary (PERF-13) — no component calls this anymore; the backend /facets
|
||||
// endpoint it targets is kept only as the /feed/summary equivalence-test oracle. Retire together.
|
||||
facets: (f: FeedFilters): Promise<{ counts: Record<string, number> }> =>
|
||||
req(`/api/facets?${filterParams(f).toString()}`),
|
||||
// The sidebar's total count + per-tag channel facets in ONE request + ONE grouped query, replacing
|
||||
// the separate feedCount + facets the feed and sidebar used to fire on every filter change (PERF-13).
|
||||
feedSummary: (f: FeedFilters): Promise<{ count: number; facets: Record<string, number> }> =>
|
||||
req(`/api/feed/summary?${filterParams(f).toString()}`),
|
||||
// idempotent: setting a state / saving a progress checkpoint is safe to replay, so these
|
||||
// recover from a transient gateway/keepalive 502 instead of surfacing it (see req()).
|
||||
setState: (id: string, status: string) =>
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
// suppresses the auto-popup on future logins (they
|
||||
// can still reopen it from Settings → Account).
|
||||
|
||||
import { withActiveAccount } from "./activeAccount";
|
||||
import { getAccountRaw, LS, setAccountRaw } from "./storage";
|
||||
|
||||
const ONBOARD_ACTIVE = "siftlode.onboarding.active"; // sessionStorage (not in LS, which is localStorage)
|
||||
@@ -36,7 +37,7 @@ export function shouldAutoOpenOnboarding(me: {
|
||||
/** Mark a grant as in progress, then send the user to Google's consent screen. */
|
||||
export function beginGrant(access: "read" | "write"): void {
|
||||
sessionStorage.setItem(ONBOARD_ACTIVE, "1");
|
||||
window.location.href = `/auth/upgrade?access=${access}`;
|
||||
window.location.href = withActiveAccount(`/auth/upgrade?access=${access}`);
|
||||
}
|
||||
|
||||
/** Leave the wizard. `dismiss` suppresses the future auto-popup (used when read is skipped). */
|
||||
|
||||
@@ -16,6 +16,8 @@ describe("qk — bare vs parameterized branch", () => {
|
||||
expect(qk.feedCount(FILTERS)).toEqual(["feed-count", FILTERS]);
|
||||
expect(qk.facets()).toEqual(["facets"]);
|
||||
expect(qk.facets(FILTERS)).toEqual(["facets", FILTERS]);
|
||||
expect(qk.feedSummary()).toEqual(["feed-summary"]);
|
||||
expect(qk.feedSummary(FILTERS)).toEqual(["feed-summary", FILTERS]);
|
||||
expect(qk.playlist()).toEqual(["playlist"]);
|
||||
expect(qk.plexShow()).toEqual(["plex-show"]);
|
||||
expect(qk.plexShow("s1")).toEqual(["plex-show", "s1"]);
|
||||
|
||||
@@ -27,6 +27,8 @@ export const qk = {
|
||||
filters !== undefined ? (["feed-count", filters] as const) : (["feed-count"] as const),
|
||||
facets: (filters?: FeedFilters) =>
|
||||
filters !== undefined ? (["facets", filters] as const) : (["facets"] as const),
|
||||
feedSummary: (filters?: FeedFilters) =>
|
||||
filters !== undefined ? (["feed-summary", filters] as const) : (["feed-summary"] as const),
|
||||
ytSearch: (q?: string | null, count?: number) =>
|
||||
q !== undefined ? (["yt-search", q, count] as const) : (["yt-search"] as const),
|
||||
videoDetail: (id: string) => ["video-detail", id] as const,
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { RELEASE_NOTES } from "./releaseNotes";
|
||||
|
||||
// DX-6: the app version is hand-maintained in two places that MUST agree — the repo-root VERSION file
|
||||
// (the single source baked into the image + served at /api/version) and the newest RELEASE_NOTES entry
|
||||
// (what the in-app "What's new" shows). A release that bumps one but forgets the other ships a wrong
|
||||
// changelog or a mismatched banner. This locks them together so the gate catches the drift.
|
||||
describe("release version consistency", () => {
|
||||
it("RELEASE_NOTES[0].version matches the VERSION file", () => {
|
||||
const version = readFileSync(new URL("../../../VERSION", import.meta.url), "utf-8").trim();
|
||||
expect(RELEASE_NOTES[0]?.version).toBe(version);
|
||||
});
|
||||
|
||||
it("release notes are ordered newest-first with unique versions", () => {
|
||||
const versions = RELEASE_NOTES.map((r) => r.version);
|
||||
expect(new Set(versions).size).toBe(versions.length); // no duplicate version entries
|
||||
});
|
||||
});
|
||||
@@ -14,6 +14,39 @@ export interface ReleaseEntry {
|
||||
}
|
||||
|
||||
export const RELEASE_NOTES: ReleaseEntry[] = [
|
||||
{
|
||||
version: "0.68.0",
|
||||
date: "2026-08-18",
|
||||
summary:
|
||||
"A big performance pass on the feed and search: pages load and search returns near-instantly, even on a large library, and marking videos no longer stutters.",
|
||||
fixes: [
|
||||
"The feed loads much faster. The default “newest” feed and channel pages now use a matching database index instead of scanning the whole video table on every page, so scrolling stays quick even with hundreds of thousands of videos (this was the likely cause of the occasional slow feed).",
|
||||
"Search is dramatically faster — a search that could take several seconds now returns in a fraction of one, and still matches both video text and channel names.",
|
||||
"The Watched / Hidden / In-progress views open faster, and deep scrolling (far down a long feed) no longer slows down page by page.",
|
||||
"Marking a video watched or hidden, and closing the player, no longer briefly reloads the whole feed — the change shows instantly and the list stays put.",
|
||||
],
|
||||
chores: [
|
||||
"The sidebar's total count and per-tag channel counts now come from a single request + one database pass instead of four, cutting the work done on every filter change.",
|
||||
"The background enrichment and maintenance scans, and the channel search branch, are now index-served instead of full-table scans — much less database load per scheduler run. Backed by new automated equivalence/paging tests and verified end-to-end.",
|
||||
],
|
||||
},
|
||||
{
|
||||
version: "0.67.0",
|
||||
date: "2026-08-18",
|
||||
summary:
|
||||
"A foundation release: reliability and security hardening under the hood, plus a few fixes you'll actually notice.",
|
||||
fixes: [
|
||||
"Canceling or pausing a download can no longer delete a video file that another download is still using — closing a rare race that could remove a file out from under a live download.",
|
||||
"Switching to an account that was signed out elsewhere (after a password reset or “log out everywhere”) now just asks you to sign back in to that one account, instead of logging you out of every account in the switcher.",
|
||||
"A failed YouTube search now shows the real reason instead of a generic “connection lost” message that silently retried and spent extra search quota.",
|
||||
"If the server's encryption key is ever misconfigured, the app now starts and degrades gracefully instead of failing to load at all.",
|
||||
],
|
||||
chores: [
|
||||
"Security hardening around account linking and email verification: a password set on an address before it was verified is no longer trusted once the real owner proves ownership (via Google or admin approval), and linking a Google account now always attaches it to the account you're actually using.",
|
||||
"A test safety net for the security-critical core — login and session revocation, feed paging, download finalization, and database-migration drift — that is required to actually run: the test lane now fails loudly instead of silently skipping when its database or tools are missing.",
|
||||
"Reproducible, audited builds: dependencies are pinned to an exact lockfile that is scanned for known vulnerabilities on every build (which caught and cleared 4 cryptography-library CVEs), the runtime moved to Node 22, a server-side CI pipeline runs the full gate on every push, and baseline security response headers were added.",
|
||||
],
|
||||
},
|
||||
{
|
||||
version: "0.66.0",
|
||||
date: "2026-08-17",
|
||||
|
||||
Reference in New Issue
Block a user