The YouTube API reads neither a user's watch history nor their Watch later list, so the Takeout
export is the only way in. `POST /api/takeout` takes one file of it (watch-history.json or
Watch later-videos.csv, told apart by content), keeps its videos in `takeout_items` and queues the
user's `takeout_import` run, whose outcome reaches their inbox.
The run marks watched the known videos the Shorts probe has cleared, at the Takeout time; a Short
is not marked, and a state the user set here (watched, hidden, or changed after that time) wins, by
the upsert's own condition. A video the catalog does not know yet waits with no outcome. Watch later
videos are fetched if unknown and appended oldest first.
An ad-hoc run can now ask for a completion notice (`run_ad_hoc(..., notify=True)`).
Feed, playlists, saved views, tags and the Home pins do no database work in their routes any more:
reads go through reader/queries (and feedquery for a feed page, its counts and facets), writes
through reader/service and discover/pins, which flush; the handlers commit. Reader is fenced below
its HTTP layer too: its owner filters go through userscope, and the catalog's jobs ask Reader
instead of joining its tables (the auto-tagger's system tags, the cleanup jobs' keep rule, the
maintenance notice). A playlist sync's catalog ingest moves to sync/videos.
VideoState, Playlist, PlaylistItem, SavedView, ComfortPin, Tag and ChannelTag leave app/models.py
for app/reader/models.py; app.tables registers them. No schema change.
Plex, library and /watch tell the operating system what plays (title, line, picture) and take its
play / pause / skip / seek through one hook (lib/mediaSession); the Plex player drives it with its own
seek and clock, so the lock screen shows the film's position, not the stream's. The YouTube player is
left alone: its iframe already publishes all of it, and the OS routes to the frame that plays.
The library and /watch players get the basic keys (lib/mediaKeys, the keymap's mediaPlayer rows) and
open where the viewer stopped (lib/resume): the library keeps the point on the server, per user and
library item (library_states, POST /api/downloads/{id}/progress, listed back on both listings), the
shared link in this browser. A started library row continues, its menu can start it over, and its
picture carries the cards' progress bar. The resume rule (too early / near the end = from the start)
is one function shared with the YouTube route.
The library dialog opens focused on its panel (Modal initialFocus "player"): on the ✕, Space closed
the film. While the media element itself has the focus, Space and the arrows stay its own controls'
— the browser acts on them whatever a listener does, and one Space played and paused.
The code cited its plan instead of saying why: `E28 review J4`, `(R6 S3b)`, `S1 justified…`,
`deep-review round 5`. That plan lives in the private ops repo, so for every reader of this one the
citation was noise, and a sentence whose subject was the plan step ("round 2 showed…") hid its
lesson behind an id. Each one is gone and the "why" kept: a bare citation is deleted, a sentence
carrying a lesson is reworded around it ("a later attempt showed…").
Measured (`scripts/audit_metrics.py`, product code): 652 lines in 270 files carried an epic code by
the definition the next commit holds (507 / 241 by the old, two-digit-only one) -> 0. Across all
code, tests, scripts and configs: ~1100 epic-code lines plus ~320 sprint/round ids.
Behaviour unchanged, checked mechanically: every changed .py file has the same AST once docstrings
are blanked; css/sh/toml/yml/json files are identical once comments are stripped; the only
non-comment changes in TS/JS are test titles, one description string in a contract table, and five
ESLint messages, whose text is all the gates snapshot diff holds. Not touched: Markdown (docs and
the changelog keep their history), Alembic migrations, TV episode notation (`S2 · E5`), DESIGN.md
ids (`C1`, `B4`, `S3h`), tracker ids (`SEC-16`, `DX-17`).
One stale claim fixed on the way: test_hls_reaper's docstring said the player checkpoints "every
10 s whether or not it is playing", which plex/stream.py calls wrong; it now says what the code does.
The demo account — one shared user that whitelisted emails entered from
the login page, reset on a schedule — goes end to end (owner decision,
E34): demo.py and the reset job, users.is_demo and demo_whitelist
(migration 0098 deletes the demo user, its audit rows, the reset job's
override and notices, then drops the column and table; the downgrade
restores the empty schema), the /auth/demo entry, the admin whitelist and
reset, the Demo tab, the banner, the login page's "Try the demo", and
every is_demo branch in routes, userscope, scheduler and the UI.
require_human only ever refused the demo account, so its 87 uses become
current_user (owner-approved); require_write_scope keeps its own check.
The login page's "or" divider now shows only beside the Google button:
without the demo entry under it, an instance without Google would have
said "or" to nothing.
A contract test (removedFeatures) fails on the word anywhere in the
backend, frontend, e2e or scripts — seen red on a planted hit. Size
ceilings tightened for the modules that shrank; admin.py is back under
the limit.
ruff gains RUF100, PLW1510 and flake8-bugbear, every hit fixed rather than
ignored. B008 stays on with FastAPI's Depends/Query declared immutable.
basedpyright moves to standard mode, pytest to --strict-markers, and a hand
run of pytest now requires the database unless SIFTLODE_TEST_REQUIRE_DB=0.
PLW1510 found a real bug: the three ffprobe probes in plex/stream.py never
looked at the exit status, so a probe that failed was read as the file's
truth (an HDR source sent down the HW-safe path, a multi-audio file collapsed
to one track, a subtitle offset taken from garbage) and the warning each
except branch promises never fired. They now run with check=True.
A broad word matched most of the library and ts_rank read every match:
"a" took 4.4 s a page on prod. The window's start is read once (0.1 s) and
rides the cursor, so a video ingested mid-scroll cannot repeat a row; the
page is read as its two halves, each on an index (80 ms + 5 ms).
The Postgres image is Alpine, whose en_US.utf8 is byte order: "Életrajzi" and
"Ötödik" sorted after "Zorró" in the feed, the channel lists, the Plex grid,
its genres and collections. `app/collation.py` names the ICU collation of the
reader's language (hu-x-icu puts "Ö" after "O" and "Cs" after "Cu"), wraps
every text ORDER BY with it - the feed's keyset cursor compares with the same
expression, so paging walks the order it sorted in - and orders the lists built
in Python (genre and studio facets, collection members, the discovery re-sort)
through the same collation. Plex title tiebreaks use the shown (HU) title.
Two rows of the canonical-patterns table had drifted, found while closing the roadmap audit:
- 63 mutation responses carried no `ok` — `{"unsubscribed": true}`, `{"status": "ok"}`,
`{"added": 3, …}` and so on. Each now leads with `"ok": true` (additive: every field a client
reads is still there).
- Twelve route helpers committed on their own. Six were just the end of their handler's work and now
flush: the playlist item add/remove and Watch later get-or-create, the Plex series/season bulk
mark, and the invite decision + account activation — which an approval used to land in separate
transactions. The other six commit on purpose (a token durable before its link is mailed, a
single-use token burned as its own unit, the Google callback's tail, a background registration
with its own session, the staged search ingest around YouTube calls), each listed with its reason.
`tests/test_route_conventions.py` holds both rules in code; CLAUDE.md's two rows point at it.
~150 HTTPExceptions carried a hand-written English sentence as their detail,
shown verbatim under the Hungarian UI — in the blocking error dialog and on
every inline surface that printed HttpError.detail (sign-in, setup wizard,
change password, the passphrase gate, live search). Four also quoted an
upstream exception's text back to the user.
- app/refusal.py: `refused(status, reason, **params)` → {code: "refused",
reason, …params}; localizeDetail translates errors.refused.<reason> with the
params interpolated. "Unknown X" 404s use the existing not_found(what).
- sysconfig.set_value raises InvalidSetting (reason + bounds) instead of an
English ValueError; the three writers answer it through setting_refused.
- The Plex connection/write/image failures log what the upstream said and
answer a reason; the search scraper's failure is logged and answered as
youtube/unavailable (it crashed refusal_reason before).
- HttpError.detail now only ever holds a translated reason: a string detail
(only the framework's own can remain) is left out, so reading .detail is
safe everywhere — including destructured, the hole the raw-detail source
scan could not see. That scan and its five-file exception list are gone.
- Enforced: test_http_details.py fails on any HTTPException with a sentence
for a detail; the contract test requires every refused reason and every
not_found "what" in both languages, and no key for one that is gone.
- local/jsx-comment-text: a // inside JSX children renders as text; the rule
is ten lines in eslint.config.js rather than eslint-plugin-react's ~1800
lockfile lines.
The two manual syncs (subscriptions import, playlist pull) had no YouTubeError
handler, so a quota, token or transport failure was a raw 500 and the blocking
"Server error" dialog (E9 F1). The other YouTube routes answered either the
upstream's English or a playlist-scoped "youtube_failed".
One helper, routes/_common.youtube_refusal: 422 {code: "youtube", reason} with
the reason sorted by what the user can do — quota (wait for the reset),
reconnect (the token step refused), unavailable (retry). Used by every route
that surfaces a YouTube failure; localizeDetail translates it.
"Reconnect" had no path: a grant Google revoked still read as granted, so
Settings and the wizard offered no connect button while every call failed.
An invalid_grant from the token endpoint now clears the stored grant.
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.
What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
degraded to a software transcode with nothing saying why; a failed audio probe
collapsed a multi-audio file to one track and the player showed no language
picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
own root, leaving every crashed session's segments on disk forever.
Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.
ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.
logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.
tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
Deep-review round 4, backend half.
- The periodic HLS reaper (round 3) applied the "finished and unfetched for 30 s" rule that only a
NEW session request may apply. A remux finishes long before the film, and a paused player with a
full buffer fetches nothing, so a one-minute pause killed the session and playback died ~30 s after
resuming. The periodic pass is `reap_abandoned` now (the long idle limit, finished or not).
- A subtitle `publish` that could not keep its output (full / read-only cache volume) returned
quietly, so every poll queued the window and the whole-container pass again. It reports failure
and the pass is backed off like a failed read; a cached VTT that cannot be READ is answered
`unavailable` (`cache_unreadable`) instead of scheduling reads that can never land.
- The subtitle routes bound `rating_key` / `sub_ord` (they become a cache file name: a 250-digit
one was ENAMETOOLONG, which pathlib's exists() re-raises -> a 500), and the cache helpers no longer
throw on such a name.
- One app-wide handler turns psycopg's "text fields cannot contain NUL" DataError into a 422: round 3
guarded only the by-rating_key lookup, and `contains=`, `contains_group=`, reorder/bulk key lists
and the collection search still 500'd.
- The hand-written `db.get(...) is None -> 404 "Unknown ..."` siblings use `get_or_404` with a
`what` (channel / video / user), so the Hungarian UI no longer shows English for a gone row.
Deep-review round 3 of bug hunt 2026-09.
- The YouTube and Plex progress bodies are bounded by POSITION_MAX_S (the int4 position columns): a
finite position of 2^31 or more was a DataError, i.e. a 500 — the sibling of round 2's non-finite
fix.
- plex/stream.py promised "a periodic reaper", but reap_idle ran only inside the NEXT session
request: after the last playback an abandoned session (its ffmpeg still working, its directory a
whole segmented copy of the film) lived until someone pressed play again. A lifespan task reaps
every 60 s.
- Tests for what round 2 left untested: the 422's body keeps the list shape the UI localizes, and
stream_session refuses non-finite start/aoff.
The JSON parser under FastAPI accepts `Infinity`/`NaN`, and both progress checkpoints truncate the
position with `int()` — a hand-written checkpoint came back as a 500 with a traceback. Their bodies
now refuse non-finite floats, and the app's validation handler — FastAPI's own shape — echoes a
non-finite input by name: the default one failed to serialise its own error (JSON has no NaN), which
was a 500 by another route. Sibling of the subtitle routes' non-finite query parameters.
Add a Year filter to the feed rail: quick chips for recent years plus a
'More…' popover listing all years with per-year video counts, sortable by
year or count. Selecting a year narrows the feed to that whole calendar
year (a published_after/before range applied in the api layer) and is
mutually exclusive with the Upload-date control.
Backend: new /api/feed/years endpoint + feed_years() — one grouped query
that honours every filter EXCEPT the publish-date narrowing (dropped), so
the year list stays a live drill-down and switching years never zeroes the
others. FE keys the list on the date-stripped filters, so picking a year
re-queries the feed but not the list.
Wires year through FeedFilters, the filterParams/URL (?year) round-trip,
the active-filter chip row, and the panel layout. i18n HU+EN.
Tests: feed_years count query (date narrowing ignored, other filters
honoured); dateFilter year-clearing; urlState year round-trip.
Filter to channels you haven't personally tagged in the channel manager
(no user-owned ChannelTag; auto language/topic tags don't count):
- Feed: a "No tag" sentinel in the "Your tags" sidebar combobox (exclusive
with real tag picks) + a removable "Untagged" chip; new `untagged` bool
feed param / `notag` URL key, persisted + coerced on every filter path.
- Channel manager: a virtual "No tag" chip in the tag-chip row filters the
table to untagged channels (client-side, shared NO_TAG_SENTINEL).
Backend `untagged` is a plain channel-level narrowing, so feed_summary keeps
it applied in the base pass — the count/facet lockstep holds with no mirror.
Note: item E (per-tag counts) and item D (Any/All toggle) were already
shipped; S1 delivers only item B.
The tag 'Match: Any/All' toggle was a single GLOBAL setting that only ever
affected topics, shown as its own row under the Topic widget. Make it
per-category — language, topic, and 'your tags' each get their own independent
any/all — and move it into each filter combobox's popover footer (like
SortControl's direction segment), reclaiming the standalone Match row.
Backend: the feed query already groups selected tags by category; replace the
single tag_mode with per-category tag_mode_language/topic/other params and apply
the conjunctive (HAVING all) branch per category. Mirrored across filtered_query,
feed_summary.passes(), and the /facets drill-down exclusion (kept in lockstep;
equivalence test updated to tag_mode_topic). No schema change.
Frontend: FeedFilters.tagMode -> tagModes {language,topic,other} (pure leaf
lib/api/tagModes with a coerceTagModes migration for the legacy global value);
serialized to the API and the shareable URL per category (legacy ?tagmode=and
still resolves into the topic slot); saved-view blobs migrate on read. Combobox
gains an optional matchMode/onMatchModeChange footer segment, shown once 2+
values are picked; the three tag comboboxes wire it to their category.
Verified: backend feed tests green (12, incl. the count/facets equivalence net);
live, the footer appears at 2+ selections, toggles aria-pressed, serializes
tagmode_topic=and to the URL, and the request carries tag_mode_topic=and (200).
374 vitest, tsc, ruff, eslint (0 err).
Review round on the E2 stack (S1-S5). No correctness bugs; the surviving
low/medium findings addressed:
- altitude (#1): extract the shared `tags_by_category` helper used by BOTH
filtered_query (SQL tag WHERE) and feed_summary (Python facet/count), removing
the category-grouping duplication, and add bidirectional "keep in lockstep —
equivalence-tested by test_feed_summary_matches_count_plus_facets" comments so
the SQL<->Python tag rule can't silently drift.
- test-coverage (#2): the /feed/summary equivalence test seeded only system tags,
leaving the `ChannelTag.user_id == user.id` visibility branch untested. Add a
user-owned Tag + link + two combos; the suite is now 346 (was 344) and the new
cases pass (feed_summary agrees with /feed/count + /facets on user-visible tags).
- dead-code (#3): annotate get_feed_count, get_facets, and the FE api.feedCount/
facets as "SUPERSEDED by /feed/summary — kept only as the equivalence oracle;
retire together" so the intentionally-retained surface reads as deliberate.
Not changed (confirmed no-op): the search channel-name resolve (#4) is
once-per-request on a 633-row table and a deliberate pre-resolve for PERF-12's
BitmapOr plan; the facet loop (#5) already computes passes() once per
(channel, category) with a distinct exclude each, so there is no recomputation.
Gate green: ruff + 346 pytest; frontend tsc/knip/prettier (FE change is a comment).
The FE fired /feed/count (1 pass) + /facets (3 passes) on every filter change =
4 full scans of the subscribed set. New /api/feed/summary does ONE grouped query
(the base filter with tags stripped, GROUP BY channel_id -> per-channel video
counts) and derives the total count + every tag's distinct-channel facet count in
Python on the ~few-hundred-row channel map, faithfully mirroring filtered_query's
channel-based tag logic + the /facets drill-down (own category dropped, unless a
conjunctive topic-AND). Returns {count, facets} — the same numbers the two old
endpoints returned.
FE: a single shared query (Feed reads .count, Sidebar reads .facets); same key +
filters, so React Query dedupes to ONE request. Invalidations repointed to
qk.feedSummary. The old /feed/count + /facets endpoints (and their now-unused FE
api/qk) are kept as the equivalence-test oracle + safety net; a later cleanup can
drop them.
New equivalence test (test_feed_views, 8 filter combos incl. cross-category,
disjunctive/conjunctive topic modes, search): /feed/summary count == /feed/count
and facets == /facets, byte-identical. Structural win: 2 requests / 4-6 queries
-> 1 request / 2-3 queries (the review measured 530 -> 144 ms). Gate green:
344 pytest, frontend tsc/eslint/knip/prettier/222 vitest.
Second-round /code-review high follow-ups:
- test_feed_cursor.py: GET /api/feed?cursor=<garbage> must return HTTP 400
(not 500) — locks the cursor-error path the refactor split across modules
(feedquery.decode_cursor raises ValueError, get_feed translates to 400).
Plus a no-cursor 200 sanity check. DB lane.
- get_feed: `raise HTTPException(...) from None` so the expected bad-input
ValueError isn't chained as __context__.
DB-lane suite 289 passed / 2 skipped; ruff clean.
Applies the three findings from the /code-review high pass on this stack:
1. to_tsquery_str moved from app.feedquery to app.utils — it's a generic
tsquery sanitizer with no feed dependency; housing it in the feed
read-model coupled the Plex browse module to the feed domain. Both feed
and plex now import it from the neutral util module.
2. feedquery.decode_cursor no longer raises fastapi.HTTPException — it raises
a plain ValueError, and routes/feed.get_feed translates it to the HTTP 400.
The read-model no longer imports fastapi, matching its "no request state"
docstring.
3. Dropped the leading underscore on the helpers that are now a cross-module
public API: feedquery {filtered_query, sort_keys, order_by, keyset_predicate,
encode_cursor, decode_cursor}; plex.query {TS_CONFIG, FIELD_COLUMNS,
field_recheck, split_csv, added_cutoff, meta_filter_conds, apply_meta_filters,
like_escape}; plex.cards {movie_card, show_card, episode_card, leaf_card}.
Genuinely-internal helpers (_channel_url, _saved_expr, _ADDED_WINDOWS,
_lib_key) keep their underscore.
Pure code motion + rename. DB-lane suite 287 passed / 2 skipped; ruff clean.
B-core-3 (2026-08-17 review): decouple two things from router modules.
1. Feed read-model → new app/feedquery.py: the pure query-building and
row-serialization layer (feed_columns, serialize, _filtered_query,
to_tsquery_str, and the keyset-pagination helpers) moves out of
routes/feed.py, which keeps only the route handlers, _feed_params, and
the request models. The three external consumers (playlists, search,
plex/browse) now import from app.feedquery instead of reaching into the
feed router.
2. admin_user sweep: 7 modules imported admin_user via app.routes.admin
(an incidental re-export of app.auth.admin_user). Repoint them all to
the definition in app.auth (audit, channels, config, quota, scheduler,
plex/admin, plex/browse); routes.admin keeps its own import for its use.
Pure code motion — no behaviour change. Full DB-lane suite 287 passed /
2 skipped; ruff clean.
R12 S5 item: the feed's FINISH_MARGIN_SECONDS (10) and the Plex player's
_FINISH_MARGIN_S (30) were flagged as drifted duplicates. They are a
deliberate divergence, not drift — the feed clears a short video's resume
bar, Plex marks long-form items (movies/episodes with credits) watched.
Cross-reference + document both so it reads as intentional.
Layering cleanup, no behaviour change (ruff clean backend+scripts, pytest 193):
- Invert the scheduler->routes dependency: reset_demo_state (+ its demo-seed
helpers) move from routes/admin.py to a new service module app/demo.py.
Both routes/admin.py and scheduler.py now import it from app.demo, so the
scheduler no longer reaches into the routes layer — its lazy import (the
routes<->scheduler cycle workaround) is gone, hoisted to a top-level import.
- Publish the private symbols that were imported across module boundaries (the
underscore convention was lying): feed._serialize -> serialize,
feed._to_tsquery_str -> to_tsquery_str, videos._insert_stubs -> insert_stubs,
videos._stub_from_playlist_item -> stub_from_playlist_item,
worker._uploader_url -> uploader_url, plex._shared._item_or_404 ->
item_or_404. Scoped renames — the 5 other route modules keep their own truly
private _serialize helpers.
- _release_asset moves from routes/downloads.py to app/downloads/service.py as
the public release_asset (asset bookkeeping is a service concern); the routes
call it via service.release_asset.
/code-review high round 1 on S1b — 5 findings, 1 fixed, 4 by-design/deferred:
- FIXED: VideoProgressIn / ItemProgressIn position_seconds & duration_seconds
typed `int` would 422 a raw float currentTime on the 10s progress checkpoint
(the old `int(payload.get(...))` truncated it). Back to `float`, truncated in
the handler; +test. The live frontend floors so this was latent, not live.
By-design / deferred (no change): the 422 detail-array isn't localized by the FE
(filed under R7); PATCH is-not-none (channels/quota, null-safe) vs model_fields_set
(nullable-clear) is a deliberate split; missing-field→422 is the point of the epic;
me/switch_account's numeric-string coercion is more correct and still session-gated.
Gate: ruff clean, pytest 172 green.
- saved_views: CreateViewIn / UpdateViewIn / ReorderViewsIn. PATCH presence via
model_fields_set; filters typed as dict (a non-object is now 422, an explicit
null on update stays a 400 "filters must be an object").
- feed: VideoStateIn(status) / VideoProgressIn(position_seconds,duration_seconds) —
the manual int()+400 in progress becomes a 422 on a non-number; the status
membership 400 and the video 404 are unchanged.
Gate: ruff clean.
Round-1 review finding #4 named only audit.py/plex.py, but the same misplaced
`from app.utils import iso` sat in channels/downloads/feed/notifications too — the
round missed them (ruff enforces no import order here, so the gate was silent).
Move them to alphabetical position for consistency. No behaviour change.
Establish the R6 API-contract infra before the S1b payload:dict migration:
- iso(dt) in utils.py replaces the ~16 hand-rolled `x.isoformat() if x else None`
serializer idioms across the routes.
- get_or_404 / owned_or_404 in routes/_common.py — one 404 + existence-leak policy;
plex's _own_playlist_or_404 and saved_views' _own_view now delegate to it.
- require_write_scope dependency in auth.py collapses the inline has_write_scope→403
checks (channels subscribe/unsubscribe, playlists push/push-plan) that had drifted
into three different messages into one; the conditional delete_playlist guard stays
inline (its message was already canonical).
No response-shape change (iso == isoformat for a present value). Gate: ruff clean,
pytest 163 green (+ test_utils for iso).
The demo account could spend real YouTube quota where the human-only guard was
missing (the channel_detail enrichment already had it):
- channels PATCH: an immediate run_recent_backfill when deep_requested is turned on;
- discovery: the channel-detail enrich on the Discover tab;
- feed get_video_detail: the off-catalog videos.list lookup for a video we don't
store (the DB-hit path stays free; demo now gets a 404 for unknown videos).
Pick any number of channels; they OR together and every other filter applies
across the lot — "the last week's unwatched from these three".
`channelId`/`channelName` are replaced outright by `channelIds: string[]`, no
legacy reader (standing rule: no back-compat unless asked — a saved view written
against the old field simply loses its channel filter). The wire and the share
URL move with it: repeated `channel_ids` params, `?channel=a,b`. Backend ORs via
`Video.channel_id.in_()`; the count endpoint shares the same params object.
Two things the migration exposed:
- SavedViewsWidget fed DB blobs straight into serializers that now index an
array — the FeedFilters type says nothing about what an older build wrote, so
the blob is coerced where it enters. It white-screened the app before this.
- Chips carried channelName to label themselves. Without it they resolve names
from the channel cache, so they now fall back to the id, not a shared
"This channel" that would render N identical chips while the cache loads.
- feed.py _filtered_query: drop the dead 2nd return element (status_expr was
used only internally for WHERE filters; all 3 callers discarded it as _status).
Now returns (query, rank_expr); fixed the stale docstring.
- youtube/client.py: extract _iter_playlist_items() — iter_my_playlist_video_ids
and iter_playlist_items_with_ids were near-identical playlistItems paging loops
(jscpd [173-187]≈[267-281]); both now map over the shared generator.
- sync/videos.py: extract _apply_video_batch() with an on_missing callback —
enrich_pending and refresh_live shared the fetch-map-apply skeleton, differing
only in the query and how they retire rows YouTube omits.
- models.py: add LIVE_OR_UPCOMING = ("live","upcoming"); replace the 4 duplicated
copies (feed.HIDDEN_LIVE, search._LIVE_HIDDEN, videos.py + channels.py inline).
Behavior-neutral. ruff clean on touched files, localdev boots, feed/worker healthy.
- Live-search results stay ephemeral: the discovery-cleanup job now also reclaims un-kept
via_search videos (no watch state / not playlisted / channel not subscribed) after a grace
period (search_grace_days), and POST /api/search/clear discards a given result set 'as if
never added' (drops the user's search-finds + deletes the now-orphaned, un-kept videos).
Admin POST /api/admin/purge-discovery runs it on demand (grace 0).
- Count selector: GET /api/search/youtube gains a 'limit' — the free scrape source pages
through continuations until that many results are gathered (no manual load-more); the API
source stays one ≤50 page (cost).
- Per-user channel blocklist (migration 0034 blocked_channels): block/unblock/list endpoints;
blocked channels' videos are dropped from live search before ingest and hidden from the feed
/ Library / explore / channel page; explore refuses a blocked channel.
- New-first ordering: results you already have (subscribed channel, watched/in-progress/saved/
playlisted) sink below genuinely-new discoveries, preserving relevance within each group.
- Channels manager: prominent channel-name search box and the 'Your tags' chips are now
clickable to filter the table by tag (replacing the hidden per-column DataTable popovers).
Both filter client-side over the status-filtered list; a focus-channel intent seeds the
search, the reset intent clears both.
- Feed 'Your tags' sidebar: count user tags in the facet endpoint (the 'other' category) and
make the widget contextual like language/topic — counts reflect the current filter and
zero-count chips hide (e.g. a source=search view with no tagged channels shows 'no matching
tags' instead of the full static list). EN/HU/DE searchPlaceholder.
Adds the backend for a dedicated channel page + ephemeral browsing of un-subscribed
channels:
- migration 0033: channels.{total_view_count,published_at,banner_url,external_links,
from_explore}, videos.via_explore, explored_channels (per-user, grace-clocked).
- enrich channels with part=brandingSettings (banner/links) + statistics.viewCount +
snippet.publishedAt — no extra quota.
- GET /api/channels/{id}: About detail + this user's relationship; lazy-enriches the new
About fields (published_at sentinel).
- POST /api/channels/{id}/explore (require_human, quota-reserve guarded): records the
exploration, flags the channel ephemeral unless followed, ingests one page of uploads
(via_explore) + enriches; returns next_page_token for load-more.
- feed: per-explorer gate — a via_explore video is visible only to users who explored or
subscribe to its channel, so exploration never leaks into everyone's catalog.
- subscribe = keep: clears from_explore + via_explore on the channel's videos.
- scheduler explore_cleanup job + explore_grace_days config: hard-delete explored-but-
unkept channels and their untouched ephemeral videos after a grace period.
Makes the local search behave more like YouTube's — finding videos by the
uploader's own keywords or the query that surfaced them, not only words in the
title. A DB-generated, weighted search_vector (migration 0032) replaces the
title-only FTS index:
- keywords: the creator's snippet.tags (free — already in the snippet we fetch),
stored on enrich.
- search_terms: distinct live-search queries that surfaced the video (across all
users), appended by the search route — folds YouTube's relevance into local
search (a video YT returned for a query becomes findable by it even without a
title match), the user's own idea.
- description (truncated) for broad recall on the existing catalog.
Weighted title(A) > keywords+queries(B) > description(C) so ts_rank keeps title
hits on top. A plain GIN index on the generated column guarantees index use (no
expression/param matching). Verified on localdev: recall 146->213 for one query;
7 'eurovision' hits via the document but not the title; index scan confirmed.
Two related search improvements:
1) Your own live-search results now belong to your Mine feed. A new per-user
search_finds table (migration 0030) records each video you surface via your
YouTube search (the route inserts them idempotently). The Mine feed becomes
'your non-hidden subscriptions OR your search finds', and the Source filter
now applies in Mine too: organic = subscriptions, search = your search finds,
all = both (default stays organic, so the main feed is unchanged). The shared
Library keeps using the global via_search flag.
2) Feed search ranks by relevance instead of a whole-phrase substring. A custom
unaccent_simple text-search config + GIN index (migration 0031) back a
YouTube-like fuzzy match: word-order-independent, multi-word AND, prefix on
the word being typed, accent-insensitive. A new 'relevance' sort orders by
ts_rank; the channel name still matches as a substring. The rank is scaled to
an integer so the keyset cursor pages it exactly (a raw float4 breaks paging).
_filtered_query returns the rank expr so only the feed list uses it.
Feed text search used plain ILIKE, which is case- but not diacritic-insensitive, so 'tiesto'
missed the many titles spelled 'Tiësto' — a search that ingested ~45 results showed only ~12.
Enable the postgres unaccent extension (migration 0029) and wrap both sides of the title/
channel match in unaccent(), so 'tiesto' now matches 'Tiësto'. Applies to feed, count and
facets alike.
Replace the binary 'show search-discovered' toggle with a 3-way Source selector in the
Library toolbar, so users can also see ONLY search-discovered videos — not just hide or
mix them. Backend: feed param library_source = organic (default, hides via_search) | all
(both) | search (only via_search), applied in scope=all. Strings in HU/EN/DE.
Add a live YouTube search that materialises results into the shared catalog so
they render with the normal feed cards + in-app player and gain per-user state.
- YouTubeClient.search_videos(): search.list (100 units), embeddable-only, returns
flat stubs + nextPageToken; surfaces liveBroadcastContent for live filtering.
- routes/search.py GET /api/search/youtube: require_human + per-user daily cap
(search_daily_limit_per_user, default 70) + can_spend pre-check (429 on either);
drops live/upcoming, upserts channel stubs (channels.list) + video stubs, enriches
(videos.list), runs the youtube.com/shorts probe, then excludes Shorts/live and
returns feed cards in relevance order with the YouTube pageToken as the cursor.
- Provenance: videos.via_search / channels.from_search (migration 0028) flag
search-discovered rows; the feed hides them from the Library (scope=all) by default
via exclude_search_discovered, leaving the Mine feed untouched.
- quota.actions_today() counts a user's per-action events today for the cap; only the
search.list call is attributed VIDEOS_SEARCH so the counter is exactly 1 per search.
- feed_columns() is the single feed-row projection, consumed by both /feed and the
playlist-detail query (was copy-pasted, drift risk on every new column).
- _replace_items_from_youtube() dedups the YouTube playlist-mirror block shared by
sync_user_playlists and repull_playlist.
- _next_playlist_position/_next_item_position/_add_item replace the append-position
idiom (x4) and the exists-then-insert blocks (x2).
Replace deep-OFFSET paging (which scans-and-discards skipped rows on the
233k-row catalog and shifts/duplicates rows as the scheduler ingests mid-
scroll) with stable keyset pagination over an opaque cursor.
A single sort registry drives both the ORDER BY and a generic, NULL-aware
keyset WHERE so they can never drift, with Video.id as the unique final
tiebreaker. Covers every sort: newest/oldest, views, duration, title,
subscribers (joined), priority (coalesced multi-key) and the seeded
shuffle. The response now returns next_cursor instead of offset.
The quota-attribution action keys were inconsistent (mixed verbs, ad-hoc names,
English-only labels). Introduce a QuotaAction constants holder (one source of
truth), rename every attribution call site to it, and add migration 0020 to
rename historical quota_events.action values. The display label now resolves
from i18n (quotaActions.<key>, EN/HU/DE) instead of a hard-coded English map.
Scheduler job ids and progress phase labels are a separate namespace and are
left untouched.
Same fix as the subscribe reconcile, applied to the remaining endpoints that
wrapped a YouTubeError in a 502: video lookup (feed) and playlist refresh,
push-plan, sync, and delete. A 502 is treated by the client as a transient
"connection lost" blip, so a real, explainable YouTube failure showed the wrong
message. 422 carries the clear detail to the error dialog instead.
Add a daily maintenance/validation job that detects videos which can't be played
anywhere and retires them safely. Two phases: re-check already-flagged videos
(recover if available again, else hard-delete once the grace period elapses,
cascading to states/playlist items), and a rolling re-validation of the
least-recently-checked currently-available videos that flags newly-unplayable
ones (hidden from the feed immediately via unavailable_since).
Detection is ~free: a video missing from the videos.list response is
deleted-or-private; an `upcoming` premiere >2 days past its scheduled start that
never went live is abandoned. A still-live broadcast is kept (legit 24/7 stream).
Enrichment now also fetches part=status to populate the status columns. Grace is
7 days for removed videos, none for abandoned. Before deleting, affected users
get one batched notification (never per-video). Interval is admin-tunable via the
Scheduler dashboard; batch size and grace are config. Quota-attributed to the
system and bounded by the same backfill reserve as the other jobs.
- Channel-row tags show only what's attached; a '+' opens a per-channel picker. Clicking a tag
filters the feed by it; a 'Your tags' sidebar widget makes that filter visible/clearable.
- Tag manager dialog (add/rename/delete; delete only confirms when the tag is in use), reachable
from the Channel manager and the feed sidebar; hovering a tag's count lists its channels, each a
link that focuses it in the Channel manager (DataTable gains an external filter input).
- Video cards get a reset action that clears all watch state (incl. an in-progress position).
- api.req() now raises the error dialog on 5xx and 400/409/422 with the server's reason.
The in-app player marks a video watched when it reaches the end, firing
POST /videos/{id}/state at the same instant as a progress checkpoint. The
progress endpoint deletes the 'new' video_states row near the end, so the
concurrent state UPDATE matched 0 rows and raised StaleDataError -> 500. The
watched status was never persisted (and Feed swallowed the error), so the
video stayed in the unwatched feed even after a refresh.
- set_video_state: atomic INSERT ... ON CONFLICT (uq_user_video) DO UPDATE for
watched/hidden, immune to a concurrent delete; tolerate StaleDataError on the
'new' branch and in the progress endpoint.
- PlayerModal: skip the redundant near-end progress checkpoint when we just
auto-marked watched, removing the self-inflicted race.
- Feed: drop the optimistic override if the server rejects the change, so a
failed request can't leave a card phantom-hidden.
1) Move the Show view filter (Unwatched/In progress/All/Watched/Hidden) up into the
toolbar chip row as its own single-select group, divided from the content-type chips;
removed the 'show' sidebar widget (sidebar now = Upload date / Language / Topic).
2) Rework ordering like the Playlists page: a sort-key dropdown (Date, Popular, Duration,
Name, Channel subscribers, Channel priority, Surprise me) + a single asc/desc arrow
toggle, instead of separate directional entries. 'Most viewed' is now 'Popular'.
Backend gains the missing directions (views_asc, title_desc, subscribers_asc,
priority_asc); the frontend maps (key, dir) -> the backend sort string, so FeedFilters
is unchanged. Trilingual (feed.sortKey.*, feed.dirAsc/dirDesc). Feed.tsx normalized to LF.
peter