perf(feed): published_at NOT NULL + range-anchored keyset for deep pages (E2 S2, PERF-15)

A nullable published_at forced the feed keyset into a non-sargable
`... OR published_at IS NULL` disjunction: with the 0067 index the planner still
applied it as a FILTER and scanned from the top of the index, so a deep `newest`
page discarded every earlier row (O(depth)).

- Migration 0068: backfill any NULLs (0 in prod) -> SET NOT NULL + DEFAULT now().
- Model: Video.published_at is now non-Optional, nullable=False, server_default now().
- feedquery: the `pub` key is nullable=False, and keyset_predicate is rebuilt
  recursively so a non-nullable leading key gets a non-strict range bound
  (`published_at <= v`) as an AND conjunct — an index range Cond instead of a
  filter. Nullable keys keep the NULL-aware OR form (a `<= v` bound would drop
  their NULL group). Equivalence proven by test_feed_keyset (all 13 sorts,
  cursor-walk == unpaged).
- Ingest paths (stub_from_playlist_item, apply_video_details, playlists sync)
  fall back to now() so they never write an explicit NULL.

Measured on the prod-sized localdev reseed at a ~50k-deep cursor (0067 index +
prod planner): OLD `Rows Removed by Filter: 50001`, 25,687 buffers / 9.16 ms ->
NEW `Index Cond: published_at <= v`, 1 row filtered, 47 buffers / 0.037 ms. SET
NOT NULL on 257k rows: 0.32s. Gate green (334 pytest).
This commit is contained in:
2026-08-18 04:30:07 +02:00
parent 3af3799b25
commit 8c08ea3133
6 changed files with 78 additions and 15 deletions
+3 -1
View File
@@ -90,7 +90,9 @@ def _ensure_videos(db: Session, yt: YouTubeClient, video_ids: list[str]) -> None
if not vid or not cid or vid in seen:
continue
seen.add(vid)
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")))
# published_at is NOT NULL (0068); apply_video_details re-fills a missing date, but seed a now()
# fallback here too so the constructor never carries an explicit NULL.
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")) or now)
apply_video_details(v, it)
v.enriched_at = now
db.add(v)
+6 -2
View File
@@ -90,7 +90,10 @@ def stub_from_playlist_item(item: dict, channel_id: str) -> dict | None:
video_id = content.get("videoId")
if not video_id:
return None
published = parse_dt(content.get("videoPublishedAt") or snippet.get("publishedAt"))
# published_at is NOT NULL (migration 0068); fall back to now() for the rare item that carries no
# date at all, so the bulk stub insert never writes a NULL. The enrich pass replaces it if the
# video's own snippet later yields a real publish date.
published = parse_dt(content.get("videoPublishedAt") or snippet.get("publishedAt")) or _now()
return {
"id": video_id,
"channel_id": channel_id,
@@ -243,7 +246,8 @@ def apply_video_details(video: Video, item: dict) -> None:
video.title = normalize_title(raw_title)
video.description = snippet.get("description")
if not video.published_at:
video.published_at = parse_dt(snippet.get("publishedAt"))
# NOT NULL (0068): a fresh insert with no date yet falls back to now() (enrich normally has one).
video.published_at = parse_dt(snippet.get("publishedAt")) or _now()
video.thumbnail_url = best_thumbnail(snippet.get("thumbnails")) or video.thumbnail_url
video.duration_seconds = parse_iso8601_duration(content.get("duration"))
video.view_count = int(stats["viewCount"]) if stats.get("viewCount") else None