perf(jobs): partial indexes for the enrich/maintenance scanners + boolean cleanup (E2 S6, PERF-17/18/3)

Two scheduler jobs full-seq-scanned the 257k-row videos table every run:
- PERF-17 enrich_pending (`WHERE enriched_at IS NULL LIMIT N`, every few minutes):
  44,680 buffers even with 0 pending -> a partial ix_videos_enrich_pending makes
  it a 1-buffer index scan (empty when nothing pending).
- PERF-18 maintenance revalidate (`WHERE unavailable_since IS NULL ORDER BY
  last_checked_at NULLS FIRST LIMIT N`, ~300 pages/run): 44,754 buffers + top-N
  sort -> a partial ix_videos_recheck_order in that exact order = a 42-buffer
  index scan. The order_by is rewritten to `asc().nulls_first()` (equivalent to
  the old `(IS NULL) DESC, ASC`) so it rides the index; the unused plain
  ix_videos_last_checked_at is dropped.

PERF-3: drop the never-picked boolean btrees (is_short, via_explore) and slim the
useful ones to partials over the queried subset (via_search=true, NOT
shorts_probed, unavailable_since IS NOT NULL) — less write amplification on the
biggest table, same lookups. Model: those 6 columns lose their inline index=True
(the partials are migration-defined, like the search_vector GIN); live_status
left alone.

Measured on the prod-sized localdev reseed: enrich 44,680 -> 1 buffer (25.9 ->
0.02 ms), maintenance 44,754 -> 42 buffers (35.5 -> 0.17 ms); all 11 CONCURRENTLY
index ops on 257k rows in 0.75s. Gate green (346 pytest incl. drift + downgrade
round-trip).
This commit is contained in:
2026-08-18 17:01:13 +02:00
parent 8b8b0ca5ab
commit 0ed4ac6958
3 changed files with 107 additions and 10 deletions
+4 -4
View File
@@ -194,10 +194,10 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
db.execute(
select(Video)
.where(Video.unavailable_since.is_(None))
.order_by(
Video.last_checked_at.is_(None).desc(), # never-checked first
Video.last_checked_at.asc(),
)
# NULLS FIRST = never-checked first, then least-recently-checked; equivalent to the old
# `(last_checked_at IS NULL) DESC, last_checked_at ASC` but in the exact shape of the
# partial ix_videos_recheck_order (migration 0069, PERF-18) so it rides that index.
.order_by(Video.last_checked_at.asc().nulls_first())
.limit(min(50, batch - checked))
)
.scalars()