perf(jobs): partial indexes for the enrich/maintenance scanners + boolean cleanup (E2 S6, PERF-17/18/3)
Two scheduler jobs full-seq-scanned the 257k-row videos table every run: - PERF-17 enrich_pending (`WHERE enriched_at IS NULL LIMIT N`, every few minutes): 44,680 buffers even with 0 pending -> a partial ix_videos_enrich_pending makes it a 1-buffer index scan (empty when nothing pending). - PERF-18 maintenance revalidate (`WHERE unavailable_since IS NULL ORDER BY last_checked_at NULLS FIRST LIMIT N`, ~300 pages/run): 44,754 buffers + top-N sort -> a partial ix_videos_recheck_order in that exact order = a 42-buffer index scan. The order_by is rewritten to `asc().nulls_first()` (equivalent to the old `(IS NULL) DESC, ASC`) so it rides the index; the unused plain ix_videos_last_checked_at is dropped. PERF-3: drop the never-picked boolean btrees (is_short, via_explore) and slim the useful ones to partials over the queried subset (via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — less write amplification on the biggest table, same lookups. Model: those 6 columns lose their inline index=True (the partials are migration-defined, like the search_vector GIN); live_status left alone. Measured on the prod-sized localdev reseed: enrich 44,680 -> 1 buffer (25.9 -> 0.02 ms), maintenance 44,754 -> 42 buffers (35.5 -> 0.17 ms); all 11 CONCURRENTLY index ops on 257k rows in 0.75s. Gate green (346 pytest incl. drift + downgrade round-trip).
This commit is contained in:
@@ -194,10 +194,10 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
|
||||
db.execute(
|
||||
select(Video)
|
||||
.where(Video.unavailable_since.is_(None))
|
||||
.order_by(
|
||||
Video.last_checked_at.is_(None).desc(), # never-checked first
|
||||
Video.last_checked_at.asc(),
|
||||
)
|
||||
# NULLS FIRST = never-checked first, then least-recently-checked; equivalent to the old
|
||||
# `(last_checked_at IS NULL) DESC, last_checked_at ASC` but in the exact shape of the
|
||||
# partial ix_videos_recheck_order (migration 0069, PERF-18) so it rides that index.
|
||||
.order_by(Video.last_checked_at.asc().nulls_first())
|
||||
.limit(min(50, batch - checked))
|
||||
)
|
||||
.scalars()
|
||||
|
||||
Reference in New Issue
Block a user