peter
8c08ea3133
A nullable published_at forced the feed keyset into a non-sargable `... OR published_at IS NULL` disjunction: with the 0067 index the planner still applied it as a FILTER and scanned from the top of the index, so a deep `newest` page discarded every earlier row (O(depth)). - Migration 0068: backfill any NULLs (0 in prod) -> SET NOT NULL + DEFAULT now(). - Model: Video.published_at is now non-Optional, nullable=False, server_default now(). - feedquery: the `pub` key is nullable=False, and keyset_predicate is rebuilt recursively so a non-nullable leading key gets a non-strict range bound (`published_at <= v`) as an AND conjunct — an index range Cond instead of a filter. Nullable keys keep the NULL-aware OR form (a `<= v` bound would drop their NULL group). Equivalence proven by test_feed_keyset (all 13 sorts, cursor-walk == unpaged). - Ingest paths (stub_from_playlist_item, apply_video_details, playlists sync) fall back to now() so they never write an explicit NULL. Measured on the prod-sized localdev reseed at a ~50k-deep cursor (0067 index + prod planner): OLD `Rows Removed by Filter: 50001`, 25,687 buffers / 9.16 ms -> NEW `Index Cond: published_at <= v`, 1 row filtered, 47 buffers / 0.037 ms. SET NOT NULL on 257k rows: 0.32s. Gate green (334 pytest).
37 lines
1.7 KiB
Python
37 lines
1.7 KiB
Python
"""videos.published_at NOT NULL DEFAULT now() (PERF-15)
|
|
|
|
Every video in practice carries a publish date (0 NULLs in the 257k-row prod-sized reseed), but the
|
|
column was nullable, which forced the feed's keyset predicate into a non-sargable
|
|
`... OR published_at IS NULL` disjunction — blocking the ix_videos_published_at_desc range so deep
|
|
`newest` pages scanned from the top of the index and filtered (51.8k buffers). Making it NOT NULL lets
|
|
the keyset drop that disjunction and take a `published_at <=` range bound (deep page -> ~0.5k buffers).
|
|
|
|
server_default now() is the safety net for the rare date-less ingest path (the enrich overwrites it with
|
|
the real date); the app-side ingest paths also fall back to now() so they never write an explicit NULL.
|
|
|
|
Note: `SET NOT NULL` validates with a brief ACCESS EXCLUSIVE lock + one table scan (it cannot run
|
|
CONCURRENTLY). On the prod row count (~235k) that is sub-second; still, it is a short write-blocking lock
|
|
rather than the ~0 of 0067's CONCURRENTLY index build.
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0068_published_at_not_null"
|
|
down_revision: Union[str, None] = "0067_feed_sort_indexes"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
# Backfill any stray NULLs first so SET NOT NULL can't fail (0 in prod, but be safe).
|
|
op.execute("UPDATE videos SET published_at = now() WHERE published_at IS NULL")
|
|
op.alter_column(
|
|
"videos", "published_at", nullable=False, server_default=sa.text("now()")
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.alter_column("videos", "published_at", nullable=True, server_default=None)
|