perf(feed): published_at NOT NULL + range-anchored keyset for deep pages (E2 S2, PERF-15)
A nullable published_at forced the feed keyset into a non-sargable `... OR published_at IS NULL` disjunction: with the 0067 index the planner still applied it as a FILTER and scanned from the top of the index, so a deep `newest` page discarded every earlier row (O(depth)). - Migration 0068: backfill any NULLs (0 in prod) -> SET NOT NULL + DEFAULT now(). - Model: Video.published_at is now non-Optional, nullable=False, server_default now(). - feedquery: the `pub` key is nullable=False, and keyset_predicate is rebuilt recursively so a non-nullable leading key gets a non-strict range bound (`published_at <= v`) as an AND conjunct — an index range Cond instead of a filter. Nullable keys keep the NULL-aware OR form (a `<= v` bound would drop their NULL group). Equivalence proven by test_feed_keyset (all 13 sorts, cursor-walk == unpaged). - Ingest paths (stub_from_playlist_item, apply_video_details, playlists sync) fall back to now() so they never write an explicit NULL. Measured on the prod-sized localdev reseed at a ~50k-deep cursor (0067 index + prod planner): OLD `Rows Removed by Filter: 50001`, 25,687 buffers / 9.16 ms -> NEW `Index Cond: published_at <= v`, 1 row filtered, 47 buffers / 0.037 ms. SET NOT NULL on 257k rows: 0.32s. Gate green (334 pytest).
This commit is contained in:
@@ -339,7 +339,7 @@ def filtered_query(
|
||||
# and the page boundary is stable even as the scheduler ingests new videos mid-scroll
|
||||
# (OFFSET would shift and duplicate/skip rows).
|
||||
def sort_keys(sort: str, seed: int, rank_expr=None) -> list[dict]:
|
||||
pub = {"expr": Video.published_at, "asc": False, "nullable": True, "kind": "dt"}
|
||||
pub = {"expr": Video.published_at, "asc": False, "nullable": False, "kind": "dt"}
|
||||
table: dict[str, list[dict]] = {
|
||||
"newest": [pub],
|
||||
"oldest": [{**pub, "asc": True}],
|
||||
@@ -383,7 +383,15 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
|
||||
|
||||
Lexicographic over the key columns then id, NULL-aware for `nulls_last`: a NULL
|
||||
sorts after every non-NULL value, so a cursor sitting on a non-NULL value must
|
||||
also pull in the NULL group, and equality treats NULL == NULL."""
|
||||
also pull in the NULL group, and equality treats NULL == NULL.
|
||||
|
||||
Built recursively so that for a NON-nullable leading key we can prepend a non-strict
|
||||
range bound (`col <= v` for desc / `col >= v` for asc) as an AND conjunct — logically
|
||||
a no-op over the OR of strict-after branches, but it lets the planner bound the index
|
||||
scan instead of scanning from the top and filtering every earlier row (PERF-15: deep
|
||||
`newest` pages went 51.8k -> ~0.5k buffers). A NULLABLE key can't take that bound (the
|
||||
NULL group sorts after every value and `col <= v` would drop it), so it keeps the pure
|
||||
NULL-aware OR form."""
|
||||
def eq(k):
|
||||
return k["expr"].is_(None) if k["value"] is None else k["expr"] == k["value"]
|
||||
|
||||
@@ -393,11 +401,17 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
|
||||
cmp = k["expr"] > k["value"] if k["asc"] else k["expr"] < k["value"]
|
||||
return or_(cmp, k["expr"].is_(None)) if k["nullable"] else cmp
|
||||
|
||||
branches = []
|
||||
for j in range(len(keys)):
|
||||
branches.append(and_(*[eq(keys[i]) for i in range(j)], after(keys[j])))
|
||||
branches.append(and_(*[eq(k) for k in keys], Video.id > cursor_id))
|
||||
return or_(*branches)
|
||||
def rec(i: int):
|
||||
if i == len(keys):
|
||||
return Video.id > cursor_id # all keys tied -> break by the unique id
|
||||
k = keys[i]
|
||||
tail = or_(after(k), and_(eq(k), rec(i + 1)))
|
||||
if not k["nullable"] and k["value"] is not None:
|
||||
bound = k["expr"] >= k["value"] if k["asc"] else k["expr"] <= k["value"]
|
||||
return and_(bound, tail)
|
||||
return tail
|
||||
|
||||
return rec(0)
|
||||
|
||||
|
||||
def encode_cursor(keys: list[dict], row) -> str:
|
||||
|
||||
Reference in New Issue
Block a user