peter
25aed6635f
Every video carries a random number drawn once (videos.shuffle_key, migration 0118, indexed with id). A seed picks a point on that ring: the feed's Shuffle sort pages along it and wraps, and the Home surprise reads a 500-video window of it and shuffles that window by the seed, so no video is favoured for sitting after a wide gap. The md5(id || seed) order read and sorted the whole feed scope on every call: ~350 ms on prod per surprise, 381-767 ms per Shuffle page on localdev, now 14 ms and 4 ms.
41 lines
1.8 KiB
Python
41 lines
1.8 KiB
Python
"""videos: a random key per video, so a seeded shuffle walks an index instead of sorting the scope
|
|
|
|
Revision ID: 0118_videos_shuffle_key
|
|
Revises: 0117_videos_length_sort_indexes
|
|
Create Date: 2026-11-07
|
|
|
|
The feed's Shuffle sort and the Home surprise ordered the whole feed scope by md5(id || seed) on every
|
|
call: ~350 ms on prod for a scope of ~213k videos, and a Shuffle page deeper in the list cost more
|
|
(767 ms on page 7 on localdev, against 4 ms for Newest). Each video now carries a random number drawn
|
|
once (`Video.shuffle_key`); a seed picks a point on that ring and the draw reads the index from there
|
|
(`feedquery.shuffle_walk`).
|
|
|
|
One ADD COLUMN with a volatile default rewrites the table and fills every row with its own value. On
|
|
localdev (278k videos) that took 28 s, against 84 s for an UPDATE of a nullable column, so the rewrite
|
|
it is: the table is locked for that time, and a deploy has the app down while it migrates anyway.
|
|
The index is (shuffle_key, id): the walk's keyset cursor compares the pair.
|
|
"""
|
|
|
|
from alembic import op
|
|
|
|
revision: str = "0118_videos_shuffle_key"
|
|
down_revision: str | None = "0117_videos_length_sort_indexes"
|
|
branch_labels: str | None = None
|
|
depends_on: str | None = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.execute(
|
|
"ALTER TABLE videos ADD COLUMN IF NOT EXISTS shuffle_key double precision "
|
|
"NOT NULL DEFAULT random()"
|
|
)
|
|
with op.get_context().autocommit_block():
|
|
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
|
|
op.execute("CREATE INDEX CONCURRENTLY ix_videos_shuffle_key ON videos (shuffle_key, id)")
|
|
|
|
|
|
def downgrade() -> None:
|
|
with op.get_context().autocommit_block():
|
|
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
|
|
op.execute("ALTER TABLE videos DROP COLUMN IF EXISTS shuffle_key")
|