A subscription pulled in from YouTube left an explored channel flagged as an exploration's, so its
later videos came in hidden from the Library with their watched marks waiting for good; a subscribe
landing while an exploration's videos came in left those flagged too. One step, settle_explored,
now holds the rule by who follows the channel now: the subscribe here, the subscription import, the
watched-channel videos and the explore page all end with it. Migration 0116 clears what the gap
left behind.
The list's "may still grow" count no longer counts a channel it already shows from a stub row, and
the comments on the cleanup's keep rule and on the in-memory daily shares say what they do.
- A channel someone follows: the videos its exploration brings in (the list's
dialog, the channel page's explore) come in as the catalog's own, not
flagged via_explore, which subscribing would never clear and which kept
their watched marks waiting for good.
- The explore cleanup keeps the row of a channel a Takeout history shows
watched (it still reclaims the videos), so a looked-at channel no longer
drops off the list.
- The Shorts job takes up only users whose list still has lengths to ask or
channels to name, by each user's own follows, blocks and dismissals; a
naming that failed is tried again, and the list says it is not complete.
- The length lookups have a daily share of their own: a large history no
longer leaves the export's Watch later file waiting a day.
- Only a channel of the user's own history can be put out of the list.
- The dialog's videos have a query root of their own: the list's
invalidations no longer re-ask (and re-explore) under an open dialog.
The import asks YouTube how long the watched videos of the candidate
channels are (five or more watched, none the user follows, blocked, put
out or owns), keeps the length on the Takeout row and names the channels
that make the list: five videos of five minutes or more. The shorts job
asks what an import could not. A channel's counted videos come in as an
exploration's when its list is opened, ready to play; a channel can be put
out of the list and back.
- An ad-hoc request made while its target's run is under way is the run after it: the scheduler
dropped a second fire of a running id, so a second Takeout file was never imported and its key
stayed queued until a restart.
- Un-marking or resetting a video is remembered (video_unmarks): an import of an older YouTube view
no longer marks it watched again; un-marks take the user's Takeout lock.
- One user's failure in the Shorts job's Takeout pass is theirs: the job keeps its own result, the
others are settled, and a failed lookup spends none of the daily share.
- The upload reads the user's id before giving the read connection back.
- A watched row waits on a discovery video (search or explore) until the catalog keeps it, so a
mark never pins it against the cleanup jobs.
- The history is decoded a window at a time, a Watch later file over 2 MiB is refused unread, and
two uploads at once across all users (takeout_server_busy).
- A Watch later list goes in as one presence read and one insert.
- Uploads: one at a time per user and ten an hour, each refused only after its body arrived;
the history read an entry at a time (32 MiB cap), a Watch later list capped at YouTube's 5000,
a user's kept videos at 200k; any unreadable shape is a 422, not a 500.
- Quota: Watch later lookups stop at the backfill reserve and at a per-user daily share; what is
not asked about waits, known videos still go in, and the shorts job retries the rest.
- Staging and settling share one per-user advisory lock (no deadlock); a run settles only the
rows it read, so a row staged meanwhile is never called gone.
- A failed run's notice names a translated failure, never the upstream's text.
- The status says when an import is queued; the page watches it to its end even when left, then
reads the feed, its counts and the playlists again.
- Settings: one live region, the zone kept (unavailable) while a run goes, focus handed back,
a missed drop kept from the browser, an oversize file refused before it is sent.
- Scheduler and run lists show Takeout runs by the importing user.
A watched video the catalog did not know at import time waits in `takeout_items`. The Shorts job now
settles those rows right after its probe (the only place a video's Short-ness becomes known), every
user's, a commit each: `reader.takeout.apply_waiting`, its count in the job's result.
`GET /api/takeout` is what the settings page reads: the user's last import run (`jobruns.last_run`)
and the kept videos by kind and outcome, "waiting" for one not known yet.
The YouTube API reads neither a user's watch history nor their Watch later list, so the Takeout
export is the only way in. `POST /api/takeout` takes one file of it (watch-history.json or
Watch later-videos.csv, told apart by content), keeps its videos in `takeout_items` and queues the
user's `takeout_import` run, whose outcome reaches their inbox.
The run marks watched the known videos the Shorts probe has cleared, at the Takeout time; a Short
is not marked, and a state the user set here (watched, hidden, or changed after that time) wins, by
the upsert's own condition. A video the catalog does not know yet waits with no outcome. Watch later
videos are fetched if unknown and appended oldest first.
An ad-hoc run can now ask for a completion notice (`run_ad_hoc(..., notify=True)`).
Channels, search and sync do no database work in their routes any more: reads go through
catalog/queries, writes through catalog/service, and a live search's ingest is catalog/search.
What a catalog route needs from Reader (a playlist's channels, a user's tag links, the feed
projection of results) it asks reader/queries and reader/service. The scheduler and the runs page
read channel titles and the work queue through catalog/queries. Catalog is fenced below its HTTP
layer: its owner filters go through userscope, it reads no other unit's tables, and its ingest
jobs commit only where they are named.
Channel, Subscription, Video, SearchFind, ExploredChannel and BlockedChannel (and LIVE_OR_UPCOMING)
leave app/models.py for app/catalog/models.py; app.tables registers them. The User↔Subscription
relationship names its other side as a type only. No schema change.
Feed, playlists, saved views, tags and the Home pins do no database work in their routes any more:
reads go through reader/queries (and feedquery for a feed page, its counts and facets), writes
through reader/service and discover/pins, which flush; the handlers commit. Reader is fenced below
its HTTP layer too: its owner filters go through userscope, and the catalog's jobs ask Reader
instead of joining its tables (the auto-tagger's system tags, the cleanup jobs' keep rule, the
maintenance notice). A playlist sync's catalog ingest moves to sync/videos.
peter