Compare commits

...
44 Commits
Author SHA1 Message Date
peter 35266fd7e8 release: 0.112.1 (bug/authlib-cve) 2026-10-28 00:08:40 +01:00
peter 6211e15611 chore(release): 0.112.1 2026-10-27 23:59:09 +01:00
peter 011dea8eb7 fix(deps): Authlib 1.7.2 -> 1.9.1 for CVE-2026-96760 and CVE-2026-114616
One pin bumped in place; pip check passes and pip-audit finds nothing.
2026-10-27 21:46:16 +01:00
peter 4aff19a9a4 Merge branch 'feature/e45-related-videos' 2026-10-25 17:35:21 +01:00
peter dc50b856fd chore(security): gitleaks knows the public InnerTube key and a fixture's page cursors are no secrets 2026-10-25 17:35:20 +01:00
peter ee6e37f032 release: 0.112.0 (feature/e45-related-videos) 2026-10-25 17:31:11 +01:00
peter 4c2fbf9b8d chore(release): 0.112.0 2026-10-25 17:22:29 +01:00
peter 9d6d7525e5 fix(feed): a live search whose first page failed offers Retry where trying again can help
Search YouTube again with the same term read the failure from the cache and sent nothing, so a busy or unreachable YouTube left the search on its error until the term changed. The first page's failure is now the error state with Retry, offered only where a retry can answer (not the day's cap, the feature turned off or the shared budget).
2026-10-25 14:05:07 +01:00
peter ee4a5b46f8 refactor(player): the watched toggle is stamped at each answer, with no hold while a write is out
The hold changed nothing a user could see (each answer stamps the toggle right after its own echo), and it would have outranked a mark made elsewhere after the click while the write was still out.
2026-10-25 13:56:40 +01:00
peter 0bc1024287 fix(player): marks are weighed when answered, lit and pressed hold under the pointer, and Restart is one glyph
The watched toggle weighed its own click at click time against marks stamped when the server answered, so the late answer of its previous click overruled the newer one; it now holds its word while any of its writes is on the way, is stamped at the answer, and drops a refused write. The hosts return the write. In a live search the feed's own answered writes stand down over the rows, which carry the marks, so a newer mark from the related list reaches a card the feed touched earlier.

A lit mode cycle got the hover layer, and a variant's hover colour outranked the pressed and lit colours: the state layer stands down on [data-lit] and those colours are important. The menus' Play from the beginning wears RotateCcw like the card's Restart, Reset wears Eraser and Redownload the retry arrow (DESIGN decision log). Tests pin the toggle's orders, the Watch later mark at its call sites, the live-search overrides, the hover exemptions and the menu glyphs.
2026-10-25 13:51:26 +01:00
peter 13f65b4844 test(db): lock probes read only their own database's locks
pg_locks is the whole server's: under the parallel lane a worker's own advisory claim on the same playlist id, or its own lock wait, was counted by another worker's probe (a leak reported where there was none, a commit made before the second tab blocked).
2026-10-25 13:51:26 +01:00
peter 2e1934689f fix(search): a search that never reached YouTube gets its slot back, and a busy one waits a bounded time
The slot was kept as soon as search.list was about to be called, so a search refused before it went out (no connection, no token) cost the user a slot for nothing. It is kept now only once the client recorded the call's units, or the request went out and its answer never came (YouTube may have counted it): YouTubeError says whether a transport failure was sent, and the client counts what it spent.

The process lock around the budget check and the call is waited for at most ten seconds; past that the search is refused as busy (search_busy, a retry can help) and gets its slot back, instead of parking a request worker behind every slow call before it. The lock test now pins one hold around both steps, and the busy test fails on a wait without a bound.
2026-10-25 13:51:25 +01:00
peter 84db570951 fix(player): marks give way to newer lists, Watch later items mark, and auto-advance has its own glyphs
A session's marks are weighed per field against when each page of a never-refetched list was read:
a page read before the mark (a Show more / Load more on its way) takes it, one read after keeps the
server's word. The live search lays them over its rows too, not only the related list.

Items of the built-in Watch later list ARE the bookmark, so adding, removing and moving them go
through features/video/videoWrites and mark the video; the contract test covers those writes and
catches a write that is named without being called on the spot.

The player's watched toggle follows a mark confirmed elsewhere since it opened (the related row's
menu, a notice's Undo); Restart on the related row after a description link goes back to it.
The file viewer tells the help sheet its keys wait under a layer; refusalReason replaces the last
two open-coded reads.

Auto-advance wears list glyphs of its own (the skip arrows are Previous / Next beside it), and the
auto-advance and loop cycles are `lit`, not toggles: no aria-pressed, the name says the mode. Both
places render from one list of controls, so the docked tooltips name the mode too. The English cap
message and admin hint quote "Show more"; the hint says refunds are limited.
2026-10-25 04:01:43 +01:00
peter b25179c06a fix(search): a paid search keeps its slot, refunds are few, and the budget check holds the call
An API-sourced live search that reached search.list has spent 100 units (the client records them
before it reads the status), so it keeps its daily slot: refunding it let a bogus cursor spend the
shared budget without ever counting against the cap. A failure before the call still gives the slot
back.

Every refund is rationed too (5 an hour per user and list): a cursor is a free string, and a request
built to fail would otherwise scrape YouTube from the server without end.

The API source asks the budget again right before search.list, under one process lock with the
call: checked apart, searches sent together at the last 100 units all passed and all spent.
2026-10-25 04:01:42 +01:00
peter 620dd8178f feat(player): the side panel's list controls and watch toggle are glyphs on one line
The queue row (previous, count, next, auto-advance, loop) wrapped onto three lines in the
narrow side panel; glyph-only, it fits one. Mark watched joins the action icons as a toggle
(check / double check). Every control keeps its full name and shortcut as its tooltip and
accessible name; the docked bar keeps its labels. The related switch moves up to the toolbar
rung (12px), and its second side reads just "Trending".
2026-10-25 02:43:01 +01:00
peter 5d009964d8 fix(i18n): the related cap message reads right at any limit
"1 requests" at a limit of 1: the number stands alone now, and the
sentence after it says what counts.
2026-10-25 02:16:20 +01:00
peter 4b5b66343d fix(player): related marks survive, Restart restarts, and refused pages offer nothing to click
- Every state and Watch-later write goes through features/video/videoWrites,
  which marks the never-refetched lists (related lists, live searches) once
  the server has it; a contract test holds it. The Notification Center's
  undo, the feed's reset and the Watch-later toggles skipped it before.
- Confirmed marks are also kept apart (qk.videoMarks) and laid over the
  related rows: a Show more page landing later rebuilt them from the pages
  it read when it started, and dropped a mark made meanwhile.
- Restart on a related video starts at 0 (the session's stored position
  won before), and on the one already playing it seeks.
- The watched toggle remembers each video's mark: Back showed the opened
  video's launch-time state after a related one was marked.
- The help sheet leaves out the player's keys while a layer above it has
  them waiting.
- After a refusal a retry cannot answer, neither Show more (related) nor
  Load more (live search) is offered; a failed live-search page keeps the
  results already read and says why under them.
- The related cap message counts requests, as the cap does.
2026-10-25 02:50:34 +02:00
peter ed51f5be7e fix(search): the API-sourced live search takes its slot under the lock too
Checked first and logged after search.list, requests sent together at the
last free slot all passed, and each spent 100 units of the shared budget.
Every live search now claims its slot (quota.claim_action) whatever the
source, and gives it back when it showed nothing. The cap counts the
zero-unit claims only, so search.list's own spend event is accounting,
not a second slot. log_action, the unlocked writer, goes.

Tests: the API source reads one page per request, is refused before the
call when the budget cannot pay, counts one slot per search and refunds a
failed one; the lock test now proves the count happens after the lock.
2026-10-25 02:50:34 +02:00
peter 005784e291 test(player): a related video's end leaves an auto-advancing queue where it was 2026-10-25 00:42:34 +02:00
peter b5045165c0 fix(player): a related video played is tracked, and its list is honest about its state
A related video played in place is the player's item now: its title and its own watched /
playlist / download controls, its progress checkpointed and resumed, marked watched at its end —
and its end does not step the queue. Back returns to the opened video, where it was left.

The related list:
- its marks announce through the shared notice (the Notification Center finds them), join the
  host's player session (the feed refetches on close), and stay per write, so an earlier answer no
  longer undoes a later mark still on its way;
- every host's state write patches the cached related lists, which are never refetched;
- kept for the session (gcTime), a retry asks for the page that failed, a list read so far with
  nothing unseen says so, a refusal a retry cannot answer offers none, and the followed mark names
  itself on hover.

Turned off on the instance (a limit of 0), the tab, the drawer button and the live-search buttons
are left out.

Keys: the player's keys and its focus reclaim defer to a layer above it (the drawer); going wide
moves the drawer's list into the panel's tab; `j` goes on from the slot of a row removed inside the
top layer; the help sheet lists the list keys over a layer that holds a list.
2026-10-25 00:32:28 +02:00
peter d53729a63d fix(youtube): a related page is empty only on YouTube's own word
The watch frame alone read as "nothing related" — so a related column YouTube moved, or a list
drawn by renderers this does not read, answered an empty list instead of an error. Now only the
"isn't available" promo YouTube draws for a video it no longer has is an empty page; anything else
it cannot read is a ScrapeError.

The flattening of a video renderer and the read of the continuation token are one each, in
innertube, for the search and the related scrape.
2026-10-25 00:32:27 +02:00
peter b04b51c909 fix(search): every related and live-search request takes a slot, and a failed one gives it back
A related list's cursor cost nothing, and a cursor is a free string: a replayed one read and
enriched without end. Every request now takes a slot of the user's daily cap — taken under a lock
before the work (`quota.claim_action`), so requests sent together cannot all pass the last free
one, and given back when the request fails outright (`quota.refund_action`). The live search's
scrape source takes its slots the same way.

A limit of 0 turns the feature off: a distinct 409 `related_disabled` / `search_disabled`, and
`/api/me` says `related_enabled` / `search_enabled` so the UI can leave it out.

A videos.list that fails no longer reads as an empty page: `ingest_candidates` raises, and the
search and the related list answer the refusal (the shared budget spent says "quota") instead of
"no results". The two routes' page loops are one (`_gather`).

`clear` deletes only what the caller found: a related list's rows carry no find, and any user
could name them.
2026-10-25 00:32:26 +02:00
peter 7d45c4f59f fix(keys): a list inside the topmost layer walks by keyboard
The list keys are the page's and an open layer silences them, so the
player's related videos could only be reached by Tab. A layer now names
its root element; with the focus on a row inside the topmost one, j/k,
the arrows and Enter walk that list, the walk stays inside the layer,
and the player's seek and volume arrows leave a focused row alone.
2026-10-24 22:45:26 +02:00
peter 5205091116 refactor(player): the related-region pref key stays private to its list 2026-10-24 17:03:01 +02:00
peter 686c673fb2 feat(player): a video's related videos, on demand beside it
The player shows what YouTube lists beside the video, without what the
user has watched or hidden. Wide, a Related tab beside the description;
stacked, a button beside the info opens them in a drawer. A row plays in
the same player (Back returns) and the list stays the opened video's.
A switch picks on-topic or Hungarian-trending lists, kept on the account.

Each related row says whether the user follows its channel; the row is
the video card's narrow form with its whole menu. VideoMenu moves to
features/video, where the player can reach it.
2026-10-24 16:57:52 +02:00
peter 8b37c470ba feat(catalog): a video's related videos, without what the user has seen
GET /api/search/related/{video_id} reads the list YouTube shows beside a
video from the website's InnerTube "next" endpoint (no quota), and brings
it into the catalog the way live-search results come in. It leaves out
what the user watched or hid, here or in their Takeout history (unless
taken back to "new" since), blocked channels and the video itself, and
pages on until 20 are gathered (at most 4 pages). region=topical|hu picks
the locale the list is filled in. Its own per-user daily cap
(related_daily_limit_per_user, one event per opened list); a failed
scrape answers 422 youtube/unavailable.

The InnerTube request (live key, consent cookie, errors) moves into
youtube/innertube.py, shared by the search scrape and the related one.
2026-10-24 16:00:37 +02:00
peter 4695e1a62f release: 0.111.0 (feature/e48-watched-channels) 2026-10-24 13:19:55 +02:00
peter 1adf59ea1b chore(release): 0.111.0 2026-10-24 13:11:09 +02:00
peter bc8fe2cc6c fix(catalog): a follow from YouTube keeps an explored channel, and the list counts only what can come
A subscription pulled in from YouTube left an explored channel flagged as an exploration's, so its
later videos came in hidden from the Library with their watched marks waiting for good; a subscribe
landing while an exploration's videos came in left those flagged too. One step, settle_explored,
now holds the rule by who follows the channel now: the subscribe here, the subscription import, the
watched-channel videos and the explore page all end with it. Migration 0116 clears what the gap
left behind.

The list's "may still grow" count no longer counts a channel it already shows from a stub row, and
the comments on the cleanup's keep rule and on the in-memory daily shares say what they do.
2026-10-24 11:29:14 +02:00
peter 33dd9ff054 fix(reader): the watched-channels list keeps its channels, its shares and its retries
- A channel someone follows: the videos its exploration brings in (the list's
  dialog, the channel page's explore) come in as the catalog's own, not
  flagged via_explore, which subscribing would never clear and which kept
  their watched marks waiting for good.
- The explore cleanup keeps the row of a channel a Takeout history shows
  watched (it still reclaims the videos), so a looked-at channel no longer
  drops off the list.
- The Shorts job takes up only users whose list still has lengths to ask or
  channels to name, by each user's own follows, blocks and dismissals; a
  naming that failed is tried again, and the list says it is not complete.
- The length lookups have a daily share of their own: a large history no
  longer leaves the export's Watch later file waiting a day.
- Only a channel of the user's own history can be put out of the list.
- The dialog's videos have a query root of their own: the list's
  invalidations no longer re-ask (and re-explore) under an open dialog.
2026-10-24 01:10:29 +02:00
peter b5f9cf14db feat(channels): a "Discover from your history" tab for the channels watched but not followed
The Channel manager lists the channels the Takeout history shows watched
often but not followed, the most watched first: twenty at a time, the
channel's About on hover, its name to its page, Follow, and Not interested
with an Undo. A row's count opens the videos it counts, ready to play one
or all. Before a history is uploaded it points to the upload; while the
import still asks YouTube it says more channels may join.
2026-10-24 00:00:50 +02:00
peter bd2340c204 feat(reader): the channels a Takeout history shows watched but not followed
The import asks YouTube how long the watched videos of the candidate
channels are (five or more watched, none the user follows, blocked, put
out or owns), keeps the length on the Takeout row and names the channels
that make the list: five videos of five minutes or more. The shorts job
asks what an import could not. A channel's counted videos come in as an
exploration's when its list is opened, ready to play; a channel can be put
out of the list and back.
2026-10-23 23:45:54 +02:00
peter 84af92aa29 release: 0.110.0 (feature/e41-history-import) 2026-10-23 22:48:06 +02:00
peter 420baff463 chore(release): 0.110.0 2026-10-23 22:39:20 +02:00
peter 9c5f81fd94 fix(reader): the Takeout import keeps a later upload, an un-mark and its memory bound
- An ad-hoc request made while its target's run is under way is the run after it: the scheduler
  dropped a second fire of a running id, so a second Takeout file was never imported and its key
  stayed queued until a restart.
- Un-marking or resetting a video is remembered (video_unmarks): an import of an older YouTube view
  no longer marks it watched again; un-marks take the user's Takeout lock.
- One user's failure in the Shorts job's Takeout pass is theirs: the job keeps its own result, the
  others are settled, and a failed lookup spends none of the daily share.
- The upload reads the user's id before giving the read connection back.
- A watched row waits on a discovery video (search or explore) until the catalog keeps it, so a
  mark never pins it against the cleanup jobs.
- The history is decoded a window at a time, a Watch later file over 2 MiB is refused unread, and
  two uploads at once across all users (takeout_server_busy).
- A Watch later list goes in as one presence read and one insert.
2026-10-23 22:07:11 +02:00
peter 9f580fd8c8 fix(reader): the Takeout import holds up under abuse, races and a run it cannot see
- Uploads: one at a time per user and ten an hour, each refused only after its body arrived;
  the history read an entry at a time (32 MiB cap), a Watch later list capped at YouTube's 5000,
  a user's kept videos at 200k; any unreadable shape is a 422, not a 500.
- Quota: Watch later lookups stop at the backfill reserve and at a per-user daily share; what is
  not asked about waits, known videos still go in, and the shorts job retries the rest.
- Staging and settling share one per-user advisory lock (no deadlock); a run settles only the
  rows it read, so a row staged meanwhile is never called gone.
- A failed run's notice names a translated failure, never the upstream's text.
- The status says when an import is queued; the page watches it to its end even when left, then
  reads the feed, its counts and the playlists again.
- Settings: one live region, the zone kept (unavailable) while a run goes, focus handed back,
  a missed drop kept from the browser, an oversize file refused before it is sent.
- Scheduler and run lists show Takeout runs by the importing user.
2026-10-23 20:48:23 +02:00
peter e0b02a0c14 fix(inbox): a Takeout import's notice opens the user's own import
The import's notice reaches a user who may not see the Scheduler, an
admin page, yet its link opened it. It now opens Settings › Account,
where the import's status is; any other run still opens the Scheduler.
The job id is held to the server's by the backend-ids contract. The
notice's Hungarian names the list as the app does elsewhere.
2026-10-23 19:15:06 +02:00
peter 3553eac8c4 feat(settings): upload a Google Takeout export in Settings › Account
A "YouTube history" section under YouTube access: the watch history and
the Watch later csv are dropped or picked (one request each, each file's
outcome said on its own), and the section then says what the import did
- only the counts it produced - or that it is running, failed or was cut
off by a restart. While it runs it asks again until it ends; a failed
re-read shows the failure, not the previous answer.

The drop zone the Files upload had inline is now ui/DropZone, used by
both. The e2e seed clears the account's kept Takeout videos.
2026-10-23 18:51:05 +02:00
peter 5d9952ab1e feat(reader): a Takeout video the catalog learns later is marked watched then; the import's status
A watched video the catalog did not know at import time waits in `takeout_items`. The Shorts job now
settles those rows right after its probe (the only place a video's Short-ness becomes known), every
user's, a commit each: `reader.takeout.apply_waiting`, its count in the job's result.

`GET /api/takeout` is what the settings page reads: the user's last import run (`jobruns.last_run`)
and the kept videos by kind and outcome, "waiting" for one not known yet.
2026-10-23 17:30:28 +02:00
peter f486a69ef2 fix(deps): brotli 1.2.1 (PYSEC-2026-4322) 2026-10-23 17:10:38 +02:00
peter 7b8505af95 feat(reader): a Google Takeout export becomes watch state and Watch later
The YouTube API reads neither a user's watch history nor their Watch later list, so the Takeout
export is the only way in. `POST /api/takeout` takes one file of it (watch-history.json or
Watch later-videos.csv, told apart by content), keeps its videos in `takeout_items` and queues the
user's `takeout_import` run, whose outcome reaches their inbox.

The run marks watched the known videos the Shorts probe has cleared, at the Takeout time; a Short
is not marked, and a state the user set here (watched, hidden, or changed after that time) wins, by
the upsert's own condition. A video the catalog does not know yet waits with no outcome. Watch later
videos are fetched if unknown and appended oldest first.

An ad-hoc run can now ask for a completion notice (`run_ad_hoc(..., notify=True)`).
2026-10-23 17:04:54 +02:00
peter bf5ceec8a1 release: 0.109.2 (bug/plex-player-server-away) 2026-10-23 15:00:27 +02:00
peter e788326247 chore(release): 0.109.2 — a server restart mid-episode is waited out by the Plex player 2026-10-23 14:52:04 +02:00
peter 38c71b7b4f fix(plex): a server that restarts mid-episode is waited out, not reported as a stream that cannot play
During a deploy the gateway answers 502 (or nothing answers at all) until the app is back. The
player took the first such answer for a broken stream: it said "Playback couldn't start" over a
picture still playing from its buffer, stopped loading, and never asked again, so the episode froze
where the buffer ended with the error still up. Now a 502/503/504 or no answer is the server being
away: the buffer keeps playing undisturbed, the session is asked for again where the viewer is
(2, 4, 8, then every 15 s), and the error is said only after three minutes away. A seek or another
item replaces a pending retry. Measured on localdev: a 45 s API stop now resumes at the viewer's
position, where it froze with the error before.
2026-10-23 14:42:17 +02:00
169 changed files with 13104 additions and 864 deletions
+8
View File
@@ -0,0 +1,8 @@
# gitleaks false positives, one fingerprint per finding (commit:file:rule:line). Each says why it is no secret.
#
# YouTube's public WEB InnerTube key: embedded in every youtube.com results page, read live and cached;
# this constant is only the fallback (backend/app/youtube/innertube.py).
8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/app/youtube/innertube.py:gcp-api-key:23
# A recorded watch-next page's continuation cursors (continuationCommand.token): page pointers, not credentials.
8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/tests/fixtures/innertube_next_first.json:generic-api-key:768
8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/tests/fixtures/innertube_next_first.json:generic-api-key:792
+48
View File
@@ -7,6 +7,54 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.112.1 — 2026-10-27
### Fixed
- Security: the sign-in library is updated, closing two published vulnerabilities.
## 0.112.0 — 2026-10-25
### New
- Player: a Related tab lists videos related to the one playing, leaving out what you have already watched (your uploaded YouTube history included). Channels you follow are marked, and you can switch between On topic and Trending.
- Play a related video in place; Back returns to the one you opened. Admins set a daily per-user limit in Config › Quota (default 200, 0 turns it off).
- Side panel: the list controls and the watched toggle are glyphs on one line, and Auto and Loop show when they are on, also under the pointer.
### Fixed
- Live search: a search that never reached YouTube no longer uses up a daily slot, a busy moment says so after a few seconds instead of waiting on, and a first page that failed offers Retry.
- Menus: Play from the beginning wears the same ↺ glyph everywhere, and Reset its own eraser glyph.
- On the day of this update, API searches made earlier that day stop counting toward the daily search limit, until Pacific midnight.
## 0.111.0 — 2026-10-24
### New
- Channels › Discover from your history: the channels your uploaded YouTube history shows you watching often (five videos of five minutes or more) but not following.
- Follow one, put it out of the list (with Undo), or open the videos you watched there and play them one by one or all in a row.
### Fixed
- A channel someone explored and then followed on YouTube is now part of the shared Library like any followed channel, and so are its videos; before, they could stay hidden and their watched marks never applied.
## 0.110.0 — 2026-10-23
### New
- YouTube history: upload your Google Takeout export in Settings › Account. What you watched on YouTube is marked watched here (never a Short, never over a state you set), and your YouTube Watch later list joins yours.
- A video the catalog does not know yet is remembered and marked once it arrives. The import's outcome reaches your inbox, and admins see each run on the Scheduler page.
### Fixed
- Scheduler: a channel's full-history backfill asked for again while one is running now runs after it, instead of being silently dropped.
## 0.109.2 — 2026-10-23
### Fixed
- Plex player: when the server restarts mid-episode (an update), playback carries on from the buffer and picks up where you are once it is back, instead of showing "Playback couldn't start" and stopping where the buffer ended.
## 0.109.1 — 2026-10-23
### New
+1 -1
View File
@@ -1 +1 @@
0.109.1
0.112.1
@@ -0,0 +1,51 @@
"""A user's Google Takeout export: what they watched on YouTube, and their Watch later list
Revision ID: 0113_takeout_items
Revises: 0112_one_watch_later
Create Date: 2026-10-23
One row per video and kind. A video the catalog does not know yet waits with no outcome, so the id
is not a foreign key; the partial index is what a newly known video looks its waiting rows up by.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0113_takeout_items"
down_revision: str | None = "0112_one_watch_later"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"takeout_items",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("kind", sa.String(12), nullable=False),
sa.Column("video_id", sa.String(16), nullable=False),
sa.Column("channel_id", sa.String(32), nullable=True),
sa.Column("at", sa.DateTime(timezone=True), nullable=False),
sa.Column("outcome", sa.String(12), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("user_id", "kind", "video_id", name="uq_takeout_item"),
sa.CheckConstraint("kind IN ('watched', 'watch_later')", name="ck_takeout_item_kind"),
)
op.create_index(
"ix_takeout_items_waiting",
"takeout_items",
["video_id"],
postgresql_where=sa.text("outcome IS NULL"),
)
def downgrade() -> None:
op.drop_index("ix_takeout_items_waiting", table_name="takeout_items")
op.drop_table("takeout_items")
@@ -0,0 +1,44 @@
"""When a user last took a video back to "new"
Revision ID: 0114_video_unmarks
Revises: 0113_takeout_items
Create Date: 2026-10-23
Un-marking or resetting a video leaves no `video_states` row, so this is what tells an import of the
user's YouTube history a video they un-marked here from one never touched.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0114_video_unmarks"
down_revision: str | None = "0113_takeout_items"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"video_unmarks",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"video_id",
sa.String(16),
sa.ForeignKey("videos.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("at", sa.DateTime(timezone=True), nullable=False),
sa.UniqueConstraint("user_id", "video_id", name="uq_video_unmark"),
)
op.create_index("ix_video_unmarks_video_id", "video_unmarks", ["video_id"])
def downgrade() -> None:
op.drop_index("ix_video_unmarks_video_id", table_name="video_unmarks")
op.drop_table("video_unmarks")
@@ -0,0 +1,49 @@
"""A watched Takeout video's length, and the channels a user put out of their list
Revision ID: 0115_takeout_channels
Revises: 0114_video_unmarks
Create Date: 2026-10-24
The "channels you watch but do not follow" list counts a user's watched videos over five minutes
by channel. The length is asked of YouTube for that list and kept on the Takeout row; a channel the
user does not want on the list is kept apart.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0115_takeout_channels"
down_revision: str | None = "0114_video_unmarks"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"takeout_items", sa.Column("looked_up_at", sa.DateTime(timezone=True), nullable=True)
)
op.add_column("takeout_items", sa.Column("duration_seconds", sa.Integer(), nullable=True))
op.create_table(
"takeout_channel_dismissals",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("channel_id", sa.String(32), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.UniqueConstraint("user_id", "channel_id", name="uq_takeout_channel_dismissal"),
)
def downgrade() -> None:
op.drop_table("takeout_channel_dismissals")
op.drop_column("takeout_items", "duration_seconds")
op.drop_column("takeout_items", "looked_up_at")
@@ -0,0 +1,34 @@
"""An explored channel someone follows is the catalog's own, and so are its videos
Revision ID: 0116_followed_explored_channels
Revises: 0115_takeout_channels
Create Date: 2026-10-24
A follow pulled in from YouTube (the subscription import) left an explored channel flagged as an
exploration's, and a subscribe landing while an exploration's videos came in left those videos
flagged on a channel that no longer was one. Both are held in the code now; this clears what they
left: the channel of a subscription is not an exploration's, and no video is an exploration's on a
channel that is not one. Data only, so the downgrade has nothing to undo.
"""
from alembic import op
revision: str = "0116_followed_explored_channels"
down_revision: str | None = "0115_takeout_channels"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.execute(
"UPDATE channels SET from_explore = false WHERE from_explore"
" AND EXISTS (SELECT 1 FROM subscriptions s WHERE s.channel_id = channels.id)"
)
op.execute(
"UPDATE videos SET via_explore = false WHERE via_explore"
" AND channel_id IN (SELECT id FROM channels WHERE NOT from_explore)"
)
def downgrade() -> None:
pass
+8
View File
@@ -72,6 +72,13 @@ DEFAULT_MAX_BODY_BYTES = 1024 * 1024
# A route belongs here ONLY IF ITS OWN CAP IS LARGER THAN THE DEFAULT. That is why
# `POST /api/diagnostics/events` is absent: its `capped_body` limit is 512 KiB, already below the
# default, so listing it could only widen what it accepts.
# A Google Takeout file (`reader.takeout`). A real watch history measured 3.7 MB for 9500 views, ~390
# bytes a view, so 32 MiB holds ~86k views. YouTube keeps the history 36 months by default on an
# account made since mid-2020 and for good on an older one, so a heavy viewer's can pass 16 MiB. The
# file's text is decoded a window at a time and read an entry at a time (`takeout._json_list`), so
# it costs its bytes and a window, not a tree of it; the route takes two at once across every user.
TAKEOUT_MAX_BYTES = 32 * 1024 * 1024
ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# Ingesting a local file. The body IS the file now — no multipart envelope to allow for —
# so this ceiling and the route's own cap count exactly the same bytes. `sysconfig`'s
@@ -95,6 +102,7 @@ ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# sending it at all, off the `upload_max_bytes` that `GET /api/files/usage` reports.
("POST", "/api/files/upload"): settings.download_upload_max_bytes,
("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
("POST", "/api/takeout"): TAKEOUT_MAX_BYTES,
}
# The refusal, byte-for-byte what `capped_body`'s `HTTPException(413, {"code": "too_large"})` renders
+9 -6
View File
@@ -71,10 +71,12 @@ class Channel(Base, TimestampMixin):
Boolean, default=False, server_default="false", index=True
)
# True when this channel exists only because a user EXPLORED it (opened its page without
# subscribing) — an ephemeral, per-explorer addition. The cleanup job hard-deletes such a
# channel (and its via_explore videos) once nobody keeps it (no subscription, no live
# ExploredChannel row, no per-user state) after a grace period. Cleared when someone
# subscribes (the "keep" signal promotes it to organic).
# subscribing) — an ephemeral, per-explorer addition. After a grace period the cleanup job
# hard-deletes its via_explore videos nobody keeps (no subscription, no live ExploredChannel
# row, no per-user state), and then the channel row itself once it holds no videos — unless a
# Takeout watch history shows it watched: the "watched but not followed" list names it from
# that row. Cleared when someone subscribes, here or on YouTube (the "keep" signal promotes it
# to organic).
from_explore: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
)
@@ -235,8 +237,9 @@ class ExploredChannel(Base, TimestampMixin):
Exploring an un-subscribed channel ingests its recent uploads (flagged via_explore) so the
user can browse them, but keeps them out of everyone else's catalog. This row (a) gates the
explorer's access to those ephemeral videos in the feed query and (b) drives the grace-period
purge: a from_explore channel with no live ExploredChannel row (and no subscription / no
per-user state on its videos) is hard-deleted by the cleanup job. Subscribing is the "keep"
purge: of a from_explore channel with no live ExploredChannel row (and no subscription) the
cleanup job hard-deletes the videos without per-user state, then the channel once it holds none
and no Takeout watch history shows it watched. Subscribing is the "keep"
signal — it promotes the channel to organic and these rows become irrelevant. `created_at`
(from TimestampMixin) is the grace clock; a re-visit refreshes it."""
+13
View File
@@ -158,6 +158,19 @@ def blocked_ids(db: Session, user: User) -> set[str]:
)
def followed_among(db: Session, user: User, channel_ids: set[str]) -> set[str]:
"""Which of these channels the user subscribes to."""
if not channel_ids:
return set()
return set(
db.scalars(
select(Subscription.channel_id).where(
owned_by(Subscription.user_id, user), Subscription.channel_id.in_(channel_ids)
)
)
)
def blocked_channels(db: Session, user: User, lang: str) -> list[Channel]:
return list(
db.execute(
+21 -9
View File
@@ -75,7 +75,8 @@ def ingest_candidates(
) -> list[str]:
"""Materialise one page of search candidates into the catalog (channel stubs + enrich,
video stubs flagged via_search, enrich, Shorts classification) and return the KEEPABLE
video ids — non-Short, non-live — in the input relevance order."""
video ids — non-Short, non-live — in the input relevance order. A videos.list that fails
raises its YouTubeError: with nothing enriched, nothing could be kept."""
ids = [it["id"] for it in candidates]
if not ids:
return []
@@ -132,13 +133,16 @@ def ingest_candidates(
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_ENRICH):
detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
except YouTubeError:
# Every candidate then stays unenriched, and an unenriched video cannot be judged live or
# Short — so it is dropped. A working search returns nothing and reads as "no results".
# An unenriched video cannot be judged live or Short, so none of this page could be kept:
# swallowed, the failure answered as a page with nothing on it — "no results" for a search
# that never ran (the shared budget spent, no API key). The stubs stay (`insert_stubs`
# committed them), as the channels' do above; the caller answers the failure (its refusal
# says which).
log.warning(
"the videos found by a search could not be enriched",
extra=logs.event("search.video_enrich_failed", candidates=len(ids)),
)
detail_by_id = {}
raise
enriched_at = datetime.now(
timezone.utc
) # when THIS enrich ran (distinct from the stub placeholder)
@@ -212,17 +216,25 @@ def clear(db: Session, user: User, video_ids: list[str]) -> int:
"""Discard these results "as if never added": drop this user's finds of them, then delete the
ones nobody kept — a `via_search` video no reader holds (`reader.queries.kept_by_a_reader`),
whose channel nobody subscribes to, and that no other user still has as a find. Returns how
many videos went."""
db.execute(
delete(SearchFind).where(
owned_by(SearchFind.user_id, user), SearchFind.video_id.in_(video_ids)
many videos went.
Only the ones this user found: a video in the catalog with no find at all (a related list's,
which records none) is in someone's list right now, and an id any user could name would let
them delete it from under that list."""
mine = list(
db.scalars(
delete(SearchFind)
.where(owned_by(SearchFind.user_id, user), SearchFind.video_id.in_(video_ids))
.returning(SearchFind.video_id)
)
)
if not mine:
return 0
subscribed = select(Subscription.id).where(Subscription.channel_id == Video.channel_id).exists()
other_find = select(SearchFind.id).where(SearchFind.video_id == Video.id).exists()
result = db.execute(
delete(Video).where(
Video.id.in_(video_ids),
Video.id.in_(mine),
Video.via_search.is_(True),
~reader_queries.kept_by_a_reader(Video.id),
~subscribed,
+18 -7
View File
@@ -96,13 +96,24 @@ def record_subscription(db: Session, user: User, channel: Channel, yt_subscripti
subscribed_at=datetime.now(timezone.utc),
)
)
if channel.from_explore:
channel.from_explore = False
db.execute(
update(Video)
.where(Video.channel_id == channel.id, Video.via_explore.is_(True))
.values(via_explore=False)
)
db.flush()
settle_explored(db, channel)
def settle_explored(db: Session, channel: Channel) -> None:
"""Hold the rule that a channel's videos are an exploration's only while nobody follows it,
decided by who follows it now, not by the flag held in memory: a subscribe in another request
may have cleared that while YouTube answered this one, and the videos that came in meanwhile
would stay flagged. A channel someone follows, however the follow came (here or pulled in from
YouTube), is the catalog's own, and so are its videos."""
if channel.from_explore and not queries.followed(db, channel.id):
return
channel.from_explore = False
db.execute(
update(Video)
.where(Video.channel_id == channel.id, Video.via_explore.is_(True))
.values(via_explore=False)
)
db.flush()
+5
View File
@@ -113,6 +113,11 @@ class Settings(BaseSettings):
# each search.list costs 100 units, so the shared budget only affords ~80-90/day total; with
# the zero-quota scrape source it's a pure anti-abuse rate limit. Admin-tunable.
search_daily_limit_per_user: int = 70
# Per-user cap on related-video requests per day — a list opened and each "Show more" (the
# InnerTube watch page, no quota; its enrich is up to ~8 units: up to 4 pages, each a videos.list
# and a channels.list). A failed one gives its slot back. 0 turns related videos off.
# Admin-tunable.
related_daily_limit_per_user: int = 200
# Source for live YouTube search: "scrape" (InnerTube, no API quota) or "api" (search.list,
# 100 units/page). Scrape is the default; flip to "api" if scraping is blocked/breaks.
search_source: SearchSource = "scrape"
+10
View File
@@ -119,6 +119,16 @@ def recent_runs(db: Session, jobs: list[str], per_job: int) -> dict[str, list[Jo
return out
def last_run(db: Session, job: str, target: str) -> JobRun | None:
"""The newest run of an ad-hoc `job` on `target` (a user's own import), or None."""
return db.execute(
select(JobRun)
.where(JobRun.job == job, JobRun.target == target)
.order_by(JobRun.started_at.desc(), JobRun.id.desc())
.limit(1)
).scalar_one_or_none()
def gc(db: Session, retention_days: int) -> int:
"""Delete finished runs older than `retention_days` (0 = keep forever). A run still going is
never reaped. The caller commits."""
+31 -13
View File
@@ -44,6 +44,7 @@ class QuotaAction:
VIDEOS_ENRICH = "videos_enrich"
VIDEOS_LOOKUP = "videos_lookup"
VIDEOS_SEARCH = "videos_search"
VIDEOS_RELATED = "videos_related"
CHANNELS_DISCOVER = "channels_discover"
CHANNELS_EXPLORE = "channels_explore"
CHANNELS_SUBSCRIBE = "channels_subscribe"
@@ -142,7 +143,7 @@ def backfill_reserve_reached() -> bool:
condition for every backfill/explore path (recent, deep, immediate full-history, explore, and
rolling revalidation).
Reads in its OWN short-lived session (like record_usage / log_action), for two reasons: it
Reads in its OWN short-lived session (like record_usage / claim_action), for two reasons: it
replaces the remaining/reserve comparison that was open-coded at every call site, and — critically
for the per-page guards in paging loops — a guard must not hold the caller's transaction open
across the network fetch that follows it. Quota is committed independently of any caller, so a
@@ -151,24 +152,40 @@ def backfill_reserve_reached() -> bool:
return remaining_today(qs) <= sysconfig.get_int(qs, "backfill_quota_reserve")
def log_action(user_id: int, action: str) -> None:
"""Record a zero-cost action event so per-user daily caps still count it.
def claim_action(user_id: int, action: str, limit: int) -> int | None:
"""Take one of this user's `limit` daily `action` slots, or None when they are all taken — the
check and the take in one transaction, under a lock per user and action. Checked first and
logged after the work, N requests sent together at the last free slot all passed the check
(the work in between is seconds of scraping). Returns the event's id, for `refund_action` when
the work then produced nothing.
`record_usage` only logs when units are actually charged; a scrape-based search spends no
quota yet must still be rate-limited per user, so it logs its event here directly.
Writes in its OWN session (like record_usage) — quota accounting is durable independent of the
caller's transaction and must never commit the caller's session (that would flush a job's
partial state mid-run, defeating its own rollback)."""
The ONE way to count a capped action: a claim is a zero-unit event, and `actions_today` counts
those alone, so a call the action then pays for (search.list's own `record_usage` event, under
the same action) is accounting, never a second slot. Own session, committed at once — quota
accounting is durable whatever the caller's transaction does next, and must never commit it."""
with SessionLocal() as qs:
qs.add(QuotaEvent(user_id=user_id, action=action, units=0))
qs.execute(select(func.pg_advisory_xact_lock(func.hashtext(action), user_id)))
if actions_today(qs, user_id, action) >= limit:
return None
event = QuotaEvent(user_id=user_id, action=action, units=0)
qs.add(event)
qs.commit()
return event.id
def refund_action(event_id: int) -> None:
"""Give a slot `claim_action` took back: the work it was taken for failed outright, and a
request that showed the user nothing must not count against them."""
with SessionLocal() as qs:
qs.execute(delete(QuotaEvent).where(QuotaEvent.id == event_id))
qs.commit()
def actions_today(db: Session, user_id: int, action: str) -> int:
"""How many quota events of `action` this user has logged so far in the current Pacific
day — for per-user, per-action daily caps (e.g. the live-search limit). Counts events,
not units, so it only works for actions charged exactly once per user action."""
"""How many slots of `action` this user has claimed so far in the current Pacific day — for
per-user, per-action daily caps (the live search, the related lists). Counts the zero-unit
claim events (`claim_action`) only: `record_usage`'s events for units the action spent are not
slots (it logs only when units were charged)."""
return (
db.scalar(
select(func.count())
@@ -176,6 +193,7 @@ def actions_today(db: Session, user_id: int, action: str) -> int:
.where(
owned_by(QuotaEvent.user_id, user_id),
QuotaEvent.action == action,
QuotaEvent.units == 0,
QuotaEvent.created_at >= pacific_day_start_utc(),
)
)
+8
View File
@@ -42,6 +42,14 @@ class RateLimiter:
with self._lock:
return sum(c for t, c in self._hits.get(key, []) if t > cutoff) >= self.max_events
def room(self, key: str) -> float:
"""How much cost ``key`` may still spend in the current window, WITHOUT counting anything —
for work counted only once it has succeeded: ask it first, ``allow`` what was done."""
cutoff = time.monotonic() - self.window
with self._lock:
used = sum(c for t, c in self._hits.get(key, []) if t > cutoff)
return max(0.0, self.max_events - used)
def allow(self, key: str, cost: float = 1) -> bool:
"""Record an attempt costing ``cost`` for ``key`` and return whether it is within the limit.
+83
View File
@@ -4,6 +4,7 @@ and the comfort shelf on Home. Only this unit writes them; another unit reads th
edge, or asks this unit's service."""
from datetime import datetime
from enum import StrEnum
from sqlalchemy import (
Boolean,
@@ -84,6 +85,21 @@ class VideoState(Base, UpdatedAtMixin):
progress_updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class VideoUnmark(Base):
"""When the user last took a video back to "new" here — un-marked it, or reset it. That leaves
no `VideoState` (its absence is "new"), so without this row a later import of what they watched
on YouTube could not tell a video they un-marked from one never touched, and marked it watched
again. Kept by user and video; the video's own deletion takes it."""
__tablename__ = "video_unmarks"
__table_args__ = (UniqueConstraint("user_id", "video_id", name="uq_video_unmark"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
video_id: Mapped[str] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"), index=True)
at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
class Playlist(Base, TimestampMixin, UpdatedAtMixin):
"""A per-user named, ordered collection of videos from the shared catalog.
@@ -213,3 +229,70 @@ class ComfortPin(Base, TimestampMixin):
# 0-based shelf order. The shelf is short and hand-curated, so a dense integer the reorder
# endpoint rewrites wholesale beats a fractional index nobody would ever exercise here.
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0", nullable=False)
class TakeoutKind(StrEnum):
"""What a Google Takeout row says the user did with a video."""
WATCHED = "watched" # history/watch-history.json
WATCH_LATER = "watch_later" # playlists/Watch later-videos.csv
class TakeoutOutcome(StrEnum):
"""What the import did with a row; NULL = still waiting for its video to be known (and, for a
watched one, probed for Shorts)."""
WATCHED = "watched" # the video is marked watched at the Takeout time
KEPT = "kept" # the state the user set here (watched, hidden, or changed later) stays
SHORT = "short" # a Short: swiping one past is not watching it
ADDED = "added" # added to Watch later
PRESENT = "present" # already in Watch later
MISSING = "missing" # YouTube no longer has the video
class TakeoutItem(Base, TimestampMixin):
"""One video from a user's Google Takeout export: watched (with when) or in their YouTube Watch
later list (with when it was added). The rows are both the import job's input and its memory: a
video the catalog does not know yet waits here, with no outcome, until it enters the catalog.
The id is not a foreign key for that reason. A watched row's channel and length feed the
"channels you watch but do not follow" list: the length is asked of YouTube for that list alone
and kept here, so its videos need not enter the shared catalog to be counted."""
__tablename__ = "takeout_items"
__table_args__ = (
UniqueConstraint("user_id", "kind", "video_id", name="uq_takeout_item"),
CheckConstraint("kind IN ('watched', 'watch_later')", name="ck_takeout_item_kind"),
# The waiting rows, by video: what a newly known video looks up.
Index(
"ix_takeout_items_waiting",
"video_id",
postgresql_where=text("outcome IS NULL"),
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
kind: Mapped[str] = mapped_column(String(12))
video_id: Mapped[str] = mapped_column(String(16))
channel_id: Mapped[str | None] = mapped_column(String(32))
# Watched: the last time in the export; Watch later: when it was added.
at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
outcome: Mapped[str | None] = mapped_column(String(12))
# When YouTube was asked for the video's length (NULL = not yet), and the length it gave; a
# video it no longer has keeps a NULL length.
looked_up_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
duration_seconds: Mapped[int | None] = mapped_column(Integer)
class TakeoutChannelDismissal(Base, TimestampMixin):
"""A channel the user put out of their "watched but not followed" list. The id is not a foreign
key: the channel may never have entered the catalog, or may leave it, and the choice stays."""
__tablename__ = "takeout_channel_dismissals"
__table_args__ = (
UniqueConstraint("user_id", "channel_id", name="uq_takeout_channel_dismissal"),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(String(32))
+412 -4
View File
@@ -1,5 +1,5 @@
"""Reader's reads: a user's labels, saved views, watch state and playlists, for the HTTP layer and
for the other units that ask about them.
"""Reader's reads: a user's labels, saved views, watch state, playlists and Takeout import, for the HTTP
layer and for the other units that ask about them.
Every read of a user's row goes through `userscope`, so "someone else's row" is the same answer as
a missing one. Reader reads Catalog's videos along its declared edge (a playlist's cover and
@@ -7,14 +7,28 @@ length, whether a video exists); another unit asks the questions at the end of t
than joining Reader's tables itself.
"""
from collections.abc import Iterable
from datetime import datetime
from sqlalchemy import ColumnElement, and_, func, or_, select
from sqlalchemy.orm import InstrumentedAttribute, Session, aliased
from app.collation import collated
from app.feedquery import feed_columns, serialize
from app.models import User
from app.catalog.models import Channel, Subscription, Video
from app.reader.models import ChannelTag, Playlist, PlaylistItem, SavedView, Tag, VideoState
from app.catalog.models import BlockedChannel, Channel, Subscription, Video
from app.reader.models import (
ChannelTag,
Playlist,
PlaylistItem,
SavedView,
Tag,
TakeoutChannelDismissal,
TakeoutItem,
TakeoutKind,
VideoState,
VideoUnmark,
)
from app.userscope import owned_by, owned_row
# --- catalog, read along the edge ------------------------------------------------------------
@@ -230,6 +244,18 @@ def kept_by_a_reader(video_id: InstrumentedAttribute[str]) -> ColumnElement[bool
)
def in_a_watch_history(channel_id: InstrumentedAttribute[str]) -> ColumnElement[bool]:
"""A channel someone's uploaded Takeout history shows watched. The "watched but not followed"
list names a channel from its catalog row, so the explore cleanup keeps the row of one (its
videos it may still reclaim: opening the list's videos brings them back). As a clause, so the
cleanup's DELETE stays one statement."""
return (
select(TakeoutItem.id)
.where(TakeoutItem.channel_id == channel_id, TakeoutItem.kind == TakeoutKind.WATCHED)
.exists()
)
def playlisted_by(db: Session, video_ids: list[str]) -> list[tuple[int, str]]:
"""(user id, video id) for every user who has one of these videos in a playlist."""
return list(
@@ -293,6 +319,36 @@ def held_by(db: Session, user: User, video_ids: list[str]) -> set[str]:
)
def seen_by(db: Session, user: User, video_ids: list[str]) -> set[str]:
"""Which of these videos the user has already seen or set aside: watched or hidden here, or in
their uploaded Takeout history unless they took it back to "new" here since. The ids need not
be in the catalog — a Takeout row names a video the catalog may never have had — so a list
of what YouTube suggests can drop them before anything is ingested."""
if not video_ids:
return set()
marked = select(VideoState.video_id).where(
owned_by(VideoState.user_id, user),
VideoState.video_id.in_(video_ids),
VideoState.status.in_(("watched", "hidden")),
)
unmarked_since = (
select(VideoUnmark.id)
.where(
owned_by(VideoUnmark.user_id, user),
VideoUnmark.video_id == TakeoutItem.video_id,
VideoUnmark.at > TakeoutItem.at,
)
.exists()
)
in_history = select(TakeoutItem.video_id).where(
owned_by(TakeoutItem.user_id, user),
TakeoutItem.kind == TakeoutKind.WATCHED,
TakeoutItem.video_id.in_(video_ids),
~unmarked_since,
)
return set(db.scalars(marked)) | set(db.scalars(in_history))
def feed_rows_by_ids(db: Session, user: User, video_ids: list[str]) -> list[dict]:
"""These videos serialized with the feed's projection (their channel, this user's watch state,
saved), in the order given."""
@@ -359,3 +415,355 @@ def user_tag_link(db: Session, user: User, channel_id: str, tag_id: int) -> Chan
owned_by(ChannelTag.user_id, user),
)
).scalar_one_or_none()
# --- Google Takeout --------------------------------------------------------------------------
def _settled_here() -> ColumnElement[bool]:
"""A video a waiting "watched" row may be marked on: one the catalog keeps of its own, not a
discovery video only a search or an explore brought in. A mark is a `VideoState`, which the
cleanup jobs read as a reader keeping it (`kept_by_a_reader`): on one of those it would keep the
video, and its explored channel, for good. Such a row waits until the video is the catalog's own
(its channel subscribed to) — or is gone."""
subscribed = select(Subscription.id).where(Subscription.channel_id == Video.channel_id).exists()
return and_(Video.via_explore.is_(False), or_(Video.via_search.is_(False), subscribed))
def takeout_watched_ready(db: Session, user: User) -> list[tuple[int, str, datetime, bool]]:
"""The user's waiting "watched" rows whose video the catalog knows and has probed for Shorts:
(row id, video id, Takeout time, is Short)."""
return list(
db.execute(
select(TakeoutItem.id, TakeoutItem.video_id, TakeoutItem.at, Video.is_short)
.join(Video, Video.id == TakeoutItem.video_id)
.where(
owned_by(TakeoutItem.user_id, user),
TakeoutItem.kind == TakeoutKind.WATCHED,
TakeoutItem.outcome.is_(None),
Video.shorts_probed,
_settled_here(),
)
).tuples()
)
def takeout_waiting(db: Session, user: User, kind: TakeoutKind) -> list[tuple[int, str]]:
"""The user's rows of `kind` the import has not settled, oldest first: (row id, video id)."""
return list(
db.execute(
select(TakeoutItem.id, TakeoutItem.video_id)
.where(
owned_by(TakeoutItem.user_id, user),
TakeoutItem.kind == kind,
TakeoutItem.outcome.is_(None),
)
.order_by(TakeoutItem.at, TakeoutItem.id)
).tuples()
)
# Ids per IN list: a list is bound one parameter an id, and a statement takes at most 65,535.
_IN_CHUNK = 5000
def known_video_ids(db: Session, ids: list[str]) -> set[str]:
"""Which of `ids` the catalog holds."""
known: set[str] = set()
for start in range(0, len(ids), _IN_CHUNK):
chunk = ids[start : start + _IN_CHUNK]
known.update(db.execute(select(Video.id).where(Video.id.in_(chunk))).scalars())
return known
def unmarked_at(db: Session, user: User, ids: list[str]) -> dict[str, datetime]:
"""When the user last took each of `ids` back to "new" here, for those they ever did."""
out: dict[str, datetime] = {}
for start in range(0, len(ids), _IN_CHUNK):
rows = db.execute(
select(VideoUnmark.video_id, VideoUnmark.at).where(
owned_by(VideoUnmark.user_id, user),
VideoUnmark.video_id.in_(ids[start : start + _IN_CHUNK]),
)
).tuples()
out.update((vid, at) for vid, at in rows)
return out
def in_playlist(db: Session, playlist_id: int, ids: list[str]) -> set[str]:
"""Which of `ids` the playlist already holds."""
held: set[str] = set()
for start in range(0, len(ids), _IN_CHUNK):
chunk = ids[start : start + _IN_CHUNK]
held.update(
db.execute(
select(PlaylistItem.video_id).where(
PlaylistItem.playlist_id == playlist_id, PlaylistItem.video_id.in_(chunk)
)
).scalars()
)
return held
def takeout_rows_besides(
db: Session, user: User, kind: TakeoutKind, video_ids: Iterable[str]
) -> int:
"""How many Takeout rows the user keeps other than `video_ids` of `kind`: what an upload of
those videos would be added to."""
total = (
db.scalar(select(func.count(TakeoutItem.id)).where(owned_by(TakeoutItem.user_id, user)))
or 0
)
ids = list(video_ids)
for start in range(0, len(ids), _IN_CHUNK):
total -= (
db.scalar(
select(func.count(TakeoutItem.id)).where(
owned_by(TakeoutItem.user_id, user),
TakeoutItem.kind == kind,
TakeoutItem.video_id.in_(ids[start : start + _IN_CHUNK]),
)
)
or 0
)
return total
def takeout_outcomes(db: Session, user: User) -> dict[tuple[str, str | None], int]:
"""How many of the user's rows each kind and outcome holds (None = waiting)."""
rows = db.execute(
select(TakeoutItem.kind, TakeoutItem.outcome, func.count())
.where(owned_by(TakeoutItem.user_id, user))
.group_by(TakeoutItem.kind, TakeoutItem.outcome)
).all()
return {(kind, outcome): n for kind, outcome, n in rows}
def takeout_summary(db: Session, user: User) -> dict[str, dict[str, int]]:
"""The user's kept Takeout videos by kind and outcome, for their settings page: `{"watched":
{"watched": n, "short": n, "waiting": n, …}, "watch_later": {…}}` — the import's outcome words,
"waiting" for a video not known yet."""
out: dict[str, dict[str, int]] = {kind.value: {} for kind in TakeoutKind}
for (kind, outcome), n in takeout_outcomes(db, user).items():
out[kind][outcome or "waiting"] = n
return out
def takeout_users_ready(db: Session) -> list[int]:
"""Every user with a waiting "watched" row whose video is now known and probed for Shorts: the
rows a later run of the Shorts probe made ready."""
return list(
db.execute(
select(TakeoutItem.user_id)
.join(Video, Video.id == TakeoutItem.video_id)
.where(
TakeoutItem.kind == TakeoutKind.WATCHED,
TakeoutItem.outcome.is_(None),
Video.shorts_probed,
_settled_here(),
)
.distinct()
).scalars()
)
def takeout_users_watch_later_waiting(db: Session) -> list[int]:
"""Every user with a Watch later row a run could not put in yet."""
return list(
db.execute(
select(TakeoutItem.user_id)
.where(TakeoutItem.kind == TakeoutKind.WATCH_LATER, TakeoutItem.outcome.is_(None))
.distinct()
).scalars()
)
# --- channels watched but not followed ---------------------------------------------------------
# A channel joins the list with this many watched videos of at least this length. A Short is three
# minutes at most, so the length alone keeps them out, and a video under five minutes says too
# little of what a channel is about to count towards it.
WATCHED_CHANNEL_MIN_VIDEOS = 5
WATCHED_CHANNEL_MIN_SECONDS = 300
def _unfollowed(user: User) -> ColumnElement[bool]:
"""A watched row whose channel the user neither follows, blocked, put out of the list, nor is."""
subscribed = select(Subscription.channel_id).where(owned_by(Subscription.user_id, user))
blocked = select(BlockedChannel.channel_id).where(owned_by(BlockedChannel.user_id, user))
dismissed = select(TakeoutChannelDismissal.channel_id).where(
owned_by(TakeoutChannelDismissal.user_id, user)
)
clauses = [
owned_by(TakeoutItem.user_id, user),
TakeoutItem.kind == TakeoutKind.WATCHED,
TakeoutItem.channel_id.is_not(None),
TakeoutItem.channel_id.not_in(subscribed),
TakeoutItem.channel_id.not_in(blocked),
TakeoutItem.channel_id.not_in(dismissed),
]
if user.yt_channel_id:
clauses.append(TakeoutItem.channel_id != user.yt_channel_id)
return and_(*clauses)
def _candidate_channels(user: User):
"""The unfollowed channels with enough watched videos to make the list once their lengths are
known: (channel id, how many watched)."""
return (
select(TakeoutItem.channel_id, func.count().label("watched"))
.where(_unfollowed(user))
.group_by(TakeoutItem.channel_id)
.having(func.count() >= WATCHED_CHANNEL_MIN_VIDEOS)
.subquery()
)
def takeout_lengths_to_ask(db: Session, user: User) -> list[str]:
"""The watched videos whose length the list still needs: those of the candidate channels not
asked about yet, the most watched channels first, so a lookup cut short fills the top."""
cand = _candidate_channels(user)
return list(
db.execute(
select(TakeoutItem.video_id)
.join(cand, cand.c.channel_id == TakeoutItem.channel_id)
.where(_unfollowed(user), TakeoutItem.looked_up_at.is_(None))
.order_by(cand.c.watched.desc(), TakeoutItem.channel_id, TakeoutItem.at.desc())
).scalars()
)
def watched_channels_pending(db: Session, user: User) -> int:
"""How many channels a later run may still add to the list: the candidates with a watched video
whose length is not known, and the ones that made it but have no catalog row to be listed from
yet. One the catalog holds without details is listed already (by the name it came in with), so
it is not one to come."""
cand = _candidate_channels(user)
unasked = db.execute(
select(TakeoutItem.channel_id)
.join(cand, cand.c.channel_id == TakeoutItem.channel_id)
.where(_unfollowed(user), TakeoutItem.looked_up_at.is_(None))
.distinct()
).scalars()
counted = _counted_channels(user)
unlisted = db.execute(
select(counted.c.channel_id)
.outerjoin(Channel, Channel.id == counted.c.channel_id)
.where(Channel.id.is_(None))
).scalars()
return len(set(unasked) | set(unlisted))
def _counted_channels(user: User):
"""The channels that make the list by their counted videos, whatever the catalog holds of them."""
return (
select(TakeoutItem.channel_id)
.where(_unfollowed(user), TakeoutItem.duration_seconds >= WATCHED_CHANNEL_MIN_SECONDS)
.group_by(TakeoutItem.channel_id)
.having(func.count() >= WATCHED_CHANNEL_MIN_VIDEOS)
.subquery()
)
def watched_channels_to_describe(db: Session, user: User) -> list[str]:
"""The channels on the list (by their counted videos) that the catalog lacks or has no details
of yet: what the list cannot name, or names with no more than a video brought."""
counted = _counted_channels(user)
return list(
db.execute(
select(counted.c.channel_id)
.outerjoin(Channel, Channel.id == counted.c.channel_id)
.where(Channel.details_synced_at.is_(None))
).scalars()
)
def watched_channels(db: Session, user: User) -> list[tuple[Channel, int, datetime]]:
"""The channels the user watched often but does not follow: (channel, how many of its videos
over five minutes they watched, when last). Most watched first; on a tie the one watched more
lately, which is the likelier interest."""
n = func.count(TakeoutItem.id)
last = func.max(TakeoutItem.at)
return list(
db.execute(
select(Channel, n, last)
.join(Channel, Channel.id == TakeoutItem.channel_id)
.where(_unfollowed(user), TakeoutItem.duration_seconds >= WATCHED_CHANNEL_MIN_SECONDS)
.group_by(Channel.id)
.having(n >= WATCHED_CHANNEL_MIN_VIDEOS)
.order_by(n.desc(), last.desc(), Channel.id)
).tuples()
)
def has_watch_history(db: Session, user: User) -> bool:
"""Whether the user has uploaded a Takeout watch history at all."""
return bool(
db.scalar(
select(
select(TakeoutItem.id)
.where(owned_by(TakeoutItem.user_id, user), TakeoutItem.kind == TakeoutKind.WATCHED)
.exists()
)
)
)
def watched_in_history(db: Session, user: User, channel_id: str) -> bool:
"""Whether the user's Takeout history shows them watching this channel at all."""
return bool(
db.scalar(
select(
select(TakeoutItem.id)
.where(
owned_by(TakeoutItem.user_id, user),
TakeoutItem.kind == TakeoutKind.WATCHED,
TakeoutItem.channel_id == channel_id,
)
.exists()
)
)
)
def watched_channel_videos(
db: Session, user: User, channel_id: str
) -> tuple[list[tuple[str, datetime]], int]:
"""The user's watched videos of one channel that the list counts, last watched first: (video
id, when). And how many YouTube no longer has."""
rows = db.execute(
select(TakeoutItem.video_id, TakeoutItem.at, TakeoutItem.duration_seconds)
.where(
owned_by(TakeoutItem.user_id, user),
TakeoutItem.kind == TakeoutKind.WATCHED,
TakeoutItem.channel_id == channel_id,
TakeoutItem.looked_up_at.is_not(None),
)
.order_by(TakeoutItem.at.desc(), TakeoutItem.video_id)
).tuples()
counted: list[tuple[str, datetime]] = []
gone = 0
for vid, at, seconds in rows:
if seconds is None:
gone += 1
elif seconds >= WATCHED_CHANNEL_MIN_SECONDS:
counted.append((vid, at))
return counted, gone
def takeout_users_with_list_work(db: Session) -> list[int]:
"""Every user whose "watched but not followed" list a run can still add to: lengths to ask, or
channels to name. Asked of each user whose history holds a channel of enough watched videos,
since what is off a list (followed, blocked, put out, their own) is that user's own — a row of
one is never asked about, and would otherwise bring the user back on every run."""
per_channel = (
select(TakeoutItem.user_id)
.where(TakeoutItem.kind == TakeoutKind.WATCHED, TakeoutItem.channel_id.is_not(None))
.group_by(TakeoutItem.user_id, TakeoutItem.channel_id)
.having(func.count() >= WATCHED_CHANNEL_MIN_VIDEOS)
.subquery()
)
users = db.execute(
select(User).where(User.id.in_(select(per_channel.c.user_id))).order_by(User.id)
).scalars()
return [user.id for user in users if watched_channels_pending(db, user)]
+220 -2
View File
@@ -9,13 +9,27 @@ unique key, a row lock taken before a presence check. The comfort shelf's writes
from datetime import datetime, timezone
from sqlalchemy import exists, func, select, text, update
from typing import cast
from sqlalchemy import Table, bindparam, delete, exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.models import User
from app.reader import queries
from app.reader.models import ChannelTag, Playlist, PlaylistItem, SavedView, Tag, VideoState
from app.reader.models import (
ChannelTag,
Playlist,
PlaylistItem,
SavedView,
Tag,
TakeoutChannelDismissal,
TakeoutItem,
TakeoutKind,
TakeoutOutcome,
VideoState,
VideoUnmark,
)
from app.sync.playlists import lock_playlists, recompute_dirty
from app.userscope import owned_by
@@ -196,6 +210,7 @@ def unmark(db: Session, user: User, video_id: str) -> None:
the in-progress state, not where the user left off); otherwise the row goes. The player fires
this beside a checkpoint that may delete the same row: the caller's commit then raises
`StaleDataError`, which is that lost race and nothing worse."""
lock_takeout(db, user) # an import settling meanwhile reads the un-mark below, or comes first
row = queries.video_state(db, user, video_id)
if row is None:
return
@@ -204,9 +219,21 @@ def unmark(db: Session, user: User, video_id: str) -> None:
row.watched_at = None
else:
db.delete(row)
_remember_unmark(db, user, video_id)
db.flush()
def _remember_unmark(db: Session, user: User, video_id: str) -> None:
"""Note that the user took the video back to "new" now: the state they set leaves no row, and
an import of their YouTube history must not mark it watched again (`apply_takeout_watched`)."""
now = datetime.now(timezone.utc)
db.execute(
pg_insert(VideoUnmark)
.values(user_id=user.id, video_id=video_id, at=now)
.on_conflict_do_update(constraint="uq_video_unmark", set_={"at": now})
)
def mark(db: Session, user: User, video_id: str, status: str) -> None:
""" "watched" or "hidden", as one atomic upsert on (user, video). The player marks "watched" at
the end of playback at the instant a checkpoint may delete the in-progress row; a SELECT then
@@ -231,9 +258,11 @@ def mark(db: Session, user: User, video_id: str, status: str) -> None:
def clear_state(db: Session, user: User, video_id: str) -> None:
"""The video back to pristine for this user: status, resume point and `watched_at` all go.
Playlist membership lives elsewhere and stays."""
lock_takeout(db, user) # as `unmark`
row = queries.video_state(db, user, video_id)
if row is not None:
db.delete(row)
_remember_unmark(db, user, video_id)
db.flush()
@@ -495,3 +524,192 @@ def detach_user_tag(db: Session, user: User, channel_id: str, tag_id: int) -> No
if row is not None:
db.delete(row)
db.flush()
# --- Google Takeout --------------------------------------------------------------------------
_TAKEOUT_BATCH = 1000
# The advisory-lock namespace of one user's Takeout rows ("TO"), so a key cannot meet another lock's.
_TAKEOUT_LOCK_NS = 0x544F
def lock_takeout(db: Session, user: User) -> None:
"""Hold the user's Takeout rows for the rest of the transaction. An upload's staging and an
import's settling each lock many of the same rows, one statement after another and in no common
order, so without one lock over both the two can deadlock. An un-mark takes it too: the import
reads the un-marks before it marks, and one landing between the two would be overwritten."""
db.execute(select(func.pg_advisory_xact_lock(_TAKEOUT_LOCK_NS, user.id)))
def stage_takeout(
db: Session, user: User, kind: TakeoutKind, rows: dict[str, tuple[datetime, str | None]]
) -> None:
"""Keep the videos of one Takeout file, `{video id: (time, channel id)}`, for the import job.
A second upload of the same export, or a newer one, adds what is new: a video already kept keeps
the later time, and a row the import has settled keeps its outcome."""
lock_takeout(db, user)
values = [
{"user_id": user.id, "kind": kind, "video_id": vid, "at": at, "channel_id": channel}
for vid, (at, channel) in rows.items()
]
for start in range(0, len(values), _TAKEOUT_BATCH):
stmt = pg_insert(TakeoutItem).values(values[start : start + _TAKEOUT_BATCH])
db.execute(
stmt.on_conflict_do_update(
constraint="uq_takeout_item",
set_={
"at": func.greatest(TakeoutItem.at, stmt.excluded.at),
"channel_id": func.coalesce(stmt.excluded.channel_id, TakeoutItem.channel_id),
},
)
)
db.flush()
def _settle(db: Session, ids: list[int], outcome: TakeoutOutcome) -> None:
for start in range(0, len(ids), _TAKEOUT_BATCH):
db.execute(
update(TakeoutItem)
.where(TakeoutItem.id.in_(ids[start : start + _TAKEOUT_BATCH]))
.values(outcome=outcome)
)
def apply_takeout_watched(db: Session, user: User) -> dict[str, int]:
"""Mark watched what the user watched on YouTube, for the videos the catalog knows and has probed
for Shorts; the rest wait. `watched_at` is the Takeout time. A Short is not marked (swiping one
past is not watching it), and a state the user set here wins: watched or hidden, anything
changed after the Takeout time, or an un-mark since then (`VideoUnmark`). The upsert's own
condition is that rule, so a state set while the import runs wins too."""
lock_takeout(db, user)
short: list[int] = []
todo: dict[str, tuple[int, datetime]] = {}
ready = queries.takeout_watched_ready(db, user)
unmarked = queries.unmarked_at(db, user, [video_id for _, video_id, _, _ in ready])
kept: list[int] = []
for row_id, video_id, at, is_short in ready:
if is_short:
short.append(row_id)
elif video_id in unmarked and unmarked[video_id] >= at:
# Un-marked here since they watched it there: theirs, as a state they set is.
kept.append(row_id)
else:
todo[video_id] = (row_id, at)
marked: set[str] = set()
items = list(todo.items())
now = datetime.now(timezone.utc)
for start in range(0, len(items), _TAKEOUT_BATCH):
stmt = pg_insert(VideoState).values(
[
{"user_id": user.id, "video_id": vid, "status": "watched", "watched_at": at}
for vid, (_, at) in items[start : start + _TAKEOUT_BATCH]
]
)
marked.update(
db.execute(
stmt.on_conflict_do_update(
constraint="uq_user_video",
set_={
"status": "watched",
"watched_at": stmt.excluded.watched_at,
"updated_at": now,
},
where=(VideoState.status == "new")
& (VideoState.updated_at < stmt.excluded.watched_at),
).returning(VideoState.video_id)
).scalars()
)
kept += [row_id for vid, (row_id, _) in items if vid not in marked]
_settle(db, [todo[vid][0] for vid in marked], TakeoutOutcome.WATCHED)
_settle(db, kept, TakeoutOutcome.KEPT)
_settle(db, short, TakeoutOutcome.SHORT)
db.flush()
return {"watched": len(marked), "kept": len(kept), "shorts": len(short)}
def apply_takeout_watch_later(
db: Session, user: User, rows: list[tuple[int, str]], asked: set[str]
) -> dict[str, int]:
"""Put the videos of the user's waiting Watch later `rows`, (row id, video id) oldest first, into
their list here, after what it already holds. `asked` is the videos YouTube was just asked for: one
of those the catalog still lacks is gone from YouTube; any other unknown one keeps waiting."""
if not rows:
return {}
lock_takeout(db, user)
ids = [vid for _, vid in rows]
known = queries.known_video_ids(db, ids)
pl = watch_later(db, user)
# What add_item does a video at a time, for the whole list at once: one presence read and one
# insert, not four statements a video (a full list is thousands). Locked before the read, as there.
next_position = next_item_position(db, pl.id)
held = queries.in_playlist(db, pl.id, [vid for vid in ids if vid in known])
done: dict[TakeoutOutcome, list[int]] = {o: [] for o in TakeoutOutcome}
for row_id, vid in rows:
if vid not in known:
if vid in asked:
done[TakeoutOutcome.MISSING].append(row_id)
elif vid in held:
done[TakeoutOutcome.PRESENT].append(row_id)
else:
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=next_position))
next_position += 1
held.add(vid)
done[TakeoutOutcome.ADDED].append(row_id)
if done[TakeoutOutcome.ADDED]:
touch(pl)
for outcome, ids in done.items():
_settle(db, ids, outcome)
db.flush()
return {
"watch_later_added": len(done[TakeoutOutcome.ADDED]),
"watch_later_present": len(done[TakeoutOutcome.PRESENT]),
"watch_later_missing": len(done[TakeoutOutcome.MISSING]),
}
# --- channels watched but not followed ---------------------------------------------------------
def record_takeout_lengths(
db: Session, user: User, asked: list[str], lengths: dict[str, int | None]
) -> None:
"""Keep what YouTube said of the user's watched videos `asked` about: each one's length, or none
for a video it no longer has (`lengths` leaves it out). Every asked row is marked asked, so a
video YouTube lost is not asked about again."""
if not asked:
return
lock_takeout(db, user) # these rows are an upload's to update too
now = datetime.now(timezone.utc)
table = cast(Table, TakeoutItem.__table__)
db.execute(
table.update()
.where(
owned_by(TakeoutItem.user_id, user),
table.c.kind == TakeoutKind.WATCHED,
table.c.video_id == bindparam("vid"),
)
.values(looked_up_at=now, duration_seconds=bindparam("seconds")),
[{"vid": vid, "seconds": lengths.get(vid)} for vid in asked],
)
db.flush()
def dismiss_watched_channel(db: Session, user: User, channel_id: str) -> None:
"""Put a channel out of the user's "watched but not followed" list (again is a no-op)."""
db.execute(
pg_insert(TakeoutChannelDismissal)
.values(user_id=user.id, channel_id=channel_id)
.on_conflict_do_nothing(constraint="uq_takeout_channel_dismissal")
)
db.flush()
def restore_watched_channel(db: Session, user: User, channel_id: str) -> None:
"""Let a channel the user put out of the list back on it."""
db.execute(
delete(TakeoutChannelDismissal).where(
owned_by(TakeoutChannelDismissal.user_id, user),
TakeoutChannelDismissal.channel_id == channel_id,
)
)
db.flush()
+497
View File
@@ -0,0 +1,497 @@
"""A Google Takeout export, imported: what the user watched on YouTube becomes watched here, and their
YouTube Watch later list joins theirs. The YouTube API reads neither of them, so the export is the
only way in.
Two files of the export are read, each uploaded on its own: `history/watch-history.json` and
`playlists/Watch later-videos.csv`. Their videos are kept (`TakeoutItem`) and the import job applies
what it can; a video the catalog does not know yet waits for it. The watched videos' channels also
make the list of channels the user watches but does not follow (`queries.watched_channels`): the
job asks YouTube how long those videos are, and what the channels are, for it.
The format, measured on a real export (2026-10): the history is a JSON list, one entry per view,
newest first. A view of a video has a `titleUrl` with `watch?v=<id>` (on www.youtube.com or
music.youtube.com) and a `time`, with or without milliseconds. The channel is the first `subtitles`
entry's url, absent for a video since removed. An ad is an entry whose `details` say "From Google
Ads". The title is in the account's language, so nothing here reads it. A Short is a plain
`watch?v=` view, told apart only once the catalog has probed the video. The Watch later csv is a
header row (in the account's language) and `<id>,<ISO time added>` rows.
"""
import codecs
import csv
import io
import json
import re
import threading
from collections.abc import Iterator
from contextlib import contextmanager
from dataclasses import dataclass, field
from datetime import datetime
from itertools import chain
from sqlalchemy.orm import Session
from app import logs, quota
from app.bodylimit import TAKEOUT_MAX_BYTES
from app.db import SessionLocal
from app.models import User
from app.ratelimit import RateLimiter
from app.reader import queries, service
from app.reader.models import TakeoutKind
from app.sync.runner import get_service_user
from app.sync.subscriptions import describe_channels
from app.sync.videos import ensure_videos, parse_iso8601_duration
from app.youtube.client import YouTubeClient, YouTubeError
log = logs.get_logger(__name__)
# The largest file the upload takes (its edge cap and the route's own are one number).
MAX_BYTES = TAKEOUT_MAX_BYTES
# The most videos a Watch later file may hold: YouTube's own cap on one playlist. Each unknown one is
# looked up on the shared API budget, so a longer list is not one YouTube made.
WATCH_LATER_MAX = 5000
# The most Takeout videos one user keeps: twenty times the real export measured here (~9,000 videos
# across both files), so it only stops uploads of made-up ones piling up for good.
ROWS_MAX = 200_000
_VIDEO_ID = re.compile(r"^[A-Za-z0-9_-]{11}$")
_WATCH_URL = re.compile(
r"^https?://(?:www\.|music\.|m\.)?youtube\.com/watch\?v=([A-Za-z0-9_-]{11})"
)
_CHANNEL_URL = re.compile(r"/channel/(UC[A-Za-z0-9_-]{22})")
class TakeoutError(ValueError):
"""The file is not one the import reads; `reason` is the client's word for why."""
def __init__(self, reason: str):
super().__init__(reason)
self.reason = reason
@dataclass
class Parsed:
"""One file's videos, `{video id: (time, channel id)}`, and what it left out, counted."""
kind: TakeoutKind
videos: dict[str, tuple[datetime, str | None]] = field(default_factory=dict)
entries: int = 0
ads: int = 0
other: int = 0 # not a view of a video (a post, a story) or a video with no link left
malformed: int = 0
def _time(raw: object) -> datetime | None:
if not isinstance(raw, str):
return None
try:
at = datetime.fromisoformat(raw.replace("Z", "+00:00"))
# silent-ok: the row is counted as malformed, and the count is in the upload's answer.
except ValueError:
return None
return at if at.tzinfo is not None else None
def _keep(parsed: Parsed, vid: str, at: datetime, channel: str | None) -> None:
old = parsed.videos.get(vid)
if old is None or at > old[0]:
parsed.videos[vid] = (at, channel or (old[1] if old else None))
def _history(data: Iterator[object]) -> Parsed:
parsed = Parsed(TakeoutKind.WATCHED)
for entry in data:
parsed.entries += 1
if not isinstance(entry, dict):
parsed.malformed += 1
continue
details = entry.get("details")
if isinstance(details, list) and any(
isinstance(d, dict) and d.get("name") == "From Google Ads" for d in details
):
parsed.ads += 1
continue
url = entry.get("titleUrl")
match = _WATCH_URL.match(url) if isinstance(url, str) else None
if match is None:
parsed.other += 1
continue
at = _time(entry.get("time"))
if at is None:
parsed.malformed += 1
continue
channel = None
subtitles = entry.get("subtitles")
if isinstance(subtitles, list) and subtitles and isinstance(subtitles[0], dict):
found = _CHANNEL_URL.search(str(subtitles[0].get("url", "")))
channel = found.group(1) if found else None
_keep(parsed, match.group(1), at, channel)
return parsed
def _watch_later(text: str) -> Parsed:
parsed = Parsed(TakeoutKind.WATCH_LATER)
rows = (row for row in csv.reader(io.StringIO(text)) if any(cell.strip() for cell in row))
first = next(rows, None)
if first is None or len(first) < 2:
raise TakeoutError("not_takeout")
# The first row is the header, in the account's language, unless it is already a video.
for row in chain([first], rows) if _VIDEO_ID.match(first[0].strip()) else rows:
parsed.entries += 1
vid = row[0].strip()
at = _time(row[1].strip()) if len(row) > 1 else None
if not _VIDEO_ID.match(vid) or at is None:
parsed.malformed += 1
continue
_keep(parsed, vid, at, None)
if len(parsed.videos) > WATCH_LATER_MAX:
raise TakeoutError("too_many")
return parsed
def _skip(text: str, at: int) -> int:
"""The first index at or after `at` that is not JSON whitespace."""
while at < len(text) and text[at] in " \t\n\r":
at += 1
return at
# How much of the file is decoded at a time. Decoded whole, the text costs up to four bytes a
# character (one emoji in a title is enough), several times the file on top of its bytes.
_WINDOW = 1024 * 1024
class _Window:
"""The file's text, decoded a window at a time: `text` holds what is not read yet from `at` on.
The bytes are the caller's, so only the window is this reader's memory."""
def __init__(self, body: bytes, start: int):
self._body, self._next = body, start
self._decoder = codecs.getincrementaldecoder("utf-8")() # strict: bad UTF-8 raises
self.text, self.at = "", 0
@property
def done(self) -> bool:
return self._next >= len(self._body)
def more(self, at_least: int = 0) -> None:
"""Decode the next window (at least `at_least` bytes), dropping what is read already."""
self.text, self.at = self.text[self.at :], 0
size = max(_WINDOW, at_least)
chunk = self._body[self._next : self._next + size]
self._next += len(chunk)
self.text += self._decoder.decode(chunk, final=self.done)
def skip(self) -> None:
"""Past the whitespace, reading on as far as it goes."""
while True:
self.at = _skip(self.text, self.at)
if self.at < len(self.text) or self.done:
return
self.more()
def startswith(self, char: str) -> bool:
self.skip()
return self.text.startswith(char, self.at)
def _decode(decoder: json.JSONDecoder, text: str, at: int) -> tuple[object, int] | None:
"""The JSON value at `at` and where it ends, or None where none reads (yet)."""
try:
return decoder.raw_decode(text, at)
# silent-ok: not a failure here — the caller reads on, or refuses the file once it has all.
except json.JSONDecodeError:
return None
def _json_list(body: bytes, start: int) -> Iterator[object]:
"""The entries of the JSON list at byte `start` (its "["), one at a time: the whole tree is never
built, nor the whole text, so a file of a million tiny entries costs a window and one entry, not
a million dicts at once."""
decoder = json.JSONDecoder()
window = _Window(body, start + 1) # past the "[" the caller saw
if window.startswith("]"):
window.at += 1
else:
while True:
window.skip()
read = _decode(decoder, window.text, window.at)
# An entry cut by the window's edge fails, or (a number) reads short: read on and try
# again, with a window twice what is left, so one entry larger than any window is read
# in a few tries, not one a window.
if read is None or (read[1] == len(window.text) and not window.done):
if window.done:
raise ValueError("not a JSON list")
window.more(2 * (len(window.text) - window.at))
continue
entry, window.at = read
yield entry
if window.startswith(","):
window.at += 1
elif window.startswith("]"):
window.at += 1
break
else:
raise ValueError("not a JSON list")
window.skip()
if window.at != len(window.text):
raise ValueError("text after the JSON list")
# The largest Watch later file read: YouTube's longest list is some 200 KB of rows, so a file ten
# times that is not one — and, unlike the history, it is decoded whole.
CSV_MAX_BYTES = 2 * 1024 * 1024
_BOM = codecs.BOM_UTF8
def parse(body: bytes) -> Parsed:
"""Read an uploaded Takeout file: the watch history (JSON) or the Watch later list (csv), told
apart by their content, not their name. Raises `TakeoutError` for anything else, and for a file
with nothing to import in it."""
start = len(_BOM) if body.startswith(_BOM) else 0
while start < len(body) and body[start] in b" \t\n\r\f\v":
start += 1
first = body[start : start + 1]
if first in (b"{", b"<"):
# Another JSON file of the export, or the history's HTML variant.
raise TakeoutError("not_takeout")
if first != b"[" and len(body) - start > CSV_MAX_BYTES:
raise TakeoutError("too_many")
try:
# A bad UTF-8 sequence is a ValueError too (UnicodeDecodeError), read below as not a file.
parsed = (
_history(_json_list(body, start))
if first == b"["
else _watch_later(body[start:].decode("utf-8"))
)
# A file the import cannot read, however it fails: bad JSON (a ValueError, as is a number too
# long to convert), nesting deeper than the decoder recurses, a csv field past csv's limit.
except (ValueError, RecursionError, csv.Error) as exc:
if isinstance(exc, TakeoutError):
raise
raise TakeoutError("not_takeout") from None
if not parsed.videos:
raise TakeoutError("no_videos")
return parsed
# One import per user at a time: a second file uploaded while the first one's run is under way runs
# after it (the scheduler's ad-hoc queue), but the Shorts job settles the same users' rows on a
# thread of its own, and the two would settle the same rows. One API process runs the scheduler, so a
# lock in this module holds it (as the channel backfill's does).
_user_locks: dict[int, threading.Lock] = {}
_user_locks_guard = threading.Lock()
@contextmanager
def _one_import(user_id: int) -> Iterator[None]:
with _user_locks_guard:
lock = _user_locks.setdefault(user_id, threading.Lock())
with lock:
yield
def run_target(user_id: int) -> str:
"""The import run's target, `job_runs.target`: one user's import."""
return f"user:{user_id}"
def target_title(db: Session, target: str) -> str | None:
"""Who an import run's target is, for the admin's run lists: the user's name, else their email."""
_, _, raw = target.partition("user:")
user = db.get(User, int(raw)) if raw.isdigit() else None
return (user.display_name or user.email) if user is not None else None
class TooMany(Exception):
"""The upload would take the user past `ROWS_MAX` kept videos."""
def stage(db: Session, user: User, parsed: Parsed) -> None:
"""Keep an uploaded file's videos for the import, unless that would take the user past
`ROWS_MAX` (`TooMany`). The caller commits."""
service.lock_takeout(db, user) # the count and the insert are one user's, not two uploads' each
if (
queries.takeout_rows_besides(db, user, parsed.kind, parsed.videos) + len(parsed.videos)
> ROWS_MAX
):
raise TooMany
service.stage_takeout(db, user, parsed.kind, parsed.videos)
# How many unknown Watch later videos one user may have looked up on the shared API budget in a day:
# two full Watch later lists (two units a hundred ids). The rest wait for the next day's runs. Both
# shares pace the work, they do not bound its cost: they are held in memory, so a restart starts
# them afresh, which only brings the rest forward — every video is asked about once (a looked-up
# row is marked), and `quota.backfill_reserve_reached` still guards the shared budget.
_lookups = RateLimiter(max_events=2 * WATCH_LATER_MAX, window_seconds=24 * 3600)
# How many watched videos' lengths one user may ask about in a day for the "watched but not followed"
# list (a unit a fifty). A share of its own: a large history asks for thousands, and on the Watch
# later share it would leave the export's Watch later file waiting a day behind it.
_lengths = RateLimiter(max_events=2 * WATCH_LATER_MAX, window_seconds=24 * 3600)
def _put_in_watch_later(db: Session, user: User) -> dict:
"""Put the user's waiting Watch later videos into theirs here: the ones the catalog knows at once,
the others once YouTube was asked for them — when a YouTube-connected account can ask, today's
budget is above the backfill reserve, and the user's own daily share of lookups allows it. A video
not asked about yet waits for a later run. The lookup commits what it fetched; the caller commits
the rest."""
waiting = queries.takeout_waiting(db, user, TakeoutKind.WATCH_LATER)
if not waiting:
return {}
ids = [vid for _, vid in waiting]
known = queries.known_video_ids(db, ids)
ask = [vid for vid in ids if vid not in known][:WATCH_LATER_MAX]
asked: set[str] = set()
fetcher = get_service_user(db) if ask else None
if (
fetcher is not None
and not quota.backfill_reserve_reached()
and _lookups.room(str(user.id)) >= len(ask)
):
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_LOOKUP):
with YouTubeClient(db, fetcher) as yt:
ensure_videos(db, yt, ask)
# Counted once done: a lookup that failed spent none of the share (one run per user at a
# time, so the room asked above is still there).
_lookups.allow(str(user.id), cost=len(ask))
asked = set(ask)
# Only the rows read above: one a second upload staged meanwhile was never asked about.
return service.apply_takeout_watch_later(db, user, waiting, asked)
def _ask_lengths(db: Session, user: User) -> dict:
"""Ask YouTube how long the user's watched videos are that the "watched but not followed" list
still needs, and keep the answer — on the same terms as the Watch later lookups, on a daily
share of its own. What is not asked yet (no YouTube-connected account, the reserve reached, the
share spent) waits for a later run, and the list says it is not complete. A YouTube failure is
logged and left for a later run too: it is no reason to fail the import it follows. The caller
commits."""
ask = queries.takeout_lengths_to_ask(db, user)
fetcher = get_service_user(db) if ask else None
if fetcher is None or quota.backfill_reserve_reached():
return {}
ask = ask[: int(_lengths.room(str(user.id)))]
if not ask:
return {}
try:
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_LOOKUP):
with YouTubeClient(db, fetcher) as yt:
items = yt.get_videos(ask)
except YouTubeError as exc:
log.warning(
"Takeout: asking the watched videos' lengths failed: %s",
exc,
extra=logs.event("takeout.lengths.failed", user_id=user.id, reason=exc.reason),
)
return {}
_lengths.allow(str(user.id), cost=len(ask))
lengths = {
it["id"]: parse_iso8601_duration(it.get("contentDetails", {}).get("duration"))
for it in items
if it.get("id")
}
service.record_takeout_lengths(db, user, ask, lengths)
return {"lengths_asked": len(ask)}
def _describe_channels(db: Session, user: User) -> dict:
"""Bring the channels the user's list holds into the catalog with their details (name, picture,
description), the ones it lacks: without them the list cannot show a channel. A YouTube failure
is logged and left for a later run. The caller commits."""
need = queries.watched_channels_to_describe(db, user)
fetcher = get_service_user(db) if need else None
if fetcher is None or quota.backfill_reserve_reached():
return {}
try:
with quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER):
with YouTubeClient(db, fetcher) as yt:
describe_channels(db, yt, need)
except YouTubeError as exc:
db.rollback()
log.warning(
"Takeout: describing the watched channels failed: %s",
exc,
extra=logs.event("takeout.channels.failed", user_id=user.id, reason=exc.reason),
)
return {}
return {"channels_described": len(need)}
def _channel_list(db: Session, user: User) -> dict:
"""The "watched but not followed" list's part of a run: the lengths, landed, then the channels.
Commits each."""
result = _ask_lengths(db, user)
db.commit()
result |= _describe_channels(db, user)
db.commit()
return result
def run_import(user_id: int) -> dict:
"""The import job: apply what the user's kept Takeout rows can say now, as one recorded run.
The watched part commits before the Watch later part asks YouTube for the videos the catalog
lacks, so a YouTube failure there loses nothing already marked; the "watched but not followed"
list's lookups come last. Returns the counts."""
with _one_import(user_id), SessionLocal() as db:
user = db.get(User, user_id)
if user is None:
return {"skipped": "no user"}
result = service.apply_takeout_watched(db, user)
db.commit()
result |= _put_in_watch_later(db, user)
db.commit()
result |= _channel_list(db, user)
outcomes = queries.takeout_outcomes(db, user)
result["waiting"] = outcomes.get((TakeoutKind.WATCHED, None), 0)
if outcomes.get((TakeoutKind.WATCH_LATER, None)):
result["watch_later_waiting"] = outcomes[(TakeoutKind.WATCH_LATER, None)]
return result
def apply_waiting(db: Session) -> dict:
"""Settle what every user's import left waiting and can be settled now, one commit each: a
watched video the catalog has come to know and the Shorts probe has cleared (by its job or a
search's), the Watch later videos a run could not put in yet, and what the "watched but not
followed" list still needs asked of YouTube. Run after each run of the
`shorts` job, which is what clears the most of them. Returns its counts, or an empty dict when
nothing was waiting that could move."""
marked = added = asked = failed = 0
ready = set(queries.takeout_users_ready(db))
later = set(queries.takeout_users_watch_later_waiting(db))
lengths = set(queries.takeout_users_with_list_work(db))
for user_id in sorted(ready | later | lengths):
user = db.get(User, user_id)
if user is None:
continue
# One user's failure (a lookup YouTube refused) is theirs: it neither fails the job that
# runs this nor keeps the users after them waiting. What it did not settle waits.
try:
with _one_import(user_id):
if user_id in ready:
marked += service.apply_takeout_watched(db, user)["watched"]
db.commit()
if user_id in later:
added += _put_in_watch_later(db, user).get("watch_later_added", 0)
db.commit()
if user_id in lengths:
asked += _channel_list(db, user).get("lengths_asked", 0)
except Exception:
db.rollback()
failed += 1
log.exception(
"Takeout settling failed for user %s",
user_id,
extra=logs.event("takeout.settle.failed", user_id=user_id),
)
out = {}
if ready:
out["takeout_watched"] = marked
if added:
out["takeout_watch_later_added"] = added
if asked:
out["takeout_lengths_asked"] = asked
if failed:
out["takeout_failed"] = failed
return out
+8
View File
@@ -124,6 +124,14 @@ async def capped_body(request: Request, limit: int) -> bytes:
return bytes(body)
async def drain_body(request: Request) -> None:
"""Read the rest of the body to nowhere, before a refusal is raised: answered mid-upload, the
client reports a network failure, not the refusal (`capped_stream_to` says why). `bodylimit`
bounds how much there can be."""
async for _ in request.stream():
pass
class ByteSink(Protocol):
"""Where `capped_stream_to` writes: a file, or anything that takes the bytes on their way to one."""
+3 -3
View File
@@ -307,9 +307,9 @@ def explore_channel(
"""Open an un-subscribed channel for browsing. Records the exploration (a per-user,
grace-clocked row that gates this user's access to the channel's ephemeral videos and
drives the later cleanup), flags the channel ephemeral unless someone organically follows
it, then ingests one page of its uploads (newest-first) marked via_explore + enriched.
Returns the next page token so the page can "load more". Guarded against the shared
backfill quota reserve."""
it, then ingests one page of its uploads (newest-first), enriched — marked via_explore while
nobody follows it. Returns the next page token so the page can "load more". Guarded against
the shared backfill quota reserve."""
channel = _channel_or_404(db, channel_id)
if queries.is_blocked(db, user, channel_id):
raise refused(403, "channel_blocked")
+229 -6
View File
@@ -1,19 +1,31 @@
import threading
from collections.abc import AsyncIterator
from datetime import date
from functools import partial
from fastapi import APIRouter, Depends, Query
from fastapi import APIRouter, Depends, Path, Query, Request
from pydantic import BaseModel
from sqlalchemy.orm import Session
from sqlalchemy.orm.exc import StaleDataError
from app import quota
from app import jobruns, quota, scheduler
from app.catalog import queries as catalog_queries, service as catalog_service
from app.auth import current_user
from app.db import get_db
from app.db import get_db, release_read_connection
from app.mailtext import lang_for_user
from app.ratelimit import RateLimiter
from app.feedquery import BadCursor, facet_counts, feed_count, feed_page, feed_summary, feed_years
from app.models import User
from app.reader import queries, service
from app.routes._common import ProgressIn, kept_position, not_found, youtube_refusal
from app.sync.videos import parse_iso8601_duration
from app.reader import queries, service, takeout
from app.routes._common import (
ProgressIn,
capped_body,
drain_body,
kept_position,
not_found,
youtube_refusal,
)
from app.sync.videos import ensure_videos, parse_iso8601_duration
from app.utils import iso
from app.youtube.client import YouTubeClient, YouTubeError
from app.refusal import refused
@@ -293,3 +305,214 @@ def get_video_detail(
items[0].get("contentDetails", {}).get("duration")
),
}
# --- Google Takeout ------------------------------------------------------------------------------
# One upload at a time per user, ten an hour, and two at once across every user: a file's bytes are
# held whole while it is read (its text a window at a time), so these bound what the accounts can
# make the process hold. The section uploads its files one after another.
_takeout_limiter = RateLimiter(max_events=10, window_seconds=3600)
_takeout_uploading: set[int] = set()
_takeout_uploading_guard = threading.Lock()
_TAKEOUT_AT_ONCE = 2
async def _takeout_body(
request: Request, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> AsyncIterator[bytes]:
# The sign-in check has run, and it was a SELECT: hand the connection back before a file of
# megabytes arrives, so a slow upload does not hold one `idle in transaction`. The user's id is
# taken first: the release expires the user, and reading it after would reload it, taking a
# connection straight back out.
uid = user.id
release_read_connection(db)
with _takeout_uploading_guard:
busy = uid in _takeout_uploading
full = not busy and len(_takeout_uploading) >= _TAKEOUT_AT_ONCE
taken = not busy and not full
if taken:
_takeout_uploading.add(uid)
if not taken or not _takeout_limiter.allow(str(uid)):
if taken:
with _takeout_uploading_guard:
_takeout_uploading.discard(uid)
# Read to the end before answering: refused mid-upload, the browser reports a network
# failure instead of the refusal (capped_stream_to says why).
await drain_body(request)
if busy:
raise refused(429, "takeout_busy")
if full:
raise refused(429, "takeout_server_busy")
raise refused(429, "takeout_cooldown")
try:
yield await capped_body(request, takeout.MAX_BYTES)
finally:
with _takeout_uploading_guard:
_takeout_uploading.discard(uid)
@router.post("/takeout")
def upload_takeout(
# The sign-in check FIRST: the body's own dependency needs the signed-in user, so a stranger's
# file is refused before a byte of it is read.
user: User = Depends(current_user),
body: bytes = Depends(_takeout_body),
db: Session = Depends(get_db),
):
"""One file of the user's Google Takeout export — the watch history or the Watch later list.
Its videos are kept and the import runs in the background; its outcome reaches the user's inbox."""
try:
parsed = takeout.parse(body)
except takeout.TakeoutError as exc:
if exc.reason == "no_videos":
raise refused(422, "takeout_no_videos") from None
if exc.reason == "too_many":
raise refused(422, "takeout_list_too_long", limit=takeout.WATCH_LATER_MAX) from None
raise refused(422, "takeout_unreadable") from None
try:
takeout.stage(db, user, parsed)
except takeout.TooMany:
db.rollback()
raise refused(422, "takeout_too_many", limit=takeout.ROWS_MAX) from None
db.commit()
scheduler.run_ad_hoc(
scheduler.TAKEOUT_IMPORT,
partial(takeout.run_import, user.id),
actor_id=user.id,
target=takeout.run_target(user.id),
notify=True,
)
return {
"ok": True,
"kind": parsed.kind,
"videos": len(parsed.videos),
"entries": parsed.entries,
"ads": parsed.ads,
"other": parsed.other,
"malformed": parsed.malformed,
}
@router.get("/takeout")
def takeout_status(user: User = Depends(current_user), db: Session = Depends(get_db)):
"""What the user's Takeout import has done: its last run (none before the first upload), and
every video kept from the export, by kind and outcome — a video the catalog did not know yet
counts as "waiting" until it does, and is marked then."""
target = takeout.run_target(user.id)
run = jobruns.last_run(db, scheduler.TAKEOUT_IMPORT, target)
return {
# Queued and not started: its run row is not written yet, and the last run is still the
# previous one, so the page waits for it rather than taking that one for this upload's.
"queued": scheduler.ad_hoc_queued(scheduler.TAKEOUT_IMPORT, target),
"max_bytes": takeout.MAX_BYTES,
"run": None
if run is None
else {
"status": run.status,
"started_at": iso(run.started_at),
"finished_at": iso(run.finished_at),
"result": run.result,
"failure": run.failure,
},
**queries.takeout_summary(db, user),
}
# --- channels watched but not followed -----------------------------------------------------------
@router.get("/takeout/channels")
def watched_channels(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
"""The channels the user's Takeout history shows them watching often but not following: each
with how many of its videos over five minutes they watched, and when last. `pending` is how many
more channels may join once the import has asked YouTube for their videos' lengths or their
names, so a list still being made does not read as the whole of it; `imported` tells "no history
uploaded" from "nothing to suggest"."""
return {
"imported": queries.has_watch_history(db, user),
"pending": queries.watched_channels_pending(db, user),
"channels": [
{
"id": ch.id,
"title": ch.title,
"handle": ch.handle,
"thumbnail_url": ch.thumbnail_url,
"subscriber_count": ch.subscriber_count,
"video_count": ch.video_count,
"description": ch.description,
"watched": int(n),
"last_watched_at": iso(last),
}
for ch, n, last in queries.watched_channels(db, user)
],
}
@router.post("/takeout/channels/{channel_id}/videos")
def watched_channel_videos(
channel_id: str, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""The videos of a channel that the user's list counts, ready to play, last watched first. The
channel is explored for the user (as its page does), and the videos the catalog lacks come in as
the exploration's: the player needs them in the catalog, the shared Library does not get them,
and the cleanup reclaims them once nobody keeps them. Of a channel someone follows they come in
as the catalog's own, as its uploads do. `gone`: how many YouTube no longer has."""
channel = catalog_queries.channel(db, channel_id)
if channel is None:
raise not_found("channel")
if catalog_queries.is_blocked(db, user, channel_id):
raise refused(403, "channel_blocked")
counted, gone = queries.watched_channel_videos(db, user, channel_id)
ids = [vid for vid, _ in counted]
known = queries.known_video_ids(db, ids)
missing = [vid for vid in ids if vid not in known]
if missing and quota.backfill_reserve_reached():
raise refused(429, "quota_reserve")
catalog_service.explore(db, user, channel)
db.commit()
if missing:
try:
with (
quota.attribute(user.id, quota.QuotaAction.CHANNELS_EXPLORE),
YouTubeClient(db, user) as yt,
):
ensure_videos(db, yt, missing, explore=channel.from_explore)
except YouTubeError as exc:
raise youtube_refusal(exc) from exc
catalog_service.settle_explored(db, channel)
db.commit()
items = queries.feed_rows_by_ids(db, user, ids)
watched_at = dict(counted)
for item in items:
item["takeout_watched_at"] = iso(watched_at[item["id"]])
# A video YouTube still had when its length was asked may have gone since.
return {"ok": True, "items": items, "gone": gone + len(ids) - len(items)}
@router.put("/takeout/channels/{channel_id}/dismissal")
def dismiss_watched_channel(
channel_id: str = Path(max_length=32),
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Put a channel out of the user's list (it stays out until restored). Only one their own history
shows them watching: a made-up id would only pile up rows every list read then has to skip."""
if not queries.watched_in_history(db, user, channel_id):
raise not_found("channel")
service.dismiss_watched_channel(db, user, channel_id)
db.commit()
return {"ok": True, "channel_id": channel_id}
@router.delete("/takeout/channels/{channel_id}/dismissal")
def restore_watched_channel(
channel_id: str = Path(max_length=32),
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Let a channel the user put out of the list back on it."""
service.restore_watched_channel(db, user, channel_id)
db.commit()
return {"ok": True, "channel_id": channel_id}
+4
View File
@@ -79,6 +79,10 @@ def get_me(
# Which live-search source runs, so the search button can say what a search costs: "scrape"
# spends no API quota, "api" spends 100 units a page.
"search_source": sysconfig.get_str(db, "search_source"),
# Whether live search and related videos are on: an admin turns either off with a daily
# limit of 0, and the UI then leaves out what would only be refused.
"search_enabled": sysconfig.get_int(db, "search_daily_limit_per_user") > 0,
"related_enabled": sysconfig.get_int(db, "related_daily_limit_per_user") > 0,
"can_read": has_read_scope(user),
"can_write": has_write_scope(user),
"pending_invites": pending_invites,
+6 -2
View File
@@ -3,7 +3,7 @@
Admin-only and read-only — rows leave the table through the retention job
(`job_run_retention_days`). Filtered on the server and paged by `_keyset`, like the server log and
the audit log beside it on the activity page. A row names what a reader needs without a second
request: the channel a user's ad-hoc run worked on, and how many log lines the run left (only
request: what a user's ad-hoc run worked on (the channel, the importing user), and how many log lines the run left (only
WARNING and up are stored, so most runs have none — the count says so before the click).
"""
@@ -116,8 +116,12 @@ def list_runs(
limit,
lambda r: r.started_at,
)
channels = {r.target for r in runs if r.job in scheduler.AD_HOC_JOBS and r.target}
channels = {r.target for r in runs if r.job == scheduler.CHANNEL_BACKFILL and r.target}
titles = catalog_queries.channel_titles(db, channels)
for target in {r.target for r in runs if r.job == scheduler.TAKEOUT_IMPORT and r.target}:
title = scheduler.ad_hoc_target_title(db, scheduler.TAKEOUT_IMPORT, target)
if title is not None:
titles[target] = title
lines: dict[int, int] = {}
if runs:
counted = db.execute(
+303 -96
View File
@@ -13,10 +13,18 @@ this search-discovered "noise" by default. Shorts and currently-live/upcoming vi
ingested this way (Shorts are confirmed by the youtube.com/shorts probe; live/upcoming are
dropped from the search response up front).
A video's related videos (`/related/{video_id}`, the list YouTube shows beside it) come in the same
way: read from the website (`related_scrape`), ingested as search finds are, under their own daily
cap — with what this user has already seen left out.
This is the HTTP side — the paging, the caps and the refusals; what a page of results leaves in the
catalog is `catalog/search`."""
from fastapi import APIRouter, Depends
import threading
from collections.abc import Callable
from typing import Literal
from fastapi import APIRouter, Depends, Path
from pydantic import BaseModel
from sqlalchemy.orm import Session
@@ -26,8 +34,10 @@ from app.catalog import queries
from app.catalog import search as catalog_search
from app.db import get_db
from app.models import User
from app.ratelimit import RateLimiter
from app.reader import queries as reader_queries
from app.youtube import search_scrape
from app.youtube import related_scrape, search_scrape
from app.youtube.innertube import ScrapeError
from app.youtube.client import YouTubeClient, YouTubeError
from app.refusal import refused
from app.routes._common import youtube_refusal
@@ -37,6 +47,115 @@ router = APIRouter(prefix="/api/search", tags=["search"])
# search.list costs this many quota units per page.
SEARCH_COST = 100
# The scrape source reads at most this many pages for one search.
SEARCH_MAX_PAGES = 12
# A related list is filled to this many unseen videos, reading at most this many pages (about 20
# videos each, measured) — the ones the user has seen are left out, so a page alone can fall short.
RELATED_TARGET = 20
RELATED_MAX_PAGES = 4
# The locale YouTube fills a related list in: on the video's topic, or with what trends in Hungary.
RELATED_REGIONS = {"topical": ("en", "US"), "hu": ("hu", "HU")}
# A request that fails outright gives its slot back, but its shape is the user's (a cursor is a
# free string): one built to fail would read YouTube for nothing, without end. So only this many
# refunds an hour per user and list; past them a failure keeps its slot, as a success does.
REFUNDS_PER_HOUR = 5
_refunds = RateLimiter(max_events=REFUNDS_PER_HOUR, window_seconds=3600)
# The API source's budget check and the search.list call it pays for run one at a time: checked
# apart, searches sent together at the last 100 units all passed, and all spent. One process
# serves the API (a single uvicorn worker, as `ratelimit` says), so a process lock holds them all.
# A search waits this long for it, then is refused as busy: each waiter holds a request worker, and
# a slow YouTube (30 s a call at worst) would otherwise park them all behind one another.
_api_search = threading.Lock()
API_SEARCH_WAIT_SECONDS = 10.0
def _gather(
db: Session,
yt: YouTubeClient,
user: User,
*,
cursor: str | None,
max_pages: int,
fetch: Callable[[str | None], dict],
candidates_of: Callable[[list[dict], list[str]], list[dict]],
enough: Callable[[list[str]], bool],
scrape_failed: Callable[[Exception], None],
ingest_failed: Callable[[int, int], None],
) -> tuple[list[str], str | None]:
"""Page through a YouTube list into the catalog: fetch a page, keep its `candidates_of`, ingest
them, and go on from its continuation until `enough` or `max_pages`. Returns the kept ids and
the cursor to read further from. A page that fails after earlier ones gathered something ends
the list with what it has; a first page that fails raises the 422 refusal.
422, not 502: this is a JSON endpoint, and the FE treats 502/503/504 as a gateway dropout
(auto-retry + "connection lost" banner — lib/api/core RETRIABLE_GATEWAY), which hides the real
message and repeats the work. A Data API failure is logged where it happened; the scraper's is
not, and its text does not ride the answer — so `scrape_failed` says it. An ingest (DB writes +
enrich) fails the same way, guarded for any exception so a write or shape failure is a 422 with
a message too, not a bare 500; `ingest_failed` logs it inside the `except`, with the trace."""
collected: list[str] = []
next_cursor = cursor
for _ in range(max_pages):
try:
page = fetch(next_cursor)
except (YouTubeError, ScrapeError) as exc:
if collected:
break # keep what earlier pages already gathered
if isinstance(exc, ScrapeError):
scrape_failed(exc)
raise youtube_refusal(exc) from exc
candidates = candidates_of(page["items"], collected)
try:
collected.extend(catalog_search.ingest_candidates(db, yt, user, candidates))
except Exception as exc: # any ingest failure, not just YouTubeError
db.rollback()
ingest_failed(len(candidates), len(collected))
if collected:
break
raise youtube_refusal(exc) from exc
next_cursor = page["next_page_token"]
if enough(collected) or not next_cursor:
break
return collected, next_cursor
def _give_back(claimed: int, feature: str, user_id: int) -> None:
"""Refund the slot of a request that failed outright, while this user has refunds left."""
if _refunds.allow(f"{feature}:{user_id}"):
quota.refund_action(claimed)
return
log.info(
"a failed request kept its slot: this hour's refunds are spent",
extra=logs.event(f"{feature}.refund_refused", per_hour=REFUNDS_PER_HOUR),
)
def _budget_spent() -> None:
"""Say that a live search was refused because the shared YouTube budget cannot pay for it."""
log.info(
"a live search was refused: the shared YouTube budget is spent",
extra=logs.event("search.refused", reason="shared_quota", cost=SEARCH_COST),
)
def _busy() -> None:
"""Say that a live search was refused because other API searches held the line too long."""
log.info(
"a live search was refused: other searches held the line",
extra=logs.event("search.refused", reason="busy", waited=API_SEARCH_WAIT_SECONDS),
)
def _cap_spent(feature: str, limit: int) -> None:
"""Say that a request was refused for the per-user daily cap. The 429 is in the request line
either way; which of the caps produced it is not, and they are answered by different things —
this one by tomorrow, the shared budget by the admin."""
log.info(
"a request was refused: the per-user daily cap is spent",
extra=logs.event(f"{feature}.refused", reason="daily_cap", limit=limit),
)
@router.get("/youtube")
@@ -59,109 +178,122 @@ def search_youtube(
# Source: "scrape" (InnerTube, no API quota) or "api" (search.list, 100 units/page).
source = sysconfig.get_str(db, "search_source")
# Per-user daily cap applies to both sources (anti-abuse); with scrape it's the only guard
# since the search itself spends no quota.
# Every request takes a slot of the per-user daily cap, a cursor's too (anti-abuse); with
# scrape it's the only guard since the search itself spends no quota. A limit of 0 turns live
# search off on this instance — a refusal tomorrow will not change.
daily_limit = sysconfig.get_int(db, "search_daily_limit_per_user")
used = quota.actions_today(db, user.id, quota.QuotaAction.VIDEOS_SEARCH)
if used >= daily_limit:
# The 429 is in the request line either way; which of the two caps produced it is not, and
# they are answered by different things — this one by tomorrow, the other by the admin.
log.info(
"a live search was refused: the per-user daily cap is spent",
extra=logs.event("search.refused", reason="daily_cap", limit=daily_limit),
)
raise refused(429, "search_daily_limit", limit=daily_limit)
# The 100-unit budget pre-check only applies to the API source.
if daily_limit <= 0:
raise refused(409, "search_disabled")
# The 100-unit budget pre-check only applies to the API source. It is asked again under the
# lock, right before the call (`fetch`); this one refuses before a slot is taken.
if source == "api" and not quota.can_spend(db, SEARCH_COST):
log.info(
"a live search was refused: the shared YouTube budget is spent",
extra=logs.event("search.refused", reason="shared_quota", cost=SEARCH_COST),
)
_budget_spent()
raise refused(429, "quota_spent")
# The check and the take are one step (`quota.claim_action`), whichever the source: checked
# first and logged after, requests sent together at the last free slot all passed — and with
# the API source each of them spent 100 units. A search that fails outright gives it back below.
claimed = quota.claim_action(user.id, quota.QuotaAction.VIDEOS_SEARCH, daily_limit)
if claimed is None:
_cap_spent("search", daily_limit)
raise refused(429, "search_daily_limit", limit=daily_limit)
# Channels this user has blocked are dropped before ingestion — never shown, never stored.
blocked = queries.blocked_ids(db, user)
collected: list[str] = []
next_cursor = cursor
MAX_PAGES = 12 # scrape safety bound
with YouTubeClient(db, user) as yt:
for _ in range(MAX_PAGES):
try:
if source == "api":
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_SEARCH):
page = yt.search_videos(term, page_token=next_cursor)
else:
page = search_scrape.search(term, continuation=next_cursor)
except (YouTubeError, search_scrape.ScrapeError) as exc:
if collected:
break # keep what we already gathered
# 422, not 502: this is a JSON endpoint, and the FE treats 502/503/504 as a gateway
# dropout (auto-retry + "connection lost" banner — lib/api/core RETRIABLE_GATEWAY),
# which hides the real message and burns a second quota-costed search. Upstream failure
# on a JSON endpoint = 422 + message; reserve 5xx for the raw byte-proxies. (B-core-1)
# A Data API failure is logged where it happened; the scraper's is not, and its text
# no longer rides the answer — so it is said here.
if isinstance(exc, search_scrape.ScrapeError):
log.warning(
"the YouTube search scrape failed",
extra=logs.event("search.scrape_failed", error=str(exc)),
)
raise youtube_refusal(exc) from exc
# Whether search.list was paid for: YouTube charges a call it refuses too, so once the client
# recorded its units — or the request went out and no answer came back — the search has spent
# the shared budget. One that never reached YouTube (no connection, no token) has not.
charged = False
# Drop currently-live/upcoming, channel-less, and blocked-channel results.
candidates = [
it
for it in page["items"]
if it["live_broadcast"] == "none"
and it["channel_id"]
and it["channel_id"] not in blocked
]
# Ingest (DB writes + enrich) runs outside the page-fetch try above; guard it too so a
# write/shape failure degrades to a 422 with a message on this JSON endpoint instead of a
# bare 500 hidden behind the FE's generic "search failed" (same rationale as B-core-1).
try:
collected.extend(catalog_search.ingest_candidates(db, yt, user, candidates))
except Exception as exc: # any ingest failure, not just YouTubeError
db.rollback()
# The term is NOT in this line. A search string is exactly what the privacy
# policy keeps out of the log unless the user turned Teljes/TRACE on for
# themselves, and `redact_text` does not cover it — it scrubs addresses and
# credential params, not what someone typed into the search box. What the
# ERROR line carries instead is the shape of the failure, which is what tells
# an ingest bug from an upstream one; the term follows at TRACE, and only
# under this user's own consent — not whenever the process runs at TRACE.
log.exception(
"live search ingest failed",
extra=logs.event(
"search.live.ingest_failed",
source=source,
candidates=len(candidates),
collected=len(collected),
),
)
if logs.consented_trace():
log.log(
logs.TRACE,
"the term the ingest failed on",
# log-pii-ok: only under this user's own Teljes consent
extra=logs.event("search.live.term", term=term),
)
if collected:
break # keep what earlier pages already gathered
raise youtube_refusal(exc) from exc
next_cursor = page["next_page_token"]
# The API source pages cost 100 units each — never auto-page; one page per request.
if source == "api" or len(collected) >= limit or not next_cursor:
break
def fetch(token: str | None) -> dict:
if source != "api":
return search_scrape.search(term, continuation=token)
if not _api_search.acquire(timeout=API_SEARCH_WAIT_SECONDS):
_busy()
raise refused(429, "search_busy")
try:
return paid_search(token)
finally:
_api_search.release()
# Count this as ONE per-user search against the daily cap. The scrape source spends no
# quota, so it never logs an event on its own; the API source already logged exactly one
# (via record_usage — it never auto-pages). Logging once here, after a search that produced
# at least one page (a total failure raises 502 above and never reaches this), keeps the cap
# counting user searches instead of internal continuation pages.
if source != "api":
quota.log_action(user.id, quota.QuotaAction.VIDEOS_SEARCH)
def paid_search(token: str | None) -> dict:
nonlocal charged
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_SEARCH):
if not quota.can_spend(db, SEARCH_COST):
_budget_spent()
raise refused(429, "quota_spent")
spent = yt.spent
try:
page = yt.search_videos(term, page_token=token)
except YouTubeError as exc:
charged = yt.spent > spent or exc.sent
raise
charged = True
return page
def candidates_of(items: list[dict], _collected: list[str]) -> list[dict]:
# Drop currently-live/upcoming, channel-less, and blocked-channel results.
return [
it
for it in items
if it["live_broadcast"] == "none"
and it["channel_id"]
and it["channel_id"] not in blocked
]
def scrape_failed(exc: Exception) -> None:
log.warning(
"the YouTube search scrape failed",
extra=logs.event("search.scrape_failed", error=str(exc)),
)
def ingest_failed(candidates: int, collected: int) -> None:
# The term is NOT in this line. A search string is exactly what the privacy policy keeps
# out of the log unless the user turned Teljes/TRACE on for themselves, and `redact_text`
# does not cover it — it scrubs addresses and credential params, not what someone typed
# into the search box. What the ERROR line carries instead is the shape of the failure,
# which is what tells an ingest bug from an upstream one; the term follows at TRACE, and
# only under this user's own consent — not whenever the process runs at TRACE.
log.exception(
"live search ingest failed",
extra=logs.event(
"search.live.ingest_failed",
source=source,
candidates=candidates,
collected=collected,
),
)
if logs.consented_trace():
log.log(
logs.TRACE,
"the term the ingest failed on",
# log-pii-ok: only under this user's own Teljes consent
extra=logs.event("search.live.term", term=term),
)
try:
with YouTubeClient(db, user) as yt:
collected, next_cursor = _gather(
db,
yt,
user,
cursor=cursor,
# The API source pages cost 100 units each — never auto-page; one page per request.
max_pages=1 if source == "api" else SEARCH_MAX_PAGES,
fetch=fetch,
candidates_of=candidates_of,
enough=lambda got: len(got) >= limit,
scrape_failed=scrape_failed,
ingest_failed=ingest_failed,
)
except Exception:
# A search that showed the user nothing does not count against them, unless search.list
# was already paid for: then the slot is what stops a failing search (a bogus cursor gets
# a 400 every time) from spending the shared budget without end. A busy refusal paid
# nothing and gives it back.
if not charged:
_give_back(claimed, "search", user.id)
raise
ordered = collected[:limit]
@@ -201,3 +333,78 @@ def clear_search(
removed = catalog_search.clear(db, user, ids)
db.commit()
return {"ok": True, "removed": removed}
@router.get("/related/{video_id}")
def related_videos(
video_id: str = Path(max_length=16),
cursor: str | None = None,
region: Literal["topical", "hu"] = "topical",
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The videos YouTube shows beside this one, materialised into the catalog like search results,
without the ones this user has watched or hidden here or in their Takeout history, and without
blocked channels. Pages on (`cursor`) until `RELATED_TARGET` are gathered; `next_cursor` reads
further. `region` picks the locale YouTube fills the list in (`RELATED_REGIONS`)."""
# Every request takes a slot, a cursor's too: each one scrapes pages and enriches what they
# hold, and a cursor is a free string — counting only first pages let a replayed cursor read
# without end. The check and the take are one step (`quota.claim_action`); a request that fails
# outright gives its slot back, a few an hour (`_give_back`). A limit of 0 turns related videos
# off on this instance.
daily_limit = sysconfig.get_int(db, "related_daily_limit_per_user")
if daily_limit <= 0:
raise refused(409, "related_disabled")
claimed = quota.claim_action(user.id, quota.QuotaAction.VIDEOS_RELATED, daily_limit)
if claimed is None:
_cap_spent("related", daily_limit)
raise refused(429, "related_daily_limit", limit=daily_limit)
hl, gl = RELATED_REGIONS[region]
blocked = queries.blocked_ids(db, user)
def candidates_of(items: list[dict], collected: list[str]) -> list[dict]:
ids = [it["id"] for it in items]
drop = reader_queries.seen_by(db, user, ids) | set(collected) | {video_id}
return [it for it in items if it["id"] not in drop and it["channel_id"] not in blocked]
def scrape_failed(exc: Exception) -> None:
log.warning(
"the related-videos scrape failed",
extra=logs.event("related.scrape_failed", error=str(exc)),
)
def ingest_failed(candidates: int, collected: int) -> None:
log.exception(
"a related list's ingest failed",
extra=logs.event("related.ingest_failed", candidates=candidates, collected=collected),
)
try:
with YouTubeClient(db, user) as yt:
collected, next_cursor = _gather(
db,
yt,
user,
cursor=cursor,
max_pages=RELATED_MAX_PAGES,
fetch=lambda token: related_scrape.related(
video_id, continuation=token, hl=hl, gl=gl
),
candidates_of=candidates_of,
enough=lambda got: len(got) >= RELATED_TARGET,
scrape_failed=scrape_failed,
ingest_failed=ingest_failed,
)
except Exception:
_give_back(claimed, "related", user.id)
raise
rows = reader_queries.feed_rows_by_ids(db, user, collected)
# A related list mixes channels the user follows with ones they never met: each row says which.
followed = queries.followed_among(db, user, {r["channel_id"] for r in rows})
return {
"items": [{**r, "followed": r["channel_id"] in followed} for r in rows],
"next_cursor": next_cursor,
"has_more": bool(next_cursor),
}
+160 -41
View File
@@ -22,6 +22,7 @@ from app.plex.tmdb import run_people_enrich
from app.plex.watch_sync import run_plex_watch_reconcile, run_plex_watch_sync
from app.notifications import create_notification
from app.runstatus import RunStatus, RunTrigger
from app.reader import takeout
from app.state import is_sync_paused
from app.sync.autotag import run_autotag_all
from app.sync.explore import purge_ephemeral
@@ -118,8 +119,9 @@ def _notify_done(
summary: str | None,
result: dict | None = None,
) -> None:
"""Post a completion notice to the triggering admin's inbox (manual runs only). Best
effort — a notification failure must never mask the job's own outcome."""
"""Post a completion notice to the inbox of who started the run: an admin's manual run, or a
user's ad-hoc run that asks for one. Best effort — a notification failure must never mask the
job's own outcome."""
try:
create_notification(
db,
@@ -272,7 +274,6 @@ def scheduler_snapshot(db) -> dict:
open_runs = _open_run_ids()
recent = jobruns.recent_runs(db, [*JOB_INTERVALS, *AD_HOC_JOBS], RECENT_RUNS)
open_runs |= _open_run_ids()
going_id = max((r for r, j in open_runs.items() if j == CHANNEL_BACKFILL), default=None)
jobs = []
for job_id, default_interval in JOB_INTERVALS.items():
a = acts.get(job_id, {})
@@ -288,30 +289,11 @@ def scheduler_snapshot(db) -> dict:
)
# Work a user starts on one target: no interval, no "run now" (it needs its target), but its
# runs are shown beside the jobs' — it spends the same quota. Its one in-memory guard is the
# target's own (sync/runner), so "running" is the runs this process has open: two channels can
# backfill at once, so a run still going counts even when a newer one already ended. That run then
# leads the row, so its channel, its "no outcome yet" and its dot are one run's, never the
# newer run's outcome under the older one's channel.
backfills = recent.get(CHANNEL_BACKFILL, [])
going = db.get(JobRun, going_id) if going_id is not None else None
if going is not None and going.status != RunStatus.RUNNING:
going = None # its outcome is written; only the bookkeeping has not caught up
if going is not None:
backfills = [going, *(r for r in backfills if r.id != going.id)][:RECENT_RUNS]
shown = backfills[0] if backfills else None
ad_hoc = [
{
"id": CHANNEL_BACKFILL,
"running": going is not None,
"target": shown.target if shown else None,
"target_title": (
catalog_queries.channel_title(db, shown.target)
if shown is not None and shown.target
else None
),
**_last_runs(backfills, open_runs),
}
]
# target's own (sync/runner, the import's per-user lock), so "running" is the runs this process
# has open: two targets can run at once, so a run still going counts even when a newer one already
# ended. That run then leads the row, so its target, its "no outcome yet" and its dot are one
# run's, never the newer run's outcome under the older one's target.
ad_hoc = [_ad_hoc_row(db, job, recent.get(job, []), open_runs) for job in AD_HOC_JOBS]
return {
"running_here": running_here,
"enabled": settings.scheduler_enabled,
@@ -333,6 +315,27 @@ def _seen(run, open_runs: dict[int, str]) -> RunStatus:
return RunStatus(run.status)
def _ad_hoc_row(db, job: str, runs: list[JobRun], open_runs: dict[int, str]) -> dict:
going_id = max((r for r, j in open_runs.items() if j == job), default=None)
going = db.get(JobRun, going_id) if going_id is not None else None
if going is not None and going.status != RunStatus.RUNNING:
going = None # its outcome is written; only the bookkeeping has not caught up
if going is not None:
runs = [going, *(r for r in runs if r.id != going.id)][:RECENT_RUNS]
shown = runs[0] if runs else None
return {
"id": job,
"running": going is not None,
"target": shown.target if shown else None,
"target_title": (
ad_hoc_target_title(db, job, shown.target)
if shown is not None and shown.target
else None
),
**_last_runs(runs, open_runs),
}
def _last_runs(runs: list, open_runs: dict[int, str]) -> dict:
"""A row's run fields from its newest runs (newest first): how the last one went, and the few
before it for the history dots."""
@@ -402,7 +405,12 @@ def _autotag_job() -> None:
def _shorts_job() -> None:
_job("shorts", run_shorts)
def work(db):
# A video waiting in someone's Takeout import becomes markable the moment the probe has
# said it is not a Short, so the import's waiting rows are settled right after it.
return run_shorts(db) | takeout.apply_waiting(db)
_job("shorts", work)
def _subscriptions_job() -> None:
@@ -608,48 +616,140 @@ def trigger_all(actor_id: int | None = None) -> list[str]:
# one channel. It runs on the same scheduler as a recorded run (trigger `ad_hoc`), so what it did and
# spent is in `job_runs` like any job's, not a background task only the log ever heard of.
CHANNEL_BACKFILL = "channel_backfill"
# A user's Google Takeout import (`reader.takeout`): its target is the user.
TAKEOUT_IMPORT = "takeout_import"
# Every ad-hoc job id: the page names each one (a frontend contract test reads this tuple).
AD_HOC_JOBS = (CHANNEL_BACKFILL,)
AD_HOC_JOBS = (CHANNEL_BACKFILL, TAKEOUT_IMPORT)
# What each one's target is called on the admin's pages: a channel's title, an importing user's name.
_TARGET_TITLES: dict[str, Callable[..., str | None]] = {
CHANNEL_BACKFILL: catalog_queries.channel_title,
TAKEOUT_IMPORT: takeout.target_title,
}
def run_ad_hoc(job: str, fn: Callable[[], dict], *, actor_id: int, target: str) -> None:
def ad_hoc_target_title(db, job: str, target: str) -> str | None:
"""The name of an ad-hoc run's target (None for a job with none, or a target since gone)."""
title = _TARGET_TITLES.get(job)
return title(db, target) if title is not None else None
# The ad-hoc runs added and not yet recorded, `job@target`: between the two the target's newest
# run row is still the previous run's, so a page watching it asks `ad_hoc_queued` too.
_queued: set[str] = set()
_queued_guard = threading.Lock()
# What the scheduler is told to run is only the key: the request itself waits here, the newest one
# per key, until a fire of that key takes it. A request made while its key's run is under way is so
# kept for the run after it, which that run asks for as it ends (`_running`): the scheduler would
# drop a second fire of an id whose run has not returned (one instance per job id).
_pending: dict[str, tuple[Callable[[], dict], int, bool]] = {}
_running: set[str] = set()
# Two at once, for the moment between a run letting go of its key and its thread returning: a fire
# then is what takes the next request, and is not dropped.
_AD_HOC_INSTANCES = 2
def ad_hoc_queued(job: str, target: str) -> bool:
"""Whether a run of `job` on `target` is queued and its run row not written yet."""
with _queued_guard:
return f"{job}@{target}" in _queued
def run_ad_hoc(
job: str, fn: Callable[[], dict], *, actor_id: int, target: str, notify: bool = False
) -> None:
"""Run `fn` now as `actor_id`'s run of `job` on `target`, a one-shot job on the scheduler. Its
own exclusion (if any) lives in `fn`, which says so in its result; one id per job and target, so
a second request before the first has even started is that same run."""
a second request before the first has even started is that same run, and one made while it runs
is the run after it. `notify` posts its outcome to the user's inbox, for work whose page does not
watch it run."""
if job not in AD_HOC_JOBS:
raise KeyError(job) # an unnamed job would show on the page as its bare id
sched = _scheduler
if sched is None:
if _scheduler is None:
# Only between shutdown and exit: start_scheduler runs at every API startup.
raise RuntimeError("the scheduler is not running")
key = f"{job}@{target}"
with _queued_guard:
_pending[key] = (fn, actor_id, notify)
_queued.add(key)
running = key in _running
if running:
log.info("ad-hoc %s for %s runs again once the run under way ends", job, target)
return
_fire(job, target)
def _fire(job: str, target: str) -> None:
"""Add the one-shot that takes `job@target`'s pending request. Raises (the request dropped) only
when the scheduler cannot take it."""
key = f"{job}@{target}"
sched = _scheduler
try:
if sched is None:
raise RuntimeError("the scheduler is not running")
sched.add_job(
_ad_hoc,
_ad_hoc_fired,
"date",
args=(job, fn, actor_id, target),
id=f"{job}@{target}",
args=(job, target),
id=key,
misfire_grace_time=None,
max_instances=_AD_HOC_INSTANCES,
)
# silent-ok: not a failure — the run for this target is already queued, and it does this work.
# silent-ok: not a failure — a fire of this key is already queued, and it takes this request.
except ConflictingIdError:
log.info("ad-hoc %s for %s already queued", job, target)
except Exception:
with _queued_guard:
_pending.pop(key, None)
_queued.discard(key)
raise
def _ad_hoc(job: str, fn: Callable[[], dict], actor_id: int, target: str) -> None:
def _ad_hoc_fired(job: str, target: str) -> None:
key = f"{job}@{target}"
with _queued_guard:
request = _pending.pop(key, None)
if request is None:
return # an earlier fire took it
_running.add(key)
fn, actor_id, notify = request
try:
_ad_hoc(job, fn, actor_id, target, notify)
finally:
with _queued_guard:
_running.discard(key)
again = key in _pending
if again:
try:
_fire(job, target)
except Exception:
# The request made meanwhile is lost with the scheduler: shut down, nothing runs.
log.exception("ad-hoc %s for %s not run again", job, target)
def _ad_hoc(
job: str, fn: Callable[[], dict], actor_id: int, target: str, notify: bool = False
) -> None:
key = f"{job}@{target}"
try:
run_id = jobruns.start(job, RunTrigger.AD_HOC, actor_id, target)
except Exception:
# No row to tell it in: the database is what failed. The log is the record.
log.exception("ad-hoc %s for %s not started: its run could not be recorded", job, target)
return
finally:
# Recorded (or never will be): the run row now speaks for it — unless another request came
# meanwhile, which is still queued.
with _queued_guard:
if key not in _pending:
_queued.discard(key)
_opened(run_id, job)
try:
_ad_hoc_work(job, fn, run_id)
_ad_hoc_work(job, fn, run_id, actor_id if notify else None)
finally:
_closed(run_id)
def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int) -> None:
def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int, notify_id: int | None) -> None:
with logs.bind(run_id=run_id):
# As in _run: only the work is inside the try.
try:
@@ -658,8 +758,23 @@ def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int) -> None:
log.exception("job %s failed", job, extra=logs.event("scheduler.job.failed", job=job))
err, failure = _failure(exc)
_close(run_id, RunStatus.ERROR, error=err, failure=failure)
# The user's notice names the failure, never the upstream's own text: that is an English
# diagnostic, for the admin's run row and the log, not a user's inbox.
_notify_ad_hoc(
notify_id, job, RunStatus.ERROR, None, failure or {"failure": "unexpected"}
)
return
_finish_ok(job, run_id, result)
summary, data = _finish_ok(job, run_id, result)
_notify_ad_hoc(notify_id, job, RunStatus.OK, summary, data)
def _notify_ad_hoc(
user_id: int | None, job: str, status: RunStatus, summary: str | None, result: dict | None
) -> None:
if user_id is None:
return
with SessionLocal() as db:
_notify_done(db, user_id, job, status, summary, result)
# Room for every job at once (a "run all") beside a few ad-hoc runs: APScheduler's default pool of
@@ -700,3 +815,7 @@ def shutdown_scheduler() -> None:
if _scheduler is not None:
_scheduler.shutdown(wait=False)
_scheduler = None
with _queued_guard:
# What it held will not run.
_queued.clear()
_pending.clear()
+16 -8
View File
@@ -14,6 +14,7 @@ from sqlalchemy import delete, or_, select
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.catalog import service as catalog_service
from app.catalog.models import Channel, ExploredChannel, SearchFind, Subscription, Video
from app.reader import queries as reader_queries
from app.sync.videos import insert_stubs, stub_from_playlist_item, apply_video_details
@@ -66,11 +67,13 @@ def _enrich_new(db: Session, yt: YouTubeClient, video_ids: list[str]) -> int:
def explore_ingest_page(
db: Session, yt: YouTubeClient, channel: Channel, page_token: str | None = None
) -> dict:
"""Ingest one page (up to 50, newest-first) of the channel's uploads, marking the new
videos via_explore, then enrich them. Returns the count ingested and the next page token
(None when the back-catalogue is exhausted) so the caller can offer "load more". Stateless
pagination: the caller round-trips the token, so the channel's own backfill cursor (used by
the organic deep backfill) is never touched."""
"""Ingest one page (up to 50, newest-first) of the channel's uploads, marking the new videos
via_explore while the channel is an exploration's (`from_explore`): of a channel someone follows
they are its uploads like any other, and a flag there would never be cleared, since subscribing
clears only an explored channel's. Then enrich them. Returns the count ingested and the next
page token (None when the back-catalogue is exhausted) so the caller can offer "load more".
Stateless pagination: the caller round-trips the token, so the channel's own backfill cursor
(used by the organic deep backfill) is never touched."""
if not channel.uploads_playlist_id:
return {"ingested": 0, "next_page_token": None}
try:
@@ -86,9 +89,10 @@ def explore_ingest_page(
for item in data.get("items", []):
stub = stub_from_playlist_item(item, channel.id)
if stub is not None:
stub["via_explore"] = True
stub["via_explore"] = channel.from_explore
stubs.append(stub)
inserted = insert_stubs(db, stubs)
catalog_service.settle_explored(db, channel)
self_enriched = _enrich_new(db, yt, [s["id"] for s in stubs])
log.info(
"explore ingest channel=%s ingested=%s enriched=%s", channel.id, inserted, self_enriched
@@ -105,7 +109,9 @@ def purge_unkept_explored(db: Session) -> dict:
- a live ExploredChannel row (someone explored it within the grace window — channel-level), or
- a video has per-user state: watched/in-progress (VideoState), in a playlist (PlaylistItem),
or surfaced by the user's own search (SearchFind) — these survive even if the channel goes.
A channel row is dropped only once it holds no videos at all. 0 quota."""
A channel row is dropped only once it holds no videos at all, and not while someone's Takeout
history shows it watched (their "watched but not followed" list names it from that row).
0 quota."""
grace_days = sysconfig.get_int(db, "explore_grace_days")
cutoff = datetime.now(timezone.utc) - timedelta(days=grace_days)
@@ -135,7 +141,9 @@ def purge_unkept_explored(db: Session) -> dict:
has_videos = select(Video.id).where(Video.channel_id == Channel.id).exists()
channels_deleted = db.execute(
delete(Channel).where(Channel.id.in_(orphan_ids), ~has_videos)
delete(Channel).where(
Channel.id.in_(orphan_ids), ~has_videos, ~reader_queries.in_a_watch_history(Channel.id)
)
).rowcount
db.commit()
return {
+20
View File
@@ -8,6 +8,7 @@ from sqlalchemy.orm import Session
from app import logs
from app.models import User
from app.catalog import service as catalog_service
from app.catalog.models import Channel, Subscription
from app.utils import normalize_description
from app.youtube.client import YouTubeClient, YouTubeError, best_thumbnail
@@ -109,6 +110,21 @@ def apply_channel_details(db: Session, items: list[dict], asked: Iterable[str])
channel.details_synced_at = now
def describe_channels(db: Session, yt: YouTubeClient, channel_ids: list[str]) -> None:
"""Make sure each given channel is in the catalog with its details: a channel it lacks is added
as a stub, then all of them are asked for (`channels.list`, 1 unit a 50). Flushes; the caller
commits."""
if not channel_ids:
return
have = set(db.execute(select(Channel.id).where(Channel.id.in_(channel_ids))).scalars())
for cid in dict.fromkeys(channel_ids):
if cid not in have:
db.add(Channel(id=cid))
db.flush()
apply_channel_details(db, yt.get_channels(channel_ids), channel_ids)
db.flush()
def should_prune(fetched_count: int, existing_count: int) -> bool:
"""Whether an import may delete the local subscriptions missing from this fetch.
@@ -182,6 +198,10 @@ def import_subscriptions(db: Session, user: User) -> dict:
db.add(sub_row)
sub_row.yt_subscription_id = sub.get("yt_subscription_id")
db.flush()
# A follow made on youtube.com keeps an explored channel as a follow made here does.
explored = select(Channel).where(Channel.id.in_(fetched), Channel.from_explore.is_(True))
for channel in db.execute(explored).scalars():
catalog_service.settle_explored(db, channel)
# Prune subscriptions removed on YouTube — but only when this fetch is trustworthy
# (see `should_prune`).
+12 -3
View File
@@ -420,11 +420,15 @@ def run_shorts_classification(db: Session, limit: int | None = None) -> dict:
return {"probed": probed, "shorts": shorts}
def ensure_videos(db: Session, yt: YouTubeClient, video_ids: list[str]) -> None:
def ensure_videos(
db: Session, yt: YouTubeClient, video_ids: list[str], *, explore: bool = False
) -> None:
"""Make sure every given video id exists in the catalog, fetching + ingesting the
missing ones (and a stub channel for any unknown channel). Deleted/private videos that
the API won't return are simply left out. The catalog's write for a playlist sync that names
videos it has never seen; commits."""
videos it has never seen; commits. `explore` adds the new ones as an exploration's
(`via_explore`): visible to who explores or follows their channel, and reclaimed by the cleanup
once nobody keeps them."""
if not video_ids:
return
have = set(db.execute(select(Video.id).where(Video.id.in_(video_ids))).scalars().all())
@@ -458,7 +462,12 @@ def ensure_videos(db: Session, yt: YouTubeClient, video_ids: list[str]) -> None:
seen.add(vid)
# published_at is NOT NULL (0068); apply_video_details re-fills a missing date, but seed a now()
# fallback here too so the constructor never carries an explicit NULL.
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")) or now)
v = Video(
id=vid,
channel_id=cid,
published_at=parse_dt(sn.get("publishedAt")) or now,
via_explore=explore,
)
apply_video_details(v, it)
v.enriched_at = now
db.add(v)
+9
View File
@@ -140,6 +140,15 @@ SPECS: tuple[ConfigSpec, ...] = (
max=100_000,
applies=LIVE,
),
ConfigSpec(
"related_daily_limit_per_user",
"int",
"quota",
"related_daily_limit_per_user",
min=0,
max=100_000,
applies=LIVE,
),
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
ConfigSpec("search_source", "choice", "quota", "search_source", applies=LIVE),
ConfigSpec(
+30 -6
View File
@@ -30,12 +30,26 @@ class YouTubeError(Exception):
message is prose, it changes under us, and it is deliberately the poorer of the two here.
`path` is the API endpoint that ANSWERED with the failure (`playlistItems`); empty when no
endpoint did — the request never got through (`transport`) or the token step refused first."""
endpoint did — the request never got through (`transport`) or the token step refused first.
def __init__(self, message: str, *, reason: str = "", path: str = ""):
`sent` is a `transport` failure that came after the request may have gone out (a read timeout,
a dropped answer): nothing answered, but YouTube may have counted the call."""
def __init__(self, message: str, *, reason: str = "", path: str = "", sent: bool = False):
super().__init__(message)
self.reason = reason
self.path = path
self.sent = sent
# The transport failures that stop a request before it reaches YouTube: nothing went out.
_UNSENT = (
httpx.ConnectError,
httpx.ConnectTimeout,
httpx.PoolTimeout,
httpx.ProxyError,
httpx.UnsupportedProtocol,
)
def _error_reason(resp: httpx.Response) -> str:
@@ -162,6 +176,9 @@ def best_thumbnail(thumbnails: dict | None) -> str | None:
class YouTubeClient:
# The quota units this client has recorded: what its calls have spent of the shared budget.
spent: int = 0
def __init__(self, db, user: User):
self.db = db
self.user = user
@@ -189,9 +206,16 @@ class YouTubeClient:
return self._http.request(method, url, **kwargs)
except httpx.HTTPError as exc:
raise YouTubeError(
f"YouTube API unreachable ({exc.__class__.__name__}: {exc})", reason="transport"
f"YouTube API unreachable ({exc.__class__.__name__}: {exc})",
reason="transport",
sent=not isinstance(exc, _UNSENT),
) from exc
def _spend(self, units: int) -> None:
"""Record what a call YouTube answered cost — a refused one too: it is charged all the same."""
quota.record_usage(units)
self.spent += units
# --- auth ---
def _access_token(self) -> str:
tok = self.user.token
@@ -269,7 +293,7 @@ class YouTubeClient:
else:
headers["Authorization"] = f"Bearer {self._access_token()}"
resp = self._send("GET", f"{API_BASE}/{path}", params=p, headers=headers)
quota.record_usage(cost)
self._spend(cost)
if resp.status_code != 200:
raise _upstream_failure("GET", path, resp, expected)
return resp.json()
@@ -401,7 +425,7 @@ class YouTubeClient:
params={"id": subscription_id},
headers={"Authorization": f"Bearer {self._access_token()}"},
)
quota.record_usage(50)
self._spend(50)
if resp.status_code not in (200, 204):
raise _upstream_failure("DELETE", "subscriptions", resp)
@@ -431,7 +455,7 @@ class YouTubeClient:
def _write(self, method: str, path: str, *, params: dict, json: dict | None = None) -> dict:
headers = {"Authorization": f"Bearer {self._access_token()}"}
resp = self._send(method, f"{API_BASE}/{path}", params=params, json=json, headers=headers)
quota.record_usage(50)
self._spend(50)
if resp.status_code not in (200, 204):
raise _upstream_failure(method, path, resp)
return resp.json() if resp.content else {}
+160
View File
@@ -0,0 +1,160 @@
"""The website's own InnerTube API (`youtubei/v1/*`), read the way the website reads it — no quota.
The zero-quota search (`search_scrape`) and the related videos (`related_scrape`) both POST here with
the public WEB key and a client version, read out of a live results page. This deliberately calls an
undocumented endpoint: if YouTube changes the JSON shape or blocks the host, each caller says so as a
`ScrapeError`, never as an empty answer.
The SOCS=CAI consent-skip cookie is sent like the Shorts probe's, so a server-side request isn't
bounced to consent.youtube.com."""
import re
import httpx
from app import logs
log = logs.get_logger(__name__)
_API_URL = "https://www.youtube.com/youtubei/v1/"
_RESULTS_URL = "https://www.youtube.com/results"
# Public WEB InnerTube key + a recent client version. These are also embedded in every results
# page; we fetch the live pair once and cache it, falling back to these constants.
_FALLBACK_KEY = "AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8" # gitleaks:allow -- the public WEB key, in every results page
_FALLBACK_CLIENT_VERSION = "2.20240101.00.00"
_HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
# SOCS skips YouTube's cookie-consent interstitial (otherwise every server-side request is
# redirected to consent.youtube.com).
"Cookie": "SOCS=CAI",
"Accept-Language": "en-US,en;q=0.9",
}
_KEY_RE = re.compile(r'"INNERTUBE_API_KEY":"([^"]+)"')
_VER_RE = re.compile(r'"INNERTUBE_CLIENT_VERSION":"([^"]+)"')
# Cached (api_key, client_version) extracted from a results page; refreshed on demand.
_config: tuple[str, str] | None = None
class ScrapeError(Exception):
"""An InnerTube read could not be completed (network, blocked, or unexpected shape)."""
def runs_text(node: dict | None) -> str | None:
"""The text of an InnerTube text node: a `simpleText`, or its `runs` joined."""
if not node:
return None
if node.get("simpleText"):
return node["simpleText"]
runs = node.get("runs") or []
if runs:
return "".join(r.get("text", "") for r in runs)
return None
def scraped_video(
vid: str, channel_id: object, channel_title: object, title: object, thumbnails: object
) -> dict:
"""One scraped video in the shape `YouTubeClient.search_videos` answers, so a page of either
source is ingested the same way. `published_at` is None (the website shows only a relative
"3 years ago"; the enrich fills the real date) and `live_broadcast` "none" (live/upcoming are
judged after the enrich, by `live_status`). The largest thumbnail is the last."""
thumbs = thumbnails if isinstance(thumbnails, list) else []
last = thumbs[-1] if thumbs else None
return {
"id": vid,
"channel_id": channel_id if isinstance(channel_id, str) and channel_id else None,
"channel_title": channel_title if isinstance(channel_title, str) else None,
"title": title if isinstance(title, str) else None,
"published_at": None,
"thumbnail_url": last.get("url") if isinstance(last, dict) else None,
"live_broadcast": "none",
}
def video_renderer(node: dict) -> dict | None:
"""A `videoRenderer` (search) or `compactVideoRenderer` (the older related column) flattened —
the two draw a video the same way. None without a video id; the channel is None when the byline
names none."""
vid = node.get("videoId")
if not isinstance(vid, str) or not vid:
return None
owner = node.get("ownerText") or node.get("longBylineText") or {}
runs = owner.get("runs") if isinstance(owner, dict) else None
first = runs[0] if isinstance(runs, list) and runs and isinstance(runs[0], dict) else {}
endpoint = (first.get("navigationEndpoint") or {}).get("browseEndpoint") or {}
return scraped_video(
vid,
endpoint.get("browseId"),
first.get("text"),
runs_text(node.get("title")),
(node.get("thumbnail") or {}).get("thumbnails"),
)
def continuation_token(node: object) -> str | None:
"""The "load more" cursor of a `continuationItemRenderer`, or None when `node` is not one."""
if not isinstance(node, dict):
return None
command = (node.get("continuationEndpoint") or {}).get("continuationCommand") or {}
token = command.get("token")
return token if isinstance(token, str) and token else None
def _get_config(client: httpx.Client) -> tuple[str, str]:
"""The current InnerTube (api_key, client_version), cached after the first fetch that FOUND them.
The fallback pair is never cached: it used to be, so one failed fetch (or a consent page with no
keys on it) at the wrong moment pinned every later search to a hard-coded key and version for
the life of the process — and when those went stale, search returned nothing, with no error
anywhere. Now the next call simply tries the live pair again."""
global _config
if _config is not None:
return _config
try:
html = client.get(_RESULTS_URL, params={"search_query": "music"}).text
except httpx.HTTPError as exc:
log.warning(
"the InnerTube config could not be fetched; falling back to the pinned key",
extra=logs.event("youtube.scrape.config_fallback", reason=type(exc).__name__),
)
return (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
key = _KEY_RE.search(html)
ver = _VER_RE.search(html)
if key and ver:
_config = (key.group(1), ver.group(1))
return _config
log.warning(
"the results page carried no InnerTube config; falling back to the pinned key",
extra=logs.event("youtube.scrape.config_fallback", reason="not_on_page"),
)
return (
key.group(1) if key else _FALLBACK_KEY,
ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
)
def post(endpoint: str, body: dict, *, hl: str = "en", gl: str = "US") -> dict:
"""POST `body` to `youtubei/v1/<endpoint>` as the WEB client in the `hl`/`gl` locale and return
the decoded JSON object. A network failure, an HTTP error or a body that is not a JSON object (a
block, consent or captcha page) raises `ScrapeError`; the caller judges the shape."""
with httpx.Client(timeout=20.0, headers=_HEADERS) as client:
api_key, client_version = _get_config(client)
context = {
"client": {"clientName": "WEB", "clientVersion": client_version, "hl": hl, "gl": gl}
}
try:
resp = client.post(
_API_URL + endpoint, params={"key": api_key}, json={"context": context, **body}
)
resp.raise_for_status()
data = resp.json()
except httpx.HTTPError as exc:
raise ScrapeError(f"InnerTube request failed: {exc}") from exc
except ValueError as exc: # non-JSON body (blocked / consent / captcha)
raise ScrapeError("InnerTube returned a non-JSON response") from exc
if not isinstance(data, dict):
raise ScrapeError("InnerTube returned an unexpected shape")
return data
+141
View File
@@ -0,0 +1,141 @@
"""A video's related videos, read from the watch page's InnerTube `next` endpoint — no quota.
The Data API has no related-videos call any more, so this is the only source: the same list the
website shows beside a playing video. A page holds about 20 videos and a continuation token for the
next 20. The returned page has the search scrape's shape (``{"items": [...], "next_page_token"}``)
so the catalog ingests it the same way (`catalog.search.ingest_candidates`).
YouTube draws each item as a `lockupViewModel` now; the older `compactVideoRenderer` is read too, so
a rollback on its side does not empty the list. An item that names no single channel — a video by
several channels draws an avatar stack, not one avatar — is left out: the catalog row needs its
channel up front, and the stack does not say which one uploaded it.
`hl`/`gl` pick the locale YouTube fills the list in: ``en``/``US`` keeps it on the video's topic,
``hu``/``HU`` mixes in what is trending in Hungary (measured)."""
from app.youtube.innertube import (
ScrapeError,
continuation_token,
post,
scraped_video,
video_renderer,
)
__all__ = ["ScrapeError", "related"]
# Where a first page and a continuation page put their item list.
_FIRST = (
"contents",
"twoColumnWatchNextResults",
"secondaryResults",
"secondaryResults",
"results",
)
_APPENDS = ("appendContinuationItemsAction", "reloadContinuationItemsCommand")
def _at(node: object, *path: str | int) -> object:
"""The value under `path` in nested dicts/lists, or None where the path stops."""
for key in path:
if isinstance(node, dict) and isinstance(key, str):
node = node.get(key)
elif isinstance(node, list) and isinstance(key, int) and -len(node) <= key < len(node):
node = node[key]
else:
return None
return node
def _lockup(node: dict) -> dict | None:
if node.get("contentType") != "LOCKUP_CONTENT_TYPE_VIDEO" or not node.get("contentId"):
return None
meta = _at(node, "metadata", "lockupMetadataViewModel")
channel_id = _at(
meta,
"image",
"decoratedAvatarViewModel",
"rendererContext",
"commandContext",
"onTap",
"innertubeCommand",
"browseEndpoint",
"browseId",
)
channel_title = _at(
meta, "metadata", "contentMetadataViewModel", "metadataRows", 0, "metadataParts", 0
)
return scraped_video(
node["contentId"],
channel_id,
_at(channel_title, "text", "content"),
_at(meta, "title", "content"),
_at(node, "contentImage", "thumbnailViewModel", "image", "sources"),
)
def _unavailable(data: dict) -> bool:
"""Whether the watch page says the video is not available: its main column draws a
`backgroundPromoRenderer` ("This video isn't available anymore") and there is no related column.
YouTube's own word for "nothing related" — the one empty page that is not a shape we lost."""
sections = _at(data, "contents", "twoColumnWatchNextResults", "results", "results", "contents")
for section in sections if isinstance(sections, list) else []:
inner = _at(section, "itemSectionRenderer", "contents")
for node in inner if isinstance(inner, list) else []:
if isinstance(node, dict) and "backgroundPromoRenderer" in node:
return True
return False
def _items(data: dict) -> list | None:
"""The page's item list (first page or continuation), or None when it has neither frame."""
first = _at(data, *_FIRST)
if isinstance(first, list):
return first
if _unavailable(data):
return []
for endpoint in data.get("onResponseReceivedEndpoints") or []:
for key in _APPENDS:
items = _at(endpoint, key, "continuationItems")
if isinstance(items, list):
return items
return None
def related(
video_id: str, continuation: str | None = None, *, hl: str = "en", gl: str = "US"
) -> dict:
"""One page of `video_id`'s related videos (`continuation` = the token a previous page gave).
A video YouTube says it no longer has is an empty page. An answer of a shape this does not know
— no related column, or one whose entries are all renderers it cannot read — is a `ScrapeError`:
not an empty page, which would read as "nothing related"."""
body = {"continuation": continuation} if continuation else {"videoId": video_id}
data = post("next", body, hl=hl, gl=gl)
items = _items(data)
if items is None:
raise ScrapeError("InnerTube returned no related list")
out: list[dict] = []
seen: set[str] = set()
token = None
known = 0 # entries drawn by a renderer read here (a video's, or the cursor)
for entry in items:
if not isinstance(entry, dict):
continue
if "lockupViewModel" in entry:
item = _lockup(entry["lockupViewModel"])
elif "compactVideoRenderer" in entry:
item = video_renderer(entry["compactVideoRenderer"])
else:
found = continuation_token(entry.get("continuationItemRenderer"))
known += found is not None
token = token or found
continue
known += 1
# A video by several channels draws an avatar stack, not one: no channel, left out.
if item and item["channel_id"] and item["id"] not in seen:
seen.add(item["id"])
out.append(item)
# A list with entries but none of them one we read: YouTube draws its items some new way.
if items and not known:
raise ScrapeError("InnerTube's related list holds no renderer this reads")
return {"items": out, "next_page_token": token}
+14 -130
View File
@@ -6,9 +6,9 @@ calls — and read the result list straight out of the JSON. The returned video
through the normal enrich path (videos.list, 1 unit / 50), exactly as the official search does,
so the search itself is effectively free and only the (cheap, shared) enrichment is charged.
The SOCS=CAI consent-skip cookie is used like the Shorts probe so a server-side request isn't
bounced to consent.youtube.com. This deliberately scrapes an undocumented endpoint: if YouTube
changes the JSON shape or blocks the host, an admin can flip ``search_source`` back to ``api``.
The request itself (the live key, the consent cookie) is `innertube`'s. This deliberately scrapes an
undocumented endpoint: if YouTube changes the JSON shape or blocks the host, an admin can flip
``search_source`` back to ``api``.
The returned page mirrors :meth:`YouTubeClient.search_videos` exactly
(``{"items": [...], "next_page_token": str | None}``) so the route is source-agnostic — the
@@ -17,110 +17,16 @@ only difference is the cursor is an InnerTube *continuation token*, not a Data A
the real date is filled by the videos.list enrich. ``live_broadcast`` is reported as ``"none"``
and live/upcoming filtering happens post-enrich via ``live_status`` (same as the API path)."""
import re
from app.youtube.innertube import ScrapeError, continuation_token, post, video_renderer
import httpx
__all__ = ["ScrapeError", "search"]
from app import logs
log = logs.get_logger(__name__)
_SEARCH_URL = "https://www.youtube.com/youtubei/v1/search"
_RESULTS_URL = "https://www.youtube.com/results"
# Public WEB InnerTube key + a recent client version. These are also embedded in every results
# page; we fetch the live pair once and cache it, falling back to these constants.
_FALLBACK_KEY = "AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"
_FALLBACK_CLIENT_VERSION = "2.20240101.00.00"
# sp / params value selecting the "type: video" search filter (base64 protobuf EgIQAQ==).
_TYPE_VIDEO = "EgIQAQ=="
_HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
# SOCS skips YouTube's cookie-consent interstitial (otherwise every server-side request is
# redirected to consent.youtube.com).
"Cookie": "SOCS=CAI",
"Accept-Language": "en-US,en;q=0.9",
}
# The top-level keys a first page (contents) and a continuation page (onResponseReceivedCommands)
# put their result list under.
_RESULT_FRAMES = {"contents", "onResponseReceivedCommands"}
_KEY_RE = re.compile(r'"INNERTUBE_API_KEY":"([^"]+)"')
_VER_RE = re.compile(r'"INNERTUBE_CLIENT_VERSION":"([^"]+)"')
# Cached (api_key, client_version) extracted from a results page; refreshed on demand.
_config: tuple[str, str] | None = None
class ScrapeError(Exception):
"""A scrape-based search could not be completed (network, blocked, or unexpected shape)."""
def _runs_text(node: dict | None) -> str | None:
if not node:
return None
if node.get("simpleText"):
return node["simpleText"]
runs = node.get("runs") or []
if runs:
return "".join(r.get("text", "") for r in runs)
return None
def _get_config(client: httpx.Client) -> tuple[str, str]:
"""The current InnerTube (api_key, client_version), cached after the first fetch that FOUND them.
The fallback pair is never cached: it used to be, so one failed fetch (or a consent page with no
keys on it) at the wrong moment pinned every later search to a hard-coded key and version for
the life of the process — and when those went stale, search returned nothing, with no error
anywhere. Now the next search simply tries the live pair again."""
global _config
if _config is not None:
return _config
try:
html = client.get(_RESULTS_URL, params={"search_query": "music"}).text
except httpx.HTTPError as exc:
log.warning(
"the InnerTube config could not be fetched; falling back to the pinned key",
extra=logs.event("youtube.scrape.config_fallback", reason=type(exc).__name__),
)
return (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
key = _KEY_RE.search(html)
ver = _VER_RE.search(html)
if key and ver:
_config = (key.group(1), ver.group(1))
return _config
log.warning(
"the results page carried no InnerTube config; falling back to the pinned key",
extra=logs.event("youtube.scrape.config_fallback", reason="not_on_page"),
)
return (
key.group(1) if key else _FALLBACK_KEY,
ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
)
def _video_item(vr: dict) -> dict | None:
"""Flatten a YouTube `videoRenderer` into the same shape as YouTubeClient.search_videos."""
vid = vr.get("videoId")
if not vid:
return None
owner = vr.get("ownerText") or vr.get("longBylineText") or {}
runs = owner.get("runs") or []
channel_id = channel_title = None
if runs:
channel_title = runs[0].get("text")
channel_id = runs[0].get("navigationEndpoint", {}).get("browseEndpoint", {}).get("browseId")
thumbs = (vr.get("thumbnail") or {}).get("thumbnails") or []
return {
"id": vid,
"channel_id": channel_id,
"channel_title": channel_title,
"title": _runs_text(vr.get("title")),
"published_at": None, # scrape exposes only a relative time; enrich fills the real date
"thumbnail_url": thumbs[-1]["url"] if thumbs else None,
"live_broadcast": "none", # live/upcoming filtered post-enrich via live_status
}
def _collect_videos(obj, out: list[dict], seen: set[str]) -> None:
@@ -130,7 +36,7 @@ def _collect_videos(obj, out: list[dict], seen: set[str]) -> None:
if isinstance(obj, dict):
vr = obj.get("videoRenderer")
if isinstance(vr, dict):
item = _video_item(vr)
item = video_renderer(vr)
if item and item["id"] not in seen:
seen.add(item["id"])
out.append(item)
@@ -144,11 +50,9 @@ def _collect_videos(obj, out: list[dict], seen: set[str]) -> None:
def _find_continuation(obj) -> str | None:
"""First continuationItemRenderer token (the 'load more' cursor), or None at the end."""
if isinstance(obj, dict):
cir = obj.get("continuationItemRenderer")
if isinstance(cir, dict):
tok = cir.get("continuationEndpoint", {}).get("continuationCommand", {}).get("token")
if tok:
return tok
tok = continuation_token(obj.get("continuationItemRenderer"))
if tok:
return tok
for v in obj.values():
tok = _find_continuation(v)
if tok:
@@ -167,34 +71,14 @@ def search(q: str, continuation: str | None = None) -> dict:
`continuation` is the InnerTube continuation token from a previous page (None for the first
page). Returns ``{"items": [...], "next_page_token": str | None}`` — the same shape as
YouTubeClient.search_videos so the route handles both sources identically."""
with httpx.Client(timeout=20.0, headers=_HEADERS) as client:
api_key, client_version = _get_config(client)
context = {
"client": {
"clientName": "WEB",
"clientVersion": client_version,
"hl": "en",
"gl": "US",
}
}
body: dict = {"context": context}
if continuation:
body["continuation"] = continuation
else:
body["query"] = q
body["params"] = _TYPE_VIDEO
try:
resp = client.post(_SEARCH_URL, params={"key": api_key}, json=body)
resp.raise_for_status()
data = resp.json()
except httpx.HTTPError as exc:
raise ScrapeError(f"InnerTube request failed: {exc}") from exc
except ValueError as exc: # non-JSON body (blocked / consent / captcha)
raise ScrapeError("InnerTube returned a non-JSON response") from exc
body: dict = (
{"continuation": continuation} if continuation else {"query": q, "params": _TYPE_VIDEO}
)
data = post("search", body)
# Even a search with no results carries its result list's frame. Without one this is not "no
# results" but a shape we do not know (YouTube moved, or a block page in JSON) — say so.
if not isinstance(data, dict) or not _RESULT_FRAMES & data.keys():
if not _RESULT_FRAMES & data.keys():
raise ScrapeError("InnerTube returned an unexpected shape")
items: list[dict] = []
_collect_videos(data, items, set())
+4 -3
View File
@@ -6,9 +6,10 @@
# bumped in place instead of a full regeneration, which would move every other package too — then
# `pip install -r requirements.lock && pip check` must pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177); yt-dlp 2026.7.4 -> 2026.8.19
# (it marks the clients that need a stream PO token, so the downloader stops fetching into a 403);
# certifi 2026.7.22 -> 2026.10.19 (GHSA-6q48-6c7w-9gw2).
# certifi 2026.7.22 -> 2026.10.19 (GHSA-6q48-6c7w-9gw2); brotli 1.2.0 -> 1.2.1 (PYSEC-2026-4322);
# Authlib 1.7.2 -> 1.9.1 (CVE-2026-96760, CVE-2026-114616).
APScheduler==3.11.3
Authlib==1.7.2
Authlib==1.9.1
Mako==1.4.1
MarkupSafe==3.0.3
PyYAML==6.0.3
@@ -21,7 +22,7 @@ anyio==4.14.2
argon2-cffi-bindings==25.1.0
argon2-cffi==25.1.0
bgutil-ytdlp-pot-provider==1.3.1
brotli==1.2.0
brotli==1.2.1
certifi==2026.10.19
cffi==2.1.1
charset-normalizer==3.5.1
+12
View File
@@ -327,3 +327,15 @@ def _reset_log_erasures():
from app import logs
logs._erased.clear()
@pytest.fixture(autouse=True)
def _reset_ad_hoc_queue():
"""The scheduler's ad-hoc queue is module-level too: a request a test left pending would read as
a later test's user's import still queued (ids restart from 1)."""
from app import scheduler
with scheduler._queued_guard:
scheduler._queued.clear()
scheduler._pending.clear()
scheduler._running.clear()
+527
View File
@@ -0,0 +1,527 @@
{
"onResponseReceivedEndpoints": [
{
"appendContinuationItemsAction": {
"continuationItems": [
{
"lockupViewModel": {
"contentImage": {
"thumbnailViewModel": {
"image": {
"sources": [
{
"url": "https://i.ytimg.com/vi/_HfdncCbMOE/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLARvB39pkdsiuUUQTsZjkhvbKrfcQ",
"width": 168,
"height": 94
},
{
"url": "https://i.ytimg.com/vi/_HfdncCbMOE/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLB15aZU5-tzwYCEbNTnZK3UX6ZCzQ",
"width": 336,
"height": 188
}
]
},
"overlays": [
{
"thumbnailBottomOverlayViewModel": {
"badges": [
{
"thumbnailBadgeViewModel": {
"text": "1:08:18",
"badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
"animationActivationTargetId": "_HfdncCbMOE",
"animatedText": "Now playing",
"animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
}
}
]
}
}
]
}
},
"metadata": {
"lockupMetadataViewModel": {
"title": {
"content": "MIT 6.S191 (2025): Large Language Models (Liquid AI)"
},
"image": {
"decoratedAvatarViewModel": {
"avatar": {
"avatarViewModel": {
"image": {
"sources": [
{
"url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
"width": 68,
"height": 68
}
]
},
"avatarImageSize": "AVATAR_SIZE_M"
}
},
"a11yLabel": "Go to channel Alexander Amini",
"rendererContext": {
"commandContext": {
"onTap": {
"innertubeCommand": {
"browseEndpoint": {
"browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
}
}
}
}
}
}
},
"metadata": {
"contentMetadataViewModel": {
"metadataRows": [
{
"metadataParts": [
{
"text": {
"content": "Alexander Amini"
}
}
]
},
{
"metadataParts": [
{
"text": {
"content": "62K"
},
"accessibilityLabel": "62 thousand views",
"leadingIcon": {
"name": "PLAY_ARROW_OUTLINED",
"height": 12,
"width": 12,
"marginEndToken": {
"spacing": 2
}
}
},
{
"text": {
"content": "1y ago"
},
"accessibilityLabel": "1 year ago"
}
]
}
],
"delimiter": " "
}
}
}
},
"contentId": "_HfdncCbMOE",
"contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
"themedPalette": {
"lightThemePalette": {
"baseBackground": 4294964720,
"raisedBackground": 4294964720,
"additiveBackground": 566850118,
"textPrimary": 4281800969,
"textSecondary": 4285876787,
"outline": 860759827,
"keyLight": 4292983720,
"collectionStack": 4291337092,
"invertedBackground": 4287578649,
"textPrimaryInverse": 4294964720,
"textDisabled": 2135828243,
"dropShadow": 432632390,
"cardOutline": 180974150,
"overlayBackground": 3426162953,
"overlayTextPrimary": 4294964720,
"solidWash": 4283311891,
"solidRim": 4281800969,
"overlaySolidWash": 4294699488,
"overlaySolidRim": 4294964720,
"tonalWash": 2583688688,
"tonalRim": 202310912,
"overlayTonalWash": 436205040,
"overlayTonalRim": 436205040,
"outlineRim": 1280190227,
"overlayOutlineRim": 871618504,
"solidWashInverse": 4294170568,
"solidRimInverse": 2147215840,
"tonalWashInverse": 654308848,
"tonalRimInverse": 654043616,
"outlineRimInverse": 654308848,
"solidBackground": 4283311891,
"solidBackgroundInverse": 4291337092,
"tonalBackground": 868785028,
"tonalBackgroundInverse": 435410888,
"outlineInverse": 872147424,
"directionalLightStopPrimary": 4294699488,
"directionalLightStopSecondary": 4294964720,
"primary": 4279174400,
"filledStateLayer": 4281800969,
"tonalTouchLight": 4294964720,
"tonalHoverLight": 3439061472,
"tonalStateLayer": 420414720,
"overlayDisabledContainer": 871618504,
"overlayTonalTouchLight": 1291577824,
"overlayTonalHoverLight": 1291577824,
"overlayTonalStateLayer": 436205040,
"disabledContainer": 420414720,
"filledTouchLight": 4285876787,
"filledHoverLight": 4284627235,
"onOverlayDisabledContainer": 2583688688,
"overlayTonalContainer": 2583423456,
"overlayOutline": 872412656,
"onFilledContainer": 4294964720,
"filledContainer": 4280027906,
"tonalContainer": 641145097,
"overlayPrimary": 4294964720,
"onOverlayFilledContainer": 4279174400,
"overlayFilledTouchLight": 4294964720,
"overlayFilledHoverLight": 4294699488,
"overlayFilledStateLayer": 4292983720,
"overlayFilledContainer": 4294170568,
"onDisabledContainer": 1712260352,
"temporaryOutline": 860759827
},
"darkThemePalette": {
"baseBackground": 4280027906,
"raisedBackground": 4281800969,
"additiveBackground": 734622278,
"textPrimary": 4294699488,
"textSecondary": 4292983720,
"outline": 1072945096,
"keyLight": 4284627235,
"collectionStack": 4285876787,
"invertedBackground": 4293107837,
"textPrimaryInverse": 4280027906,
"textDisabled": 2146686920,
"dropShadow": 432632390,
"cardOutline": 180974150,
"overlayBackground": 3426162953,
"overlayTextPrimary": 4294964720,
"solidWash": 4294699488,
"solidRim": 4294964720,
"overlaySolidWash": 4294699488,
"overlaySolidRim": 4294964720,
"tonalWash": 217307080,
"tonalRim": 218101232,
"overlayTonalWash": 436205040,
"overlayTonalRim": 436205040,
"outlineRim": 871618504,
"overlayOutlineRim": 871618504,
"solidWashInverse": 4285876787,
"solidRimInverse": 871618504,
"tonalWashInverse": 872412656,
"tonalRimInverse": 870431656,
"outlineRimInverse": 1291843056,
"solidBackground": 4291337092,
"solidBackgroundInverse": 4283311891,
"tonalBackground": 435410888,
"tonalBackgroundInverse": 868785028,
"outlineInverse": 860759827,
"directionalLightStopPrimary": 4281800969,
"directionalLightStopSecondary": 4280027906,
"primary": 4294964720,
"filledStateLayer": 4292983720,
"tonalTouchLight": 1073474016,
"tonalHoverLight": 654043616,
"tonalStateLayer": 436205040,
"overlayDisabledContainer": 871618504,
"overlayTonalTouchLight": 1291577824,
"overlayTonalHoverLight": 1291577824,
"overlayTonalStateLayer": 436205040,
"disabledContainer": 436205040,
"filledTouchLight": 4294964720,
"filledHoverLight": 4294699488,
"onOverlayDisabledContainer": 2583688688,
"overlayTonalContainer": 2583423456,
"overlayOutline": 872412656,
"onFilledContainer": 4279174400,
"filledContainer": 4294170568,
"tonalContainer": 435410888,
"overlayPrimary": 4294964720,
"onOverlayFilledContainer": 4279174400,
"overlayFilledTouchLight": 4294964720,
"overlayFilledHoverLight": 4294699488,
"overlayFilledStateLayer": 4292983720,
"overlayFilledContainer": 4294170568,
"onDisabledContainer": 1728050672,
"temporaryOutline": 871618504
}
}
}
},
{
"lockupViewModel": {
"contentImage": {
"thumbnailViewModel": {
"image": {
"sources": [
{
"url": "https://i.ytimg.com/vi/P7Hkh2zOGQ0/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLAHInNtMierSAgtrGl5rLSKc-vdzQ",
"width": 168,
"height": 94
},
{
"url": "https://i.ytimg.com/vi/P7Hkh2zOGQ0/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLDXrm5g-rdjRIqqK309ImTdqpw4CA",
"width": 336,
"height": 188
}
]
},
"overlays": [
{
"thumbnailBottomOverlayViewModel": {
"badges": [
{
"thumbnailBadgeViewModel": {
"text": "1:08:19",
"badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
"animationActivationTargetId": "P7Hkh2zOGQ0",
"animatedText": "Now playing",
"animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
}
}
]
}
}
]
}
},
"metadata": {
"lockupMetadataViewModel": {
"title": {
"content": "MIT 6.S191 (2024): Google - Generative AI for Media"
},
"image": {
"decoratedAvatarViewModel": {
"avatar": {
"avatarViewModel": {
"image": {
"sources": [
{
"url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
"width": 68,
"height": 68
}
]
},
"avatarImageSize": "AVATAR_SIZE_M"
}
},
"a11yLabel": "Go to channel Alexander Amini",
"rendererContext": {
"commandContext": {
"onTap": {
"innertubeCommand": {
"browseEndpoint": {
"browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
}
}
}
}
}
}
},
"metadata": {
"contentMetadataViewModel": {
"metadataRows": [
{
"metadataParts": [
{
"text": {
"content": "Alexander Amini"
}
}
]
},
{
"metadataParts": [
{
"text": {
"content": "49K"
},
"accessibilityLabel": "49 thousand views",
"leadingIcon": {
"name": "PLAY_ARROW_OUTLINED",
"height": 12,
"width": 12,
"marginEndToken": {
"spacing": 2
}
}
},
{
"text": {
"content": "2y ago"
},
"accessibilityLabel": "2 years ago"
}
]
}
],
"delimiter": " "
}
}
}
},
"contentId": "P7Hkh2zOGQ0",
"contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
"themedPalette": {
"lightThemePalette": {
"baseBackground": 4293981951,
"raisedBackground": 4293981951,
"additiveBackground": 558201801,
"textPrimary": 4278655801,
"textSecondary": 4280831104,
"outline": 856631890,
"keyLight": 4288855783,
"collectionStack": 4286094803,
"invertedBackground": 4279847311,
"textPrimaryInverse": 4293981951,
"textDisabled": 2131700306,
"dropShadow": 423984073,
"cardOutline": 172325833,
"overlayBackground": 3423017785,
"overlayTextPrimary": 4293981951,
"solidWash": 4279183954,
"solidRim": 4278655801,
"overlaySolidWash": 4292864764,
"overlaySolidRim": 4293981951,
"tonalWash": 2582705919,
"tonalRim": 201328143,
"overlayTonalWash": 435222271,
"overlayTonalRim": 435222271,
"outlineRim": 1276062290,
"overlayOutlineRim": 868669942,
"solidWashInverse": 4291222006,
"solidRimInverse": 2145381116,
"tonalWashInverse": 653326079,
"tonalRimInverse": 652208892,
"outlineRimInverse": 653326079,
"solidBackground": 4279183954,
"solidBackgroundInverse": 4286094803,
"tonalBackground": 863542739,
"tonalBackgroundInverse": 432462326,
"outlineInverse": 870312700,
"directionalLightStopPrimary": 4292864764,
"directionalLightStopSecondary": 4293981951,
"primary": 4278191631,
"filledStateLayer": 4278655801,
"tonalTouchLight": 4293981951,
"tonalHoverLight": 3437226748,
"tonalStateLayer": 419431951,
"overlayDisabledContainer": 868669942,
"overlayTonalTouchLight": 1289743100,
"overlayTonalHoverLight": 1289743100,
"overlayTonalStateLayer": 435222271,
"disabledContainer": 419431951,
"filledTouchLight": 4280831104,
"filledHoverLight": 4280040297,
"onOverlayDisabledContainer": 2582705919,
"overlayTonalContainer": 2581588732,
"overlayOutline": 871429887,
"onFilledContainer": 4293981951,
"filledContainer": 4278324509,
"tonalContainer": 637999929,
"overlayPrimary": 4293981951,
"onOverlayFilledContainer": 4278191631,
"overlayFilledTouchLight": 4293981951,
"overlayFilledHoverLight": 4292864764,
"overlayFilledStateLayer": 4288855783,
"overlayFilledContainer": 4291222006,
"onDisabledContainer": 1711277583,
"temporaryOutline": 856631890
},
"darkThemePalette": {
"baseBackground": 4278324509,
"raisedBackground": 4278655801,
"additiveBackground": 725973961,
"textPrimary": 4292864764,
"textSecondary": 4288855783,
"outline": 1069996534,
"keyLight": 4280040297,
"collectionStack": 4280831104,
"invertedBackground": 4286359267,
"textPrimaryInverse": 4278324509,
"textDisabled": 2143738358,
"dropShadow": 423984073,
"cardOutline": 172325833,
"overlayBackground": 3423017785,
"overlayTextPrimary": 4293981951,
"solidWash": 4292864764,
"solidRim": 4293981951,
"overlaySolidWash": 4292864764,
"overlaySolidRim": 4293981951,
"tonalWash": 214358518,
"tonalRim": 217118463,
"overlayTonalWash": 435222271,
"overlayTonalRim": 435222271,
"outlineRim": 868669942,
"overlayOutlineRim": 868669942,
"solidWashInverse": 4280831104,
"solidRimInverse": 868669942,
"tonalWashInverse": 871429887,
"tonalRimInverse": 866303719,
"outlineRimInverse": 1290860287,
"solidBackground": 4286094803,
"solidBackgroundInverse": 4279183954,
"tonalBackground": 432462326,
"tonalBackgroundInverse": 863542739,
"outlineInverse": 856631890,
"directionalLightStopPrimary": 4278655801,
"directionalLightStopSecondary": 4278324509,
"primary": 4293981951,
"filledStateLayer": 4288855783,
"tonalTouchLight": 1071639292,
"tonalHoverLight": 652208892,
"tonalStateLayer": 435222271,
"overlayDisabledContainer": 868669942,
"overlayTonalTouchLight": 1289743100,
"overlayTonalHoverLight": 1289743100,
"overlayTonalStateLayer": 435222271,
"disabledContainer": 435222271,
"filledTouchLight": 4293981951,
"filledHoverLight": 4292864764,
"onOverlayDisabledContainer": 2582705919,
"overlayTonalContainer": 2581588732,
"overlayOutline": 871429887,
"onFilledContainer": 4278191631,
"filledContainer": 4291222006,
"tonalContainer": 432462326,
"overlayPrimary": 4293981951,
"onOverlayFilledContainer": 4278191631,
"overlayFilledTouchLight": 4293981951,
"overlayFilledHoverLight": 4292864764,
"overlayFilledStateLayer": 4288855783,
"overlayFilledContainer": 4291222006,
"onDisabledContainer": 1727067903,
"temporaryOutline": 868669942
}
}
}
},
{
"continuationItemRenderer": {
"trigger": "CONTINUATION_TRIGGER_ON_ITEM_SHOWN",
"continuationEndpoint": {
"commandMetadata": {
"webCommandMetadata": {
"sendPost": true,
"apiUrl": "/youtubei/v1/next"
}
},
"continuationCommand": {
"token": "CCgSDRILU2RUWkFNREtyTlkYACqPDE1nb0lodVQ4eG9EdXVLSUYyczZMNnd6NENBcjFDQW9EOGo0QUNnN0NQZ3NJbk1XcnZzbWo2Y2UyQVFvRDhqNEFDZzNDUGdvSW90N0czWnVCMy1KSENnUHlQZ0FLRGNJLUNnaTVfOGVUbjV5cTJSd0tBX0ktQUFvT3dqNExDSkQydWJfNXp2ZXZfUUVLQV9JLUFBb053ajRLQ1BXWjVQakt2cjZWU2dvRDhqNEFDZzNDUGdvSW9aTDd1OUtGdF85NkNnUHlQZ0FLRGNJLUNnaUh3N2FLMzd6YzVIZ0tBX0ktQUFvT3dqNExDUE94aHRQTWtfVzB1UUVLQV9JLUFBb093ajRMQ092eTBzaXNsSWZScGdFS0FfSS1BQW9Pd2o0TENKS1FxdHpBeE5lZHJBRUtBX0ktQUFvTndqNEtDTFdteWE2cHNPMkJPZ29EOGo0QUNnM0NQZ29JanYtdy1wQ3k1YVozQ2dQeVBnQUtEc0ktQ3dqMmtmVF9tYlB1XzRvQkNnUHlQZ0FLRHNJLUN3aVpuYWZjc0tQVmpZUUJDZ1B5UGdBS0RjSS1DZ2lxM3BxM3FyX2s0U0VLQV9JLUFBb053ajRLQ0phNXU5eVJvTmJOS0FvRDhqNEFDZzNDUGdvSXFMeWR3T1NhNGJkV0NnUHlQZ0FLRGNJLUNnakh3X2ZucktlSDdXUUtBX0ktQUFvTndqNEtDUG4xak1mcXg4QzJNQW9EOGo0QUNnM0NQZ29JeWRpdXV0VDQ3UHNhQ2dQeVBnQUtEc0ktQ3dqaDRleUUzTFAzdV93QkNnUHlQZ0FLRGNJLUNnaU5zcmptOXBENTJEOEtBX0ktQUFvTndqNEtDTExudjR5ZXg3U1dTUW9EOGo0QUNnM0NQZ29JNnZHODhaR1JpTWNnQ2dQeVBnQUtEc0ktQ3dpRF85S3Z5WS1hbThJQkNnUHlQZ0FLRHNJLUN3alppNmJYbWR6YWxZb0JDZ1B5UGdBS0RzSS1Dd2lkemJfS2dPdWN0YUFCQ2dQeVBnQUtEY0ktQ2dqMXktMjYyb2JQeFdNS0FfSS1BQW9Pd2o0TENPLUE4SWVaMUlESXJRRUtBX0ktQUFvTndqNEtDUHlZNDVhUzhkeWpJZ29EOGo0QUNnM0NQZ29Jb191aWtfSE05ZHdTQ2dQeVBnQUtEc0ktQ3dqaTBiemlrdnpBbkxvQkNnUHlQZ0FLRHNJLUN3aW05cXI1eXNtTWlzOEJDZ1B5UGdBS0RjSS1DZ2k3M2V5NXlzQ1pzMG9LQV9JLUFBb093ajRMQ0xmT281WFU1X2JPbFFFS0FfSS1BQW9Od2o0S0NLTEVwYXlXOTliWkdnb0Q4ajRBQ2c3Q1Bnc0k2dWZSMnRPM3hLem9BUW9EOGo0QUNnN0NQZ3NJN2ZlZXZQRHV2WlRXQVFvRDhqNEFDZzNDUGdvSXFibS0zYXVBdDVWcUNnUHlQZ0FLRGNJLUNnaUc1UHpHZ082NG9nVVNLQUFDQkFZSUNnd09FQklVRmhnYUhCNGdJaVFtS0Nvc0xqQXlORFk0T2p3LVFFSkVSa2hLVEU0YUJBZ0FFQUVhQkFnQ0VBTWFCQWdFRUFVYUJBZ0dFQWNhQkFnSUVBa2FCQWdLRUFzYUJBZ01FQTBhQkFnT0VBOGFCQWdRRUJFYUJBZ1NFQk1hQkFnVUVCVWFCQWdXRUJjYUJBZ1lFQmthQkFnYUVCc2FCQWdjRUIwYUJBZ2VFQjhhQkFnZ0VDRWFCQWdpRUNNYUJBZ2tFQ1VhQkFnbUVDY2FCQWdvRUNrYUJBZ3FFQ3NhQkFnc0VDMGFCQWd1RUM4YUJBZ3dFREVhQkFneUVETWFCQWcwRURVYUJBZzJFRGNhQkFnNEVEa2FCQWc2RURzYUJBZzhFRDBhQkFnLUVEOGFCQWhBRUVFYUJBaENFRU1hQkFoRUVFVWFCQWhHRUVjYUJBaElFRWthQkFoS0VFc2FCQWhNRUUwYUJBaE9FRThxS0FBQ0JBWUlDZ3dPRUJJVUZoZ2FIQjRnSWlRbUtDb3NMakF5TkRZNE9qdy1RRUpFUmtoS1RFNGoPd2F0Y2gtbmV4dC1mZWVk",
"request": "CONTINUATION_REQUEST_TYPE_WATCH_NEXT"
}
}
}
}
]
}
}
]
}
+805
View File
@@ -0,0 +1,805 @@
{
"contents": {
"twoColumnWatchNextResults": {
"secondaryResults": {
"secondaryResults": {
"results": [
{
"lockupViewModel": {
"contentImage": {
"thumbnailViewModel": {
"image": {
"sources": [
{
"url": "https://i.ytimg.com/vi/to-lHJfK4pw/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLBKP_oFChvoqvwVrKAy_PG9bn2_AA",
"width": 168,
"height": 94
},
{
"url": "https://i.ytimg.com/vi/to-lHJfK4pw/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLDyH3Z8DJhXMmFJ5eKEWMufk7ql9g",
"width": 336,
"height": 188
}
]
},
"overlays": [
{
"thumbnailBottomOverlayViewModel": {
"badges": [
{
"thumbnailBadgeViewModel": {
"text": "1:02:00",
"badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
"animationActivationTargetId": "to-lHJfK4pw",
"animatedText": "Now playing",
"animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
}
}
]
}
}
]
}
},
"metadata": {
"lockupMetadataViewModel": {
"title": {
"content": "MIT 6.S191 (2025): Reinforcement Learning"
},
"image": {
"decoratedAvatarViewModel": {
"avatar": {
"avatarViewModel": {
"image": {
"sources": [
{
"url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
"width": 68,
"height": 68
}
]
},
"avatarImageSize": "AVATAR_SIZE_M"
}
},
"a11yLabel": "Go to channel Alexander Amini",
"rendererContext": {
"commandContext": {
"onTap": {
"innertubeCommand": {
"browseEndpoint": {
"browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
}
}
}
}
}
}
},
"metadata": {
"contentMetadataViewModel": {
"metadataRows": [
{
"metadataParts": [
{
"text": {
"content": "Alexander Amini"
}
}
]
},
{
"metadataParts": [
{
"text": {
"content": "128K"
},
"accessibilityLabel": "128 thousand views",
"leadingIcon": {
"name": "PLAY_ARROW_OUTLINED",
"height": 12,
"width": 12,
"marginEndToken": {
"spacing": 2
}
}
},
{
"text": {
"content": "1y ago"
},
"accessibilityLabel": "1 year ago"
}
]
}
],
"delimiter": " "
}
}
}
},
"contentId": "to-lHJfK4pw",
"contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
"themedPalette": {
"lightThemePalette": {
"baseBackground": 4293981439,
"raisedBackground": 4293981439,
"additiveBackground": 558196425,
"textPrimary": 4278850870,
"textSecondary": 4282009454,
"outline": 857088843,
"keyLight": 4289509597,
"collectionStack": 4287338944,
"invertedBackground": 4279842703,
"textPrimaryInverse": 4293981439,
"textDisabled": 2132157259,
"dropShadow": 423978697,
"cardOutline": 172320457,
"overlayBackground": 3423212854,
"overlayTextPrimary": 4293981439,
"solidWash": 4279640907,
"solidRim": 4278850870,
"overlaySolidWash": 4292929530,
"overlaySolidRim": 4293981439,
"tonalWash": 2582705407,
"tonalRim": 201327631,
"overlayTonalWash": 435221759,
"overlayTonalRim": 435221759,
"outlineRim": 1276519243,
"overlayOutlineRim": 868930802,
"solidWashInverse": 4291482866,
"solidRimInverse": 2145445882,
"tonalWashInverse": 653325567,
"tonalRimInverse": 652273658,
"outlineRimInverse": 653325567,
"solidBackground": 4279640907,
"solidBackgroundInverse": 4287338944,
"tonalBackground": 864786880,
"tonalBackgroundInverse": 432723186,
"outlineInverse": 870377466,
"directionalLightStopPrimary": 4292929530,
"directionalLightStopSecondary": 4293981439,
"primary": 4278191119,
"filledStateLayer": 4278850870,
"tonalTouchLight": 4293981439,
"tonalHoverLight": 3437291514,
"tonalStateLayer": 419431439,
"overlayDisabledContainer": 868930802,
"overlayTonalTouchLight": 1289807866,
"overlayTonalHoverLight": 1289807866,
"overlayTonalStateLayer": 435221759,
"disabledContainer": 419431439,
"filledTouchLight": 4282009454,
"filledHoverLight": 4280825181,
"onOverlayDisabledContainer": 2582705407,
"overlayTonalContainer": 2581653498,
"overlayOutline": 871429375,
"onFilledContainer": 4293981439,
"filledContainer": 4278323484,
"tonalContainer": 638194998,
"overlayPrimary": 4293981439,
"onOverlayFilledContainer": 4278191119,
"overlayFilledTouchLight": 4293981439,
"overlayFilledHoverLight": 4292929530,
"overlayFilledStateLayer": 4289509597,
"overlayFilledContainer": 4291482866,
"onDisabledContainer": 1711277071,
"temporaryOutline": 857088843
},
"darkThemePalette": {
"baseBackground": 4278323484,
"raisedBackground": 4278850870,
"additiveBackground": 725968585,
"textPrimary": 4292929530,
"textSecondary": 4289509597,
"outline": 1070257394,
"keyLight": 4280825181,
"collectionStack": 4282009454,
"invertedBackground": 4286355427,
"textPrimaryInverse": 4278323484,
"textDisabled": 2143999218,
"dropShadow": 423978697,
"cardOutline": 172320457,
"overlayBackground": 3423212854,
"overlayTextPrimary": 4293981439,
"solidWash": 4292929530,
"solidRim": 4293981439,
"overlaySolidWash": 4292929530,
"overlaySolidRim": 4293981439,
"tonalWash": 214619378,
"tonalRim": 217117951,
"overlayTonalWash": 435221759,
"overlayTonalRim": 435221759,
"outlineRim": 868930802,
"overlayOutlineRim": 868930802,
"solidWashInverse": 4282009454,
"solidRimInverse": 868930802,
"tonalWashInverse": 871429375,
"tonalRimInverse": 866957533,
"outlineRimInverse": 1290859775,
"solidBackground": 4287338944,
"solidBackgroundInverse": 4279640907,
"tonalBackground": 432723186,
"tonalBackgroundInverse": 864786880,
"outlineInverse": 857088843,
"directionalLightStopPrimary": 4278850870,
"directionalLightStopSecondary": 4278323484,
"primary": 4293981439,
"filledStateLayer": 4289509597,
"tonalTouchLight": 1071704058,
"tonalHoverLight": 652273658,
"tonalStateLayer": 435221759,
"overlayDisabledContainer": 868930802,
"overlayTonalTouchLight": 1289807866,
"overlayTonalHoverLight": 1289807866,
"overlayTonalStateLayer": 435221759,
"disabledContainer": 435221759,
"filledTouchLight": 4293981439,
"filledHoverLight": 4292929530,
"onOverlayDisabledContainer": 2582705407,
"overlayTonalContainer": 2581653498,
"overlayOutline": 871429375,
"onFilledContainer": 4278191119,
"filledContainer": 4291482866,
"tonalContainer": 432723186,
"overlayPrimary": 4293981439,
"onOverlayFilledContainer": 4278191119,
"overlayFilledTouchLight": 4293981439,
"overlayFilledHoverLight": 4292929530,
"overlayFilledStateLayer": 4289509597,
"overlayFilledContainer": 4291482866,
"onDisabledContainer": 1727067391,
"temporaryOutline": 868930802
}
}
}
},
{
"lockupViewModel": {
"contentImage": {
"thumbnailViewModel": {
"image": {
"sources": [
{
"url": "https://i.ytimg.com/vi/R8V8CbuxryI/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLDqXMjrpgDyQKHL2lWrxczwxKoefw",
"width": 168,
"height": 94
},
{
"url": "https://i.ytimg.com/vi/R8V8CbuxryI/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLB7JX9Z238s6DGqIS5-fhcEIkUxKw",
"width": 336,
"height": 188
}
]
},
"overlays": [
{
"thumbnailBottomOverlayViewModel": {
"badges": [
{
"thumbnailBadgeViewModel": {
"text": "49:36",
"badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
"animationActivationTargetId": "R8V8CbuxryI",
"animatedText": "Now playing",
"animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
}
}
]
}
}
]
}
},
"metadata": {
"lockupMetadataViewModel": {
"title": {
"content": "MIT 6.S191: Deep Generative Modeling"
},
"image": {
"decoratedAvatarViewModel": {
"avatar": {
"avatarViewModel": {
"image": {
"sources": [
{
"url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
"width": 68,
"height": 68
}
]
},
"avatarImageSize": "AVATAR_SIZE_M"
}
},
"a11yLabel": "Go to channel Alexander Amini",
"rendererContext": {
"commandContext": {
"onTap": {
"innertubeCommand": {
"browseEndpoint": {
"browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
}
}
}
}
}
}
},
"metadata": {
"contentMetadataViewModel": {
"metadataRows": [
{
"metadataParts": [
{
"text": {
"content": "Alexander Amini"
}
}
]
},
{
"metadataParts": [
{
"text": {
"content": "38K"
},
"accessibilityLabel": "38 thousand views",
"leadingIcon": {
"name": "PLAY_ARROW_OUTLINED",
"height": 12,
"width": 12,
"marginEndToken": {
"spacing": 2
}
}
},
{
"text": {
"content": "6mo ago"
},
"accessibilityLabel": "6 months ago"
}
]
}
],
"delimiter": " "
}
}
}
},
"contentId": "R8V8CbuxryI",
"contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
"themedPalette": {
"lightThemePalette": {
"baseBackground": 4294965488,
"raisedBackground": 4294965488,
"additiveBackground": 566856261,
"textPrimary": 4281737226,
"textSecondary": 4285420858,
"outline": 860565782,
"keyLight": 4292724140,
"collectionStack": 4290815883,
"invertedBackground": 4287583769,
"textPrimaryInverse": 4294965488,
"textDisabled": 2135634198,
"dropShadow": 432638533,
"cardOutline": 180980293,
"overlayBackground": 3426099210,
"overlayTextPrimary": 4294965488,
"solidWash": 4283117846,
"solidRim": 4281737226,
"overlaySolidWash": 4294635232,
"overlaySolidRim": 4294965488,
"tonalWash": 2583689456,
"tonalRim": 202311680,
"overlayTonalWash": 436205808,
"overlayTonalRim": 436205808,
"outlineRim": 1279996182,
"overlayOutlineRim": 871554762,
"solidWashInverse": 4294106826,
"solidRimInverse": 2147151584,
"tonalWashInverse": 654309616,
"tonalRimInverse": 653979360,
"outlineRimInverse": 654309616,
"solidBackground": 4283117846,
"solidBackgroundInverse": 4290815883,
"tonalBackground": 868263819,
"tonalBackgroundInverse": 435347146,
"outlineInverse": 872083168,
"directionalLightStopPrimary": 4294635232,
"directionalLightStopSecondary": 4294965488,
"primary": 4279175168,
"filledStateLayer": 4281737226,
"tonalTouchLight": 4294965488,
"tonalHoverLight": 3438997216,
"tonalStateLayer": 420415488,
"overlayDisabledContainer": 871554762,
"overlayTonalTouchLight": 1291513568,
"overlayTonalHoverLight": 1291513568,
"overlayTonalStateLayer": 436205808,
"disabledContainer": 420415488,
"filledTouchLight": 4285420858,
"filledHoverLight": 4284302120,
"onOverlayDisabledContainer": 2583689456,
"overlayTonalContainer": 2583359200,
"overlayOutline": 872413424,
"onFilledContainer": 4294965488,
"filledContainer": 4280029186,
"tonalContainer": 641081354,
"overlayPrimary": 4294965488,
"onOverlayFilledContainer": 4279175168,
"overlayFilledTouchLight": 4294965488,
"overlayFilledHoverLight": 4294635232,
"overlayFilledStateLayer": 4292724140,
"overlayFilledContainer": 4294106826,
"onDisabledContainer": 1712261120,
"temporaryOutline": 860565782
},
"darkThemePalette": {
"baseBackground": 4280029186,
"raisedBackground": 4281737226,
"additiveBackground": 734628421,
"textPrimary": 4294635232,
"textSecondary": 4292724140,
"outline": 1072881354,
"keyLight": 4284302120,
"collectionStack": 4285420858,
"invertedBackground": 4293112700,
"textPrimaryInverse": 4280029186,
"textDisabled": 2146623178,
"dropShadow": 432638533,
"cardOutline": 180980293,
"overlayBackground": 3426099210,
"overlayTextPrimary": 4294965488,
"solidWash": 4294635232,
"solidRim": 4294965488,
"overlaySolidWash": 4294635232,
"overlaySolidRim": 4294965488,
"tonalWash": 217243338,
"tonalRim": 218102000,
"overlayTonalWash": 436205808,
"overlayTonalRim": 436205808,
"outlineRim": 871554762,
"overlayOutlineRim": 871554762,
"solidWashInverse": 4285420858,
"solidRimInverse": 871554762,
"tonalWashInverse": 872413424,
"tonalRimInverse": 870172076,
"outlineRimInverse": 1291843824,
"solidBackground": 4290815883,
"solidBackgroundInverse": 4283117846,
"tonalBackground": 435347146,
"tonalBackgroundInverse": 868263819,
"outlineInverse": 860565782,
"directionalLightStopPrimary": 4281737226,
"directionalLightStopSecondary": 4280029186,
"primary": 4294965488,
"filledStateLayer": 4292724140,
"tonalTouchLight": 1073409760,
"tonalHoverLight": 653979360,
"tonalStateLayer": 436205808,
"overlayDisabledContainer": 871554762,
"overlayTonalTouchLight": 1291513568,
"overlayTonalHoverLight": 1291513568,
"overlayTonalStateLayer": 436205808,
"disabledContainer": 436205808,
"filledTouchLight": 4294965488,
"filledHoverLight": 4294635232,
"onOverlayDisabledContainer": 2583689456,
"overlayTonalContainer": 2583359200,
"overlayOutline": 872413424,
"onFilledContainer": 4279175168,
"filledContainer": 4294106826,
"tonalContainer": 435347146,
"overlayPrimary": 4294965488,
"onOverlayFilledContainer": 4279175168,
"overlayFilledTouchLight": 4294965488,
"overlayFilledHoverLight": 4294635232,
"overlayFilledStateLayer": 4292724140,
"overlayFilledContainer": 4294106826,
"onDisabledContainer": 1728051440,
"temporaryOutline": 871554762
}
}
}
},
{
"lockupViewModel": {
"contentImage": {
"thumbnailViewModel": {
"image": {
"sources": [
{
"url": "https://i.ytimg.com/vi/HLKo4fJx_7k/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLAQcvMQEX1ny9Z8qu2mIMogpEMaVA",
"width": 168,
"height": 94
},
{
"url": "https://i.ytimg.com/vi/HLKo4fJx_7k/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLCkoPSkHifs5ifpWXXcb96XG_Ly1w",
"width": 336,
"height": 188
}
]
},
"overlays": [
{
"thumbnailBottomOverlayViewModel": {
"badges": [
{
"thumbnailBadgeViewModel": {
"text": "47:36",
"badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
"animationActivationTargetId": "HLKo4fJx_7k",
"animatedText": "Now playing",
"animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
}
}
]
}
}
]
}
},
"metadata": {
"lockupMetadataViewModel": {
"title": {
"content": "MIT 6.S191 (2025): Language Models and New Frontiers"
},
"image": {
"decoratedAvatarViewModel": {
"avatar": {
"avatarViewModel": {
"image": {
"sources": [
{
"url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
"width": 68,
"height": 68
}
]
},
"avatarImageSize": "AVATAR_SIZE_M"
}
},
"a11yLabel": "Go to channel Alexander Amini",
"rendererContext": {
"commandContext": {
"onTap": {
"innertubeCommand": {
"browseEndpoint": {
"browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
}
}
}
}
}
}
},
"metadata": {
"contentMetadataViewModel": {
"metadataRows": [
{
"metadataParts": [
{
"text": {
"content": "Alexander Amini"
}
}
]
},
{
"metadataParts": [
{
"text": {
"content": "63K"
},
"accessibilityLabel": "63 thousand views",
"leadingIcon": {
"name": "PLAY_ARROW_OUTLINED",
"height": 12,
"width": 12,
"marginEndToken": {
"spacing": 2
}
}
},
{
"text": {
"content": "1y ago"
},
"accessibilityLabel": "1 year ago"
}
]
}
],
"delimiter": " "
}
}
}
},
"contentId": "HLKo4fJx_7k",
"contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
"themedPalette": {
"lightThemePalette": {
"baseBackground": 4294964208,
"raisedBackground": 4294964208,
"additiveBackground": 566847045,
"textPrimary": 4281734410,
"textSecondary": 4285417018,
"outline": 860561942,
"keyLight": 4292720812,
"collectionStack": 4290812043,
"invertedBackground": 4287575065,
"textPrimaryInverse": 4294964208,
"textDisabled": 2135630358,
"dropShadow": 432629317,
"cardOutline": 180971077,
"overlayBackground": 3426096394,
"overlayTextPrimary": 4294964208,
"solidWash": 4283114006,
"solidRim": 4281734410,
"overlaySolidWash": 4294633184,
"overlaySolidRim": 4294964208,
"tonalWash": 2583688176,
"tonalRim": 202310656,
"overlayTonalWash": 436204528,
"overlayTonalRim": 436204528,
"outlineRim": 1279992342,
"overlayOutlineRim": 871551946,
"solidWashInverse": 4294104010,
"solidRimInverse": 2147149536,
"tonalWashInverse": 654308336,
"tonalRimInverse": 653977312,
"outlineRimInverse": 654308336,
"solidBackground": 4283114006,
"solidBackgroundInverse": 4290812043,
"tonalBackground": 868259979,
"tonalBackgroundInverse": 435344330,
"outlineInverse": 872081120,
"directionalLightStopPrimary": 4294633184,
"directionalLightStopSecondary": 4294964208,
"primary": 4279174144,
"filledStateLayer": 4281734410,
"tonalTouchLight": 4294964208,
"tonalHoverLight": 3438995168,
"tonalStateLayer": 420414464,
"overlayDisabledContainer": 871551946,
"overlayTonalTouchLight": 1291511520,
"overlayTonalHoverLight": 1291511520,
"overlayTonalStateLayer": 436204528,
"disabledContainer": 420414464,
"filledTouchLight": 4285417018,
"filledHoverLight": 4284298280,
"onOverlayDisabledContainer": 2583688176,
"overlayTonalContainer": 2583357152,
"overlayOutline": 872412144,
"onFilledContainer": 4294964208,
"filledContainer": 4280027394,
"tonalContainer": 641078538,
"overlayPrimary": 4294964208,
"onOverlayFilledContainer": 4279174144,
"overlayFilledTouchLight": 4294964208,
"overlayFilledHoverLight": 4294633184,
"overlayFilledStateLayer": 4292720812,
"overlayFilledContainer": 4294104010,
"onDisabledContainer": 1712260096,
"temporaryOutline": 860561942
},
"darkThemePalette": {
"baseBackground": 4280027394,
"raisedBackground": 4281734410,
"additiveBackground": 734619205,
"textPrimary": 4294633184,
"textSecondary": 4292720812,
"outline": 1072878538,
"keyLight": 4284298280,
"collectionStack": 4285417018,
"invertedBackground": 4293105020,
"textPrimaryInverse": 4280027394,
"textDisabled": 2146620362,
"dropShadow": 432629317,
"cardOutline": 180971077,
"overlayBackground": 3426096394,
"overlayTextPrimary": 4294964208,
"solidWash": 4294633184,
"solidRim": 4294964208,
"overlaySolidWash": 4294633184,
"overlaySolidRim": 4294964208,
"tonalWash": 217240522,
"tonalRim": 218100720,
"overlayTonalWash": 436204528,
"overlayTonalRim": 436204528,
"outlineRim": 871551946,
"overlayOutlineRim": 871551946,
"solidWashInverse": 4285417018,
"solidRimInverse": 871551946,
"tonalWashInverse": 872412144,
"tonalRimInverse": 870168748,
"outlineRimInverse": 1291842544,
"solidBackground": 4290812043,
"solidBackgroundInverse": 4283114006,
"tonalBackground": 435344330,
"tonalBackgroundInverse": 868259979,
"outlineInverse": 860561942,
"directionalLightStopPrimary": 4281734410,
"directionalLightStopSecondary": 4280027394,
"primary": 4294964208,
"filledStateLayer": 4292720812,
"tonalTouchLight": 1073407712,
"tonalHoverLight": 653977312,
"tonalStateLayer": 436204528,
"overlayDisabledContainer": 871551946,
"overlayTonalTouchLight": 1291511520,
"overlayTonalHoverLight": 1291511520,
"overlayTonalStateLayer": 436204528,
"disabledContainer": 436204528,
"filledTouchLight": 4294964208,
"filledHoverLight": 4294633184,
"onOverlayDisabledContainer": 2583688176,
"overlayTonalContainer": 2583357152,
"overlayOutline": 872412144,
"onFilledContainer": 4279174144,
"filledContainer": 4294104010,
"tonalContainer": 435344330,
"overlayPrimary": 4294964208,
"onOverlayFilledContainer": 4279174144,
"overlayFilledTouchLight": 4294964208,
"overlayFilledHoverLight": 4294633184,
"overlayFilledStateLayer": 4292720812,
"overlayFilledContainer": 4294104010,
"onDisabledContainer": 1728050160,
"temporaryOutline": 871551946
}
}
}
},
{
"continuationItemRenderer": {
"trigger": "CONTINUATION_TRIGGER_ON_ITEM_SHOWN",
"continuationEndpoint": {
"commandMetadata": {
"webCommandMetadata": {
"sendPost": true,
"apiUrl": "/youtubei/v1/next"
}
},
"continuationCommand": {
"token": "CBQSDRILU2RUWkFNREtyTlkYACqXBk1nb0l5ZGl1dXRUNDdQc2EyczZMNnd5LUJBcTdCQW9EOGo0QUNnN0NQZ3NJbk1XcnZzbWo2Y2UyQVFvRDhqNEFDZzNDUGdvSW90N0czWnVCMy1KSENnUHlQZ0FLRGNJLUNnaTVfOGVUbjV5cTJSd0tBX0ktQUFvT3dqNExDSkQydWJfNXp2ZXZfUUVLQV9JLUFBb053ajRLQ1BXWjVQakt2cjZWU2dvRDhqNEFDZzNDUGdvSW9aTDd1OUtGdF85NkNnUHlQZ0FLRGNJLUNnaUh3N2FLMzd6YzVIZ0tBX0ktQUFvT3dqNExDUE94aHRQTWtfVzB1UUVLQV9JLUFBb093ajRMQ092eTBzaXNsSWZScGdFS0FfSS1BQW9Pd2o0TENKS1FxdHpBeE5lZHJBRUtBX0ktQUFvTndqNEtDTFdteWE2cHNPMkJPZ29EOGo0QUNnM0NQZ29JanYtdy1wQ3k1YVozQ2dQeVBnQUtEc0ktQ3dqMmtmVF9tYlB1XzRvQkNnUHlQZ0FLRHNJLUN3aVpuYWZjc0tQVmpZUUJDZ1B5UGdBS0RjSS1DZ2lxM3BxM3FyX2s0U0VLQV9JLUFBb053ajRLQ0phNXU5eVJvTmJOS0FvRDhqNEFDZzNDUGdvSXFMeWR3T1NhNGJkV0NnUHlQZ0FLRGNJLUNnakh3X2ZucktlSDdXUUtBX0ktQUFvTndqNEtDUG4xak1mcXg4QzJNQW9EOGo0QUNnM0NQZ29JeWRpdXV0VDQ3UHNhRWhRQUFnUUdDQW9NRGhBU0ZCWVlHaHdlSUNJa0pob0VDQUFRQVJvRUNBSVFBeG9FQ0FRUUJSb0VDQVlRQnhvRUNBZ1FDUm9FQ0FvUUN4b0VDQXdRRFJvRUNBNFFEeG9FQ0JBUUVSb0VDQklRRXhvRUNCUVFGUm9FQ0JZUUZ4b0VDQmdRR1JvRUNCb1FHeG9FQ0J3UUhSb0VDQjRRSHhvRUNDQVFJUm9FQ0NJUUl4b0VDQ1FRSlJvRUNDWVFKeW9VQUFJRUJnZ0tEQTRRRWhRV0dCb2NIaUFpSkNZag93YXRjaC1uZXh0LWZlZWQ%3D",
"request": "CONTINUATION_REQUEST_TYPE_WATCH_NEXT"
}
},
"button": {
"buttonRenderer": {
"style": "STYLE_SUGGESTIVE",
"size": "SIZE_DEFAULT",
"isDisabled": false,
"text": {
"runs": [
{
"text": "Show more"
}
]
},
"command": {
"commandMetadata": {
"webCommandMetadata": {
"sendPost": true,
"apiUrl": "/youtubei/v1/next"
}
},
"continuationCommand": {
"token": "CBQSDRILU2RUWkFNREtyTlkYACqXBk1nb0l5ZGl1dXRUNDdQc2EyczZMNnd5LUJBcTdCQW9EOGo0QUNnN0NQZ3NJbk1XcnZzbWo2Y2UyQVFvRDhqNEFDZzNDUGdvSW90N0czWnVCMy1KSENnUHlQZ0FLRGNJLUNnaTVfOGVUbjV5cTJSd0tBX0ktQUFvT3dqNExDSkQydWJfNXp2ZXZfUUVLQV9JLUFBb053ajRLQ1BXWjVQakt2cjZWU2dvRDhqNEFDZzNDUGdvSW9aTDd1OUtGdF85NkNnUHlQZ0FLRGNJLUNnaUh3N2FLMzd6YzVIZ0tBX0ktQUFvT3dqNExDUE94aHRQTWtfVzB1UUVLQV9JLUFBb093ajRMQ092eTBzaXNsSWZScGdFS0FfSS1BQW9Pd2o0TENKS1FxdHpBeE5lZHJBRUtBX0ktQUFvTndqNEtDTFdteWE2cHNPMkJPZ29EOGo0QUNnM0NQZ29JanYtdy1wQ3k1YVozQ2dQeVBnQUtEc0ktQ3dqMmtmVF9tYlB1XzRvQkNnUHlQZ0FLRHNJLUN3aVpuYWZjc0tQVmpZUUJDZ1B5UGdBS0RjSS1DZ2lxM3BxM3FyX2s0U0VLQV9JLUFBb053ajRLQ0phNXU5eVJvTmJOS0FvRDhqNEFDZzNDUGdvSXFMeWR3T1NhNGJkV0NnUHlQZ0FLRGNJLUNnakh3X2ZucktlSDdXUUtBX0ktQUFvTndqNEtDUG4xak1mcXg4QzJNQW9EOGo0QUNnM0NQZ29JeWRpdXV0VDQ3UHNhRWhRQUFnUUdDQW9NRGhBU0ZCWVlHaHdlSUNJa0pob0VDQUFRQVJvRUNBSVFBeG9FQ0FRUUJSb0VDQVlRQnhvRUNBZ1FDUm9FQ0FvUUN4b0VDQXdRRFJvRUNBNFFEeG9FQ0JBUUVSb0VDQklRRXhvRUNCUVFGUm9FQ0JZUUZ4b0VDQmdRR1JvRUNCb1FHeG9FQ0J3UUhSb0VDQjRRSHhvRUNDQVFJUm9FQ0NJUUl4b0VDQ1FRSlJvRUNDWVFKeW9VQUFJRUJnZ0tEQTRRRWhRV0dCb2NIaUFpSkNZag93YXRjaC1uZXh0LWZlZWQ%3D",
"request": "CONTINUATION_REQUEST_TYPE_WATCH_NEXT"
}
}
}
}
}
}
]
}
}
}
}
}
+9
View File
@@ -159,6 +159,15 @@ def test_the_run_list_filters_pages_and_names_the_channel_and_the_log_lines(db,
assert facets["total"] == 4
def test_the_run_list_names_the_user_a_takeout_import_ran_for(db, make_user, as_user):
admin = make_user(role="admin")
importer = make_user("importer@example.com")
imported = _run(db, "takeout_import", "ok", importer.id, T0, f"user:{importer.id}", "ad_hoc")
db.commit()
(row,) = as_user(admin).get("/api/admin/runs").json()["items"]
assert (row["id"], row["target_title"]) == (imported, "importer@example.com")
def test_the_quota_breakdown_covers_its_range_and_one_account(db, make_user, as_user):
admin = make_user(role="admin")
now = datetime.now(timezone.utc)
+5
View File
@@ -23,6 +23,7 @@ from app import bodylimit, debugsession, sysconfig
from app.config import settings
from app.models import SystemConfig
from app.bodylimit import DEFAULT_MAX_BODY_BYTES, BodyLimitMiddleware
from app.reader import takeout
from app.routes import diagnostics as diagnostics_routes
SMALL = b"x" * 16
@@ -221,6 +222,7 @@ def test_the_default_ceiling_clears_the_largest_body_a_default_route_takes():
CAPPED_BODY_SITES = {
("diagnostics.py", "_parsed_events", "capped_body", "BODY_MAX_BYTES"),
("debug_sessions.py", "_image_body", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
("feed.py", "_takeout_body", "capped_body", "takeout.MAX_BYTES"),
("files.py", "spool_upload", "capped_stream_to", "cap"),
}
@@ -368,6 +370,9 @@ def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
# The upload site's limit is the sysconfig knob, which is bounded BY the edge ceiling rather
# than being a constant to compare against — that pairing is its own test below.
assert ("POST", "/api/files/upload") in bodylimit.ROUTE_MAX_BODY_BYTES
# A Takeout file is larger than the default: without its ceiling every real history is a 413.
assert takeout.MAX_BYTES > DEFAULT_MAX_BODY_BYTES
assert bodylimit.ROUTE_MAX_BODY_BYTES[("POST", "/api/takeout")] == takeout.MAX_BYTES
def test_release_read_connection_ends_the_transaction_it_was_called_for(db):
+86
View File
@@ -12,6 +12,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from app.catalog import queries, search, service
from app.db import SessionLocal
from app.catalog.models import (
BlockedChannel,
Channel,
@@ -22,6 +23,8 @@ from app.catalog.models import (
)
from app.reader import queries as reader_queries
from app.reader import service as reader_service
from app.sync import explore as sync_explore
from app.sync import subscriptions as sync_subscriptions
def _channel(db, cid="UCc", **kw):
@@ -141,6 +144,89 @@ def test_subscribing_keeps_an_explored_channel_and_its_videos(db, make_user):
assert queries.subscription(db, user, "UCc") is None
def test_exploring_a_channel_someone_follows_brings_its_uploads_in_as_the_catalogs_own(
db, make_user, monkeypatch
):
follower, explorer = make_user(), make_user()
ch = _channel(db, uploads_playlist_id="UUc")
db.add(Subscription(user_id=follower.id, channel_id="UCc"))
db.commit()
service.explore(db, explorer, ch)
db.commit()
class _Yt:
def get_playlist_items(self, playlist_id, page_token):
item = {"contentDetails": {"videoId": "v1", "videoPublishedAt": "2026-05-01T00:00:00Z"}}
return {"items": [item]}
monkeypatch.setattr(sync_explore, "_enrich_new", lambda session, yt, ids: 0)
sync_explore.explore_ingest_page(db, _Yt(), ch)
db.commit()
db.expire_all()
# The channel is followed, so not the exploration's: a video flagged as one would stay flagged
# (subscribing clears only an explored channel's) and its watched marks would wait for good.
assert (ch.from_explore, db.get(Video, "v1").via_explore) == (False, False)
def test_a_follow_pulled_in_from_youtube_makes_an_explored_channel_the_catalogs_own(
db, make_user, monkeypatch
):
explorer, follower = make_user(), make_user()
ch = _channel(db, details_synced_at=datetime.now(timezone.utc))
service.explore(db, explorer, ch)
db.commit()
_video(db, "v1", via_explore=True)
class _Yt:
def __init__(self, session, user):
pass
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def iter_subscriptions(self):
yield {"channel_id": "UCc", "title": "UCc", "yt_subscription_id": "sub-UCc"}
def get_channels(self, ids):
return []
monkeypatch.setattr(sync_subscriptions, "YouTubeClient", _Yt)
sync_subscriptions.import_subscriptions(db, follower)
db.commit()
db.expire_all()
# Followed on youtube.com is followed all the same: left flagged, the channel's later videos
# came in as an exploration's, hidden from the Library and their watched marks waiting for good.
assert (db.get(Channel, "UCc").from_explore, db.get(Video, "v1").via_explore) == (False, False)
def test_a_follow_landing_while_an_explored_page_comes_in_leaves_none_of_it_flagged(
db, make_user, monkeypatch
):
explorer, follower = make_user(), make_user()
ch = _channel(db, uploads_playlist_id="UUc")
service.explore(db, explorer, ch)
db.commit()
class _Yt:
def get_playlist_items(self, playlist_id, page_token):
# Someone subscribes, in a request of their own, while YouTube answers this one.
with SessionLocal() as other:
service.record_subscription(other, follower, other.get(Channel, "UCc"), "sub-UCc")
other.commit()
item = {"contentDetails": {"videoId": "v1", "videoPublishedAt": "2026-05-01T00:00:00Z"}}
return {"items": [item]}
monkeypatch.setattr(sync_explore, "_enrich_new", lambda session, yt, ids: 0)
sync_explore.explore_ingest_page(db, _Yt(), ch)
db.commit()
db.expire_all()
# The subscribe cleared what was flagged before it; a page that lands after it must not be.
assert (db.get(Channel, "UCc").from_explore, db.get(Video, "v1").via_explore) == (False, False)
def test_a_block_is_once_and_ends_the_exploration(db, make_user):
user = make_user()
ch = _channel(db)
+8 -2
View File
@@ -1105,10 +1105,16 @@ def test_two_overlapping_reorders_never_share_a_slot(db, as_user):
worker = threading.Thread(target=second_tab)
worker.start()
# Commit A only once B is blocked on a row lock, so B's read is the stale snapshot.
# Commit A only once B is blocked on a row lock, so B's read is the stale snapshot. pg_locks is
# the whole server's: only a wait in this database is B's (a parallel worker's is not).
waiting = 0
for _ in range(200):
waiting = probe.scalar(text("SELECT count(*) FROM pg_locks WHERE NOT granted"))
waiting = probe.scalar(
text(
"SELECT count(*) FROM pg_locks WHERE NOT granted AND pid IN"
" (SELECT pid FROM pg_stat_activity WHERE datname = current_database())"
)
)
probe.rollback()
if waiting:
break
+23 -1
View File
@@ -1,5 +1,6 @@
"""`/api/me` names the live-search source, so the search button's tooltip can say what a search costs
under the source that actually runs (the default website source spends no API quota)."""
under the source that actually runs (the default website source spends no API quota), and says
whether live search and related videos are on at all."""
from app import sysconfig
@@ -24,3 +25,24 @@ def test_me_names_the_running_search_source(client, db, make_user):
sysconfig.set_value(db, "search_source", "api")
db.commit()
assert _me(client)["search_source"] == "api"
def test_me_says_whether_live_search_and_related_videos_are_on(client, db, make_user):
# A daily limit of 0 turns the feature off; the UI leaves it out instead of offering a refusal.
user = make_user(password=_PW)
login = client.post("/auth/password-login", json={"email": user.email, "password": _PW})
assert login.status_code == 200
me = _me(client)
assert (me["search_enabled"], me["related_enabled"]) == (True, True)
sysconfig.set_value(db, "search_daily_limit_per_user", 0)
db.commit()
me = _me(client)
assert (me["search_enabled"], me["related_enabled"]) == (False, True)
sysconfig.set_value(db, "search_daily_limit_per_user", 5)
sysconfig.set_value(db, "related_daily_limit_per_user", 0)
db.commit()
me = _me(client)
assert (me["search_enabled"], me["related_enabled"]) == (True, False)
+50
View File
@@ -0,0 +1,50 @@
"""Migration 0116's data step, DB-lane: an explored channel someone already follows becomes the
catalog's own, and no video stays flagged as an exploration's on a channel that is not one.
The schema half is checked by test_migrations.py; a flag left on is invisible there.
"""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def test_followed_explored_channels_and_their_videos_are_the_catalogs_own(db, make_user):
follower = make_user().id
cfg = Config("alembic.ini")
command.downgrade(cfg, "0115_takeout_channels")
try:
with engine.begin() as conn:
def channel(cid, explored):
conn.execute(
text("INSERT INTO channels (id, from_explore) VALUES (:c, :e)"),
{"c": cid, "e": explored},
)
conn.execute(
text(
"INSERT INTO videos (id, channel_id, published_at, via_explore)"
" VALUES (:v, :c, now(), true)"
),
{"v": f"v-{cid}", "c": cid},
)
channel("UCfollowed", True) # explored, then followed on youtube.com
channel("UCexplored", True) # explored, nobody follows it: stays the exploration's
channel("UCorganic", False) # a video flagged after a subscribe had cleared the channel
conn.execute(
text("INSERT INTO subscriptions (user_id, channel_id) VALUES (:u, 'UCfollowed')"),
{"u": follower},
)
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
channels = dict(conn.execute(text("SELECT id, from_explore FROM channels")).tuples().all())
videos = dict(
conn.execute(text("SELECT channel_id, via_explore FROM videos")).tuples().all()
)
assert channels == {"UCfollowed": False, "UCexplored": True, "UCorganic": False}
assert videos == {"UCfollowed": False, "UCexplored": True, "UCorganic": False}
+3 -3
View File
@@ -75,9 +75,9 @@ def test_trimming_read_notifications_trims_only_mine(db, make_user):
def test_todays_actions_count_only_mine(db, make_user):
me, other = make_user(), make_user()
quota.log_action(me.id, quota.QuotaAction.VIDEOS_SEARCH)
quota.log_action(other.id, quota.QuotaAction.VIDEOS_SEARCH)
quota.log_action(other.id, quota.QuotaAction.VIDEOS_SEARCH)
quota.claim_action(me.id, quota.QuotaAction.VIDEOS_SEARCH, 5)
quota.claim_action(other.id, quota.QuotaAction.VIDEOS_SEARCH, 5)
quota.claim_action(other.id, quota.QuotaAction.VIDEOS_SEARCH, 5)
assert quota.actions_today(db, me.id, quota.QuotaAction.VIDEOS_SEARCH) == 1
+6 -2
View File
@@ -89,9 +89,13 @@ def test_a_claim_is_released_when_its_block_raises(db, make_user):
except RuntimeError:
pass
# Asked of pg_locks, not by claiming again: a lock leaked on a pooled connection would be granted
# again to that same connection (advisory locks are re-entrant per session).
# again to that same connection (advisory locks are re-entrant per session). pg_locks is the whole
# server's, so only this database's: a parallel worker's own claim on the same id counted here.
held = db.scalar(
text("SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND objid = :id"),
text(
"SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND objid = :id"
" AND database = (SELECT oid FROM pg_database WHERE datname = current_database())"
),
{"id": pl.id},
)
assert held == 0
+3 -3
View File
@@ -44,15 +44,15 @@ def test_reads_see_separate_session_spend_mid_transaction(db):
assert after - before == 7
def test_log_action_survives_caller_rollback(db):
def test_a_claimed_slot_survives_caller_rollback(db):
user = User(email="searcher@example.com")
db.add(user)
db.commit() # a committed user to attribute the (free) action to
db.add(User(email="pending@example.com")) # uncommitted caller work
quota.log_action(user.id, quota.QuotaAction.VIDEOS_SEARCH)
assert quota.claim_action(user.id, quota.QuotaAction.VIDEOS_SEARCH, 5) is not None
db.rollback()
# The zero-cost action event survived (per-user daily caps must still count it) ...
# The zero-cost claim event survived (per-user daily caps must still count it) ...
events = db.execute(select(QuotaEvent)).scalars().all()
assert len(events) == 1
assert events[0].user_id == user.id
+428
View File
@@ -0,0 +1,428 @@
"""`GET /api/search/related/{video_id}` — a video's related videos, without what the user has seen.
The scrape is replaced by pages the test hands out, and YouTube's videos.list by an offline enrich
(30-minute videos, so no Shorts probe goes out). What is held: the list leaves out what this user
watched or hid — here or in their Takeout history, unless taken back to "new" since — blocked
channels and the video itself; it pages on until it has enough; it lands in the catalog the way
search finds do; and a failure, or the day's cap, says so instead of answering an empty list."""
import threading
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import func, select, text
from app import quota, sysconfig
from app.catalog.models import BlockedChannel, Channel, Subscription, Video
from app.models import QuotaEvent
from app.reader.models import TakeoutItem, TakeoutKind, VideoState, VideoUnmark
from app.routes import search as search_routes
from app.youtube.client import YouTubeError
from app.youtube.innertube import ScrapeError
T0 = datetime(2026, 10, 1, tzinfo=timezone.utc)
SOURCE = "srcvideo001"
def _vid(n: int) -> str:
return f"rel{n:08d}"
def _item(n: int, channel: str = "UCrelated01") -> dict:
return {
"id": _vid(n),
"channel_id": channel,
"channel_title": f"Chan {channel}",
"title": f"Related {n}",
"published_at": None,
"thumbnail_url": None,
"live_broadcast": "none",
}
class _FakeYT:
def __init__(self, *_a, **_kw):
pass
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def get_channels(self, ids):
return []
def get_videos(self, ids):
return [
{
"id": i,
"snippet": {"title": f"T {i}", "publishedAt": "2025-01-01T00:00:00Z"},
"contentDetails": {"duration": "PT30M"},
"statistics": {},
"status": {},
}
for i in ids
]
@pytest.fixture
def pages(monkeypatch, db):
"""The scrape answers with the pages in `state["pages"]`, one per call, recording each call."""
sysconfig.set_value(db, "shorts_probe_max_seconds", 60)
state: dict = {"pages": [], "calls": []}
def related(video_id, continuation=None, *, hl="en", gl="US"):
state["calls"].append((video_id, continuation, hl, gl))
page = state["pages"].pop(0)
if isinstance(page, Exception):
raise page
return page
monkeypatch.setattr(search_routes.related_scrape, "related", related)
monkeypatch.setattr(search_routes, "YouTubeClient", _FakeYT)
return state
def _page(items: list[dict], token: str | None = "NEXT") -> dict:
return {"items": items, "next_page_token": token}
def _ids(resp) -> list[str]:
assert resp.status_code == 200, resp.text
return [it["id"] for it in resp.json()["items"]]
def _known(db, *ns: int, channel: str = "UCrelated01") -> None:
"""These related videos already in the catalog (a state needs the row)."""
if db.get(Channel, channel) is None:
db.add(Channel(id=channel, title="Chan"))
for n in ns:
db.add(Video(id=_vid(n), channel_id=channel, title=f"Related {n}", published_at=T0))
db.commit()
def test_a_related_list_comes_into_the_catalog_as_search_finds_do(db, make_user, as_user, pages):
user = make_user()
pages["pages"] = [_page([_item(n) for n in range(1, 21)])]
resp = as_user(user).get(f"/api/search/related/{SOURCE}")
assert _ids(resp) == [_vid(n) for n in range(1, 21)]
assert resp.json()["next_cursor"] == "NEXT" and resp.json()["has_more"] is True
video = db.get(Video, _vid(1))
# Flagged like a search find: hidden from the shared Library, reclaimed when nobody keeps it.
assert video.via_search is True and video.enriched_at is not None
assert pages["calls"] == [(SOURCE, None, "en", "US")]
def test_what_the_user_watched_or_hid_here_is_left_out(db, make_user, as_user, pages):
user = make_user()
other = make_user()
_known(db, 1, 2, 3, 4)
db.add_all(
[
VideoState(user_id=user.id, video_id=_vid(1), status="watched", watched_at=T0),
VideoState(user_id=user.id, video_id=_vid(2), status="hidden"),
# In progress is not "seen": it stays, to be finished.
VideoState(user_id=user.id, video_id=_vid(3), status="new", position_seconds=120),
# Someone else's watch is theirs.
VideoState(user_id=other.id, video_id=_vid(4), status="watched", watched_at=T0),
]
)
db.commit()
pages["pages"] = [_page([_item(n) for n in range(1, 6)], token=None)]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [_vid(3), _vid(4), _vid(5)]
def test_what_the_takeout_history_shows_watched_is_left_out_unless_taken_back(
db, make_user, as_user, pages
):
user = make_user()
other = make_user()
_known(db, 2)
db.add_all(
[
# Not in the catalog at all: the history alone names it.
TakeoutItem(user_id=user.id, kind=TakeoutKind.WATCHED, video_id=_vid(1), at=T0),
# Watched on YouTube, then taken back to "new" here: it may come again.
TakeoutItem(user_id=user.id, kind=TakeoutKind.WATCHED, video_id=_vid(2), at=T0),
VideoUnmark(user_id=user.id, video_id=_vid(2), at=T0 + timedelta(days=1)),
# Only in Watch later: not watched.
TakeoutItem(user_id=user.id, kind=TakeoutKind.WATCH_LATER, video_id=_vid(3), at=T0),
# Another user's history.
TakeoutItem(user_id=other.id, kind=TakeoutKind.WATCHED, video_id=_vid(4), at=T0),
]
)
db.commit()
pages["pages"] = [_page([_item(n) for n in range(1, 5)], token=None)]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [_vid(2), _vid(3), _vid(4)]
# Left out before the ingest: what the user has seen is not stored for a list that hides it.
assert db.get(Video, _vid(1)) is None
def test_an_unmark_older_than_the_history_row_does_not_bring_it_back(db, make_user, as_user, pages):
user = make_user()
_known(db, 1)
db.add_all(
[
VideoUnmark(user_id=user.id, video_id=_vid(1), at=T0 - timedelta(days=1)),
TakeoutItem(user_id=user.id, kind=TakeoutKind.WATCHED, video_id=_vid(1), at=T0),
]
)
db.commit()
pages["pages"] = [_page([_item(1), _item(2)], token=None)]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [_vid(2)]
def test_another_users_unmark_does_not_bring_it_back(db, make_user, as_user, pages):
user = make_user()
other = make_user()
_known(db, 1)
db.add_all(
[
TakeoutItem(user_id=user.id, kind=TakeoutKind.WATCHED, video_id=_vid(1), at=T0),
VideoUnmark(user_id=other.id, video_id=_vid(1), at=T0 + timedelta(days=1)),
]
)
db.commit()
pages["pages"] = [_page([_item(1), _item(2)], token=None)]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [_vid(2)]
def test_blocked_channels_and_the_video_itself_are_left_out_and_never_stored(
db, make_user, as_user, pages
):
user = make_user()
db.add(Channel(id="UCblocked01", title="Blocked"))
db.commit()
db.add(BlockedChannel(user_id=user.id, channel_id="UCblocked01"))
db.commit()
own = {**_item(9), "id": SOURCE}
pages["pages"] = [_page([_item(1), _item(2, "UCblocked01"), own], token=None)]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [_vid(1)]
assert db.get(Video, _vid(2)) is None
def test_it_pages_on_until_enough_unseen_videos_are_gathered(db, make_user, as_user, pages):
user = make_user()
_known(db, *range(1, 16))
for n in range(1, 16):
db.add(VideoState(user_id=user.id, video_id=_vid(n), status="watched", watched_at=T0))
db.commit()
pages["pages"] = [
_page([_item(n) for n in range(1, 21)], token="page2"), # 15 seen, 5 new
_page([_item(n) for n in range(21, 41)], token="page3"), # 20 new: 25 now
_page([_item(n) for n in range(41, 61)], token="page4"), # never asked
]
ids = _ids(resp := as_user(user).get(f"/api/search/related/{SOURCE}"))
assert ids == [_vid(n) for n in range(16, 41)]
assert [c[1] for c in pages["calls"]] == [None, "page2"]
assert resp.json()["next_cursor"] == "page3"
def test_a_repeat_on_a_later_page_counts_once(db, make_user, as_user, pages):
user = make_user()
pages["pages"] = [
_page([_item(n) for n in range(1, 11)], token="page2"),
_page([_item(n) for n in range(5, 25)], token=None),
]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [
_vid(n) for n in range(1, 25)
]
def test_it_reads_at_most_four_pages(db, make_user, as_user, pages):
user = make_user()
pages["pages"] = [_page([_item(n)], token=f"page{n + 1}") for n in range(1, 7)]
ids = _ids(resp := as_user(user).get(f"/api/search/related/{SOURCE}"))
assert ids == [_vid(n) for n in range(1, 5)]
assert len(pages["calls"]) == 4 and resp.json()["next_cursor"] == "page5"
def test_the_cursor_and_the_hungarian_locale_reach_the_scrape(db, make_user, as_user, pages):
user = make_user()
pages["pages"] = [_page([_item(1)], token=None)]
resp = as_user(user).get(f"/api/search/related/{SOURCE}?cursor=C1&region=hu")
assert _ids(resp) == [_vid(1)]
assert pages["calls"] == [(SOURCE, "C1", "hu", "HU")]
assert resp.json()["has_more"] is False
def test_an_unknown_region_is_refused(db, make_user, as_user, pages):
user = make_user()
assert as_user(user).get(f"/api/search/related/{SOURCE}?region=xx").status_code == 422
assert pages["calls"] == []
def test_a_failed_scrape_says_youtube_is_unavailable(db, make_user, as_user, pages):
user = make_user()
pages["pages"] = [ScrapeError("blocked")]
resp = as_user(user).get(f"/api/search/related/{SOURCE}")
assert resp.status_code == 422
assert resp.json()["detail"] == {"code": "youtube", "reason": "unavailable"}
def test_a_failure_on_a_later_page_keeps_what_came_before(db, make_user, as_user, pages):
user = make_user()
pages["pages"] = [_page([_item(1), _item(2)], token="page2"), ScrapeError("blocked")]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [_vid(1), _vid(2)]
def test_one_list_is_one_event_against_the_days_cap(db, make_user, as_user, pages):
user = make_user()
sysconfig.set_value(db, "related_daily_limit_per_user", 1)
pages["pages"] = [
_page([_item(1)], token="page2"),
_page([_item(2)], token=None),
]
client = as_user(user)
assert client.get(f"/api/search/related/{SOURCE}").status_code == 200
events = db.scalars(
select(QuotaEvent).where(
QuotaEvent.user_id == user.id, QuotaEvent.action == quota.QuotaAction.VIDEOS_RELATED
)
).all()
assert len(events) == 1
refused = client.get(f"/api/search/related/{SOURCE}")
assert refused.status_code == 429
assert refused.json()["detail"] == {
"code": "refused",
"reason": "related_daily_limit",
"limit": 1,
}
assert len(pages["calls"]) == 2 # the refused one asked YouTube nothing
def test_reading_further_takes_a_slot_too(db, make_user, as_user, pages):
# Each request scrapes and enriches; a cursor is a free string, so one that cost nothing let a
# replayed cursor read without end.
user = make_user()
sysconfig.set_value(db, "related_daily_limit_per_user", 1)
pages["pages"] = [_page([_item(1)], token=None), _page([_item(2)], token=None)]
client = as_user(user)
assert client.get(f"/api/search/related/{SOURCE}").status_code == 200
refused = client.get(f"/api/search/related/{SOURCE}?cursor=more")
assert refused.status_code == 429
assert refused.json()["detail"]["reason"] == "related_daily_limit"
assert len(pages["calls"]) == 1
def test_a_list_that_failed_gives_its_slot_back(db, make_user, as_user, pages):
user = make_user()
sysconfig.set_value(db, "related_daily_limit_per_user", 1)
pages["pages"] = [ScrapeError("blocked"), _page([_item(1)], token=None)]
client = as_user(user)
assert client.get(f"/api/search/related/{SOURCE}").status_code == 422
assert quota.actions_today(db, user.id, quota.QuotaAction.VIDEOS_RELATED) == 0
assert _ids(client.get(f"/api/search/related/{SOURCE}")) == [_vid(1)]
def test_failures_past_the_hours_refunds_keep_their_slot(db, make_user, as_user, pages):
# A cursor is a free string: a request built to fail would otherwise read YouTube without end.
user = make_user()
sysconfig.set_value(db, "related_daily_limit_per_user", 50)
n = search_routes.REFUNDS_PER_HOUR
pages["pages"] = [ScrapeError("bad cursor") for _ in range(n + 1)]
client = as_user(user)
for _ in range(n + 1):
assert client.get(f"/api/search/related/{SOURCE}?cursor=bogus").status_code == 422
assert quota.actions_today(db, user.id, quota.QuotaAction.VIDEOS_RELATED) == 1
def test_a_zero_limit_turns_related_videos_off(db, make_user, as_user, pages):
user = make_user()
sysconfig.set_value(db, "related_daily_limit_per_user", 0)
resp = as_user(user).get(f"/api/search/related/{SOURCE}")
assert resp.status_code == 409
assert resp.json()["detail"] == {"code": "refused", "reason": "related_disabled"}
assert pages["calls"] == []
def test_the_last_slot_is_taken_under_a_lock(db, make_user):
# Checked first and logged after seconds of scraping, requests sent together at the last free
# slot all passed. The claim waits for one already taking a slot for this user and action.
user = make_user()
action = quota.QuotaAction.VIDEOS_RELATED
db.execute(select(func.pg_advisory_xact_lock(func.hashtext(action), user.id)))
got: list[int | None] = []
t = threading.Thread(target=lambda: got.append(quota.claim_action(user.id, action, 1)))
t.start()
t.join(timeout=1)
assert t.is_alive() # it waits for the one holding the lock
# ...and counts only once it has the lock: the one holding it takes the last slot meanwhile.
db.add(QuotaEvent(user_id=user.id, action=action, units=0))
db.commit()
t.join(timeout=10)
assert got == [None]
def _videos_list_fails(monkeypatch, *, on_call: int = 1, reason: str = "quotaExceeded") -> None:
"""YouTube's videos.list refuses from its `on_call`-th call on."""
calls = {"n": 0}
real = _FakeYT.get_videos
def get_videos(self, ids):
calls["n"] += 1
if calls["n"] >= on_call:
raise YouTubeError("videos -> 403", reason=reason, path="videos")
return real(self, ids)
monkeypatch.setattr(_FakeYT, "get_videos", get_videos)
def test_a_lookup_that_fails_says_why_not_no_videos(db, make_user, as_user, pages, monkeypatch):
# Nothing it found can be kept unenriched; an empty list would read as "nothing related".
user = make_user()
_videos_list_fails(monkeypatch)
pages["pages"] = [_page([_item(1)], token=None)]
resp = as_user(user).get(f"/api/search/related/{SOURCE}")
assert resp.status_code == 422
assert resp.json()["detail"] == {"code": "youtube", "reason": "quota"}
assert quota.actions_today(db, user.id, quota.QuotaAction.VIDEOS_RELATED) == 0
def test_a_lookup_failing_on_a_later_page_keeps_what_came_before(
db, make_user, as_user, pages, monkeypatch
):
user = make_user()
_videos_list_fails(monkeypatch, on_call=2)
pages["pages"] = [_page([_item(1)], token="page2"), _page([_item(2)], token=None)]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [_vid(1)]
def test_an_ingest_that_breaks_the_session_is_rolled_back(
db, make_user, as_user, pages, monkeypatch
):
# A database error leaves the session unusable until it is rolled back: the rows of the pages
# before could not be read for the answer.
user = make_user()
real = search_routes.catalog_search.ingest_candidates
calls = {"n": 0}
def ingest(db_, yt, user_, candidates):
calls["n"] += 1
if calls["n"] == 2:
db_.execute(text("SELECT 1/0"))
return real(db_, yt, user_, candidates)
monkeypatch.setattr(search_routes.catalog_search, "ingest_candidates", ingest)
pages["pages"] = [_page([_item(1)], token="page2"), _page([_item(2)], token=None)]
assert _ids(as_user(user).get(f"/api/search/related/{SOURCE}")) == [_vid(1)]
def test_each_row_says_whether_the_user_follows_its_channel(db, make_user, as_user, pages):
user = make_user()
stranger = make_user()
_known(db, 1, channel="UCfollowed1")
_known(db, 2, channel="UCstranger1")
db.add(Subscription(user_id=user.id, channel_id="UCfollowed1"))
db.add(Subscription(user_id=stranger.id, channel_id="UCstranger1"))
db.commit()
pages["pages"] = [
_page([_item(1, "UCfollowed1"), _item(2, "UCstranger1"), _item(3, "UCnobody01")], None)
]
rows = as_user(user).get(f"/api/search/related/{SOURCE}").json()["items"]
assert {r["id"]: r["followed"] for r in rows} == {
_vid(1): True,
_vid(2): False, # another user's follow is not this one's
_vid(3): False,
}
+232
View File
@@ -0,0 +1,232 @@
"""youtube/related_scrape — a video's related videos from the InnerTube `next` endpoint.
The two pages under `fixtures/` are real answers (2026-10-24), trimmed to three items and the
continuation: they hold the `lockupViewModel` shape YouTube draws today. Everything here breaks
silently when YouTube moves — a parser that stops finding videos, a cursor that stops paging — so a
shape it does not know must say so, never read as "nothing related". The network is an httpx
MockTransport."""
import copy
import json
from pathlib import Path
import httpx
import pytest
from app.youtube import innertube, related_scrape
FIXTURES = Path(__file__).parent / "fixtures"
FIRST = json.loads((FIXTURES / "innertube_next_first.json").read_text(encoding="utf-8"))
CONTINUATION = json.loads(
(FIXTURES / "innertube_next_continuation.json").read_text(encoding="utf-8")
)
RESULTS_HTML = (
'<script>{"INNERTUBE_API_KEY":"live-key","INNERTUBE_CLIENT_VERSION":"2.20261001"}</script>'
)
def _results(page: dict) -> list:
return page["contents"]["twoColumnWatchNextResults"]["secondaryResults"]["secondaryResults"][
"results"
]
@pytest.fixture
def net(monkeypatch):
"""Route every httpx.Client the InnerTube module opens through a handler the test sets."""
calls: list[httpx.Request] = []
state: dict = {"handler": None, "calls": calls}
def transport(request: httpx.Request) -> httpx.Response:
calls.append(request)
if request.url.path == "/results":
return httpx.Response(200, text=RESULTS_HTML)
return state["handler"](request)
real = httpx.Client
monkeypatch.setattr(
innertube.httpx,
"Client",
lambda **kw: real(transport=httpx.MockTransport(transport), **kw),
)
monkeypatch.setattr(innertube, "_config", None)
return state
def _answer(net, page, status=200):
net["handler"] = lambda request: (
httpx.Response(status, text=page)
if isinstance(page, str)
else httpx.Response(status, json=page)
)
def test_a_first_page_gives_its_videos_with_their_channel_and_the_next_cursor(net):
_answer(net, FIRST)
page = related_scrape.related("SdTZAMDKrNY")
assert [it["id"] for it in page["items"]] == ["to-lHJfK4pw", "R8V8CbuxryI", "HLKo4fJx_7k"]
first = page["items"][0]
assert first["channel_id"] == "UCtslD4DGH6PKyG_1gFAX7sg"
assert first["channel_title"] == "Alexander Amini"
assert first["title"] == "MIT 6.S191 (2025): Reinforcement Learning"
assert first["thumbnail_url"].startswith("https://i.ytimg.com/vi/to-lHJfK4pw/")
assert first["published_at"] is None and first["live_broadcast"] == "none"
assert page["next_page_token"]
body = json.loads(net["calls"][-1].content)
assert body["videoId"] == "SdTZAMDKrNY" and "continuation" not in body
assert net["calls"][-1].url.path == "/youtubei/v1/next"
def test_the_largest_thumbnail_is_taken(net):
_answer(net, FIRST)
item = related_scrape.related("SdTZAMDKrNY")["items"][0]
sources = _results(FIRST)[0]["lockupViewModel"]["contentImage"]["thumbnailViewModel"]["image"][
"sources"
]
assert item["thumbnail_url"] == sources[-1]["url"]
def test_a_continuation_page_reads_its_videos_and_sends_only_the_cursor(net):
_answer(net, CONTINUATION)
page = related_scrape.related("SdTZAMDKrNY", continuation="TOKEN", hl="hu", gl="HU")
assert len(page["items"]) == 2
assert all(it["channel_id"].startswith("UC") for it in page["items"])
assert page["next_page_token"]
body = json.loads(net["calls"][-1].content)
assert body["continuation"] == "TOKEN" and "videoId" not in body
assert body["context"]["client"]["hl"] == "hu" and body["context"]["client"]["gl"] == "HU"
def test_the_locale_defaults_to_the_topical_one(net):
_answer(net, FIRST)
related_scrape.related("SdTZAMDKrNY")
client = json.loads(net["calls"][-1].content)["context"]["client"]
assert (client["hl"], client["gl"]) == ("en", "US")
def test_a_video_by_several_channels_is_left_out(net):
page = copy.deepcopy(FIRST)
meta = _results(page)[1]["lockupViewModel"]["metadata"]["lockupMetadataViewModel"]
meta["image"] = {"avatarStackViewModel": {"avatars": []}}
_answer(net, page)
ids = [it["id"] for it in related_scrape.related("SdTZAMDKrNY")["items"]]
assert ids == ["to-lHJfK4pw", "HLKo4fJx_7k"]
def test_what_is_not_a_video_is_left_out_and_a_repeat_counts_once(net):
page = copy.deepcopy(FIRST)
results = _results(page)
playlist = copy.deepcopy(results[0])
playlist["lockupViewModel"]["contentType"] = "LOCKUP_CONTENT_TYPE_PLAYLIST"
playlist["lockupViewModel"]["contentId"] = "PLxyz"
results.insert(0, playlist)
results.insert(2, copy.deepcopy(results[1]))
results.insert(0, {"adSlotRenderer": {}})
_answer(net, page)
ids = [it["id"] for it in related_scrape.related("SdTZAMDKrNY")["items"]]
assert ids == ["to-lHJfK4pw", "R8V8CbuxryI", "HLKo4fJx_7k"]
def test_the_previous_compact_renderer_is_still_read(net):
page = copy.deepcopy(FIRST)
_results(page).insert(
0,
{
"compactVideoRenderer": {
"videoId": "oldshape001",
"title": {"simpleText": "Old shape"},
"longBylineText": {
"runs": [
{
"text": "Old Chan",
"navigationEndpoint": {"browseEndpoint": {"browseId": "UColdchan"}},
}
]
},
"thumbnail": {"thumbnails": [{"url": "s.jpg"}, {"url": "b.jpg"}]},
}
},
)
_answer(net, page)
first = related_scrape.related("SdTZAMDKrNY")["items"][0]
assert first == {
"id": "oldshape001",
"channel_id": "UColdchan",
"channel_title": "Old Chan",
"title": "Old shape",
"published_at": None,
"thumbnail_url": "b.jpg",
"live_broadcast": "none",
}
def test_the_last_page_has_no_cursor(net):
page = copy.deepcopy(FIRST)
_results(page).pop() # the continuationItemRenderer
_answer(net, page)
assert related_scrape.related("SdTZAMDKrNY")["next_page_token"] is None
def test_a_video_youtube_no_longer_has_is_an_empty_page(net):
_answer(
net,
{
"contents": {
"twoColumnWatchNextResults": {
"results": {
"results": {
"contents": [
{
"itemSectionRenderer": {
"contents": [{"backgroundPromoRenderer": {}}]
}
}
]
}
}
}
}
},
)
assert related_scrape.related("aaaaaaaaaaa") == {"items": [], "next_page_token": None}
def test_a_watch_frame_with_no_related_column_and_no_word_why_is_an_error(net):
# The frame alone, its related column moved somewhere this does not look: a lost shape, not
# "nothing related".
page = copy.deepcopy(FIRST)
watch = page["contents"]["twoColumnWatchNextResults"]
watch["sideColumn"] = watch.pop("secondaryResults")
_answer(net, page)
with pytest.raises(related_scrape.ScrapeError):
related_scrape.related("SdTZAMDKrNY")
def test_a_related_list_drawn_by_renderers_this_does_not_read_is_an_error(net):
page = copy.deepcopy(FIRST)
results = _results(page)
results[:] = [{"videoTileViewModel": entry} for entry in results]
_answer(net, page)
with pytest.raises(related_scrape.ScrapeError):
related_scrape.related("SdTZAMDKrNY")
@pytest.mark.parametrize(
"answer",
[
{"responseContext": {}}, # no watch page frame at all
{"onResponseReceivedEndpoints": [{"somethingElse": {}}]}, # a continuation we do not know
"<html>consent</html>", # not JSON
[1, 2], # JSON, not an object
],
)
def test_an_answer_of_an_unknown_shape_is_an_error_not_an_empty_list(net, answer):
_answer(net, answer)
with pytest.raises(related_scrape.ScrapeError):
related_scrape.related("SdTZAMDKrNY")
def test_a_refused_request_is_an_error_whatever_its_body_says(net):
_answer(net, FIRST, status=429)
with pytest.raises(related_scrape.ScrapeError):
related_scrape.related("SdTZAMDKrNY")
+17
View File
@@ -291,6 +291,23 @@ def test_search_results_cleared_through_the_route_are_gone(db, make_user, as_use
assert db.execute(select(SearchFind)).first() is None
def test_clearing_leaves_a_video_the_caller_never_found(db, make_user, as_user):
# A related list ingests its rows with no find: they are in someone's list right now, and an id
# any user could name must not delete them from under it.
user = make_user()
_video(db, "s1", via_search=True)
_video(db, "s2", via_search=True)
db.add(SearchFind(user_id=user.id, video_id="s1"))
db.commit()
r = as_user(user).post("/api/search/clear", json={"video_ids": ["s1", "s2"]})
assert r.json() == {"ok": True, "removed": 1}
db.expire_all()
assert db.get(Video, "s1") is None and db.get(Video, "s2") is not None
# Naming only ids it never found removes nothing.
r = as_user(user).post("/api/search/clear", json={"video_ids": ["s2"]})
assert r.json() == {"ok": True, "removed": 0}
# --- Plex show and season marks ---------------------------------------------------------------
+66 -5
View File
@@ -6,6 +6,7 @@ through (at the top of _job): holding _activity_lock across the check-and-set me
caller can hold a job's "running" claim at a time, and it's claimable again once released."""
import logging
import threading
import time
from datetime import datetime, timedelta, timezone
@@ -342,6 +343,29 @@ def test_an_ad_hoc_run_is_the_users_run_of_its_target_with_its_spend_and_failure
assert (failed.status, failed.error) == ("error", "quota exceeded")
def _ad_hoc_row(db, job: str) -> dict:
rows = scheduler.scheduler_snapshot(db)["ad_hoc"]
assert [r["id"] for r in rows] == list(scheduler.AD_HOC_JOBS) # every one has its row
return next(r for r in rows if r["id"] == job)
def test_a_takeout_import_has_its_row_named_by_the_importing_user(db, make_user):
user = make_user("importer@example.com")
user.display_name = "Ida Importer"
db.commit()
assert _ad_hoc_row(db, "takeout_import")["recent"] == []
scheduler._ad_hoc("takeout_import", lambda: {"watched": 2}, user.id, f"user:{user.id}")
row = _ad_hoc_row(db, "takeout_import")
assert (row["running"], row["target"], row["target_title"], row["status"]) == (
False,
f"user:{user.id}",
"Ida Importer",
"ok",
)
assert _ad_hoc_row(db, "channel_backfill")["recent"] == []
def test_the_page_shows_the_ad_hoc_runs_beside_the_jobs_with_their_channel(db, make_user):
user = make_user("shown@example.com")
db.add(Channel(id="UC_shown", title="BBC Earth"))
@@ -349,7 +373,7 @@ def test_the_page_shows_the_ad_hoc_runs_beside_the_jobs_with_their_channel(db, m
scheduler._ad_hoc("channel_backfill", lambda: {"skipped": "at capacity"}, user.id, "UC_none")
scheduler._ad_hoc("channel_backfill", lambda: {"videos_added": 69}, user.id, "UC_shown")
(row,) = scheduler.scheduler_snapshot(db)["ad_hoc"]
row = _ad_hoc_row(db, "channel_backfill")
assert (row["id"], row["running"], row["target"], row["target_title"]) == (
"channel_backfill",
False,
@@ -367,12 +391,12 @@ def test_the_page_shows_the_ad_hoc_runs_beside_the_jobs_with_their_channel(db, m
seen = []
scheduler._ad_hoc(
"channel_backfill",
lambda: seen.append(scheduler.scheduler_snapshot(db)["ad_hoc"][0]["running"]) or {},
lambda: seen.append(_ad_hoc_row(db, "channel_backfill")["running"]) or {},
user.id,
"UC_shown",
)
assert seen == [True]
assert scheduler.scheduler_snapshot(db)["ad_hoc"][0]["running"] is False
assert _ad_hoc_row(db, "channel_backfill")["running"] is False
def _finished(db, job) -> list[JobRun]:
@@ -397,9 +421,46 @@ def test_a_second_ad_hoc_request_before_the_first_starts_is_that_same_run(
"channel_backfill@UC_x",
"channel_backfill@UC_y",
]
# Queued, so its target's newest run row is not this run's yet: a page watching it can tell.
assert scheduler.ad_hoc_queued(scheduler.CHANNEL_BACKFILL, "UC_x")
assert not scheduler.ad_hoc_queued(scheduler.CHANNEL_BACKFILL, "UC_z")
live_scheduler.remove_all_jobs() # never run: a run finishing after the test would land in the next
def test_a_request_made_while_its_run_is_under_way_is_the_run_after_it(
db, make_user, live_scheduler
):
"""The scheduler drops a second fire of an id whose run has not returned: the second Takeout file
of a pair, uploaded while the first one's import ran, was never imported, and its key stayed
queued until a restart (the page spun on "Processing…" for good)."""
user = make_user("again@example.com")
first_runs, release = threading.Event(), threading.Event()
calls = []
def first():
first_runs.set()
assert release.wait(10)
calls.append("first")
return {"n": 1}
def second():
calls.append("second")
return {"n": 2}
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, first, actor_id=user.id, target="UC_again")
assert first_runs.wait(10)
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, second, actor_id=user.id, target="UC_again")
assert scheduler.ad_hoc_queued(scheduler.CHANNEL_BACKFILL, "UC_again")
release.set()
deadline = time.monotonic() + 10
while time.monotonic() < deadline and len(_finished(db, "channel_backfill")) < 2:
time.sleep(0.05)
assert calls == ["first", "second"]
assert sorted(r.result["n"] for r in _finished(db, "channel_backfill")) == [1, 2]
assert not scheduler.ad_hoc_queued(scheduler.CHANNEL_BACKFILL, "UC_again")
def test_a_manual_run_leaves_no_actor_behind_on_its_thread(db, make_user):
"""The scheduler's pool threads are reused: an actor a manual run left set made the next
scheduled run on that thread an admin's "manual" run, with a notice in their inbox."""
@@ -475,7 +536,7 @@ def test_a_run_whose_outcome_cannot_be_written_is_not_called_failed(db, make_use
scheduler._ad_hoc("channel_backfill", lambda: {"videos_added": 1}, user.id, "UC_blip")
assert calls == ["ok"]
# Its row is still `running`, but nothing here is: the page does not call it going.
(row,) = scheduler.scheduler_snapshot(db)["ad_hoc"]
row = _ad_hoc_row(db, "channel_backfill")
assert (row["running"], row["status"], row["recent"][0]["status"]) == (
False,
"interrupted",
@@ -529,7 +590,7 @@ def test_an_ad_hoc_run_still_going_shows_although_a_newer_one_ended(db, make_use
return {}
scheduler._ad_hoc("channel_backfill", long_one, user.id, "UC_long")
(row,) = rows
(row,) = [r for r in rows if r["id"] == "channel_backfill"]
assert (row["running"], row["target"], row["target_title"]) == (True, "UC_long", "Long one")
# The outcome beside it is the long one's (none yet), never the short one's skip.
assert (row["status"], row["result"], row["last_error"]) == (None, None, None)
+361
View File
@@ -0,0 +1,361 @@
"""`GET /api/search/youtube` — the per-user daily cap of the live search, both sources.
The scrape and search.list are replaced by pages the test hands out, and YouTube's videos.list by an
offline enrich. What is held: every request takes a slot, a cursor's too; a search that failed
outright gives its slot back; a zero limit turns live search off, which says so instead of "try
again tomorrow"; a lookup that fails says why instead of answering "no results"; only a few
failures an hour get their slot back; and the API source — 100 units a page — reads one page per
request, is refused before the call when the shared budget cannot pay for it (checked again under
one lock with the call, whose wait is bounded), counts one slot per search though search.list logs
its own event, and keeps the slot of a search search.list was already paid for — but not of one
that never reached YouTube."""
import threading
import pytest
from app import quota, sysconfig
from app.routes import search as search_routes
from app.youtube.client import YouTubeError
from app.youtube.innertube import ScrapeError
ACTION = quota.QuotaAction.VIDEOS_SEARCH
def _item(n: int) -> dict:
return {
"id": f"srch{n:07d}",
"channel_id": "UCsearch001",
"channel_title": "Chan",
"title": f"Found {n}",
"published_at": None,
"thumbnail_url": None,
"live_broadcast": "none",
}
class _FakeYT:
fail_videos = False
def __init__(self, *_a, **_kw):
self.spent = 0
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def get_channels(self, ids):
return []
# search.list: the pages the test hands out, each logged as the real client does (100 units,
# under the action the route attributes it to) — a refusal YouTube answered too, but not a
# failure no endpoint answered (no connection, no token).
api_pages: list = []
api_calls = 0
def search_videos(self, q, page_token=None, order="relevance"):
_FakeYT.api_calls += 1
page = _FakeYT.api_pages.pop(0)
if not (isinstance(page, YouTubeError) and not page.path):
quota.record_usage(100)
self.spent += 100
if isinstance(page, Exception):
raise page
return page
def get_videos(self, ids):
if _FakeYT.fail_videos:
raise YouTubeError("videos -> 403", reason="quotaExceeded", path="videos")
return [
{
"id": i,
"snippet": {"title": f"T {i}", "publishedAt": "2025-01-01T00:00:00Z"},
"contentDetails": {"duration": "PT30M"},
"statistics": {},
"status": {},
}
for i in ids
]
@pytest.fixture
def pages(monkeypatch, db):
"""The scrape answers with the pages in `state["pages"]`, one per call."""
sysconfig.set_value(db, "shorts_probe_max_seconds", 60)
sysconfig.set_value(db, "search_source", "scrape")
state: dict = {"pages": [], "calls": 0}
def search(term, continuation=None):
state["calls"] += 1
page = state["pages"].pop(0)
if isinstance(page, Exception):
raise page
return page
monkeypatch.setattr(search_routes.search_scrape, "search", search)
monkeypatch.setattr(search_routes, "YouTubeClient", _FakeYT)
monkeypatch.setattr(_FakeYT, "fail_videos", False)
monkeypatch.setattr(_FakeYT, "api_pages", [])
monkeypatch.setattr(_FakeYT, "api_calls", 0)
return state
def _page(n: int) -> dict:
return {"items": [_item(n)], "next_page_token": None}
def test_every_request_takes_a_slot_a_cursors_too(db, make_user, as_user, pages):
user = make_user()
sysconfig.set_value(db, "search_daily_limit_per_user", 1)
pages["pages"] = [_page(1), _page(2)]
client = as_user(user)
assert client.get("/api/search/youtube?q=cats").status_code == 200
refused = client.get("/api/search/youtube?q=cats&cursor=more")
assert refused.status_code == 429
assert refused.json()["detail"]["reason"] == "search_daily_limit"
assert pages["calls"] == 1
def test_a_search_that_failed_gives_its_slot_back(db, make_user, as_user, pages):
user = make_user()
sysconfig.set_value(db, "search_daily_limit_per_user", 1)
pages["pages"] = [ScrapeError("blocked"), _page(1)]
client = as_user(user)
assert client.get("/api/search/youtube?q=cats").status_code == 422
assert quota.actions_today(db, user.id, ACTION) == 0
assert client.get("/api/search/youtube?q=cats").status_code == 200
assert quota.actions_today(db, user.id, ACTION) == 1
def test_a_zero_limit_turns_live_search_off(db, make_user, as_user, pages):
user = make_user()
sysconfig.set_value(db, "search_daily_limit_per_user", 0)
resp = as_user(user).get("/api/search/youtube?q=cats")
assert resp.status_code == 409
assert resp.json()["detail"] == {"code": "refused", "reason": "search_disabled"}
assert pages["calls"] == 0
def test_a_zero_limit_turns_the_api_source_off_too(db, make_user, as_user, pages):
user = make_user()
sysconfig.set_value(db, "search_source", "api")
sysconfig.set_value(db, "search_daily_limit_per_user", 0)
resp = as_user(user).get("/api/search/youtube?q=cats")
assert resp.status_code == 409
assert resp.json()["detail"]["reason"] == "search_disabled"
def test_a_lookup_that_fails_says_why_not_no_results(db, make_user, as_user, pages, monkeypatch):
user = make_user()
monkeypatch.setattr(_FakeYT, "fail_videos", True)
pages["pages"] = [_page(1)]
resp = as_user(user).get("/api/search/youtube?q=cats")
assert resp.status_code == 422
assert resp.json()["detail"] == {"code": "youtube", "reason": "quota"}
assert quota.actions_today(db, user.id, ACTION) == 0
def _api(db) -> None:
sysconfig.set_value(db, "search_source", "api")
sysconfig.set_value(db, "quota_daily_budget", 10000)
def test_the_api_source_reads_one_page_per_request(db, make_user, as_user, pages):
# Each page is 100 units: a short page is answered as it is, with its cursor, never paged on.
_api(db)
_FakeYT.api_pages = [{"items": [_item(1)], "next_page_token": "nextpage"}, _page(2)]
resp = as_user(make_user()).get("/api/search/youtube?q=cats&limit=20")
assert resp.status_code == 200
assert _FakeYT.api_calls == 1
assert resp.json()["next_cursor"] == "nextpage"
def test_the_api_source_counts_one_slot_per_search(db, make_user, as_user, pages):
# search.list logs its own event under the action (record_usage): accounting, not a second slot.
user = make_user()
_api(db)
sysconfig.set_value(db, "search_daily_limit_per_user", 2)
_FakeYT.api_pages = [_page(1), _page(2), _page(3)]
client = as_user(user)
assert client.get("/api/search/youtube?q=cats").status_code == 200
assert quota.actions_today(db, user.id, ACTION) == 1
assert client.get("/api/search/youtube?q=dogs").status_code == 200
refused = client.get("/api/search/youtube?q=birds")
assert refused.status_code == 429
assert refused.json()["detail"]["reason"] == "search_daily_limit"
assert _FakeYT.api_calls == 2
def test_the_api_source_takes_its_slot_under_the_lock(db, make_user, as_user, pages, monkeypatch):
user = make_user()
_api(db)
claimed: list = []
real = quota.claim_action
def claim(uid, action, limit):
claimed.append(action)
return real(uid, action, limit)
monkeypatch.setattr(search_routes.quota, "claim_action", claim)
_FakeYT.api_pages = [_page(1)]
assert as_user(user).get("/api/search/youtube?q=cats").status_code == 200
assert claimed == [ACTION]
def test_the_api_source_is_refused_before_the_call_when_the_budget_cannot_pay(
db, make_user, as_user, pages
):
user = make_user()
_api(db)
sysconfig.set_value(db, "quota_daily_budget", 50)
_FakeYT.api_pages = [_page(1)]
resp = as_user(user).get("/api/search/youtube?q=cats")
assert resp.status_code == 429
assert resp.json()["detail"]["reason"] == "quota_spent"
assert _FakeYT.api_calls == 0
assert quota.actions_today(db, user.id, ACTION) == 0
def test_an_api_search_that_search_list_refused_keeps_its_slot(db, make_user, as_user, pages):
# YouTube charges a refused call too (the client records the units first), so a search that
# reached search.list has spent the shared budget: refunding it let a bogus cursor spend it all.
user = make_user()
_api(db)
sysconfig.set_value(db, "search_daily_limit_per_user", 1)
_FakeYT.api_pages = [
YouTubeError("search -> 400", reason="invalidPageToken", path="search"),
_page(1),
]
client = as_user(user)
assert client.get("/api/search/youtube?q=cats&cursor=bogus").status_code == 422
assert quota.actions_today(db, user.id, ACTION) == 1
refused = client.get("/api/search/youtube?q=cats")
assert refused.status_code == 429
assert refused.json()["detail"]["reason"] == "search_daily_limit"
assert _FakeYT.api_calls == 1
def test_an_api_search_the_budget_refused_at_the_call_gives_its_slot_back(
db, make_user, as_user, pages, monkeypatch
):
user = make_user()
_api(db)
answers = iter([True, False]) # the pre-check passes; the budget is gone by the call
monkeypatch.setattr(search_routes.quota, "can_spend", lambda _db, _u: next(answers))
_FakeYT.api_pages = [_page(1)]
resp = as_user(user).get("/api/search/youtube?q=cats")
assert resp.status_code == 429
assert resp.json()["detail"]["reason"] == "quota_spent"
assert _FakeYT.api_calls == 0
assert quota.actions_today(db, user.id, ACTION) == 0
def test_the_budget_check_and_the_call_it_pays_for_hold_one_lock(
db, make_user, as_user, pages, monkeypatch
):
# Checked apart, searches sent together at the last 100 units all passed, and all spent: the
# check and the call sit inside ONE hold of the lock, not two.
user = make_user()
_api(db)
events: list[str] = []
real_check = quota.can_spend
real_call = _FakeYT.search_videos
class _Recorded:
def __init__(self):
self._lock = threading.Lock()
def acquire(self, timeout=-1):
events.append("acquire")
return self._lock.acquire(timeout=timeout)
def release(self):
events.append("release")
self._lock.release()
def can_spend(db_, units):
events.append("check")
return real_check(db_, units)
def search_videos(self, q, page_token=None, order="relevance"):
events.append("call")
return real_call(self, q, page_token=page_token, order=order)
monkeypatch.setattr(search_routes, "_api_search", _Recorded())
monkeypatch.setattr(search_routes.quota, "can_spend", can_spend)
monkeypatch.setattr(_FakeYT, "search_videos", search_videos)
_FakeYT.api_pages = [_page(1)]
assert as_user(user).get("/api/search/youtube?q=cats").status_code == 200
assert events == ["check", "acquire", "check", "call", "release"]
def test_a_search_kept_waiting_for_the_lock_is_refused_as_busy_and_keeps_nothing(
db, make_user, as_user, pages, monkeypatch
):
# Each waiter holds a request worker: a slow YouTube must not park them all behind one another.
user = make_user()
_api(db)
monkeypatch.setattr(search_routes, "API_SEARCH_WAIT_SECONDS", 0.05)
_FakeYT.api_pages = [_page(1)]
# Held by another search for a while: a wait without its bound would get it, and be answered.
assert search_routes._api_search.acquire(timeout=1)
later = threading.Timer(2.0, search_routes._api_search.release)
later.start()
try:
resp = as_user(user).get("/api/search/youtube?q=cats")
finally:
later.join()
assert resp.status_code == 429
assert resp.json()["detail"]["reason"] == "search_busy"
assert _FakeYT.api_calls == 0
assert quota.actions_today(db, user.id, ACTION) == 0
@pytest.mark.parametrize(
"failure",
[
YouTubeError("YouTube API unreachable (ConnectError)", reason="transport"),
YouTubeError("User has no stored OAuth token"),
],
ids=["no-connection", "no-token"],
)
def test_an_api_search_that_never_reached_youtube_gives_its_slot_back(
db, make_user, as_user, pages, failure
):
user = make_user()
_api(db)
sysconfig.set_value(db, "search_daily_limit_per_user", 1)
_FakeYT.api_pages = [failure, _page(1)]
client = as_user(user)
assert client.get("/api/search/youtube?q=cats").status_code == 422
assert quota.actions_today(db, user.id, ACTION) == 0
assert client.get("/api/search/youtube?q=cats").status_code == 200
def test_an_api_search_whose_answer_never_came_keeps_its_slot(db, make_user, as_user, pages):
# The request went out (a read timeout): YouTube may have counted it.
user = make_user()
_api(db)
sysconfig.set_value(db, "search_daily_limit_per_user", 1)
_FakeYT.api_pages = [
YouTubeError("YouTube API unreachable (ReadTimeout)", reason="transport", sent=True),
_page(1),
]
client = as_user(user)
assert client.get("/api/search/youtube?q=cats").status_code == 422
assert quota.actions_today(db, user.id, ACTION) == 1
assert client.get("/api/search/youtube?q=cats").status_code == 429
def test_failures_past_the_hours_refunds_keep_their_slot(db, make_user, as_user, pages):
# A cursor is a free string: a request built to fail would otherwise read YouTube without end.
user = make_user()
sysconfig.set_value(db, "search_daily_limit_per_user", 50)
n = search_routes.REFUNDS_PER_HOUR
pages["pages"] = [ScrapeError("bad cursor") for _ in range(n + 1)]
client = as_user(user)
for _ in range(n + 1):
assert client.get("/api/search/youtube?q=cats&cursor=bogus").status_code == 422
assert quota.actions_today(db, user.id, ACTION) == 1
+4 -4
View File
@@ -9,7 +9,7 @@ import json
import httpx
import pytest
from app.youtube import search_scrape
from app.youtube import innertube, search_scrape
RESULTS_HTML = (
'<script>{"INNERTUBE_API_KEY":"live-key","INNERTUBE_CLIENT_VERSION":"2.20261001"}</script>'
@@ -79,11 +79,11 @@ def net(monkeypatch):
real = httpx.Client
monkeypatch.setattr(
search_scrape.httpx,
innertube.httpx,
"Client",
lambda **kw: real(transport=httpx.MockTransport(transport), **kw),
)
monkeypatch.setattr(search_scrape, "_config", None)
monkeypatch.setattr(innertube, "_config", None)
state["calls"] = calls
return state
@@ -171,7 +171,7 @@ def test_a_failed_key_fetch_falls_back_for_now_and_tries_again_next_time(net, re
with no error anywhere."""
net["handler"] = _serve(results_html=results_html)
search_scrape.search("a")
assert net["calls"][-1].url.params["key"] == search_scrape._FALLBACK_KEY
assert net["calls"][-1].url.params["key"] == innertube._FALLBACK_KEY
net["handler"] = _serve()
search_scrape.search("b")
@@ -13,10 +13,13 @@ a placeholder never becomes the video's permanent date.
from datetime import datetime, timezone
import pytest
from app import sysconfig
from app.models import User
from app.catalog.models import Video
from app.catalog.search import ingest_candidates
from app.youtube.client import YouTubeError
class _FakeYT:
@@ -97,6 +100,25 @@ def test_scrape_ingest_enrich_miss_keeps_nonnull_placeholder(db):
assert c.enriched_at is None
class _FailingYT(_FakeYT):
def get_videos(self, ids):
raise YouTubeError("videos -> 403", reason="quotaExceeded", path="videos")
def test_a_videos_list_failure_raises_and_keeps_the_stubs(db):
# Swallowed, it answered as a page with nothing on it — "no results" for a search whose lookup
# never ran. The stubs stay: the FK targets, backfilled later.
user = User(email="searcher3@example.com")
db.add(user)
db.commit()
with pytest.raises(YouTubeError):
ingest_candidates(db, _FailingYT({}), user, [_candidate("vid_ddddddd", "UC_chan_d")])
db.rollback() # what the caller does: the stubs were committed before the lookup
assert db.get(Video, "vid_ddddddd") is not None
def test_search_youtube_route_returns_422_when_ingest_fails(db, monkeypatch, override_dep):
# Route-level guard (B-core-1): an ingest failure with nothing yet collected must surface as a 422
# + message on this JSON endpoint (the FE hides 5xx behind a generic "connection lost" + auto-retry,
File diff suppressed because it is too large Load Diff
+10
View File
@@ -44,6 +44,9 @@ UNIT_MODELS = {
"PlaylistItem",
"SavedView",
"ComfortPin",
"TakeoutItem",
"TakeoutChannelDismissal",
"VideoUnmark",
"Tag",
"ChannelTag",
},
@@ -360,6 +363,13 @@ UNIT_COMMITS = {
"sync/playlists.py:pull_user_playlists",
"sync/playlists.py:merge_playlist",
"sync/playlists.py:repull_playlist",
# The Takeout import is a recorded run with its own session: the watched part lands before
# the Watch later part asks YouTube (which commits the videos it fetched).
"reader/takeout.py:run_import",
# Run inside the Shorts job, on its session: each user's waiting rows land on their own.
"reader/takeout.py:apply_waiting",
# Inside either of those: the lengths YouTube gave land before the channels are asked for.
"reader/takeout.py:_channel_list",
},
# The catalog's ingest jobs commit per page or batch, a YouTube call between each: a page
# fetched and stored is not lost when the next call fails, and an enrich that fails rolls back
+538
View File
@@ -0,0 +1,538 @@
"""The channels a user's Takeout history shows them watching but not following: which channels make
the list (five videos of five minutes or more, none they follow, block, put out or own), what the
import asks YouTube for it and when, and the channel's watched videos made ready to play."""
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import select, update
from app import quota
from app.catalog import service as catalog_service
from app.catalog.models import BlockedChannel, Channel, ExploredChannel, Subscription, Video
from app.db import SessionLocal
from app.reader import queries, service, takeout
from app.reader.models import TakeoutChannelDismissal, TakeoutItem, TakeoutKind
from app.routes import feed as feed_routes
from app.sync import explore as sync_explore
from app.sync import videos as sync_videos
from app.youtube.client import YouTubeError
T0 = datetime(2026, 5, 1, 12, 0, tzinfo=timezone.utc)
LONG = 600
SHORT = 120
def _ch(n: int) -> str:
return f"UC{n:022d}"
def _vid(channel: int, n: int) -> str:
return f"c{channel:02d}v{n:07d}"
def _watched(db, user, channel: int, lengths: list[int | None], *, at=T0, asked=True):
"""Stage `len(lengths)` watched videos of a channel, the newest first, each with its length
already asked (None: YouTube no longer has it) — or not asked yet."""
service.stage_takeout(
db,
user,
TakeoutKind.WATCHED,
{_vid(channel, i): (at - timedelta(days=i), _ch(channel)) for i in range(len(lengths))},
)
db.flush()
if asked:
ids = [_vid(channel, i) for i in range(len(lengths))]
service.record_takeout_lengths(
db, user, ids, {v: s for v, s in zip(ids, lengths, strict=True) if s is not None}
)
db.commit()
def _channel(db, n: int, title: str | None = None):
db.add(Channel(id=_ch(n), title=title or f"Channel {n}", details_synced_at=T0))
db.commit()
def _listed(db, user) -> list[tuple[str, int]]:
return [(ch.id, n) for ch, n, _ in queries.watched_channels(db, user)]
# --- which channels make the list --------------------------------------------------------------
def test_a_channel_makes_the_list_with_five_watched_videos_of_five_minutes_or_more(db, make_user):
user = make_user("list@example.com")
for n in (1, 2, 3):
_channel(db, n)
_watched(db, user, 1, [LONG] * 5 + [SHORT] * 3) # five long, three short: on it with 5
_watched(db, user, 2, [LONG] * 4 + [SHORT] * 6) # ten watched, only four long: off it
_watched(db, user, 3, [299, 300, 300, 300, 300, 300]) # five at exactly five minutes: on it
# A tie: channel 1's counted videos were watched later (channel 3's newest is under five minutes).
assert _listed(db, user) == [(_ch(1), 5), (_ch(3), 5)]
def test_a_video_youtube_no_longer_has_does_not_count(db, make_user):
user = make_user("gone@example.com")
_channel(db, 1)
_watched(db, user, 1, [LONG] * 4 + [None] * 3)
assert _listed(db, user) == []
def test_the_most_watched_lead_and_a_tie_goes_to_the_one_watched_more_lately(db, make_user):
user = make_user("order@example.com")
for n in (1, 2, 3):
_channel(db, n)
_watched(db, user, 1, [LONG] * 6, at=T0 - timedelta(days=100))
_watched(db, user, 2, [LONG] * 9, at=T0 - timedelta(days=300))
_watched(db, user, 3, [LONG] * 6, at=T0)
assert _listed(db, user) == [(_ch(2), 9), (_ch(3), 6), (_ch(1), 6)]
last = {ch.id: when for ch, _, when in queries.watched_channels(db, user)}
assert last[_ch(3)] == T0 # the last watched of the counted ones
def test_a_channel_the_user_follows_blocked_put_out_or_owns_is_not_on_the_list(db, make_user):
user = make_user("off@example.com", yt_channel_id=_ch(4))
for n in (1, 2, 3, 4, 5):
_channel(db, n)
_watched(db, user, n, [LONG] * 5)
db.add(Subscription(user_id=user.id, channel_id=_ch(1)))
db.add(BlockedChannel(user_id=user.id, channel_id=_ch(2)))
db.commit()
service.dismiss_watched_channel(db, user, _ch(3))
db.commit()
assert _listed(db, user) == [(_ch(5), 5)]
def test_one_users_follows_and_dismissals_leave_anothers_list_alone(db, make_user):
me, other = make_user("me@example.com"), make_user("other@example.com")
_channel(db, 1)
for user in (me, other):
_watched(db, user, 1, [LONG] * 5)
db.add(Subscription(user_id=other.id, channel_id=_ch(1)))
db.commit()
service.dismiss_watched_channel(db, other, _ch(1))
db.commit()
assert _listed(db, me) == [(_ch(1), 5)]
assert _listed(db, other) == []
def test_a_channel_whose_lengths_are_not_all_known_is_pending_until_they_are(db, make_user):
user = make_user("pending@example.com")
_channel(db, 1)
_watched(db, user, 1, [LONG] * 6, asked=False)
_watched(db, user, 2, [LONG] * 4, asked=False) # four watched: it can never make the list
assert queries.watched_channels_pending(db, user) == 1
assert _listed(db, user) == []
ids = [_vid(1, i) for i in range(6)]
service.record_takeout_lengths(db, user, ids, dict.fromkeys(ids, LONG))
db.commit()
assert queries.watched_channels_pending(db, user) == 0
assert _listed(db, user) == [(_ch(1), 6)]
def test_a_new_upload_of_the_history_keeps_the_lengths_already_asked(db, make_user):
user = make_user("again@example.com")
_channel(db, 1)
_watched(db, user, 1, [LONG] * 5)
_watched(db, user, 1, [LONG] * 5, at=T0 + timedelta(days=1), asked=False)
assert _listed(db, user) == [(_ch(1), 5)]
assert queries.takeout_lengths_to_ask(db, user) == []
# --- what the import asks YouTube for ----------------------------------------------------------
class FakeYouTube:
"""Stands in for `YouTubeClient`: answers `videos.list` with the lengths it was given and
`channels.list` with a title per channel, and records what it was asked."""
def __init__(self, lengths: dict[str, int] | None = None, *, fail: bool = False):
self.lengths = lengths or {}
self.fail = fail
self.fail_channels = False
self.videos_asked: list[list[str]] = []
self.channels_asked: list[list[str]] = []
self.actions: list[tuple[int | None, str]] = []
def __call__(self, db, user):
return self
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def get_videos(self, ids):
self.videos_asked.append(list(ids))
self.actions.append((quota._actor_id.get(), quota._action.get()))
if self.fail:
raise YouTubeError("refused", reason="quotaExceeded", path="videos")
return [
{"id": v, "contentDetails": {"duration": f"PT{self.lengths[v]}S"}}
for v in ids
if v in self.lengths
]
def get_channels(self, ids):
self.channels_asked.append(list(ids))
self.actions.append((quota._actor_id.get(), quota._action.get()))
if self.fail_channels:
raise YouTubeError("refused", reason="backendError", path="channels")
return [{"id": c, "snippet": {"title": f"Named {c[-2:]}"}} for c in ids]
@pytest.fixture
def youtube(monkeypatch, db, make_user):
fetcher = make_user("fetcher@example.com")
fake = FakeYouTube()
monkeypatch.setattr(takeout, "YouTubeClient", fake)
monkeypatch.setattr(takeout, "get_service_user", lambda session: fetcher)
monkeypatch.setattr(takeout.quota, "backfill_reserve_reached", lambda: False)
return fake
def test_the_import_asks_the_lengths_the_list_needs_and_names_its_channels(db, make_user, youtube):
user = make_user("import@example.com")
_watched(db, user, 1, [None] * 7, asked=False) # the most watched channel
_watched(db, user, 2, [None] * 5, asked=False)
_watched(db, user, 3, [None] * 4, asked=False) # too few to ever make it: not asked about
youtube.lengths = {_vid(1, i): LONG for i in range(6)} | {_vid(2, i): SHORT for i in range(5)}
result = takeout.run_import(user.id)
asked = [_vid(1, i) for i in range(7)] + [_vid(2, i) for i in range(5)]
assert youtube.videos_asked == [asked] # the top channel first
assert result["lengths_asked"] == 12
# Only channel 1 made it (six long; its seventh is gone, channel 2's are short): it is named.
assert youtube.channels_asked == [[_ch(1)]]
assert result["channels_described"] == 1
assert {
(user.id, quota.QuotaAction.VIDEOS_LOOKUP),
(user.id, quota.QuotaAction.CHANNELS_DISCOVER),
} == set(youtube.actions)
db.expire_all()
assert [(ch.title, n) for ch, n, _ in queries.watched_channels(db, user)] == [("Named 01", 6)]
assert queries.watched_channels_pending(db, user) == 0
# Asked once: a second run asks nothing again, the gone video included.
youtube.videos_asked.clear()
youtube.channels_asked.clear()
assert "lengths_asked" not in takeout.run_import(user.id)
assert youtube.videos_asked == [] and youtube.channels_asked == []
def test_the_lookups_stop_at_the_users_daily_share_and_the_rest_wait(
db, make_user, youtube, monkeypatch
):
user = make_user("share@example.com")
_watched(db, user, 1, [None] * 8, asked=False)
_watched(db, user, 2, [None] * 6, asked=False)
monkeypatch.setattr(takeout, "_lengths", takeout.RateLimiter(10, 3600))
takeout.run_import(user.id)
assert youtube.videos_asked == [[_vid(1, i) for i in range(8)] + [_vid(2, 0), _vid(2, 1)]]
db.expire_all()
assert queries.watched_channels_pending(db, user) == 1 # channel 2 still has some to ask
youtube.videos_asked.clear()
takeout.run_import(user.id)
assert youtube.videos_asked == [] # the share is spent for today
@pytest.mark.parametrize("why", ["no_fetcher", "reserve"])
def test_nothing_is_asked_without_a_connected_account_or_past_the_reserve(
db, make_user, youtube, monkeypatch, why
):
user = make_user(f"{why}@example.com")
_watched(db, user, 1, [None] * 5, asked=False)
if why == "no_fetcher":
monkeypatch.setattr(takeout, "get_service_user", lambda session: None)
else:
monkeypatch.setattr(takeout.quota, "backfill_reserve_reached", lambda: True)
assert "lengths_asked" not in takeout.run_import(user.id)
assert youtube.videos_asked == []
assert queries.watched_channels_pending(db, user) == 1
def test_a_youtube_failure_asking_lengths_fails_neither_the_import_nor_its_marks(
db, make_user, youtube
):
user = make_user("fail@example.com")
_watched(db, user, 1, [None] * 5, asked=False)
youtube.fail = True
result = takeout.run_import(user.id)
assert "lengths_asked" not in result
assert queries.takeout_lengths_to_ask(db, user) == [_vid(1, i) for i in range(5)]
db.expire_all()
rows = db.execute(select(TakeoutItem).where(TakeoutItem.user_id == user.id)).scalars()
assert all(r.looked_up_at is None for r in rows) # left to ask again
def test_the_shorts_job_asks_what_an_import_could_not(db, make_user, youtube):
user = make_user("later@example.com")
_watched(db, user, 1, [None] * 5, asked=False)
youtube.lengths = {_vid(1, i): LONG for i in range(5)}
assert takeout.apply_waiting(db)["takeout_lengths_asked"] == 5
db.expire_all()
assert [n for _, n, _ in queries.watched_channels(db, user)] == [5]
@pytest.mark.parametrize("why", ["follows", "blocked", "dismissed", "own"])
def test_a_channel_off_the_list_is_not_asked_about_nor_looked_at_again(
db, make_user, youtube, monkeypatch, why
):
user = make_user(f"off-{why}@example.com", yt_channel_id=_ch(1) if why == "own" else None)
_channel(db, 1)
_watched(db, user, 1, [None] * 5, asked=False)
if why == "follows":
db.add(Subscription(user_id=user.id, channel_id=_ch(1)))
elif why == "blocked":
db.add(BlockedChannel(user_id=user.id, channel_id=_ch(1)))
elif why == "dismissed":
service.dismiss_watched_channel(db, user, _ch(1))
db.commit()
runs = []
monkeypatch.setattr(takeout, "_channel_list", lambda session, u: runs.append(u.id) or {})
assert takeout.apply_waiting(db) == {}
# Its rows are never asked about, so they would stay "to ask" for good: the Shorts job must not
# take the user up again for them on every run.
assert runs == []
assert youtube.videos_asked == []
def test_naming_the_channels_is_tried_again_and_meanwhile_the_list_says_it_is_not_complete(
db, make_user, youtube
):
user = make_user("unnamed@example.com")
_watched(db, user, 1, [None] * 5, asked=False)
youtube.lengths = {_vid(1, i): LONG for i in range(5)}
youtube.fail_channels = True
takeout.run_import(user.id)
db.expire_all()
# Every length is known, the channel could not be named: not on the list, and not complete.
assert queries.takeout_lengths_to_ask(db, user) == []
assert _listed(db, user) == []
assert queries.watched_channels_pending(db, user) == 1
youtube.fail_channels = False
takeout.apply_waiting(db)
db.expire_all()
assert [(ch.title, n) for ch, n, _ in queries.watched_channels(db, user)] == [("Named 01", 5)]
assert queries.watched_channels_pending(db, user) == 0
def test_a_listed_channel_still_to_be_described_is_not_counted_as_one_to_come(db, make_user):
user = make_user("stub@example.com")
# A stub the catalog made when one of its videos came in: a title, no details yet.
db.add(Channel(id=_ch(1), title="From a video"))
db.commit()
_watched(db, user, 1, [LONG] * 5)
assert queries.watched_channels_to_describe(db, user) == [_ch(1)]
assert _listed(db, user) == [(_ch(1), 5)]
# It is on the list already: "the list may grow" would promise one more that cannot come.
assert queries.watched_channels_pending(db, user) == 0
def test_the_lengths_leave_the_watch_later_lookups_their_own_share(
db, make_user, youtube, monkeypatch
):
user = make_user("shares@example.com")
monkeypatch.setattr(takeout, "_lookups", takeout.RateLimiter(10, 3600))
monkeypatch.setattr(takeout, "_lengths", takeout.RateLimiter(10, 3600))
_watched(db, user, 1, [None] * 15, asked=False)
takeout.run_import(user.id) # the history: its lengths spend all they may today
youtube.videos_asked.clear()
service.stage_takeout(db, user, TakeoutKind.WATCH_LATER, {"later000001": (T0, None)})
db.commit()
takeout.run_import(user.id) # then the export's Watch later file
assert ["later000001"] in youtube.videos_asked
# --- the channel's watched videos, ready to play ------------------------------------------------
def _fake_ensure(calls):
def ensure(session, yt, ids, *, explore=False):
calls.append((list(ids), explore))
for vid in ids:
if vid.endswith("4"):
continue # gone from YouTube since its length was asked
session.add(Video(id=vid, channel_id=_ch(1), published_at=T0, via_explore=explore))
session.commit()
return ensure
def test_a_channels_watched_videos_come_ready_to_play_and_explored(
db, make_user, as_user, monkeypatch
):
user = make_user("play@example.com")
_channel(db, 1)
_watched(db, user, 1, [LONG, SHORT, LONG, None, LONG, LONG, LONG])
db.add(Video(id=_vid(1, 0), channel_id=_ch(1), published_at=T0)) # already in the catalog
db.commit()
calls = []
monkeypatch.setattr(feed_routes, "ensure_videos", _fake_ensure(calls))
monkeypatch.setattr(feed_routes.quota, "backfill_reserve_reached", lambda: False)
resp = as_user(user).post(f"/api/takeout/channels/{_ch(1)}/videos")
assert resp.status_code == 200, resp.text
body = resp.json()
# Only the counted ones, newest first; the short one and the gone ones are left out.
assert calls == [([_vid(1, 2), _vid(1, 4), _vid(1, 5), _vid(1, 6)], True)]
assert [it["id"] for it in body["items"]] == [_vid(1, 0), _vid(1, 2), _vid(1, 5), _vid(1, 6)]
assert body["items"][0]["takeout_watched_at"].startswith("2026-05-01")
assert body["gone"] == 2 # one YouTube had lost before, one since
assert body["ok"] is True
db.expire_all()
assert db.execute(
select(ExploredChannel).where(
ExploredChannel.user_id == user.id, ExploredChannel.channel_id == _ch(1)
)
).scalar_one()
assert (
db.get(Channel, _ch(1)).from_explore is True
) # nobody follows it: the cleanup's to reclaim
def test_a_followed_channels_watched_videos_come_in_as_the_catalogs_own(
db, make_user, as_user, monkeypatch
):
user, follower = make_user("viewer@example.com"), make_user("follower@example.com")
_channel(db, 1)
_watched(db, user, 1, [LONG] * 5)
db.add(Subscription(user_id=follower.id, channel_id=_ch(1)))
db.commit()
calls = []
monkeypatch.setattr(feed_routes, "ensure_videos", _fake_ensure(calls))
monkeypatch.setattr(feed_routes.quota, "backfill_reserve_reached", lambda: False)
assert as_user(user).post(f"/api/takeout/channels/{_ch(1)}/videos").status_code == 200
# Someone follows it, so it is no exploration's to reclaim: a video flagged as one would never
# be cleared again (subscribing clears only an explored channel's) and never marked watched.
assert calls == [([_vid(1, i) for i in range(5)], False)]
db.expire_all()
assert db.get(Channel, _ch(1)).from_explore is False
def test_a_follow_landing_while_the_videos_come_in_leaves_none_of_them_flagged(
db, make_user, as_user, monkeypatch
):
user, follower = make_user("racer@example.com"), make_user("late@example.com")
_channel(db, 1)
_watched(db, user, 1, [LONG] * 5)
fake = _fake_ensure(calls := [])
def ensure(session, yt, ids, *, explore=False):
# Someone subscribes, in a request of their own, while YouTube answers this one.
with SessionLocal() as other:
ch = other.get(Channel, _ch(1))
catalog_service.record_subscription(other, follower, ch, "sub-late")
other.commit()
fake(session, yt, ids, explore=explore)
monkeypatch.setattr(feed_routes, "ensure_videos", ensure)
monkeypatch.setattr(feed_routes.quota, "backfill_reserve_reached", lambda: False)
assert as_user(user).post(f"/api/takeout/channels/{_ch(1)}/videos").status_code == 200
assert calls[0][1] is True # asked as the exploration's
db.expire_all()
flagged = db.scalars(select(Video.id).where(Video.via_explore.is_(True))).all()
assert (db.get(Channel, _ch(1)).from_explore, flagged) == (False, [])
def test_the_cleanup_keeps_a_channel_a_watch_history_lists_and_reclaims_its_videos(db, make_user):
user = make_user("kept@example.com")
_channel(db, 1)
_watched(db, user, 1, [LONG] * 5)
catalog_service.explore(db, user, db.get(Channel, _ch(1)))
db.add(Video(id=_vid(1, 0), channel_id=_ch(1), published_at=T0, via_explore=True))
db.add(Channel(id=_ch(2), from_explore=True)) # explored, in nobody's history
db.commit()
db.execute(update(ExploredChannel).values(created_at=T0 - timedelta(days=365)))
db.commit()
out = sync_explore.purge_unkept_explored(db)
# The unplayed video goes (opening the list's videos brings it back); the channel stays listed.
assert (out["videos_deleted"], out["channels_deleted"]) == (1, 1)
db.expire_all()
assert db.get(Channel, _ch(2)) is None
assert _listed(db, user) == [(_ch(1), 5)]
def test_a_blocked_or_unknown_channels_videos_are_refused(db, make_user, as_user):
user = make_user("refused@example.com")
_channel(db, 1)
db.add(BlockedChannel(user_id=user.id, channel_id=_ch(1)))
db.commit()
client = as_user(user)
assert client.post(f"/api/takeout/channels/{_ch(1)}/videos").status_code == 403
assert client.post(f"/api/takeout/channels/{_ch(9)}/videos").status_code == 404
def test_ensure_videos_adds_an_explorations_videos_as_its_own(db):
db.add(Channel(id=_ch(1), title="C"))
db.commit()
class YT:
def get_videos(self, ids):
return [{"id": v, "snippet": {"channelId": _ch(1)}} for v in ids]
sync_videos.ensure_videos(db, YT(), ["exploredvid"], explore=True)
sync_videos.ensure_videos(db, YT(), ["organicvid0"])
db.expire_all()
assert db.get(Video, "exploredvid").via_explore is True
assert db.get(Video, "organicvid0").via_explore is False
# --- the routes -----------------------------------------------------------------------------------
def test_the_list_route_says_what_it_holds_and_whether_a_history_was_uploaded(
db, make_user, as_user
):
user = make_user("route@example.com")
client = as_user(user)
assert client.get("/api/takeout/channels").json() == {
"imported": False,
"pending": 0,
"channels": [],
}
_channel(db, 1, "Some channel")
_watched(db, user, 1, [LONG] * 5)
_watched(db, user, 2, [None] * 5, asked=False)
body = client.get("/api/takeout/channels").json()
assert body["imported"] is True and body["pending"] == 1
assert [(c["id"], c["title"], c["watched"]) for c in body["channels"]] == [
(_ch(1), "Some channel", 5)
]
assert body["channels"][0]["last_watched_at"].startswith("2026-05-01")
def test_a_channel_put_out_of_the_list_comes_back_when_restored(db, make_user, as_user):
user = make_user("dismiss@example.com")
_channel(db, 1)
_watched(db, user, 1, [LONG] * 5)
client = as_user(user)
url = f"/api/takeout/channels/{_ch(1)}/dismissal"
assert client.put(url).json() == {"ok": True, "channel_id": _ch(1)}
assert client.put(url).status_code == 200 # again: no error
assert client.get("/api/takeout/channels").json()["channels"] == []
assert client.delete(url).json() == {"ok": True, "channel_id": _ch(1)}
assert [c["id"] for c in client.get("/api/takeout/channels").json()["channels"]] == [_ch(1)]
db.expire_all()
assert db.execute(select(TakeoutChannelDismissal)).scalars().all() == []
def test_only_a_channel_of_the_users_own_history_can_be_put_out_of_the_list(db, make_user, as_user):
user, other = make_user("mine@example.com"), make_user("theirs@example.com")
_channel(db, 1)
_watched(db, other, 1, [LONG] * 5)
client = as_user(user)
assert client.put(f"/api/takeout/channels/{_ch(1)}/dismissal").status_code == 404
assert client.put(f"/api/takeout/channels/{_ch(9)}/dismissal").status_code == 404
db.expire_all()
assert db.execute(select(TakeoutChannelDismissal)).scalars().all() == []
def test_a_dismissal_of_an_overlong_channel_id_is_refused_before_the_database(
db, make_user, as_user
):
user = make_user("long@example.com")
assert as_user(user).put(f"/api/takeout/channels/{'U' * 33}/dismissal").status_code == 422
+42
View File
@@ -193,3 +193,45 @@ def test_any_other_token_refusal_keeps_the_grant(monkeypatch):
c._access_token()
assert tok.scopes == "s"
assert refusal_reason(caught.value) == "unavailable"
@pytest.mark.parametrize(
("failure", "sent"),
[
(httpx.ConnectError("refused"), False),
(httpx.ConnectTimeout("slow connect"), False),
(httpx.ProxyError("proxy down"), False),
(httpx.ReadTimeout("no answer"), True),
(httpx.RemoteProtocolError("dropped"), True),
],
)
def test_a_transport_failure_says_whether_the_request_went_out(failure, sent):
"""A search that never reached YouTube spent nothing and gets its slot back; one whose answer
never came may have been counted, and keeps it (`routes/search`)."""
def request(*_a, **_k):
raise failure
c = yt.YouTubeClient.__new__(yt.YouTubeClient)
c._http = SimpleNamespace(request=request)
with pytest.raises(YouTubeError) as caught:
c._send("GET", "https://example.invalid")
assert caught.value.reason == "transport"
assert caught.value.sent is sent
def test_a_call_youtube_answered_is_spent_a_refused_one_too(monkeypatch):
resp = httpx.Response(
400,
json={"error": {"errors": [{"reason": "invalidPageToken"}]}},
request=httpx.Request("GET", f"{yt.API_BASE}/search"),
)
recorded: list[int] = []
c = yt.YouTubeClient.__new__(yt.YouTubeClient)
c.db, c.spent = None, 0
monkeypatch.setattr(yt.sysconfig, "get_str", lambda d, key: "a-key")
monkeypatch.setattr(yt.quota, "record_usage", recorded.append)
monkeypatch.setattr(c, "_send", lambda *a, **k: resp)
with pytest.raises(YouTubeError):
c._get("search", {}, cost=100)
assert (recorded, c.spent) == ([100], 100)
+34 -5
View File
@@ -225,7 +225,10 @@ Also in force:
tag chip is `Chip`; a sortable row's grip is `DragHandle`. On the always-dark player chrome both
`Button` and `IconButton` use the `overlay` variant (white, a white layer), whatever the theme; the Plex
control row is IconButton's `player` rung (36, a 20px glyph, round, 44 on touch). A pressed toggle is
`pressed`: the solid accent on a `Button`, the accent glyph on an `IconButton`. A raw `<button>` (or a bare
`pressed`: the solid accent on a `Button`, the accent glyph on an `IconButton`, kept under the pointer
(no hover layer or hover colour over it). A mode cycle (the player's auto-advance and loop) is no
two-state switch: `lit` gives it the same look when it is not off, without `aria-pressed`, and its name
says the mode. A raw `<button>` (or a bare
`FocusSafeButton`) outside the control primitives (eslint `BUTTON_PRIMITIVES`) is a lint error. A text-less button must have an accessible name, and the type enforces
it. [WCAG 4.1.2]
- **The chip family** (`Chip`, E28 S3b): rounded-full, never under 24 (A3 grows it to 44 on touch).
@@ -251,8 +254,13 @@ Also in force:
- A button that starts slow work shows `loading` (a spinner, `aria-busy`) and can't be pressed twice.
[Nielsen #1]
- **A glyph means one thing.** Play (▷) starts playback, or resumes a paused transfer; a retry is the
refresh arrow (`RefreshCw`); a reset or revert is the counter-clockwise arrow (`RotateCcw`). A failed
download's Retry had worn the Play glyph, which promised a video. [Nielsen #4]
refresh arrow (`RefreshCw`), and so is doing a download again; a reset or revert is the
counter-clockwise arrow (`RotateCcw`), and so is playing from the beginning, on the card and in its
menu alike; clearing a video's progress and status is the eraser (`Eraser`). A failed download's Retry
had worn the Play glyph, which promised a video. The skip arrows (⏮ ⏭) are Previous /
Next; the auto-advance mode beside them wears list glyphs (`ListEnd` next, `ListStart` previous,
`Shuffle` random; off is `ListEnd`, not lit) — a mode in the skip arrows put two identical ⏭ on the
panel's glyph-only row. [Nielsen #4]
## Surfaces and page chrome — _Decided_ (batch B)
@@ -498,7 +506,10 @@ of a full-width panel and nothing on the right.
## Interaction rules — _Decided_
- **Keyboard:** everything works by keyboard, in visual order, with no traps. A shortcut is a row in
`lib/keymap.ts`; list and grid navigation is `NavGrid`/`keyboardNav`. [WCAG 2.1.1, 2.4.3, 2.1.2]
`lib/keymap.ts`; list and grid navigation is `NavGrid`/`keyboardNav`. An open dialog or player
silences the page's list keys, but a list inside the topmost one walks while the focus is in it
(the player's related videos); the walk stays inside that layer, and its arrows there move the list,
not the video. [WCAG 2.1.1, 2.4.3, 2.1.2]
- **Focus:** every control has a visible focus ring (the base ring in `index.css`; don't add a second). A
sticky or fixed layer never covers the focused element. A list item the keyboard lands on rings 2 px
outside itself where it has a gap round it (a card); a table row, a log line or a row flush in a
@@ -556,7 +567,8 @@ of a full-width panel and nothing on the right.
row is not a card and keeps its one play button: started, that button continues, its "…" has Continue +
"Play from the beginning", and its picture carries the cards' progress bar.
- **The YouTube modal:** its chrome uses the `Button` rungs (44 on touch); Previous / Next live in the panel
(its transport row, or the shut panel's strip) and on Shift+← / →, with no strips beside the card. Close is
(its transport row: glyphs only — Previous, the count, Next, auto-advance, loop on one line, each named in
its tooltip, the modes with their current one — or the shut panel's strip) and on Shift+← / →, with no strips beside the card. Close is
an icon at the top right of the panel (wide: its head, or its strip) and at the end of the title row
under the video (stacked). Wide, the panel shows the description outright, clamped to three
lines with a More (`ClampedText`; a Tab onto a link below the cut opens it), and collapses to a 2.75rem
@@ -564,6 +576,18 @@ of a full-width panel and nothing on the right.
open. Stacked, the description opens from an ⓘ, because a hover popover never opens on a tap (measured
live). The wheel: over the video it sets the volume; over a panel with something to scroll it scrolls
only, never the volume, even at its end; over one with nothing to scroll it sets the volume. [WCAG 1.4.13]
- **Related videos (owner's picks A, A, a, a):** on demand, never open by itself — the modal has no room to
spare. Wide, a "Related" tab beside the description in the side panel; stacked, a button beside the ⓘ
opens them in a drawer over the player. A row is the thumbnailed video row in its narrow form (a 6rem
picture, the relative time only, the "…" its one control — every toggle is a row of that menu), with
a check before the channel the user follows (named on hover). A click plays it in the same player
(Back returns to the opened video), and the list stays the opened video's. Played, it is the item it
now is (owner's pick D1 a): its title, its own watched / playlist / download controls, its progress
kept, marked watched at its end — and its end does not step the queue. What the user watched or hid
is never in it; a list read so far with nothing unseen says so above Show more. Over it, a two-way
switch: on topic or trending (the Hungarian chart, labelled just "Trending"), kept on the account. Turned off on the instance (a daily
limit of 0, owner's pick D2 a), there is no tab and no button; a refusal a retry cannot answer (the
day's cap, turned off, the shared budget) offers no Retry.
- **Clip mode (both players, owner's picks A + P1 + a):** a scissors in the control row (Plex: the C5 row,
before the ⋯; YouTube: the actions beside Download, and the docked bar) or `C` opens a clip panel IN the
player; `[` / `]` mark the start / end at the playhead and open it when shut. The video being watched is the
@@ -795,3 +819,8 @@ of a full-width panel and nothing on the right.
| 2026-10-17 | Text on glass (owner ★) | Light `--muted` 15% toward `--fg` in every scheme (≥ 6:1 on the bare page); the item count, the badges and the current Files row on opaque tints mixed into the surface | Only the two failing components; the item count's number in `--fg` | Pixel-measured over every scheme × mode × backdrop, the count fell to 3.8:1 (dark too: thumbnails scroll under the toolbar's glass) and Channels' tabs to 4.1:1; an `--fg` count outshouts the filters beside it |
| 2026-10-18 | Scheduler run history (owner D4, E1, E2 ★) | A row's last three runs as dots, newest first (interrupted: a hollow warn ring, set apart from skipped and failed), and a history button that opens the job's runs on the Activity page; a user's ad-hoc runs in their own block under the jobs, with no interval and no run button | An inline expandable history per row; a dark-grey interrupted dot; ad-hoc runs only on the Activity page | Three dots answer "did it run, did it work" at a glance; the whole history needs filters, which the Activity page has |
| 2026-10-18 | Activity page (owner D1, D2, G1–G3 ★) | One page, "Napló" / "Activity": four tabs (Runs, Admin actions, Server log, Quota) under one filter bar (text, a period of 1 h · 24 h · 7 d · 30 d · 90 d defaulting to 7 days, an account), the view in the URL; runs and admin actions as dense rows that open in place as a log line does, filtered and paged on the server; a run's or a request's log lines one pivot away, as a history entry | One merged timeline; a page per source; a default period per tab; card rows; the audit log's client-side table | Each source keeps its own table and retention, read in one place; one bar keeps what you narrowed across tabs; about 1.8k runs a day need a log's density and a server-side filter |
| 2026-10-24 | Related videos (owner U1–U3 ★, D1 A, D3) | On demand: a tab beside the description (wide), a drawer (stacked); the narrow thumbnailed row, its menu alone, a followed check; plays in place and the list stays the opened video's; an on-topic / Hungarian-trending switch kept on the account | Always shown; the end-of-video grid alone; a dialog; a big card or a text row; a list that follows the playing video | The modal is full on a small screen; a picture is what a choice among strangers rests on; a list that stays put is tried one by one, and spends no more of the day's cap |
| 2026-10-25 | Related: one played; off (owner D1 a, D2 a) | Played from the list, it is the player's item: its title, controls and progress, marked watched at its end, the queue left where it was; a daily limit of 0 takes the tab and the button away | A linked video's in-place play (no controls, nothing kept); a tab that only shows a refusal | What is watched should count as watched, wherever it was started from; a control that can only be refused is not offered |
| 2026-10-25 | Player panel glyphs (owner 1A, 2A, 3) | In the side panel the queue row is glyph-only on one line, and Mark watched is a glyph toggle among the actions (✓ / ✓✓); the docked bar keeps its labels; the related switch at the default size, "Trending" | Labelled buttons wrapping onto three lines; glyphs on the docked bar too; Mark watched on its own line | The panel is narrow and the list controls are the same few every time; each glyph keeps its full name and shortcut in its tooltip; the docked bar has the width for words |
| 2026-10-25 | Auto-advance glyphs (owner E A, H a) | Auto-advance wears list glyphs (`ListEnd` next and off, `ListStart` previous, `Shuffle` random), in the panel and the docked bar; the auto-advance and loop cycles are `lit`, not `pressed` | The skip arrows for the mode; an X'd list for off; `aria-pressed` on a cycle | Beside Previous / Next the skip arrows read as a second Next; a cycle of three or four modes is no two-state switch, and its name already says which |
| 2026-10-25 | Restart and Reset glyphs (owner L A) | A menu's Play from the beginning wears `RotateCcw`, as the card's own Restart does; Reset (clear progress and status) wears `Eraser`; Redownload wears `RefreshCw`, the retry arrow | The skip arrow ⏮ on the menu's Restart; `RotateCcw` on both Restart and Reset; `TimerReset` for Restart | ⏮ is Previous beside the player; one action wore two glyphs, and two actions in one menu would have worn one |
+4 -2
View File
@@ -1,9 +1,9 @@
import { test as setup, expect, type Page } from "@playwright/test";
import { execFileSync } from "node:child_process";
import { mkdirSync, readFileSync } from "node:fs";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { MEMBER_STORAGE } from "./helpers";
import { MEMBER_STORAGE, SEED_FILE } from "./helpers";
import { retryConnect } from "./retryConnect";
/**
@@ -99,6 +99,8 @@ setup("seed + authenticate", async ({ page, browser }) => {
{ input: readFileSync(SEED_SCRIPT) },
);
console.log(`[e2e seed] ${out.toString().trim()}`);
// What the seed chose, for the specs that need one of its videos (helpers.seeded).
writeFileSync(SEED_FILE, out);
} catch (err: unknown) {
const e = err as { stderr?: Buffer; message?: string };
throw new Error(`E2E seed failed: ${e.stderr?.toString() || e.message}`);
+10
View File
@@ -1,8 +1,18 @@
import { readFileSync } from "node:fs";
import { expect, type Page } from "@playwright/test";
/** The plain member's saved session (global.setup.ts signs it in); `member.spec.ts` and
* `devices.spec.ts` run from it. */
export const MEMBER_STORAGE = "e2e/.auth/member.json";
// The seed's own answer (scripts/e2e_seed.py), written by global.setup: the videos it chose.
export const SEED_FILE = "e2e/.auth/seed.json";
/** The videos the seed chose for the main account under `key` in its answer. */
export function seeded(key: string): string[] {
const ids = (JSON.parse(readFileSync(SEED_FILE, "utf-8")) as Record<string, unknown>)[key];
if (!Array.isArray(ids) || !ids.length) throw new Error(`the seed chose no ${key} videos`);
return ids as string[];
}
/** Open the app and wait for the authenticated shell (the nav rail) to be ready. */
export async function openApp(page: Page, path = "/"): Promise<void> {
+59
View File
@@ -0,0 +1,59 @@
import { test, expect } from "@playwright/test";
import { openApp, gotoPage, seeded } from "./helpers";
// Settings › Account › YouTube history: a Google Takeout watch history, uploaded through the real
// picker, is read, imported in the background, and the section says what it did — a video the
// catalog knows is marked watched, one it does not know is remembered for later. The seed clears
// the account's kept Takeout videos each run, but the projects share one account: a later one's
// import of the same video adds nothing new, so what each adds is counted against what was there
// before it.
test("a Takeout watch history uploads, imports, and the section says what it did", async ({
page,
}) => {
await openApp(page);
await gotoPage(page, "settings");
await page.getByRole("tab", { name: "Account" }).click();
await expect(page.getByRole("heading", { name: "YouTube history" })).toBeVisible();
// The first project's upload is an import already made: its "New import" opens the drop zone.
const zone = page.getByRole("button", { name: "Drop the files here, or choose them" });
const again = page.getByRole("button", { name: "New import" });
await expect(zone.or(again)).toBeVisible();
if (await again.isVisible()) await again.click();
await expect(zone).toBeVisible();
const before = (await (await page.request.get("/api/takeout")).json()) as {
watched?: Record<string, number>;
};
const watchedBefore = before.watched?.watched ?? 0;
// A video the catalog knows and the Shorts probe has cleared, chosen by the seed (one not probed
// yet would wait for the probe): the next one no earlier project of this run has imported.
const known = seeded("takeout_known")[watchedBefore];
expect(known, "the seed chose too few videos for every project").toBeTruthy();
// Watched now: later than any state the seed set, which the import would otherwise leave alone.
const now = new Date().toISOString();
const history = [
{ titleUrl: `https://www.youtube.com/watch?v=${known}`, time: now },
{ titleUrl: "https://www.youtube.com/watch?v=e2eUnknown0", time: now },
{
titleUrl: "https://www.youtube.com/watch?v=e2eAdvert00",
time: now,
details: [{ name: "From Google Ads" }],
},
];
const uploaded = page.waitForResponse(
(r) => r.url().endsWith("/api/takeout") && r.request().method() === "POST",
);
await page.locator('input[type="file"][accept=".json,.csv"]').setInputFiles({
name: "watch-history.json",
mimeType: "application/json",
buffer: Buffer.from(JSON.stringify(history)),
});
expect((await uploaded).ok()).toBeTruthy();
await expect(page.getByText("watch-history.json: 2 videos read")).toBeVisible();
const line = page.getByTestId("takeout-last");
await expect(line).toContainText("Last import:");
await expect(line).toContainText(`${watchedBefore + 1} watched`);
await expect(line).toContainText("1 remembered");
await expect(page.getByRole("button", { name: "New import" })).toBeVisible();
});
@@ -34,6 +34,7 @@ import AddToPlaylist, { AddToPlaylistAt } from "./features/video/AddToPlaylist";
import PlexPlaylistAdd from "./features/plexCards/PlexPlaylistAdd";
import TagManager from "./features/channel/TagManager";
import { api, HttpError, type PlexCard } from "./lib/api";
import { qk } from "./lib/queryKeys";
beforeAll(() => {
class Stub {
@@ -157,6 +158,52 @@ describe("the add-to-playlist popover", () => {
await waitFor(() => expect(lists.mock.calls.length).toBeGreaterThan(1));
});
// The built-in Watch later list's items ARE the bookmark: ticked here, the video is saved wherever
// it is shown (a related list, a live search), and a user list's tick marks nothing.
it("marks the video saved when Watch later is ticked, and unsaved when it is unticked", async () => {
// The server's side: the lists the video is on, so the refetch after a write agrees with it.
const on = new Set<number>();
vi.spyOn(api, "playlists").mockImplementation(
async () =>
[
{ id: 9, name: "Watch later", kind: "watch_later", has_video: on.has(9) },
{ id: 3, name: "Mix", kind: "user", has_video: on.has(3) },
] as never,
);
vi.spyOn(api, "addToPlaylist").mockImplementation(async (id: number) => {
on.add(id);
return {} as never;
});
vi.spyOn(api, "removeFromPlaylist").mockImplementation(async (id: number) => {
on.delete(id);
return {} as never;
});
const qc = client();
const saved = () =>
qc.getQueryData<Record<string, { saved?: { value: boolean } }>>(qk.videoMarks())?.v1?.saved
?.value;
render(
<QueryClientProvider client={qc}>
<AddToPlaylist videoId="v1" />
</QueryClientProvider>,
);
fireEvent.click(screen.getByLabelText("playlists.addToPlaylist"));
const row = async (name: string) =>
(await screen.findAllByRole("menuitemcheckbox", { hidden: true })).find((r) =>
r.textContent?.includes(name),
)!;
fireEvent.click(await row("Mix"));
await waitFor(() => expect(api.addToPlaylist).toHaveBeenCalledTimes(1));
expect(saved()).toBeUndefined();
fireEvent.click(await row("playlists.watchLater"));
await waitFor(() => expect(saved()).toBe(true));
await waitFor(async () =>
expect((await row("playlists.watchLater")).getAttribute("aria-checked")).toBe("true"),
);
fireEvent.click(await row("playlists.watchLater"));
await waitFor(() => expect(saved()).toBe(false));
});
it("says a failed load of the playlists as one, not as 'no playlists yet'", async () => {
vi.spyOn(api, "playlists").mockRejectedValue(new Error("down"));
render(
@@ -52,6 +52,18 @@ describe("Button primitive", () => {
}
});
it("a lit cycle paints the pressed look without announcing a toggle", () => {
render(
<Button variant="overlay" lit>
Loop: All
</Button>,
);
const btn = screen.getByRole("button", { name: "Loop: All" });
expect(btn.hasAttribute("aria-pressed")).toBe(false);
expect(btn.hasAttribute("data-lit")).toBe(true);
expect(btn.className).toContain("data-lit:bg-accent");
});
it("a toggle announces aria-pressed and paints the pressed state", () => {
const { rerender } = render(
<Button variant="outline" pressed={false}>
+6 -1
View File
@@ -30,6 +30,9 @@ interface ButtonOwnProps extends Omit<ButtonHTMLAttributes<HTMLButtonElement>, "
loading?: boolean;
/** A toggle: sets aria-pressed and paints the pressed state (the solid accent). */
pressed?: boolean;
/** A mode cycle that is not off: the pressed look without aria-pressed — its name says the mode (a
* cycle is no two-state switch). */
lit?: boolean;
/** Not accepted: a toggle is `pressed` (JSX lets any undeclared `aria-*` through, so it is `never`). */
"aria-pressed"?: never;
/** Sits on a picture (a thumbnail, a poster) or inside a pill field: the pill shape. Implied by the
@@ -55,6 +58,7 @@ const ButtonImpl = forwardRef<HTMLButtonElement, ButtonOwnProps>(function Button
label,
loading = false,
pressed,
lit,
round = false,
fullWidth = false,
className,
@@ -84,9 +88,10 @@ const ButtonImpl = forwardRef<HTMLButtonElement, ButtonOwnProps>(function Button
onBlur={onBlur}
aria-label={label}
aria-pressed={pressed}
data-lit={lit || undefined}
title={label}
className={clsx(
buttonClass({ variant, size: resolvedSize, round, fullWidth, pressed }),
buttonClass({ variant, size: resolvedSize, round, fullWidth, pressed, lit }),
className,
)}
{...rest}
@@ -0,0 +1,64 @@
// @vitest-environment jsdom
//
// ui/DropZone: a drop or a pick hands over the files (only the first unless `multiple`), the drag
// highlight survives the cursor crossing the zone's own children, and a disabled zone takes nothing.
import { afterEach, describe, expect, it, vi } from "vitest";
import { cleanup, fireEvent, render, screen } from "@testing-library/react";
import { DropZone } from "./DropZone";
afterEach(cleanup);
const files = (...names: string[]) => names.map((n) => new File(["x"], n));
function mount(props: Partial<Parameters<typeof DropZone>[0]> = {}) {
const onFiles = vi.fn();
render(
<DropZone onFiles={onFiles} {...props}>
{(over) => <span data-testid="inner">{over ? "over" : "idle"}</span>}
</DropZone>,
);
return { onFiles, zone: screen.getByRole("button"), inner: screen.getByTestId("inner") };
}
describe("DropZone", () => {
it("a drop hands over the first file, or all of them when multiple", () => {
const one = mount();
fireEvent.drop(one.zone, { dataTransfer: { files: files("a", "b") } });
expect(one.onFiles.mock.calls[0]![0].map((f: File) => f.name)).toEqual(["a"]);
cleanup();
const many = mount({ multiple: true });
fireEvent.drop(many.zone, { dataTransfer: { files: files("a", "b") } });
expect(many.onFiles.mock.calls[0]![0].map((f: File) => f.name)).toEqual(["a", "b"]);
});
it("a pick hands over the picked files; an empty pick hands over nothing", () => {
const { onFiles } = mount({ multiple: true });
const input = document.querySelector<HTMLInputElement>('input[type="file"]')!;
expect(input.multiple).toBe(true);
fireEvent.change(input, { target: { files: [] } });
expect(onFiles).not.toHaveBeenCalled();
fireEvent.change(input, { target: { files: files("c") } });
expect(onFiles.mock.calls[0]![0].map((f: File) => f.name)).toEqual(["c"]);
});
it("stays highlighted while the cursor crosses its own children, and clears on the last leave", () => {
const { zone, inner } = mount();
fireEvent.dragEnter(zone);
fireEvent.dragEnter(inner); // into a child: the zone gets its leave after this enter
fireEvent.dragLeave(zone);
expect(inner.textContent).toBe("over");
fireEvent.dragLeave(inner);
expect(inner.textContent).toBe("idle");
});
it("a disabled zone takes no drop", () => {
const { zone, onFiles } = mount({ disabled: true });
fireEvent.drop(zone, { dataTransfer: { files: files("a") } });
expect(onFiles).not.toHaveBeenCalled();
});
it("an invalid zone reads as invalid and points at the reason", () => {
const { zone } = mount({ invalid: true, describedBy: "why" });
expect(zone.getAttribute("aria-invalid")).toBe("true");
expect(zone.getAttribute("aria-describedby")).toBe("why");
});
});
+101
View File
@@ -0,0 +1,101 @@
import { useRef, useState, type ReactNode, type Ref } from "react";
import clsx from "clsx";
import { Pressable } from "./Pressable";
// A place to drop files onto, which is also the button that opens the native file dialog: the
// dashed surface an upload starts from. It owns what each such surface needed and the first one
// got right only after a flicker: dragenter/dragleave fire as the cursor crosses the zone's OWN
// children (an icon, a line of text), so a plain boolean flickers — enter and leave are counted
// and the highlight clears at depth 0. The caller draws the inside, told whether a drag is over it.
export function DropZone({
ref,
onFiles,
multiple = false,
accept,
disabled = false,
invalid = false,
describedBy,
className,
children,
}: {
/** The zone's button, for a caller that moves the focus to it. */
ref?: Ref<HTMLButtonElement>;
/** The files dropped or picked, never empty. One unless `multiple`. */
onFiles: (files: [File, ...File[]]) => void;
multiple?: boolean;
/** The picker's filter (`<input accept>`); a drop is not filtered by it. */
accept?: string;
disabled?: boolean;
/** What the zone holds is refused: the border turns danger and `describedBy` says why. */
invalid?: boolean;
describedBy?: string;
/** Lays the zone out (its padding); the border and the drag paint are the zone's. */
className?: string;
children: (dragOver: boolean) => ReactNode;
}) {
const [dragOver, setDragOver] = useState(false);
const depth = useRef(0);
const inputRef = useRef<HTMLInputElement>(null);
const take = (list: FileList | null | undefined) => {
const [first, ...rest] = [...(list ?? [])];
if (first) onFiles(multiple ? [first, ...rest] : [first]);
};
return (
<>
<Pressable
ref={ref}
align="center"
onClick={() => inputRef.current?.click()}
onDragEnter={(e) => {
e.preventDefault();
depth.current += 1;
setDragOver(true);
}}
onDragOver={(e) => e.preventDefault()}
onDragLeave={() => {
depth.current = Math.max(0, depth.current - 1);
if (depth.current === 0) setDragOver(false);
}}
onDrop={(e) => {
e.preventDefault();
depth.current = 0;
setDragOver(false);
if (!disabled) take(e.dataTransfer.files);
}}
disabled={disabled}
// The refusal is tied to the control it invalidates, not left to a one-shot announcement:
// the zone reads as invalid and points at the sentence, so a reader who tabs back to it
// hears WHY. (Measured, because `aria-invalid` is not in `button`'s ARIA-supported list:
// Chromium's accessibility node for a native button does carry `invalid` from it.) The
// signal that is NOT colour is the caller's sentence — the border only reinforces it.
aria-invalid={invalid}
aria-describedby={describedBy}
// `dragOver` outranks `invalid`: a drag is transient and the refusal repaints the instant
// the drop lands, whereas a zone that stays red under the pointer tells the user their drop
// target is dead — in the one state where they are trying to replace the file.
className={clsx(
"w-full flex flex-col items-center justify-center gap-2 rounded-xl border-2 border-dashed transition disabled:opacity-60",
dragOver
? "border-accent bg-accent/5"
: invalid
? "border-danger"
: "border-border hover:border-fg/40",
className,
)}
>
{children(dragOver)}
</Pressable>
<input
ref={inputRef}
type="file"
className="hidden"
multiple={multiple}
accept={accept}
onChange={(e) => {
take(e.target.files);
e.target.value = ""; // let re-picking the same file fire onChange again
}}
/>
</>
);
}
@@ -59,6 +59,25 @@ describe("IconButton", () => {
expect(screen.getByRole("button", { name: "Saved" }).getAttribute("aria-pressed")).toBe("true");
});
// A mode cycle (auto-advance, loop) is no two-state switch: it looks on, its name says the mode.
it("a lit cycle tints its glyph without announcing a toggle", () => {
const { rerender } = render(
<IconButton label="Auto: Next" lit>
<svg />
</IconButton>,
);
const btn = screen.getByRole("button", { name: "Auto: Next" });
expect(btn.hasAttribute("aria-pressed")).toBe(false);
expect(btn.hasAttribute("data-lit")).toBe(true);
expect(btn.className).toContain("data-lit:text-chosen");
rerender(
<IconButton label="Auto: Off" lit={false}>
<svg />
</IconButton>,
);
expect(btn.hasAttribute("data-lit")).toBe(false);
});
it("loading swaps the glyph for a spinner and swallows the click", () => {
const onClick = vi.fn();
render(
+6 -1
View File
@@ -23,6 +23,9 @@ export interface IconButtonProps
/** A toggle: sets aria-pressed and tints the glyph with the chosen colour (the accent; the media
* amber on the player chrome). */
pressed?: boolean;
/** A mode cycle that is not off: the pressed tint without aria-pressed — its `label` says the mode
* (a cycle is no two-state switch). */
lit?: boolean;
/** Not accepted: a toggle is `pressed` (JSX lets any undeclared `aria-*` through, so it is `never`). */
"aria-pressed"?: never;
/** Sits on a picture or inside a pill field: round (A2). Implied by the `media` variant. */
@@ -40,6 +43,7 @@ export const IconButton = forwardRef<HTMLButtonElement, IconButtonProps>(functio
variant = "ghost",
size = "sm",
pressed,
lit,
round = false,
loading = false,
noTitle = false,
@@ -56,8 +60,9 @@ export const IconButton = forwardRef<HTMLButtonElement, IconButtonProps>(functio
busy={loading}
aria-label={label}
aria-pressed={pressed}
data-lit={lit || undefined}
title={noTitle ? undefined : label}
className={clsx(iconButtonClass({ variant, size, round, pressed }), className)}
className={clsx(iconButtonClass({ variant, size, round, pressed, lit }), className)}
{...rest}
>
{loading ? <Loader2 className="animate-spin" /> : children}
@@ -9,6 +9,7 @@ import KeyboardHelp from "./KeyboardHelp";
import { useContextMenu } from "./useContextMenu";
import {
closeKeyboardHelp,
setScopeCoveredProbe,
useGlobalKeymap,
useHelpConditions,
useKeyScope,
@@ -16,7 +17,8 @@ import {
} from "../../lib/keymap";
import { pushLayer, removeLayer } from "../../lib/layerStack";
import { useMediaKeys } from "../../lib/mediaKeys";
import { useState } from "react";
import { useRef, useState } from "react";
import { navItemAttrs, useListNav } from "../../lib/listNav";
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
@@ -144,6 +146,67 @@ describe("KeyboardHelp", () => {
expect(descriptions()).toContain("keys.help");
});
// The player's related videos are a list inside the layer: their keys work there (`inTopLayer`), so
// the sheet opened over the player lists them — and over a layer with no list, still does not.
it("lists the list and menu keys over a layer that holds a list of its own", () => {
function LayerList() {
const ref = useRef<HTMLDivElement>(null);
useListNav(ref, { enabled: true });
useContextMenu<string>();
return (
<div data-testid="layer">
<div ref={ref}>
<div {...navItemAttrs(0)}>row</div>
</div>
</div>
);
}
render(
<Host>
<LayerList />
</Host>,
);
const withList = pushLayer({ root: () => screen.getByTestId("layer") });
pressH();
expect(descriptions()).toContain("keys.listStep");
expect(descriptions()).toContain("keys.contextMenu");
act(() => closeKeyboardHelp());
removeLayer(withList);
const bare = pushLayer();
pressH();
removeLayer(bare);
expect(descriptions()).not.toContain("keys.listStep");
expect(descriptions()).not.toContain("keys.contextMenu");
});
// The YouTube player's keys wait while a layer sits above it (its related drawer, a dialog): a
// sheet opened there leaves them out, and one opened over the player itself lists them.
it("leaves out a surface's keys while a layer above it has them waiting", () => {
let covered = true;
setScopeCoveredProbe("ytPlayer", () => covered);
try {
render(
<Host>
<Surface scope="ytPlayer" />
</Host>,
);
const id = pushLayer();
pressH();
removeLayer(id);
expect(descriptions()).not.toContain("keys.playPause");
expect(descriptions()).toContain("keys.help");
act(() => closeKeyboardHelp());
covered = false;
pressH();
expect(descriptions()).toContain("keys.playPause");
} catch (e) {
setScopeCoveredProbe("ytPlayer", null);
throw e;
}
setScopeCoveredProbe("ytPlayer", null);
});
// Two live context menus (the Playlists page mounts one, its rail another) must not double the row.
it("lists the context-menu chords once however many surfaces register", () => {
function WithMenu() {
+12 -5
View File
@@ -10,6 +10,7 @@ import {
useHelpConditionValues,
useKeyboardHelpOpen,
useKeyboardHelpOverLayer,
useKeyboardHelpCovered,
useNavigablePages,
} from "../../lib/keymap";
@@ -33,6 +34,7 @@ export default function KeyboardHelp() {
const scopes = useActiveScopes();
const conditions = useHelpConditionValues();
const overLayer = useKeyboardHelpOverLayer();
const covered = useKeyboardHelpCovered();
const pages = useNavigablePages();
if (!open) return null;
@@ -41,11 +43,16 @@ export default function KeyboardHelp() {
// Opened over a player or a dialog, the page-level keys (`s`, `g …`, `j`/`k`) are quiet — listing
// them there would promise keys that do nothing until the overlay closes. And a `g` chord is listed
// only for a module this account can actually reach.
// The context-menu key goes too: every `useContextMenu` is a page list, and with an overlay up the
// focus Shift+F10 needs is inside the overlay (found in UAT).
const live = overLayer
? scopes.filter((s) => !isPageLevelScope(s) && s !== "contextMenu")
: scopes;
// The context-menu key goes too, with the page's lists. Over a layer that holds a list of its own
// (the player's related videos) the list keys and the menu key work on THAT list, so they stay —
// but not Delete: no list inside a layer takes it, and a page's remover behind does not reach.
const inLayer = (s: (typeof scopes)[number]) =>
(!isPageLevelScope(s) && s !== "contextMenu") ||
(overLayer === "layerWithList" && (s === "list" || s === "contextMenu"));
// A surface under a layer of its own (the player under its related drawer) waits: not listed.
const live = (overLayer === "page" ? scopes : scopes.filter(inLayer)).filter(
(s) => !covered.has(s),
);
const lines = helpLines(live, pages).filter((l) => !l.when || !!conditions[l.when]);
const words = { Space: t("keys.cap.space"), ContextMenu: t("keys.cap.menu") };
return (
+1 -1
View File
@@ -147,7 +147,7 @@ export default function Modal({
// An effect event, so a new handler each render never re-registers: the stack id is assigned once per mount.
const close = useEffectEvent(onClose);
useEffect(() => {
const id = pushLayer();
const id = pushLayer({ root: () => rootRef.current });
const onKey = (e: KeyboardEvent) => {
if (takeEscape(e, id)) {
e.stopPropagation();
+7 -7
View File
@@ -35,13 +35,13 @@ Counter-examples, deliberately NOT here: `PosterTile` (knows about Plex montage
## What lives here
| area | components |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| overlays & anchoring | `AnchoredPopover` + `anchoredPopoverHooks` (the popover engine: arrows, typeahead, Escape, anchor-flip), `ActionMenu`, `MembershipMenu` (the body of a "which sets is it in" menu: checkbox rows + a field naming a new set), `ContextMenu` + `useContextMenu` (the SAME menu, opened by right-click / long-press / Shift+F10 / a row's `MoreButton` "…" via `buttonProps`; `anchor` for a panel a row opens; `PageMenu` for a page's background menu), `Modal` (+ `ModalFooter`; `initialFocus` by purpose, C4), `Overlay`, `SidePanel`, `Tooltip`, `Toaster` (+ `toastLevels`, `overlayOrder`; `lib/toastDock` docks one control under the stack, in every host), `CommandPalette`, `KeyboardHelp`, `ConfirmProvider` + `confirmContext` (`useConfirm` / `useChoice`: the app's promise-based confirm, mounted once at the root; a decision with more than two outcomes is one `useChoice`, never two chained confirms), `dialogFocus` (the focus half of the dialog contract: `Modal`'s, and the wizard's own shell), `menuGroups` (one menu from its groups, a rule only between non-empty ones), `pageMenuSlot` + `pageMenuItems` (a page's background menu, mirroring its own toolbar) |
| actions & inputs | `Button`, `IconButton`, `Pressable` (`selectable` when its text is a title or a name), `Chip` + `chipStyles`, `DragHandle` + `OrderButtons` / `useOrderItems` (the non-drag reorder, WCAG 2.5.7) (one per button family, docs/DESIGN.md; a raw `<button>` or a bare `FocusSafeButton` outside the control primitives — eslint `BUTTON_PRIMITIVES` — is a lint error), `Input` + `fieldStyles`, `RenameField` (a name typed in place of the name: Enter or blur saves, Escape cancels; the caller hands the keyboard back), `Meter` (how full something is against a limit — one danger line for every place showing the same measure), `TimecodeField` + `useTimecodeField` / `TimecodeError` (a cut's time typed in: the draft, held by the caller, until Enter or blur takes it, then the value as it landed; unreadable stays once offered, marked invalid, and the caller places the line saying how to type one in its flow, outside a label; a draft goes when the value moves under it), `NativeControls`, `MenuCheck` (the drawn check of a menu row or a whole-row checkbox button; its tick is the `chosen` colour, which `media-marks` turns amber on the player chrome), `mediaChrome` (`MEDIA_CHOSEN`, a chosen tab or toggle on the chrome), `SelectTrigger` (the field family; a raw `<input>` is a lint error), `useReturnFocus`, `surfaces` (the B5 glass roles; a bare glass class is a lint error), `Badge` (a static label), `LinkButton` / `IconLink` (a link wearing the button face) over `buttonStyles`, `Skeleton` (B8 first loads, through `LoadingState shape`), `NavRail` (B7; `NavRailGroup` folds a set of rare pages into one row with a panel), `form`, `DraftSaveBar`, `Combobox`, `FilterSelect`, `SegmentedToggle`, `SortControl`, `ColumnSortControl`, `PriorityStepper`, `UndoToolbar`, `ViewSwitcher` (a list's display modes behind one toolbar icon), `copyItems` (`useCopyItems`: the "Copy title" / "Copy link" rows every object's menu ends with) + `useCopyFeedback` (a Copy button's "Copied" moment), `MatchToggle` (a multi-value filter's any/all switch), `LanguageSwitcher` (presentational: the caller decides what a change does), `sortableA11y` (what a screen reader hears during a keyboard reorder) |
| data display | `DataTable` (+ `progressiveRows`: a long page mounts in steps), `tableHeader` + `tableAlign`, `HiddenFilterCue` (names a filter whose column is off screen), `Pager` + `pagerContent`, `VirtualGrid` + `gridColumns` (the column/width maths, incl. the ultrawide `maxCols` cap), `readAhead`, `Carousel`, `MediaCard` + `mediaCardStyles` (the one media card, docs/DESIGN.md "Media card") + `MediaPreview` / `useMediaPreview` (its mouse-rest preview), `NavGrid` (a plain grid, list or table body the keyboard walks — `DataTable`, `VirtualGrid` and `Carousel` take `keyboardNav`), `RemoteImage` (every network picture: no Referer, retries past the negative cache, a fallback instead of the broken-image mark — a bare `<img>` is linted out), `Avatar`, `CountBadge`, `ItemCount`, `FilterChipGroup`, `RelativeTime` ("3d ago" with the exact minute on hover, docs/DESIGN.md "Copy"), `ClampedText` (a long text clamped to a few lines; More / Less only when it is cut), `StatTiles` + `StatTile` / `StatBars` (number tiles, a breakdown as horizontal bars), `RangeTrack` + `trackPosition` + `RulerTicks` (a time track over a window of a media item: regions, slider ends the arrows step, a playhead, a background slot — the editor's cuts, the players' clip panel), `CropFrame` (a crop frame over a picture, on the picture and not its bars: drag, edges, corners, the arrows; the editor's crop, the players' clip crop), `reveal` (a card's hover chrome: hover or keyboard focus on a mouse, never on touch) |
| page & panel structure | `PageShell` + `pageToolbarSlot` + `pageHeaderSlot` (`PageHeader`: a page fills the floating header) + `pageHeading`, `PageScroller`, `Headings` (`PageHeading`, `SectionLabel`), `SkipLink` + `landmarks`, `FilterFold` (B4), `scrollFold` (`useScrollFold`: a fixed header that folds once its content scrolls), `PanelGroup`, `PanelGroups`, `Tabs` + `tabPanel`, `BackToTop` + `backToTopState` (PageScroller keeps the page's end clear of it), `FilterRail` (the filter-rail shell a rail fills with its groups: title bar, customize, reset), `stickyTop` (a sticky header at the scroller's top turns its top fade off) |
| state & messaging | `QueryState`, `Banner`, `InlineNotice`, `CrashBoundary` |
| area | components |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| overlays & anchoring | `AnchoredPopover` + `anchoredPopoverHooks` (the popover engine: arrows, typeahead, Escape, anchor-flip), `ActionMenu`, `MembershipMenu` (the body of a "which sets is it in" menu: checkbox rows + a field naming a new set), `ContextMenu` + `useContextMenu` (the SAME menu, opened by right-click / long-press / Shift+F10 / a row's `MoreButton` "…" via `buttonProps`; `anchor` for a panel a row opens; `PageMenu` for a page's background menu), `Modal` (+ `ModalFooter`; `initialFocus` by purpose, C4), `Overlay`, `SidePanel`, `Tooltip`, `Toaster` (+ `toastLevels`, `overlayOrder`; `lib/toastDock` docks one control under the stack, in every host), `CommandPalette`, `KeyboardHelp`, `ConfirmProvider` + `confirmContext` (`useConfirm` / `useChoice`: the app's promise-based confirm, mounted once at the root; a decision with more than two outcomes is one `useChoice`, never two chained confirms), `dialogFocus` (the focus half of the dialog contract: `Modal`'s, and the wizard's own shell), `menuGroups` (one menu from its groups, a rule only between non-empty ones), `pageMenuSlot` + `pageMenuItems` (a page's background menu, mirroring its own toolbar) |
| actions & inputs | `Button`, `IconButton`, `Pressable` (`selectable` when its text is a title or a name), `Chip` + `chipStyles`, `DragHandle` + `OrderButtons` / `useOrderItems` (the non-drag reorder, WCAG 2.5.7) (one per button family, docs/DESIGN.md; a raw `<button>` or a bare `FocusSafeButton` outside the control primitives — eslint `BUTTON_PRIMITIVES` — is a lint error), `Input` + `fieldStyles`, `RenameField` (a name typed in place of the name: Enter or blur saves, Escape cancels; the caller hands the keyboard back), `DropZone` (the dashed place a file upload starts from: a drop, or a click that opens the file dialog), `Meter` (how full something is against a limit — one danger line for every place showing the same measure), `TimecodeField` + `useTimecodeField` / `TimecodeError` (a cut's time typed in: the draft, held by the caller, until Enter or blur takes it, then the value as it landed; unreadable stays once offered, marked invalid, and the caller places the line saying how to type one in its flow, outside a label; a draft goes when the value moves under it), `NativeControls`, `MenuCheck` (the drawn check of a menu row or a whole-row checkbox button; its tick is the `chosen` colour, which `media-marks` turns amber on the player chrome), `mediaChrome` (`MEDIA_CHOSEN`, a chosen tab or toggle on the chrome), `SelectTrigger` (the field family; a raw `<input>` is a lint error), `useReturnFocus`, `surfaces` (the B5 glass roles; a bare glass class is a lint error), `Badge` (a static label), `LinkButton` / `IconLink` (a link wearing the button face) over `buttonStyles`, `Skeleton` (B8 first loads, through `LoadingState shape`), `NavRail` (B7; `NavRailGroup` folds a set of rare pages into one row with a panel), `form`, `DraftSaveBar`, `Combobox`, `FilterSelect`, `SegmentedToggle`, `SortControl`, `ColumnSortControl`, `PriorityStepper`, `UndoToolbar`, `ViewSwitcher` (a list's display modes behind one toolbar icon), `copyItems` (`useCopyItems`: the "Copy title" / "Copy link" rows every object's menu ends with) + `useCopyFeedback` (a Copy button's "Copied" moment), `MatchToggle` (a multi-value filter's any/all switch), `LanguageSwitcher` (presentational: the caller decides what a change does), `sortableA11y` (what a screen reader hears during a keyboard reorder) |
| data display | `DataTable` (+ `progressiveRows`: a long page mounts in steps), `tableHeader` + `tableAlign`, `HiddenFilterCue` (names a filter whose column is off screen), `Pager` + `pagerContent`, `VirtualGrid` + `gridColumns` (the column/width maths, incl. the ultrawide `maxCols` cap), `readAhead`, `Carousel`, `MediaCard` + `mediaCardStyles` (the one media card, docs/DESIGN.md "Media card") + `MediaPreview` / `useMediaPreview` (its mouse-rest preview), `NavGrid` (a plain grid, list or table body the keyboard walks — `DataTable`, `VirtualGrid` and `Carousel` take `keyboardNav`), `RemoteImage` (every network picture: no Referer, retries past the negative cache, a fallback instead of the broken-image mark — a bare `<img>` is linted out), `Avatar`, `CountBadge`, `ItemCount`, `FilterChipGroup`, `RelativeTime` ("3d ago" with the exact minute on hover, docs/DESIGN.md "Copy"), `ClampedText` (a long text clamped to a few lines; More / Less only when it is cut), `StatTiles` + `StatTile` / `StatBars` (number tiles, a breakdown as horizontal bars), `RangeTrack` + `trackPosition` + `RulerTicks` (a time track over a window of a media item: regions, slider ends the arrows step, a playhead, a background slot — the editor's cuts, the players' clip panel), `CropFrame` (a crop frame over a picture, on the picture and not its bars: drag, edges, corners, the arrows; the editor's crop, the players' clip crop), `reveal` (a card's hover chrome: hover or keyboard focus on a mouse, never on touch) |
| page & panel structure | `PageShell` + `pageToolbarSlot` + `pageHeaderSlot` (`PageHeader`: a page fills the floating header) + `pageHeading`, `PageScroller`, `Headings` (`PageHeading`, `SectionLabel`), `SkipLink` + `landmarks`, `FilterFold` (B4), `scrollFold` (`useScrollFold`: a fixed header that folds once its content scrolls), `PanelGroup`, `PanelGroups`, `Tabs` + `tabPanel`, `BackToTop` + `backToTopState` (PageScroller keeps the page's end clear of it), `FilterRail` (the filter-rail shell a rail fills with its groups: title bar, customize, reset), `stickyTop` (a sticky header at the scroller's top turns its top fade off) |
| state & messaging | `QueryState`, `Banner`, `InlineNotice`, `CrashBoundary` |
## The size contract (E10 S1c)
+19 -4
View File
@@ -57,9 +57,16 @@ const BUTTON_VARIANTS: Record<ButtonVariant, string> = {
};
// A pressed toggle (`pressed`) is the solid accent in every variant (B3: "selected stays the solid
// accent"); the layer utilities leave it alone, so it stays solid under the pointer too.
// accent"); the layer utilities leave it alone, and its colours are marked important (`!`) because a
// variant's hover colour (`act:hover:text-fg`, the overlay's `act:hover:bg-white/15`) is the more
// specific selector — without it the accent turned grey, or white, under the pointer.
const PRESSED =
"aria-pressed:bg-accent aria-pressed:text-accent-fg aria-pressed:border-accent aria-pressed:no-underline";
"aria-pressed:bg-accent! aria-pressed:text-accent-fg! aria-pressed:border-accent! aria-pressed:no-underline!";
// A mode cycle that is not off (`lit`: auto-advance Next, loop One) looks pressed but is no toggle —
// its name says the mode, and aria-pressed would announce a two-state switch whose name changes on
// every press. The same look as PRESSED, under the pointer too: the layers stand down on `[data-lit]`.
const LIT =
"data-lit:bg-accent! data-lit:text-accent-fg! data-lit:border-accent! data-lit:no-underline!";
// B1, the whole ladder: explicit heights, so a toolbar's `sm` button, search pill, dropdown trigger and
// small icon button all line up at 28. A call site picks a rung; it does not invent one. rem-derived, so
@@ -120,8 +127,10 @@ const ICON_VARIANTS: Record<IconButtonVariant, string> = {
};
// An icon toggle that is on shows it in the glyph (watched, saved, a selected track), not as a solid
// fill: a solid accent square in a row of quiet glyphs outshouts the row's primary action.
const ICON_PRESSED = "aria-pressed:text-chosen";
// fill: a solid accent square in a row of quiet glyphs outshouts the row's primary action. Important
// for the same reason as PRESSED: the variant's hover colour would take the tint off under the pointer.
const ICON_PRESSED = "aria-pressed:text-chosen!";
const ICON_LIT = "data-lit:text-chosen!";
const ICON_SIZES: Record<IconButtonSize, string> = {
sm: "size-7 [&>svg]:size-4",
@@ -138,17 +147,20 @@ export function buttonClass({
round = false,
fullWidth = false,
pressed,
lit,
}: {
variant: ButtonVariant;
size: ButtonSize;
round?: boolean;
fullWidth?: boolean;
pressed?: boolean;
lit?: boolean;
}) {
return clsx(
BASE,
BUTTON_VARIANTS[variant],
pressed !== undefined && PRESSED,
lit !== undefined && LIT,
SIZES[size],
size !== "none" && (round || variant === "media" ? "rounded-full" : "rounded-lg"),
// The coarse-pointer target (DESIGN.md A3) lives HERE, not in the call sites — on every variant.
@@ -165,17 +177,20 @@ export function iconButtonClass({
size,
round = false,
pressed,
lit,
}: {
variant: IconButtonVariant;
size: IconButtonSize;
round?: boolean;
pressed?: boolean;
lit?: boolean;
}) {
return clsx(
"inline-grid shrink-0 place-items-center transition touch-target aria-expanded:text-chosen disabled:opacity-50 disabled:cursor-not-allowed aria-disabled:opacity-50 aria-disabled:cursor-not-allowed",
ICON_VARIANTS[variant],
round || variant === "media" ? "rounded-full" : "rounded-lg",
pressed !== undefined && ICON_PRESSED,
lit !== undefined && ICON_LIT,
ICON_SIZES[size],
);
}
@@ -52,10 +52,11 @@ const PLAIN: Record<string, [number, string]> = {
"modules/settings/SettingsPanel.tsx": [1, "a colour swatch"],
"modules/channels/SyncStatus.tsx": [1, "the status pill trigger"],
"features/channel/TagManager.tsx": [1, "a suggestion in a picker"],
"modules/files/UploadDialog.tsx": [1, "the drop zone"],
"components/ui/DropZone.tsx": [1, "the drop zone (its text is its own prompt)"],
"modules/files/VideoEditor.tsx": [2, "timeline glyphs"],
"shell/Welcome.tsx": [1, "a screenshot (zoom)"],
"modules/playlists/playlistColumns.tsx": [1, "the thumbnail"],
"modules/channels/WatchedVideosDialog.tsx": [1, "the thumbnail (its title sits beside it)"],
"modules/plex/plexDetailUi.tsx": [2, "a switch row and the poster"],
"modules/plex/plexViews.tsx": [1, "the rating filter (a number)"],
"components/ui/ActionMenu.tsx": [1, "a menu row"],
@@ -1,4 +1,54 @@
[
{
"version": "0.112.1",
"date": "2026-10-27",
"new": [],
"fixed": ["Security: the sign-in library is updated, closing two published vulnerabilities."]
},
{
"version": "0.112.0",
"date": "2026-10-25",
"new": [
"Player: a Related tab lists videos related to the one playing, leaving out what you have already watched (your uploaded YouTube history included). Channels you follow are marked, and you can switch between On topic and Trending.",
"Play a related video in place; Back returns to the one you opened. Admins set a daily per-user limit in Config › Quota (default 200, 0 turns it off).",
"Side panel: the list controls and the watched toggle are glyphs on one line, and Auto and Loop show when they are on, also under the pointer."
],
"fixed": [
"Live search: a search that never reached YouTube no longer uses up a daily slot, a busy moment says so after a few seconds instead of waiting on, and a first page that failed offers Retry.",
"Menus: Play from the beginning wears the same ↺ glyph everywhere, and Reset its own eraser glyph.",
"On the day of this update, API searches made earlier that day stop counting toward the daily search limit, until Pacific midnight."
]
},
{
"version": "0.111.0",
"date": "2026-10-24",
"new": [
"Channels › Discover from your history: the channels your uploaded YouTube history shows you watching often (five videos of five minutes or more) but not following.",
"Follow one, put it out of the list (with Undo), or open the videos you watched there and play them one by one or all in a row."
],
"fixed": [
"A channel someone explored and then followed on YouTube is now part of the shared Library like any followed channel, and so are its videos; before, they could stay hidden and their watched marks never applied."
]
},
{
"version": "0.110.0",
"date": "2026-10-23",
"new": [
"YouTube history: upload your Google Takeout export in Settings › Account. What you watched on YouTube is marked watched here (never a Short, never over a state you set), and your YouTube Watch later list joins yours.",
"A video the catalog does not know yet is remembered and marked once it arrives. The import's outcome reaches your inbox, and admins see each run on the Scheduler page."
],
"fixed": [
"Scheduler: a channel's full-history backfill asked for again while one is running now runs after it, instead of being silently dropped."
]
},
{
"version": "0.109.2",
"date": "2026-10-23",
"new": [],
"fixed": [
"Plex player: when the server restarts mid-episode (an update), playback carries on from the buffer and picks up where you are once it is back, instead of showing \"Playback couldn't start\" and stopping where the buffer ended."
]
},
{
"version": "0.109.1",
"date": "2026-10-23",
@@ -1,4 +1,56 @@
[
{
"version": "0.112.1",
"date": "2026-10-27",
"new": [],
"fixed": [
"Biztonság: frissült a bejelentkezési könyvtár, ami két nyilvánosan ismert sérülékenységet zár be."
]
},
{
"version": "0.112.0",
"date": "2026-10-25",
"new": [
"Lejátszó: a Kapcsolódók fül a lejátszott videóhoz kapcsolódó videókat mutatja, a már megnézettek nélkül (a feltöltött YouTube-előzményeiddel együtt). A követett csatornák jelölve vannak, és válthatsz a Témába vágó és a Trend között.",
"A kapcsolódó videó helyben játszható, a Vissza a megnyitott videóhoz visz. Az adminok a Konfig › Kvóta alatt adhatnak napi felhasználónkénti limitet (alapból 200, a 0 kikapcsolja).",
"Oldalpanel: a listavezérlők és a megnézve-kapcsoló glifként egy sorban vannak, az Auto és a Loop bekapcsolt állapota az egérmutató alatt is látszik."
],
"fixed": [
"Élő keresés: a YouTube-ig el sem jutott keresés már nem fogyaszt napi keretet, túlterheléskor pár másodperc után üzenet jön a végtelen várakozás helyett, a hibás első oldal pedig Újra gombot ad.",
"Menük: a Lejátszás az elejétől mindenhol ugyanazt a ↺ glifet kapta, az Alaphelyzet saját radír glifet.",
"A frissítés napján az aznap korábban indított API-s keresések nem számítanak bele a napi keresési limitbe, a csendes-óceáni éjfélig."
]
},
{
"version": "0.111.0",
"date": "2026-10-24",
"new": [
"Csatornák › Felfedezés az előzményekből: azok a csatornák, amelyeket a feltöltött YouTube-előzményeid szerint sokat nézel (öt legalább ötperces videó), de nem követsz.",
"Követheted, kiveheted a listából (visszavonható), vagy megnyithatod az ott megnézett videóit, és egyenként vagy sorban lejátszhatod őket."
],
"fixed": [
"Ha egy felfedezett csatornát valaki a YouTube-on kezd követni, az most a többi követett csatornához hasonlóan a közös Könyvtár része lesz a videóival együtt; eddig rejtve maradhattak, és a megnézett jelöléseik sem érvényesültek."
]
},
{
"version": "0.110.0",
"date": "2026-10-23",
"new": [
"YouTube-előzmények: a Google Takeout exportod a Beállítások › Fiók oldalon tölthető fel. Amit a YouTube-on megnéztél, itt megnézettnek jelölődik (Short soha, és amit itt beállítottál, azt nem írja felül), a „Megnézem később” listád pedig az ittenihez adódik.",
"Egy videót, amit a katalógus még nem ismer, megjegyez, és megjelöli, amint bekerül. Az import eredménye a postafiókodba érkezik, az adminok pedig minden futását látják az Ütemező oldalon."
],
"fixed": [
"Ütemező: ha egy csatorna teljes előzményének letöltését futás közben újra kérik, az most utána lefut, nem vész el csendben."
]
},
{
"version": "0.109.2",
"date": "2026-10-23",
"new": [],
"fixed": [
"Plex-lejátszó: ha a szerver epizód közben újraindul (frissítés), a lejátszás a pufferből megy tovább, és ahol tartasz, ott folytatódik, amint a szerver visszajött – nem írja ki, hogy „Nem sikerült elindítani a lejátszást”, és nem áll meg a puffer végén."
]
},
{
"version": "0.109.1",
"date": "2026-10-23",
File diff suppressed because it is too large Load Diff
+259 -67
View File
@@ -23,6 +23,7 @@ import {
CheckCheck,
ExternalLink,
Info,
ListVideo,
Maximize,
Minimize,
Minimize2,
@@ -49,11 +50,13 @@ import {
anyLayerAbove,
escapeHandled,
} from "../../lib/layerStack";
import { focusInNavList } from "../../lib/listNav";
import { trapTab, useDialogChrome } from "../../components/ui/dialogFocus";
import DownloadButton from "../downloads/DownloadButton";
import ClipPanel, { ClipCropFrame, type ClipCrop } from "./ClipPanel";
import { markInAt, markOutAt, openAt, type ClipRange } from "../../lib/clipRange";
import { api, type Video, type VideoStatus } from "../../lib/api";
import { markedValue, useVideoMarks } from "../../lib/progressCache";
import type { YTPlayer } from "../../lib/youtube";
import {
channelYouTubeUrl,
@@ -63,7 +66,7 @@ import {
relativeTime,
} from "../../lib/format";
import { canFullscreenElements, fullscreenElement } from "../../lib/fullscreen";
import { dispatch, isActivationTarget, useKeyScope } from "../../lib/keymap";
import { dispatch, isActivationTarget, setScopeCoveredProbe, useKeyScope } from "../../lib/keymap";
import { useBackToClose } from "../../lib/history";
import ChannelNameLink from "../video/ChannelNameLink";
import Toaster from "../../components/ui/Toaster";
@@ -83,6 +86,11 @@ import { SURFACE } from "../../components/ui/surfaces";
import { IconLink, LinkButton } from "../../components/ui/LinkButton";
import { SectionLabel } from "../../components/ui/Headings";
import { useMediaQuery } from "../../lib/useMediaQuery";
import { useMe } from "../../lib/useMe";
import Modal from "../../components/ui/Modal";
import Tabs from "../../components/ui/Tabs";
import { useTabPanel } from "../../components/ui/tabPanel";
import { RelatedVideos } from "./RelatedVideos";
// Experiment (branch experiment/inline-player): play the video in-app via the
// YouTube IFrame Player API (not a bare embed) so we can read playback position
@@ -155,6 +163,25 @@ const META_OWN_LINE = "w-full [&>span:first-child>[data-meta-sep]]:hidden";
// the line wraps and the meta, or the actions after it, move down instead of squeezing.
const META_MIN = "min-w-48 flex-1";
// The watched toggle's own word on an item: what it last asked for (`seq`, its write), and when — its
// click, then the server's last answer to any of its writes.
type OwnMark = { status: VideoStatus; seq: number; at: number };
// One of the toggle's writes was answered. Its answer has just come back as a session mark too (the
// write marks before it resolves), so the toggle's word is stamped after it and outranks it; a later
// mark from elsewhere still outranks the toggle. The latest write refused, the toggle's word is
// dropped and the item shows what the server holds.
function settled(marks: Record<string, OwnMark>, id: string, seq: number, ok: boolean) {
const own = marks[id];
if (!own) return marks;
if (!ok && own.seq === seq) {
const rest = { ...marks };
delete rest[id];
return rest;
}
return { ...marks, [id]: { ...own, at: Date.now() } };
}
export default function PlayerModal({
video,
startAt,
@@ -162,6 +189,7 @@ export default function PlayerModal({
startIndex,
onClose,
onState,
onWrite,
onRemoveFromList,
}: {
video: Video;
@@ -174,7 +202,13 @@ export default function PlayerModal({
queue?: Video[];
startIndex?: number;
onClose: () => void;
onState: (id: string, status: VideoStatus) => void;
/** The player's own marks (its watched toggle, the auto-mark at the end), for the item it is on —
* named by `video`, which may be none of the host's (a related video played in place). Returns
* the write: the toggle weighs its own word at the server's answer, and drops a refused one. */
onState: (id: string, status: VideoStatus, video?: Video) => Promise<unknown>;
/** A write the related list made itself, for a host that refetches once the player closes and its
* writes have landed (the feed's player session). */
onWrite?: (write: Promise<unknown>) => void;
/** Remove the playing video from the list it was opened from (the Playlists module passes its own
* `removeItem`; the feed passes nothing — there is no list to remove it from). Returns whether it
* actually happened, because the CONSEQUENCE is the player's: its launch queue is frozen, so it
@@ -230,6 +264,23 @@ export default function PlayerModal({
// startAt only applies to the item we opened with; advanced items resume their own pos.
const resumeAt =
startAt != null && active.id === video.id ? startAt : active.position_seconds || 0;
// A related video played in place (the owner's pick) is the item it now is: its progress is
// checkpointed, it is marked watched at its end, and its own watched / playlist / download
// controls show — while the queue stays where it was, and Back returns to the opened item. It is
// the SUBJECT of the player from here down; `active` stays the queue's item, which the related
// list, the counter and prev/next belong to. A queue step leaves it.
// `startAt` null resumes it where it was left (this session's stored position first); a number is
// where the user asked it to start — Restart's 0 — and beats that position.
const [related, setRelated] = useState<{ video: Video; startAt: number | null } | null>(null);
const [relatedOf, setRelatedOf] = useState(active.id);
if (relatedOf !== active.id) {
setRelatedOf(active.id);
setRelated(null);
}
const subject = related?.video ?? active;
const subjectResume = related
? (related.startAt ?? related.video.position_seconds ?? 0)
: resumeAt;
const mountRef = useRef<HTMLDivElement | null>(null);
const playerRef = useRef<YTPlayer | null>(null);
// Keyboard/wheel controls. We keep focus on the modal card (not the cross-origin player
@@ -245,11 +296,16 @@ export default function PlayerModal({
// useYtNativeControls, useYtBigView) where a fresh identity each render would spread into their
// dep lists too.
const focusModal = useCallback(() => cardRef.current?.focus(), []);
// The player's place on the layer stack (registered below), for what must defer to a layer above.
const layerIdRef = useRef<number | null>(null);
// What the player does on its own (turning ready, the pointer leaving the video, the window
// regaining focus, a wheel over it, leaving fullscreen) takes the focus back only when it may have
// slipped into the iframe or off the card. Taking it off a control the viewer tabbed to restarted
// their Tab from the first button.
// their Tab from the first button. Nor while a layer that owns the keyboard is open above the
// player (the related drawer, a dialog): its focus is outside the card by design, and the window
// regaining focus pulled it onto the card behind the scrim.
const reclaimFocus = useCallback(() => {
if (layerIdRef.current != null && anyBlockingLayerAbove(layerIdRef.current)) return;
const card = cardRef.current;
const active = document.activeElement;
if (
@@ -345,7 +401,7 @@ export default function PlayerModal({
// currently-playing id may differ from the active item.
const [currentVideoId, setCurrentVideoId] = useState(active.id);
const currentIdRef = useRef(active.id);
const navigated = currentVideoId !== active.id;
const navigated = currentVideoId !== subject.id;
// Clip mode: the range being marked of the video that is playing, or null. Its panel sits under
// the video, which is its preview; another video starts without one.
const [clip, setClip] = useState<ClipRange | null>(null);
@@ -365,7 +421,7 @@ export default function PlayerModal({
// The picture the crop frame is drawn on. The iframe never tells its video's size: a short is
// 9:16, anything else is taken as the player's 16:9, and the cut fits the frame to the real picture
// (downloads/clip) — so a 4:3 video's frame drawn over its bars loses only what lies on the bars.
const picture = !navigated && active.is_short ? 9 / 16 : 16 / 9;
const picture = !navigated && subject.is_short ? 9 / 16 : 16 / 9;
// The docked clip panel's height, measured (it wraps with the window and the text size): the video
// above it ends there.
const clipDockRef = useRef<HTMLDivElement>(null);
@@ -482,7 +538,7 @@ export default function PlayerModal({
// Title/author of a navigated-to video, read from the player (free, no API call).
const [liveData, setLiveData] = useState<{ title?: string; author?: string } | null>(null);
// What is playing, by name: the panel's title, and the dialog's name (`titleId`) in either layout.
const heading = navigated ? (liveData?.title ?? t("player.loading")) : active.title;
const heading = navigated ? (liveData?.title ?? t("player.loading")) : subject.title;
// IFrame Player API error code, when the current video can't play in the embed (e.g. an
// auto-generated Topic "Art Track" with embedding disabled → 101/150, or removed/private →
// 100). We overlay a friendly message + "Open on YouTube" instead of YouTube's bare error.
@@ -493,9 +549,13 @@ export default function PlayerModal({
const loadVideo = useStableCallback((id: string, start: number | null) => {
const p = playerRef.current;
if (!p || typeof p.loadVideoById !== "function") return;
// Explicit start wins; otherwise resume the active item, start others at 0.
// Explicit start wins; otherwise resume the item the player is on, start others at 0.
const startSeconds =
start != null ? start : id === active.id ? (sessionPosRef.current.get(id) ?? resumeAt) : 0;
start != null
? start
: id === subject.id
? (sessionPosRef.current.get(id) ?? subjectResume)
: 0;
setPlayerError(null);
p.loadVideoById({ videoId: id, startSeconds: startSeconds || 0 });
currentIdRef.current = id;
@@ -504,23 +564,41 @@ export default function PlayerModal({
});
// Local mirror of watch status so the toggle reacts instantly; changes are propagated to the feed
// (and server) via onState. The mirror carries the id it DESCRIBES, and a write for any other id
// is ignored rather than displayed — that is the enforcement, and it has to be in code because
// one writer arrives late by construction: useYtProgress's cleanup for the OUTGOING item runs
// after the commit that already advanced `active`, and near the end of a clip it auto-marks that
// item watched (playerModalHooks.ts `maybeAutoWatch`). Keyed, that stray write is simply not this
// item's, so the toggle falls back to `active.status` and no ordering has to be argued about.
// Unkeyed it landed last and painted the NEXT video as watched; a corrective click then sent
// onState(next, "new") for an item that was already new.
const [status, setStatus] = useState<{ id: string; value: VideoStatus }>({
id: active.id,
value: active.status,
});
const watched = (status.id === active.id ? status.value : active.status) === "watched";
// (and server) via onState. Each mark is kept under the id it was MADE for, and that has to be in
// code because one writer arrives late by construction: useYtProgress's cleanup for the OUTGOING
// item runs after the commit that already advanced `active`, and near the end of a clip it
// auto-marks that item watched (playerModalHooks.ts `maybeAutoWatch`). Keyed, that late write is
// recorded for the item it belongs to (shown again if the user steps back to it) and never
// painted on the next one; unkeyed it landed last and painted the NEXT video as watched, and a
// corrective click then sent onState(next, "new") for an item that was already new.
// The newest word wins: this toggle's own mark, or one the session confirmed elsewhere since the
// player opened (the related row's own menu, a notice's Undo — `useVideoMarks`), else the item as
// it was opened. The toggle alone missed those, and its first click re-sent the same status.
// The toggle's own writes come back as session marks too, stamped at their answer, so a click
// weighed only at its own time lost to the late answer of the click before it: each answer stamps
// the toggle's word again (`settled`). A refused one is dropped, so the toggle does not keep showing
// what the server turned down.
const [marks, setMarks] = useState<Record<string, OwnMark>>({});
const ownSeq = useRef(0);
const [openedAt] = useState(() => Date.now());
const sessionMarks = useVideoMarks();
const own = marks[subject.id];
const confirmed = markedValue(sessionMarks, subject.id, "status");
const elsewhere = confirmed && confirmed.at >= openedAt ? confirmed : undefined;
const status =
own && (!elsewhere || own.at >= elsewhere.at)
? own.status
: (elsewhere?.value ?? subject.status);
const watched = status === "watched";
const setWatched = (on: boolean) => {
const id = subject.id;
const next = on ? "watched" : "new";
setStatus({ id: active.id, value: next });
onState(active.id, next);
const seq = ++ownSeq.current;
setMarks((m) => ({ ...m, [id]: { status: next, seq, at: Date.now() } }));
Promise.resolve(onState(id, next, subject)).then(
() => setMarks((m) => settled(m, id, seq, true)),
() => setMarks((m) => settled(m, id, seq, false)),
);
};
// On queue advance (active changes), point `currentVideoId` at the new item — DURING that render,
@@ -531,9 +609,9 @@ export default function PlayerModal({
// The watch status is NOT synced here: it is keyed by id above, which is both stale-proof and
// immune to the late writer this block cannot see (see that comment).
const [syncedTo, setSyncedTo] = useState(active.id);
if (syncedTo !== active.id) {
setSyncedTo(active.id);
setCurrentVideoId(active.id);
if (syncedTo !== subject.id) {
setSyncedTo(subject.id);
setCurrentVideoId(subject.id);
}
// `currentIdRef` must NOT move into the block above, and the reason is EFFECT ORDERING, which the
@@ -547,8 +625,8 @@ export default function PlayerModal({
// frozen launch queue's stale position — which is the exact bug the session-position map exists
// to prevent (748 instead of 830, in the test's numbers).
useEffect(() => {
currentIdRef.current = active.id;
}, [active.id]);
currentIdRef.current = subject.id;
}, [subject.id]);
const seekTo = useStableCallback((seconds: number) => {
const p = playerRef.current;
@@ -640,6 +718,8 @@ export default function PlayerModal({
if (clipOpenRef.current) return;
const { autoMode: a, loopMode: l } = modeRef.current;
if (l === "one") return replayCurrent();
// A related video's end is not the queue's: the list beside it is where the next one is picked.
if (related) return;
const { queue: q, index: i } = navRef.current;
if (!q || q.length === 0) return;
const wrap = l === "all";
@@ -669,6 +749,35 @@ export default function PlayerModal({
// a tablet could not read a description at all. Anchored ABOVE, so it grows upward over the player
// rather than off-screen; `flip` still rescues it when there is no room up there.
const [showDesc, setShowDesc] = useState(false);
// The related videos, on demand (the owner's pick: the modal has little room to spare). Wide, they
// take the side panel's place of the description, a tab beside it; stacked, a drawer over the video
// opened from a button beside the ⓘ. The list is the OPENED video's: one played from it leaves the
// list in place, to try the next one (the owner's pick).
const [panelTab, setPanelTab] = useState<"description" | "related">("description");
const [relatedOpen, setRelatedOpen] = useState(false);
// The drawer is the stacked layout's: going wide while it is open, the list moves into the panel's
// tab instead. Left open, it reopened unasked when the window narrowed again.
if (wide && relatedOpen) {
setRelatedOpen(false);
setPanelTab("related");
}
const panelTabs = useTabPanel();
// Turned off on this instance (a daily limit of 0): no tab, no button — nothing that could only
// be refused.
const relatedOn = useMe().related_enabled;
const playRelated = (v: Video, startAt: number | null) => {
setRelatedOpen(false);
// The video already playing (its row stays in the list): a new player for the same id would
// never be built (one per video), so a Restart is a seek, and a plain pick just keeps playing.
// Unless a description link put another video in the player meanwhile (`navigated`): then the
// pick goes back to this one — the seek moved the linked video, and the pick did nothing.
if (v.id === subject.id) {
if (navigated) loadVideo(subject.id, startAt);
else if (startAt != null) seekTo(startAt);
return;
}
setRelated({ video: v, startAt });
};
const descPop = useAnchoredPopover({
open: showDesc,
onOpenChange: setShowDesc,
@@ -693,10 +802,10 @@ export default function PlayerModal({
// over the player by doing so.
const overlayOrder = useOverlayOrder();
// This player is a dismissable layer: register it so a Modal or popover opened ABOVE it (e.g. the
// download dialog, or an anchored menu) takes Escape first and this player defers.
const layerIdRef = useRef<number | null>(null);
// download dialog, or an anchored menu) takes Escape first and this player defers. Its root lets
// the keyboard walk a list inside it (the related videos).
useEffect(() => {
const id = pushLayer();
const id = pushLayer({ root: () => dialogRef.current });
layerIdRef.current = id;
return () => {
removeLayer(id);
@@ -705,6 +814,15 @@ export default function PlayerModal({
}, []);
useKeyScope("ytPlayer");
// Its keys wait under a layer above it (the drawer, a dialog — see onPlayerKey), so the help sheet
// opened there must not list them.
useEffect(() => {
setScopeCoveredProbe(
"ytPlayer",
() => layerIdRef.current != null && anyBlockingLayerAbove(layerIdRef.current),
);
return () => setScopeCoveredProbe("ytPlayer", null);
}, []);
// Keyboard shortcuts. The chords live in the shared keymap (`ytPlayer`
// scope) so the help sheet lists them; only the behaviour behind each id is here.
@@ -724,7 +842,9 @@ export default function PlayerModal({
// there: deferring would scroll something invisible and leave the viewer with no volume control at
// all. Both flags, not just fullscreen: since F stopped taking the browser's lock, maximise is the
// common case and checking only `isFullscreen` made the keys dead exactly where they matter most.
// On a row of a list in the player (the related videos) the arrows walk the list instead.
const volume = (e: KeyboardEvent, up: boolean) => {
if (focusInNavList()) return;
const covered = isFullscreenRef.current || maximized;
if (!covered && canScrollBy(document.activeElement as HTMLElement | null, up ? -1 : 1))
return;
@@ -732,6 +852,7 @@ export default function PlayerModal({
nudgeVolume(up ? 5 : -5);
};
const seek = (e: KeyboardEvent, secs: number) => {
if (focusInNavList()) return;
e.preventDefault();
nudgeSeek(secs);
};
@@ -743,6 +864,15 @@ export default function PlayerModal({
if (dialogRef.current) trapTab(e, dialogRef.current, cardRef.current);
return;
}
// A layer that owns the keyboard above the player (the related drawer, a menu, a dialog) has the
// keys: on its ✕ the arrows changed the volume and seeked the video behind it. Escape is the
// exception, and `close` decides for itself which layer it closes.
if (
e.key !== "Escape" &&
layerIdRef.current != null &&
anyBlockingLayerAbove(layerIdRef.current)
)
return;
dispatch(e, "ytPlayer", {
clip: () => {
if (clip) closeClip();
@@ -817,8 +947,9 @@ export default function PlayerModal({
// the end — recreated per active item. The once-bound event closures read fresh queue/mode/volume via
// the refs passed in. See lib/playerModalHooks.
useYtProgress({
activeId: active.id,
resumeAt,
activeId: subject.id,
resumeAt: subjectResume,
exactStart: related?.startAt != null,
mountRef,
playerRef,
currentIdRef,
@@ -1030,9 +1161,9 @@ export default function PlayerModal({
{/* Both work in fullscreen now: add-to-playlist's popover and download's Modal both
portal into the fullscreen element rather than <body> (a body-portaled node is
painted underneath the element holding the lock, i.e. invisible). */}
{!navigated && <AddToPlaylist videoId={active.id} variant="overlay" />}
{!navigated && <AddToPlaylist videoId={subject.id} variant="overlay" />}
{!navigated && (
<DownloadButton videoId={active.id} title={active.title} variant="overlay" />
<DownloadButton videoId={subject.id} title={subject.title} variant="overlay" />
)}
{/* Clip mode here: the clip panel takes this bar's place. */}
{playerError == null && (
@@ -1212,7 +1343,7 @@ export default function PlayerModal({
</div>
)}
{hasQueue && (
<div className="flex flex-wrap items-center justify-center gap-x-4 gap-y-1.5 px-4 py-2 border-b border-border text-sm">
<div className="flex flex-wrap items-center justify-center gap-x-2 gap-y-1.5 px-4 py-2 border-b border-border text-sm">
<PlayerTransportControls
variant="modal"
index={index}
@@ -1245,11 +1376,36 @@ export default function PlayerModal({
<Info />
</IconButton>
)}
{navigated && (
{!wide && relatedOn && (
<IconButton
variant="ghost"
onClick={() => setRelatedOpen(true)}
aria-expanded={relatedOpen}
label={t("player.related.title")}
>
<ListVideo />
</IconButton>
)}
{/* Inside the card, whose click stops at its edge: a portal's events still bubble
along the React tree, and from the backdrop a click on a row closed the player. */}
{relatedOpen && !wide && (
<Modal
title={t("player.related.title")}
onClose={() => setRelatedOpen(false)}
placement="drawer"
maxWidth="max-w-sm"
initialFocus="close"
>
<RelatedVideos videoId={active.id} onPlay={playRelated} onWrite={onWrite} />
</Modal>
)}
{(navigated || related) && (
<Button
variant="ghost"
size="sm"
onClick={() => loadVideo(active.id, null)}
// One step back: from a linked video to the one it was linked from, from a
// related video to the opened item.
onClick={() => (navigated ? loadVideo(subject.id, null) : setRelated(null))}
title={t("player.backToOriginal")}
icon={<ArrowLeft />}
>
@@ -1282,8 +1438,8 @@ export default function PlayerModal({
<div className="flex flex-wrap items-center gap-x-3 gap-y-2 mt-3">
{!navigated && (
<Avatar
src={active.channel_thumbnail}
fallback={active.channel_title ?? ""}
src={subject.channel_thumbnail}
fallback={subject.channel_title ?? ""}
className="w-9 h-9 rounded-full shrink-0"
/>
)}
@@ -1309,19 +1465,19 @@ export default function PlayerModal({
className={`flex items-center gap-1.5 min-w-0 ${wide ? "flex-1" : "shrink-0"}`}
>
<ChannelNameLink
channelId={active.channel_id}
title={active.channel_title}
channelId={subject.channel_id}
title={subject.channel_title}
onOpenInApp={() => {
// openChannel closes this player first and pushes only once its
// Back-to-close entry is reclaimed — otherwise that reclaim undoes the push
// and clicking the name merely closes the player
// (lib/history.closeOverlaysThenNavigate).
openChannel(active.channel_id, active.channel_title ?? undefined);
openChannel(subject.channel_id, subject.channel_title ?? undefined);
}}
className="font-medium hover:text-accent truncate text-left"
/>
<IconLink
href={active.channel_url}
href={subject.channel_url}
target="_blank"
rel="noreferrer"
label={t("channels.row.openOnYouTube")}
@@ -1335,27 +1491,27 @@ export default function PlayerModal({
<div
className={`flex flex-wrap items-center gap-x-2 gap-y-0.5 text-sm text-muted ${wide ? META_OWN_LINE : META_MIN}`}
>
{active.view_count != null && (
{subject.view_count != null && (
<span>
{metaSep}
{t("player.views", {
count: active.view_count,
formattedCount: formatCompact(active.view_count, lang),
count: subject.view_count,
formattedCount: formatCompact(subject.view_count, lang),
})}
</span>
)}
<span>
{metaSep}
{relativeTime(active.published_at, lang, t)}
{active.published_at && ` · ${fullDate(active.published_at)}`}
{relativeTime(subject.published_at, lang, t)}
{subject.published_at && ` · ${fullDate(subject.published_at)}`}
</span>
{active.duration_seconds != null && (
{subject.duration_seconds != null && (
<span>
{metaSep}
{formatDuration(active.duration_seconds)}
{formatDuration(subject.duration_seconds)}
</span>
)}
{active.live_status === "was_live" && (
{subject.live_status === "was_live" && (
<span className="bg-accent text-accent-fg text-3xs font-semibold uppercase tracking-wide px-1.5 py-0.5 rounded-sm">
{t("player.stream")}
</span>
@@ -1408,10 +1564,10 @@ export default function PlayerModal({
>
<Maximize />
</IconButton>
<AddToPlaylist videoId={active.id} variant="outline" size="md" />
<AddToPlaylist videoId={subject.id} variant="outline" size="md" />
<DownloadButton
videoId={active.id}
title={active.title}
videoId={subject.id}
title={subject.title}
variant="outline"
size="md"
/>
@@ -1428,7 +1584,16 @@ export default function PlayerModal({
<Scissors />
</IconButton>
)}
{onRemoveFromList && (
<IconButton
variant="outline"
size="md"
pressed={watched}
onClick={() => setWatched(!watched)}
label={watched ? t("player.watchedUnmark") : t("player.markWatched")}
>
{watched ? <CheckCheck /> : <Check />}
</IconButton>
{onRemoveFromList && !related && (
<IconButton
variant="outline-danger"
size="md"
@@ -1441,20 +1606,10 @@ export default function PlayerModal({
<Trash2 />
</IconButton>
)}
<Button
variant="outline"
size="sm"
pressed={watched}
onClick={() => setWatched(!watched)}
title={watched ? t("player.watchedUnmark") : t("player.markWatched")}
icon={watched ? <CheckCheck /> : <Check />}
>
{watched ? t("player.watched") : t("player.markWatched")}
</Button>
</div>
)}
</div>
{wide && (
{wide && !relatedOn && (
<section className="mt-5" data-testid="player-description">
<SectionLabel level={3} className="mb-2">
{t("player.description")}
@@ -1466,6 +1621,43 @@ export default function PlayerModal({
/>
</section>
)}
{wide && relatedOn && (
<div className="mt-5">
{/* The class replaces Tabs' own, so it keeps the row's layout and gap with
its tighter margin. */}
<Tabs
tabs={[
{ id: "description", label: t("player.description") },
{ id: "related", label: t("player.related.title") },
]}
active={panelTab}
onChange={(id) => setPanelTab(id === "related" ? "related" : "description")}
idBase={panelTabs.idBase}
label={t("player.panelTabs")}
className="flex flex-wrap gap-1 mb-3"
/>
<div {...panelTabs.panelProps(panelTab)}>
{panelTab === "description" ? (
<section data-testid="player-description">
<PlayerDescription
className="text-sm whitespace-pre-wrap wrap-break-word leading-relaxed"
clamp
{...descriptionProps}
/>
</section>
) : (
// Out into the panel's padding: in 20rem, every rem is the title's.
<div className="-mx-3">
<RelatedVideos
videoId={active.id}
onPlay={playRelated}
onWrite={onWrite}
/>
</div>
)}
</div>
</div>
)}
</div>
</div>
)}
@@ -0,0 +1,410 @@
// @vitest-environment jsdom
//
// A video's related videos, as the player shows them. What is held: a row says when the user follows
// its channel; the region the user picks is asked for and kept on the account; a video marked seen
// here leaves the list (and comes back when the server refuses the mark); reading further sends the
// cursor; and a failure says so rather than drawing an empty list.
import { afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
import { FloatingTree } from "@floating-ui/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
return {
...actual,
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
};
});
vi.mock("../navigation/navigationContext", () => ({
useNavigationActions: () => ({ openChannel: vi.fn() }),
}));
vi.mock("../video/AddToPlaylist", () => ({ default: () => null }));
import { RelatedVideos } from "./RelatedVideos";
import { api, HttpError, type RelatedVideo, type Video } from "../../lib/api";
import { clearAll, getNotifications } from "../../lib/notifications";
import { qk } from "../../lib/queryKeys";
import { setHintsEnabled } from "../../lib/hints";
import { writeVideoState } from "../video/videoWrites";
beforeAll(() => {
class Stub {
observe() {}
unobserve() {}
disconnect() {}
takeRecords() {
return [];
}
}
globalThis.ResizeObserver ??= Stub as unknown as typeof ResizeObserver;
globalThis.IntersectionObserver ??= Stub as unknown as typeof IntersectionObserver;
});
afterEach(() => {
cleanup();
vi.restoreAllMocks();
clearAll();
});
const row = (id: string, extra: Partial<RelatedVideo> = {}): RelatedVideo =>
({
id,
title: `Video ${id}`,
channel_id: `UC${id}`,
channel_title: `Channel ${id}`,
watch_url: `https://www.youtube.com/watch?v=${id}`,
status: "new",
saved: false,
position_seconds: 0,
followed: false,
...extra,
}) as RelatedVideo;
const page = (items: RelatedVideo[], next: string | null = null) => ({
items,
has_more: next != null,
next_cursor: next,
});
function setup(
onPlay: (v: Video, startAt: number | null) => void = () => {},
onWrite?: (write: Promise<unknown>) => void,
qc = new QueryClient({ defaultOptions: { queries: { retry: false } } }),
) {
vi.spyOn(api, "comfortPins").mockResolvedValue({ items: [] } as never);
vi.spyOn(api, "downloadIndex").mockResolvedValue({});
const saved = vi.spyOn(api, "savePrefs").mockResolvedValue({} as never);
render(
<QueryClientProvider client={qc}>
<FloatingTree>
<RelatedVideos videoId="SRC" onPlay={onPlay} onWrite={onWrite} />
</FloatingTree>
</QueryClientProvider>,
);
return { saved, qc };
}
// A row's title (its picture is a link of the same name).
const title = async (name: string) =>
(await screen.findAllByRole("link", { name })).find((a) => a.hasAttribute("data-nav-open"))!;
const titles = () =>
screen
.queryAllByRole("link")
.filter((a) => a.hasAttribute("data-nav-open"))
.map((a) => a.textContent);
// The menu's surface is visibility:hidden in jsdom: an item is matched on its text.
const menuItem = (label: string) =>
screen.queryAllByRole("menuitem", { hidden: true }).find((b) => b.textContent === label)!;
describe("RelatedVideos", () => {
it("marks the rows whose channel the user follows, and only those", async () => {
vi.spyOn(api, "related").mockResolvedValue(
page([row("A", { followed: true }), row("B"), row("C", { followed: true })]),
);
setup();
await title("Video A");
const marks = screen.getAllByRole("img", { name: "player.related.followed" });
const owners = marks.map((m) => m.closest("[data-video-id]")?.getAttribute("data-video-id"));
expect(owners).toEqual(["A", "C"]);
});
it("asks for the region picked, and keeps the pick on the account", async () => {
const asked = vi.spyOn(api, "related").mockResolvedValue(page([row("A")]));
const { saved } = setup();
await title("Video A");
expect(asked.mock.calls).toEqual([["SRC", "topical", null]]);
fireEvent.click(screen.getByRole("button", { name: "player.related.hu" }));
await waitFor(() => expect(asked.mock.calls).toContainEqual(["SRC", "hu", null]));
expect(saved).toHaveBeenCalledWith({ relatedRegion: "hu" });
});
it("sets the region switch at the toolbar rung, not the 10px caption one", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A")]));
setup();
await title("Video A");
const track = screen.getByRole("group", { name: "player.related.region" });
expect(track.className).toMatch(/\btext-xs\b/);
});
// A plain pick resumes (null: the player knows the freshest position); the row's own position
// travels on the video it hands over.
it("plays a row from where the user left it", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A", { position_seconds: 95 })]));
const onPlay = vi.fn();
setup(onPlay);
fireEvent.click(await title("Video A"));
expect(onPlay).toHaveBeenCalledWith(
expect.objectContaining({ id: "A", position_seconds: 95 }),
null,
);
});
it("drops a video hidden here at once, and brings it back when the server refuses", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A"), row("B")]));
let refuse: (e: Error) => void = () => {};
const write = vi
.spyOn(api, "setState")
.mockReturnValue(new Promise((_, reject) => (refuse = reject)) as never);
setup();
fireEvent.contextMenu((await title("Video A")).closest("[data-video-id]")!);
fireEvent.click(menuItem("card.hide"));
await waitFor(() => expect(write).toHaveBeenCalledWith("A", "hidden"));
expect(titles()).toEqual(["Video B"]);
await act(async () => refuse(new Error("offline")));
expect(titles()).toEqual(["Video A", "Video B"]);
});
it("keeps a video marked watched out of the list once the server has it", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A"), row("B")]));
vi.spyOn(api, "setState").mockResolvedValue(undefined as never);
setup();
fireEvent.contextMenu((await title("Video B")).closest("[data-video-id]")!);
fireEvent.click(menuItem("card.markWatched"));
await waitFor(() => expect(titles()).toEqual(["Video A"]));
});
it("reads further with the cursor the last page gave", async () => {
const asked = vi
.spyOn(api, "related")
.mockResolvedValueOnce(page([row("A")], "CUR1"))
.mockResolvedValueOnce(page([row("B")]));
setup();
await title("Video A");
fireEvent.click(screen.getByRole("button", { name: "player.related.more" }));
await title("Video B");
expect(asked.mock.calls[1]).toEqual(["SRC", "topical", "CUR1"]);
expect(screen.queryByRole("button", { name: "player.related.more" })).toBeNull();
});
it("says so when YouTube refuses, rather than drawing an empty list", async () => {
vi.spyOn(api, "related").mockRejectedValue(
new HttpError(422, "YouTube isn't reachable", undefined),
);
setup();
expect((await screen.findByRole("alert")).textContent).toContain("YouTube isn't reachable");
expect(screen.queryByText("player.related.empty")).toBeNull();
});
// A deferred answer, settled when the test says.
const deferred = () => {
let resolve: () => void = () => {};
let reject: (e: Error) => void = () => {};
const promise = new Promise<never>((res, rej) => {
resolve = res as () => void;
reject = rej;
});
return { promise, resolve, reject };
};
it("keeps a row hidden while its hide is on its way, though an earlier save landed", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A"), row("B")]));
const save = deferred();
const saving = vi.spyOn(api, "watchLaterAdd").mockReturnValue(save.promise as never);
const hide = deferred();
const hiding = vi.spyOn(api, "setState").mockReturnValue(hide.promise as never);
setup();
const card = (await title("Video A")).closest("[data-video-id]")!;
fireEvent.contextMenu(card);
fireEvent.click(menuItem("card.saveForLater"));
await waitFor(() => expect(saving).toHaveBeenCalledWith("A"));
fireEvent.contextMenu(card);
fireEvent.click(menuItem("card.hide"));
await waitFor(() => expect(hiding).toHaveBeenCalledWith("A", "hidden"));
expect(titles()).toEqual(["Video B"]);
await act(async () => save.resolve());
expect(titles()).toEqual(["Video B"]);
});
it("announces a hide with the video it names, for the notification center to find", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A")]));
vi.spyOn(api, "setState").mockResolvedValue(undefined as never);
const writes: Promise<unknown>[] = [];
setup(undefined, (w) => void writes.push(w));
fireEvent.contextMenu((await title("Video A")).closest("[data-video-id]")!);
fireEvent.click(menuItem("card.hide"));
await waitFor(() => expect(getNotifications().length).toBeGreaterThan(0));
expect(getNotifications().map((n) => n.meta)).toContainEqual({
kind: "video-hidden",
videoId: "A",
title: "Video A",
channelId: "UCA",
channelName: "Channel A",
});
// And the write reached the host's session.
expect(writes).toHaveLength(1);
});
it("flips a row's bookmark at once, and back when the server refuses", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A")]));
const save = deferred();
vi.spyOn(api, "watchLaterAdd").mockReturnValue(save.promise as never);
setup();
const card = (await title("Video A")).closest("[data-video-id]")!;
// What the menu offers is the row's state: un-save once saved.
const offers = (label: string) => {
fireEvent.contextMenu(card);
return !!menuItem(label);
};
expect(offers("card.saveForLater")).toBe(true);
fireEvent.click(menuItem("card.saveForLater"));
await waitFor(() => expect(offers("card.menu.unsave")).toBe(true));
fireEvent.keyDown(document.activeElement ?? document.body, { key: "Escape" });
await act(async () => save.reject(new Error("offline")));
await waitFor(() => expect(offers("card.saveForLater")).toBe(true));
});
it("is kept for the session, not collected when the list closes", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A")]));
const { qc } = setup();
await title("Video A");
const [cached] = qc.getQueryCache().findAll({ queryKey: qk.related() });
expect(cached!.options.gcTime).toBe(Infinity);
});
it("says the next page failed under the rows it has, and retries that page", async () => {
const asked = vi
.spyOn(api, "related")
.mockResolvedValueOnce(page([row("A")], "CUR1"))
.mockRejectedValueOnce(new HttpError(422, "YouTube isn't reachable", undefined))
.mockResolvedValueOnce(page([row("B")]));
setup();
await title("Video A");
fireEvent.click(screen.getByRole("button", { name: "player.related.more" }));
expect((await screen.findByRole("alert")).textContent).toContain("YouTube isn't reachable");
expect(titles()).toEqual(["Video A"]);
fireEvent.click(screen.getByRole("button", { name: "common.retry" }));
await title("Video B");
expect(asked.mock.calls.map((c) => c[2])).toEqual([null, "CUR1", "CUR1"]);
});
it("with nothing unseen read so far, says so above Show more, and retries the page it failed", async () => {
const asked = vi
.spyOn(api, "related")
.mockResolvedValueOnce(page([], "CUR1"))
.mockRejectedValueOnce(new HttpError(422, "YouTube isn't reachable", undefined))
.mockResolvedValueOnce(page([row("B")]));
setup();
expect(await screen.findByText("player.related.emptySoFar")).toBeTruthy();
expect(screen.queryByText("player.related.empty")).toBeNull();
fireEvent.click(screen.getByRole("button", { name: "player.related.more" }));
await screen.findByRole("alert");
fireEvent.click(screen.getByRole("button", { name: "common.retry" }));
await title("Video B");
expect(asked.mock.calls.map((c) => c[2])).toEqual([null, "CUR1", "CUR1"]);
});
it.each([
["the day's cap", { code: "refused", reason: "related_daily_limit", limit: 5 }],
["turned off", { code: "refused", reason: "related_disabled" }],
["the shared budget", { code: "youtube", reason: "quota" }],
])("offers no retry a retry cannot answer: %s", async (_name, detail) => {
vi.spyOn(api, "related").mockRejectedValue(new HttpError(429, "Refused", detail));
setup();
expect((await screen.findByRole("alert")).textContent).toContain("Refused");
expect(screen.queryByRole("button", { name: "common.retry" })).toBeNull();
});
it("offers a retry where trying again can help", async () => {
vi.spyOn(api, "related").mockRejectedValue(
new HttpError(422, "YouTube isn't reachable", { code: "youtube", reason: "unavailable" }),
);
setup();
await screen.findByRole("alert");
expect(screen.getByRole("button", { name: "common.retry" })).toBeTruthy();
});
it("names the followed mark on hover", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A", { followed: true })]));
setup();
await title("Video A");
const mark = screen.getByRole("img", { name: "player.related.followed" });
fireEvent.mouseEnter(mark.parentElement!);
fireEvent.pointerEnter(mark.parentElement!, { pointerType: "mouse" });
expect(await screen.findByText("player.related.followed")).toBeTruthy();
});
// An infinite query builds its next page from the pages it read when that fetch started: a mark
// that landed in between was dropped with them, for the session.
it("keeps a hide that landed while Show more was on its way", async () => {
let land: (p: ReturnType<typeof page>) => void = () => {};
vi.spyOn(api, "related")
.mockResolvedValueOnce(page([row("A"), row("C")], "CUR1"))
.mockReturnValueOnce(new Promise((r) => (land = r)) as never);
const write = vi.spyOn(api, "setState").mockResolvedValue({ ok: true } as never);
setup();
await title("Video A");
fireEvent.click(screen.getByRole("button", { name: "player.related.more" }));
fireEvent.contextMenu((await title("Video A")).closest("[data-video-id]")!);
fireEvent.click(menuItem("card.hide"));
await waitFor(() => expect(write).toHaveBeenCalledWith("A", "hidden"));
await act(async () => {});
await act(async () => land(page([row("B")])));
await title("Video B");
expect(titles()).toEqual(["Video C", "Video B"]);
});
// Hidden in this session, then un-hidden on another device: a list read after the mark carries
// the server's word, and the mark does not paint over it.
it("shows a list read after a mark as the server has it", async () => {
vi.spyOn(api, "setState").mockResolvedValue({ ok: true } as never);
vi.spyOn(api, "related").mockResolvedValue(page([row("A"), row("B")]));
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
await writeVideoState(qc, "A", "hidden");
await new Promise((r) => setTimeout(r, 5)); // the list is read a moment after the mark
setup(undefined, undefined, qc);
await title("Video B");
expect(titles()).toEqual(["Video A", "Video B"]);
});
// The Notification Center's undo, the feed's reset, a card elsewhere: every write reaches it.
it("follows a video's state written anywhere else", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([row("A"), row("B")]));
vi.spyOn(api, "setState").mockResolvedValue({ ok: true } as never);
const { qc } = setup();
await title("Video A");
await act(async () => void (await writeVideoState(qc, "A", "hidden")));
await waitFor(() => expect(titles()).toEqual(["Video B"]));
await act(async () => void (await writeVideoState(qc, "A", "new")));
await waitFor(() => expect(titles()).toEqual(["Video A", "Video B"]));
});
// Show more is the request Retry would make: refused for good, it is not offered either.
it("offers no Show more once the next page was refused for good", async () => {
vi.spyOn(api, "related")
.mockResolvedValueOnce(page([row("A")], "CUR1"))
.mockRejectedValueOnce(
new HttpError(429, "Limit", { code: "refused", reason: "related_daily_limit", limit: 3 }),
);
setup();
await title("Video A");
fireEvent.click(screen.getByRole("button", { name: "player.related.more" }));
expect((await screen.findByRole("alert")).textContent).toContain("Limit");
expect(screen.queryByRole("button", { name: "player.related.more" })).toBeNull();
expect(screen.queryByRole("button", { name: "common.retry" })).toBeNull();
expect(titles()).toEqual(["Video A"]);
});
it("names the followed mark on hover with hints turned off too", async () => {
setHintsEnabled(false);
try {
vi.spyOn(api, "related").mockResolvedValue(page([row("A", { followed: true })]));
setup();
await title("Video A");
const mark = screen.getByRole("img", { name: "player.related.followed" });
fireEvent.mouseEnter(mark.parentElement!);
fireEvent.pointerEnter(mark.parentElement!, { pointerType: "mouse" });
expect(await screen.findByText("player.related.followed")).toBeTruthy();
} catch (e) {
setHintsEnabled(true);
throw e;
}
setHintsEnabled(true);
});
it("says when nothing is left that the user has not seen", async () => {
vi.spyOn(api, "related").mockResolvedValue(page([]));
setup();
expect(await screen.findByText("player.related.empty")).toBeTruthy();
expect(screen.queryByRole("alert")).toBeNull();
});
});
@@ -0,0 +1,236 @@
import { useRef, useState } from "react";
import { useTranslation } from "react-i18next";
import { useInfiniteQuery, useQueryClient } from "@tanstack/react-query";
import { CircleCheck } from "lucide-react";
import {
api,
HttpError,
refusalReason,
type RelatedRegion,
type Video,
type VideoStatus,
} from "../../lib/api";
import { qk } from "../../lib/queryKeys";
import { nextPageOnClick } from "../../lib/nextPage";
import { usePref, type UsePrefOptions } from "../../lib/usePref";
import { markedRows, stampFetch, useVideoMarks } from "../../lib/progressCache";
import { notifyActionError } from "../../lib/actionErrors";
import { Button } from "../../components/ui/Button";
import SegmentedToggle from "../../components/ui/SegmentedToggle";
import { LoadingState, StateMessage } from "../../components/ui/QueryState";
import { useContextMenu } from "../../components/ui/useContextMenu";
import NavGrid from "../../components/ui/NavGrid";
import Tooltip from "../../components/ui/Tooltip";
import VideoCard from "../video/VideoCard";
import { VideoMenu } from "../video/VideoMenu";
import { announceVideoState } from "../video/stateNotice";
import { writeVideoState, writeWatchLater } from "../video/videoWrites";
/** Which list YouTube fills in — on the video's topic or with what trends in Hungary — is a mood,
* decided at the list, and the account keeps the last one (the owner's pick). */
const RELATED_REGION_PREF = "relatedRegion";
const RELATED_REGION_OPTIONS: UsePrefOptions<RelatedRegion> = {
parse: (raw) => (raw === "hu" ? "hu" : "topical"),
};
// The list leaves out what the user has seen (the server does, as it fetches); a video marked here
// leaves it the same way, at once.
const isSeen = (v: Video) => v.status === "watched" || v.status === "hidden";
/** The refusals a retry cannot answer: the day's cap (tomorrow does), the feature turned off on
* this instance and the shared budget spent (the admin, or tomorrow). */
const FINAL_REASONS = new Set(["related_daily_limit", "related_disabled", "quota"]);
const isFinal = (e: unknown) => FINAL_REASONS.has(refusalReason(e) ?? "");
// A write not yet confirmed: its change shows at once, and leaves when the server answers.
type Pending = { seq: number; id: string; change: Partial<Video> };
/** The videos YouTube shows beside `videoId`, without the ones the user has watched or hidden. A
* click plays one in this player (`onPlay`); the list stays the opened video's while it does. Its
* rows carry the feed's whole menu. */
export function RelatedVideos({
videoId,
onPlay,
onWrite,
}: {
videoId: string;
/** `startAt` null resumes the video where it was left; a number (Restart's 0) starts there. */
onPlay: (v: Video, startAt: number | null) => void;
/** Each write this list makes, for the host's player session (see PlayerModal `onWrite`). */
onWrite?: (write: Promise<unknown>) => void;
}) {
const { t } = useTranslation();
const qc = useQueryClient();
const [region, setRegion] = usePref(RELATED_REGION_PREF, "topical", RELATED_REGION_OPTIONS);
// Kept for the session — never refetched, never collected while the app is open: the list
// unmounts whenever the drawer closes or the tab changes, and a refetch is another scrape and
// another slot of the day's cap.
const query = useInfiniteQuery({
queryKey: qk.related(videoId, region),
queryFn: ({ pageParam }) => stampFetch(() => api.related(videoId, region, pageParam)),
initialPageParam: null as string | null,
getNextPageParam: (last) => (last.has_more ? last.next_cursor : undefined),
staleTime: Infinity,
gcTime: Infinity,
retry: false,
});
// The marks on their way, one entry per WRITE: a row leaves (or its bookmark flips) at once, and
// comes back if the server refuses that write. Kept per video, the first answer to land cleared
// the video's whole entry, and a save then a hide put the hidden row back until the hide landed.
const [pending, setPending] = useState<Pending[]>([]);
const seqRef = useRef(0);
const begin = (id: string, change: Partial<Video>) => {
const seq = ++seqRef.current;
setPending((p) => [...p, { seq, id, change }]);
return seq;
};
const settle = (seq: number) => setPending((p) => p.filter((w) => w.seq !== seq));
const onState = (id: string, status: VideoStatus) => {
const v = rows.find((x) => x.id === id);
const seq = begin(id, { status });
const write = writeVideoState(qc, id, status);
onWrite?.(write);
write
.then(() => {
settle(seq);
// Stale-mark the feed, as its own marks do: the video may be in it too.
void qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" });
announceVideoState(id, status, v, () => onState(id, "new"));
})
.catch((e: unknown) => {
notifyActionError(e, t("errors.writeFailed.videoState"));
settle(seq);
});
};
const onToggleSave = (id: string, saved: boolean) => {
const seq = begin(id, { saved });
const write = writeWatchLater(qc, id, saved);
onWrite?.(write);
write
.then(() => {
settle(seq);
void qc.invalidateQueries({ queryKey: qk.playlists() });
void qc.invalidateQueries({ queryKey: qk.playlist() });
})
.catch((e: unknown) => {
notifyActionError(
e,
t(saved ? "errors.writeFailed.watchLaterAdd" : "errors.writeFailed.watchLaterRemove"),
);
settle(seq);
});
};
const open = (v: Video, startAt?: number | null) => onPlay(v, startAt ?? null);
const menu = useContextMenu<Video>();
const openId = menu.open ? menu.target?.id : undefined;
// What this session confirmed about a row since its page was read (here or anywhere else), then
// what is on its way.
const marks = useVideoMarks();
const rows = markedRows(query.data?.pages, marks)
.map((v) => pending.reduce((acc, w) => (w.id === v.id ? { ...acc, ...w.change } : acc), v))
.filter((v) => !isSeen(v));
const followed = new Set(
(query.data?.pages ?? []).flatMap((p) => p.items.filter((v) => v.followed).map((v) => v.id)),
);
const error =
query.error instanceof HttpError
? query.error.detail || t("player.related.error")
: query.isError
? t("player.related.error")
: null;
// A refused page is the one to try again: with pages loaded, a refetch started over from the first
// (another scrape and another slot) instead of asking for the page that failed.
const final = isFinal(query.error);
const retry = final
? undefined
: query.data?.pages.length
? () => void query.fetchNextPage(nextPageOnClick(query.isFetchingNextPage))
: () => void query.refetch();
return (
<section data-testid="player-related" aria-label={t("player.related.title")}>
<SegmentedToggle
options={[
{ value: "topical", label: t("player.related.topical") },
{ value: "hu", label: t("player.related.hu") },
]}
value={region}
onChange={setRegion}
ariaLabel={t("player.related.region")}
fullWidth
className="mb-3"
/>
{query.isPending ? (
<LoadingState shape="list" />
) : error && !rows.length ? (
<StateMessage tone="error" compact onRetry={retry}>
{error}
</StateMessage>
) : !rows.length && !query.hasNextPage ? (
<StateMessage compact>{t("player.related.empty")}</StateMessage>
) : (
<>
{/* What was read so far was all seen (or marked here since): the list goes on, it is not
empty — said, so Show more is not a lone button under nothing. */}
{!rows.length && <StateMessage compact>{t("player.related.emptySoFar")}</StateMessage>}
<NavGrid className="flex flex-col gap-1.5">
{rows.map((v, i) => (
<VideoCard
key={v.id}
navIndex={i}
video={v}
view="rows"
narrow
width={0}
onState={onState}
onToggleSave={onToggleSave}
onOpen={open}
menuProps={menu.triggerProps}
menuButtonProps={menu.buttonProps}
menuOpen={openId === v.id}
channelMark={
followed.has(v.id) && (
// A bare mark means nothing to a mouse: it names itself on hover, whatever
// the hints setting, since the caption is all it has.
<Tooltip hint={t("player.related.followed")} always>
<CircleCheck
className="w-3.5 h-3.5 shrink-0 text-success"
aria-label={t("player.related.followed")}
role="img"
/>
</Tooltip>
)
}
/>
))}
</NavGrid>
{/* Show more is the same request a Retry makes: refused for good, neither is offered. */}
{query.hasNextPage && !final && (
<div className="mt-3 flex justify-center">
<Button
variant="outline"
size="sm"
disabled={query.isFetchingNextPage}
onClick={() => void query.fetchNextPage(nextPageOnClick(query.isFetchingNextPage))}
>
{query.isFetchingNextPage ? t("player.related.loading") : t("player.related.more")}
</Button>
</div>
)}
{error && rows.length > 0 && (
<StateMessage tone="error" compact onRetry={retry}>
{error}
</StateMessage>
)}
</>
)}
<VideoMenu menu={menu} handlers={{ onState, onToggleSave, onOpen: open }} />
</section>
);
}
@@ -0,0 +1,74 @@
// @vitest-environment jsdom
//
// The queue transport cluster, in both of its places. The auto-advance mode wears glyphs of its own:
// on the side panel's glyph-only row the skip arrows are Previous / Next, and a mode that wore them
// too put two identical ⏭ side by side, the second one silently changing a saved setting. The modes
// are cycles, not switches (lit, no aria-pressed), and every control's tooltip names it and its mode
// in both places. i18n is key-passthrough.
import { afterEach, describe, expect, it, vi } from "vitest";
import { cleanup, render, screen } from "@testing-library/react";
vi.mock("react-i18next", async (importOriginal) => ({
...(await importOriginal<typeof import("react-i18next")>()),
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
}));
import { PlayerTransportControls } from "./playerModalControls";
import { AUTO_MODES, LOOP_MODES, type AutoMode, type LoopMode } from "../../lib/playerModes";
afterEach(cleanup);
function mount(variant: "modal" | "docked", autoMode: AutoMode, loopMode: LoopMode = "off") {
render(
<PlayerTransportControls
variant={variant}
index={1}
queueLength={3}
autoMode={autoMode}
loopMode={loopMode}
onPrev={() => {}}
onNext={() => {}}
onCycleAuto={() => {}}
onCycleLoop={() => {}}
/>,
);
}
const glyph = (el: HTMLElement) => el.querySelector("svg")!.getAttribute("class");
const button = (name: string | RegExp) => screen.getByRole("button", { name });
describe("PlayerTransportControls", () => {
for (const variant of ["modal", "docked"] as const)
for (const mode of AUTO_MODES)
it(`${variant}: auto-advance ${mode} wears neither Previous's glyph nor Next's`, () => {
mount(variant, mode);
const auto = glyph(button(/^player\.autoAdvance\.label: /));
expect(auto).not.toBe(glyph(button("player.previous")));
expect(auto).not.toBe(glyph(button("player.next")));
});
for (const variant of ["modal", "docked"] as const)
it(`${variant}: the modes are lit when on, and announce no toggle`, () => {
mount(variant, "next", "off");
const auto = button(/^player\.autoAdvance\.label: /);
const loop = button(/^player\.loop\.label: /);
expect(auto.hasAttribute("aria-pressed")).toBe(false);
expect(loop.hasAttribute("aria-pressed")).toBe(false);
expect(auto.hasAttribute("data-lit")).toBe(true);
expect(loop.hasAttribute("data-lit")).toBe(false);
});
for (const variant of ["modal", "docked"] as const)
it(`${variant}: each mode's tooltip names it and the mode it is in`, () => {
for (const loopMode of LOOP_MODES) {
mount(variant, "random", loopMode);
expect(button(/^player\.autoAdvance\.label: /).title).toBe(
"player.autoAdvance.label: player.autoAdvance.random · player.autoAdvance.hint",
);
expect(button(/^player\.loop\.label: /).title).toBe(
`player.loop.label: player.loop.${loopMode} · player.loop.hint`,
);
cleanup();
}
});
});
@@ -1,9 +1,10 @@
import type { CSSProperties, Ref } from "react";
import type { CSSProperties, ReactNode, Ref } from "react";
import { useTranslation } from "react-i18next";
import { Repeat, Repeat1, Shuffle, SkipBack, SkipForward } from "lucide-react";
import { ListEnd, ListStart, Repeat, Repeat1, Shuffle, SkipBack, SkipForward } from "lucide-react";
import type { VideoDetail } from "../../lib/api";
import type { AutoMode, LoopMode } from "../../lib/playerModes";
import { Button } from "../../components/ui/Button";
import { IconButton } from "../../components/ui/IconButton";
import { StateMessage } from "../../components/ui/QueryState";
import { ClampedText } from "../../components/ui/ClampedText";
import { renderDescription } from "./descriptionLinks";
@@ -12,11 +13,16 @@ import { renderDescription } from "./descriptionLinks";
// the sibling of plexPlayerControls.tsx. No state or effects here; all behaviour comes in via props.
// The auto-advance / loop modes it renders are defined in `lib/playerModes`.
/** The queue transport cluster — prev / counter / next, then the auto-advance and loop mode toggles.
* Rendered in two places with identical logic: the modal's queue bar (`variant:"modal"`, normal glass
* theme, labels always shown) and the big-view docked control bar (`variant:"docked"`, white-on-dark,
* labels hidden until a breakpoint). Only the theme + label-visibility differ; the behaviour is one,
* so a mode/icon change lands once. The wrapper (fragment vs. bordered row) stays at the call site. */
/** The queue transport cluster — prev / counter / next, then the auto-advance and loop modes.
* Rendered in two places from ONE list of controls: the modal's queue bar (`variant:"modal"`, normal
* glass theme, glyphs only so the whole cluster fits one line of the side panel) and the big-view
* docked control bar (`variant:"docked"`, white-on-dark, labels shown from a breakpoint up). What a
* control does, its glyph and its tooltip are said once; only the primitive and the label's
* visibility differ. Every control's tooltip carries its full name (and the shortcut, where it has
* one), so a glyph-only one still says what it does and which mode it is in. The modes are cycles,
* not switches: `lit` when not off, and the name says which. Auto-advance wears list glyphs of its
* own — the skip arrows are Previous / Next, beside it (DESIGN.md "A glyph means one thing"). The
* wrapper (fragment vs. bordered row) stays at the call site. */
export function PlayerTransportControls({
variant,
index,
@@ -40,69 +46,105 @@ export function PlayerTransportControls({
}) {
const { t } = useTranslation();
const docked = variant === "docked";
// The docked bar sits on the video (always dark): the overlay variant; the modal's is a page surface.
const chrome = docked ? "overlay" : "ghost";
const navLabel = docked ? "hidden sm:inline" : "";
const counter = docked ? "tabular-nums text-white/70" : "text-muted tabular-nums";
const divider = docked ? "h-4 w-px bg-white/20" : "w-px h-4 bg-border";
const toggleLabel = docked ? "hidden md:inline" : "";
const autoName = `${t("player.autoAdvance.label")}: ${t(`player.autoAdvance.${autoMode}`)}`;
const loopName = `${t("player.loop.label")}: ${t(`player.loop.${loopMode}`)}`;
type Control = {
key: string;
glyph: ReactNode;
name: string;
title: string;
onClick: () => void;
disabled?: boolean;
lit?: boolean;
/** From which width the docked bar shows the name beside the glyph. */
labelFrom: "sm" | "md";
/** The glyph after the name (Next points the way it goes). */
trailing?: boolean;
};
const step: Control[] = [
{
key: "prev",
glyph: <SkipBack />,
name: t("player.previous"),
title: `${t("player.previous")} · Shift+←`,
onClick: onPrev,
disabled: index === 0,
labelFrom: "sm",
},
{
key: "next",
glyph: <SkipForward />,
name: t("player.next"),
title: `${t("player.next")} · Shift+→`,
onClick: onNext,
disabled: index === queueLength - 1,
labelFrom: "sm",
trailing: true,
},
];
// Persistent playback settings — cycle on click, saved to your account.
const modes: Control[] = [
{
key: "auto",
glyph:
autoMode === "prev" ? <ListStart /> : autoMode === "random" ? <Shuffle /> : <ListEnd />,
name: autoName,
title: `${autoName} · ${t("player.autoAdvance.hint")}`,
onClick: onCycleAuto,
lit: autoMode !== "off",
labelFrom: "md",
},
{
key: "loop",
glyph: loopMode === "one" ? <Repeat1 /> : <Repeat />,
name: loopName,
title: `${loopName} · ${t("player.loop.hint")}`,
onClick: onCycleLoop,
lit: loopMode !== "off",
labelFrom: "md",
},
];
const render = (c: Control) =>
docked ? (
<Button
key={c.key}
variant="overlay"
size="sm"
icon={c.trailing ? undefined : c.glyph}
onClick={c.onClick}
disabled={c.disabled}
lit={c.lit}
label={c.name}
title={c.title}
>
<span className={c.labelFrom === "sm" ? "hidden sm:inline" : "hidden md:inline"}>
{c.name}
</span>
{c.trailing && c.glyph}
</Button>
) : (
<IconButton
key={c.key}
onClick={c.onClick}
disabled={c.disabled}
lit={c.lit}
label={c.name}
title={c.title}
>
{c.glyph}
</IconButton>
);
return (
<>
<Button
variant={chrome}
size="sm"
icon={<SkipBack />}
onClick={onPrev}
disabled={index === 0}
label={t("player.previous")}
title={`${t("player.previous")} · Shift+←`}
>
<span className={navLabel}>{t("player.previous")}</span>
</Button>
{render(step[0]!)}
<span className={counter}>
{index + 1} / {queueLength}
</span>
<Button
variant={chrome}
size="sm"
onClick={onNext}
disabled={index === queueLength - 1}
label={t("player.next")}
title={`${t("player.next")} · Shift+→`}
>
<span className={navLabel}>{t("player.next")}</span>
<SkipForward />
</Button>
{render(step[1]!)}
<span className={divider} />
{/* Persistent playback settings — cycle on click, saved to your account. */}
<Button
variant={chrome}
size="sm"
pressed={autoMode !== "off"}
icon={
autoMode === "prev" ? <SkipBack /> : autoMode === "random" ? <Shuffle /> : <SkipForward />
}
onClick={onCycleAuto}
label={`${t("player.autoAdvance.label")}: ${t(`player.autoAdvance.${autoMode}`)}`}
title={t("player.autoAdvance.hint")}
>
<span className={toggleLabel}>
{t("player.autoAdvance.label")}: {t(`player.autoAdvance.${autoMode}`)}
</span>
</Button>
<Button
variant={chrome}
size="sm"
pressed={loopMode !== "off"}
icon={loopMode === "one" ? <Repeat1 /> : <Repeat />}
onClick={onCycleLoop}
label={`${t("player.loop.label")}: ${t(`player.loop.${loopMode}`)}`}
title={t("player.loop.hint")}
>
<span className={toggleLabel}>
{t("player.loop.label")}: {t(`player.loop.${loopMode}`)}
</span>
</Button>
{modes.map(render)}
</>
);
}
@@ -2,7 +2,7 @@ import { useLayoutEffect, useState } from "react";
import { useTranslation } from "react-i18next";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { qk } from "../../lib/queryKeys";
import { api, HttpError, type PlexCollection } from "../../lib/api";
import { api, refusalReason, type PlexCollection } from "../../lib/api";
import { notifyActionError } from "../../lib/actionErrors";
import { PopoverSurface } from "../../components/ui/AnchoredPopover";
import { useAnchoredPopover } from "../../components/ui/anchoredPopoverHooks";
@@ -123,8 +123,7 @@ export default function PlexCollectionAddAt({
/** A create refused only because the read after it failed: the collection is on Plex, with this
* title in it, and its row is kept — the name is taken now, so the field clears as on success. */
function madeOnPlex(e: unknown): boolean {
const d = e instanceof HttpError ? (e.detailData as { reason?: string } | undefined) : undefined;
return d?.reason === "collection_pending_sync";
return refusalReason(e) === "collection_pending_sync";
}
function flipContains<T extends { collections: { id: string; contains: boolean }[] }>(
@@ -1,7 +1,7 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { useQueryClient } from "@tanstack/react-query";
import { Check, Info, Layers, ListPlus, Play, SkipBack, Tv2, Undo2 } from "lucide-react";
import { Check, Info, Layers, ListPlus, Play, RotateCcw, Tv2, Undo2 } from "lucide-react";
import { type PlexCard } from "../../lib/api";
import { qk } from "../../lib/queryKeys";
@@ -98,7 +98,7 @@ export function usePlexCardActions(handlers: PlexActionHandlers, open: boolean)
{
id: "restart",
label: t("card.restartTitle"),
icon: <SkipBack className={icon} />,
icon: <RotateCcw className={icon} />,
onSelect: () => onRestart(c),
},
]
@@ -5,6 +5,7 @@ import { qk } from "../../lib/queryKeys";
import { ListPlus } from "lucide-react";
import { YouTubeIcon } from "../../lib/brandIcons";
import { api, type Playlist } from "../../lib/api";
import { addPlaylistItem, removePlaylistItem } from "./videoWrites";
import { notifyActionError } from "../../lib/actionErrors";
import { playlistName } from "../../lib/playlistName";
import { PopoverSurface } from "../../components/ui/AnchoredPopover";
@@ -115,8 +116,8 @@ function MembershipPanel({
if (busy) return;
setBusy(true);
try {
if (pl.has_video) await api.removeFromPlaylist(pl.id, videoId);
else await api.addToPlaylist(pl.id, videoId);
if (pl.has_video) await removePlaylistItem(qc, pl, videoId);
else await addPlaylistItem(qc, pl, videoId);
qc.setQueryData<Playlist[]>(qk.playlistMembership(videoId), (old) =>
old?.map((p) => (p.id === pl.id ? { ...p, has_video: !p.has_video } : p)),
);
@@ -133,7 +134,7 @@ function MembershipPanel({
let made = false;
try {
const pl = await api.createPlaylist(name);
await api.addToPlaylist(pl.id, videoId);
await addPlaylistItem(qc, pl, videoId);
made = true;
} catch (e) {
notifyActionError(e, t("playlists.addFailed"));
+41 -2
View File
@@ -374,6 +374,8 @@ function VideoCard({
menuOpen = false,
navIndex,
shelf = false,
narrow = false,
channelMark,
}: {
video: Video;
view: FeedView;
@@ -397,6 +399,12 @@ function VideoCard({
navIndex?: number;
/** On a shelf (a horizontal strip): a one-line title over the channel alone (ui/mediaCardStyles). */
shelf?: boolean;
/** The thumbnailed row in a side panel's ~20rem column: a smaller picture, and the "…" alone at the
* row's end — the hover toggles leave no room for the title there, and every one of them is a row
* of the menu. */
narrow?: boolean;
/** Something the list says about the channel (a "followed" mark), set before its name. */
channelMark?: ReactNode;
}) {
const { t, i18n } = useTranslation();
const lang = i18n.language;
@@ -610,15 +618,46 @@ function VideoCard({
const textBlock = (
<>
{title}
<div className="mt-0.5">{channelLink}</div>
<div className="mt-0.5">
{channelMark ? (
<div className="flex items-center gap-1 min-w-0">
{channelMark}
<div className="min-w-0">{channelLink}</div>
</div>
) : (
channelLink
)}
</div>
<div className="text-xs text-muted mt-0.5">
{views}
{views && " · "}
{published}
{/* The narrow row keeps the relative time and moves the date into its tooltip, as the small
card does: in a 20rem panel the full pair wrapped to a line of its own. */}
{narrow ? <RelativeTime at={video.published_at} /> : published}
</div>
</>
);
if (spec.family === "row" && narrow) {
return (
<div
data-testid="video-card"
data-video-id={video.id}
{...menuProps?.(video)}
{...(navIndex !== undefined && navItemAttrs(navIndex))}
className={clsx(rowShell, "flex gap-2 p-1.5", watched && "opacity-55")}
>
<Thumb video={video} className="aspect-video shrink-0 w-24" onOpen={onOpen} small />
<div className="min-w-0 flex-1 flex flex-col">{textBlock}</div>
{more && (
<div className="shrink-0 self-start">
<MoreButton label={t("card.menu.more")} expanded={more.open} menuProps={more.props} />
</div>
)}
</div>
);
}
if (spec.family === "row") {
return (
<div
@@ -2,7 +2,7 @@ import { useTranslation } from "react-i18next";
import { type Video } from "../../lib/api";
import { ContextMenu } from "../../components/ui/ContextMenu";
import { type ContextMenuController } from "../../components/ui/useContextMenu";
import { useVideoActions, type VideoActionHandlers } from "../../features/video/useVideoActions";
import { useVideoActions, type VideoActionHandlers } from "./useVideoActions";
/** A video list's one menu (the feed grid and a channel page): every trigger — right-click,
* long-press, Shift+F10 and each card's "…" — opens it with the video's whole action list. Render
@@ -0,0 +1,38 @@
import i18n from "../../i18n";
import { notify, resolveVideo } from "../../lib/notifications";
import type { Video, VideoStatus } from "../../lib/api";
/** What a notice names the video by; any of it may be unknown to the surface that marked it. */
export type NoticeVideo = Pick<Video, "title" | "channel_id" | "channel_title">;
/** Say that a video's mark landed, once the server has it: "Hidden" / "Marked watched" with the
* Undo (`undo` puts it back to new) and the meta the Notification Center keeps it under, so the
* notice still finds the video after a reload. A mark back to new quietly resolves a hide or watch
* notice still standing for it, so it does not linger with a dead action. A claim of success:
* never called before the write is confirmed. One definition, because every list that marks a
* video must announce it the same way (the feed's cards, a related list). */
export function announceVideoState(
id: string,
status: VideoStatus,
video: NoticeVideo | undefined,
undo: () => void,
): void {
if (status === "new") {
resolveVideo(id);
return;
}
const hidden = status === "hidden";
notify({
message: video?.title
? i18n.t(hidden ? "feed.hiddenNamed" : "feed.markedWatchedNamed", { title: video.title })
: i18n.t(hidden ? "feed.hidden" : "feed.markedWatched"),
action: { label: i18n.t(hidden ? "feed.undo" : "feed.unwatch"), onClick: undo },
meta: {
kind: hidden ? "video-hidden" : "video-watched",
videoId: id,
title: video?.title ?? i18n.t("feed.thisVideo"),
channelId: video?.channel_id ?? "",
channelName: video?.channel_title ?? "",
},
});
}
@@ -7,6 +7,7 @@ import {
Check,
Download,
Eye,
Eraser,
EyeOff,
Filter,
FilterX,
@@ -14,7 +15,6 @@ import {
ListPlus,
Play,
RotateCcw,
SkipBack,
Tv,
Undo2,
} from "lucide-react";
@@ -104,7 +104,7 @@ export function useVideoActions(handlers: VideoActionHandlers, open: boolean) {
{
id: "restart",
label: t("card.restartTitle"),
icon: <SkipBack className={icon} />,
icon: <RotateCcw className={icon} />,
onSelect: () => onOpen(v, 0),
},
]
@@ -193,7 +193,7 @@ export function useVideoActions(handlers: VideoActionHandlers, open: boolean) {
{
id: "reset",
label: t("card.resetState"),
icon: <RotateCcw className={icon} />,
icon: <Eraser className={icon} />,
onSelect: () => onResetState(v.id),
},
]
@@ -0,0 +1,44 @@
import { readdirSync, readFileSync } from "node:fs";
import { describe, expect, it } from "vitest";
// The whole app source: a state write can be made from any page.
const root = new URL("../../", import.meta.url);
const sources = () =>
readdirSync(root, { recursive: true, encoding: "utf8" })
.map((f) => f.replace(/\\/g, "/"))
.filter((f) => /\.tsx?$/.test(f) && !f.includes(".test."))
.map((f) => ({ f, src: readFileSync(new URL(f, root), "utf8") }));
// The api's own definitions of these writes, and the one module that may call them.
const ALLOWED = new Set([
"lib/api/feed.ts",
"lib/api/playlists.ts",
"features/video/videoWrites.ts",
]);
// A related list and a live search are never refetched, so a state, Watch-later or playlist-item
// write that does not reach them (`markVideo`) leaves them showing the old mark for the session —
// the built-in Watch later list's items ARE the mark. videoWrites makes those writes and the mark
// together; a direct api call skips the mark.
const WRITES =
/\bapi\s*\.\s*(setState|clearState|watchLaterAdd|watchLaterRemove|addToPlaylist|removeFromPlaylist|moveToPlaylist)\b/;
describe("a video's state, Watch-later and playlist-item writes", () => {
it("go through features/video/videoWrites, never the api directly", () => {
const offenders = sources()
.filter(({ f }) => !ALLOWED.has(f))
// Only the api object's members count (React's own `setState` is not ours) — named, not only
// called: `(saved ? api.watchLaterAdd : api.watchLaterRemove)(id)` makes the write too. The
// dot may sit on the next line of a chain.
.filter(({ src }) => WRITES.test(src))
.map(({ f }) => f);
expect(offenders).toEqual([]);
});
it("catches a write named without being called on the spot, and a playlist item's", () => {
expect(WRITES.test("(saved ? api.watchLaterAdd : api.watchLaterRemove)(id)")).toBe(true);
expect(WRITES.test("await api\n .removeFromPlaylist(listId, id)")).toBe(true);
expect(WRITES.test("api.moveToPlaylist(a, id, b)")).toBe(true);
expect(WRITES.test("this.setState(next); api.playlists()")).toBe(false);
});
});
@@ -0,0 +1,95 @@
// @vitest-environment jsdom
//
// Each write reaches the lists that are never refetched once the server has it — and not before:
// a refused write leaves them as they were.
import { afterEach, describe, expect, it, vi } from "vitest";
import { QueryClient } from "@tanstack/react-query";
import { api } from "../../lib/api";
import { qk } from "../../lib/queryKeys";
import {
addPlaylistItem,
movePlaylistItem,
removePlaylistItem,
resetVideoState,
writeVideoState,
writeWatchLater,
} from "./videoWrites";
afterEach(() => vi.restoreAllMocks());
// The marks' values, without when each was confirmed.
const marks = (qc: QueryClient) => {
const m = qc.getQueryData<Record<string, Record<string, { value: unknown }>>>(qk.videoMarks());
return m
? Object.fromEntries(
Object.entries(m).map(([id, f]) => [
id,
Object.fromEntries(Object.entries(f).map(([k, x]) => [k, x.value])),
]),
)
: undefined;
};
const WATCH_LATER = { id: 1, kind: "watch_later" as const };
const MINE = { id: 2, kind: "user" as const };
describe("videoWrites", () => {
it("marks a state written", async () => {
vi.spyOn(api, "setState").mockResolvedValue({ ok: true } as never);
const qc = new QueryClient();
await writeVideoState(qc, "X", "hidden");
expect(marks(qc)).toEqual({ X: { status: "hidden" } });
});
it("marks a reset as new, from the start", async () => {
vi.spyOn(api, "clearState").mockResolvedValue({ ok: true } as never);
const qc = new QueryClient();
await resetVideoState(qc, "X");
expect(marks(qc)).toEqual({ X: { status: "new", position_seconds: 0 } });
});
it("marks a Watch-later save and its removal", async () => {
vi.spyOn(api, "watchLaterAdd").mockResolvedValue({ ok: true } as never);
vi.spyOn(api, "watchLaterRemove").mockResolvedValue({ ok: true } as never);
const qc = new QueryClient();
await writeWatchLater(qc, "X", true);
expect(marks(qc)).toEqual({ X: { saved: true } });
await writeWatchLater(qc, "X", false);
expect(marks(qc)).toEqual({ X: { saved: false } });
});
it("marks an item of the Watch later list as the bookmark it is", async () => {
vi.spyOn(api, "addToPlaylist").mockResolvedValue({ ok: true } as never);
vi.spyOn(api, "removeFromPlaylist").mockResolvedValue({ ok: true } as never);
const qc = new QueryClient();
await addPlaylistItem(qc, WATCH_LATER, "X");
expect(marks(qc)).toEqual({ X: { saved: true } });
await removePlaylistItem(qc, WATCH_LATER, "X");
expect(marks(qc)).toEqual({ X: { saved: false } });
});
it("marks nothing for an item of any other list", async () => {
vi.spyOn(api, "addToPlaylist").mockResolvedValue({ ok: true } as never);
vi.spyOn(api, "removeFromPlaylist").mockResolvedValue({ ok: true } as never);
const qc = new QueryClient();
await addPlaylistItem(qc, MINE, "X");
await removePlaylistItem(qc, { id: 3, kind: undefined }, "X");
expect(marks(qc)).toBeUndefined();
});
it("marks a move into and out of the Watch later list", async () => {
vi.spyOn(api, "moveToPlaylist").mockResolvedValue({ ok: true } as never);
const qc = new QueryClient();
await movePlaylistItem(qc, MINE, "X", WATCH_LATER);
expect(marks(qc)).toEqual({ X: { saved: true } });
await movePlaylistItem(qc, WATCH_LATER, "X", MINE);
expect(marks(qc)).toEqual({ X: { saved: false } });
});
it("marks nothing the server refused", async () => {
vi.spyOn(api, "setState").mockRejectedValue(new Error("offline"));
const qc = new QueryClient();
await expect(writeVideoState(qc, "X", "hidden")).rejects.toThrow("offline");
expect(marks(qc)).toBeUndefined();
});
});
@@ -0,0 +1,69 @@
import type { QueryClient } from "@tanstack/react-query";
import { api, type Playlist, type VideoStatus } from "../../lib/api";
import { markVideo } from "../../lib/progressCache";
// A video's user state, its Watch-later mark and its playlist items, written to the server and —
// once it has the change — into the lists that are never refetched (`markVideo`: every related
// list, every live search). These are the ONE way to make those writes, and
// `videoWrites.contract.test.ts` holds it: four writes that called the api directly (the
// Notification Center's undo, the feed's reset, the feed's and Home's Watch-later toggles) left a
// related list hiding a video the user had un-hidden, or showing it unsaved after a save. A
// playlist item is one of them because the built-in Watch later list IS the mark (`saved`): ticked
// in Add to playlist or taken out on the Playlists page, it left the bookmark as it was. Each
// returns the server's answer, so a caller keeps its own optimistic layer, notices and rollback.
export function writeVideoState(qc: QueryClient, id: string, status: VideoStatus) {
return api.setState(id, status).then((r) => {
markVideo(qc, id, { status });
return r;
});
}
/** Back to pristine: the state row and its resume position both go. */
export function resetVideoState(qc: QueryClient, id: string) {
return api.clearState(id).then((r) => {
markVideo(qc, id, { status: "new", position_seconds: 0 });
return r;
});
}
export function writeWatchLater(qc: QueryClient, id: string, saved: boolean) {
return (saved ? api.watchLaterAdd(id) : api.watchLaterRemove(id)).then((r) => {
markVideo(qc, id, { saved });
return r;
});
}
/** The list a playlist item write goes to: on the built-in Watch later list (`kind`) the item IS
* the video's Watch-later mark. A kind the caller cannot tell (undefined) marks nothing. */
type ListRef = { id: number; kind: Playlist["kind"] | undefined };
const markIfWatchLater = (qc: QueryClient, list: ListRef, id: string, saved: boolean) => {
if (list.kind === "watch_later") markVideo(qc, id, { saved });
};
/** Add a video to a playlist; `position` (an index in the stored order) puts it back where a
* removal took it from. */
export function addPlaylistItem(qc: QueryClient, list: ListRef, id: string, position?: number) {
return api.addToPlaylist(list.id, id, position).then((r) => {
markIfWatchLater(qc, list, id, true);
return r;
});
}
export function removePlaylistItem(qc: QueryClient, list: ListRef, id: string) {
return api.removeFromPlaylist(list.id, id).then((r) => {
markIfWatchLater(qc, list, id, false);
return r;
});
}
/** Move a video from one playlist to another, in one request (see `api.moveToPlaylist`). */
export function movePlaylistItem(qc: QueryClient, from: ListRef, id: string, to: ListRef) {
return api.moveToPlaylist(from.id, id, to.id).then((r) => {
markIfWatchLater(qc, from, id, false);
markIfWatchLater(qc, to, id, true);
return r;
});
}
@@ -305,6 +305,37 @@ describe("backend ids have their names in both languages", () => {
expect([...RUN_STATUSES].sort()).toEqual([...runStatuses].sort());
});
it("the Takeout import's notice is known by the server's job id", () => {
// Read as text: the slice pulls in the request layer, which needs a browser.
const feedTs = readFileSync(new URL("../lib/api/feed.ts", import.meta.url), "utf8");
const ours = /^export const TAKEOUT_JOB = "(\w+)";$/m.exec(feedTs)?.[1];
expect(ours).toBeTruthy();
expect(/^TAKEOUT_IMPORT = "(\w+)"$/m.exec(schedulerPy)?.[1]).toBe(ours);
});
it("the Takeout import's files and outcomes are the server's", () => {
const feedTs = readFileSync(new URL("../lib/api/feed.ts", import.meta.url), "utf8");
const ours = (name: string) =>
[
...(
new RegExp(`^export const ${name} = \\[([^\\]]*)\\]`, "m").exec(feedTs)?.[1] ?? ""
).matchAll(/"(\w+)"/g),
].map((m) => m[1]);
const models = backend("reader/models.py");
const theirs = (cls: string) =>
[
...(
new RegExp(`^class ${cls}\\(StrEnum\\):\\n([\\s\\S]*?)(?=\\n\\S)`, "m").exec(
models,
)?.[1] ?? ""
).matchAll(/^\s+[A-Z_]+ = "(\w+)"/gm),
].map((m) => m[1]);
expect(theirs("TakeoutKind").length).toBeGreaterThan(0);
expect(ours("TAKEOUT_KINDS")).toEqual(theirs("TakeoutKind"));
expect(theirs("TakeoutOutcome").length).toBeGreaterThan(0);
expect(ours("TAKEOUT_OUTCOMES")).toEqual(theirs("TakeoutOutcome"));
});
for (const lang of ["en", "hu"]) {
const keys = keysOf(lang);
const missing = (wanted: string[]) => wanted.filter((k) => !keys.has(k));
@@ -7,6 +7,7 @@
"tabs": {
"subscribed": "Subscriptions",
"discovery": "Discover from playlists",
"watched": "Discover from your history",
"hidden": "Hidden",
"blocked": "Blocked"
},
@@ -154,5 +155,38 @@
"hint": "Their videos are kept out of your personal Feed, but still appear in the shared Library.",
"empty": "No hidden channels. Hide a subscription (the eye icon on its row) to keep its uploads out of your personal Feed without unsubscribing.",
"unhide": "Unhide"
},
"watched": {
"intro": "Channels you often watched on YouTube but don't follow, from the watch history you uploaded. Only videos of five minutes or more count, so Shorts stay out. Follow a channel to get its new uploads in your Feed, or take it off the list.",
"loading": "Finding channels…",
"empty": "No suggestions — you follow every channel you watched five or more longer videos from.",
"notImported": "Upload your YouTube watch history to see the channels you watch but don't follow.",
"openTakeout": "Open YouTube history",
"pending_one": "{{count}} more channel is still being looked at — the list may grow.",
"pending_other": "{{count}} more channels are still being looked at — the list may grow.",
"more_one": "Show {{count}} more",
"more_other": "Show {{count}} more",
"shown": "{{shown}} of {{total}}",
"dismiss": "Not interested",
"dismissHint": "Take this channel off the list. You can undo it right after.",
"dismissed": "{{name}} is off the list",
"cols": {
"watched": "Watched",
"watchedHint_one": "Show the {{count}} video of five minutes or more you watched from this channel",
"watchedHint_other": "Show the {{count}} videos of five minutes or more you watched from this channel",
"last": "Last watched"
},
"videos": {
"title": "Videos you watched from {{name}}",
"summary_one": "{{count}} video of five minutes or more · last watched {{when}}",
"summary_other": "{{count}} videos of five minutes or more · last watched {{when}}",
"loading": "Getting the videos ready…",
"gone_one": "{{count}} video is no longer on YouTube",
"gone_other": "{{count}} videos are no longer on YouTube",
"empty": "None of these videos is on YouTube any more.",
"colTitle": "Video",
"colLength": "Length",
"colWatched": "Watched"
}
}
}
+4
View File
@@ -54,6 +54,10 @@
"label": "Backfill quota reserve",
"hint": "Units kept in reserve so scheduled backfill never starves interactive syncs / enrichment."
},
"related_daily_limit_per_user": {
"label": "Related videos — daily limit per user",
"hint": "Max related-video requests a user can make per day — opening a list counts one, and so does every “Show more”. They come from YouTube's website and spend no search quota; looking up the videos in one costs up to about 8 units (up to 4 pages, each a video and a channel lookup). A failed one doesn't count (up to 5 an hour). Set 0 to turn related videos off."
},
"search_daily_limit_per_user": {
"label": "Live search — daily limit per user",
"hint": "Max live YouTube searches per user per day. With the API source each search costs 100 units (so the shared budget affords only ~80-90/day total); with the scrape source it's a pure anti-abuse limit. Set 0 to disable live search."
+16 -2
View File
@@ -87,6 +87,7 @@
"purgeDiscovery": "Couldn't clear the discovery data",
"plexWatchLink": "Couldn't change the Plex watch-state sync",
"plexWatchImport": "Couldn't import the watch state from Plex",
"takeoutUpload": "Couldn't upload the Takeout file",
"shareLinkDownload": "Couldn't change whether the link allows downloading",
"shareLinkRevoke": "Couldn't revoke the link",
"shareLinkPassword": "Couldn't change the link's password",
@@ -114,7 +115,9 @@
"apiToken": {
"create": "Couldn't create the token",
"revoke": "Couldn't revoke the token"
}
},
"dismissWatchedChannel": "Couldn't take the channel off the list",
"restoreWatchedChannel": "Couldn't put the channel back on the list"
},
"ok": "OK",
"offline": {
@@ -185,6 +188,10 @@
"quota_spent": "Today's shared YouTube budget is spent. Try again tomorrow.",
"recipient_required": "A recipient email is required",
"registration_closed": "Registration is currently closed",
"related_disabled": "Related videos are turned off on this instance.",
"related_daily_limit": "You've reached today's related videos limit ({{limit}}). Opening a list counts as one request, and so does every “Show more”. Try again tomorrow.",
"search_busy": "Too many searches are running right now. Try again in a few seconds.",
"search_disabled": "Live YouTube search is turned off on this instance.",
"search_daily_limit": "You've reached today's YouTube search limit ({{limit}}). Try again tomorrow.",
"search_term_required": "Enter something to search for",
"session_gone": "The playback session ended",
@@ -249,6 +256,13 @@
"api_token_preset_grants": "The Hub token gets its folder by itself — leave the folders out",
"api_token_no_grants": "Choose at least one folder the token may use",
"api_token_name_required": "Give the token a name",
"api_token_limit": "You already have {{limit}} tokens — revoke one first"
"api_token_limit": "You already have {{limit}} tokens — revoke one first",
"takeout_unreadable": "This isn't a Takeout file the import reads. Upload watch-history.json (JSON format) or Watch later-videos.csv.",
"takeout_no_videos": "This file has no videos to import",
"takeout_busy": "Another of your files is still being uploaded — wait for it to finish",
"takeout_server_busy": "Other imports are being received right now — try again in a minute",
"takeout_cooldown": "You have uploaded many files in the last hour — try again later",
"takeout_too_many": "That would keep more than {{limit, number}} videos from your export — more than a real one holds",
"takeout_list_too_long": "This list holds more than {{limit, number}} videos — more than a YouTube list can"
}
}
+1
View File
@@ -16,6 +16,7 @@
"openScheduler": "Open Scheduler",
"openFile": "Open the file",
"openTokens": "Open API tokens",
"openTakeout": "Open YouTube history",
"revertFailed": "Couldn't undo that — please try again",
"downloadDone": {
"title": "Download ready"

Some files were not shown because too many files have changed in this diff Show More