GET /api/search/related/{video_id} reads the list YouTube shows beside a
video from the website's InnerTube "next" endpoint (no quota), and brings
it into the catalog the way live-search results come in. It leaves out
what the user watched or hid, here or in their Takeout history (unless
taken back to "new" since), blocked channels and the video itself, and
pages on until 20 are gathered (at most 4 pages). region=topical|hu picks
the locale the list is filled in. Its own per-user daily cap
(related_daily_limit_per_user, one event per opened list); a failed
scrape answers 422 youtube/unavailable.
The InnerTube request (live key, consent cookie, errors) moves into
youtube/innertube.py, shared by the search scrape and the related one.
A subscription pulled in from YouTube left an explored channel flagged as an exploration's, so its
later videos came in hidden from the Library with their watched marks waiting for good; a subscribe
landing while an exploration's videos came in left those flagged too. One step, settle_explored,
now holds the rule by who follows the channel now: the subscribe here, the subscription import, the
watched-channel videos and the explore page all end with it. Migration 0116 clears what the gap
left behind.
The list's "may still grow" count no longer counts a channel it already shows from a stub row, and
the comments on the cleanup's keep rule and on the in-memory daily shares say what they do.
- A channel someone follows: the videos its exploration brings in (the list's
dialog, the channel page's explore) come in as the catalog's own, not
flagged via_explore, which subscribing would never clear and which kept
their watched marks waiting for good.
- The explore cleanup keeps the row of a channel a Takeout history shows
watched (it still reclaims the videos), so a looked-at channel no longer
drops off the list.
- The Shorts job takes up only users whose list still has lengths to ask or
channels to name, by each user's own follows, blocks and dismissals; a
naming that failed is tried again, and the list says it is not complete.
- The length lookups have a daily share of their own: a large history no
longer leaves the export's Watch later file waiting a day.
- Only a channel of the user's own history can be put out of the list.
- The dialog's videos have a query root of their own: the list's
invalidations no longer re-ask (and re-explore) under an open dialog.
The import asks YouTube how long the watched videos of the candidate
channels are (five or more watched, none the user follows, blocked, put
out or owns), keeps the length on the Takeout row and names the channels
that make the list: five videos of five minutes or more. The shorts job
asks what an import could not. A channel's counted videos come in as an
exploration's when its list is opened, ready to play; a channel can be put
out of the list and back.
- An ad-hoc request made while its target's run is under way is the run after it: the scheduler
dropped a second fire of a running id, so a second Takeout file was never imported and its key
stayed queued until a restart.
- Un-marking or resetting a video is remembered (video_unmarks): an import of an older YouTube view
no longer marks it watched again; un-marks take the user's Takeout lock.
- One user's failure in the Shorts job's Takeout pass is theirs: the job keeps its own result, the
others are settled, and a failed lookup spends none of the daily share.
- The upload reads the user's id before giving the read connection back.
- A watched row waits on a discovery video (search or explore) until the catalog keeps it, so a
mark never pins it against the cleanup jobs.
- The history is decoded a window at a time, a Watch later file over 2 MiB is refused unread, and
two uploads at once across all users (takeout_server_busy).
- A Watch later list goes in as one presence read and one insert.
- Uploads: one at a time per user and ten an hour, each refused only after its body arrived;
the history read an entry at a time (32 MiB cap), a Watch later list capped at YouTube's 5000,
a user's kept videos at 200k; any unreadable shape is a 422, not a 500.
- Quota: Watch later lookups stop at the backfill reserve and at a per-user daily share; what is
not asked about waits, known videos still go in, and the shorts job retries the rest.
- Staging and settling share one per-user advisory lock (no deadlock); a run settles only the
rows it read, so a row staged meanwhile is never called gone.
- A failed run's notice names a translated failure, never the upstream's text.
- The status says when an import is queued; the page watches it to its end even when left, then
reads the feed, its counts and the playlists again.
- Settings: one live region, the zone kept (unavailable) while a run goes, focus handed back,
a missed drop kept from the browser, an oversize file refused before it is sent.
- Scheduler and run lists show Takeout runs by the importing user.
A watched video the catalog did not know at import time waits in `takeout_items`. The Shorts job now
settles those rows right after its probe (the only place a video's Short-ness becomes known), every
user's, a commit each: `reader.takeout.apply_waiting`, its count in the job's result.
`GET /api/takeout` is what the settings page reads: the user's last import run (`jobruns.last_run`)
and the kept videos by kind and outcome, "waiting" for one not known yet.
The YouTube API reads neither a user's watch history nor their Watch later list, so the Takeout
export is the only way in. `POST /api/takeout` takes one file of it (watch-history.json or
Watch later-videos.csv, told apart by content), keeps its videos in `takeout_items` and queues the
user's `takeout_import` run, whose outcome reaches their inbox.
The run marks watched the known videos the Shorts probe has cleared, at the Takeout time; a Short
is not marked, and a state the user set here (watched, hidden, or changed after that time) wins, by
the upsert's own condition. A video the catalog does not know yet waits with no outcome. Watch later
videos are fetched if unknown and appended oldest first.
An ad-hoc run can now ask for a completion notice (`run_ad_hoc(..., notify=True)`).
- /auth/google/connect is a navigation and carries no headers: the tab names the account it
was built for in `expect`, checked by the same comparison as the request header
(built_for_another_account); a mismatch lands on /?account=changed
- the link marker names the OAuth state of its own flow: an older flow coming back no longer
spends the marker of the connect still open (which then returned as a plain sign-in)
- a 401 account_changed reloads the tab with a one-time flag, and the reloaded tab says it
now shows the account signed in
- a refused switch to a suspended or not-yet-approved account says so in the signed-in
app's words, naming the address a request was recorded for
- the prefs merge locks FOR NO KEY UPDATE, so a transaction that only added a row referring
to the account does not hold the save up; a test pins the lock
- giving up a lost page closes a channel page open over it
- e2e README names the chromium-only specs
A tab names the account it was claimed for on every request, beacon and upload
(X-Siftlode-Expect-Account); current_user and /auth/logout refuse a mismatch with
401 account_changed and keep the session, so a tab left showing the account
signed in before reloads instead of writing to the new one.
/auth/login clears a connect marker left by an abandoned connect, and a link
completes only on a session that still holds. The prefs merge reads the row
under a lock. A tab's sessionStorage records its own owner. A page the account
loses is given up, address bar and remembered page included. The panel layouts
are preferences; the storage contract also refuses a key handed to the Storage
API or to usePref outside its kind. A refused sign-in as someone else and a
failed sign-out say so. The chromium-only e2e specs are in the gate register,
the member walk waits for its requests, and devices.spec checks the save itself.
/auth/link and /auth/upgrade become /auth/google/connect?access=identity|read|write: one handler, one
table of what each access asks Google for, the same callback markers. An unknown access is refused
(422) instead of quietly meaning read, and an instance without Google answers ?link=error for a
grant too, not a silent return to the app. The frontend builds the URL in lib/googleConnect, and a
contract test holds it to the backend route and its access values.
The session held a wallet of up to six accounts, and a request picked one by a header or an
?account= parameter. It now holds one: user_id and its epoch. A sign-in replaces whoever was
signed in, sign-out and account deletion end the session, and an account refused mid-session
(revoked, suspended, deactivated) clears it. /api/me/accounts and /api/me/switch are gone, and
with them their refusal codes and the UI copy for them.
A cookie the multi-account build wrote carries no single epoch and is refused: its holder signs
in once more. A Google link finishing in a browser now signed in as someone else is refused, as
before for an account outside the wallet. A test holds that no app code reads the wallet keys or
the account header again.
The test's own writes are committed before a request, so a handler's Core DML can no longer hide
in a transaction the test had already opened. The entry-point probe answers on a marked line the
API's logging cannot mix into. A listed commit whose function no longer commits fails the fence.
CLAUDE.md and the backend README now name what an AST cannot see; the route-commit tests no longer
claim their read-back holds the commit.
A season marked watched, a pull from Plex and a YouTube resume point made their first row by a read
then an add, and failed on the unique key when a first checkpoint or the watched mark made it in
between. They go through the same insert-then-read as the checkpoint (the resume point is one
upsert); a pull that meets a local change still waiting for its push leaves it.
Folding duplicate Watch later lists moved their videos but not a Home pin, which the cascade then
deleted; the first pin of a dropped copy moves onto the kept list.
Marking one notice read trimmed before its own flag was flushed, so the trim left one past the cap.
The route test now crosses the cap and reads the result from a session of its own.
Each handler that now calls a service and commits is driven through HTTP once.
A second account's watch state, Watch later list and private download profile
are never read or changed. The autotag job's orphan prune is seen committed.
The first checkpoint of an item and the final flush beside it both read no
state and both inserted; the loser failed on uq_plex_user_item. Two first saves
each made a Watch later list. Both are created by an ON CONFLICT DO NOTHING
insert and then read; a partial unique index holds one Watch later per user, and
the migration folds an existing second list into the first. Plex keeps one
per-item state map (rollup.item_state_map).
The owner-filter check missed an aliased() model, an import alias, .in_() and
filter_by; four live inline filters on aliased VideoState go through owned_by now.
Cross-unit writes are seen under any import name, through sa.update and via
__table__. Core modules may commit only where CORE_COMMITS names the reason;
trim_read and audit.gc flush now and their callers commit. The pin target, the
debug-session header and a link's target fetch through owned_row. The docs
claim what the fence holds, and say what is beyond it.
A db.execute(update/delete) fires no flush and stages nothing, so a handler whose
only write was Core DML could forget its commit and stay green. The check now
asks Postgres whether the request opened a writing transaction. It found one
real case: a repeated issue send returned with the session row still locked.
The models now live in their units' packages, and a core model names a unit's by
string (User.subscriptions -> Subscription). The download child imports neither
catalog nor app.tables before db.get, so mapper configuration failed and every
download, clip, edit and upload job died with child_exit. Each entry point now
imports app.tables, and a test imports each one in a clean interpreter.
The per-file pin list and the unit-by-unit switch go: every route file is held
to zero database work, every unit's package to its edges, and core code says
whose a row is through userscope too. CLAUDE.md, the backend README and the
gate register describe the units as they now are.
Tokens, notifications, quota, debug sessions and the reset-link burn filtered
their owner inline. Ten of the fifteen filters had no test that noticed one
going missing; each now has a test where another account's rows must not
count, show or change.
The last Plex HTTP files do no database work of their own, so ALLOWED is empty: no route builds
SQL, touches the session or a unit's table. Discover's rails move whole into plex/discover.py; a
user's playlists, watch-state writes (the race-safe finish, an early checkpoint, a status, a
season or show at once) and the watch-sync link go through plex/queries.py and plex/service.py.
The plex package joins FENCED: it reads no other unit's tables, says whose a row is through
userscope (the rollup and the watch sync's 8 inline filters moved to owned_by), and commits only in
its named jobs — the library sync, TMDB enrichment, the watch sync and its background pushes.
A playlist's has-item probe keeps the plain key lookup, so a NUL in it is still the 422 a test pins.
Pins 178 -> 0; basedpyright baseline 118 -> 106.
The HTTP files browse, detail, people, collections and _shared do no database work of their own
now. New plex/queries.py (the rows a page shows, a user's watch state through userscope),
plex/service.py (the info pages' enrichment, a collection's editable flag, the row kept after a
failed follow-up read, the person bio's background fetch; none commits), plex/library.py (the
unified browse feed, its search and facets) and plex/collections.py (a collection's source,
editability, card, certificate and member order). Every watch-state join in the moved code names
its owner through owned_by.
Pins 354 -> 178 in 4 files; basedpyright baseline 132 -> 118.
PlexLibrary, PlexShow, PlexSeason, PlexItem, PlexPerson, PlexCollection, PlexState, PlexLink,
PlexPlaylist and PlexPlaylistItem leave app/models.py for app/plex/models.py; app.tables registers
them. Every import is re-pointed; no schema change.
Channels, search and sync do no database work in their routes any more: reads go through
catalog/queries, writes through catalog/service, and a live search's ingest is catalog/search.
What a catalog route needs from Reader (a playlist's channels, a user's tag links, the feed
projection of results) it asks reader/queries and reader/service. The scheduler and the runs page
read channel titles and the work queue through catalog/queries. Catalog is fenced below its HTTP
layer: its owner filters go through userscope, it reads no other unit's tables, and its ingest
jobs commit only where they are named.
Channel, Subscription, Video, SearchFind, ExploredChannel and BlockedChannel (and LIVE_OR_UPCOMING)
leave app/models.py for app/catalog/models.py; app.tables registers them. The User↔Subscription
relationship names its other side as a type only. No schema change.
A reorder's left-out item, a playlisted video beside one not asked about, another user's tag, and
the user's own channel link committed before the auto-tagger reads the table.
Feed, playlists, saved views, tags and the Home pins do no database work in their routes any more:
reads go through reader/queries (and feedquery for a feed page, its counts and facets), writes
through reader/service and discover/pins, which flush; the handlers commit. Reader is fenced below
its HTTP layer too: its owner filters go through userscope, and the catalog's jobs ask Reader
instead of joining its tables (the auto-tagger's system tags, the cleanup jobs' keep rule, the
maintenance notice). A playlist sync's catalog ingest moves to sync/videos.
VideoState, Playlist, PlaylistItem, SavedView, ComfortPin, Tag and ChannelTag leave app/models.py
for app/reader/models.py; app.tables registers them. No schema change.
Capture's routes (downloads, files, public watch links) and the token list no longer build SQL, call
the session or set a model's fields: reads are downloads/queries, writes the unit's service (service,
folders, links, quota), and each write flushes for the handler to commit. Whose a row is goes
through userscope (owned_by, owned_row), which owned_or_404 now uses too.
The fence holds Capture below its routes as well: it reads only Catalog's and Plex's tables, writes
no other unit's, filters owners through userscope and commits only in its jobs and the upload
ingest. A test per service use case, among them that the tray lists only the caller's files.
The nine Capture tables (files, jobs, folders, shares, links, quota, cookies, profiles, library
resume points) leave app/models.py for the unit's own module; app.tables registers every unit's
tables for the migrations, the test database and the ownership check. No schema change.
The fence is switched on unit by unit: each route file's database work (SQL built, session data
calls, model attributes set, unit tables imported) is pinned at today's count and may only shrink.
A core route may not reach into a unit's tables. Every model has an owner.
- The scheduler keeps one registry of the runs this process has open, for the interval jobs and the
ad-hoc runs alike. A row the table still says is `running` but no run here has open (its outcome
failed to write) reads as interrupted, as the next start's sweep will close it: no pulsing dot
beside "No run recorded yet".
- Migration 0109's downgrade sums a still-running run's quota events into its spend before the link
to them is dropped, as the startup sweep would.
- A test holds that a dialog closing onto the same address leaves an open log line open.
- The channel-backfill row says "running" from the runs this process has open, not from the table:
a row whose outcome failed to write no longer reads as running until a restart. The run still
going leads the row, so a newer run's outcome never sits under the older one's channel.
- Migration 0109's downgrade carries every run's trigger, spend and start in the audit row, and the
upgrade restores them: a manual run whose admin was deleted stays manual.
- Back/Forward is read by the page's one popstate listener, both halves in one render; the server
log adopts its half during that render, so no read goes out with one entry's filters and the
other's range.
- The server log's failed facets use the other tabs' words (the menus may be short); the accounts
message shows only when the menu really lacks them.
Backend
- A run's outcome write sits outside the job's own try: a database blip after a run that
succeeded no longer records it as failed (the row waits for the next start's sweep).
- An interrupted run keeps what it spent (the sweep sums its quota rows, as finish does).
- The Scheduler's ad-hoc row says running while any backfill runs, not only the newest.
- recent_runs reads each job's last runs off its index instead of ranking the whole table.
- An interval change is stored with its audit row in one transaction, then rescheduled.
- One API instance per database is the stated contract (.env.example, jobruns).
- Turning full history on for a channel already complete records no run.
- Migration 0109 round-trips interrupted and ad-hoc runs; the run words' DB CHECKs are
test-held to the enums; the account filter is one helper for the four activity reads.
- An audit row says how many of its request's log lines are still kept.
Frontend
- An "All" period reaches every row still kept (runs and admin actions outlive 90 days).
- Back/Forward is followed in place by the server log: a closing dialog no longer turns
Live off; a pivot still opens a fresh log.
- Runs moved over from the audit log read their one-liner in the reader's language.
- Filter choices or accounts that failed to load say so beside the filters, with a retry.
- An old server-log link keeps its account, text and range; a remembered retired page
opens its tab.
- A request whose lines are gone says so instead of linking to an empty list.
- The log lines and the history rows share one row frame, and the server log one list
frame with the other tabs; an empty list is announced; each day's quota is readable.
The audit log, the server log and the admin quota view were three pages (the quota one a Stats tab),
each with its own filters, and the background runs had no list at all. Now one page ("Napló" /
"Activity") reads all four, each still its own table on the server: a shared bar (text, a period of
1 h to 90 days defaulting to 7, an account) and each tab's own filters, the whole view in the URL.
The old page ids open their tab.
Server side: the audit log and the new run list (`/api/admin/runs`) filter and page like the server
log, through one keyset helper (`routes/_keyset`) the log now shares; an audit row keeps the request
it was made in (migration 0111), so its log lines are a pivot away, as a run's are (`lrun`). The
quota breakdown takes since/until and one account. The Scheduler rows get a history button to their
job's runs.
Also fixed on the way: re-saving the value in force on the Configuration page staged its override
without committing it (the commit sat inside the audit branch), so the save was lost.
An interval override was a row of its own table beside the one store every other admin knob uses; it
is now the `job_interval_<job>` sysconfig key (migration 0110 copies them over and drops the table).
The Scheduler page stays its one editor: the Configuration page and the setup wizard neither list nor
write those keys, since a write there would skip the reschedule.
Each ConfigSpec now states `applies` (required, no default). Every key is `live`, held by a scan that
finds no read at import time; a `restart` key is refused until the page can say so. A second scan
requires a reader for every key on the Configuration page — `sponsorblock_default` had none since it
was added, so it leaves the registry, the translations and the store.
A row's one dot said how the LAST run went, and an interrupted run fell through to "idle" — a grey dot
that read as "no run recorded". The row now leads with its last three runs, newest first, each named
for a screen reader and in its tooltip (how it went, when, who started it); an interrupted run is a
ring, set apart from a skipped run and a failure, and its line says the server restarted under it.
A channel's full-history backfill gets its own "Ad-hoc runs" block: its runs, its channel and what
it fetched, no interval and no run button — it needs its target. AD_HOC_JOBS names every such job;
the backend-id contract holds their names in both languages, and run_ad_hoc refuses any other id.
The one-channel deep backfill ran as a FastAPI BackgroundTask: what it fetched, what it spent and
how it failed reached only the log. It is now a one-shot run on the scheduler (`channel_backfill`,
trigger ad_hoc, actor = the user, target = the channel) with its quota tagged to the run. Its
per-channel exclusion with the scheduled backfill and the two-run cap are unchanged, and a run
they turn away says why in its result instead of returning nothing. A failure is no longer
swallowed: the run records it.
The scheduler snapshot carries each job's last three runs (one windowed query), so the page can
show an interrupted or skipped run instead of the latest outcome alone.
What a job last did lived in the API process's memory and was gone at
every restart; the audit log kept a run only when it was manual, failed
or changed something. Now each run is a job_runs row written at its
start and its end in its own session: who started it, its outcome as
data, what it spent of the YouTube quota (quota_events.job_run_id) and
the lines it logged (log_events.run_id, filterable in the log console).
- a run a stopped process left open is marked interrupted at startup
- "run now" is a one-shot job on the scheduler, which now always starts
(interval jobs only with SCHEDULER_ENABLED); the manual actor is reset
after the run, since pool threads are reused
- the dashboard's last run comes from job_runs, so it survives a deploy
- runs leave the audit log: migration 0109 moves the old job_run rows
over (and back on downgrade); the audit page's run folding goes
- job_run_retention_days (default 30) prunes old runs in audit_gc
- A password reset or change, "log out everywhere" and a revoked untrusted password revoke all of the
account's API tokens too (audited): one minted from a stolen session no longer outlives the answer.
- A bearer token or link grant with a non-ASCII character is refused (401/403), not a 500.
- Making a token's grants or a folder link holds the folder, so a racing delete cannot fail the
foreign key; a link's view count is one atomic UPDATE.
- A folder link's card counts exactly the files its page lists (one `links.folder_files`).
- A file with no extension is served without a trailing dot.
- 0106's downgrade names a Hub folder so it never collides with one of the user's folders.
- The speed baseline reads share tokens through the app (the database keeps them hashed/encrypted).
- Tests: the token scope on the real session stack, the folder poster, the Hub refused for a link,
0106's downgrade data step, 0107 without a key; docs/comments follow (three folder kinds, folder
links, key change, the cleanup job's token warnings).
A /watch link may now open one of its owner's own folders (migration 0108: folder_id beside job_id,
exactly one set; a folder's links go with it). The page shows what is in the folder when it is
opened, newest first: a video or a song plays in the same player a file link has, above the list,
each keeping its own resume point; a document only downloads, never opens as a page. A file moved
out stops opening through the link at its next request. Password, expiry and stream-only work as on
a file link; the unfurl names the folder and counts its files, never their names. Source folders
and the Hub cannot be shared.
The folder menu gets "Copy link" and "Share with a link…" (the Share dialog's link half); the rail
marks a folder a live link opens.
Two fixes in the Share dialog on the way: revoking the last link no longer mints a new one at once,
and the new-link form goes back to downloadable after making a link.
A /watch link's token is the credential, and it sat in the database as written. Migration 0107
replaces the column with the token's SHA-256 (the lookup) and a Fernet-encrypted copy (so the owner
can copy the same link again); every existing address keeps working, and the downgrade decrypts
them back. Without a copy this key can read, a link still opens but its address cannot be shown:
"Copy link" then takes the newest link that has one or makes a new one, and the Share dialog says
so in that row instead of a broken URL. Making, changing and revoking a link are audited, never
with its token or password.
peter