Compare commits

...
43 Commits
Author SHA1 Message Date
peter 909732b3c1 release: 0.108.1 (improvement/e42-backend-units) 2026-10-20 02:12:49 +02:00
peter 215f022fe3 chore(release): 0.108.1 — the backend in four units, each owning its tables; a first Watch later and a first resume point survive a write beside them 2026-10-20 02:04:56 +02:00
peter 6ea750a698 test: the commit check starts every request clean, and the fence docs claim what it holds
The test's own writes are committed before a request, so a handler's Core DML can no longer hide
in a transaction the test had already opened. The entry-point probe answers on a marked line the
API's logging cannot mix into. A listed commit whose function no longer commits fails the fence.
CLAUDE.md and the backend README now name what an AST cannot see; the route-commit tests no longer
claim their read-back holds the commit.
2026-10-20 01:56:31 +02:00
peter c5f121139b fix: every first watch state survives a write beside it, and a folded Watch later keeps its pin
A season marked watched, a pull from Plex and a YouTube resume point made their first row by a read
then an add, and failed on the unique key when a first checkpoint or the watched mark made it in
between. They go through the same insert-then-read as the checkpoint (the resume point is one
upsert); a pull that meets a local change still waiting for its push leaves it.

Folding duplicate Watch later lists moved their videos but not a Home pin, which the cascade then
deleted; the first pin of a dropped copy moves onto the kept list.

Marking one notice read trimmed before its own flag was flushed, so the trim left one past the cap.
The route test now crosses the cap and reads the result from a session of its own.
2026-10-20 01:56:30 +02:00
peter b04b2cbd62 test: the rewritten write handlers commit, and three more owner filters hold
Each handler that now calls a service and commits is driven through HTTP once.
A second account's watch state, Watch later list and private download profile
are never read or changed. The autotag job's orphan prune is seen committed.
2026-10-20 00:45:22 +02:00
peter 3a5212cd8a fix: a first watch state and a first Watch later survive two requests at once
The first checkpoint of an item and the final flush beside it both read no
state and both inserted; the loser failed on uq_plex_user_item. Two first saves
each made a Watch later list. Both are created by an ON CONFLICT DO NOTHING
insert and then read; a partial unique index holds one Watch later per user, and
the migration folds an existing second list into the first. Plex keeps one
per-item state map (rollup.item_state_map).
2026-10-20 00:45:22 +02:00
peter b376373af5 fix(files): a folder's kind is answered before its new name, as before
A blank name on a source folder answered 400 name_required; it answers 409
not_own again.
2026-10-20 00:45:22 +02:00
peter 8206384c91 test(fence): owner filters in any form, statement writes under any name, core commits by list
The owner-filter check missed an aliased() model, an import alias, .in_() and
filter_by; four live inline filters on aliased VideoState go through owned_by now.
Cross-unit writes are seen under any import name, through sa.update and via
__table__. Core modules may commit only where CORE_COMMITS names the reason;
trim_read and audit.gc flush now and their callers commit. The pin target, the
debug-session header and a link's target fetch through owned_row. The docs
claim what the fence holds, and say what is beyond it.
2026-10-20 00:45:21 +02:00
peter e27656ee7e test: the request commit check sees Core DML and row locks
A db.execute(update/delete) fires no flush and stages nothing, so a handler whose
only write was Core DML could forget its commit and stay green. The check now
asks Postgres whether the request opened a writing transaction. It found one
real case: a repeated issue send returned with the session row still locked.
2026-10-20 00:45:20 +02:00
peter d73cdcee97 fix(capture): every process registers every table before its first query
The models now live in their units' packages, and a core model names a unit's by
string (User.subscriptions -> Subscription). The download child imports neither
catalog nor app.tables before db.get, so mapper configuration failed and every
download, clip, edit and upload job died with child_exit. Each entry point now
imports app.tables, and a test imports each one in a clean interpreter.
2026-10-20 00:45:20 +02:00
peter 5ce292cd28 test(fence): the unit fence has no exemptions, and the owner filter is held in every module
The per-file pin list and the unit-by-unit switch go: every route file is held
to zero database work, every unit's package to its edges, and core code says
whose a row is through userscope too. CLAUDE.md, the backend README and the
gate register describe the units as they now are.
2026-10-19 23:36:01 +02:00
peter ad6ccbac05 refactor(core): the core's owner filters go through userscope, each held by a second account
Tokens, notifications, quota, debug sessions and the reset-link burn filtered
their owner inline. Ten of the fifteen filters had no test that noticed one
going missing; each now has a test where another account's rows must not
count, show or change.
2026-10-19 23:36:01 +02:00
peter 531eb2eef1 refactor(plex): playlists, playback, Discover and admin through the unit; the plex package is fenced
The last Plex HTTP files do no database work of their own, so ALLOWED is empty: no route builds
SQL, touches the session or a unit's table. Discover's rails move whole into plex/discover.py; a
user's playlists, watch-state writes (the race-safe finish, an early checkpoint, a status, a
season or show at once) and the watch-sync link go through plex/queries.py and plex/service.py.

The plex package joins FENCED: it reads no other unit's tables, says whose a row is through
userscope (the rollup and the watch sync's 8 inline filters moved to owned_by), and commits only in
its named jobs — the library sync, TMDB enrichment, the watch sync and its background pushes.
A playlist's has-item probe keeps the plain key lookup, so a NUL in it is still the 422 a test pins.

Pins 178 -> 0; basedpyright baseline 118 -> 106.
2026-10-19 22:42:46 +02:00
peter 2cf1b688d9 test(plex): a collection's editable flag and the mirror's reads, committed through HTTP
The editable endpoint had no route test: dropping its commit left every test green.
2026-10-19 22:11:57 +02:00
peter 3b0e365e26 refactor(plex): the browse, detail, people and collection pages read through queries and write through the service
The HTTP files browse, detail, people, collections and _shared do no database work of their own
now. New plex/queries.py (the rows a page shows, a user's watch state through userscope),
plex/service.py (the info pages' enrichment, a collection's editable flag, the row kept after a
failed follow-up read, the person bio's background fetch; none commits), plex/library.py (the
unified browse feed, its search and facets) and plex/collections.py (a collection's source,
editability, card, certificate and member order). Every watch-state join in the moved code names
its owner through owned_by.

Pins 354 -> 178 in 4 files; basedpyright baseline 132 -> 118.
2026-10-19 22:11:20 +02:00
peter 533571f572 refactor(plex): the unit's tables move into plex/models
PlexLibrary, PlexShow, PlexSeason, PlexItem, PlexPerson, PlexCollection, PlexState, PlexLink,
PlexPlaylist and PlexPlaylistItem leave app/models.py for app/plex/models.py; app.tables registers
them. Every import is re-pointed; no schema change.
2026-10-19 21:42:07 +02:00
peter 4b6db47083 test(catalog): the channel manager's local writes, committed through HTTP 2026-10-19 02:36:19 +02:00
peter 5229fd96be refactor(catalog): the HTTP layer reads through queries and writes through the service
Channels, search and sync do no database work in their routes any more: reads go through
catalog/queries, writes through catalog/service, and a live search's ingest is catalog/search.
What a catalog route needs from Reader (a playlist's channels, a user's tag links, the feed
projection of results) it asks reader/queries and reader/service. The scheduler and the runs page
read channel titles and the work queue through catalog/queries. Catalog is fenced below its HTTP
layer: its owner filters go through userscope, it reads no other unit's tables, and its ingest
jobs commit only where they are named.
2026-10-19 02:32:20 +02:00
peter 238e6f8a61 refactor(catalog): the unit's tables move into catalog/models
Channel, Subscription, Video, SearchFind, ExploredChannel and BlockedChannel (and LIVE_OR_UPCOMING)
leave app/models.py for app/catalog/models.py; app.tables registers them. The User↔Subscription
relationship names its other side as a type only. No schema change.
2026-10-19 02:03:14 +02:00
peter a35eece652 fix(e2e): the seed imports Reader's tables from reader/models 2026-10-19 01:33:32 +02:00
peter b703234c17 test(reader): the service tests hold each rule their mutant broke
A reorder's left-out item, a playlisted video beside one not asked about, another user's tag, and
the user's own channel link committed before the auto-tagger reads the table.
2026-10-19 01:28:38 +02:00
peter dbb4065e7b refactor(reader): the HTTP layer reads through queries and writes through the service
Feed, playlists, saved views, tags and the Home pins do no database work in their routes any more:
reads go through reader/queries (and feedquery for a feed page, its counts and facets), writes
through reader/service and discover/pins, which flush; the handlers commit. Reader is fenced below
its HTTP layer too: its owner filters go through userscope, and the catalog's jobs ask Reader
instead of joining its tables (the auto-tagger's system tags, the cleanup jobs' keep rule, the
maintenance notice). A playlist sync's catalog ingest moves to sync/videos.
2026-10-19 01:20:14 +02:00
peter e82f87acb0 refactor(reader): the unit's tables move into reader/models
VideoState, Playlist, PlaylistItem, SavedView, ComfortPin, Tag and ChannelTag leave app/models.py
for app/reader/models.py; app.tables registers them. No schema change.
2026-10-19 00:53:44 +02:00
peter e9c60f72cd refactor(capture): the HTTP layer reads through queries and writes through the service
Capture's routes (downloads, files, public watch links) and the token list no longer build SQL, call
the session or set a model's fields: reads are downloads/queries, writes the unit's service (service,
folders, links, quota), and each write flushes for the handler to commit. Whose a row is goes
through userscope (owned_by, owned_row), which owned_or_404 now uses too.

The fence holds Capture below its routes as well: it reads only Catalog's and Plex's tables, writes
no other unit's, filters owners through userscope and commits only in its jobs and the upload
ingest. A test per service use case, among them that the tray lists only the caller's files.
2026-10-19 00:19:03 +02:00
peter 3a688fa0f5 refactor(capture): the unit's tables move into downloads/models
The nine Capture tables (files, jobs, folders, shares, links, quota, cookies, profiles, library
resume points) leave app/models.py for the unit's own module; app.tables registers every unit's
tables for the migrations, the test database and the ownership check. No schema change.
2026-10-18 23:45:53 +02:00
peter d66174ed16 test(arch): a unit's HTTP layer does no database work beyond its pin
The fence is switched on unit by unit: each route file's database work (SQL built, session data
calls, model attributes set, unit tables imported) is pinned at today's count and may only shrink.
A core route may not reach into a unit's tables. Every model has an owner.
2026-10-18 23:01:12 +02:00
peter c85ae42b31 release: 0.108.0 (feature/e40-job-history) 2026-10-18 21:48:23 +02:00
peter cb9a89682a chore(release): 0.108.0 — one Activity page for runs, admin actions, the server log and quota; every background run has a history 2026-10-18 21:40:52 +02:00
peter b842e4e871 fix(ui): a looping pulse stops for reduced motion; the e2e waits for skeleton bones, not for any pulse
- Seven looping pulses (a running job's dot, the scheduler's progress bar, a queued download, the
  download tray, a playlist upload, a channel's full-history action, the setup wizard's test mail)
  ran regardless of prefers-reduced-motion, which docs/DESIGN.md promises to respect: each is
  motion-safe now, held by a contract test.
- The a11y suite's "settled" wait took every aria-hidden pulse for a loading skeleton, so the
  Scheduler legend's running dot, which pulses for good, kept it waiting until it timed out. A
  skeleton bone now says it is one (data-skeleton) and the wait looks for that.
2026-10-18 21:40:52 +02:00
peter 5082636f15 fix(activity): a run nothing here has open reads as interrupted, a rollback sums a stopped run's spend
- The scheduler keeps one registry of the runs this process has open, for the interval jobs and the
  ad-hoc runs alike. A row the table still says is `running` but no run here has open (its outcome
  failed to write) reads as interrupted, as the next start's sweep will close it: no pulsing dot
  beside "No run recorded yet".
- Migration 0109's downgrade sums a still-running run's quota events into its spend before the link
  to them is dropped, as the startup sweep would.
- A test holds that a dialog closing onto the same address leaves an open log line open.
2026-10-18 21:21:19 +02:00
peter 13b7577d9f fix(activity): an ad-hoc run's row is one run's, a rollback keeps a run whole, Back reads one entry's filters
- The channel-backfill row says "running" from the runs this process has open, not from the table:
  a row whose outcome failed to write no longer reads as running until a restart. The run still
  going leads the row, so a newer run's outcome never sits under the older one's channel.
- Migration 0109's downgrade carries every run's trigger, spend and start in the audit row, and the
  upgrade restores them: a manual run whose admin was deleted stays manual.
- Back/Forward is read by the page's one popstate listener, both halves in one render; the server
  log adopts its half during that render, so no read goes out with one entry's filters and the
  other's range.
- The server log's failed facets use the other tabs' words (the menus may be short); the accounts
  message shows only when the menu really lacks them.
2026-10-18 17:32:22 +02:00
peter fad656988a fix(activity): runs keep their true outcome and spend, the page reaches every kept row, a closing dialog leaves the server log alone
Backend
- A run's outcome write sits outside the job's own try: a database blip after a run that
  succeeded no longer records it as failed (the row waits for the next start's sweep).
- An interrupted run keeps what it spent (the sweep sums its quota rows, as finish does).
- The Scheduler's ad-hoc row says running while any backfill runs, not only the newest.
- recent_runs reads each job's last runs off its index instead of ranking the whole table.
- An interval change is stored with its audit row in one transaction, then rescheduled.
- One API instance per database is the stated contract (.env.example, jobruns).
- Turning full history on for a channel already complete records no run.
- Migration 0109 round-trips interrupted and ad-hoc runs; the run words' DB CHECKs are
  test-held to the enums; the account filter is one helper for the four activity reads.
- An audit row says how many of its request's log lines are still kept.

Frontend
- An "All" period reaches every row still kept (runs and admin actions outlive 90 days).
- Back/Forward is followed in place by the server log: a closing dialog no longer turns
  Live off; a pivot still opens a fresh log.
- Runs moved over from the audit log read their one-liner in the reader's language.
- Filter choices or accounts that failed to load say so beside the filters, with a retry.
- An old server-log link keeps its account, text and range; a remembered retired page
  opens its tab.
- A request whose lines are gone says so instead of linking to an empty list.
- The log lines and the history rows share one row frame, and the server log one list
  frame with the other tabs; an empty list is announced; each day's quota is readable.
2026-10-18 15:38:32 +02:00
peter 3021be5166 fix(activity): a run's quota reads with its unit, its outcome keeps room at a large text size, a token's audit rows read in Hungarian
The run list showed the quota spend as a bare figure and, at the largest text size on a 1024-wide
screen, its fixed columns left the outcome column no width at all. The outcome now keeps 10rem and
the trigger column gives way; the audit list's summary keeps the same floor. A run that stored no
log line says so instead of linking to an empty list. An API token revoke and a token's file work
rendered the server's English line under the Hungarian UI; both are rebuilt in the reader's
language, and the test holds the server's wording.
2026-10-18 04:22:52 +02:00
peter c288ba147b fix(activity): an old audit/log link opens its tab, times read relative, one name for admin actions
Seen in the browser, not in the page's own tests: the navigation provider rewrites a retired page id
to `activity` before the page's chunk reads the URL, so `?page=audit` landed on the Runs tab. The
retired-page map now carries the params that open the successor's part (`at`), and the provider
writes them as it rewrites the page — on load and on Back alike.

A list time is relative with the exact second on hover (the DESIGN copy rule); a run moved over from
the audit log, whose start and end are one instant, shows no length instead of an invented "1 sec".
The audit log is called "admin actions" everywhere it is named (the clear button and its confirm, the
Configuration group, the cleanup job, which also prunes the run history its description left out).
2026-10-18 04:00:38 +02:00
peter fe6308d844 refactor(activity): the window's hours stay private to its URL state 2026-10-18 03:48:30 +02:00
peter 24448e7c6f feat(activity): one Activity page — runs, admin actions, server log and quota under one filter bar
The audit log, the server log and the admin quota view were three pages (the quota one a Stats tab),
each with its own filters, and the background runs had no list at all. Now one page ("Napló" /
"Activity") reads all four, each still its own table on the server: a shared bar (text, a period of
1 h to 90 days defaulting to 7, an account) and each tab's own filters, the whole view in the URL.
The old page ids open their tab.

Server side: the audit log and the new run list (`/api/admin/runs`) filter and page like the server
log, through one keyset helper (`routes/_keyset`) the log now shares; an audit row keeps the request
it was made in (migration 0111), so its log lines are a pivot away, as a run's are (`lrun`). The
quota breakdown takes since/until and one account. The Scheduler rows get a history button to their
job's runs.

Also fixed on the way: re-saving the value in force on the Configuration page staged its override
without committing it (the commit sat inside the audit branch), so the save was lost.
2026-10-18 03:45:18 +02:00
peter d13483a593 feat(config): job intervals live in sysconfig, every setting says when it applies, the unread SponsorBlock default goes
An interval override was a row of its own table beside the one store every other admin knob uses; it
is now the `job_interval_<job>` sysconfig key (migration 0110 copies them over and drops the table).
The Scheduler page stays its one editor: the Configuration page and the setup wizard neither list nor
write those keys, since a write there would skip the reschedule.

Each ConfigSpec now states `applies` (required, no default). Every key is `live`, held by a scan that
finds no read at import time; a `restart` key is refused until the page can say so. A second scan
requires a reader for every key on the Configuration page — `sponsorblock_default` had none since it
was added, so it leaves the registry, the translations and the store.
2026-10-18 02:23:15 +02:00
peter 3bab2e7dc7 docs(scheduler): the snapshot docstring names the ad-hoc runs it now carries 2026-10-18 01:29:46 +02:00
peter 1b1e3cdcf2 feat(scheduler): each row shows its last three runs, an interrupted run as such, and the ad-hoc runs beside the jobs
A row's one dot said how the LAST run went, and an interrupted run fell through to "idle" — a grey dot
that read as "no run recorded". The row now leads with its last three runs, newest first, each named
for a screen reader and in its tooltip (how it went, when, who started it); an interrupted run is a
ring, set apart from a skipped run and a failure, and its line says the server restarted under it.

A channel's full-history backfill gets its own "Ad-hoc runs" block: its runs, its channel and what
it fetched, no interval and no run button — it needs its target. AD_HOC_JOBS names every such job;
the backend-id contract holds their names in both languages, and run_ad_hoc refuses any other id.
2026-10-18 01:20:53 +02:00
peter 27c2fcbd2e feat(scheduler): a channel's "get full history" backfill is a recorded run, and a row knows its last three runs
The one-channel deep backfill ran as a FastAPI BackgroundTask: what it fetched, what it spent and
how it failed reached only the log. It is now a one-shot run on the scheduler (`channel_backfill`,
trigger ad_hoc, actor = the user, target = the channel) with its quota tagged to the run. Its
per-channel exclusion with the scheduled backfill and the two-run cap are unchanged, and a run
they turn away says why in its result instead of returning nothing. A failure is no longer
swallowed: the run records it.

The scheduler snapshot carries each job's last three runs (one windowed query), so the page can
show an interrupted or skipped run instead of the latest outcome alone.
2026-10-18 01:05:24 +02:00
peter 911527a548 fix(scheduler): the dashboard no longer says a job has not run "this session" — runs are kept across restarts 2026-10-17 23:22:13 +02:00
peter 8fd91b7b15 fix(scheduler): keep run history 180 days, as long as the audit log kept the runs it moved over 2026-10-17 23:15:17 +02:00
peter 45c0ee7d58 feat(scheduler): every background run is a durable row
What a job last did lived in the API process's memory and was gone at
every restart; the audit log kept a run only when it was manual, failed
or changed something. Now each run is a job_runs row written at its
start and its end in its own session: who started it, its outcome as
data, what it spent of the YouTube quota (quota_events.job_run_id) and
the lines it logged (log_events.run_id, filterable in the log console).

- a run a stopped process left open is marked interrupted at startup
- "run now" is a one-shot job on the scheduler, which now always starts
  (interval jobs only with SCHEDULER_ENABLED); the manual actor is reset
  after the run, since pool threads are reused
- the dashboard's last run comes from job_runs, so it survives a deploy
- runs leave the audit log: migration 0109 moves the old job_run rows
  over (and back on downgrade); the audit page's run folding goes
- job_run_retention_days (default 30) prunes old runs in audit_gc
2026-10-17 23:08:55 +02:00
303 changed files with 16621 additions and 8214 deletions
+5 -3
View File
@@ -68,9 +68,11 @@ SMTP_FROM=
# --- Scheduler ---
# The background scheduler (subscription sync, backfill, enrichment) runs inside the app.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. If you ever run
# more than one instance against the same database, keep SCHEDULER_ENABLED=true on exactly one
# of them and false on the rest, to avoid double quota use and write races.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. Run ONE app instance
# per database: a starting instance closes every run still open as interrupted (it assumes the
# previous process is gone), and "is this job already running" is held in the process. A second
# instance would cut the first one's runs short and could run the same job twice.
# SCHEDULER_ENABLED=false keeps the recurring jobs off; "run now" still works.
SCHEDULER_ENABLED=true
# --- Files library & downloads ---
+24
View File
@@ -7,6 +7,30 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.108.1 — 2026-10-20
### New
- Development: the backend is split into four units (downloads, reader, catalog, Plex), each owning its tables. A contract test holds what each one reads and writes, and that every query names whose rows it asks for.
### Fixed
- Watch later: saving the first two videos at once no longer makes two lists. An existing second list is merged into the first, its videos and its comfort-shelf pin with it.
- Playback: the first resume point of a video or a Plex item no longer fails when it is saved together with the watched mark, a whole season marked at once, or a Plex sync.
## 0.108.0 — 2026-10-18
### New
- Activity: one admin page for background runs, admin actions, the server log and quota, with one time range, account and search. A run opens to its outcome and its log lines, an admin action to its request's lines; old audit and log links open their tab.
- Scheduler: each job shows its last three runs (a run cut short by a restart is marked) and a button opens all of them; turning on a channel's full history is now a run of its own, listed under Ad-hoc runs.
- Configuration: how long the run history is kept (Activity › Run history, 180 days by default). Stats keeps your own numbers; the system quota view moved to Activity › Quota.
### Fixed
- Configuration: saving a setting with the value already in force now keeps it, instead of dropping the change.
- Configuration: the unused SponsorBlock default setting is gone.
## 0.107.3 — 2026-10-17
### New
+38 -31
View File
@@ -63,9 +63,16 @@ how it gets built. Concretely:
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`,
`downloads/`, `reader/`, `discover/`, `catalog/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
The domain is four **units**, each owning its tables: Capture (`downloads/`), Reader (`reader/`, `discover/`, the playlist
sync), Catalog (`catalog/`, the `sync/` ingests) and Plex (`plex/`). A unit's route does no database work: it reads through
the unit's `queries`, writes through its `service` (which flushes) and commits. A unit's code reads another unit's tables
only along a declared edge and writes them only through that unit's service; core reaches a unit's tables through its
`queries`. **Fence-held, with no exemptions** (`backend/tests/test_unit_fences.py`) — as far as an AST sees: what it
cannot (an ORM write on another unit's loaded row, SQL in `text()`, a commit by `begin()`) is review's to catch, listed in
[`backend/app/README.md`](backend/app/README.md) with the conventions.
Frontend: `main.tsx` provider tree → `App.tsx` shell (a shared `/watch` link boots from its own
entry, `watch.html` → `watchMain.tsx`, held app-free by a contract test) → `moduleRegistry` route-chunks → module
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
@@ -82,36 +89,36 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
## Canonical patterns — when you have a choice, pick THIS
| Area | Use | Not |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect | a bare `catch {}` or `notify()` with no error attached |
| FE floating or full-screen layer | `ui/Overlay` (portals to `<body>`) or the popover engine (`AnchoredPopover` portals itself); stacking by the z-index scale (`z-menu` … `z-popover`, docs/DESIGN.md "Layers") | one rendered in place inside a virtualized row or a masked/transformed scroller — clipped or painted over; a magic `z-[n]` |
| FE player on a native media element | `lib/mediaSession` (`useMediaSession`) + `lib/mediaKeys` + `lib/resume` (`useResume`), the element held as STATE | a private `navigator.mediaSession` / `keydown` / position-saving effect per player |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+); a handler or timer: `useLatest` / `useStableCallback` | a render-phase `ref.current = x` — the React Compiler skips the component |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
| FE popover / scroll-fade ref | destructure it: `const { setReference } = pop.refs`, then `ref={setReference}` | `ref={pop.refs.setReference}` / `ref={fade.ref}` — the compiler then treats the whole object as a ref and skips |
| FE async handler with cleanup | `try { … } catch { … }`, then the cleanup after it (steps that `return` early go in their own function); a conditional, loop or `?.` the `try` needs goes before it or into a module-level helper | `try/finally`, or a `?:` / `??` / loop / `?.` inside `try` — the React Compiler skips the whole component (and `react-hooks/todo` misses most of these) |
| Area | Use | Not |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `userscope.owned_by` (a filter) / `owned_row` (a row by id): a unit's `queries.own_x` returns None and its route answers `not_found`; a core route `owned_or_404`. A composite rule (own or shared, built-in or own) says so in its `queries` | an inline owner filter — `Model.user_id == …`, an alias's, `.in_()`, `filter_by(user_id=…)` (fence-held, every module) — or a `db.get` and a compare where `owned_row` fits |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; a unit's `service` and every helper `flush` (fence-held, every module) | a helper that commits — only a listed one (what owns its session, or a core step that must land whatever its caller does next), each with its reason: the fence's `UNIT_COMMITS` / `CORE_COMMITS`, a route's `COMMITTING_HELPERS` (`test_route_conventions`) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect | a bare `catch {}` or `notify()` with no error attached |
| FE floating or full-screen layer | `ui/Overlay` (portals to `<body>`) or the popover engine (`AnchoredPopover` portals itself); stacking by the z-index scale (`z-menu` … `z-popover`, docs/DESIGN.md "Layers") | one rendered in place inside a virtualized row or a masked/transformed scroller — clipped or painted over; a magic `z-[n]` |
| FE player on a native media element | `lib/mediaSession` (`useMediaSession`) + `lib/mediaKeys` + `lib/resume` (`useResume`), the element held as STATE | a private `navigator.mediaSession` / `keydown` / position-saving effect per player |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+); a handler or timer: `useLatest` / `useStableCallback` | a render-phase `ref.current = x` — the React Compiler skips the component |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
| FE popover / scroll-fade ref | destructure it: `const { setReference } = pop.refs`, then `ref={setReference}` | `ref={pop.refs.setReference}` / `ref={fade.ref}` — the compiler then treats the whole object as a ref and skips |
| FE async handler with cleanup | `try { … } catch { … }`, then the cleanup after it (steps that `return` early go in their own function); a conditional, loop or `?.` the `try` needs goes before it or into a module-level helper | `try/finally`, or a `?:` / `??` / loop / `?.` inside `try` — the React Compiler skips the whole component (and `react-hooks/todo` misses most of these) |
Not repeated here, because a lint rule refuses the wrong one and its message names the right one:
buttons (`ui/Button`, `IconButton`, `Pressable`, `Chip`, `DragHandle`), text fields and native
+1 -1
View File
@@ -1 +1 @@
0.107.3
0.108.1
-492
View File
@@ -630,41 +630,7 @@
}
}
],
"./app/routes/admin_logs.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 15,
"endColumn": 47,
"lineCount": 1
}
}
],
"./app/routes/audit.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 17,
"endColumn": 98,
"lineCount": 1
}
},
{
"code": "reportAssignmentType",
"range": {
"startColumn": 17,
"endColumn": 98,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 22,
"endColumn": 97,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
@@ -674,48 +640,6 @@
}
}
],
"./app/routes/channels.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 11,
"endColumn": 11,
"lineCount": 6
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 15,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 49,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 38,
"endColumn": 44,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 33,
"endColumn": 39,
"lineCount": 1
}
}
],
"./app/routes/discover.py": [
{
"code": "reportInvalidTypeForm",
@@ -726,277 +650,7 @@
}
}
],
"./app/routes/downloads.py": [
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 17,
"endColumn": 19,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 82,
"endColumn": 84,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 87,
"endColumn": 89,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 30,
"endColumn": 32,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 49,
"endColumn": 54,
"lineCount": 1
}
}
],
"./app/routes/playlists.py": [
{
"code": "reportOptionalOperand",
"range": {
"startColumn": 8,
"endColumn": 9,
"lineCount": 5
}
},
{
"code": "reportOptionalOperand",
"range": {
"startColumn": 8,
"endColumn": 9,
"lineCount": 5
}
},
{
"code": "reportCallIssue",
"range": {
"startColumn": 13,
"endColumn": 5,
"lineCount": 7
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 15,
"lineCount": 5
}
},
{
"code": "reportCallIssue",
"range": {
"startColumn": 16,
"endColumn": 5,
"lineCount": 11
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 15,
"lineCount": 9
}
},
{
"code": "reportCallIssue",
"range": {
"startColumn": 13,
"endColumn": 5,
"lineCount": 5
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 15,
"lineCount": 3
}
},
{
"code": "reportCallIssue",
"range": {
"startColumn": 12,
"endColumn": 31,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 23,
"endColumn": 27,
"lineCount": 1
}
},
{
"code": "reportCallIssue",
"range": {
"startColumn": 12,
"endColumn": 34,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 26,
"endColumn": 30,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 12,
"endColumn": 28,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 23,
"endColumn": 27,
"lineCount": 1
}
}
],
"./app/routes/plex/browse.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 26,
"endColumn": 61,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 26,
"endColumn": 61,
"lineCount": 1
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 11,
"endColumn": 13,
"lineCount": 1
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 25,
"endColumn": 27,
"lineCount": 1
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 11,
"endColumn": 13,
"lineCount": 1
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 25,
"endColumn": 27,
"lineCount": 1
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 15,
"endColumn": 17,
"lineCount": 1
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 28,
"endColumn": 30,
"lineCount": 1
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 15,
"endColumn": 17,
"lineCount": 1
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 28,
"endColumn": 30,
"lineCount": 1
}
}
],
"./app/routes/plex/collections.py": [
{
"code": "reportAssignmentType",
"range": {
"startColumn": 11,
"endColumn": 39,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 76,
"endColumn": 80,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 75,
"endColumn": 79,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 80,
"endColumn": 84,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
@@ -1054,117 +708,7 @@
}
}
],
"./app/routes/plex/discover.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 4,
"endColumn": 63,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 18,
"endColumn": 48,
"lineCount": 1
}
},
{
"code": "reportCallIssue",
"range": {
"startColumn": 4,
"endColumn": 63,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 18,
"endColumn": 48,
"lineCount": 1
}
}
],
"./app/routes/plex/playback.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 51,
"endColumn": 54,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 51,
"endColumn": 54,
"lineCount": 1
}
}
],
"./app/routes/plex/playlists.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 77,
"endColumn": 83,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 77,
"endColumn": 83,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 9,
"lineCount": 5
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 9,
"lineCount": 5
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 10,
"endColumn": 18,
"lineCount": 1
}
}
],
"./app/routes/scheduler.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 42,
"endColumn": 55,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
@@ -1182,42 +726,6 @@
}
}
],
"./app/routes/search.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 25,
"endColumn": 31,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
}
],
"./app/routes/sync.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 42,
"endColumn": 65,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 42,
"endColumn": 50,
"lineCount": 1
}
}
],
"./app/sync/explore.py": [
{
"code": "reportAttributeAccessIssue",
+1 -1
View File
@@ -4,7 +4,7 @@ from sqlalchemy import engine_from_config, pool
from app.config import settings
from app.db import Base
from app.logs import configure_logging
import app.models # noqa: F401 (import so models register on Base.metadata)
import app.tables # noqa: F401 (import so every table registers on Base.metadata)
config = context.config
config.set_main_option("sqlalchemy.url", settings.database_url)
+160
View File
@@ -0,0 +1,160 @@
"""Background runs get a durable history
Revision ID: 0109_job_runs
Revises: 0108_folder_links
Create Date: 2026-10-17
What a scheduler job last did lived in the API process's memory, gone at every restart; the audit
log kept a row for a run only when it was manual, failed, or changed something. Now every run is a
`job_runs` row, written when it starts and when it ends. A run's quota spend points at it
(`quota_events.job_run_id`), and so do the lines it logged (`log_events.run_id`, no foreign key: the
log writer runs behind the event).
Runs are not admin actions, so the audit log stops carrying them: the existing `job_run` audit rows
MOVE into `job_runs` (start = end = when the row was written, the only time it kept), and leave the
audit log. The downgrade moves every run back into the audit log as such a row (one still running as
interrupted, what the audit row has no column for in its `after`), then drops the table.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0109_job_runs"
down_revision: str | None = "0108_folder_links"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"job_runs",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("job", sa.String(40), nullable=False),
sa.Column("trigger", sa.String(16), nullable=False),
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="SET NULL")),
sa.Column("target", sa.String(128)),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True)),
sa.Column("summary", sa.Text()),
sa.Column("result", postgresql.JSONB()),
sa.Column("error", sa.Text()),
sa.Column("failure", postgresql.JSONB()),
sa.Column("quota_units", sa.Integer(), nullable=False, server_default="0"),
sa.CheckConstraint(
"status IN ('running', 'ok', 'error', 'skipped', 'interrupted')",
name="ck_job_runs_status",
),
sa.CheckConstraint(
"trigger IN ('scheduled', 'manual', 'ad_hoc')", name="ck_job_runs_trigger"
),
sa.CheckConstraint(
"(status = 'running') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
op.create_index("ix_job_runs_job_started", "job_runs", ["job", "started_at"])
op.create_index("ix_job_runs_actor_id", "job_runs", ["actor_id"])
op.create_index("ix_job_runs_started_at", "job_runs", ["started_at"])
op.add_column(
"quota_events",
sa.Column("job_run_id", sa.BigInteger(), sa.ForeignKey("job_runs.id", ondelete="SET NULL")),
)
op.create_index("ix_quota_events_job_run_id", "quota_events", ["job_run_id"])
op.add_column("log_events", sa.Column("run_id", sa.BigInteger()))
op.create_index(
"ix_log_events_run",
"log_events",
["run_id"],
postgresql_where=sa.text("run_id IS NOT NULL"),
)
# The audit trail's runs move over: the three statuses the scheduler wrote, and `interrupted`,
# which a downgrade writes back (as it does a run's trigger, target, spend and start, which the
# audit row has no column for: its actor alone cannot say "manual" once that admin is deleted),
# so a rollback and a redeploy keep every run as it was. A status the run table does
# not know would fail the CHECK.
op.execute(
"""
INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at, finished_at,
summary, result, error, failure, quota_units)
SELECT target_id,
coalesce(after::jsonb ->> 'trigger',
CASE WHEN actor_id IS NULL THEN 'scheduled' ELSE 'manual' END),
actor_id,
after::jsonb ->> 'target',
after::jsonb ->> 'status',
coalesce((after::jsonb ->> 'started_at')::timestamptz, created_at),
created_at,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb -> 'result' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb -> 'result' END,
coalesce((after::jsonb ->> 'quota_units')::int, 0)
FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
ORDER BY id
"""
)
op.execute(
"""
DELETE FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
"""
)
def downgrade() -> None:
# A run still `running` is one the rollback stopped: it goes back as interrupted. What the audit
# row's own columns cannot say rides in `after` (its trigger, target, spend and start), so the
# upgrade restores them.
op.execute(
"""
INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after, created_at)
SELECT actor_id,
'job_run',
'job',
job,
left(job || ': ' || st
|| coalesce(' — ' || CASE WHEN st = 'error' THEN error ELSE summary END, ''),
255),
(jsonb_build_object(
'status', st,
'summary', CASE WHEN st = 'error' THEN error ELSE summary END)
|| CASE
WHEN st = 'error' AND failure IS NOT NULL
THEN jsonb_build_object('result', failure)
WHEN st <> 'error' AND result IS NOT NULL
THEN jsonb_build_object('result', result)
ELSE '{}'::jsonb
END
|| jsonb_build_object('trigger', trigger, 'quota_units', units,
'started_at', started_at)
|| CASE WHEN target IS NOT NULL
THEN jsonb_build_object('target', target) ELSE '{}'::jsonb END)::json,
coalesce(finished_at, started_at)
FROM (SELECT *, CASE WHEN status = 'running' THEN 'interrupted' ELSE status END AS st,
-- A run stopped mid-way has not summed its spend yet: sum it as the startup
-- sweep would, before the link to it is dropped.
CASE WHEN status = 'running'
THEN (SELECT coalesce(sum(q.units), 0) FROM quota_events q
WHERE q.job_run_id = job_runs.id)
ELSE quota_units END AS units
FROM job_runs) AS r
ORDER BY id
"""
)
op.drop_index("ix_log_events_run", table_name="log_events")
op.drop_column("log_events", "run_id")
op.drop_index("ix_quota_events_job_run_id", table_name="quota_events")
op.drop_column("quota_events", "job_run_id")
op.drop_index("ix_job_runs_started_at", table_name="job_runs")
op.drop_index("ix_job_runs_actor_id", table_name="job_runs")
op.drop_index("ix_job_runs_job_started", table_name="job_runs")
op.drop_table("job_runs")
@@ -0,0 +1,79 @@
"""Job intervals move into sysconfig; the unread SponsorBlock default goes
Revision ID: 0110_intervals_in_sysconfig
Revises: 0109_job_runs
Create Date: 2026-10-18
An admin's interval override lived in its own table, `scheduler_settings`, beside the one store every
other admin knob uses. Each override becomes the `job_interval_<job>` sysconfig key and the table
goes. Only the jobs that still exist are copied: a row for any other id never did anything.
The `sponsorblock_default` setting was offered on the Configuration page but nothing ever read it, so
its key is gone and its stored value with it — the downgrade cannot bring that value back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0110_intervals_in_sysconfig"
down_revision: str | None = "0109_job_runs"
branch_labels: str | None = None
depends_on: str | None = None
# The interval jobs as of this revision (a migration does not read the live code's list).
_JOBS = (
"rss_poll",
"enrich",
"backfill",
"autotag",
"shorts",
"subscriptions",
"playlist_sync",
"maintenance",
"explore_cleanup",
"download_gc",
"plex_sync",
"plex_watch_sync",
"plex_watch_reconcile",
"tmdb_enrich",
"people_enrich",
"audit_gc",
"log_gc",
)
_PREFIX = "job_interval_"
def upgrade() -> None:
op.execute(
sa.text(
"INSERT INTO system_config (key, value, encrypted)"
" SELECT :prefix || job_id, interval_minutes::text, false FROM scheduler_settings"
" WHERE interval_minutes IS NOT NULL AND job_id IN :jobs"
" ON CONFLICT (key) DO NOTHING"
).bindparams(sa.bindparam("jobs", expanding=True), prefix=_PREFIX, jobs=list(_JOBS))
)
op.drop_table("scheduler_settings")
op.execute("DELETE FROM system_config WHERE key = 'sponsorblock_default'")
def downgrade() -> None:
op.create_table(
"scheduler_settings",
sa.Column("job_id", sa.String(40), primary_key=True),
sa.Column("interval_minutes", sa.Integer(), nullable=False),
)
op.execute(
sa.text(
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
" SELECT substr(key, :start), value::int FROM system_config WHERE key IN :keys"
).bindparams(
sa.bindparam("keys", expanding=True),
start=len(_PREFIX) + 1,
keys=[_PREFIX + j for j in _JOBS],
)
)
op.execute(
sa.text("DELETE FROM system_config WHERE key IN :keys").bindparams(
sa.bindparam("keys", expanding=True), keys=[_PREFIX + j for j in _JOBS]
)
)
@@ -0,0 +1,28 @@
"""An audit row keeps the request that made it
Revision ID: 0111_audit_request_id
Revises: 0110_intervals_in_sysconfig
Create Date: 2026-10-18
The activity page opens an admin action's log lines from its row: `audit_log.request_id` is the
request the change was made in, the same id `log_events.request_id` carries. Older rows have none —
the request that made them was never recorded.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0111_audit_request_id"
down_revision: str | None = "0110_intervals_in_sysconfig"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("audit_log", sa.Column("request_id", sa.String(64)))
op.create_index("ix_audit_log_request_id", "audit_log", ["request_id"])
def downgrade() -> None:
op.drop_index("ix_audit_log_request_id", table_name="audit_log")
op.drop_column("audit_log", "request_id")
@@ -0,0 +1,70 @@
"""One Watch later list per user, held by the database
Revision ID: 0112_one_watch_later
Revises: 0111_audit_request_id
Create Date: 2026-10-20
Watch later was made on a user's first save by a read then an insert, so two first saves at once
each made a list, and the reader then picked between them at random. A partial unique index holds
"one per user" now. An instance that already has a second list keeps the oldest: the others' videos
move into it (after its own, in their order; one it already holds is not added twice), a Home pin
of one of them moves to it (unless it is pinned itself; of several, the first on the shelf), then
they go — a pin left on them goes with them, as the kept list already holds one.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0112_one_watch_later"
down_revision: str | None = "0111_audit_request_id"
branch_labels: str | None = None
depends_on: str | None = None
_EXTRA = """
SELECT p.id, k.keep_id
FROM playlists p
JOIN (
SELECT user_id, min(id) AS keep_id FROM playlists WHERE kind = 'watch_later' GROUP BY user_id
) k ON k.user_id = p.user_id
WHERE p.kind = 'watch_later' AND p.id <> k.keep_id
"""
def upgrade() -> None:
op.execute(
f"""
INSERT INTO playlist_items (playlist_id, video_id, position, added_at)
SELECT e.keep_id, i.video_id,
(SELECT coalesce(max(position), -1) FROM playlist_items WHERE playlist_id = e.keep_id)
+ row_number() OVER (PARTITION BY e.keep_id ORDER BY i.playlist_id, i.position, i.id),
i.added_at
FROM playlist_items i
JOIN ({_EXTRA}) e ON e.id = i.playlist_id
ON CONFLICT ON CONSTRAINT uq_playlist_video DO NOTHING
"""
)
op.execute(
f"""
UPDATE comfort_pins c SET playlist_id = m.keep_id
FROM (
SELECT DISTINCT ON (e.keep_id) p.id AS pin_id, e.keep_id
FROM comfort_pins p
JOIN ({_EXTRA}) e ON e.id = p.playlist_id
WHERE NOT EXISTS (SELECT 1 FROM comfort_pins k WHERE k.playlist_id = e.keep_id)
ORDER BY e.keep_id, p.position, p.id
) m
WHERE c.id = m.pin_id
"""
)
op.execute(f"DELETE FROM playlists WHERE id IN (SELECT id FROM ({_EXTRA}) e)")
op.create_index(
"uq_playlist_watch_later",
"playlists",
["user_id"],
unique=True,
postgresql_where=sa.text("kind = 'watch_later'"),
)
def downgrade() -> None:
op.drop_index("uq_playlist_watch_later", table_name="playlists")
+61 -19
View File
@@ -1,9 +1,8 @@
# Backend conventions
The rules below are already followed across `backend/app` — they lived only in a few module
docstrings (`userscope.py`, `routes/_common.py`) until this file collected them (E3 S3,
B-core-20). When you add code, match these so the next reader doesn't have to guess "which pattern do
I copy".
docstrings (`userscope.py`, `routes/_common.py`) until this file collected them. When you add code,
match these so the next reader doesn't have to guess "which pattern do I copy".
## Module layering
@@ -13,15 +12,17 @@ Three layers, by *who* runs the code — not by feature:
response models, status-code mapping, serialization. It owns logic that runs *only* on a request.
Business logic that a request happens to trigger does **not** live here if anything else runs it too.
2. **`plex/`, `downloads/`, `sync/`, `issues/` — domain logic shared by more than one caller.** Anything the
**scheduler or the worker** runs (not just a request) lives in one of these domain packages, so the
route, the worker, and the scheduler share one definition. Examples from this codebase:
`downloads/service.py` owns enqueue + the job state transitions (routes call it); `plex/sync.py` is
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline;
`issues/bundle.py` is the one definition of a bug report's files, for the zip export and the tracker.
2. **The units and the other domain packages — logic shared by more than one caller.** A unit's
database work (below), and anything the **scheduler or the worker** runs (not just a request), lives
in a domain package, so the route, the worker, and the scheduler share one definition. Examples from
this codebase: `downloads/service.py` owns enqueue + the job state transitions (routes call it);
`plex/sync.py` is run by both the admin route and the scheduler; `downloads/pipeline.py` is the
worker's child pipeline; `issues/bundle.py` is the one definition of a bug report's files, for the
zip export and the tracker.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth` (with `apitokens`, a machine's
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`,
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`, `jobruns` (every
background run's durable row, its words in `runstatus`),
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
@@ -33,19 +34,59 @@ Three layers, by *who* runs the code — not by feature:
a layer — into the relevant domain package, or a flat core module if it's cross-cutting. Don't
pre-abstract for a single caller; don't leave shared logic duplicated once there are two.
## Units
The domain is four units, each owning its tables (`models.py` in its package):
| Unit | Package | Its routes |
| --- | --- | --- |
| Capture | `downloads/`, the worker | `downloads.py`, `files.py`, `public.py` |
| Reader | `reader/`, `discover/`, `feedquery.py`, the playlist sync and tag suggestions in `sync/` | `feed.py`, `playlists.py`, `saved_views.py`, `discover.py`, `tags.py` |
| Catalog | `catalog/`, the ingests in `sync/` | `channels.py`, `search.py`, `sync.py` |
| Plex | `plex/` | `plex/` |
- **A unit's route does no database work.** It validates, reads through the unit's `queries.py`,
writes through its `service.py`, serializes, and commits. It may name a unit's model as a type
(under `TYPE_CHECKING`), never use one as a value, and it passes a kind (`"movie"`), not a model, to
a query.
- **A unit's code keeps to its edges.** It reads another unit's tables only along a declared edge
(Capture and Reader read Catalog's and Plex's: the feed joins `Video` with `VideoState` in one
query) and writes them only through the owner's service. Core code reaches a unit's tables through
that unit's `queries`.
- **Whose a row is, `userscope` says** — in every module, core included: `owned_by(Model.user_id,
user)` for a filter, `owned_row(db, Model, id, user)` for a row by id (someone else's row is the
same answer as a missing one). A unit's `queries.own_x` returns None and its route answers
`not_found`; a core route calls `owned_or_404`. A rule `owned_row` cannot say — a job its owner
shared, a built-in profile nobody owns, a folder read under `FOR SHARE` — is written out in the
unit's own code, beside the reason.
All of it is held by `tests/test_unit_fences.py`, with no exemptions: the unit map, each unit's
packages and edges (`UNIT_PACKAGES`, `READS`) and what may commit (`UNIT_COMMITS`, `CORE_COMMITS`)
are listed there. It reads the code's AST, so it holds the forms written here: an owner column
compared inline (a model's, an import alias's or an `aliased()` one's, by `==`, `!=`, `.in_()`, or a
`filter_by(user_id=…)`), a model built or an `insert`/`update`/`delete` (under any import name, of a
model or its `__table__`) across units, and a `.commit()` call off the lists all fail it. What an AST
cannot tell is review's to catch: an ORM write on another unit's loaded row (an attribute, a
`db.delete`), SQL in `text()`, a column or model reached another way (bound to a variable, through
`__table__.c` or a module path), and a commit made by a `begin()` block or a `commit` handed in.
## Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
to the **operation owner**; anything called *inside* another transaction only stages its change
This follows CLAUDE.md's canonical rule — **the handler commits; a unit's `service` and every helper
flush** — so read it that way, not as an exception to it. The commit belongs to the **operation
owner**; anything called *inside* another transaction only stages its change
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
mutation it records, so an audit entry can never exist without its change (or vice-versa).
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
- The **operation owner** is the route handler, or a job. A unit's `service` flushes even when it
makes a whole transition (`service.cancel`): the handler commits it, together with whatever else the
request changed. What owns its OWN session commits it: `quota` and `jobruns`' run rows, the
scheduler's jobs, a unit's jobs (the worker's `jobdb` writes, the syncs and ingests that commit per
page or batch so a later failure loses nothing already stored), and a few core steps that must land
whatever their caller does next (a token's last-used stamp, a refreshed YouTube grant) — the fence's
`UNIT_COMMITS` and `CORE_COMMITS` name each with its reason, `test_route_conventions` a route's own
committing helpers. Everything else flushes.
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
attribute — see `service.decr_ref_count`.
@@ -67,8 +108,9 @@ mutation it records, so an audit entry can never exist without its change (or vi
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
startup, not user-tunable at runtime.
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
slated to fold into `sysconfig` — do not add more like it.)
goes in `SPECS`; don't invent a fourth store. Each spec states when a change takes effect
(`applies`, required): `live` keys are read where they are used, never at import (test-held). The
scheduler's job intervals are `SPECS` keys too, edited on the Scheduler page, not the Configuration page.
## The gate
+4 -1
View File
@@ -26,6 +26,7 @@ from fastapi import Depends, Request
from sqlalchemy import delete, or_, select, update
from sqlalchemy.orm import Session
from app.userscope import owned_by
from app import audit, debugsession, logs
from app.audit import AuditAction
from app.auth import current_user
@@ -223,7 +224,9 @@ def revoke_all(db: Session, user_id: int) -> int:
change, "log out everywhere", a revoked untrusted password — ends its tokens too: one minted
from a stolen session must not outlive the owner's answer to that theft. Returns how many."""
gone = (
db.execute(delete(ApiToken).where(ApiToken.user_id == user_id).returning(ApiToken.id))
db.execute(
delete(ApiToken).where(owned_by(ApiToken.user_id, user_id)).returning(ApiToken.id)
)
.scalars()
.all()
)
+8 -9
View File
@@ -1,9 +1,9 @@
"""Admin/scheduler audit trail: an append-only record of who changed what and when.
"""Admin audit trail: an append-only record of who changed what and when.
Design (locked): log at ACTION / job-run granularity, NEVER per-row — a scheduler run that
touches thousands of videos is ONE audit row summarising the outcome. Automatic job runs are
only logged when they actually changed something (or errored); all admin actions and all manual
runs are always logged. Secrets are never stored (config changes log the key name only).
Design (locked): log at ACTION granularity, NEVER per-row — an action that touches thousands of
rows is ONE audit row summarising the outcome. Background runs are not actions: they have their
own history (app/jobruns.py). Secrets are never stored (config changes log the key name only).
Each row keeps the request that made it, so the request's log lines are one click away.
`record()` only db.add()s the row — the caller commits it in the SAME transaction as the
mutation it describes, so the trail can't drift from reality (and a rolled-back mutation drops
@@ -15,6 +15,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from sqlalchemy.orm import Session
from app import logs
from app.models import AuditLog
@@ -48,8 +49,6 @@ class AuditAction:
MAINTENANCE_TUNE = "maintenance_tune"
SYNC_PAUSE = "sync_pause"
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
@@ -104,16 +103,16 @@ def record(
summary=(summary or None) and summary[:255],
before=before,
after=after,
request_id=logs.current_request_id(),
)
)
def gc(db: Session, retention_days: int) -> int:
"""Delete audit rows older than `retention_days`. Returns the number removed. A
retention_days <= 0 disables pruning (keep forever)."""
retention_days <= 0 disables pruning (keep forever). Flushes; the job commits."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
result = db.execute(delete(AuditLog).where(AuditLog.created_at < cutoff))
db.commit()
return result.rowcount or 0
+2 -1
View File
@@ -4,6 +4,7 @@ from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from starlette.requests import HTTPConnection
from app.userscope import owned_by
from app import debugsession
from app import email as email_mod
from app import logs
@@ -230,7 +231,7 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
# request's access line, a queued row, a later background task would otherwise put rows back
# under the id after this DELETE (logs.forget_user).
logs.forget_user(user_id)
db.execute(delete(LogEvent).where(LogEvent.user_id == user_id))
db.execute(delete(LogEvent).where(owned_by(LogEvent.user_id, user_id)))
# Every canonically-equal invite goes, not just the literal spelling: is_allowed matches on the
# canonical form, so leaving a sibling row behind would keep granting access to a mailbox whose
# account was just erased — and prod really does carry approved pairs for one mailbox.
View File
+265
View File
@@ -0,0 +1,265 @@
"""Catalog's tables: the shared YouTube world every user reads — channels and their videos — and
each user's footprint in it: what they subscribe to, block, explored or found by search. Only this
unit writes them; another unit reads them along a declared edge, or asks this unit's service."""
from datetime import datetime
from typing import TYPE_CHECKING
from sqlalchemy import (
BigInteger,
Boolean,
Computed,
DateTime,
ForeignKey,
Integer,
JSON,
String,
Text,
UniqueConstraint,
func,
)
from sqlalchemy.dialects.postgresql import TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db import Base
from app.models import TimestampMixin
if TYPE_CHECKING:
from app.models import User
class Channel(Base, TimestampMixin):
"""A YouTube channel. Shared across all users (one channel's videos are the same
for everyone), so its expensive metadata is fetched and stored only once."""
__tablename__ = "channels"
id: Mapped[str] = mapped_column(String(32), primary_key=True) # "UC..." channel id
title: Mapped[str | None] = mapped_column(String(255))
handle: Mapped[str | None] = mapped_column(String(255))
description: Mapped[str | None] = mapped_column(Text)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
uploads_playlist_id: Mapped[str | None] = mapped_column(String(64))
subscriber_count: Mapped[int | None] = mapped_column(BigInteger)
video_count: Mapped[int | None] = mapped_column(BigInteger)
# Channel "About" metadata (fetched with the same channels.list call, no extra quota):
# lifetime view count, the channel's join date, banner image, and external links.
total_view_count: Mapped[int | None] = mapped_column(BigInteger)
published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
banner_url: Mapped[str | None] = mapped_column(String(1024))
external_links: Mapped[list | None] = mapped_column(JSON)
topic_categories: Mapped[list | None] = mapped_column(JSON)
default_language: Mapped[str | None] = mapped_column(String(16))
country: Mapped[str | None] = mapped_column(String(8))
# Creator keyword tags (brandingSettings.channel.keywords) — a single space-separated string,
# multi-word tags quoted; the client parses it into chips for the About tab.
keywords: Mapped[str | None] = mapped_column(Text)
# Sync bookkeeping.
details_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
recent_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
last_rss_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# When the newest uploads were last read through the Data API because the RSS feed would not
# answer (`sync/runner.py` `run_feed_fallback`): the once-per-`rss_fallback_hours` clock.
feed_fallback_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
backfill_cursor: Mapped[str | None] = mapped_column(String(128))
backfill_done: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this channel only entered the catalog because a live YouTube search
# surfaced one of its videos (nobody subscribes to it / it has no organic upload sync).
# Lets the Library hide search-only "noise" by default.
from_search: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
)
# True when this channel exists only because a user EXPLORED it (opened its page without
# subscribing) — an ephemeral, per-explorer addition. The cleanup job hard-deletes such a
# channel (and its via_explore videos) once nobody keeps it (no subscription, no live
# ExploredChannel row, no per-user state) after a grace period. Cleared when someone
# subscribes (the "keep" signal promotes it to organic).
from_explore: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
)
videos: Mapped[list["Video"]] = relationship(back_populates="channel")
subscriptions: Mapped[list["Subscription"]] = relationship(back_populates="channel")
class Subscription(Base, TimestampMixin):
"""Per-user link to a channel, with the user's own priority/visibility overrides."""
__tablename__ = "subscriptions"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
# YouTube's subscription resource id, needed to unsubscribe via the API.
yt_subscription_id: Mapped[str | None] = mapped_column(String(128))
priority: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
hidden: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Per-user opt-in to full-history (deep) backfill of this channel. Default off so a
# new user's unique channels don't trigger a big shared-quota burst; the deep
# scheduler only picks up channels at least one user has requested.
deep_requested: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
subscribed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
user: Mapped["User"] = relationship(back_populates="subscriptions")
channel: Mapped["Channel"] = relationship(back_populates="subscriptions")
# Video.live_status values that mean "currently a live/upcoming broadcast" — transient states
# hidden from feeds/search and revisited by the refresh job until they settle into a VOD.
LIVE_OR_UPCOMING = ("live", "upcoming")
class Video(Base, TimestampMixin):
"""A YouTube video. Shared across users; per-user state lives elsewhere."""
__tablename__ = "videos"
id: Mapped[str] = mapped_column(String(16), primary_key=True) # video id
# No standalone channel_id index: the composite `ix_videos_channel_pub_cover`
# (channel_id, published_at DESC NULLS LAST, id) from migration 0095 leads with channel_id, so it
# serves every channel_id lookup (FK cascade, sync upsert, channel-detail) and the per-channel feed
# sort. Its INCLUDEd columns make the channel manager's sums and the feed sidebar's counts Index Only
# Scans (no heap): a column those queries start reading belongs in it (test_channel_list_index_only).
# Like the search_vector GIN, this composite is migration-defined, not an inline `index=True`.
channel_id: Mapped[str] = mapped_column(ForeignKey("channels.id", ondelete="CASCADE"))
title: Mapped[str | None] = mapped_column(Text) # normalized for display (see app.titles)
original_title: Mapped[str | None] = mapped_column(Text) # raw YouTube title, preserved
description: Mapped[str | None] = mapped_column(Text)
# NOT NULL (migration 0068): in reality every video carries a publish date, and a nullable key
# forced the feed's keyset predicate into a non-sargable `... OR published_at IS NULL` disjunction
# that blocked the ix_videos_published_at_desc range (deep pages scanned from the top and filtered —
# PERF-15). server_default now() is the safety net for the rare date-less ingest (the enrich fills
# the real date); the ix_videos_published_at index stays for the `oldest` sort.
published_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), index=True, nullable=False, server_default=func.now()
)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
# The feed's views / duration / title sorts each ride a migration-defined index in their exact
# ORDER BY, one per direction (and per ICU collation for title): migration 0096, held to
# `feedquery.order_by` by test_feed_sort_indexes.
duration_seconds: Mapped[int | None] = mapped_column(Integer)
view_count: Mapped[int | None] = mapped_column(BigInteger)
like_count: Mapped[int | None] = mapped_column(BigInteger)
category_id: Mapped[int | None] = mapped_column(Integer)
topic_categories: Mapped[list | None] = mapped_column(JSON)
default_language: Mapped[str | None] = mapped_column(String(16))
detected_language: Mapped[str | None] = mapped_column(String(16))
# Creator-supplied keyword tags (snippet.tags joined by spaces) — fetched for free with the
# snippet we already request. Indexed into search_vector so the feed search matches the
# uploader's own keywords, not just words in the title.
keywords: Mapped[str | None] = mapped_column(Text)
# Accumulated distinct live-search queries that surfaced this video (across all users) —
# folds YouTube's own relevance judgement into the local search: a video YouTube returned
# for "ti amo magyarul" becomes findable by that query locally even if its title lacks it.
search_terms: Mapped[str | None] = mapped_column(Text)
# No btree here or on the flag columns below: migration 0069 (PERF-3) dropped the never-picked
# boolean btrees and replaced the useful ones with partial indexes over the target subset
# (via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — migration-defined, like the
# search_vector GIN.
is_short: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Whether the youtube.com/shorts/<id> probe has run for this video.
shorts_probed: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this video first entered the catalog via a live YouTube search (not an
# organic channel/playlist sync). Used to hide search-discovered "noise" from the
# Library by default; left False by the organic ingest paths.
via_search: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this video entered the catalog because a user EXPLORED its (un-subscribed)
# channel. Like via_search it's hidden from the shared Library by default, but it's
# additionally PER-USER private: a via_explore video is only visible to users who
# explored or subscribe to its channel (a gate in the feed query), so one user's
# curiosity never floods everyone else's catalog. Left False by the organic ingest paths.
via_explore: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# none | live | upcoming | premiere | was_live
live_status: Mapped[str] = mapped_column(
String(16), default="none", server_default="none", index=True
)
enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# videos.list?part=status fields (populated during enrichment); used by the
# maintenance/validation job to decide whether a video is playable anywhere.
embeddable: Mapped[bool | None] = mapped_column(Boolean)
privacy_status: Mapped[str | None] = mapped_column(String(16)) # public|unlisted|private
upload_status: Mapped[str | None] = mapped_column(
String(16)
) # processed|failed|rejected|deleted
# Maintenance/validation lifecycle. last_checked_at drives the rolling re-validation
# (least-recently-checked first); unavailable_since marks a video as currently
# unplayable (hidden from the feed immediately, hard-deleted after a grace period);
# unavailable_reason records why (deleted|private|paywalled|abandoned|stuck_live).
# last_checked_at: no plain btree — migration 0069 (PERF-18) replaced it with the partial
# ix_videos_recheck_order (last_checked_at NULLS FIRST WHERE unavailable_since IS NULL) that the
# revalidate job's exact ORDER BY rides; unavailable_since is a partial WHERE IS NOT NULL (PERF-3).
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
unavailable_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
unavailable_reason: Mapped[str | None] = mapped_column(String(24))
# Weighted full-text search document (DB-generated, never written by the app): title (A) >
# creator keywords + search queries (B) > truncated description (C). ts_rank honours the
# weights so title matches outrank description matches. Backed by a GIN index (migration
# 0032). The accent-insensitive `unaccent_simple` config comes from migration 0031.
search_vector: Mapped[object | None] = mapped_column(
TSVECTOR,
Computed(
"setweight(to_tsvector('public.unaccent_simple', coalesce(title, '')), 'A') || "
"setweight(to_tsvector('public.unaccent_simple', coalesce(keywords, '') || ' ' || "
"coalesce(search_terms, '')), 'B') || "
"setweight(to_tsvector('public.unaccent_simple', left(coalesce(description, ''), 1000)), 'C')",
persisted=True,
),
)
channel: Mapped["Channel"] = relationship(back_populates="videos")
class SearchFind(Base, TimestampMixin):
"""Per-user record of a video the user surfaced via their own live YouTube search.
Lets "videos I searched for" appear in that user's own (Mine) feed and the Mine Source
filter — distinct from the GLOBAL `Video.via_search` provenance, which marks a video as
search-discovered by *anyone* and drives the shared Library's Source filter."""
__tablename__ = "search_finds"
__table_args__ = (UniqueConstraint("user_id", "video_id", name="uq_user_search_video"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
video_id: Mapped[str] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"), index=True)
class ExploredChannel(Base, TimestampMixin):
"""Per-user record of a channel the user opened the page of WITHOUT subscribing.
Exploring an un-subscribed channel ingests its recent uploads (flagged via_explore) so the
user can browse them, but keeps them out of everyone else's catalog. This row (a) gates the
explorer's access to those ephemeral videos in the feed query and (b) drives the grace-period
purge: a from_explore channel with no live ExploredChannel row (and no subscription / no
per-user state on its videos) is hard-deleted by the cleanup job. Subscribing is the "keep"
signal — it promotes the channel to organic and these rows become irrelevant. `created_at`
(from TimestampMixin) is the grace clock; a re-visit refreshes it."""
__tablename__ = "explored_channels"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_explored_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
class BlockedChannel(Base, TimestampMixin):
"""Per-user channel blocklist. Videos from a blocked channel are excluded from this user's
live YouTube search (and not ingested on their behalf) and hidden from their feed / explore
views — a personal "never show me this channel again" that doesn't affect other users."""
__tablename__ = "blocked_channels"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_blocked_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
+296
View File
@@ -0,0 +1,296 @@
"""Catalog's reads: channels and videos, and each user's footprint in them (subscriptions, blocks,
explorations), for the HTTP layer and for the core code that asks about them.
A user's row is read through `userscope`, so someone else's is the same answer as a missing one.
Catalog does not read another unit's tables: what it needs from Reader (a playlist's channels, a
user's tag links) it asks `reader.queries`.
"""
from datetime import datetime
from sqlalchemy import Select, and_, case, func, or_, select
from sqlalchemy.orm import Session
from app.catalog.models import (
LIVE_OR_UPCOMING,
BlockedChannel,
Channel,
ExploredChannel,
Subscription,
Video,
)
from app.collation import collated
from app.models import User
from app.sync.explore import followed_channel
from app.userscope import owned_by
# --- channels --------------------------------------------------------------------------------
def channel(db: Session, channel_id: str) -> Channel | None:
return db.get(Channel, channel_id)
def channel_title(db: Session, channel_id: str) -> str | None:
return db.scalar(select(Channel.title).where(Channel.id == channel_id))
def channel_titles(db: Session, ids: set[str]) -> dict[str, str]:
"""Each channel's title by id; a channel with no title yet is left out."""
if not ids:
return {}
rows = db.execute(select(Channel.id, Channel.title).where(Channel.id.in_(ids)))
return {cid: title for cid, title in rows.tuples() if title is not None}
def channel_aggregates(channel_ids: list[str]) -> Select:
"""Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
alone; a test holds that (test_channel_list_index_only)."""
return (
select(
Video.channel_id,
func.count(Video.id),
func.max(Video.published_at),
func.coalesce(func.sum(Video.duration_seconds), 0),
)
.where(Video.channel_id.in_(channel_ids))
.group_by(Video.channel_id)
)
def aggregates(db: Session, channel_ids: list[str]) -> list[tuple[str, int, datetime | None, int]]:
"""`channel_aggregates`, run: (channel id, stored videos, last upload, total seconds)."""
if not channel_ids:
return []
return list(db.execute(channel_aggregates(channel_ids)).tuples())
def video_counts(db: Session, channel_id: str) -> tuple[int, int, int]:
"""A channel's stored videos, its Shorts and its live/upcoming ones, in one grouped pass."""
n_all, n_short, n_live = db.execute(
select(
func.count(Video.id),
func.count(Video.id).filter(Video.is_short.is_(True)),
func.count(Video.id).filter(Video.live_status.in_(LIVE_OR_UPCOMING)),
).where(Video.channel_id == channel_id)
).one()
return int(n_all or 0), int(n_short or 0), int(n_live or 0)
def deep_queued(db: Session, channel_ids: list[str]) -> set[str]:
"""Which of these channels at least one user (any user) has asked full history for."""
if not channel_ids:
return set()
return set(
db.execute(
select(Subscription.channel_id)
.where(
Subscription.channel_id.in_(channel_ids),
Subscription.deep_requested.is_(True),
)
.distinct()
).scalars()
)
def followed(db: Session, channel_id: str) -> bool:
"""Whether anyone subscribes to the channel."""
return (
db.execute(
select(Subscription.id).where(Subscription.channel_id == channel_id).limit(1)
).first()
is not None
)
# --- a user's subscriptions, blocks and explorations ----------------------------------------
def subscriptions(db: Session, user: User, lang: str) -> list[tuple[Subscription, Channel]]:
"""The user's subscriptions with their channels, highest priority first, then by title in the
reader's alphabet."""
return list(
db.execute(
select(Subscription, Channel)
.join(Channel, Channel.id == Subscription.channel_id)
.where(owned_by(Subscription.user_id, user))
.order_by(Subscription.priority.desc(), collated(Channel.title, lang))
).tuples()
)
def subscription(db: Session, user: User, channel_id: str) -> Subscription | None:
return db.execute(
select(Subscription).where(
owned_by(Subscription.user_id, user), Subscription.channel_id == channel_id
)
).scalar_one_or_none()
def subscribed_channels(user: User):
"""The channel ids the user subscribes to, as a subquery."""
return select(Subscription.channel_id).where(owned_by(Subscription.user_id, user))
def exploration(db: Session, user: User, channel_id: str) -> ExploredChannel | None:
return db.execute(
select(ExploredChannel).where(
owned_by(ExploredChannel.user_id, user), ExploredChannel.channel_id == channel_id
)
).scalar_one_or_none()
def is_blocked(db: Session, user: User, channel_id: str) -> bool:
return (
db.execute(
select(BlockedChannel.id).where(
owned_by(BlockedChannel.user_id, user), BlockedChannel.channel_id == channel_id
)
).first()
is not None
)
def blocked_ids(db: Session, user: User) -> set[str]:
return set(
db.scalars(select(BlockedChannel.channel_id).where(owned_by(BlockedChannel.user_id, user)))
)
def blocked_channels(db: Session, user: User, lang: str) -> list[Channel]:
return list(
db.execute(
select(Channel)
.join(BlockedChannel, BlockedChannel.channel_id == Channel.id)
.where(owned_by(BlockedChannel.user_id, user))
.order_by(collated(Channel.title, lang))
).scalars()
)
# --- sync progress ---------------------------------------------------------------------------
def catalog_totals(db: Session, user: User) -> dict:
"""The instance's catalog size and this user's subscription count (the sync status card)."""
return {
"subscriptions": db.scalar(
select(func.count())
.select_from(Subscription)
.where(owned_by(Subscription.user_id, user))
),
"channels_total": db.scalar(select(func.count()).select_from(Channel)),
"channels_backfilling": db.scalar(
select(func.count()).select_from(Channel).where(Channel.backfill_done.is_(False))
),
"videos_total": db.scalar(select(func.count()).select_from(Video)),
"pending_enrich": db.scalar(
select(func.count()).select_from(Video).where(Video.enriched_at.is_(None))
),
}
def my_progress(db: Session, user: User) -> tuple[int, int, int, int]:
"""Over the channels this user subscribes to: how many, how many have details, how many their
recent uploads, how many their whole history."""
total, details_synced, recent_synced, all_deep_done = db.execute(
select(
func.count(Channel.id),
func.count(Channel.details_synced_at),
func.count(Channel.recent_synced_at),
func.sum(case((Channel.backfill_done.is_(True), 1), else_=0)),
)
.select_from(Channel)
.join(
Subscription,
and_(Subscription.channel_id == Channel.id, owned_by(Subscription.user_id, user)),
)
).one()
return total or 0, details_synced or 0, recent_synced or 0, int(all_deep_done or 0)
def my_deep_requested(db: Session, user: User) -> list[Channel]:
"""The channels this user asked full history for."""
return list(
db.execute(
select(Channel).join(
Subscription,
and_(
Subscription.channel_id == Channel.id,
owned_by(Subscription.user_id, user),
Subscription.deep_requested.is_(True),
),
)
).scalars()
)
def my_video_count(db: Session, user: User) -> int:
"""The videos of the channels this user subscribes to and has not hidden."""
return (
db.scalar(
select(func.count(Video.id)).join(
Subscription,
and_(
Subscription.channel_id == Video.channel_id,
owned_by(Subscription.user_id, user),
Subscription.hidden.is_(False),
),
)
)
or 0
)
def video_total(db: Session) -> int:
return db.scalar(select(func.count(Video.id))) or 0
def deep_backlog(db: Session) -> list[Channel]:
"""Every channel someone asked full history for that does not have it yet."""
deep_requested = (
select(Subscription.id)
.where(Subscription.channel_id == Channel.id, Subscription.deep_requested.is_(True))
.exists()
)
return list(
db.execute(
select(Channel).where(Channel.backfill_done.is_(False), deep_requested)
).scalars()
)
def work_queue(db: Session) -> dict:
"""What the catalog's jobs still have queued, instance-wide (the scheduler card)."""
return {
# The same selection run_recent_backfill works through, or the card counts channels the
# job will never touch and never reaches zero.
"channels_recent_pending": db.scalar(
select(func.count())
.select_from(Channel)
.where(Channel.recent_synced_at.is_(None), followed_channel())
)
or 0,
"videos_pending_enrich": db.scalar(
select(func.count()).select_from(Video).where(Video.enriched_at.is_(None))
)
or 0,
"videos_pending_shorts": db.scalar(
select(func.count())
.select_from(Video)
.where(Video.shorts_probed.is_(False), Video.enriched_at.is_not(None))
)
or 0,
"videos_live_refresh": db.scalar(
select(func.count())
.select_from(Video)
.where(
or_(
Video.live_status.in_(("live", "upcoming")),
and_(Video.live_status == "was_live", Video.duration_seconds.is_(None)),
)
)
)
or 0,
}
+232
View File
@@ -0,0 +1,232 @@
"""Live YouTube search, the catalog's side: what a page of results leaves in the shared catalog.
A result becomes a catalog row so it renders with the normal feed cards and the in-app player and
gains per-user state. The ingest runs in steps with a YouTube call between each, and each step's
rows are committed before the next call: an enrich that fails rolls back to the stubs, which stay
(the FK targets for what follows, backfilled later). Results are flagged (`videos.via_search`,
`channels.from_search`) so the Library can hide this search-discovered noise; Shorts and
live/upcoming videos are never kept. The HTTP layer (`routes/search.py`) runs the paging, the caps
and the refusals.
"""
from collections.abc import Sequence
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from sqlalchemy import delete, func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app import logs, quota, sysconfig
from app.catalog import queries
from app.catalog.models import LIVE_OR_UPCOMING, Channel, SearchFind, Subscription, Video
from app.models import User
from app.reader import queries as reader_queries
from app.sync.subscriptions import apply_channel_details
from app.sync.videos import apply_video_details, insert_stubs, parse_dt
from app.userscope import owned_by
from app.utils import like_escape
from app.youtube.client import YouTubeClient, YouTubeError
from app.youtube.shorts import make_client, probe_is_short
log = logs.get_logger(__name__)
def classify_shorts(db: Session, videos: Sequence[Video]) -> None:
"""Confirm/deny Shorts for the freshly-ingested videos so none slip into the feed.
Mirrors the scheduler's run_shorts_classification, but scoped to just these rows and run
synchronously (the search is an explicit, awaited action). Non-candidates (too long, no
duration, or live) are finalised without a probe; the rest hit youtube.com/shorts (no
quota). Undetermined probes are left for the scheduler to retry."""
probe_max = sysconfig.get_int(db, "shorts_probe_max_seconds")
to_probe: list[Video] = []
for v in videos:
if v.shorts_probed:
continue
if v.enriched_at is None:
# Enrichment failed (proxy blip) or YouTube omitted the row — duration is unknown,
# so DON'T finalise shorts_probed off a null duration; leave it for the scheduler
# (which only classifies enriched rows) to retry. Otherwise a real Short would leak.
continue
if v.live_status != "none" or v.duration_seconds is None or v.duration_seconds > probe_max:
v.is_short = False
v.shorts_probed = True
else:
to_probe.append(v)
if to_probe:
with make_client() as client:
def work(v: Video):
return v, probe_is_short(client, v.id)
with ThreadPoolExecutor(max_workers=16) as pool:
for v, result in pool.map(work, to_probe):
if result is None:
continue
v.is_short = result
v.shorts_probed = True
db.commit()
def ingest_candidates(
db: Session, yt: YouTubeClient, user: User, candidates: list[dict]
) -> list[str]:
"""Materialise one page of search candidates into the catalog (channel stubs + enrich,
video stubs flagged via_search, enrich, Shorts classification) and return the KEEPABLE
video ids — non-Short, non-live — in the input relevance order."""
ids = [it["id"] for it in candidates]
if not ids:
return []
# 1) Channels are the FK target — create stubs for unknown ones (flagged from_search),
# then enrich the new ones (channels.list, 1 unit). Stubs survive an enrich failure.
channel_ids = list({it["channel_id"] for it in candidates})
existing = set(db.scalars(select(Channel.id).where(Channel.id.in_(channel_ids))).all())
new_channel_ids = [c for c in channel_ids if c not in existing]
if new_channel_ids:
titles = {it["channel_id"]: it["channel_title"] for it in candidates}
for cid in new_channel_ids:
db.add(Channel(id=cid, title=titles.get(cid), from_search=True))
db.commit()
try:
with quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER):
details = yt.get_channels(new_channel_ids)
apply_channel_details(db, details, new_channel_ids)
db.commit()
except YouTubeError:
# The stubs remain and the details can be backfilled later — but until they are, those
# channels sit in the results with no thumbnail and no subscriber count, looking like
# channels that have neither.
log.warning(
"the channels found by a search could not be enriched",
extra=logs.event("search.channel_enrich_failed", channels=len(new_channel_ids)),
)
db.rollback() # details can be backfilled later; the stubs remain
# 2) Video stubs, flagged via_search. Existing rows keep their flags (ON CONFLICT DO
# NOTHING), so an organically-synced video stays organic.
# published_at is NOT NULL (migration 0068); the scrape source exposes no publish date
# (only a relative "3 years ago"), so fall back to now() to satisfy the constraint — the
# enrich in step 3 overwrites it with the video's real snippet date.
now = datetime.now(timezone.utc)
insert_stubs(
db,
[
{
"id": it["id"],
"channel_id": it["channel_id"],
"title": it["title"],
"published_at": parse_dt(it["published_at"]) or now,
"thumbnail_url": it["thumbnail_url"],
"via_search": True,
}
for it in candidates
],
)
# 3) Enrich (videos.list, 1 unit/50) so cards have duration/stats and we can judge live.
videos = db.scalars(select(Video).where(Video.id.in_(ids))).all()
try:
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_ENRICH):
detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
except YouTubeError:
# Every candidate then stays unenriched, and an unenriched video cannot be judged live or
# Short — so it is dropped. A working search returns nothing and reads as "no results".
log.warning(
"the videos found by a search could not be enriched",
extra=logs.event("search.video_enrich_failed", candidates=len(ids)),
)
detail_by_id = {}
enriched_at = datetime.now(
timezone.utc
) # when THIS enrich ran (distinct from the stub placeholder)
for v in videos:
item = detail_by_id.get(v.id)
if item is None:
continue
apply_video_details(v, item)
v.enriched_at = enriched_at
db.commit()
# 4) Confirm/deny Shorts (no quota) so none enter via search.
classify_shorts(db, videos)
keep = {
v.id
for v in videos
if v.enriched_at is not None and not v.is_short and v.live_status not in LIVE_OR_UPCOMING
}
return [vid for vid in ids if vid in keep]
def _fold_term(db: Session, video_ids: list[str], term: str) -> None:
"""Append `term` to each video's space-joined `search_terms`, unless it is already one of them.
"Already one of them" is a whole-term match, as a literal: a substring test skipped `cat` on a
video already found for `category`, and an unescaped one read the `_` in `my_channel` as a
wildcard — either way the term was never folded in, and local search lost that relevance."""
padded = func.concat(" ", func.coalesce(Video.search_terms, ""), " ")
db.execute(
update(Video)
.where(Video.id.in_(video_ids))
.where(padded.notilike(f"% {like_escape(term)} %"))
.values(
search_terms=func.trim(
func.coalesce(Video.search_terms, "").op("||")(" ").op("||")(term)
)
)
)
def new_first(db: Session, user: User, ids: list[str]) -> list[str]:
"""Reorder results so genuinely-new discoveries come first and things the user already has
(a subscribed channel, or a video they've watched / in-progress / saved / playlisted) sink
to the back — keeping the relevance order within each group."""
if not ids:
return ids
have = set(
db.scalars(
select(Video.id).where(
Video.id.in_(ids), Video.channel_id.in_(queries.subscribed_channels(user))
)
)
) | reader_queries.held_by(db, user, ids)
return [i for i in ids if i not in have] + [i for i in ids if i in have]
def record_finds(db: Session, user: User, video_ids: list[str], term: str) -> None:
"""These results as THIS user's search finds (the Mine feed, the Source filter), and the term
folded into each one's `search_terms` so local full-text search inherits YouTube's relevance.
Idempotent."""
db.execute(
pg_insert(SearchFind)
.values([{"user_id": user.id, "video_id": vid} for vid in video_ids])
.on_conflict_do_nothing(index_elements=["user_id", "video_id"])
)
_fold_term(db, video_ids, term)
def clear(db: Session, user: User, video_ids: list[str]) -> int:
"""Discard these results "as if never added": drop this user's finds of them, then delete the
ones nobody kept — a `via_search` video no reader holds (`reader.queries.kept_by_a_reader`),
whose channel nobody subscribes to, and that no other user still has as a find. Returns how
many videos went."""
db.execute(
delete(SearchFind).where(
owned_by(SearchFind.user_id, user), SearchFind.video_id.in_(video_ids)
)
)
subscribed = select(Subscription.id).where(Subscription.channel_id == Video.channel_id).exists()
other_find = select(SearchFind.id).where(SearchFind.video_id == Video.id).exists()
result = db.execute(
delete(Video).where(
Video.id.in_(video_ids),
Video.via_search.is_(True),
~reader_queries.kept_by_a_reader(Video.id),
~subscribed,
~other_find,
)
)
return result.rowcount if isinstance(result, CursorResult) else 0
+133
View File
@@ -0,0 +1,133 @@
"""Catalog's writes a request makes: a user's subscription settings, what they explore or block, a
subscription recorded or dropped, a channel's sync markers reset.
Every function here flushes and leaves the commit to its caller (the handler). The YouTube side of a
subscribe or an unsubscribe is the handler's (it needs the user's client and its refusals); what
lands here is the local record of it. The ingest jobs (`sync/`, `catalog/search`) write the
catalog's shared rows themselves.
"""
from datetime import datetime, timezone
from sqlalchemy import delete, update
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app.catalog import queries
from app.catalog.models import BlockedChannel, Channel, ExploredChannel, Subscription, Video
from app.models import User
from app.userscope import owned_by
def remember_own_channel(user: User, channel_id: str) -> None:
"""The user's own YouTube channel, so discovery never offers it to them."""
user.yt_channel_id = channel_id
def update_subscription(
db: Session,
sub: Subscription,
*,
priority: int | None = None,
hidden: bool | None = None,
deep_requested: bool | None = None,
) -> bool:
"""Change the settings given (None leaves one as it is). Returns whether full history was just
turned ON — the caller then pulls the channel now rather than at the scheduler's next tick."""
turned_on = False
if priority is not None:
sub.priority = priority
if hidden is not None:
sub.hidden = hidden
if deep_requested is not None:
# Turning it off won't undo already-fetched videos; it stops further deep paging if nobody
# else still wants it.
turned_on = deep_requested and not sub.deep_requested
sub.deep_requested = deep_requested
db.flush()
return turned_on
def set_deep_all(db: Session, user: User, on: bool) -> int:
"""Every one of the user's channels in (or out of) full-history backfill; how many rows moved."""
result = db.execute(
update(Subscription).where(owned_by(Subscription.user_id, user)).values(deep_requested=on)
)
return result.rowcount if isinstance(result, CursorResult) else 0
def explore(db: Session, user: User, channel: Channel) -> None:
"""Record (or refresh the grace clock of) the user's exploration of a channel, and flag the
channel ephemeral unless someone follows it — the cleanup job's keep rule reads both."""
existing = queries.exploration(db, user, channel.id)
if existing is None:
db.add(ExploredChannel(user_id=user.id, channel_id=channel.id))
else:
existing.created_at = datetime.now(timezone.utc)
if not channel.from_explore and not queries.followed(db, channel.id):
channel.from_explore = True
db.flush()
def reset_backfill(db: Session, channel: Channel) -> None:
"""Fetch the channel again from scratch: its backfill markers cleared, and full history asked
for on EVERY subscription to it — the deep scheduler gates on an EXISTS over those, so the admin
who resets it need not subscribe themselves."""
channel.backfill_done = False
channel.backfill_cursor = None
channel.recent_synced_at = None
db.execute(
update(Subscription)
.where(Subscription.channel_id == channel.id)
.values(deep_requested=True)
)
db.flush()
def record_subscription(db: Session, user: User, channel: Channel, yt_subscription_id: str) -> None:
"""The subscription YouTube now holds, recorded here. Subscribing is the "keep" signal: an
explored channel becomes an organic one, so the cleanup job leaves it alone and its videos join
the shared Library like any subscribed channel's uploads."""
db.add(
Subscription(
user_id=user.id,
channel_id=channel.id,
yt_subscription_id=yt_subscription_id or None,
subscribed_at=datetime.now(timezone.utc),
)
)
if channel.from_explore:
channel.from_explore = False
db.execute(
update(Video)
.where(Video.channel_id == channel.id, Video.via_explore.is_(True))
.values(via_explore=False)
)
db.flush()
def drop_subscription(db: Session, sub: Subscription) -> None:
db.delete(sub)
db.flush()
def block(db: Session, user: User, channel_id: str) -> None:
"""Block a channel for this user (idempotent), and end any exploration of it so the cleanup job
can reclaim it."""
if queries.is_blocked(db, user, channel_id):
return
db.add(BlockedChannel(user_id=user.id, channel_id=channel_id))
db.execute(
delete(ExploredChannel).where(
owned_by(ExploredChannel.user_id, user), ExploredChannel.channel_id == channel_id
)
)
db.flush()
def unblock(db: Session, user: User, channel_id: str) -> None:
db.execute(
delete(BlockedChannel).where(
owned_by(BlockedChannel.user_id, user), BlockedChannel.channel_id == channel_id
)
)
+4 -2
View File
@@ -268,10 +268,12 @@ class Settings(BaseSettings):
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
quota_retention_days: int = 365
# Background-run history (job_runs): days kept (0 = forever). The audit log's 180, which is how
# long a run lived there before it moved here: ~1.8k rows a day at the default intervals
# (measured 2026-10-17), ~320k at the limit. The same daily cleanup job prunes it.
job_run_retention_days: int = 180
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: DownloadLayout = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
sponsorblock_default: bool = False
# --- Diagnostics / logging ---
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
+7 -8
View File
@@ -26,6 +26,7 @@ from uuid import uuid4
from sqlalchemy import and_, delete, func, not_, select, update
from sqlalchemy.orm import Session, defer
from app.userscope import owned_by, owned_row
from app import diagnostics
from app.models import DebugScreenshot, DebugSession, LogEvent, User
@@ -76,7 +77,9 @@ def open_session(db: Session, user: User) -> DebugSession | None:
def _unclosed(db: Session, user: User) -> DebugSession | None:
return db.scalar(
select(DebugSession).where(DebugSession.user_id == user.id, DebugSession.ended_at.is_(None))
select(DebugSession).where(
owned_by(DebugSession.user_id, user), DebugSession.ended_at.is_(None)
)
)
@@ -126,12 +129,8 @@ def from_header(db: Session, user: User, value: str | None) -> DebugSession | No
"""The session a request names, if it is the caller's own and still live; else None."""
if not value or len(value) != _ID_LEN or not all(c in "0123456789abcdef" for c in value):
return None
session = db.get(DebugSession, value)
if (
session is None
or session.user_id != user.id
or not is_live(session, datetime.now(timezone.utc))
):
session = owned_row(db, DebugSession, value, user)
if session is None or not is_live(session, datetime.now(timezone.utc)):
return None
return session
@@ -354,7 +353,7 @@ def stored_screenshot_bytes(db: Session, user_id: int) -> int:
db.scalar(
select(func.coalesce(func.sum(DebugScreenshot.size), 0))
.join(DebugSession, DebugSession.id == DebugScreenshot.session_id)
.where(DebugSession.user_id == user_id)
.where(owned_by(DebugSession.user_id, user_id))
)
or 0
)
+2 -1
View File
@@ -22,7 +22,8 @@ to the client, which has both the native payload and the translations.
"""
from app.discover.budget import remaining_seconds
from app.models import Channel, Playlist
from app.catalog.models import Channel
from app.reader.models import Playlist
# Plex artwork is a poster for the things you browse and a landscape still for the things you play.
_POSTER_KINDS = ("movie", "show")
+38 -7
View File
@@ -22,10 +22,14 @@ from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.discover import cards, queries
from app.models import Channel, ComfortPin, PlexItem, PlexShow, Playlist, User, Video
from app.models import User
from app.plex.models import PlexItem, PlexShow
from app.catalog.models import Channel, Video
from app.reader.models import ComfortPin, Playlist
from app.plex import rollup
from app.plex.cards import SHOW_NEW, leaf_card, show_card
from app.plex.lookups import by_rating_key
from app.userscope import owned_by, owned_row
# kind -> the ComfortPin column that holds it. The client's vocabulary and the schema's, in one place.
PIN_COLUMNS: dict[str, str] = {
@@ -68,11 +72,11 @@ def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, obje
# such pin target", which is the same answer this function gives for one that does not exist.
except ValueError:
return None
pl = db.get(Playlist, pid)
# A playlist is per-user, unlike the other four (which are shared catalog rows whose PRIVACY
# lives in the state tables). Someone else's list must read as "no such thing", not as a 403 —
# the same reason `owned_or_404` 404s rather than 403s.
return (col, pl.id) if pl is not None and pl.user_id == user.id else None
pl = owned_row(db, Playlist, pid, user)
return (col, pl.id) if pl is not None else None
def reorder(db: Session, user: User, ids: list[int]) -> None:
@@ -92,7 +96,7 @@ def reorder(db: Session, user: User, ids: list[int]) -> None:
# old slots — and one global lock order means two reorders cannot deadlock.
locked = db.execute(
select(ComfortPin)
.where(ComfortPin.user_id == user.id)
.where(owned_by(ComfortPin.user_id, user.id))
.order_by(ComfortPin.id)
.with_for_update()
.execution_options(populate_existing=True)
@@ -117,7 +121,7 @@ def list_pins(db: Session, user: User) -> list[ComfortPin]:
return list(
db.execute(
select(ComfortPin)
.where(ComfortPin.user_id == user.id)
.where(owned_by(ComfortPin.user_id, user.id))
.order_by(ComfortPin.position.asc(), ComfortPin.id.asc())
)
.scalars()
@@ -132,17 +136,44 @@ def next_position(db: Session, user: User) -> int:
deletes the row, and every FK cascade silently removes a pin when the thing it points at goes
away — so past the first hole a count under-counts and would drop the new pin into the MIDDLE of
the shelf, or onto a position another pin already holds (nothing in the schema forbids that).
Same idiom as `routes/playlists._next_playlist_position`."""
Same idiom as a new playlist's place in the rail (`reader.service`)."""
return (
db.scalar(
select(func.coalesce(func.max(ComfortPin.position), -1)).where(
ComfortPin.user_id == user.id
owned_by(ComfortPin.user_id, user.id)
)
)
+ 1
)
def own_pin(db: Session, user: User, pin_id: int) -> ComfortPin | None:
return owned_row(db, ComfortPin, pin_id, user)
def pin_of(db: Session, user: User, column: str, value: object) -> ComfortPin | None:
"""The user's pin of this thing (`column`/`value` from `resolve_ref`), if it is on the shelf."""
return db.execute(
select(ComfortPin).where(
owned_by(ComfortPin.user_id, user), getattr(ComfortPin, column) == value
)
).scalar_one_or_none()
def add(db: Session, user: User, column: str, value: object) -> ComfortPin:
"""A new pin at the end of the shelf. A second pin of the same thing (another tab, at the same
moment) raises `IntegrityError` at the flush: the unique constraint is the guard."""
pin = ComfortPin(user_id=user.id, position=next_position(db, user), **{column: value})
db.add(pin)
db.flush()
return pin
def remove(db: Session, pin: ComfortPin) -> None:
db.delete(pin)
db.flush()
@dataclass
class _Batch:
"""What the shelf resolves for every pinned channel and playlist in one go."""
+18 -19
View File
@@ -6,7 +6,7 @@ Two rules shape this module:
**Reuse the query layer, not the route.** The YouTube half calls `feedquery.filtered_query` /
`sort_keys`, so the Home surface inherits the feed's own definitions of subscribed, blocked,
explore-visible and in-progress for free, and cannot drift from them. The Plex half calls
`plex.rollup` and the `plex.cards` serializers rather than `routes/plex/discover.py`: that route
`plex.rollup` and the `plex.cards` serializers rather than `plex/discover.py`: that read-model
narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
sidebar and wrong for a landing page that does not.
@@ -29,22 +29,17 @@ from app.feedquery import (
serialize,
sort_keys,
)
from app.models import (
Channel,
PlaylistItem,
PlexItem,
PlexShow,
PlexState,
User,
Video,
VideoState,
)
from app.models import User
from app.plex.models import PlexItem, PlexShow, PlexState
from app.catalog.models import Channel, Video
from app.reader.models import PlaylistItem, VideoState
from app.plex import rollup
from app.plex.cards import display_title, leaf_card, movie_card, show_card
from app.plex.query import apply_show_state_filter, content_libs
from app.userscope import owned_by
# How many rows a single rail may carry — Home's rails and the Plex module's Discover rails alike
# (`routes/plex/discover.py` imports it): the two surfaces should feel like the same app, and a rail
# (`plex/discover.py` imports it): the two surfaces should feel like the same app, and a rail
# is a rail. One definition, because a comment was all that held two copies in step.
RAIL_LIMIT = 20
@@ -98,7 +93,7 @@ def yt_in_progress(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[tup
q = (
select(*feed_columns(user, st), st.progress_updated_at.label("progressed_at"))
.join(Channel, Channel.id == Video.channel_id)
.join(st, and_(st.video_id == Video.id, st.user_id == user.id))
.join(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(
st.position_seconds > 0,
st.status.notin_(("watched", "hidden")),
@@ -141,7 +136,7 @@ def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
row = db.execute(
select(*feed_columns(user, st))
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
.outerjoin(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(Video.id == video_id)
).first()
return serialize(row) if row is not None else None
@@ -184,7 +179,7 @@ def playlists_summary(
select(*feed_columns(user, st), PlaylistItem.playlist_id.label("_playlist"))
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
.outerjoin(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(PlaylistItem.playlist_id.in_(playlist_ids))
.order_by(PlaylistItem.playlist_id, PlaylistItem.position.asc())
).all()
@@ -209,7 +204,7 @@ def _unseen_movie_conds(uid: int):
.select_from(PlexState)
.where(
PlexState.item_id == PlexItem.id,
PlexState.user_id == uid,
owned_by(PlexState.user_id, uid),
or_(
PlexState.status.in_(("watched", "hidden")),
# Started counts as "not a surprise" too — see the note in `yt_surprise`.
@@ -226,7 +221,7 @@ def plex_in_progress(
) -> list[tuple[object, dict]]:
"""In-progress movies + partially-watched shows as (progressed_at, card).
Mirrors `routes/plex/discover.py::_continue_watching` — same `PlexState` predicate for movies (a
Mirrors `plex/discover.py::_continue_watching` — same `PlexState` predicate for movies (a
NULL status still counts as started, so `notin_` alone would drop it), and the same shared
`rollup.agg_subquery` + `apply_show_state_filter` for shows, so the two rails cannot disagree on
what "in progress" means. What it does NOT inherit is that route's `LibraryFilters` narrowing:
@@ -237,7 +232,9 @@ def plex_in_progress(
if movie_libs:
mq = (
select(PlexItem, PlexState)
.join(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.join(
PlexState, and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid))
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_libs),
@@ -331,7 +328,9 @@ def next_episode(db: Session, uid: int, show_id: int, *, lang: str) -> dict | No
)
base = (
select(PlexItem, PlexState)
.outerjoin(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.outerjoin(
PlexState, and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid))
)
.where(PlexItem.show_id == show_id, PlexItem.kind == "episode")
.limit(1)
)
+3 -2
View File
@@ -11,7 +11,8 @@ from sqlalchemy.orm import Session
from app import sysconfig
from app.discover import cards, queries
from app.discover.queries import RAIL_LIMIT
from app.models import PlexShow, User
from app.models import User
from app.plex.models import PlexShow
from app.plex.lookups import by_rating_key
from app.plex.rollup import NO_TS
@@ -75,7 +76,7 @@ def surprise(
) -> list[dict]:
"""The hero's draw pool: unseen things that fit the budget, from both sources under ONE seed.
Interleaved rather than concatenated — the same alternation `routes/plex/discover.py` uses for its
Interleaved rather than concatenated — the same alternation `plex/discover.py` uses for its
movie/show split — so a short pool never comes back all-YouTube or all-Plex just because one side
happened to be queried first.
+1 -1
View File
@@ -22,7 +22,7 @@ from app.db import SessionLocal
from app.downloads import adapters
from app.downloads import edit as editmod
from app.downloads import formats, service
from app.models import MediaAsset
from app.downloads.models import MediaAsset
log = logs.get_logger(__name__)
+4 -3
View File
@@ -33,8 +33,9 @@ from app import logs
from app.downloads import clip as clipmod
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential, MediaAsset
from app.downloads.models import DownloadCredential, MediaAsset
from app.security import decrypt_optional, encrypt
from app.userscope import owned_by
log = logs.get_logger(__name__)
@@ -157,7 +158,7 @@ def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredential | None:
return db.execute(
select(DownloadCredential).where(
DownloadCredential.user_id == user_id, DownloadCredential.host_key == host_key
owned_by(DownloadCredential.user_id, user_id), DownloadCredential.host_key == host_key
)
).scalar_one_or_none()
@@ -214,7 +215,7 @@ def status_list(db: Session, user_id: int) -> list[dict]:
have = {
c.host_key: c
for c in db.execute(
select(DownloadCredential).where(DownloadCredential.user_id == user_id)
select(DownloadCredential).where(owned_by(DownloadCredential.user_id, user_id))
).scalars()
}
out = []
+36 -3
View File
@@ -32,7 +32,8 @@ from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobStatus
from app.models import DownloadJob, FileFolder, MediaAsset
from app.downloads.models import DownloadJob, FileFolder, MediaAsset
from app.userscope import owned_by
class FolderKind(StrEnum):
@@ -106,7 +107,9 @@ def source_key_for(db: Session, job: DownloadJob) -> str:
def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
"""The user's source folder for `key`, made on first use. A race on the unique (user, key) is
settled by re-selecting, inside a SAVEPOINT so the caller's own pending writes survive."""
stmt = select(FileFolder).where(FileFolder.user_id == user_id, FileFolder.source_key == key)
stmt = select(FileFolder).where(
owned_by(FileFolder.user_id, user_id), FileFolder.source_key == key
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
@@ -131,7 +134,7 @@ def hub_folder(db: Session, user_id: int) -> FileFolder:
"""The user's Hub folder, made on first use; a race on its one-per-user index is settled like
`source_folder`'s."""
stmt = select(FileFolder).where(
FileFolder.user_id == user_id, FileFolder.kind == FolderKind.HUB
owned_by(FileFolder.user_id, user_id), FileFolder.kind == FolderKind.HUB
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
@@ -310,3 +313,33 @@ def delete_own(db: Session, folder: FileFolder) -> int:
db.flush() # the files leave before the folder goes: its foreign key RESTRICTs the delete
db.delete(folder)
return len(jobs)
def create_own(db: Session, user_id: int, name: str) -> FileFolder:
"""A new folder of the user's own; FolderError("file_folder_name_taken") when they already have
one by that name (the unique (user, lower(name)))."""
folder = FileFolder(user_id=user_id, kind=FolderKind.OWN, name=name)
try:
with db.begin_nested():
db.add(folder)
db.flush()
except IntegrityError as e:
raise FolderError("file_folder_name_taken") from e
return folder
def renamable(folder: FileFolder) -> bool:
"""Only a folder of the user's own making takes a name: a source folder's is its origin's."""
return folder.kind == FolderKind.OWN
def rename(db: Session, folder: FileFolder, name: str) -> None:
"""Rename one of the user's own folders (a source folder's name is its origin's)."""
if not renamable(folder):
raise FolderError("file_folder_not_own")
folder.name = name
try:
with db.begin_nested():
db.flush()
except IntegrityError as e:
raise FolderError("file_folder_name_taken") from e
+1 -1
View File
@@ -31,7 +31,7 @@ from app import apitokens, logs, sysconfig
from app.config import settings
from app.downloads import quota, service, storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
from app.notifications import create_notification
log = logs.get_logger(__name__)
+1 -1
View File
@@ -6,7 +6,7 @@ from sqlalchemy import text
from app.config import settings
from app.db import SessionLocal
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
# --- small DB helpers (short-lived sessions; the download itself holds no transaction) -------
+124 -6
View File
@@ -15,20 +15,24 @@ grant appended to the file URL, so the raw password never rides in the media URL
import hashlib
import hmac
import secrets
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from sqlalchemy import update as sql_update
from sqlalchemy.orm import Session
from app import security
from app import audit, security
from app.config import settings
from app.downloads import storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.models import User
from app.utils import iso
from app.userscope import owned_by
_GRANT_TTL = 6 * 3600 # a watch session's grant is good for 6h, then the viewer re-unlocks
# The most files a folder link lists: as many as the owner's own Files list shows (`routes.files.
# MAX_LIST`; a test holds the two equal — this layer may not import that one).
# The most files a folder link lists: as many as the owner's own Files list shows
# (`queries.MAX_LIST`; a test holds the two equal — `queries` imports this module).
FOLDER_MAX = 500
@@ -59,7 +63,7 @@ def in_folder(folder: FileFolder) -> tuple:
moved out stops opening through the link at its next request; one moved in starts."""
return (
DownloadJob.folder_id == folder.id,
DownloadJob.user_id == folder.user_id,
owned_by(DownloadJob.user_id, folder.user_id),
DownloadJob.status == JobStatus.DONE,
)
@@ -170,3 +174,117 @@ def public_meta(
"file_url": file_url,
"poster_url": poster_url,
}
# --- the owner's writes --------------------------------------------------------------------
# Each flushes and records its audit row; the handler commits.
def expiry(days: int | None) -> datetime | None:
"""None/0/negative → no expiry; otherwise N days out, capped at ~10 years."""
if not days or days <= 0:
return None
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
def audit_view(link: DownloadLink) -> dict:
"""A link's settings as the audit records them: whether it has a password, never which."""
return {
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": iso(link.expires_at),
}
def create(
db: Session,
user: User,
*,
password: str,
allow_download: bool,
expires_days: int | None,
**target: int,
) -> tuple[DownloadLink, str]:
"""A link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and the
token it is opened by — which is in hand only now."""
token = new_token()
link = DownloadLink(
**target,
created_by=user.id,
allow_download=allow_download,
password_hash=security.hash_password(password) if password else None,
expires_at=expiry(expires_days),
)
seal(link, token)
db.add(link)
db.flush()
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_CREATE,
target_type="download_link",
target_id=link.id,
after={**target, **audit_view(link)},
)
return link, token
def update(
db: Session,
user: User,
link: DownloadLink,
*,
allow_download: bool | None = None,
expires_days: int | None = None,
password: str | None = None,
sent: frozenset[str] = frozenset(),
) -> None:
"""Change what the caller sent (`sent` names the fields, so `expires_days=None` can mean "never
expires"). Setting, clearing or changing the password bumps its version, so any grant issued
under the previous one stops working now rather than living out its TTL."""
before = audit_view(link)
if "allow_download" in sent:
link.allow_download = bool(allow_download)
if "expires_days" in sent:
link.expires_at = expiry(expires_days)
if "password" in sent:
pw = (password or "").strip()
link.password_hash = security.hash_password(pw) if pw else None
link.password_version += 1
after = audit_view(link)
if after != before or "password" in sent:
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
target_type="download_link",
target_id=link.id,
before=before,
after=after,
)
db.flush()
def revoke(db: Session, user: User, link: DownloadLink) -> None:
link_id = link.id
db.delete(link)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
target_type="download_link",
target_id=link_id,
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
)
db.flush()
def count_view(db: Session, link: DownloadLink) -> None:
"""One more opening of the link. In SQL, not `+= 1` on the loaded row: two opens at once would
both write the same count."""
db.execute(
sql_update(DownloadLink)
.where(DownloadLink.id == link.id)
.values(view_count=DownloadLink.view_count + 1)
.execution_options(synchronize_session=False)
)
+385
View File
@@ -0,0 +1,385 @@
"""Capture's tables: the downloaded files (`MediaAsset`), each user's jobs over them and the folders
they sit in, the shares and public links that hand them on, a user's quota, cookies and format
profiles, and where they stopped playing one. Only this unit writes them; another unit reads them
along a declared edge, or asks this unit's service."""
from datetime import date, datetime, timezone
from sqlalchemy import (
BigInteger,
Boolean,
CheckConstraint,
Date,
DateTime,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.orm import Mapped, mapped_column, validates
from app.db import Base
from app.models import DISPLAY_TEXT_MAX, TimestampMixin, UpdatedAtMixin
class LibraryState(Base):
"""Where a user stopped in a library item (their own download, or one shared with them) — the
library player's resume point. Keyed by the JOB, not the file: a job is the user's library entry,
a re-download keeps it, and deleting the job deletes the row; the shared file under it may outlive
both. A revoked share leaves its recipient's row behind, unread — only a job the user can still
reach is ever read or written — and it is back if the item is shared again. Rows exist only for a
position worth keeping (see `routes._common.kept_position`); no row means "from the start"."""
__tablename__ = "library_states"
__table_args__ = (UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
job_id: Mapped[int] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
position_seconds: Mapped[int] = mapped_column(Integer, nullable=False)
progress_updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
class MediaAsset(Base, TimestampMixin):
"""A physical downloaded media file — the SHARED cache layer of the download center.
Mirrors the app's "shared catalog, private per-user state" model: one row per unique
``(source_kind, source_ref, format_sig)`` is the global cache. When a second user requests
the same video with a format that hashes to the same `format_sig`, their `download_job`
links to this ready asset instantly — no re-download, no cost. Only the SOURCE download is
shared; per-user edited derivatives (phase 2) are never cached here.
`rel_path` is relative to `settings.files_root` (the mount), so the physical filename is
system-decided and bound to the source id — never renamed (the user's custom name is display
metadata on `download_job`, applied only when downloading to their own machine). The naming
fields (`title`/`uploader`/`upload_date`) are denormalized so the Plex-tree path can be built
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here) frees
the file with its last holder, and `last_access_at` orders the GC's LRU eviction. How long a file
stays is not the asset's: each holder's job carries its own `expires_at` (its library folder's)."""
__tablename__ = "media_assets"
__table_args__ = (
UniqueConstraint(
"source_kind", "source_ref", "format_sig", name="uq_media_asset_cache_key"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512)) # video id, or full URL for ad-hoc
format_sig: Mapped[str] = mapped_column(String(64)) # hash of the output-affecting spec
status: Mapped[str] = mapped_column(
String(16), default="pending", server_default="pending", index=True
) # "pending" | "ready" | "error"
rel_path: Mapped[str | None] = mapped_column(Text) # relative to files_root
storyboard_path: Mapped[str | None] = mapped_column(Text) # phase-2 filmstrip mosaic
# A locally-generated poster frame (ffmpeg), set only when the source had NO thumbnail so the
# card / watch page / link preview can still show an image. Relative to files_root.
poster_path: Mapped[str | None] = mapped_column(Text)
size_bytes: Mapped[int | None] = mapped_column(BigInteger)
duration_s: Mapped[int | None] = mapped_column(Integer)
width: Mapped[int | None] = mapped_column(Integer)
height: Mapped[int | None] = mapped_column(Integer)
container: Mapped[str | None] = mapped_column(String(16))
vcodec: Mapped[str | None] = mapped_column(String(32))
acodec: Mapped[str | None] = mapped_column(String(32))
# "video" | "audio" | "file" — the media-kind discriminator that drives which library actions
# apply (play/edit for media; download-only for a generic uploaded file). Null while an asset is
# still pending (kind is known once the file is probed at finalize). It is the single stored home
# of the invariant `downloads.mediakind.derive` used to compute — never re-derived in prose (an
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Denormalized naming/metadata for the Plex tree + ad-hoc display.
title: Mapped[str | None] = mapped_column(Text)
uploader: Mapped[str | None] = mapped_column(String(255))
# yt-dlp's channel/uploader page URL (channel_url|uploader_url) — lets an auto-filled channel
# render as a clickable link. Null for sources without one (e.g. reddit) or older rows.
uploader_url: Mapped[str | None] = mapped_column(Text)
upload_date: Mapped[date | None] = mapped_column(Date)
thumbnail_url: Mapped[str | None] = mapped_column(Text)
# yt-dlp's canonical page URL for the source (webpage_url) — the clean "downloaded from"
# reference shown in the Files library. Null until the worker extracts it (older/queued rows
# fall back to a URL derived from source_kind+source_ref).
source_webpage_url: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
ref_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
last_access_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The file's SHA-256 (hex), so a machine fetching it can check what it got. Hashed while an upload
# streams in, otherwise on the first ask (`routes.files`); NULL = not hashed yet.
sha256: Mapped[str | None] = mapped_column(String(64))
@validates("rel_path")
def _forget_hash(self, _key: str, rel_path: str | None) -> str | None:
# A hash names the bytes at `rel_path`: a new file there (a conversion, a retry) is not them.
self.sha256 = None
return rel_path
class DownloadProfile(Base, TimestampMixin):
"""A named download preset — a reusable format profile.
`user_id` NULL + `is_builtin` marks the shipped presets (Best MP4, 1080p, audio-only, …);
a user's saved custom profiles carry their `user_id`. `spec` is the format definition
(mode av/v/a, max_height, container, audio_format, codec prefs, embed subs/chapters/
thumbnail, sponsorblock) that `downloads.formats` turns into a yt-dlp selector + a cache
`format_sig`."""
__tablename__ = "download_profiles"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
) # NULL for a builtin preset
name: Mapped[str] = mapped_column(String(80))
spec: Mapped[dict] = mapped_column(JSON)
is_builtin: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
sort_order: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class FileFolder(Base, TimestampMixin):
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
__tablename__ = "file_folders"
__table_args__ = (
UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
Index(
"uq_file_folder_own_name",
"user_id",
func.lower(text("name")),
unique=True,
postgresql_where=text("name IS NOT NULL"),
),
# One Hub folder per user.
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
kind: Mapped[str] = mapped_column(String(16)) # folders.FolderKind
# "site:" + a host name (DNS caps one at 253 characters).
source_key: Mapped[str | None] = mapped_column(String(260))
name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
retention_days: Mapped[int | None] = mapped_column(Integer)
def _ready_on_insert(context) -> datetime | None:
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
done = context.get_current_parameters().get("status") == "done"
return datetime.now(timezone.utc) if done else None
class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
"""A user's request to download one source — the per-user layer over `MediaAsset`.
The worker claims `queued` jobs with ``FOR UPDATE SKIP LOCKED`` and writes live
`progress`/`speed_bps`/`eta_s`/`phase` here (the frontend polls via useLiveQuery: there is
no server push channel, and the worker is a separate process anyway). `profile_snapshot`
freezes the spec at enqueue so later edits/deletion of the profile don't change this job.
`display_name` is the user's own name for the file (display + the Content-Disposition on
local download); the on-disk asset is never renamed. `asset_id` links to the shared file
(SET NULL if the asset is GC'd — the job then shows as expired).
Phase 2 (editor): `job_kind='edit'` jobs derive a new per-user clip from an already-downloaded
ready job. They still hang off a `MediaAsset` (with `source_kind='edit'`, a per-user cache key)
so all the file-serve/ref-count/GC/quota machinery is reused; the worker branches on the
asset's `source_kind` and runs ffmpeg instead of yt-dlp. `source_job_id`/`source_asset_id`
point at the parent download the clip was cut from; `edit_spec` is the (trim/crop) recipe."""
__tablename__ = "download_jobs"
__table_args__ = (
Index("ix_download_jobs_claim", "status", "queue_pos"),
# Every way a job becomes done stamps when (`folders.stamp_done`, `finalize_if_running`).
CheckConstraint(
"status <> 'done' OR ready_at IS NOT NULL", name="ck_download_jobs_ready_at"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL"), index=True
)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512))
# "download" (default), "edit" (a per-user trim/crop derivative of another job), "upload" (a
# user's own file, converted by the worker when it is not phone-safe — see `downloads.upload`) or
# "clip" (a range cut straight from a YouTube video or a Plex item — see `downloads.clip`).
job_kind: Mapped[str] = mapped_column(String(16), default="download", server_default="download")
# For edit jobs: the parent download this clip is cut from (SET NULL if the parent is deleted).
source_job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="SET NULL")
)
source_asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL")
)
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
edit_spec: Mapped[dict | None] = mapped_column(JSON)
profile_id: Mapped[int | None] = mapped_column(
ForeignKey("download_profiles.id", ondelete="SET NULL")
)
profile_snapshot: Mapped[dict] = mapped_column(JSON)
display_name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
# Per-user display overrides + extras (like display_name — metadata only, never the file).
# `display_uploader`/`display_uploader_url` override the asset's auto channel name/link;
# `extra_links` is an optional list of extra reference URLs shown on the card + watch page.
display_uploader: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
display_uploader_url: Mapped[str | None] = mapped_column(Text)
extra_links: Mapped[list | None] = mapped_column(JSON)
status: Mapped[str] = mapped_column(
String(16), default="queued", server_default="queued", index=True
) # queued | running | paused | done | error | canceled
progress: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
speed_bps: Mapped[int | None] = mapped_column(BigInteger)
eta_s: Mapped[int | None] = mapped_column(Integer)
phase: Mapped[str | None] = mapped_column(String(32))
error: Mapped[str | None] = mapped_column(Text) # raw yt-dlp/ffmpeg text (technical detail)
error_code: Mapped[str | None] = mapped_column(
String(24)
) # category, see downloads.errors.classify
# The host-specific failure-catalog entry on top of the category (`facebook.login_wall`), or NULL.
error_detail: Mapped[str | None] = mapped_column(String(48)) # see downloads.errors.diagnose
queue_pos: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# The Files library: the folder this file sits in (`downloads.folders.place` at creation; an own
# folder is emptied back into the source folders before it is deleted, hence RESTRICT), whether
# the user keeps it past its folder's retention, and when it goes (stamped once done; NULL =
# never). `expiry_warned` marks the one warning the GC sends before it deletes the file.
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="RESTRICT"), index=True
)
# The origin the file came from (the source folder's key), recorded where it is placed: an edit
# traces its origin through its parent, and that parent may be deleted or expire later — the
# file must still know which source folder it may go back to. NULL only for a file placed before
# the column existed outside a source folder; `folders.source_key_for` works those out.
source_key: Mapped[str | None] = mapped_column(String(260))
kept: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# When the file became done — what the library's "Added" means: a download that failed for days
# and then went through is new to its user, so it sorts as new. `created_at` stays when it was
# asked for (the queue's order). Set again when a file is made anew (a retry, a re-download).
ready_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), default=_ready_on_insert
)
class DownloadShare(Base, TimestampMixin):
"""A user→user grant: `to_user` may access `from_user`'s downloaded file (private by
default). Since assets are already deduped/shared on disk, a share is just an ACL row."""
__tablename__ = "download_shares"
__table_args__ = (UniqueConstraint("job_id", "to_user_id", name="uq_download_share"),)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int] = mapped_column(ForeignKey("download_jobs.id", ondelete="CASCADE"))
from_user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
)
to_user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
class DownloadQuota(Base, UpdatedAtMixin):
"""Per-user download limits (admin-managed). A row exists only when an admin customizes a
user; otherwise the sysconfig `download_default_*` values apply. `unlimited` bypasses the
byte cap entirely (e.g. for the admin's own account)."""
__tablename__ = "download_quotas"
user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), primary_key=True
)
max_bytes: Mapped[int] = mapped_column(BigInteger)
max_concurrent: Mapped[int] = mapped_column(Integer)
max_jobs: Mapped[int] = mapped_column(Integer)
unlimited: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class DownloadLink(Base, TimestampMixin):
"""A public capability link to one download, or to one of its owner's own folders — the "share
by link" surface.
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
the folder at the time it is opened. The token itself is never stored as written: a link is
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
grants a *registered* user access via the in-app "Shared with me" list."""
__tablename__ = "download_links"
__table_args__ = (
# A link opens exactly one thing: a file or a folder.
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
password_hash: Mapped[str | None] = mapped_column(Text)
# Bumped whenever the password changes; folded into the signed watch grant so rotating the
# password immediately invalidates any grants issued under the old one (instead of them living
# out the 6h grant TTL).
password_version: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
view_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class DownloadCredential(Base, TimestampMixin, UpdatedAtMixin):
"""A user's stored login cookies for ONE auth-walled download host (Facebook, Instagram, …).
Per-user, per-host: each user pastes their OWN browser cookies.txt (Netscape format) for a host
where they're logged in (e.g. a closed Facebook group they belong to), and a download of a
yt-dlp-supported, auth-walled link on that host is retried with the ACTING user's cookie. The
blob is Fernet-encrypted at rest (`cookies_enc`) and is NEVER returned by any API/log/audit —
the routes are write-only (host + updated_at only). See `downloads.credentials`.
Two load-bearing security controls live around this row, NOT on it:
* **cache privacy** — a cookie-authed download SALTS its MediaAsset `format_sig` with the
user id, so the authed asset is private to that user and can never be cache-served to
another user (`downloads.credentials.cookie_salt`).
* **host scope (SEC-12)** — a cookie is attached ONLY when the download URL's netloc matches
this row's `host_key` (`downloads.credentials.host_key_for_url`), so an arbitrary `?url=`
can't exfiltrate the user's session to a look-alike host.
`host_key` is a short canonical key from the supported-host registry (e.g. "facebook"), not the
raw netloc — one credential can cover several domains of the same site (facebook.com/fb.watch)."""
__tablename__ = "download_credentials"
__table_args__ = (UniqueConstraint("user_id", "host_key", name="uq_download_credential"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
host_key: Mapped[str] = mapped_column(String(64))
cookies_enc: Mapped[str] = mapped_column(Text) # Fernet ciphertext — never returned/logged
+1 -1
View File
@@ -16,7 +16,7 @@ from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import links as linksmod
from app.downloads import storage
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
+3 -1
View File
@@ -28,7 +28,9 @@ from app.downloads.jobdb import (
)
from app.downloads.stated import StatedFailure
from app.downloads.status import JobStatus
from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset, PlexItem
from app.models import DISPLAY_TEXT_MAX
from app.plex.models import PlexItem
from app.downloads.models import DownloadJob, MediaAsset
from app.plex import lookups as plex_lookups
from app.plex import paths as plex_paths
from app.notifications import create_notification
+404
View File
@@ -0,0 +1,404 @@
"""Capture's reads: what the library, the download tray, the share dialogs and the public watch pages
show. Each returns rows or plain values and writes nothing; whose a row is goes through
`userscope.owned_by` / `owned_row`, so a row of someone else's reads as a missing one.
The HTTP layer (`routes.downloads`, `routes.files`, `routes.public`) asks here and serializes; the
writes are the service's (`service`, `folders`, `links`, `quota`)."""
from collections.abc import Sequence
from datetime import datetime, timezone
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app.apitokens import FilesCaller, Right
from app.collation import collated
from app.downloads import folders, links, quota
from app.downloads.models import (
DownloadJob,
DownloadLink,
DownloadProfile,
DownloadQuota,
DownloadShare,
FileFolder,
LibraryState,
MediaAsset,
)
from app.downloads.status import FINISHED, JobStatus
from app.models import User
from app.userscope import owned_by, owned_row
from app.utils import like_escape
# Cap a listing so a heavy library can't return an unbounded result set (the old Download Center's
# own cap; far beyond any real library).
MAX_LIST = 500
# --- jobs and their files ------------------------------------------------------------------
def own_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
return owned_row(db, DownloadJob, job_id, user)
def job(db: Session, job_id: int) -> DownloadJob | None:
"""A job by id, whoever's — for a public link, which already names its job."""
return db.get(DownloadJob, job_id)
def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""A file the caller may read: their own, or one shared with them."""
row = db.get(DownloadJob, job_id)
if row is None:
return None
if row.user_id == user.id or shared_with(db, job_id, user) is not None:
return row
return None
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob | None:
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
own in a folder it may read."""
if caller.grants is None:
return accessible_job(db, caller.user, job_id)
row = own_job(db, caller.user, job_id)
if row is None or not caller.may(row.folder_id, Right.READ):
return None
return row
def asset_of(db: Session, job: DownloadJob | None) -> MediaAsset | None:
"""The file under a job, if it has one yet."""
return db.get(MediaAsset, job.asset_id) if job is not None and job.asset_id else None
def assets_for(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, MediaAsset]:
ids = {j.asset_id for j in jobs if j.asset_id}
if not ids:
return {}
rows = db.execute(select(MediaAsset).where(MediaAsset.id.in_(ids))).scalars()
return {a.id: a for a in rows}
def positions_for(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, int]:
"""`user`'s resume point in each of `jobs` that has one (the library player's `library_states`)."""
ids = [j.id for j in jobs]
if not ids:
return {}
rows = db.execute(
select(LibraryState.job_id, LibraryState.position_seconds).where(
owned_by(LibraryState.user_id, user), LibraryState.job_id.in_(ids)
)
).all()
return {job_id: pos for job_id, pos in rows}
def source_keys(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, str]:
"""Where each file came from (its source folder's key), for the list's Source column, the
"back to its source folder" move and the download tray's site: a file in a source folder has its
folder's key; one moved into an own folder (or someone else's, shared) is traced back by
`folders.source_key_for`."""
ids = {j.folder_id for j in jobs if j.folder_id is not None}
by_id = (
{f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
if ids
else {}
)
out: dict[int, str] = {}
for j in jobs:
folder = by_id.get(j.folder_id) if j.folder_id is not None else None
if folder is not None and folder.kind == folders.FolderKind.SOURCE and folder.source_key:
out[j.id] = folder.source_key
else:
out[j.id] = folders.source_key_for(db, j)
return out
def queue(db: Session, user: User, recent: datetime) -> list[DownloadJob]:
"""The download tray: what is waiting, running, paused or failed, and what finished (done or
canceled) since `recent`. A done job is timed from when it became done, never from its last
write: a rename, a move or the GC's expiry warning touches an old file's row, and the tray would
announce it as just finished."""
return list(
db.execute(
select(DownloadJob)
.where(
owned_by(DownloadJob.user_id, user),
DownloadJob.status.not_in(FINISHED)
| ((DownloadJob.status == JobStatus.DONE) & (DownloadJob.ready_at >= recent))
| ((DownloadJob.status == JobStatus.CANCELED) & (DownloadJob.updated_at >= recent)),
)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
).scalars()
)
def statuses_by_source(db: Session, user: User, statuses: Sequence[str]) -> list[tuple[str, str]]:
"""(source_ref, status) of each of `user`'s jobs in one of `statuses`."""
return list(
db.execute(
select(DownloadJob.source_ref, DownloadJob.status).where(
owned_by(DownloadJob.user_id, user), DownloadJob.status.in_(tuple(statuses))
)
).tuples()
)
def library(
db: Session,
user: User,
*,
folder_id: int | None,
listable: set[int] | None,
shared: bool,
term: str,
sort: str,
order: str,
lang: str,
) -> list[DownloadJob]:
"""The caller's done files (optionally one folder's, or only in the folders a token may list),
or the ones shared with them; searched by name and channel, sorted by when they were added, by
name (in the reader's alphabet) or by size. A file is added to its owner's list when it is done,
not when it was asked for (a download that failed for days and then went through is new); one
shared with the caller, when it was shared."""
stmt = (
select(DownloadJob)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.status == JobStatus.DONE)
)
if shared:
stmt = stmt.join(DownloadShare, DownloadShare.job_id == DownloadJob.id).where(
owned_by(DownloadShare.to_user_id, user)
)
else:
stmt = stmt.where(owned_by(DownloadJob.user_id, user))
if folder_id is not None:
stmt = stmt.where(DownloadJob.folder_id == folder_id)
elif listable is not None:
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
if term:
like = f"%{like_escape(term)}%"
stmt = stmt.where(
or_(
name.ilike(like),
func.coalesce(DownloadJob.display_uploader, MediaAsset.uploader, "").ilike(like),
)
)
key = {
"added": DownloadShare.created_at if shared else DownloadJob.ready_at,
"name": collated(name, lang),
"size": MediaAsset.size_bytes,
}[sort]
first = key.asc() if order == "asc" else key.desc()
tie = DownloadJob.id.asc() if order == "asc" else DownloadJob.id.desc()
return list(db.execute(stmt.order_by(first.nulls_last(), tie).limit(MAX_LIST)).scalars())
def recent_jobs(db: Session) -> list[DownloadJob]:
"""Everyone's newest jobs, for the admin's list."""
return list(
db.execute(
select(DownloadJob)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
).scalars()
)
# --- shares --------------------------------------------------------------------------------
def shared_with(db: Session, job_id: int, user: User | int) -> DownloadShare | None:
"""The grant that hands this job to `user`, if there is one."""
return db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, owned_by(DownloadShare.to_user_id, user)
)
).scalar_one_or_none()
def shared_at(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, datetime]:
rows = db.execute(
select(DownloadShare.job_id, DownloadShare.created_at).where(
owned_by(DownloadShare.to_user_id, user),
DownloadShare.job_id.in_([j.id for j in jobs]),
)
).all()
return {job_id: at for job_id, at in rows}
# --- folders -------------------------------------------------------------------------------
def own_folder(db: Session, user: User, folder_id: int) -> FileFolder | None:
return owned_row(db, FileFolder, folder_id, user)
def folders_by_id(db: Session, ids: set[int]) -> dict[int, FileFolder]:
"""These folders, by id — for a reader that already knows which ones (a token's grants)."""
if not ids:
return {}
return {f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
def folders_with_totals(
db: Session, user: User, listable: set[int] | None
) -> list[tuple[FileFolder, int, int]]:
"""The user's folders (only `listable` ones, when given) with how many done files each holds and
their size. A source folder its files have all left stays (count 0)."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
stmt = select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
).where(owned_by(FileFolder.user_id, user))
if listable is not None:
stmt = stmt.where(FileFolder.id.in_(listable))
rows = db.execute(
stmt.outerjoin(DownloadJob, done)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.group_by(FileFolder.id)
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
).tuples()
return [(f, count, int(size or 0)) for f, count, size in rows]
def folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
count, size = db.execute(
select(func.count(DownloadJob.id), func.coalesce(func.sum(MediaAsset.size_bytes), 0))
.select_from(DownloadJob)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.folder_id == folder_id, DownloadJob.status == JobStatus.DONE)
).one()
return count, int(size)
def linked_folders(db: Session, folder_ids: list[int]) -> set[int]:
"""Which of these folders a live public link opens (one not past its expiry)."""
now = datetime.now(timezone.utc)
rows = db.execute(
select(DownloadLink.folder_id).where(
DownloadLink.folder_id.in_(folder_ids),
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
)
).scalars()
return {folder_id for folder_id in rows if folder_id is not None}
# --- links ---------------------------------------------------------------------------------
def own_link(db: Session, user: User, link_id: int) -> DownloadLink | None:
"""A link to one of the user's files or folders."""
link = db.get(DownloadLink, link_id)
if link is None:
return None
target = (
owned_row(db, DownloadJob, link.job_id, user)
if link.job_id is not None
else owned_row(db, FileFolder, link.folder_id, user)
)
return link if target is not None else None
def links_to(
db: Session, *, job_id: int | None = None, folder_id: int | None = None
) -> list[DownloadLink]:
"""The links to one file or one folder, newest first."""
target = (
DownloadLink.job_id == job_id if job_id is not None else DownloadLink.folder_id == folder_id
)
return list(
db.execute(select(DownloadLink).where(target).order_by(DownloadLink.id.desc())).scalars()
)
def link_folder(db: Session, link: DownloadLink) -> FileFolder | None:
return db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
def folder_job(db: Session, folder: FileFolder, job_id: int) -> DownloadJob | None:
"""One of the files a link to `folder` reaches (`links.in_folder`), while it is in it."""
return db.execute(
select(DownloadJob).where(DownloadJob.id == job_id, *links.in_folder(folder))
).scalar_one_or_none()
def owner_suspended(db: Session, job: DownloadJob) -> bool:
"""The job's owner is suspended (or gone): the worker leaves their queued jobs where they are."""
owner = db.get(User, job.user_id)
return owner is None or owner.is_suspended
# --- profiles ------------------------------------------------------------------------------
def profiles_for(db: Session, user: User) -> list[DownloadProfile]:
"""The built-in format profiles, then the user's own."""
return list(
db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True) | owned_by(DownloadProfile.user_id, user))
.order_by(
DownloadProfile.is_builtin.desc(), DownloadProfile.sort_order, DownloadProfile.id
)
).scalars()
)
def own_profile(db: Session, user: User, profile_id: int) -> DownloadProfile | None:
"""One of the user's own profiles (a built-in is no one's to change)."""
p = owned_row(db, DownloadProfile, profile_id, user)
return None if p is None or p.is_builtin else p
def usable_profile(db: Session, user: User, profile_id: int) -> DownloadProfile | None:
"""A profile the user may enqueue with: a built-in, or their own."""
p = db.get(DownloadProfile, profile_id)
if p is None or (not p.is_builtin and p.user_id != user.id):
return None
return p
def default_profile(db: Session) -> DownloadProfile | None:
"""The first built-in, for an enqueue that named no format."""
return db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True))
.order_by(DownloadProfile.sort_order)
.limit(1)
).scalar_one_or_none()
# --- storage and quota (admin) -------------------------------------------------------------
def storage_totals(db: Session) -> tuple[int, int, int]:
"""(files on disk, their bytes, every asset row). On disk is `quota.ON_DISK`: ready files, and
uploads waiting for or failed in their conversion — the same disk the footprint and the GC cap
count."""
on_disk = db.execute(
select(func.count(), func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).one()
total_assets = db.execute(select(func.count()).select_from(MediaAsset)).scalar_one()
return on_disk[0], int(on_disk[1]), total_assets
def done_files_per_user(db: Session) -> dict[int, int]:
"""How many finished files each user's Files lists."""
return dict(
db.execute(
select(DownloadJob.user_id, func.count())
.where(DownloadJob.status == JobStatus.DONE)
.group_by(DownloadJob.user_id)
)
.tuples()
.all()
)
def custom_quota(db: Session, user_id: int) -> DownloadQuota | None:
"""The admin's own limits for this user, if any were set."""
return db.get(DownloadQuota, user_id)
+47 -3
View File
@@ -26,7 +26,8 @@ from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobKind, JobStatus
from app.models import DownloadJob, DownloadQuota, MediaAsset
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
from app.userscope import owned_by
# Jobs a user is considered to be "holding" (occupy a job slot / footprint). Terminal failures
# (error, canceled) don't count.
@@ -85,7 +86,7 @@ def footprint(db: Session, user_id: int) -> int:
held_assets = (
select(DownloadJob.asset_id)
.where(
DownloadJob.user_id == user_id,
owned_by(DownloadJob.user_id, user_id),
or_(
DownloadJob.status.in_(_HOLDING),
and_(DownloadJob.job_kind == JobKind.UPLOAD, DownloadJob.status == JobStatus.ERROR),
@@ -109,7 +110,7 @@ def _count(db: Session, user_id: int, states) -> int:
db.execute(
select(func.count())
.select_from(DownloadJob)
.where(DownloadJob.user_id == user_id, DownloadJob.status.in_(states))
.where(owned_by(DownloadJob.user_id, user_id), DownloadJob.status.in_(states))
).scalar()
or 0
)
@@ -201,3 +202,46 @@ def usage(db: Session, user_id: int) -> dict:
"max_jobs": lim.max_jobs,
"unlimited": lim.unlimited,
}
def set_custom(
db: Session,
user_id: int,
*,
max_bytes: int | None = None,
max_concurrent: int | None = None,
max_jobs: int | None = None,
unlimited: bool | None = None,
) -> None:
"""The admin's own limits for a user: starts from what applies now, then each value given (a
size floored at 0, a count at 1). The handler commits."""
row = db.get(DownloadQuota, user_id)
if row is None:
cur = resolve(db, user_id)
row = DownloadQuota(
user_id=user_id,
max_bytes=cur.max_bytes,
max_concurrent=cur.max_concurrent,
max_jobs=cur.max_jobs,
unlimited=cur.unlimited,
)
db.add(row)
if max_bytes is not None:
row.max_bytes = max(0, max_bytes)
if max_concurrent is not None:
row.max_concurrent = max(1, max_concurrent)
if max_jobs is not None:
row.max_jobs = max(1, max_jobs)
if unlimited is not None:
row.unlimited = unlimited
db.flush()
def reset_custom(db: Session, user_id: int) -> bool:
"""Back to the defaults; False when the user had no limits of their own."""
row = db.get(DownloadQuota, user_id)
if row is None:
return False
db.delete(row)
db.flush()
return True
+2 -1
View File
@@ -16,6 +16,7 @@ import shutil
import subprocess
import sys
import app.tables # noqa: F401 — a fresh interpreter: every table, before the first query
from app import logs
from app import logsink
from app.config import settings
@@ -23,7 +24,7 @@ from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import formats
from app.downloads.status import JobKind
from app.models import DownloadJob
from app.downloads.models import DownloadJob
log = logs.get_logger(__name__)
+201 -22
View File
@@ -1,16 +1,19 @@
"""Enqueue + cache resolution for the download center (the per-user layer over MediaAsset).
"""Capture's writes: enqueue + cache resolution, the job transitions, and what the library's
dialogs change (profiles, resume points, shares, a file's display fields).
`enqueue` is the single entry point used by the API: it resolves (or creates) the shared
MediaAsset for the requested source+format, links a per-user DownloadJob to it, and — on a
cache hit (asset already ready) — marks the job done instantly with no re-download. The worker
handles everything else. State transitions used by the routes (pause/resume/cancel/delete) and
the worker live in `worker.py` / the routes; this module owns enqueue + asset bookkeeping.
handles everything else. Every function here flushes and leaves the commit to its caller (the
handler); the folder, link and quota writes are `folders`, `links` and `quota`.
"""
import uuid
from datetime import datetime, timezone
from sqlalchemy import delete as delete_rows
from sqlalchemy import func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -21,17 +24,19 @@ from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import folders, formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobKind, JobStatus
from app.models import (
DISPLAY_TEXT_MAX,
Channel,
from app.models import DISPLAY_TEXT_MAX, User
from app.plex.models import PlexItem, PlexShow
from app.catalog.models import Channel, Video
from app.downloads.models import (
DownloadJob,
DownloadProfile,
DownloadShare,
LibraryState,
MediaAsset,
PlexItem,
PlexShow,
Video,
)
from app.plex import lookups as plex_lookups
from app.plex import paths as plex_paths
from app.userscope import owned_by, owned_row
class EditError(Exception):
@@ -230,8 +235,7 @@ def enqueue(
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
@@ -310,8 +314,7 @@ def enqueue_edit(
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
@@ -399,8 +402,7 @@ def enqueue_clip(
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
@@ -508,8 +510,8 @@ def release_asset(db: Session, job: DownloadJob) -> None:
# --- Job state transitions ---------------------------------------------------------------------
# The single home for the pause/resume/cancel/delete/redownload transitions + their asset
# bookkeeping (release, failed-asset reset, fresh-asset rebuild), so the routes are thin callers and
# the worker/GC could share the same definitions (B-sub-8). Each commits its own transition; the
# route keeps only auth + HTTP mapping + serialization.
# the worker/GC could share the same definitions (B-sub-8). Each flushes its transition and the caller
# commits it; the route keeps only auth + HTTP mapping + serialization.
def pause(db: Session, job: DownloadJob) -> None:
@@ -517,7 +519,7 @@ def pause(db: Session, job: DownloadJob) -> None:
in any other state."""
if job.status in (JobStatus.QUEUED, JobStatus.RUNNING):
job.status = JobStatus.PAUSED
db.commit()
db.flush()
def resume(db: Session, job: DownloadJob) -> None:
@@ -541,7 +543,7 @@ def resume(db: Session, job: DownloadJob) -> None:
asset.status = "pending"
asset.error = None
job.status = JobStatus.QUEUED
db.commit()
db.flush()
def _follow_the_cookie(db: Session, job: DownloadJob, asset: MediaAsset) -> MediaAsset:
@@ -576,14 +578,14 @@ def cancel(db: Session, job: DownloadJob) -> None:
if job.status not in (JobStatus.DONE, JobStatus.CANCELED):
release_asset(db, job) # release while the job still counts as holding
job.status = JobStatus.CANCELED
db.commit()
db.flush()
def delete(db: Session, job: DownloadJob) -> None:
"""Delete a job, releasing its asset hold first (which frees the file iff this was the last holder)."""
release_asset(db, job)
db.delete(job)
db.commit()
db.flush()
def redownload(db: Session, job: DownloadJob) -> None:
@@ -632,4 +634,181 @@ def redownload(db: Session, job: DownloadJob) -> None:
job.queue_pos = next_queue_pos(db) # tail of the queue, not its stale original slot
job.expires_at = None # stamped again when the rebuild is done
job.expiry_warned = False
db.commit()
db.flush()
# --- the library's writes ------------------------------------------------------------------
# What the HTTP layer changes for a user: each flushes, the handler commits.
def lock_own_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""One of the user's jobs, re-read under a row lock, for a transition that moves its asset's ref
count (resume, cancel, delete, redownload) or writes the status those decide on (pause). Two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds. None: not
theirs, or gone.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`finalize_if_running`), so the two can never deadlock."""
while True:
owned = owned_row(db, DownloadJob, job_id, user)
if owned is None:
return None
asset_id = owned.asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
return None
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
def hold_for_share(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
rather than failing the link's foreign key — or, done first, leaves None."""
if owned_row(db, DownloadJob, job_id, user) is None:
return None
return db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update(read=True)
.execution_options(populate_existing=True)
).scalar_one_or_none()
def create_profile(db: Session, user: User, name: str, spec: dict) -> DownloadProfile:
"""A format profile of the user's own, after their others."""
maxpos = db.execute(
select(func.coalesce(func.max(DownloadProfile.sort_order), 100)).where(
owned_by(DownloadProfile.user_id, user)
)
).scalar_one()
p = DownloadProfile(
user_id=user.id, name=name[:80], spec=spec, is_builtin=False, sort_order=maxpos + 1
)
db.add(p)
db.flush()
return p
def update_profile(profile: DownloadProfile, *, name: str | None, spec: dict | None) -> None:
"""Rename a profile and/or replace its spec; None leaves that one as it is."""
if name is not None:
profile.name = name[:80]
if spec is not None:
profile.spec = spec
def delete_profile(db: Session, profile: DownloadProfile) -> None:
db.delete(profile)
db.flush()
def set_position(db: Session, user: User, job_id: int, position: int) -> None:
"""The library player's resume point in a file the user can play; 0 clears it.
A clear is a statement, not a loaded row's delete: the periodic checkpoint and the pagehide beacon
of one stop may both clear it, and the second then deletes nothing rather than failing. A set is
an upsert, one row per (user, job) even when two checkpoints race, not a read-then-insert that
the second would fail on the unique key."""
if not position:
db.execute(
delete_rows(LibraryState).where(
owned_by(LibraryState.user_id, user), LibraryState.job_id == job_id
)
)
return
now = datetime.now(timezone.utc)
db.execute(
pg_insert(LibraryState)
.values(user_id=user.id, job_id=job_id, position_seconds=position, progress_updated_at=now)
.on_conflict_do_update(
constraint="uq_library_user_job",
set_={"position_seconds": position, "progress_updated_at": now},
)
)
def share(db: Session, job: DownloadJob, sender: User, recipient: User) -> bool:
"""Hand a file to another user, with a notice for them; False when it was already theirs."""
from app.notifications import create_notification
exists = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job.id, owned_by(DownloadShare.to_user_id, recipient)
)
).first()
if exists is not None:
return False
db.add(DownloadShare(job_id=job.id, from_user_id=sender.id, to_user_id=recipient.id))
create_notification(
db,
user_id=recipient.id,
type="download_shared",
title=(job.display_name or "A download") + " was shared with you",
# The client names the notice in the reader's language from `data`; the title is the
# English fallback only.
data={
"kind": "download_shared",
"from": sender.email,
"job_id": job.id,
"title": job.display_name,
},
commit=False,
)
db.flush()
return True
def unshare(db: Session, job_id: int, recipient: User) -> bool:
"""Take a file back from one user (its owner's act, or the recipient's own dismissal); False
when it was not theirs."""
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, owned_by(DownloadShare.to_user_id, recipient)
)
).scalar_one_or_none()
if row is None:
return False
db.delete(row)
db.flush()
return True
# The display fields a user may set on their file; the file on disk keeps its own name.
DISPLAY_FIELDS = frozenset(
{"display_name", "display_uploader", "display_uploader_url", "extra_links"}
)
def set_display(job: DownloadJob, **fields: object) -> None:
"""Set a file's display fields (each already cleaned by the caller); a field not given is left."""
unknown = set(fields) - DISPLAY_FIELDS
if unknown:
raise ValueError(f"not a display field: {sorted(unknown)}")
for name, value in fields.items():
setattr(job, name, value)
def touch(asset: MediaAsset) -> None:
"""The file was just served: the cache's idle clock starts again."""
asset.last_access_at = datetime.now(timezone.utc)
def store_sha256(db: Session, asset_id: int, digest: str) -> None:
"""Keep a file's hash once it is known — only where none is, so a concurrent hash of the same
file never overwrites the first."""
db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id, MediaAsset.sha256.is_(None))
.values(sha256=digest)
)
+2 -1
View File
@@ -37,7 +37,8 @@ from app.config import settings
from app.downloads import edit as editmod
from app.downloads import folders, mediakind, playable, probe, quota, service, storage
from app.downloads.status import JobKind, JobStatus
from app.models import DownloadJob, MediaAsset, display_text
from app.models import display_text
from app.downloads.models import DownloadJob, MediaAsset
def _title_from_filename(filename: str) -> str:
+117 -24
View File
@@ -1,10 +1,11 @@
"""Feed read-model: the pure query-building and row-serialization layer behind the feed.
"""Feed read-model: Reader's query-building, paging and row serialization behind the feed.
Extracted from `routes/feed.py` so the read-model has one home instead of living inside the
router module — the feed route handlers and the playlist-detail query share these helpers (the
projection, the filter builder, and keyset paging). Nothing here defines a route or reaches for
request state; it's all `(db, user, params) -> query`, and a malformed cursor raises a plain
`ValueError` for the route handler to translate (the tsquery sanitizer lives in `app.utils`).
One home for the read-model rather than the router module: the feed routes, the playlist-detail
query and Home share these helpers (the projection, the filter builder, and keyset paging), and the
feed's routes only validate and answer — a page, its counts and its facets are built here. Nothing
here defines a route or reaches for request state; it's all `(db, user, params) -> query`, and a
malformed cursor raises `BadCursor` for the route handler to translate (the tsquery sanitizer lives
in `app.utils`).
"""
import base64
@@ -16,22 +17,19 @@ from sqlalchemy import BigInteger, Select, and_, case, cast, false, func, or_, s
from sqlalchemy.orm import Session, aliased
from app.collation import collated
from app.models import (
from app.models import User
from app.catalog.models import (
LIVE_OR_UPCOMING,
BlockedChannel,
Channel,
ChannelTag,
ExploredChannel,
Playlist,
PlaylistItem,
SearchFind,
Subscription,
Tag,
User,
Video,
VideoState,
)
from app.reader.models import ChannelTag, Playlist, PlaylistItem, Tag, VideoState
from app.utils import iso, like_escape, to_tsquery_str
from app.userscope import owned_by
def _channel_url(channel_id: str, handle: str | None) -> str:
@@ -48,7 +46,7 @@ def _saved_expr(user: User):
.select_from(PlaylistItem)
.join(Playlist, Playlist.id == PlaylistItem.playlist_id)
.where(
Playlist.user_id == user.id,
owned_by(Playlist.user_id, user.id),
Playlist.kind == "watch_later",
PlaylistItem.video_id == Video.id,
)
@@ -103,7 +101,7 @@ def serialize(row) -> dict:
def _tag_visible(user: User):
"""A ChannelTag row this user may see: a shared system tag (user_id NULL) or their own."""
return or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
return or_(ChannelTag.user_id.is_(None), owned_by(ChannelTag.user_id, user.id))
# The pseudo-category of a selected tag id no Tag row answers (a saved view or the feed state outlives
@@ -122,7 +120,7 @@ def tags_by_category(db: Session, user: User, tag_ids: list[int]) -> dict[str, l
if not tag_ids:
return {} # skip the pointless `WHERE id IN ()` round-trip on the no-tag-filter hot path
out: dict[str, list[int]] = {}
visible = or_(Tag.user_id.is_(None), Tag.user_id == user.id)
visible = or_(Tag.user_id.is_(None), owned_by(Tag.user_id, user.id))
for tid, cat in db.execute(
select(Tag.id, Tag.category).where(Tag.id.in_(tag_ids), visible)
).all():
@@ -142,7 +140,7 @@ def not_blocked_cond(user: User):
position — and the one query that copied only PART of this predicate set is exactly how that
video came back as the landing page's suggestion after the user had said "never again"."""
return Video.channel_id.notin_(
select(BlockedChannel.channel_id).where(BlockedChannel.user_id == user.id)
select(BlockedChannel.channel_id).where(owned_by(BlockedChannel.user_id, user.id))
)
@@ -187,7 +185,7 @@ def filtered_query(
Subscription,
and_(
Subscription.channel_id == Video.channel_id,
Subscription.user_id == user.id,
owned_by(Subscription.user_id, user.id),
),
)
else:
@@ -199,12 +197,12 @@ def filtered_query(
Subscription,
and_(
Subscription.channel_id == Video.channel_id,
Subscription.user_id == user.id,
owned_by(Subscription.user_id, user.id),
Subscription.hidden.is_(False),
),
).outerjoin(
SearchFind,
and_(SearchFind.video_id == Video.id, SearchFind.user_id == user.id),
and_(SearchFind.video_id == Video.id, owned_by(SearchFind.user_id, user.id)),
)
# The state-gated views (watched/hidden/in_progress) only ever return videos that HAVE a
@@ -214,7 +212,7 @@ def filtered_query(
# exactly. 'unwatched'/'all' include row-less 'new' videos, so they keep the outer join.
state_required = show in ("watched", "hidden", "in_progress")
join_state = query.join if state_required else query.outerjoin
query = join_state(state, and_(state.video_id == Video.id, state.user_id == user.id))
query = join_state(state, and_(state.video_id == Video.id, owned_by(state.user_id, user)))
# Hide videos the maintenance job has flagged as currently unplayable (deleted/private/
# abandoned). They're either deleted after a grace period or unhidden if they recover;
@@ -227,8 +225,8 @@ def filtered_query(
# catalog. Applied in BOTH scopes; the channel page reaches its ephemeral videos by
# asking for scope=all + channel_ids (the channel is in this accessible set).
accessible_explore = (
select(Subscription.channel_id).where(Subscription.user_id == user.id)
).union(select(ExploredChannel.channel_id).where(ExploredChannel.user_id == user.id))
select(Subscription.channel_id).where(owned_by(Subscription.user_id, user.id))
).union(select(ExploredChannel.channel_id).where(owned_by(ExploredChannel.user_id, user.id)))
query = query.where(
or_(
Video.via_explore.is_(False),
@@ -346,7 +344,7 @@ def filtered_query(
if untagged:
query = query.where(
Video.channel_id.notin_(
select(ChannelTag.channel_id).where(ChannelTag.user_id == user.id)
select(ChannelTag.channel_id).where(owned_by(ChannelTag.user_id, user.id))
)
)
@@ -696,3 +694,98 @@ def decode_cursor(keys: list[dict], cursor: str) -> tuple[list, str, datetime |
# Framework-free: the route handler translates this into an HTTP 400 so the
# read-model stays web-layer-agnostic (see routes/feed.get_feed).
raise ValueError("Invalid feed cursor") from exc
class BadCursor(ValueError):
"""A `cursor` this module did not write (or one from another sort): the caller's bad input."""
def feed_page(
db: Session,
user: User,
params: dict,
*,
sort: str,
seed: int,
limit: int,
cursor: str | None,
lang: str,
) -> dict:
"""One page of the feed, and the cursor for the next. A malformed cursor raises `BadCursor`."""
query, rank_expr = filtered_query(db, user, **params)
keys = sort_keys(sort, seed, rank_expr, lang)
vals, cursor_id, window = None, None, None
if cursor:
try:
vals, cursor_id, window = decode_cursor(keys, cursor)
except ValueError as e:
raise BadCursor(str(e)) from e
ranked = sort == "relevance" and rank_expr is not None
if ranked and not cursor:
window = rank_window_start(db, query)
if ranked and window is not None:
keys = sort_keys(sort, seed, windowed_rank(rank_expr, window), lang)
# Expose each key column on the row so we can build the next cursor from the last item.
query = query.add_columns(*[k["expr"].label(f"_k{i}") for i, k in enumerate(keys)])
if vals is not None:
for k, v in zip(keys, vals, strict=True):
k["value"] = v
if ranked and window is not None:
rows = windowed_relevance_rows(db, query, keys, window, cursor_id, limit + 1)
else:
query = query.order_by(*order_by(keys))
if cursor_id is not None:
query = query.where(keyset_predicate(keys, cursor_id))
rows = db.execute(query.limit(limit + 1)).all()
has_more = len(rows) > limit
page = rows[:limit]
return {
"items": [serialize(r) for r in page],
"next_cursor": (
encode_cursor(keys, page[-1], window if ranked else None)
if (has_more and page)
else None
),
"has_more": has_more,
"limit": limit,
}
def feed_count(db: Session, user: User, **params) -> int:
"""How many videos the filters match — the reference oracle `feed_summary`'s count is tested
against."""
query, _rank = filtered_query(db, user, **params)
return db.scalar(select(func.count()).select_from(query.subquery())) or 0
def facet_counts(db: Session, user: User, **params) -> dict[int, int]:
"""Per-tag channel counts for the current filter context, one pass per category — the reference
oracle `feed_summary`'s facets are tested against.
Each category is counted with every other filter applied but its own selections ignored
(drill-down faceting: selecting one topic doesn't zero out the rest). A conjunctive (AND)
category keeps them applied, so each remaining chip narrows to channels that ALSO have every
selected tag, and tags that can't co-occur drop to zero."""
visible = _tag_visible(user)
counts: dict[int, int] = {}
for category in ("language", "topic", "other"):
conjunctive = params.get("tag_modes", {}).get(category) == "and"
base, _rank = filtered_query(
db,
user,
**{**params, "exclude_tag_category": None if conjunctive else category},
)
channels = base.with_only_columns(Video.channel_id).distinct().subquery()
rows = db.execute(
select(ChannelTag.tag_id, func.count(func.distinct(ChannelTag.channel_id)))
.select_from(channels)
.join(ChannelTag, ChannelTag.channel_id == channels.c.channel_id)
.join(Tag, and_(Tag.id == ChannelTag.tag_id, Tag.category == category))
.where(visible)
.group_by(ChannelTag.tag_id)
).all()
for tag_id, count in rows:
counts[tag_id] = count
return counts
+131
View File
@@ -0,0 +1,131 @@
"""The durable history of background runs — the `job_runs` table.
A run is written twice: a `running` row when it starts, its outcome when it ends. Each write is its
OWN short session, committed at once, for the reason `quota.record_usage` gives: the run's own work
may roll back (its `except: db.rollback()`), and the history of what happened must not go with it —
nor may this bookkeeping commit the job's half-done session.
Only the API process runs jobs (the worker never starts the scheduler), and one API process serves a
database (`.env.example`, the Scheduler section: the single-run guard is in its memory too), so at its
startup nothing can still be running: `interrupt_leftovers` closes what a stopped process left open.
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete, func, select, union_all, update
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import JobRun, QuotaEvent
from app.runstatus import RunStatus, RunTrigger
def start(job: str, trigger: RunTrigger, actor_id: int | None, target: str | None = None) -> int:
"""Open a run's row; returns its id."""
with SessionLocal() as s:
row = JobRun(
job=job,
trigger=trigger,
actor_id=actor_id,
target=target,
status=RunStatus.RUNNING,
started_at=datetime.now(timezone.utc),
)
s.add(row)
s.commit()
return row.id
def finish(
run_id: int,
status: RunStatus,
*,
summary: str | None = None,
result: dict | None = None,
error: str | None = None,
failure: dict | None = None,
) -> None:
"""Close a run with its outcome and what it spent. Conditional on the row still running: a row
the startup sweep already called interrupted is not rewritten by a thread that outlived it."""
with SessionLocal() as s:
units = s.scalar(
select(func.coalesce(func.sum(QuotaEvent.units), 0)).where(
QuotaEvent.job_run_id == run_id
)
)
s.execute(
update(JobRun)
.where(JobRun.id == run_id, JobRun.status == RunStatus.RUNNING)
.values(
status=status,
finished_at=datetime.now(timezone.utc),
summary=summary,
result=result,
error=error,
failure=failure,
quota_units=units or 0,
)
)
s.commit()
def interrupt_leftovers() -> int:
"""Every row a previous process left `running` → `interrupted`, with what it spent until then
(its quota rows outlive it, as `finish` sums them). Called once at startup, before the scheduler
can open a new run. Returns how many."""
spent = (
select(func.coalesce(func.sum(QuotaEvent.units), 0))
.where(QuotaEvent.job_run_id == JobRun.id)
.scalar_subquery()
)
with SessionLocal() as s:
res = s.execute(
update(JobRun)
.where(JobRun.status == RunStatus.RUNNING)
.values(
status=RunStatus.INTERRUPTED,
finished_at=datetime.now(timezone.utc),
quota_units=spent,
)
)
s.commit()
return res.rowcount if isinstance(res, CursorResult) else 0
def recent_runs(db: Session, jobs: list[str], per_job: int) -> dict[str, list[JobRun]]:
"""Each of `jobs`' last `per_job` runs, newest first, by job id — one query, each job's part a
short walk of `ix_job_runs_job_started`, so a poll costs the same at any history length (a window
over the whole table grew with every run kept)."""
if not jobs:
return {}
newest = union_all(
*(
select(JobRun.id)
.where(JobRun.job == job)
.order_by(JobRun.started_at.desc(), JobRun.id.desc())
.limit(per_job)
for job in jobs
)
).subquery()
rows = db.execute(
select(JobRun)
.where(JobRun.id.in_(select(newest.c.id)))
.order_by(JobRun.job, JobRun.started_at.desc(), JobRun.id.desc())
).scalars()
out: dict[str, list[JobRun]] = {}
for r in rows:
out.setdefault(r.job, []).append(r)
return out
def gc(db: Session, retention_days: int) -> int:
"""Delete finished runs older than `retention_days` (0 = keep forever). A run still going is
never reaped. The caller commits."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
res = db.execute(
delete(JobRun).where(JobRun.started_at < cutoff, JobRun.status != RunStatus.RUNNING)
)
return res.rowcount if isinstance(res, CursorResult) else 0
+15 -2
View File
@@ -10,8 +10,8 @@ the logs were both unjoinable and leaky. `configure_logging()` replaces all four
Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
ts · level · sev · source · logger · event · msg · attrs
plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
`job_id`) and the app version. That shape is what the admin log console filters on and what a Gitea
issue attachment will carry, so it is deliberately flat and stable.
`job_id`, `run_id`) and the app version. That shape is what the admin log console filters on and
what a Gitea issue attachment will carry, so it is deliberately flat and stable.
Call sites stay ordinary stdlib logging:
@@ -81,15 +81,28 @@ _debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
"log_debug_session", default=None
)
_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
# The background run (app/jobruns.py) this context is doing; quota's spend is tagged with it too.
_run_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_run_id", default=None)
_CONTEXT_VARS = {
"request_id": _request_id,
"user_id": _user_id,
"debug_session_id": _debug_session,
"job_id": _job_id,
"run_id": _run_id,
}
def current_request_id() -> str | None:
"""The HTTP request this context is serving, if any — the request middleware sets it."""
return _request_id.get()
def current_run_id() -> int | None:
"""The background run this context is doing, if any — `bind(run_id=…)` sets it."""
return _run_id.get()
class _RequestScope:
"""What the auth dependency learns about a request, for every line logged after it.
+1
View File
@@ -362,6 +362,7 @@ def serialize(row: LogEvent) -> dict:
"request_id": row.request_id,
"user_id": row.user_id,
"job_id": row.job_id,
"run_id": row.run_id,
"debug_session_id": row.debug_session_id,
"version": row.version,
}
+3
View File
@@ -28,6 +28,7 @@ log = logs.get_logger(__name__)
from app import bodylimit, collation, logsink, state, sysconfig # noqa: E402 — logging is configured first
from app.config import settings # noqa: E402 — logging is configured first
from app.db import SessionLocal, engine # noqa: E402 — logging is configured first
import app.tables # noqa: E402 — every table, before the first query
from app.downloads import service as download_service # noqa: E402 — logging is configured first
from app.downloads import storage # noqa: E402 — logging is configured first
from app.plex import stream as plex_stream # noqa: E402 — logging is configured first
@@ -52,6 +53,7 @@ from app.routes import ( # noqa: E402 — logging is configured first
plex as plex_routes,
public as public_routes,
quota,
runs as runs_routes,
saved_views,
scheduler as scheduler_routes,
search as search_routes,
@@ -345,6 +347,7 @@ app.include_router(config_routes.router)
app.include_router(scheduler_routes.router)
app.include_router(audit_routes.router)
app.include_router(admin_logs_routes.router)
app.include_router(runs_routes.router)
app.include_router(quota.router)
app.include_router(version.router)
app.include_router(setup_routes.router)
+61 -1228
View File
File diff suppressed because it is too large Load Diff
+5 -3
View File
@@ -9,6 +9,7 @@ table can't grow without bound for a heavy user.
from sqlalchemy import delete, select
from sqlalchemy.orm import Session
from app.userscope import owned_by
from app.models import Notification
# Keep at most this many READ notifications per user; older read ones are pruned. Unread
@@ -48,11 +49,13 @@ def create_notification(
def trim_read(db: Session, user_id: int, cap: int = READ_SOFT_CAP) -> int:
"""Delete a user's oldest READ notifications beyond `cap`. Returns the number removed.
Idempotent and cheap; safe to call after marking things read."""
Idempotent and cheap; safe to call after marking things read. Flushes first, so a notice the
caller has just marked read counts; the caller commits."""
db.flush()
ids = (
db.execute(
select(Notification.id)
.where(Notification.user_id == user_id, Notification.read.is_(True))
.where(owned_by(Notification.user_id, user_id), Notification.read.is_(True))
.order_by(Notification.created_at.desc())
.offset(cap)
)
@@ -63,5 +66,4 @@ def trim_read(db: Session, user_id: int, cap: int = READ_SOFT_CAP) -> int:
return 0
# One bulk DELETE rather than a get+delete per row (the ids are already in hand).
db.execute(delete(Notification).where(Notification.id.in_(ids)))
db.commit()
return len(ids)
+1 -1
View File
@@ -5,7 +5,7 @@ router. Pure functions — no DB, no request state."""
from typing import NamedTuple
from app.models import PlexItem, PlexShow, PlexState
from app.plex.models import PlexItem, PlexShow, PlexState
class ShowState(NamedTuple):
+122
View File
@@ -0,0 +1,122 @@
"""What a collection is to the app: where it comes from, whether it may be edited, its card, its
effective content rating, and how its members order — shared by the collection pages, the info
page's collection strips and the admin editor."""
from collections.abc import Callable
from datetime import date, datetime
from typing import Any, NamedTuple
from app.plex.models import PlexCollection
# The most-restrictive → least ordering used to pick a collection's effective content rating from its
# members (Plex shows one cert on the collection; we don't store it, so derive it).
_CERT_RANK = {
"G": 1,
"TV-Y": 1,
"TV-Y7": 1,
"TV-G": 1,
"PG": 2,
"TV-PG": 2,
"PG-13": 3,
"TV-14": 3,
"R": 4,
"TV-MA": 4,
"NC-17": 5,
}
def collection_cert(certs: list[str | None]) -> str | None:
ranked = [(c, _CERT_RANK.get(c, 0)) for c in certs if c]
return max(ranked, key=lambda x: x[1])[0] if ranked else None
_STRIP_PROVIDERS = ("imdb", "tmdb", "tvdb", "trakt")
def collection_source(col: PlexCollection) -> str:
first = (col.title or "").strip().split(" ", 1)[0].lower()
if first in _STRIP_PROVIDERS:
return first
if col.smart:
return "smart"
return "collection"
def collection_editable(c: PlexCollection) -> bool:
"""Effective editability: the admin marked it editable AND it's a plain manual collection — never
smart, never an external auto-list (IMDb/TMDb/…), which are managed outside Plex."""
return bool(c.editable) and not c.smart and collection_source(c) == "collection"
def collection_card(c: PlexCollection) -> dict:
return {
"id": c.rating_key,
"title": c.title,
"summary": c.summary,
"thumb": f"/api/plex/image/{c.rating_key}" if c.thumb_key else None,
"child_count": c.child_count,
"smart": c.smart,
"source": collection_source(c),
"editable": c.editable,
"can_edit": collection_editable(
c
), # whether the edit controls should be offered (admin UI)
}
class Member(NamedTuple):
"""One collection member flattened to everything the Sort control can order by. The page's two
legs are different tables (movies and shows), so there is no single query to ORDER BY — the
ordering is a pure list sort over these, which is also what makes it unit-testable."""
kind: str # "movie" | "show"
obj: Any # PlexItem | PlexShow, carried through to the card serializer
title: str # lower-cased; never NULL (the column is not nullable)
year: int | None
rating: float | None
added: datetime | None
release: date | None
duration: int | None # movies only — a show has no single runtime
last_watched: datetime | None
# Sort key -> what to read off a Member. Pinned to `PLEX_SORT_KEYS` by a test: the collection page
# shows the SAME Sort control as the browse grid, so a key this map is missing would be an option
# that silently does nothing here.
MEMBER_SORT_KEYS: dict[str, Callable[[Member], Any]] = {
"added": lambda m: m.added,
"lastwatched": lambda m: m.last_watched,
"release": lambda m: m.release,
"year": lambda m: m.year,
"rating": lambda m: m.rating,
"duration": lambda m: m.duration,
"title": lambda m: m.title,
}
# What the page ordered by before it had a control, and still its default: a film series wants to be
# read in release order, not by whenever Plex happened to ingest it.
DEFAULT_MEMBER_SORT = "year"
def sort_members(
members: list[Member], sort: str, sort_dir: str, alpha: dict[str, int] | None = None
) -> list[Member]:
"""Order a collection's members by one of `PLEX_SORT_KEYS`.
Two properties worth stating because they are easy to get wrong with a plain `reverse=True`:
members with no value for the key sink to the END in BOTH directions (matching the browse grid's
`nullslast()` — a movie with no rating is not "the worst rated"), and ties always resolve
title-ascending, so flipping the direction reverses the ranking rather than the alphabet inside
it. An unknown key falls back to the page's default instead of returning an arbitrary order.
`alpha` ranks each title in the reader's alphabet (`collation.sorted_words`); without it, titles
compare as Python strings, which puts every accented initial after "z"."""
by_title = (lambda m: alpha[m.title]) if alpha is not None else (lambda m: m.title)
key = MEMBER_SORT_KEYS.get(sort, MEMBER_SORT_KEYS[DEFAULT_MEMBER_SORT])
if key is MEMBER_SORT_KEYS["title"]:
key = by_title
ordered = sorted(members, key=by_title) # stable secondary, applied first
have = [m for m in ordered if key(m) is not None]
missing = [m for m in ordered if key(m) is None]
have.sort(key=key, reverse=sort_dir != "asc")
return have + missing
+5 -4
View File
@@ -1,10 +1,11 @@
"""Parse the "info page" rich metadata out of a full Plex metadata dict, and persist the subset that
the detail endpoints serve from the DB row.
Shared by the live info-page path (routes.plex.browse) and the background rich re-sync (plex.sync) so
the two never diverge — the same one-parser rule that keeps `people.py`'s cast handling consistent.
`rich_detail` is the pure parser; `apply_rich_detail` writes the rich-ONLY fields onto a movie/show row
(genres/directors/studio/content_rating already land from the cheap listing at `_apply_facet_fields`)."""
Shared by the live info-page path (`plex.service.enrich_movie` / `enrich_show`) and the background
rich re-sync (plex.sync) so the two never diverge — the same one-parser rule that keeps `people.py`'s
cast handling consistent. `rich_detail` is the pure parser; `apply_rich_detail` writes the rich-ONLY
fields onto a movie/show row (genres/directors/studio/content_rating already land from the cheap
listing at `_apply_facet_fields`)."""
from app.plex import people
from app.plex.plaintext import plain_text
+536
View File
@@ -0,0 +1,536 @@
"""Plex Discover's reads: the rail-surface read-model.
A consolidated payload for the dedicated Discover landing — a stack of horizontal rails:
Continue watching · Recently added · Browse-by-genre tiles · Because you watched X — plus a
separate, reshuffling Surprise-me sample. Serve-from-DB only: the recommendation rail reads the
recs the TMDB engine already persisted (`PlexItem.recommendations`), never a live TMDB call, and
every card comes from the same `movie_card`/`show_card` serializers the grid uses (so rails inherit
their `art` + rating/summary for free). The rails carry EN metadata (like the other non-grid
card paths); the HU overlay coalesces only in the main grid.
The sidebar metadata filters (genre / rating / year / collection / …) narrow EVERY rail too, so the
Discover surface behaves like the Browse grid — the same `LibraryFilters` / `meta_filter_conds` the
grid uses, applied to each rail's movie + show query (a rail may legitimately come back empty, which
the client hides). Genre-language matches the grid; duration is movie-only.
"""
from datetime import datetime
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
from app.collation import sorted_words
from app.discover.queries import RAIL_LIMIT
from app.models import User
from app.plex import rollup
from app.plex.cards import SHOW_NEW, display_title, leaf_card, movie_card, show_card
from app.plex.models import PlexItem, PlexShow, PlexState
from app.plex.query import (
LibraryFilters,
apply_meta_filters,
apply_show_state_filter,
content_libs,
genres_col,
meta_filter_conds,
top_art_keys,
)
from app.plex.tmdb import resolve_recs
from app.userscope import owned_by
GENRE_TILES = 12
SURPRISE_PER_KIND = 3 # Surprise-me: at most this many movies + this many shows (6 total)
def _scope_libs(db: Session, scope: str) -> tuple[list[int], list[int]]:
"""The movie/show library ids visible for this scope (empty list = out of scope)."""
movie_ids, show_ids = content_libs(db)
return (
movie_ids if scope in ("movie", "both") else [],
show_ids if scope in ("show", "both") else [],
)
def _movie_meta(q, p: dict, lang: str, f: LibraryFilters):
"""Apply the shared metadata filters + movie-only duration to a movie query (same as the grid)."""
q = apply_meta_filters(q, PlexItem, p, lang)
if f.duration_min is not None:
q = q.where(PlexItem.duration_s >= f.duration_min)
if f.duration_max is not None:
q = q.where(PlexItem.duration_s <= f.duration_max)
return q
def _movie_cards(db: Session, uid: int, items: list[PlexItem], lang: str) -> list[dict]:
sts = rollup.item_state_map(db, uid, [it.id for it in items])
return [movie_card(it, sts.get(it.id), lang=lang) for it in items]
def _show_cards(db: Session, uid: int, shows: list[PlexShow], lang: str) -> list[dict]:
statuses = rollup.state_map(db, uid, [s.id for s in shows])
return [show_card(s, statuses.get(s.id, SHOW_NEW), lang=lang) for s in shows]
def _recently_added(
db: Session,
uid: int,
movie_ids: list[int],
show_ids: list[int],
p: dict,
lang: str,
f: LibraryFilters,
) -> list[dict]:
"""Newest movies + shows, interleaved by `added_at` desc into one rail."""
movies = (
db.execute(
_movie_meta(
select(PlexItem).where(
PlexItem.kind == "movie", PlexItem.library_id.in_(movie_ids)
),
p,
lang,
f,
)
.order_by(PlexItem.added_at.desc().nullslast())
.limit(RAIL_LIMIT)
)
.scalars()
.all()
if movie_ids
else []
)
shows = (
db.execute(
apply_meta_filters(
select(PlexShow).where(PlexShow.library_id.in_(show_ids)), PlexShow, p, lang
)
.order_by(PlexShow.added_at.desc().nullslast())
.limit(RAIL_LIMIT)
)
.scalars()
.all()
if show_ids
else []
)
# Merge the two type-homogeneous lists by added_at, then serialize each side in one batch so the
# per-card state lookups stay batched (no N+1). Same `or NO_TS` key as the two rails below — a
# review found this one still on the older `(is not None, value)` form, i.e. one rule in two
# spellings, with the two comments explaining it differently.
tagged = [(m.added_at, "movie", m) for m in movies] + [(s.added_at, "show", s) for s in shows]
tagged.sort(key=lambda t: t[0] or rollup.NO_TS, reverse=True)
top = tagged[:RAIL_LIMIT]
# Key the serialized cards back by rating_key (globally unique in Plex — a movie and a show never
# share one), so the merge doesn't depend on object identity staying alive.
top_movies = [o for _, kind, o in top if kind == "movie"]
top_shows = [o for _, kind, o in top if kind == "show"]
mcards = {
m.rating_key: c
for m, c in zip(top_movies, _movie_cards(db, uid, top_movies, lang), strict=True)
}
scards = {
s.rating_key: c
for s, c in zip(top_shows, _show_cards(db, uid, top_shows, lang), strict=True)
}
return [mcards[o.rating_key] if kind == "movie" else scards[o.rating_key] for _, kind, o in top]
def _recently_watched(
db: Session,
uid: int,
movie_ids: list[int],
show_ids: list[int],
p: dict,
lang: str,
f: LibraryFilters,
) -> list[dict]:
"""Recently watched rail: the titles the user last FULLY watched, newest first. A movie is its own
card; a show contributes the specific EPISODE watched (carrying its show title). Movies honour the
sidebar filters (like the other rails); episodes carry no filterable metadata, so they don't."""
rows: list[tuple[datetime | None, dict]] = [] # (watched_at, card)
if movie_ids:
mq = (
select(PlexItem, PlexState)
.join(
PlexState, and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid))
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.status == "watched",
)
)
for it, st in db.execute(
_movie_meta(mq, p, lang, f)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all():
rows.append((st.watched_at, leaf_card(it, st, lang=lang)))
if show_ids:
eq = (
select(PlexItem, PlexState, PlexShow)
.join(
PlexState, and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid))
)
.join(PlexShow, PlexShow.id == PlexItem.show_id)
.where(
PlexItem.kind == "episode",
PlexItem.library_id.in_(show_ids),
PlexState.status == "watched",
)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(RAIL_LIMIT)
)
for ep, st, sh in db.execute(eq).all():
card = leaf_card(
ep, st, show_title=display_title(sh, lang), show_rk=sh.rating_key, lang=lang
)
rows.append((st.watched_at, card))
rows.sort(key=lambda r: r[0] or rollup.NO_TS, reverse=True)
return [card for _, card in rows[:RAIL_LIMIT]]
def _continue_watching(
db: Session,
uid: int,
movie_ids: list[int],
show_ids: list[int],
p: dict,
lang: str,
f: LibraryFilters,
) -> list[dict]:
"""Resume rail: in-progress movies + partially-watched shows, most-recently-progressed first.
A movie's resume time is its own `progress_updated_at`; a show's is the max across its episodes."""
rows: list[tuple[datetime | None, dict]] = [] # (resumed_at, card)
if movie_ids:
mq = (
select(PlexItem, PlexState)
.join(
PlexState,
and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid)),
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.position_seconds > 0,
# A NULL status is still "started" (mirrors apply_movie_state_filter's in_progress) —
# `notin_` alone would drop it, since `NULL NOT IN (...)` is NULL, not TRUE.
or_(
PlexState.status.is_(None),
PlexState.status.notin_(["watched", "hidden"]),
),
)
)
mrows = db.execute(
_movie_meta(mq, p, lang, f)
.order_by(PlexState.progress_updated_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all()
for it, st in mrows:
rows.append((st.progress_updated_at, movie_card(it, st, lang=lang)))
if show_ids:
# Reuse the shared per-show episode rollup (total/watched/inprog) rather than re-inlining it, plus
# its `resumed_at` = max(progress_updated_at); the in-progress predicate comes from the same shared
# `apply_show_state_filter` the grid uses, so the rail can't drift from the grid's definition.
prog = rollup.agg_subquery(db, uid, include_resumed=True).subquery()
sq = apply_show_state_filter(
select(PlexShow, prog.c.resumed_at, prog.c.total, prog.c.watched, prog.c.inprog).join(
prog, prog.c.show_id == PlexShow.id
),
prog.c.total,
prog.c.watched,
prog.c.inprog,
"in_progress",
).where(PlexShow.library_id.in_(show_ids))
srows = db.execute(
apply_meta_filters(sq, PlexShow, p, lang)
.order_by(prog.c.resumed_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all()
for sh, resumed_at, total, watched, inprog in srows:
state = rollup.show_state(total, watched, inprog)
rows.append((resumed_at, show_card(sh, state, lang=lang)))
rows.sort(key=lambda r: r[0] or rollup.NO_TS, reverse=True)
return [card for _, card in rows[:RAIL_LIMIT]]
def _genre_tiles(
db: Session, movie_ids: list[int], show_ids: list[int], p: dict, lang: str
) -> list[dict]:
"""The most-common genres across the scope, as {value, count} tiles → tap to filter the grid.
The genre vocabulary is language-specific (coalesce(genres_hu, genres) in HU), matching the grid.
The active metadata filters narrow the counts too, so the tiles agree with the filtered rails."""
counts: dict[str, int] = {}
for model, ids in ((PlexItem, movie_ids), (PlexShow, show_ids)):
if not ids:
continue
gcol = genres_col(model, lang)
base = select(func.jsonb_array_elements_text(gcol).label("g")).where(
model.library_id.in_(ids),
func.jsonb_typeof(gcol) == "array",
*meta_filter_conds(model, p, lang),
)
if model is PlexItem:
base = base.where(model.kind == "movie")
expanded = base.subquery()
for g, c in db.execute(select(expanded.c.g, func.count()).group_by(expanded.c.g)).all():
counts[g] = counts.get(g, 0) + c
alpha = {w: i for i, w in enumerate(sorted_words(db, list(counts), lang))}
top = sorted(counts.items(), key=lambda kv: (-kv[1], alpha[kv[0]]))[:GENRE_TILES]
# A 2×2 poster collage per tile: the top movies in that genre (color hash is the client fallback).
art = top_art_keys(
db,
PlexItem,
movie_ids,
genres_col(PlexItem, lang),
[g for g, _ in top],
extra_conds=(PlexItem.kind == "movie", *meta_filter_conds(PlexItem, p, lang)),
)
return [{"value": g, "count": c, "art": art.get(g, [])} for g, c in top]
def _because_you_watched(
db: Session,
uid: int,
movie_ids: list[int],
show_ids: list[int],
p: dict,
lang: str,
f: LibraryFilters,
) -> dict | None:
"""Recommendations anchored on the user's most-recently-watched, TMDB-enriched movie. Only the
recs that resolve to OUR library (by guid) become cards — the rest aren't actionable here. Returns
None (rail hidden) when there's no enriched watch history or none of its recs are in the library.
The metadata filters narrow the resolved recs (not the anchor pick) so the rail matches the grid."""
if not movie_ids:
return None
anchor = (
db.execute(
select(PlexItem)
.join(
PlexState,
and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid)),
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.status == "watched",
PlexItem.recommendations.isnot(None),
)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(1)
)
.scalars()
.first()
)
if anchor is None:
return None
resolved = resolve_recs(db, anchor.recommendations, lang=lang)
# Preserve TMDB's recommendation order, keep only in-library hits, and drop the anchor itself.
rks = [
r["rating_key"]
for r in resolved
if r.get("in_library") and r.get("rating_key") != anchor.rating_key
]
if not rks:
return None
movies = {
m.rating_key: m
for m in db.execute(
_movie_meta(
select(PlexItem).where(PlexItem.kind == "movie", PlexItem.rating_key.in_(rks)),
p,
lang,
f,
)
)
.scalars()
.all()
}
shows = {
s.rating_key: s
for s in db.execute(
apply_meta_filters(
select(PlexShow).where(PlexShow.rating_key.in_(rks)), PlexShow, p, lang
)
)
.scalars()
.all()
}
ordered_movies = [movies[rk] for rk in rks if rk in movies]
ordered_shows = [shows[rk] for rk in rks if rk in shows]
# rating_key is globally unique in Plex, so it keys the serialized cards back safely (no id()).
mcards = {
m.rating_key: c
for m, c in zip(ordered_movies, _movie_cards(db, uid, ordered_movies, lang), strict=True)
}
scards = {
s.rating_key: c
for s, c in zip(ordered_shows, _show_cards(db, uid, ordered_shows, lang), strict=True)
}
items: list[dict] = []
for rk in rks:
if rk in mcards:
items.append(mcards[rk])
elif rk in scards:
items.append(scards[rk])
if not items:
return None
return {
"anchor": {"id": anchor.rating_key, "title": display_title(anchor, lang)},
"items": items,
}
def discover_page(db: Session, user: User, scope: str, lang: str, f: LibraryFilters) -> dict:
"""The Discover landing payload — every rail in one pass (serve-from-DB), narrowed by the active
sidebar filters. Empty rails come back as empty lists (the client hides them); `because_you_watched`
is null when there's no anchor. `lang` is already normalised."""
p = f.as_dict()
movie_ids, show_ids = _scope_libs(db, scope)
return {
"scope": scope,
"continue_watching": _continue_watching(db, user.id, movie_ids, show_ids, p, lang, f),
"recently_watched": _recently_watched(db, user.id, movie_ids, show_ids, p, lang, f),
"recently_added": _recently_added(db, user.id, movie_ids, show_ids, p, lang, f),
"genres": _genre_tiles(db, movie_ids, show_ids, p, lang),
"because_you_watched": _because_you_watched(db, user.id, movie_ids, show_ids, p, lang, f),
}
def _draw(db: Session, stmt, key_col, per: int, exclude: list[str]) -> list:
"""`per` random rows of `stmt`, the previous draw's (`exclude`) left out while the pool allows: a
reshuffle over a small filtered pool kept dealing the same few titles. What the fresh rows leave
short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail."""
fresh = (
db.execute(stmt.where(key_col.not_in(exclude)).order_by(func.random()).limit(per))
.scalars()
.all()
if exclude
else []
)
if len(fresh) == per:
return list(fresh)
taken = [key_col.not_in([r.rating_key for r in fresh])] if fresh else []
rest = db.execute(stmt.where(*taken).order_by(func.random()).limit(per - len(fresh))).scalars()
return [*fresh, *rest.all()]
def _kept(
db: Session, uid: int, movie_ids: list[int], show_ids: list[int], keep: list[str], lang: str
) -> list[dict]:
"""The `keep` titles' cards in their given order, as they stand now: a title watched since the draw
stays in it, marked watched, and one gone from the scope's libraries drops out."""
movies = (
db.execute(
select(PlexItem).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexItem.rating_key.in_(keep),
)
)
.scalars()
.all()
if movie_ids
else []
)
shows = (
db.execute(
select(PlexShow).where(PlexShow.library_id.in_(show_ids), PlexShow.rating_key.in_(keep))
)
.scalars()
.all()
if show_ids
else []
)
cards = {
c["id"]: c
for c in [
*_movie_cards(db, uid, list(movies), lang),
*_show_cards(db, uid, list(shows), lang),
]
}
return [cards[k] for k in keep if k in cards]
def surprise(
db: Session,
user: User,
scope: str,
lang: str,
limit: int,
exclude: list[str],
keep: list[str],
f: LibraryFilters,
) -> dict:
"""A fresh random sample for the reshuffling Surprise-me rail (a random draw each call), narrowed by
the active filters. Only titles the user has NOT seen — a watched movie / a fully-watched show is
excluded (the point is to surface something to watch). Capped at SURPRISE_PER_KIND movies +
SURPRISE_PER_KIND shows, drawn separately then interleaved so a lopsided library doesn't crowd out
one type."""
p = f.as_dict()
movie_ids, show_ids = _scope_libs(db, scope)
if keep:
return {"scope": scope, "items": _kept(db, user.id, movie_ids, show_ids, keep, lang)}
per = min(limit, SURPRISE_PER_KIND)
movies = []
if movie_ids:
# Exclude movies this user has watched.
watched_movie = (
select(PlexState.id)
.where(
PlexState.item_id == PlexItem.id,
owned_by(PlexState.user_id, user),
PlexState.status == "watched",
)
.exists()
)
movies = _draw(
db,
_movie_meta(
select(PlexItem).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
~watched_movie,
),
p,
lang,
f,
),
PlexItem.rating_key,
per,
exclude,
)
shows = []
if show_ids:
# Exclude fully-watched shows (LEFT JOIN so a never-started show — no rollup row — still counts
# as unseen). `total IS NULL / == 0 / watched < total` == "not fully watched".
prog = rollup.agg_subquery(db, user.id).subquery()
not_fully_watched = or_(
prog.c.total.is_(None), prog.c.total == 0, prog.c.watched < prog.c.total
)
shows = _draw(
db,
apply_meta_filters(
select(PlexShow)
.outerjoin(prog, prog.c.show_id == PlexShow.id)
.where(PlexShow.library_id.in_(show_ids), not_fully_watched),
PlexShow,
p,
lang,
),
PlexShow.rating_key,
per,
exclude,
)
mcards = _movie_cards(db, user.id, movies, lang)
scards = _show_cards(db, user.id, shows, lang)
# Interleave movie/show cards, then trim to `limit`.
items: list[dict] = []
for i in range(max(len(mcards), len(scards))):
if i < len(mcards):
items.append(mcards[i])
if i < len(scards):
items.append(scards[i])
return {"scope": scope, "items": items[:limit]}
+1 -1
View File
@@ -15,7 +15,7 @@ from fastapi import Response
from sqlalchemy.orm import Session
from app import logs
from app.models import PlexCollection, PlexItem, PlexSeason, PlexShow
from app.plex.models import PlexCollection, PlexItem, PlexSeason, PlexShow
from app.plex.cache import CACHE_DIR
from app.plex.lookups import by_rating_key
+473
View File
@@ -0,0 +1,473 @@
"""The Plex browse grid's reads: the unified movies + shows feed, its search, and its facets.
A user's watch state is joined through `userscope`; the feed and its facets share every filter
(`plex.query`), so a facet count is the size of what the grid would show.
"""
from sqlalchemy import Boolean, Integer, String, and_, cast, func, literal, null, or_, select
from sqlalchemy.orm import Session, aliased
from app.collation import collated, sorted_words
from app.models import User
from app.plex import rollup
from app.plex.cards import norm_lang
from app.plex.models import PlexItem, PlexShow, PlexState
from app.plex.query import (
FIELD_COLUMNS,
PLAYBACK_CLASSES,
TS_CONFIG,
LibraryFilters,
apply_meta_filters,
apply_movie_state_filter,
apply_show_state_filter,
content_libs,
field_recheck,
genres_col,
meta_filter_conds,
movie_status_case,
playable_conds,
playback_class_expr,
show_status_case,
split_csv,
summary_col,
title_col,
vector_col,
vector_match,
)
from app.userscope import owned_by
from app.utils import to_tsquery_str
def sort_column(u, sort: str, lang: str = "en"):
"""The browse grid's sort key -> orderable column on the union subquery. Module-level (not inline
in the handler) so a test can pin its vocabulary to `PLEX_SORT_KEYS` — a key the Sort control
offers but this cannot order would quietly fall back to `added_at` and look like a broken control.
An unknown key (a stale bookmark, a hand-edited URL) deliberately keeps that fallback. The title sorts
in the reader's alphabet (`collation`)."""
return {
"added": u.c.added_at,
"title": collated(u.c.title, lang),
"year": u.c.year,
"rating": u.c.rating,
"duration": u.c.duration_s,
"release": u.c.rel,
"lastwatched": u.c.last_watched,
}.get(sort, u.c.added_at)
def library_page(
db: Session,
user: User,
*,
scope: str,
q: str | None,
field: str,
sort: str,
sort_dir: str,
show: str,
lang: str,
offset: int,
limit: int,
f: LibraryFilters,
) -> tuple[int, list, list]:
"""One page of the unified cross-library browse: (total, rows, episodes). `rows` mixes movies
and shows (`kind` says which), scoped to movie|show|both, with the shared filters/search/watch
state (a show's state is the aggregate of its episodes). On a search that also matches episodes
(and shows are in scope), `episodes` holds the matching ones as (item, state, show) — the
grouped 'Episodes' section. `lang` is already normalised."""
p = f.as_dict()
movie_lib_ids, show_lib_ids = content_libs(db)
want_movies = scope in ("movie", "both") and bool(movie_lib_ids)
want_shows = scope in ("show", "both") and bool(show_lib_ids)
ts = to_tsquery_str(q) if q else None
tsq = func.to_tsquery(TS_CONFIG, ts) if ts else None
if field not in FIELD_COLUMNS and field != "all":
field = "all"
# `episode` scope targets episode rows only — the movie/show grid stays empty; the recheck for
# the specific title/people/summary fields is a strict narrowing on top of the GIN prefilter.
grid_field = field != "episode"
recheck = field_recheck if tsq is not None and field not in ("all", "episode") else None
selects = []
if want_movies and grid_field:
st = aliased(PlexState)
m_status = movie_status_case(st)
mq = (
select(
PlexItem.rating_key.label("rk"),
literal("movie").label("kind"),
title_col(PlexItem, lang).label("title"),
PlexItem.year.label("year"),
PlexItem.rating.label("rating"),
# Truncated so the card hover-overview costs a few hundred bytes, not the full synopsis.
func.left(summary_col(PlexItem, lang), 300).label("summary"),
PlexItem.added_at.label("added_at"),
PlexItem.originally_available_at.label("rel"),
PlexItem.duration_s.label("duration_s"),
PlexItem.playable.label("playable"),
PlexItem.transcode_hw.label("transcode_hw"),
cast(null(), Integer).label("season_count"),
func.coalesce(st.position_seconds, 0).label("position_seconds"),
m_status.label("status"),
literal(False).label("resumable"),
# "last watched" sort key: when the movie was last resumed, else when it was marked
# watched. NULL for never-touched movies → nullslast sinks them (see the sort dict).
func.coalesce(st.progress_updated_at, st.watched_at).label("last_watched"),
(
func.ts_rank(vector_col(PlexItem, lang), tsq)
if tsq is not None
else literal(0.0)
).label("rank"),
)
.outerjoin(st, and_(st.item_id == PlexItem.id, owned_by(st.user_id, user)))
.where(PlexItem.kind == "movie", PlexItem.library_id.in_(movie_lib_ids))
)
mq = apply_movie_state_filter(mq, st, show)
mq = apply_meta_filters(mq, PlexItem, p, lang)
if f.duration_min is not None:
mq = mq.where(PlexItem.duration_s >= f.duration_min)
if f.duration_max is not None:
mq = mq.where(PlexItem.duration_s <= f.duration_max)
# Playback-class filter (movie-only — a show is an aggregate with no single class).
mq = mq.where(*playable_conds(split_csv(f.playable)))
if tsq is not None:
mq = mq.where(vector_match(PlexItem, lang, tsq))
if recheck is not None and (narrow := recheck(PlexItem, field, tsq, lang)) is not None:
mq = mq.where(narrow)
selects.append(mq)
if want_shows and grid_field:
agg = rollup.agg_subquery(db, user.id, include_resumed=True).subquery()
tot = func.coalesce(agg.c.total, 0)
wat = func.coalesce(agg.c.watched, 0)
inp = func.coalesce(agg.c.inprog, 0)
s_status = show_status_case(tot, wat, inp)
sq = (
select(
PlexShow.rating_key.label("rk"),
literal("show").label("kind"),
title_col(PlexShow, lang).label("title"),
PlexShow.year.label("year"),
PlexShow.rating.label("rating"),
func.left(summary_col(PlexShow, lang), 300).label("summary"),
PlexShow.added_at.label("added_at"),
PlexShow.originally_available_at.label("rel"),
cast(null(), Integer).label("duration_s"),
cast(null(), String).label("playable"),
cast(null(), Boolean).label("transcode_hw"),
PlexShow.child_count.label("season_count"),
literal(0).label("position_seconds"),
s_status.label("status"),
# An episode holds a resume point (`ShowState.resumable`, the card's Continue + Restart).
(inp > 0).label("resumable"),
# Show leg's "last watched": max(resume/watched time) across the show's episodes (see
# rollup.agg_subquery include_resumed). NULL for a never-watched show → nullslast.
agg.c.resumed_at.label("last_watched"),
(
func.ts_rank(vector_col(PlexShow, lang), tsq)
if tsq is not None
else literal(0.0)
).label("rank"),
)
.outerjoin(agg, agg.c.show_id == PlexShow.id)
.where(PlexShow.library_id.in_(show_lib_ids))
)
sq = apply_show_state_filter(sq, tot, wat, inp, show)
sq = apply_meta_filters(sq, PlexShow, p, lang)
if tsq is not None:
sq = sq.where(vector_match(PlexShow, lang, tsq))
if recheck is not None and (narrow := recheck(PlexShow, field, tsq, lang)) is not None:
sq = sq.where(narrow)
selects.append(sq)
total = 0
rows: list = []
if selects:
union = selects[0] if len(selects) == 1 else selects[0].union_all(*selects[1:])
u = union.subquery()
total = db.scalar(select(func.count()).select_from(u)) or 0
if tsq is not None:
order = [u.c.rank.desc()]
else:
col = sort_column(u, sort, lang)
order = [(col.asc() if sort_dir == "asc" else col.desc()).nullslast()]
order.append(u.c.rk.desc())
rows = list(db.execute(select(u).order_by(*order).offset(offset).limit(limit)).all())
# Episode "grouped result" section: shown for Everywhere (full-vector match, today's behaviour)
# and for the Episode scope (title-only match, empty item grid). The title/people/summary scopes
# target the movie/show grid, so they suppress episodes.
eps: list = []
if tsq is not None and want_shows and field in ("all", "episode"):
ep_st = aliased(PlexState)
# In HU an episode is found by its Hungarian title too (the overlay fills it) — and
# still by its English one, like `vector_match` does for the grid.
if field == "episode":
ep_vec = func.to_tsvector(TS_CONFIG, func.coalesce(PlexItem.title, ""))
ep_vec_hu = func.to_tsvector(TS_CONFIG, func.coalesce(PlexItem.title_hu, ""))
else:
ep_vec, ep_vec_hu = PlexItem.search_vector, PlexItem.search_vector_hu
hit = ep_vec.op("@@")(tsq)
if lang == "hu":
hit = or_(ep_vec_hu.op("@@")(tsq), hit)
ep_vec = ep_vec_hu
eps = list(
db.execute(
select(PlexItem, ep_st, PlexShow)
.outerjoin(ep_st, and_(ep_st.item_id == PlexItem.id, owned_by(ep_st.user_id, user)))
.outerjoin(PlexShow, PlexShow.id == PlexItem.show_id)
.where(
PlexItem.kind == "episode",
PlexItem.library_id.in_(show_lib_ids),
hit,
)
.order_by(func.ts_rank(ep_vec, tsq).desc())
.limit(24)
).all()
)
return total, rows, eps
def facets(db: Session, user: User, *, scope: str, show: str, lang: str, f: LibraryFilters) -> dict:
"""Available filter values for the unified library (scope movie|show|both), ADAPTED to the active
filters (faceted search): each group's values/bounds are computed with all the OTHER active
filters applied but NOT its own — so selecting a filter narrows what the other groups offer, while
a multi-select group still lets you add more. Merged across the scope's libraries; duration is
movie-only. (Search text is not factored in — facets track the sidebar filters.) `lang` is
normalised here, after the empty answer."""
empty = {
"genres": [],
"content_ratings": [],
"studios": [],
"playable": [],
"year_min": None,
"year_max": None,
"rating_max": None,
"duration_min": None,
"duration_max": None,
}
all_movie_ids, all_show_ids = content_libs(db)
movie_ids = all_movie_ids if scope in ("movie", "both") else []
show_ids = all_show_ids if scope in ("show", "both") else []
if not movie_ids and not show_ids:
return empty
lang = norm_lang(lang)
p = f.as_dict()
uid = user.id
# The shows that pass the watch-state scope, asked ONCE. The per-user episode rollup behind it is a
# full pass over the user's episodes (~10 ms on a prod-sized library), and each of the show groups
# (genres, ratings, studios, years, rating max) asked it again: half of a filtered facets call.
# The same filter on the same rows, so the counts do not change
# (test_plex_facets_grid_parity).
shows_in_scope: list[int] | None = None
if show_ids and show in ("watched", "in_progress", "unwatched"):
agg = rollup.agg_subquery(db, uid).subquery()
in_scope = select(PlexShow.id).outerjoin(agg, agg.c.show_id == PlexShow.id)
in_scope = apply_show_state_filter(
in_scope,
func.coalesce(agg.c.total, 0),
func.coalesce(agg.c.watched, 0),
func.coalesce(agg.c.inprog, 0),
show,
)
shows_in_scope = list(db.scalars(in_scope.where(PlexShow.library_id.in_(show_ids))))
def wsq(query, model, ids, pp, kind_movie, with_dur):
"""Apply the (self-excluded) metadata filters + library scope + the per-user watch-state to a
facet aggregation query, so the facets reflect exactly the visible result set."""
query = query.where(model.library_id.in_(ids), *meta_filter_conds(model, pp, lang))
if kind_movie:
query = query.where(model.kind == "movie")
if with_dur and pp.get("duration_min") is not None:
query = query.where(model.duration_s >= pp["duration_min"])
if with_dur and pp.get("duration_max") is not None:
query = query.where(model.duration_s <= pp["duration_max"])
# Playback-class filter (movie-only) — applied for the OTHER groups so an active class
# narrows them; the playable facet itself passes pp with playable=None (self-exclusion).
query = query.where(*playable_conds(split_csv(pp.get("playable"))))
# Join watch-state and reuse the grid's exact movie state filter (unified_library) —
# hidden movies never appear in the grid, so they must not inflate facet counts/bounds either.
st = aliased(PlexState)
query = query.outerjoin(st, and_(st.item_id == model.id, owned_by(st.user_id, uid)))
query = apply_movie_state_filter(query, st, show)
elif shows_in_scope is not None:
query = query.where(model.id.in_(shows_in_scope))
return query
genre_counts: dict[str, int] = {}
cr_counts: dict[str, int] = {}
studio_counts: dict[str, int] = {}
playable_counts: dict[str, int] = {}
years: list[int] = []
ratings: list[float] = []
durs: list[int] = []
def per_table(model, ids, kind_movie: bool) -> None:
if not ids:
return
# Genres — in "any"/OR mode, exclude the genre filter itself so each offered genre is a valid
# OR-addition that only widens (keep adding). In "all"/AND mode, KEEP the filter applied so we
# offer only genres that CO-OCCUR on the current result set — a non-co-occurring genre would AND
# the result down to zero, i.e. a dead-end chip. (The already-selected genres still appear, since
# every matching title carries them, so they stay togglable.)
genre_pp = p if p.get("genre_mode") == "all" else {**p, "genres": None}
# Only rows whose `genres` is a JSON array can be unnested. A genre-less item stores JSON
# `null` (SQLAlchemy's JSONB persists a Python None as JSON null, not SQL NULL), and
# jsonb_array_elements_text() raises on a scalar ("cannot extract elements from a scalar") —
# so a single metadata-poor library (e.g. one bare Shows library with no genres) 500'd the
# whole facets endpoint. Mirror the exact guard people.py already applies to the same columns.
# In HU the vocabulary comes from the active-language genre column (coalesce(genres_hu, genres)).
gcol = genres_col(model, lang)
expanded = (
wsq(
select(func.jsonb_array_elements_text(gcol).label("g")),
model,
ids,
genre_pp,
kind_movie,
True,
)
.where(func.jsonb_typeof(gcol) == "array")
.subquery()
)
for g, c in db.execute(select(expanded.c.g, func.count()).group_by(expanded.c.g)).all():
genre_counts[g] = genre_counts.get(g, 0) + c
# Content ratings — exclude its own.
crq = (
wsq(
select(model.content_rating, func.count()),
model,
ids,
{**p, "content_ratings": None},
kind_movie,
True,
)
.where(model.content_rating.isnot(None))
.group_by(model.content_rating)
)
for cr, c in db.execute(crq).all():
cr_counts[cr] = cr_counts.get(cr, 0) + c
# Studios — exclude its own (each offered studio is a valid OR-addition that only widens).
stq = (
wsq(
select(model.studio, func.count()),
model,
ids,
{**p, "studios": None},
kind_movie,
True,
)
.where(model.studio.isnot(None))
.group_by(model.studio)
)
for s, c in db.execute(stq).all():
studio_counts[s] = studio_counts.get(s, 0) + c
# Playback class (movie-only) — exclude its own so each class stays an addable OR option.
if kind_movie:
pcls = playback_class_expr()
pcq = (
wsq(
select(pcls.label("cls"), func.count()),
model,
ids,
{**p, "playable": None},
kind_movie,
True,
)
.where(pcls.isnot(None))
.group_by(pcls)
)
for cls, c in db.execute(pcq).all():
playable_counts[cls] = playable_counts.get(cls, 0) + c
# Year bounds — exclude the year filter.
by = db.execute(
wsq(
select(func.min(model.year), func.max(model.year)),
model,
ids,
{**p, "year_min": None, "year_max": None},
kind_movie,
True,
)
).one()
if by[0] is not None:
years.append(by[0])
if by[1] is not None:
years.append(by[1])
# Rating max — exclude the rating filter. Coalesce(imdb, audience) to match the score the
# rating filter and the UI badge use (IMDb preferred, audience fallback).
br = db.scalar(
wsq(
select(func.max(func.coalesce(model.imdb_rating, model.rating))),
model,
ids,
{**p, "rating_min": None},
kind_movie,
True,
)
)
if br is not None:
ratings.append(float(br))
# Duration bounds (movies) — exclude the duration filter.
if kind_movie:
bd = db.execute(
wsq(
select(func.min(model.duration_s), func.max(model.duration_s)),
model,
ids,
p,
kind_movie,
False,
)
).one()
if bd[0] is not None:
durs.append(bd[0])
if bd[1] is not None:
durs.append(bd[1])
per_table(PlexItem, movie_ids, True)
per_table(PlexShow, show_ids, False)
# Always surface the actively-selected genres (even a zero-match AND combo, where they'd otherwise
# be absent): the sidebar renders genres solely from this list and hides the whole genre section —
# chips AND the Any/All toggle — when it's empty, which would trap the user with no per-chip way to
# undo a zero-result selection. Count 0 is fine (genre chips don't display counts).
for g in split_csv(f.genres):
genre_counts.setdefault(g, 0)
# Same for the actively-selected studios, so a selected studio stays removable even at zero matches.
for s in split_csv(f.studios):
studio_counts.setdefault(s, 0)
# Keep an actively-selected playback class visible even at zero matches (the sidebar hides the whole
# section when its facet list is empty, which would trap a zero-result selection with no way to undo).
for c in split_csv(f.playable):
if c in PLAYBACK_CLASSES:
playable_counts.setdefault(c, 0)
# Names in the reader's alphabet: the genres are Hungarian words for a Hungarian reader.
alpha = {w: i for i, w in enumerate(sorted_words(db, [*genre_counts, *studio_counts], lang))}
return {
"genres": [
{"value": g, "count": c}
for g, c in sorted(genre_counts.items(), key=lambda kv: alpha[kv[0]])
],
"content_ratings": [
{"value": cr, "count": c} for cr, c in sorted(cr_counts.items(), key=lambda kv: -kv[1])
],
"studios": [
{"value": s, "count": c}
for s, c in sorted(studio_counts.items(), key=lambda kv: (-kv[1], alpha[kv[0]]))
],
# Fixed good→heavy order (not by count), so the class list reads as a severity scale.
"playable": [
{"value": c, "count": playable_counts[c]}
for c in PLAYBACK_CLASSES
if c in playable_counts
],
"year_min": min(years) if years else None,
"year_max": max(years) if years else None,
"rating_max": max(ratings) if ratings else None,
"duration_min": min(durs) if durs else None,
"duration_max": max(durs) if durs else None,
}
+463
View File
@@ -0,0 +1,463 @@
"""Plex's tables: a self-contained catalog mirrored from a local Plex server — libraries, shows,
seasons, items, people, collections, the server's playlists — and each user's private part of it:
watch state, the account link, their own playlists. Only this unit writes them; another unit reads
them along a declared edge, or asks this unit's service."""
from datetime import datetime
from sqlalchemy import (
BigInteger,
Boolean,
Computed,
Date,
DateTime,
Float,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column
from app.db import Base
from app.models import TimestampMixin, UpdatedAtMixin
# --- Plex integration (optional) -------------------------------------------------------------
# A dedicated, self-contained catalog mirrored from a locally-reachable Plex server. Playback is
# from the LOCAL physical file (Plex is metadata only); per-user watch state lives in plex_states
# so it works for users without a Plex account. Mirrors the app's "shared catalog, private
# per-user state" model. The weighted FTS search_vector uses the same unaccent_simple config as
# `videos` (migrations 0031/0032).
class PlexLibrary(Base, TimestampMixin, UpdatedAtMixin):
"""A mirrored Plex library section. Only `enabled` sections are exposed in the feed.
`plex_key` is the section id on the Plex server."""
__tablename__ = "plex_libraries"
id: Mapped[int] = mapped_column(primary_key=True)
plex_key: Mapped[str] = mapped_column(String(32), unique=True, index=True)
title: Mapped[str] = mapped_column(String(255))
kind: Mapped[str] = mapped_column(String(16)) # "movie" | "show"
enabled: Mapped[bool] = mapped_column(Boolean, default=True, server_default="true")
synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# HU-overlay support. `metadata_lang` tags the section's metadata language. An OVERLAY
# library (overlay_of set) is the Hungarian-metadata twin of another section: it is never browsable
# on its own — its titles' HU fields are written onto the matched EN rows (paired by `guid`).
metadata_lang: Mapped[str] = mapped_column(String(8), default="en", server_default="en")
overlay_of: Mapped[int | None] = mapped_column(
ForeignKey("plex_libraries.id", ondelete="SET NULL"), index=True
)
class PlexShow(Base, TimestampMixin, UpdatedAtMixin):
"""A TV show (grandparent of episodes) — holds poster + summary for the drill-down page, plus
the same filterable metadata as movies (genres/rating/content_rating/studio/people) so the TV
grid can be filtered/sorted like the movie grid. All mirrored cheaply from the show listing."""
__tablename__ = "plex_shows"
__table_args__ = (
Index("ix_plex_shows_genres_gin", "genres", postgresql_using="gin"),
Index("ix_plex_shows_directors_gin", "directors", postgresql_using="gin"),
Index("ix_plex_shows_cast_gin", "cast_names", postgresql_using="gin"),
Index("ix_plex_shows_genres_hu_gin", "genres_hu", postgresql_using="gin"),
# ix_plex_shows_collections_gin already created with collections (migration 0047).
)
id: Mapped[int] = mapped_column(primary_key=True)
rating_key: Mapped[str] = mapped_column(String(32), unique=True, index=True)
library_id: Mapped[int] = mapped_column(
ForeignKey("plex_libraries.id", ondelete="CASCADE"), index=True
)
# Canonical external id (e.g. "tmdb:12345") — the key that pairs this EN row with its HU-overlay
# twin during sync. Indexed for the guid->row overlay match.
guid: Mapped[str | None] = mapped_column(String(64), index=True)
title: Mapped[str] = mapped_column(String(512))
summary: Mapped[str | None] = mapped_column(Text)
year: Mapped[int | None] = mapped_column(Integer)
thumb_key: Mapped[str | None] = mapped_column(
String(512)
) # Plex thumb path (proxied, not stored)
art_key: Mapped[str | None] = mapped_column(String(512))
child_count: Mapped[int | None] = mapped_column(Integer) # seasons
collection_keys: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # Plex collections this show is in
added_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
# Filterable / orderable metadata mirrored from the show section listing (parallels PlexItem).
rating: Mapped[float | None] = mapped_column(Float, index=True) # Plex audienceRating (~IMDb)
content_rating: Mapped[str | None] = mapped_column(String(16), index=True)
studio: Mapped[str | None] = mapped_column(String(255), index=True) # network/studio
originally_available_at: Mapped[Date | None] = mapped_column(Date, index=True) # first air date
# none_as_null: an empty tag list must persist as SQL NULL, not JSON `null`. Without it a Python
# None round-trips as a JSON scalar that jsonb_array_elements_text() can't unnest ("cannot extract
# elements from a scalar") — the trap a metadata-poor library sprang on /facets. Same on plex_items.
genres: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
directors: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
cast_names: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
people_text: Mapped[str | None] = mapped_column(
Text
) # cast+directors, folded into search_vector
# Set once the show's FULL cast has been rich-fetched by the backfill; NULL = still the listing top-3.
people_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Rich-fetch-only detail fields, persisted by the same backfill that fills the full cast so
# the info page serves from this row instead of a live plex.metadata() call. NULL until enriched.
imdb_rating: Mapped[float | None] = mapped_column(
Float
) # IMDb-specific score (vs `rating` = audienceRating)
imdb_id: Mapped[str | None] = mapped_column(String(16)) # tt… → the clickable IMDb link
tagline: Mapped[str | None] = mapped_column(Text)
full_cast: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # ordered [{name, role, thumb}]
# --- TMDB enrichment: see the identical block on plex_items for the full rationale. Additive;
# served from the row; NULL until enriched. (`budget`/`revenue` stay NULL for shows — TV has none.) ---
tmdb_id: Mapped[int | None] = mapped_column(Integer)
tmdb_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
tmdb_rating: Mapped[float | None] = mapped_column(Float) # TMDb vote_average (0–10)
tmdb_vote_count: Mapped[int | None] = mapped_column(Integer)
budget: Mapped[int | None] = mapped_column(BigInteger)
revenue: Mapped[int | None] = mapped_column(BigInteger)
keywords: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
crew: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # [{name, job, tmdb_id, thumb}]
studios: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # TMDB production companies [str]
production_countries: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
recommendations: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
backdrops: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
# Hungarian-metadata twin fields, written by the overlay sync. NULL = no HU twin matched,
# so serving falls back to the EN columns. `people_text` is language-neutral, reused for both vectors.
title_hu: Mapped[str | None] = mapped_column(String(512))
tagline_hu: Mapped[str | None] = mapped_column(Text) # the HU overlay's tagline
summary_hu: Mapped[str | None] = mapped_column(Text)
genres_hu: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
search_vector: Mapped[object | None] = mapped_column(
TSVECTOR,
Computed(
"setweight(to_tsvector('public.unaccent_simple', coalesce(title, '')), 'A') || "
"setweight(to_tsvector('public.unaccent_simple', coalesce(people_text, '')), 'B') || "
"setweight(to_tsvector('public.unaccent_simple', left(coalesce(summary, ''), 1000)), 'C')",
persisted=True,
),
)
search_vector_hu: Mapped[object | None] = mapped_column(
TSVECTOR,
Computed(
"setweight(to_tsvector('public.unaccent_simple', coalesce(title_hu, '')), 'A') || "
"setweight(to_tsvector('public.unaccent_simple', coalesce(people_text, '')), 'B') || "
"setweight(to_tsvector('public.unaccent_simple', left(coalesce(summary_hu, ''), 1000)), 'C')",
persisted=True,
),
)
class PlexSeason(Base, TimestampMixin, UpdatedAtMixin):
"""A season under a show — used to order episodes for prev/next stepping."""
__tablename__ = "plex_seasons"
id: Mapped[int] = mapped_column(primary_key=True)
rating_key: Mapped[str] = mapped_column(String(32), unique=True, index=True)
show_id: Mapped[int] = mapped_column(
ForeignKey("plex_shows.id", ondelete="CASCADE"), index=True
)
title: Mapped[str | None] = mapped_column(String(255))
season_number: Mapped[int | None] = mapped_column(Integer, index=True)
thumb_key: Mapped[str | None] = mapped_column(String(512))
class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
"""A playable LEAF — a movie or a single episode. This is what the feed lists and the player
plays. Movies carry `library_id` and no parents; episodes carry show/season refs +
season/episode numbers. `file_path` is the Plex-side absolute path of the physical media,
mapped through `plex_path_map` to a local mount at playback time. `playable` is the
ffprobe-derived browser-compatibility class (direct | remux | transcode)."""
__tablename__ = "plex_items"
__table_args__ = (
Index("ix_plex_items_show_order", "show_id", "season_number", "episode_number"),
Index("ix_plex_items_genres_gin", "genres", postgresql_using="gin"),
Index("ix_plex_items_directors_gin", "directors", postgresql_using="gin"),
Index("ix_plex_items_cast_gin", "cast_names", postgresql_using="gin"),
Index("ix_plex_items_collections_gin", "collection_keys", postgresql_using="gin"),
Index("ix_plex_items_genres_hu_gin", "genres_hu", postgresql_using="gin"),
)
id: Mapped[int] = mapped_column(primary_key=True)
rating_key: Mapped[str] = mapped_column(String(32), unique=True, index=True)
library_id: Mapped[int] = mapped_column(
ForeignKey("plex_libraries.id", ondelete="CASCADE"), index=True
)
kind: Mapped[str] = mapped_column(String(16), index=True) # "movie" | "episode"
# Canonical external id (e.g. "tmdb:12345") — pairs a MOVIE row with its HU-overlay twin during
# sync; indexed for the guid->row match. Episodes carry it too but it's unused for them.
guid: Mapped[str | None] = mapped_column(String(64), index=True)
show_id: Mapped[int | None] = mapped_column(
ForeignKey("plex_shows.id", ondelete="CASCADE"), index=True
)
season_id: Mapped[int | None] = mapped_column(
ForeignKey("plex_seasons.id", ondelete="CASCADE"), index=True
)
season_number: Mapped[int | None] = mapped_column(Integer)
episode_number: Mapped[int | None] = mapped_column(Integer)
title: Mapped[str] = mapped_column(String(512))
summary: Mapped[str | None] = mapped_column(Text)
year: Mapped[int | None] = mapped_column(Integer)
duration_s: Mapped[int | None] = mapped_column(Integer)
thumb_key: Mapped[str | None] = mapped_column(String(512))
art_key: Mapped[str | None] = mapped_column(String(512))
cast: Mapped[list | None] = mapped_column(JSON) # [role names]
# Physical media (from the Plex Part).
file_path: Mapped[str | None] = mapped_column(Text) # Plex-side absolute path
part_key: Mapped[str | None] = mapped_column(String(255)) # /library/parts/... (fallback fetch)
container: Mapped[str | None] = mapped_column(String(16))
codec_video: Mapped[str | None] = mapped_column(String(32))
codec_audio: Mapped[str | None] = mapped_column(String(32))
size_bytes: Mapped[int | None] = mapped_column(BigInteger)
# Browser-playability class from ffprobe: "direct" | "remux" | "transcode" (None = unprobed).
playable: Mapped[str | None] = mapped_column(String(12), index=True)
probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Transcode HW-capability facet (the playback-class filter). NULL for non-transcode items; for a
# `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
# AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
# breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
# set codec-only at sync (bit depth unknown from the cheap listing), then refined by the bit-depth
# backfill (`_probe_media_batch`) which downgrades a 10-bit/HDR HEVC to False. The one filterable
# home of the invariant `_want_hw`'s source gate computes at play time — kept in sync via the shared
# classifier so the catalog facet and the runtime encoder choice never disagree by codec.
transcode_hw: Mapped[bool | None] = mapped_column(Boolean, index=True)
# Video bit depth (8/10/12…) from the Plex media Stream, filled by the bit-depth backfill. NULL =
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw`.
video_bit_depth: Mapped[int | None] = mapped_column(Integer)
# Watermark for the bit-depth backfill: when this item's media Stream was last probed for
# bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are
# probed (the rest are SW-only by codec and need no bit-depth); see `_probe_media_batch`.
media_probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
added_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
# Filterable / orderable metadata mirrored from the Plex section listing (no extra API calls;
# `rating` = Plex audienceRating, which is ~the IMDb score). The arrays back genre/person/studio
# filters (GIN-indexed) and double as content signals for a future watch-habit recommender.
rating: Mapped[float | None] = mapped_column(Float, index=True)
content_rating: Mapped[str | None] = mapped_column(String(16), index=True)
studio: Mapped[str | None] = mapped_column(String(255), index=True)
originally_available_at: Mapped[Date | None] = mapped_column(Date, index=True)
genres: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
directors: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
cast_names: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
# rating_keys of the Plex collections this movie belongs to (GIN — filter + card-strip siblings).
collection_keys: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
# Cast + director names, space-joined — folded into search_vector (weight B) so the search box
# finds titles by a person's name (and ranks them above summary-only mentions).
people_text: Mapped[str | None] = mapped_column(Text)
# Set once the movie's FULL cast has been rich-fetched by the backfill; NULL = still the listing top-3.
people_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Rich-fetch-only detail fields, persisted by the same backfill that fills the full cast so
# the info page serves from this row instead of a live plex.metadata() call. NULL until enriched.
imdb_rating: Mapped[float | None] = mapped_column(
Float
) # IMDb-specific score (vs `rating` = audienceRating)
imdb_id: Mapped[str | None] = mapped_column(String(16)) # tt… → the clickable IMDb link
tagline: Mapped[str | None] = mapped_column(Text)
full_cast: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # ordered [{name, role, thumb}]
# --- TMDB enrichment: matched by the canonical `guid` ("tmdb:12345"), filled by the TMDB
# backfill job, SERVED FROM THIS ROW (never a live TMDB call). Additive — imdb_rating/rating
# above stay Plex-sourced; `tmdb_rating` is the distinct TMDb community score. NULL until enriched;
# `tmdb_enriched_at` is the "tried" watermark (stamped on a definitive no-match too, so an
# unresolvable title isn't re-fetched every run; left NULL only on a transient fetch error). ---
tmdb_id: Mapped[int | None] = mapped_column(Integer)
tmdb_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
tmdb_rating: Mapped[float | None] = mapped_column(Float) # TMDb vote_average (0–10)
tmdb_vote_count: Mapped[int | None] = mapped_column(Integer)
budget: Mapped[int | None] = mapped_column(BigInteger) # USD (movies; TV has none)
revenue: Mapped[int | None] = mapped_column(BigInteger)
keywords: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
crew: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # [{name, job, tmdb_id, thumb}]
studios: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # TMDB production companies [str]
production_countries: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
# [{tmdb_id, title, poster, media_type}] — resolved to library rows (by guid) at serve time.
recommendations: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
backdrops: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # TMDB backdrop file_paths [str] (crossfade pool)
# Hungarian-metadata twin fields, written by the overlay sync. NULL = no HU twin matched
# (serving falls back to the EN columns). `people_text` is language-neutral, reused for both vectors.
title_hu: Mapped[str | None] = mapped_column(String(512))
tagline_hu: Mapped[str | None] = mapped_column(Text) # the HU overlay's tagline
summary_hu: Mapped[str | None] = mapped_column(Text)
genres_hu: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
search_vector: Mapped[object | None] = mapped_column(
TSVECTOR,
Computed(
"setweight(to_tsvector('public.unaccent_simple', coalesce(title, '')), 'A') || "
"setweight(to_tsvector('public.unaccent_simple', coalesce(people_text, '')), 'B') || "
"setweight(to_tsvector('public.unaccent_simple', left(coalesce(summary, ''), 1000)), 'C')",
persisted=True,
),
)
search_vector_hu: Mapped[object | None] = mapped_column(
TSVECTOR,
Computed(
"setweight(to_tsvector('public.unaccent_simple', coalesce(title_hu, '')), 'A') || "
"setweight(to_tsvector('public.unaccent_simple', coalesce(people_text, '')), 'B') || "
"setweight(to_tsvector('public.unaccent_simple', left(coalesce(summary_hu, ''), 1000)), 'C')",
persisted=True,
),
)
class PlexPerson(Base, TimestampMixin, UpdatedAtMixin):
"""A small cast/crew headshot cache, keyed by name, so person filter-chips (the cards for the
actors currently in the filter) can show a photo + count without a live per-card metadata fetch.
Populated opportunistically when an item's RICH metadata is fetched — the info-page view and the
periodic rich re-sync — since the cheap section listing that drives the main sync only carries the
top few cast. Rows whose name no longer appears in ANY item's cast_names/directors (a library
removal orphaned them) are pruned by the maintenance sweep, mirroring how Plex drops orphaned
metadata. The filterable cast lives in plex_items/plex_shows.cast_names (self-cleaning with the
row); this table is the people's own facts: the headshot link, the TMDB link and the TMDB bio
(the headshot FILE lives in the image cache, filled by the people_enrich job)."""
__tablename__ = "plex_people"
id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str] = mapped_column(String(255), unique=True, index=True)
photo_thumb: Mapped[str | None] = mapped_column(Text) # proxied /person-image URL, or None
# TMDB person id, linked by name from a title's TMDB credits (the /person
# biography fetch builds on it). NULL until a TMDB-enriched title names this person.
tmdb_person_id: Mapped[int | None] = mapped_column(Integer)
# --- TMDB person enrichment: fetched from /person/{tmdb_person_id} by the people_enrich job,
# or by the first person-page open the job hasn't reached, then SERVED FROM THE ROW. `tmdb_enriched_at` is the watermark (set even on a
# no-op fetch so it isn't retried every open). `known_for` = top combined-credits [{tmdb_id,
# media_type, title}], resolved to library rows by guid at serve time (like recommendations). ---
biography: Mapped[str | None] = mapped_column(Text)
# TMDB's Hungarian biography, from the same request's `translations`; a Hungarian reader is
# served it, and the instance-language `biography` when TMDB has none. Birthplace, aliases and
# keywords have no Hungarian on TMDB, so they stay in the instance language.
biography_hu: Mapped[str | None] = mapped_column(Text)
birthday: Mapped[str | None] = mapped_column(String(10)) # "YYYY-MM-DD", served as-is
deathday: Mapped[str | None] = mapped_column(String(10))
place_of_birth: Mapped[str | None] = mapped_column(Text)
known_for_department: Mapped[str | None] = mapped_column(String(64))
also_known_as: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
known_for: Mapped[list | None] = mapped_column(
JSONB(none_as_null=True)
) # [{tmdb_id, media_type, title}]
tmdb_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class PlexCollection(Base, TimestampMixin, UpdatedAtMixin):
"""A mirrored Plex collection (a grouping of movies or shows — franchises like Avatar, curated
sets like MCU, or auto/community lists like IMDb Top 250). Mirrored in full by the background
sync so reads never hit Plex's slow collection queries. Membership is stored as `collection_keys`
on the member movies (plex_items) / shows (plex_shows). `smart` = Plex's query-based flag;
`editable` = we may write changes back to Plex (default False = read-only; set for curated ones)."""
__tablename__ = "plex_collections"
id: Mapped[int] = mapped_column(primary_key=True)
rating_key: Mapped[str] = mapped_column(String(32), unique=True, index=True)
library_id: Mapped[int] = mapped_column(
ForeignKey("plex_libraries.id", ondelete="CASCADE"), index=True
)
title: Mapped[str] = mapped_column(String(512))
summary: Mapped[str | None] = mapped_column(Text)
thumb_key: Mapped[str | None] = mapped_column(String(512))
child_count: Mapped[int | None] = mapped_column(Integer)
smart: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
editable: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class PlexState(Base, UpdatedAtMixin):
"""Per-user watch state for a Plex item — mirrors VideoState. Lives in Siftlode's own DB so
it works for users WITHOUT a Plex account (the whole point of the access-layer design).
`synced_to_plex` marks whether this state was pushed back to a linked Plex account."""
__tablename__ = "plex_states"
__table_args__ = (UniqueConstraint("user_id", "item_id", name="uq_plex_user_item"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
item_id: Mapped[int] = mapped_column(
ForeignKey("plex_items.id", ondelete="CASCADE"), index=True
)
status: Mapped[str] = mapped_column(String(12), default="new", server_default="new")
position_seconds: Mapped[int] = mapped_column(
Integer, default=0, server_default="0", nullable=False
)
watched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
progress_updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
synced_to_plex: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class PlexLink(Base, UpdatedAtMixin):
"""Links a Siftlode user to a Plex account for two-way watch-state sync (one row per user).
For the owner (the MVP scope) the row uses the server's admin token — `uses_admin=True`,
`plex_token_enc=None` — so no separate Plex login is needed; the admin token already reads/writes
the owner account's watch state. `plex_account_id` is that Plex account's numeric id, used to
filter the shared watch-history feed to this user (Phase C). Reserved for later: a friend can link
their OWN Plex account via PIN OAuth, storing a personal `plex_token_enc` (uses_admin=False) — the
sync loop then iterates link rows uniformly.
`initial_import_done` guards the one-time "Plex is master" import; `last_watch_sync_at` is the
high-water mark for the incremental Plex→Siftlode pass."""
__tablename__ = "plex_link"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), unique=True, index=True
)
uses_admin: Mapped[bool] = mapped_column(Boolean, default=True, server_default="true")
plex_token_enc: Mapped[str | None] = mapped_column(Text) # P5b friend tokens (encrypted)
plex_account_id: Mapped[int | None] = mapped_column(Integer)
plex_username: Mapped[str | None] = mapped_column(String(255))
sync_enabled: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
initial_import_done: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false"
)
last_watch_sync_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
linked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class PlexPlaylist(Base, TimestampMixin, UpdatedAtMixin):
"""A per-user ORDERED watch-list of Plex items — Siftlode-native (lives in our own DB, works for
users WITHOUT a Plex account, like plex_states). Unlike collections (shared, library-level), a
playlist is owned by one user and its items are ordered. Optional Plex-direction sync is a later
phase (`plex_rating_key` set once pushed back to the owner's Plex account)."""
__tablename__ = "plex_playlists"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
title: Mapped[str] = mapped_column(String(512))
plex_rating_key: Mapped[str | None] = mapped_column(String(32))
class PlexPlaylistItem(Base):
"""One ordered entry in a playlist. `position` (0-based) orders playback."""
__tablename__ = "plex_playlist_items"
__table_args__ = (UniqueConstraint("playlist_id", "item_id", name="uq_plex_playlist_item"),)
id: Mapped[int] = mapped_column(primary_key=True)
playlist_id: Mapped[int] = mapped_column(ForeignKey("plex_playlists.id", ondelete="CASCADE"))
item_id: Mapped[int] = mapped_column(
ForeignKey("plex_items.id", ondelete="CASCADE"), index=True
)
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0", nullable=False)
+1 -1
View File
@@ -13,7 +13,7 @@ from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.models import PlexItem, PlexLibrary, PlexShow
from app.plex.models import PlexItem, PlexLibrary, PlexShow
from app.plex import sync as plex_sync
from app.plex.client import PlexClient
from app.plex.genres import hu_genres
+1 -1
View File
@@ -15,7 +15,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.models import PlexPerson
from app.plex.models import PlexPerson
from app.plex import imgcache
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
+507
View File
@@ -0,0 +1,507 @@
"""Plex's reads for the HTTP layer: the mirrored rows a page shows, and a user's watch state on them.
A user's own rows (`PlexState`) are read through `userscope`, so "someone else's row" is the same
answer as a missing one. A rating key with a NUL in it names no row (`lookups.by_rating_key`).
"""
from typing import Literal
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app.collation import collated
from app.models import User
from app.plex import rollup
from app.plex.lookups import by_rating_key
from app.plex.models import (
PlexCollection,
PlexItem,
PlexLibrary,
PlexLink,
PlexPerson,
PlexPlaylist,
PlexPlaylistItem,
PlexSeason,
PlexShow,
PlexState,
)
from app.plex.query import content_libs, runtime_by_key, title_col, top_art_keys
from app.userscope import owned_by, owned_row
from app.utils import like_escape
# --- one row ---------------------------------------------------------------------------------
def item(db: Session, rating_key: str) -> PlexItem | None:
return by_rating_key(db, PlexItem, rating_key)
def movie(db: Session, rating_key: str) -> PlexItem | None:
"""The movie with this key — an episode's key is not one."""
it = item(db, rating_key)
return it if it is not None and it.kind == "movie" else None
def show(db: Session, rating_key: str) -> PlexShow | None:
return by_rating_key(db, PlexShow, rating_key)
def collection(db: Session, rating_key: str) -> PlexCollection | None:
return by_rating_key(db, PlexCollection, rating_key)
def season(db: Session, rating_key: str) -> PlexSeason | None:
return by_rating_key(db, PlexSeason, rating_key)
def season_episodes(db: Session, se: PlexSeason) -> list[PlexItem]:
return list(
db.execute(
select(PlexItem).where(PlexItem.season_id == se.id, PlexItem.kind == "episode")
).scalars()
)
def show_by_id(db: Session, show_id: int) -> PlexShow | None:
return db.get(PlexShow, show_id)
def season_by_id(db: Session, season_id: int) -> PlexSeason | None:
return db.get(PlexSeason, season_id)
def library(db: Session, library_id: int) -> PlexLibrary | None:
return db.get(PlexLibrary, library_id)
def library_by_key(db: Session, plex_key: str) -> PlexLibrary | None:
return (
db.execute(select(PlexLibrary).where(PlexLibrary.plex_key == str(plex_key)))
.scalars()
.first()
)
def library_key(db: Session, library_id: int | None) -> str | None:
"""The Plex section key for a library id — the info/show page passes it to the admin collection
editor (in the unified cross-library view there's no single library prop otherwise)."""
if library_id is None:
return None
lib = db.get(PlexLibrary, library_id)
return lib.plex_key if lib else None
def person(db: Session, name: str) -> PlexPerson | None:
return db.execute(select(PlexPerson).where(PlexPerson.name == name)).scalars().first()
# --- a user's watch state --------------------------------------------------------------------
def state(db: Session, user: User | int, item_id: int) -> PlexState | None:
return (
db.execute(
select(PlexState).where(owned_by(PlexState.user_id, user), PlexState.item_id == item_id)
)
.scalars()
.first()
)
# --- a show ----------------------------------------------------------------------------------
def seasons(db: Session, sh: PlexShow) -> list[PlexSeason]:
return list(
db.execute(
select(PlexSeason)
.where(PlexSeason.show_id == sh.id)
.order_by(PlexSeason.season_number.asc().nullsfirst())
).scalars()
)
def episodes(db: Session, sh: PlexShow) -> list[PlexItem]:
"""A show's episodes in season/episode order."""
return list(
db.execute(
select(PlexItem)
.where(PlexItem.show_id == sh.id, PlexItem.kind == "episode")
.order_by(
PlexItem.season_number.asc().nullsfirst(),
PlexItem.episode_number.asc().nullsfirst(),
)
).scalars()
)
def episode_keys(db: Session, show_id: int) -> list[str]:
"""The rating keys of a show's episodes, in the order `episodes` gives."""
return list(
db.execute(
select(PlexItem.rating_key)
.where(PlexItem.show_id == show_id, PlexItem.kind == "episode")
.order_by(
PlexItem.season_number.asc().nullsfirst(),
PlexItem.episode_number.asc().nullsfirst(),
)
).scalars()
)
def shows_by_keys(db: Session, rating_keys: list[str], exclude: int) -> dict[str, PlexShow]:
"""The mirrored shows among these keys, but `exclude`, by key."""
return {
s.rating_key: s
for s in db.execute(
select(PlexShow).where(PlexShow.rating_key.in_(rating_keys), PlexShow.id != exclude)
).scalars()
}
# --- libraries -------------------------------------------------------------------------------
def browsable_libraries(db: Session, lang: str) -> list[tuple[PlexLibrary, int]]:
"""The mirrored libraries a user can browse, each with its title count (movies, or shows). An
HU overlay library is left out: it is never browsed, it only feeds its EN twin's Hungarian
metadata."""
libs = db.execute(
select(PlexLibrary)
.where(PlexLibrary.overlay_of.is_(None))
.order_by(PlexLibrary.kind.desc(), collated(PlexLibrary.title, lang))
).scalars()
out = []
for lib in libs:
if lib.kind == "movie":
count = db.scalar(
select(func.count(PlexItem.id)).where(
PlexItem.library_id == lib.id, PlexItem.kind == "movie"
)
)
else:
count = db.scalar(select(func.count(PlexShow.id)).where(PlexShow.library_id == lib.id))
out.append((lib, count or 0))
return out
# --- people ----------------------------------------------------------------------------------
def person_photos(db: Session, names: list[str]) -> dict[str, str | None]:
return {
p.name: p.photo_thumb
for p in db.execute(select(PlexPerson).where(PlexPerson.name.in_(names))).scalars()
}
def cast_count(db: Session, name: str, movie_lib_ids: list[int], show_lib_ids: list[int]) -> int:
"""How many movies (in `movie_lib_ids`) and shows (in `show_lib_ids`) name this person in their
cast. An empty id list leaves its side out."""
count = 0
if movie_lib_ids:
count += (
db.scalar(
select(func.count(PlexItem.id)).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_lib_ids),
PlexItem.cast_names.contains([name]),
)
)
or 0
)
if show_lib_ids:
count += (
db.scalar(
select(func.count(PlexShow.id)).where(
PlexShow.library_id.in_(show_lib_ids),
PlexShow.cast_names.contains([name]),
)
)
or 0
)
return count
def filmography(
db: Session,
kind: Literal["movie", "show"],
name: str,
lib_ids: list[int],
lang: str,
limit: int,
) -> list:
"""The movies or shows in these libraries whose cast or directors name this person, newest
first."""
model: type[PlexItem] | type[PlexShow] = PlexItem if kind == "movie" else PlexShow
query = select(model).where(
model.library_id.in_(lib_ids),
or_(model.cast_names.contains([name]), model.directors.contains([name])),
)
if model is PlexItem:
query = query.where(PlexItem.kind == "movie")
return list(
db.execute(
query.order_by(
model.year.desc().nullslast(), collated(title_col(model, lang), lang)
).limit(limit)
).scalars()
)
# --- collections -----------------------------------------------------------------------------
def collections_page(
db: Session, library: str | None, q: str | None, keys: list[str] | None, lang: str
) -> list[PlexCollection]:
"""Collections, most-populated first: one library's (by its plex key), else every enabled
library's; name-filtered by `q`; narrowed to `keys` when given (an empty list asks for none)."""
if library is not None:
lib = library_by_key(db, library)
if lib is None:
return []
query = select(PlexCollection).where(PlexCollection.library_id == lib.id)
else:
enabled_ids = list(
db.execute(
select(PlexLibrary.id).where(
PlexLibrary.enabled.is_(True), PlexLibrary.overlay_of.is_(None)
)
).scalars()
)
if not enabled_ids:
return []
query = select(PlexCollection).where(PlexCollection.library_id.in_(enabled_ids))
term = (q or "").strip()
if term:
query = query.where(PlexCollection.title.ilike(f"%{like_escape(term)}%"))
if keys is not None:
# An explicit but empty `keys=` asks for nothing, which is not the same as no `keys` (every
# collection): answered here without a query. The empty IN () would also match nothing.
if not keys:
return []
query = query.where(PlexCollection.rating_key.in_(keys))
return list(
db.execute(
query.order_by(
PlexCollection.child_count.desc().nullslast(), collated(PlexCollection.title, lang)
)
).scalars()
)
def library_collections(db: Session, library_id: int, lang: str) -> list[PlexCollection]:
return list(
db.execute(
select(PlexCollection)
.where(PlexCollection.library_id == library_id)
.order_by(collated(PlexCollection.title, lang))
).scalars()
)
def collections_by_keys(db: Session, keys: list[str]) -> list[PlexCollection]:
return list(
db.execute(select(PlexCollection).where(PlexCollection.rating_key.in_(keys))).scalars()
)
def collection_named(db: Session, library_id: int, title: str) -> PlexCollection | None:
"""A collection of this library with this name, case aside."""
return (
db.execute(
select(PlexCollection).where(
PlexCollection.library_id == library_id,
func.lower(PlexCollection.title) == title.lower(),
)
)
.scalars()
.first()
)
def collection_members(
db: Session, rating_key: str, lang: str
) -> tuple[list[PlexItem], list[PlexShow]]:
"""A collection's movies and shows, each by year then title."""
movies = db.execute(
select(PlexItem)
.where(PlexItem.kind == "movie", PlexItem.collection_keys.contains([rating_key]))
.order_by(PlexItem.year.asc().nullslast(), collated(title_col(PlexItem, lang), lang))
).scalars()
shows = db.execute(
select(PlexShow)
.where(PlexShow.collection_keys.contains([rating_key]))
.order_by(PlexShow.year.asc().nullslast(), collated(title_col(PlexShow, lang), lang))
).scalars()
return list(movies), list(shows)
def library_movies_in(
db: Session, library_id: int, rating_key: str, lang: str, limit: int
) -> list[PlexItem]:
"""The movies of one library in this collection, by year then title."""
return list(
db.execute(
select(PlexItem)
.where(
PlexItem.library_id == library_id,
PlexItem.kind == "movie",
PlexItem.collection_keys.contains([rating_key]),
)
.order_by(PlexItem.year.asc().nullslast(), collated(title_col(PlexItem, lang), lang))
.limit(limit)
).scalars()
)
def show_rollups(db: Session, user_id: int, show_ids: list[int]) -> list:
"""The user's episode rollup (total, watched, in progress, last resumed) of each of these shows
that has episodes."""
if not show_ids:
return []
return list(
db.execute(
rollup.agg_subquery(db, user_id, include_resumed=True).where(
PlexItem.show_id.in_(show_ids)
)
).all()
)
def collection_tiles(
db: Session, keys: list[str]
) -> tuple[dict[str, list[str]], dict[str, list[str]], dict[str, int]]:
"""What each collection's tile shows: the top two movie members' keys, the top two show members'
keys (a collection lives in one library, so exactly one side has any), and its movies' total
runtime (a show has no single duration)."""
movie_ids, show_ids = content_libs(db)
art_m = top_art_keys(db, PlexItem, movie_ids, PlexItem.collection_keys, keys, per=2)
art_s = top_art_keys(db, PlexShow, show_ids, PlexShow.collection_keys, keys, per=2)
runtime = runtime_by_key(db, PlexItem, movie_ids, PlexItem.collection_keys, keys)
return art_m, art_s, runtime
# --- the admin page --------------------------------------------------------------------------
def sample_file_path(db: Session) -> str | None:
"""Any mirrored item's Plex-side file path — a sample to test the mount and path map against."""
return db.scalar(select(PlexItem.file_path).where(PlexItem.file_path.isnot(None)))
def watch_link(db: Session, user: User | int) -> PlexLink | None:
"""The user's Plex watch-sync link, if they have one."""
return db.execute(select(PlexLink).where(owned_by(PlexLink.user_id, user))).scalars().first()
# --- a user's playlists ----------------------------------------------------------------------
def own_playlist(
db: Session, user: User, playlist_id: int, *, lock: bool = False
) -> PlexPlaylist | None:
"""The user's playlist with this id; someone else's is the same answer as a missing one. `lock`
takes it FOR UPDATE: every add, remove and reorder does, so two of them never read the same max
position, and the list row is always locked before its items."""
return owned_row(db, PlexPlaylist, playlist_id, user, lock=lock)
def playlists_of(db: Session, user: User) -> list[PlexPlaylist]:
"""The user's playlists, newest-touched first."""
return list(
db.execute(
select(PlexPlaylist)
.where(owned_by(PlexPlaylist.user_id, user))
.order_by(PlexPlaylist.updated_at.desc())
).scalars()
)
def playlist_count(db: Session, playlist_id: int) -> int:
return (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(
PlexPlaylistItem.playlist_id == playlist_id
)
)
or 0
)
def _playlist_rows(playlist_id: int):
return (
select(PlexPlaylistItem, PlexItem)
.join(PlexItem, PlexItem.id == PlexPlaylistItem.item_id)
.where(PlexPlaylistItem.playlist_id == playlist_id)
.order_by(PlexPlaylistItem.position, PlexPlaylistItem.id)
)
def playlist_cover(db: Session, playlist_id: int) -> str | None:
"""The rating key of a playlist's first item — its cover."""
first = db.execute(_playlist_rows(playlist_id)).first()
return first[1].rating_key if first else None
def playlist_items(db: Session, playlist_id: int) -> list[PlexItem]:
"""A playlist's items in play order."""
return [it for _row, it in db.execute(_playlist_rows(playlist_id)).all()]
def playlist_holds(db: Session, playlist_id: int, item_ids: set[int]) -> int:
"""How many of these items the playlist holds."""
if not item_ids:
return 0
return (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(
PlexPlaylistItem.playlist_id == playlist_id,
PlexPlaylistItem.item_id.in_(item_ids),
)
)
or 0
)
def item_id(db: Session, rating_key: str) -> int | None:
"""The id of the mirrored item with this key. Unlike `item`, a NUL in the key reaches the query,
which the app answers as a 422 (`main._nul_in_text`), as for every other compared string."""
return db.scalar(select(PlexItem.id).where(PlexItem.rating_key == str(rating_key)))
def item_ids(db: Session, rating_keys: list[str]) -> set[int]:
"""The ids of the mirrored items with these keys."""
if not rating_keys:
return set()
return set(
db.execute(select(PlexItem.id).where(PlexItem.rating_key.in_(rating_keys))).scalars()
)
def items_by_keys(db: Session, rating_keys: list) -> list[PlexItem]:
"""The mirrored items with these keys, in the given order; an unknown key and a repeat drop out."""
order = [str(k) for k in rating_keys if str(k)]
if not order:
return []
by_rk = {
it.rating_key: it
for it in db.execute(select(PlexItem).where(PlexItem.rating_key.in_(order))).scalars()
}
seen: set[str] = set()
out: list[PlexItem] = []
for rk in order:
it = by_rk.get(rk)
if it is not None and rk not in seen:
seen.add(rk)
out.append(it)
return out
def shows_by_ids(db: Session, show_ids: set[int]) -> dict[int, PlexShow]:
if not show_ids:
return {}
return {
sh.id: sh for sh in db.execute(select(PlexShow).where(PlexShow.id.in_(show_ids))).scalars()
}
+2 -2
View File
@@ -10,7 +10,7 @@ from sqlalchemy import and_, case, func, or_, select, type_coerce
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Session
from app.models import PlexItem, PlexLibrary
from app.plex.models import PlexItem, PlexLibrary
from app.plex.genres import hu_genre
TS_CONFIG = "public.unaccent_simple"
@@ -236,7 +236,7 @@ def apply_meta_filters(q, model, p: dict, lang: str = "en"):
# The 4 classes the feed filter + facet expose, derived from the stored `playable` + `transcode_hw`.
# ONE definition (this SQL CASE) so the filter WHERE and the facet counts can't drift: `playable_conds`
# filters on it, `facets` groups by it. transcode_hw IS NOT True groups NULL (unprobed) with software.
# Public — it is the shared source of truth imported by routes.plex.browse (facet order + zero-fill).
# Public — it is the shared source of truth imported by plex.library (facet order + zero-fill).
PLAYBACK_CLASSES = ("direct", "remux", "transcode_hw", "transcode_sw")
+8 -4
View File
@@ -8,7 +8,9 @@ from datetime import datetime, timezone
from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.orm import Session, aliased
from app.models import PlexItem, PlexState
from app.models import User
from app.userscope import owned_by
from app.plex.models import PlexItem, PlexState
from app.plex.cards import ShowState
# Sort sentinel for a rail row whose timestamp is missing, so `key=lambda r: r[0] or NO_TS` reads the
@@ -92,7 +94,7 @@ def agg_subquery(db: Session, user_id: int, include_resumed: bool = False):
)
return (
select(*cols)
.outerjoin(ps, and_(ps.item_id == PlexItem.id, ps.user_id == user_id))
.outerjoin(ps, and_(ps.item_id == PlexItem.id, owned_by(ps.user_id, user_id)))
.where(PlexItem.kind == "episode", PlexItem.show_id.isnot(None))
.group_by(PlexItem.show_id)
)
@@ -105,7 +107,7 @@ def state_map(db: Session, user_id: int, show_ids: list[int]) -> dict[int, ShowS
return {r.show_id: show_state(r.total, r.watched, r.inprog) for r in rows}
def item_state_map(db: Session, user_id: int, item_ids: list[int]) -> dict[int, PlexState]:
def item_state_map(db: Session, user_id: User | int, item_ids: list[int]) -> dict[int, PlexState]:
"""A user's PlexState rows for a set of leaf items (movies or episodes), keyed by item_id — the
leaf analogue of `state_map` (which rolls a show's episodes up to a `ShowState`). One batched
fetch, so the caller serializes cards without a per-card query. Empty in → empty dict (no `IN ()`)."""
@@ -114,6 +116,8 @@ def item_state_map(db: Session, user_id: int, item_ids: list[int]) -> dict[int,
return {
s.item_id: s
for s in db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id.in_(item_ids))
select(PlexState).where(
owned_by(PlexState.user_id, user_id), PlexState.item_id.in_(item_ids)
)
).scalars()
}
+413
View File
@@ -0,0 +1,413 @@
"""Plex's writes for the HTTP layer, one function per use case. None of them commits: the handler
does, so a page that fails to save an enrichment still renders, and a Plex write and its local
mirror land together.
The rest of the mirror's writes stay with the sync (`plex.sync`): a collection's targeted re-sync
after a write-back, its local cleanup after a delete.
"""
from datetime import datetime, timezone
from sqlalchemy import CursorResult, delete, func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app import logs
from app.db import SessionLocal
from app.plex import detail as plex_detail
from app.plex import people as plex_people
from app.plex import tmdb as plex_tmdb
from app.plex.lookups import by_rating_key
from app.models import User
from app.plex import queries, rollup
from app.plex.models import (
PlexCollection,
PlexItem,
PlexLibrary,
PlexLink,
PlexPerson,
PlexPlaylist,
PlexPlaylistItem,
PlexShow,
PlexState,
)
from app.userscope import owned_by
log = logs.get_logger(__name__)
def enrich_show(db: Session, sh: PlexShow, meta: dict) -> None:
"""Fill a not-yet-enriched show from its live metadata: the full cast (filterable), IMDb and
tagline, stamped so later views serve from the row."""
plex_people.enrich_row_people(db, sh, plex_people.cast_from_meta(meta))
plex_detail.apply_rich_detail(sh, meta)
sh.people_enriched_at = datetime.now(timezone.utc)
def enrich_movie(db: Session, it: PlexItem, meta: dict) -> None:
"""The same for a movie. The listing sync stored only its top ~3 cast; the live metadata
carries all of them."""
cast = plex_people.cast_from_meta(meta)
if cast:
plex_people.enrich_row_people(db, it, cast)
plex_detail.apply_rich_detail(it, meta)
it.people_enriched_at = datetime.now(timezone.utc)
def set_collection_editable(col: PlexCollection, editable: bool) -> None:
col.editable = editable
def keep_created_collection(
db: Session, lib: PlexLibrary, rating_key: str, title: str, seed: str
) -> None:
"""A collection Plex made, whose follow-up read failed: its row is kept, editable. The full sync
preserves `editable` but creates a missing row read-only, so the collection would drop out of
every "add to" list; and the duplicate-name check reads these rows, so a retry of the same name
would make a second one on Plex. Plex made it with exactly the seed in it, so the seed's row says
so too — otherwise the title showed unticked in the collection it was just added to, and the
collection empty, until that sync."""
db.add(
PlexCollection(
rating_key=rating_key, library_id=lib.id, title=title, editable=True, child_count=1
)
)
Model = PlexItem if lib.kind == "movie" else PlexShow
if row := by_rating_key(db, Model, seed):
row.collection_keys = list(dict.fromkeys([*(row.collection_keys or []), rating_key]))
db.flush()
def enrich_person_bg(person_id: int) -> None:
"""Fetch + persist one person's TMDB bio OFF the request path — a FastAPI background task that runs
AFTER the response is sent, so the outbound TMDB call never sits on the serve latency (the module's
"no live TMDB call at serve time" invariant). The request's session is already closed by then, so
open a fresh one. Best-effort: a TMDB failure just leaves `tmdb_enriched_at` NULL to retry on the
next open, and the bio lands from the row by then."""
db = SessionLocal()
try:
person = db.get(PlexPerson, person_id)
if person is not None:
plex_tmdb.enrich_person(db, person)
except Exception: # noqa: BLE001 — background best-effort; never surfaces to a user
db.rollback()
log.warning("background person bio enrich failed for id=%s", person_id, exc_info=True)
finally:
db.close()
def set_watch_link(db: Session, user: User, enabled: bool) -> PlexLink:
"""Turn the user's two-way watch sync on or off, creating their link on first use. The link uses
the server admin token (`uses_admin`), so it needs no Plex login of its own."""
link = queries.watch_link(db, user)
if link is None:
link = PlexLink(user_id=user.id, uses_admin=True, linked_at=datetime.now(timezone.utc))
db.add(link)
link.sync_enabled = enabled
db.flush()
return link
# --- a user's playlists ----------------------------------------------------------------------
def create_playlist(db: Session, user: User, title: str, seed: PlexItem | None) -> PlexPlaylist:
"""A new playlist of the user's, holding `seed` when one is given."""
pl = PlexPlaylist(user_id=user.id, title=title)
db.add(pl)
db.flush()
if seed is not None:
db.add(PlexPlaylistItem(playlist_id=pl.id, item_id=seed.id, position=0))
db.flush()
return pl
def rename_playlist(pl: PlexPlaylist, title: str) -> None:
pl.title = title
def delete_playlist(db: Session, pl: PlexPlaylist) -> None:
db.delete(pl)
db.flush()
def _touch(pl: PlexPlaylist) -> None:
pl.updated_at = datetime.now(timezone.utc)
def _last_position(db: Session, playlist_id: int) -> int:
return db.execute(
select(func.coalesce(func.max(PlexPlaylistItem.position), -1)).where(
PlexPlaylistItem.playlist_id == playlist_id
)
).scalar_one()
def add_item(db: Session, pl: PlexPlaylist, it: PlexItem) -> None:
"""Append `it` to the (locked) playlist; an item already in it stays where it is."""
exists = db.execute(
select(PlexPlaylistItem.id).where(
PlexPlaylistItem.playlist_id == pl.id, PlexPlaylistItem.item_id == it.id
)
).first()
if exists is None:
pos = _last_position(db, pl.id) + 1
db.add(PlexPlaylistItem(playlist_id=pl.id, item_id=it.id, position=pos))
_touch(pl)
db.flush()
def remove_item(db: Session, pl: PlexPlaylist, it: PlexItem) -> None:
db.execute(
delete(PlexPlaylistItem).where(
PlexPlaylistItem.playlist_id == pl.id, PlexPlaylistItem.item_id == it.id
)
)
_touch(pl)
db.flush()
def add_items(db: Session, pl: PlexPlaylist, items: list[PlexItem]) -> int:
"""Append many items in their given order (a whole season or show); those already in the list
are skipped. Says how many were added."""
existing = set(
db.execute(
select(PlexPlaylistItem.item_id).where(PlexPlaylistItem.playlist_id == pl.id)
).scalars()
)
pos = _last_position(db, pl.id)
added = 0
for it in items:
if it.id in existing:
continue
pos += 1
db.add(PlexPlaylistItem(playlist_id=pl.id, item_id=it.id, position=pos))
existing.add(it.id)
added += 1
if added:
_touch(pl)
db.flush()
return added
def insert_items_at(db: Session, pl: PlexPlaylist, placed: list[tuple[int, PlexItem]]) -> int:
"""Insert new rows at their indices (ascending, so each index is where the row stood when the
rows before it were back), renumber the whole list 0..n-1, and say how many were added. An item
already in the list does not move: the add stays idempotent."""
rows = list(
db.execute(
select(PlexPlaylistItem)
.where(PlexPlaylistItem.playlist_id == pl.id)
.order_by(PlexPlaylistItem.position, PlexPlaylistItem.id)
).scalars()
)
present = {r.item_id for r in rows}
added = 0
for at, it in sorted(placed, key=lambda p: p[0]):
if it.id in present:
continue
row = PlexPlaylistItem(playlist_id=pl.id, item_id=it.id, position=0)
db.add(row)
rows.insert(min(at, len(rows)), row)
present.add(it.id)
added += 1
for i, r in enumerate(rows):
r.position = i
if added:
_touch(pl)
db.flush()
return added
def remove_items(db: Session, pl: PlexPlaylist, item_ids: list[int]) -> int:
"""Remove these items from the playlist; says how many were there."""
if not item_ids:
return 0
result = db.execute(
delete(PlexPlaylistItem).where(
PlexPlaylistItem.playlist_id == pl.id, PlexPlaylistItem.item_id.in_(item_ids)
),
execution_options={"synchronize_session": False},
)
removed = result.rowcount if isinstance(result, CursorResult) else 0
if removed:
_touch(pl)
db.flush()
return removed
def reorder_playlist(db: Session, pl: PlexPlaylist, rating_keys: list[str]) -> None:
"""Set the play order from an explicit list of item keys (0-based positions)."""
id_by_rk = {
it.rating_key: it.id
for it in db.execute(
select(PlexItem).where(PlexItem.rating_key.in_(rating_keys or [""]))
).scalars()
}
rows = {
r.item_id: r
for r in db.execute(
select(PlexPlaylistItem).where(PlexPlaylistItem.playlist_id == pl.id)
).scalars()
}
pos = 0
seen: set[int] = set()
for rk in rating_keys:
row = rows.get(id_by_rk.get(rk, -1))
if row is not None and row.item_id not in seen:
row.position = pos
pos += 1
seen.add(row.item_id)
# Rows the payload omitted (added in another tab meanwhile) keep their relative order after
# the given ones, so no two rows share a position (the YouTube reorder does the same).
for row in sorted(
(r for iid, r in rows.items() if iid not in seen), key=lambda r: (r.position, r.id)
):
row.position = pos
pos += 1
_touch(pl)
db.flush()
# --- a user's watch state --------------------------------------------------------------------
def mark_finished(db: Session, user_id: int, item_id: int, now: datetime) -> bool:
"""Mark the item watched (position cleared) and say whether THIS call made it so. Two near-end
checkpoints arrive together routinely (the 10 s tick and the debounced `seeked` flush), and a
read-then-write let both see "not watched" and both scrobble. The transition is decided by the
write itself: an insert that did not conflict, or an update conditional on the old status —
under READ COMMITTED the second writer re-reads the committed row and matches nothing."""
values = {
"status": "watched",
"watched_at": now,
"position_seconds": 0,
"progress_updated_at": now,
"synced_to_plex": False,
}
inserted = db.execute(
pg_insert(PlexState)
.values(user_id=user_id, item_id=item_id, **values)
.on_conflict_do_nothing(constraint="uq_plex_user_item")
.returning(PlexState.id)
).first()
if inserted is not None:
return True
mine = owned_by(PlexState.user_id, user_id) & (PlexState.item_id == item_id)
result = db.execute(update(PlexState).where(mine, PlexState.status != "watched").values(values))
if isinstance(result, CursorResult) and result.rowcount:
return True
# Already watched: a re-finish still restamps the row, as it always did — just no scrobble.
db.execute(update(PlexState).where(mine).values(values))
return False
def state_row(
db: Session,
user: User | int,
item_id: int,
st: PlexState | None = None,
*,
synced: bool = False,
) -> PlexState:
"""The user's state row on this item, made if the read before found none. Made by the insert
itself (ON CONFLICT DO NOTHING) and then read: the first checkpoint of an item and the `final`
flush beside it routinely both read "none", and a plain add let the second one fail on
`uq_plex_user_item` — a 500, and the push it was carrying lost. Every first row goes through
here: a checkpoint, a status, a season marked at once, a pull from Plex. `synced` is what a row
made now says about Plex; a row another writer made first keeps its own."""
if st is not None:
return st
db.execute(
pg_insert(PlexState)
.values(
user_id=user if isinstance(user, int) else user.id,
item_id=item_id,
synced_to_plex=synced,
)
.on_conflict_do_nothing(constraint="uq_plex_user_item")
)
made = queries.state(db, user, item_id)
assert made is not None # inserted now, or committed by the request that won
return made
def clear_progress(db: Session, st: PlexState | None, now: datetime) -> None:
"""A checkpoint too early to keep: a state that held nothing else goes, any other forgets its
position."""
if st is not None and st.status == "new":
db.delete(st)
elif st is not None:
st.position_seconds = 0
st.progress_updated_at = now
db.flush()
def checkpoint(
db: Session,
user: User,
item_id: int,
st: PlexState | None,
position: int,
now: datetime,
*,
dirty: bool,
) -> PlexState:
"""Record a resume position. `dirty` marks it not yet mirrored to Plex — set before a push, so
the reconcile re-pushes it if the push fails and the push can settle it once Plex confirms."""
st = state_row(db, user, item_id, st)
st.position_seconds = position
st.progress_updated_at = now
if dirty:
st.synced_to_plex = False
db.flush()
return st
def forget_state(db: Session, st: PlexState) -> None:
db.delete(st)
db.flush()
def set_status(
db: Session, user: User, item_id: int, st: PlexState | None, status: str
) -> PlexState:
"""Set a watched or hidden status; watched also clears the resume point."""
st = state_row(db, user, item_id, st)
st.status = status
if status == "watched":
st.watched_at = datetime.now(timezone.utc)
st.position_seconds = 0
st.synced_to_plex = False
db.flush()
return st
def bulk_state(
db: Session, user_id: int, eps: list[PlexItem], watched: bool
) -> list[tuple[int, str, str]]:
"""Mark every episode in `eps` watched (or unwatched) for a user and say what changed, as the
(item id, rating key, action) pushes a linked Plex account needs. Unwatching keeps the user's
private "hidden" marks."""
now = datetime.now(timezone.utc)
existing = rollup.item_state_map(db, user_id, [e.id for e in eps])
to_push: list[tuple[int, str, str]] = []
for e in eps:
st = existing.get(e.id)
prev = st.status if st is not None else "new"
if watched:
st = state_row(db, user_id, e.id, st)
prev = st.status # a first checkpoint beside this may have made it meanwhile
st.status = "watched"
st.watched_at = now
st.position_seconds = 0
st.synced_to_plex = False
if prev != "watched":
to_push.append((e.id, e.rating_key, "watched"))
else: # → new: drop any state (but keep the user's private "hidden" marks)
if st is not None and prev != "hidden":
db.delete(st)
if prev != "new":
to_push.append((e.id, e.rating_key, "unwatched"))
db.flush()
return to_push
+1 -1
View File
@@ -33,7 +33,7 @@ from sqlalchemy.orm import Session as DbSession
from app import logs, sysconfig
from app.config import settings
from app.downloads.edit import encode_thread_cap
from app.models import PlexItem
from app.plex.models import PlexItem
from app.plex import paths
from app.plex.sync import HDR_TRANSFERS, HW_DECODABLE_CODECS
+1 -1
View File
@@ -47,7 +47,7 @@ from sqlalchemy.orm import Session
from app import logs
from app.db import SessionLocal
from app.models import PlexItem
from app.plex.models import PlexItem
from app.plex import paths as plex_paths
from app.plex.cache import CACHE_DIR
from app.plex.client import PlexClient, PlexError
+1 -1
View File
@@ -21,7 +21,7 @@ from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.models import PlexCollection, PlexItem, PlexLibrary, PlexSeason, PlexShow
from app.plex.models import PlexCollection, PlexItem, PlexLibrary, PlexSeason, PlexShow
from app.plex import detail, overlay, people
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.genres import hu_genres
+4 -4
View File
@@ -22,7 +22,7 @@ from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.config import settings
from app.models import PlexItem, PlexPerson, PlexShow
from app.plex.models import PlexItem, PlexPerson, PlexShow
from app.plex.cards import display_title
from app.plex.people import people_by_credits, warm_portraits
@@ -619,9 +619,9 @@ def enrich_person(db: Session, person: PlexPerson) -> bool:
if not person.tmdb_person_id or not sysconfig.get_str(db, "tmdb_api_key"):
return False
try:
# Called from a background task (off the request path — see people._enrich_person_bg), so it no
# longer blocks a serve; still cap the outbound TMDB call with a short timeout so a hanging TMDB
# doesn't tie up a threadpool worker for the default 30s.
# Called from a background task (off the request path — see `plex.service.enrich_person_bg`),
# so it no longer blocks a serve; still cap the outbound TMDB call with a short timeout so a
# hanging TMDB doesn't tie up a threadpool worker for the default 30s.
with TmdbClient(db, timeout=8.0) as tmdb:
_fetch_person_into(tmdb, person)
except TmdbError as exc:
+14 -22
View File
@@ -29,8 +29,11 @@ from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.db import SessionLocal
from app.models import PlexItem, PlexLibrary, PlexLink, PlexState
from app.userscope import owned_by
from app.plex.models import PlexItem, PlexLibrary, PlexLink, PlexState
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex import queries
from app.plex.service import state_row
from app.plex.sync import (
_TYPE_EPISODE,
_TYPE_MOVIE,
@@ -196,7 +199,7 @@ def link_for_push(db: Session, user_id: int) -> PlexLink | None:
"""The active owner watch-push link for this user, or None when push is off / not the owner.
Push is gated on `initial_import_done` so we never push a half-known local state over Plex's
authoritative one before the one-time "Plex is master" import has reconciled them."""
link = db.execute(select(PlexLink).where(PlexLink.user_id == user_id)).scalars().first()
link = db.execute(select(PlexLink).where(owned_by(PlexLink.user_id, user_id))).scalars().first()
if link and link.uses_admin and link.sync_enabled and link.initial_import_done:
return link
return None
@@ -251,13 +254,7 @@ def push_state_to_plex(
# Flag the row synced — but an "unwatched" via status=new deletes the row, so there may be
# nothing to flag; that's fine (unscrobble still happened). Skip the flag if the row was
# re-edited since we captured `expect_ts` (leave it dirty for its own push / the reconcile).
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.first()
)
st = queries.state(db, user_id, item_id)
if st is not None and _row_matches(st, expect_ts):
st.synced_to_plex = True
db.commit()
@@ -318,7 +315,7 @@ def push_bulk_state_to_plex(user_id: int, changes: list[tuple[int, str, str]]) -
if synced_ids:
db.execute(
update(PlexState)
.where(PlexState.user_id == user_id, PlexState.item_id.in_(synced_ids))
.where(owned_by(PlexState.user_id, user_id), PlexState.item_id.in_(synced_ids))
.values(synced_to_plex=True)
)
db.commit()
@@ -391,13 +388,7 @@ def _pull_apply(
"""Bring one Plex-side positive signal into Siftlode under last-write-wins. Returns
"watched"/"in_progress" if it changed the row, else None. Never clears a row (un-watch is the
full reconcile's job) — this only pulls in watched/resume."""
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.first()
)
st = queries.state(db, user_id, item_id)
if _same_state(st, status, pos):
if st is not None and not st.synced_to_plex:
st.synced_to_plex = True # already matches Plex → just settle the flag, no churn
@@ -418,8 +409,9 @@ def _pull_apply(
# "almost always already pushed" assumption above excludes.
return None
if st is None:
st = PlexState(user_id=user_id, item_id=item_id)
db.add(st)
st = state_row(db, user_id, item_id, synced=True)
if not st.synced_to_plex: # a local change made it first, beside this pull: the guard above
return None
if status == "watched":
st.status, st.position_seconds, st.watched_at, st.progress_updated_at = (
"watched",
@@ -449,7 +441,7 @@ def _repush_dirty(db: Session, plex: PlexClient, link: PlexLink) -> int:
dirty = (
db.execute(
select(PlexState).where(
PlexState.user_id == link.user_id,
owned_by(PlexState.user_id, link.user_id),
PlexState.synced_to_plex.is_(False),
PlexState.status != "hidden",
)
@@ -505,7 +497,7 @@ def _reconcile_inprogress_movies(
and aren't affected). Note: like the rest of this job, `metadata()` reads the TOKEN account (the
owner) — a HTPC playing under a managed Plex user (accountID>1) is out of scope here as elsewhere."""
conds = [
PlexState.user_id == link.user_id,
owned_by(PlexState.user_id, link.user_id),
PlexState.status.notin_(("watched", "hidden")),
PlexState.position_seconds > _PROGRESS_MIN_S,
PlexItem.kind == "movie",
@@ -627,7 +619,7 @@ def _clear_unwatched(db: Session, user_id: int, plex_states: dict, settled: set[
select(PlexState)
.join(PlexItem, PlexItem.id == PlexState.item_id)
.join(PlexLibrary, PlexLibrary.id == PlexItem.library_id)
.where(PlexState.user_id == user_id, PlexLibrary.plex_key.in_(settled))
.where(owned_by(PlexState.user_id, user_id), PlexLibrary.plex_key.in_(settled))
)
for st in rows.scalars().all():
if st.item_id in plex_states or st.status == "hidden" or not st.synced_to_plex:
+13 -4
View File
@@ -16,7 +16,8 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app import sysconfig
from app.userscope import owned_by
from app import logs, sysconfig
from app.db import SessionLocal
from app.models import ApiQuotaUsage, QuotaEvent
@@ -173,7 +174,7 @@ def actions_today(db: Session, user_id: int, action: str) -> int:
select(func.count())
.select_from(QuotaEvent)
.where(
QuotaEvent.user_id == user_id,
owned_by(QuotaEvent.user_id, user_id),
QuotaEvent.action == action,
QuotaEvent.created_at >= pacific_day_start_utc(),
)
@@ -204,6 +205,14 @@ def record_usage(units: int) -> None:
)
with SessionLocal() as qs:
qs.execute(stmt)
# Audit detail: who/what spent it (per-user attribution; NULL actor = system).
qs.add(QuotaEvent(user_id=_actor_id.get(), action=_action.get(), units=units))
# Audit detail: who/what spent it (per-user attribution; NULL actor = system), and the
# background run it belongs to, so the run's row can say what it cost.
qs.add(
QuotaEvent(
user_id=_actor_id.get(),
action=_action.get(),
units=units,
job_run_id=logs.current_run_id(),
)
)
qs.commit()
View File
+215
View File
@@ -0,0 +1,215 @@
"""Reader's tables: what a user does with the shared catalog — where they are in a video
(`VideoState`), their playlists, their saved views, their labels on channels (`Tag`, `ChannelTag`)
and the comfort shelf on Home. Only this unit writes them; another unit reads them along a declared
edge, or asks this unit's service."""
from datetime import datetime
from sqlalchemy import (
Boolean,
CheckConstraint,
DateTime,
Float,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db import Base
from app.models import TimestampMixin, UpdatedAtMixin
class Tag(Base, TimestampMixin):
"""A label. System tags (user_id NULL) are computed automatically and shared by all
users; user tags (user_id set) are personal. category groups them in the UI."""
__tablename__ = "tags"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
)
name: Mapped[str] = mapped_column(String(64))
color: Mapped[str | None] = mapped_column(String(16))
# language | topic | other
category: Mapped[str] = mapped_column(String(16), default="other", server_default="other")
class ChannelTag(Base):
"""Links a tag to a channel. source='auto' rows are managed by the auto-tagger and
regenerated freely; source='user' rows are never touched by the system."""
__tablename__ = "channel_tags"
__table_args__ = (UniqueConstraint("channel_id", "tag_id", "user_id", name="uq_channel_tag"),)
id: Mapped[int] = mapped_column(primary_key=True)
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
tag_id: Mapped[int] = mapped_column(ForeignKey("tags.id", ondelete="CASCADE"), index=True)
# NULL = system/auto link visible to everyone; set = a specific user's link.
user_id: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
source: Mapped[str] = mapped_column(String(8), default="auto", server_default="auto")
confidence: Mapped[float | None] = mapped_column(Float)
class VideoState(Base, UpdatedAtMixin):
"""Per-user watch state for a video. Rows exist only for non-default states; the
absence of a row means 'new/unseen'."""
__tablename__ = "video_states"
__table_args__ = (UniqueConstraint("user_id", "video_id", name="uq_user_video"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
video_id: Mapped[str] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"), index=True)
# new | watched | hidden ("saved" is now Watch-later playlist membership, not a status)
status: Mapped[str] = mapped_column(String(12), default="new", server_default="new")
watched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Resume position (seconds into the video) for the in-app player. Orthogonal to
# status: a partially-watched video keeps status "new" but a non-zero position, so
# it can render a progress bar and match the "in progress" feed filter. 0 = not
# started / finished (trivially-early and near-finished positions are not stored).
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
position_seconds: Mapped[int] = mapped_column(
Integer, default=0, server_default="0", nullable=False
)
progress_updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class Playlist(Base, TimestampMixin, UpdatedAtMixin):
"""A per-user named, ordered collection of videos from the shared catalog.
`kind`: 'user' (normal) or 'watch_later' (built-in, undeletable, one per user).
`source`: 'local' (created here) or 'youtube' (mirrored from the user's YouTube
playlists). `yt_playlist_id` links a mirrored/exported playlist to its YouTube id.
`dirty` marks a YouTube-linked playlist with local edits not yet pushed, so the read
sync won't clobber them. (source/yt_playlist_id/dirty are forward-looking for the YT
sync phases; the foundation phase only uses local playlists.)"""
__tablename__ = "playlists"
__table_args__ = (
# One Watch later per user, held by the database: two first saves at once each made one.
Index(
"uq_playlist_watch_later",
"user_id",
unique=True,
postgresql_where=text("kind = 'watch_later'"),
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
name: Mapped[str] = mapped_column(String(255))
kind: Mapped[str] = mapped_column(String(16), default="user", server_default="user")
source: Mapped[str] = mapped_column(String(16), default="local", server_default="local")
yt_playlist_id: Mapped[str | None] = mapped_column(String(64))
# True when the current items/name differ from `synced_fingerprint` (unpushed edits).
dirty: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# SHA-256 of name + ordered video ids as last synced from / pushed to YouTube; the
# baseline `dirty` is derived against. NULL = no baseline yet (treated as dirty).
synced_fingerprint: Mapped[str | None] = mapped_column(String)
# JSON list of the ordered video ids as of the last sync — the 3-way-merge baseline the
# two-way YouTube sync classifies each item against (local-add vs YT-add vs real delete).
# NULL = no baseline yet; the first merge then does a lossless union (never deletes).
synced_video_ids: Mapped[str | None] = mapped_column(Text)
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
items: Mapped[list["PlaylistItem"]] = relationship(
back_populates="playlist",
cascade="all, delete-orphan",
order_by="PlaylistItem.position",
)
class PlaylistItem(Base):
"""A video's membership in a playlist, with its order within that playlist."""
__tablename__ = "playlist_items"
__table_args__ = (UniqueConstraint("playlist_id", "video_id", name="uq_playlist_video"),)
id: Mapped[int] = mapped_column(primary_key=True)
playlist_id: Mapped[int] = mapped_column(ForeignKey("playlists.id", ondelete="CASCADE"))
video_id: Mapped[str] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"), index=True)
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
playlist: Mapped["Playlist"] = relationship(back_populates="items")
class SavedView(Base, TimestampMixin, UpdatedAtMixin):
"""A per-user named snapshot of a module's filter/sort state (a "smart view").
`source` names the module the view belongs to ("feed" | "plex"); `filters` is that module's
serialized filter JSON the frontend applies with one click. `position` orders them in the sidebar
(drag-reorder, per source); `is_default` marks the one view applied automatically on load — at most
one per (user, source), enforced by the API on set. The name is unique per (user, source)."""
__tablename__ = "saved_views"
__table_args__ = (
UniqueConstraint("user_id", "source", "name", name="uq_user_source_saved_view"),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
source: Mapped[str] = mapped_column(String(16), default="feed", server_default="feed")
name: Mapped[str] = mapped_column(String(80))
filters: Mapped[dict] = mapped_column(JSON)
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
is_default: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class ComfortPin(Base, TimestampMixin):
"""One entry on a user's comfort shelf — the anti-paralysis fallback: "just put on X".
Deliberately NOT a boolean on `VideoState`/`PlexState`, and NOT a polymorphic `(kind, ref)` string
pair. A flag cannot ORDER a shelf and cannot hold a channel or a show — and comfort media is
usually "put on X", a series or a channel, not one particular video. A string ref would order and
hold anything, but it would also lose the foreign key: a Plex item dropped from the library, or an
unsubscribed channel, would leave a dead tile on the one surface whose whole job is to be
reliable. So instead: one nullable CASCADE-ing FK per pinnable kind, and a check constraint that
exactly one of them is set.
That constraint is the house rule applied to a five-way union — the invariant lives in the schema,
not in a comment every insert site has to remember. `kind`/`ref` below derive FROM the columns, so
there is no second vocabulary to keep in sync.
NULLs are DISTINCT in a Postgres unique index, so the five per-kind unique constraints each stop a
double-pin of the same thing without colliding across kinds."""
__tablename__ = "comfort_pins"
__table_args__ = (
CheckConstraint(
"num_nonnulls(video_id, channel_id, plex_item_id, plex_show_id, playlist_id) = 1",
name="ck_comfort_pin_one_ref",
),
UniqueConstraint("user_id", "video_id", name="uq_comfort_pin_video"),
UniqueConstraint("user_id", "channel_id", name="uq_comfort_pin_channel"),
UniqueConstraint("user_id", "plex_item_id", name="uq_comfort_pin_plex_item"),
UniqueConstraint("user_id", "plex_show_id", name="uq_comfort_pin_plex_show"),
UniqueConstraint("user_id", "playlist_id", name="uq_comfort_pin_playlist"),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
video_id: Mapped[str | None] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"))
channel_id: Mapped[str | None] = mapped_column(ForeignKey("channels.id", ondelete="CASCADE"))
plex_item_id: Mapped[int | None] = mapped_column(
ForeignKey("plex_items.id", ondelete="CASCADE")
)
plex_show_id: Mapped[int | None] = mapped_column(
ForeignKey("plex_shows.id", ondelete="CASCADE")
)
playlist_id: Mapped[int | None] = mapped_column(ForeignKey("playlists.id", ondelete="CASCADE"))
# 0-based shelf order. The shelf is short and hand-curated, so a dense integer the reorder
# endpoint rewrites wholesale beats a fractional index nobody would ever exercise here.
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0", nullable=False)
+361
View File
@@ -0,0 +1,361 @@
"""Reader's reads: a user's labels, saved views, watch state and playlists, for the HTTP layer and
for the other units that ask about them.
Every read of a user's row goes through `userscope`, so "someone else's row" is the same answer as
a missing one. Reader reads Catalog's videos along its declared edge (a playlist's cover and
length, whether a video exists); another unit asks the questions at the end of this module rather
than joining Reader's tables itself.
"""
from sqlalchemy import ColumnElement, and_, func, or_, select
from sqlalchemy.orm import InstrumentedAttribute, Session, aliased
from app.collation import collated
from app.feedquery import feed_columns, serialize
from app.models import User
from app.catalog.models import Channel, Subscription, Video
from app.reader.models import ChannelTag, Playlist, PlaylistItem, SavedView, Tag, VideoState
from app.userscope import owned_by, owned_row
# --- catalog, read along the edge ------------------------------------------------------------
def video(db: Session, video_id: str) -> Video | None:
return db.get(Video, video_id)
# --- tags ------------------------------------------------------------------------------------
def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int]]:
"""The tags this user sees (the shared system ones and their own), each with how many channel
links it has, by category then name."""
return [
(tag, count)
for tag, count in db.execute(
select(Tag, func.count(ChannelTag.id))
.outerjoin(ChannelTag, ChannelTag.tag_id == Tag.id)
.where(or_(Tag.user_id.is_(None), owned_by(Tag.user_id, user)))
.group_by(Tag.id)
.order_by(Tag.category, Tag.name)
).tuples()
]
def own_tag(db: Session, user: User, tag_id: int) -> Tag | None:
"""One of the user's own tags. A system tag (no owner) is nobody's to edit, so it reads as
missing too."""
return owned_row(db, Tag, tag_id, user)
# --- saved views -----------------------------------------------------------------------------
def views(db: Session, user: User, source: str) -> list[SavedView]:
return list(
db.execute(
select(SavedView)
.where(owned_by(SavedView.user_id, user), SavedView.source == source)
.order_by(SavedView.position, SavedView.id)
).scalars()
)
def all_views(db: Session, user: User) -> list[SavedView]:
return list(db.execute(select(SavedView).where(owned_by(SavedView.user_id, user))).scalars())
def own_view(db: Session, user: User, view_id: int) -> SavedView | None:
return owned_row(db, SavedView, view_id, user)
# --- watch state -----------------------------------------------------------------------------
def video_state(db: Session, user: User, video_id: str) -> VideoState | None:
return db.execute(
select(VideoState).where(
owned_by(VideoState.user_id, user), VideoState.video_id == video_id
)
).scalar_one_or_none()
# --- playlists -------------------------------------------------------------------------------
def own_playlist(
db: Session, user: User, playlist_id: int, *, lock: bool = False
) -> Playlist | None:
return owned_row(db, Playlist, playlist_id, user, lock=lock)
def playlists_for(db: Session, user: User) -> list[Playlist]:
"""The user's playlists in their rail order."""
return list(
db.execute(
select(Playlist)
.where(owned_by(Playlist.user_id, user))
.order_by(Playlist.position, Playlist.created_at)
).scalars()
)
def watch_later(db: Session, user: User) -> Playlist | None:
return db.scalar(
select(Playlist).where(owned_by(Playlist.user_id, user), Playlist.kind == "watch_later")
)
def item_counts(db: Session, ids: list[int]) -> dict[int, int]:
return dict(
db.execute(
select(PlaylistItem.playlist_id, func.count())
.where(PlaylistItem.playlist_id.in_(ids))
.group_by(PlaylistItem.playlist_id)
)
.tuples()
.all()
)
def durations(db: Session, ids: list[int]) -> dict[int, int]:
"""Each playlist's running time: the sum of its videos' lengths."""
return {
pid: int(total or 0)
for pid, total in db.execute(
select(PlaylistItem.playlist_id, func.coalesce(func.sum(Video.duration_seconds), 0))
.join(Video, Video.id == PlaylistItem.video_id)
.where(PlaylistItem.playlist_id.in_(ids))
.group_by(PlaylistItem.playlist_id)
).tuples()
}
def containing(db: Session, ids: list[int], video_id: str) -> set[int]:
"""Which of these playlists already hold the video (the add-to-playlist popover's ticks)."""
return set(
db.execute(
select(PlaylistItem.playlist_id).where(
PlaylistItem.playlist_id.in_(ids), PlaylistItem.video_id == video_id
)
).scalars()
)
def covers(db: Session, ids: list[int]) -> dict[int, str | None]:
"""Each playlist's cover: its first item's thumbnail, for every list in ONE window query rather
than a lookup per playlist."""
rn = (
func.row_number()
.over(partition_by=PlaylistItem.playlist_id, order_by=PlaylistItem.position)
.label("rn")
)
first_items = (
select(PlaylistItem.playlist_id.label("pid"), Video.thumbnail_url.label("thumb"), rn)
.join(Video, Video.id == PlaylistItem.video_id)
.where(PlaylistItem.playlist_id.in_(ids))
.subquery()
)
return dict(
db.execute(select(first_items.c.pid, first_items.c.thumb).where(first_items.c.rn == 1))
.tuples()
.all()
)
def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None]:
"""One playlist's item count, running time and cover. An outer join, so an empty list still
answers (0, 0, None)."""
count, total = db.execute(
select(func.count(PlaylistItem.id), func.coalesce(func.sum(Video.duration_seconds), 0))
.select_from(PlaylistItem)
.outerjoin(Video, Video.id == PlaylistItem.video_id)
.where(PlaylistItem.playlist_id == playlist_id)
).one()
cover = db.scalar(
select(Video.thumbnail_url)
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.where(PlaylistItem.playlist_id == playlist_id)
.order_by(PlaylistItem.position)
.limit(1)
)
return count or 0, total or 0, cover
def playlist_rows(db: Session, user: User, playlist_id: int) -> list:
"""A playlist's videos as feed rows (the same columns the feed serializes), in list order."""
state = aliased(VideoState)
return list(
db.execute(
select(*feed_columns(user, state))
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(state, and_(state.video_id == Video.id, owned_by(state.user_id, user)))
.where(PlaylistItem.playlist_id == playlist_id)
# `id` breaks a position tie the same way on every read (the renumbering add does too).
.order_by(PlaylistItem.position, PlaylistItem.id)
).all()
)
def items(db: Session, playlist_id: int) -> list[PlaylistItem]:
"""A playlist's item rows, in list order."""
return list(
db.execute(
select(PlaylistItem)
.where(PlaylistItem.playlist_id == playlist_id)
.order_by(PlaylistItem.position, PlaylistItem.id)
).scalars()
)
def item(db: Session, playlist_id: int, video_id: str) -> PlaylistItem | None:
return db.scalar(
select(PlaylistItem).where(
PlaylistItem.playlist_id == playlist_id, PlaylistItem.video_id == video_id
)
)
# --- for the other units ---------------------------------------------------------------------
def kept_by_a_reader(video_id: InstrumentedAttribute[str]) -> ColumnElement[bool]:
"""A video someone holds: watched, in progress or hidden (a `VideoState`), or in one of their
playlists. The catalog's cleanup jobs keep such a video; as a clause, so their DELETE stays
one statement."""
return or_(
select(VideoState.id).where(VideoState.video_id == video_id).exists(),
select(PlaylistItem.id).where(PlaylistItem.video_id == video_id).exists(),
)
def playlisted_by(db: Session, video_ids: list[str]) -> list[tuple[int, str]]:
"""(user id, video id) for every user who has one of these videos in a playlist."""
return list(
db.execute(
select(Playlist.user_id, PlaylistItem.video_id)
.join(PlaylistItem, PlaylistItem.playlist_id == Playlist.id)
.where(PlaylistItem.video_id.in_(video_ids))
).tuples()
)
def has_auto_tags(channel_id: InstrumentedAttribute[str]) -> ColumnElement[bool]:
"""A channel the auto-tagger has already labelled, as a clause for its "only the missing" scan."""
return (
select(ChannelTag.id)
.where(
ChannelTag.channel_id == channel_id,
ChannelTag.source == "auto",
ChannelTag.user_id.is_(None),
)
.exists()
)
def view_tag_ids(filters: object) -> list[int]:
"""The tag ids a stored view's filters name; anything else in that slot names none."""
tags = filters.get("tags") if isinstance(filters, dict) else None
if not isinstance(tags, list):
return []
return [t for t in tags if type(t) is int] # `type`, not isinstance: True is not a tag id
def tags_in_feed_views(db: Session) -> set[int]:
"""Every tag id any user's saved feed view filters on. Read defensively (`view_tag_ids`): a view
stored before its route checked the shape must not raise for everyone's job."""
return {
t
for (filters,) in db.execute(select(SavedView.filters).where(SavedView.source == "feed"))
for t in view_tag_ids(filters)
}
def held_by(db: Session, user: User, video_ids: list[str]) -> set[str]:
"""Which of these videos the user already holds: watched, in progress or hidden, or in one of
their playlists (a live search sinks them below the new finds)."""
if not video_ids:
return set()
has_state = (
select(VideoState.id)
.where(VideoState.video_id == Video.id, owned_by(VideoState.user_id, user))
.exists()
)
in_playlist = (
select(PlaylistItem.id)
.join(Playlist, Playlist.id == PlaylistItem.playlist_id)
.where(PlaylistItem.video_id == Video.id, owned_by(Playlist.user_id, user))
.exists()
)
return set(
db.scalars(select(Video.id).where(Video.id.in_(video_ids), or_(has_state, in_playlist)))
)
def feed_rows_by_ids(db: Session, user: User, video_ids: list[str]) -> list[dict]:
"""These videos serialized with the feed's projection (their channel, this user's watch state,
saved), in the order given."""
if not video_ids:
return []
state = aliased(VideoState)
rows = db.execute(
select(*feed_columns(user, state))
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(state, and_(state.video_id == Video.id, owned_by(state.user_id, user)))
.where(Video.id.in_(video_ids))
).all()
by_id = {r.id: serialize(r) for r in rows}
return [by_id[i] for i in video_ids if i in by_id]
def playlist_channels(db: Session, user: User, exclude: str | None, lang: str) -> list:
"""The channels of the videos in the user's playlists that they don't subscribe to (`exclude`:
their own channel), each with how many of those videos it has and across how many playlists;
most present first, then by title in the reader's alphabet."""
subscribed = select(Subscription.channel_id).where(owned_by(Subscription.user_id, user))
q = (
select(
Channel,
func.count(func.distinct(PlaylistItem.video_id)),
func.count(func.distinct(Playlist.id)),
)
.select_from(PlaylistItem)
.join(Playlist, Playlist.id == PlaylistItem.playlist_id)
.join(Video, Video.id == PlaylistItem.video_id)
.join(Channel, Channel.id == Video.channel_id)
.where(owned_by(Playlist.user_id, user), Channel.id.not_in(subscribed))
)
if exclude:
q = q.where(Channel.id != exclude)
return list(
db.execute(
q.group_by(Channel.id).order_by(
func.count(func.distinct(PlaylistItem.video_id)).desc(),
collated(Channel.title, lang),
)
).all()
)
def user_tag_links(db: Session, user: User, channel_id: str | None = None) -> dict[str, list[int]]:
"""The user's own tag links by channel (one channel's, when given)."""
stmt = select(ChannelTag.channel_id, ChannelTag.tag_id).where(
owned_by(ChannelTag.user_id, user)
)
if channel_id is not None:
stmt = stmt.where(ChannelTag.channel_id == channel_id)
out: dict[str, list[int]] = {}
for cid, tag_id in db.execute(stmt).tuples():
out.setdefault(cid, []).append(tag_id)
return out
def user_tag_link(db: Session, user: User, channel_id: str, tag_id: int) -> ChannelTag | None:
return db.execute(
select(ChannelTag).where(
ChannelTag.channel_id == channel_id,
ChannelTag.tag_id == tag_id,
owned_by(ChannelTag.user_id, user),
)
).scalar_one_or_none()
+497
View File
@@ -0,0 +1,497 @@
"""Reader's writes: a user's labels and saved views, where they are in a video, their playlists'
items and order — and the system tags the catalog's auto-tagger hands over.
Every function here flushes and leaves the commit to its caller (the handler, or the job that owns
its session). A write that must not double up keeps its guard where it is written: an upsert on the
unique key, a row lock taken before a presence check. The comfort shelf's writes are Home's
(`discover.pins`); a playlist's YouTube sync is `sync.playlists`.
"""
from datetime import datetime, timezone
from sqlalchemy import exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.models import User
from app.reader import queries
from app.reader.models import ChannelTag, Playlist, PlaylistItem, SavedView, Tag, VideoState
from app.sync.playlists import lock_playlists, recompute_dirty
from app.userscope import owned_by
TAG_CATEGORIES = ("language", "topic", "other")
VIEW_NAME_MAX = 80
# --- tags ------------------------------------------------------------------------------------
def create_tag(db: Session, user: User, name: str, category: str, color: str | None) -> Tag:
"""A tag of the user's own. An unknown category falls back to "other" rather than refusing. A
name the user already has raises `IntegrityError` at the flush."""
tag = Tag(
user_id=user.id,
name=name,
color=color,
category=category if category in TAG_CATEGORIES else "other",
)
db.add(tag)
db.flush()
return tag
def update_tag(tag: Tag, *, sent: set[str], name: str | None, color: str | None) -> None:
"""Change the fields the caller sent: `color` sent as None clears it."""
if "name" in sent and name is not None:
tag.name = name
if "color" in sent:
tag.color = color
def delete_tag(db: Session, tag: Tag) -> None:
db.delete(tag) # its channel links cascade
db.flush()
def ensure_system_tag(db: Session, name: str, category: str, color: str | None) -> Tag:
"""The shared system tag of this name, made the first time a channel earns it."""
tag = db.execute(
select(Tag).where(Tag.user_id.is_(None), Tag.name == name)
).scalar_one_or_none()
if tag is None:
tag = Tag(user_id=None, name=name, category=category, color=color)
db.add(tag)
db.flush()
return tag
def set_auto_tags(db: Session, channel_id: str, desired: dict[int, float | None]) -> None:
"""Make a channel's automatic tag links exactly `desired` (tag id → confidence). Only the
auto-tagger's own links move; a user's links on the channel are never touched."""
existing = {
ct.tag_id: ct
for ct in db.execute(
select(ChannelTag).where(
ChannelTag.channel_id == channel_id,
ChannelTag.user_id.is_(None),
ChannelTag.source == "auto",
)
).scalars()
}
for tag_id, ct in existing.items():
if tag_id not in desired:
db.delete(ct)
for tag_id, conf in desired.items():
ct = existing.get(tag_id)
if ct is None:
db.add(
ChannelTag(
channel_id=channel_id,
tag_id=tag_id,
user_id=None,
source="auto",
confidence=conf,
)
)
else:
ct.confidence = conf
db.flush()
def prune_orphan_system_tags(db: Session) -> int:
"""Delete the system tags no channel links to any more (after a mapping change) — but not one a
saved feed view filters on. Views hold tag IDS, and a deleted tag comes back under a new id when
a channel earns it again, so the view's filter would point at nothing; the feed drops an unknown
id, and the view showed the whole feed under its old name. The kept ids are filtered here rather
than in SQL: a `NOT IN` over every view's ids could pass the driver's bind-parameter limit."""
in_views = queries.tags_in_feed_views(db)
orphans = [
tag
for tag in db.execute(
select(Tag).where(Tag.user_id.is_(None), ~exists().where(ChannelTag.tag_id == Tag.id))
).scalars()
if tag.id not in in_views
]
for tag in orphans:
db.delete(tag)
db.flush()
return len(orphans)
# --- saved views -----------------------------------------------------------------------------
def create_view(db: Session, user: User, source: str, name: str, filters: dict) -> SavedView:
"""A view appended after the user's others of the same module (each module numbers its own
0..N). A name the user already has there raises `IntegrityError` at the flush."""
maxpos = db.execute(
select(func.coalesce(func.max(SavedView.position), -1)).where(
owned_by(SavedView.user_id, user), SavedView.source == source
)
).scalar_one()
view = SavedView(
user_id=user.id,
source=source,
name=name[:VIEW_NAME_MAX],
filters=filters,
position=maxpos + 1,
)
db.add(view)
db.flush()
return view
def update_view(
db: Session,
view: SavedView,
*,
sent: set[str],
name: str | None = None,
filters: dict | None = None,
is_default: bool | None = None,
) -> None:
"""Change the fields the caller sent. Making a view the default clears the flag on the user's
other views of the same module first: at most one default per (user, source)."""
if "name" in sent and name is not None:
view.name = name[:VIEW_NAME_MAX]
if "filters" in sent and filters is not None:
view.filters = filters
if "is_default" in sent:
make_default = bool(is_default)
if make_default:
db.execute(
update(SavedView)
.where(
owned_by(SavedView.user_id, view.user_id),
SavedView.source == view.source,
SavedView.id != view.id,
)
.values(is_default=False)
)
view.is_default = make_default
db.flush()
def delete_view(db: Session, view: SavedView) -> None:
db.delete(view)
db.flush()
def reorder_views(db: Session, user: User, ids: list[int]) -> None:
"""Number the listed views 0..n in the listed order; ids that are not the user's are skipped."""
by_id = {v.id: v for v in queries.all_views(db, user)}
pos = 0
for vid in ids:
v = by_id.get(vid)
if v is not None:
v.position = pos
pos += 1
db.flush()
# --- watch state -----------------------------------------------------------------------------
def unmark(db: Session, user: User, video_id: str) -> None:
"""Back to "new". A row that still holds a resume point keeps it (un-marking "watched" restores
the in-progress state, not where the user left off); otherwise the row goes. The player fires
this beside a checkpoint that may delete the same row: the caller's commit then raises
`StaleDataError`, which is that lost race and nothing worse."""
row = queries.video_state(db, user, video_id)
if row is None:
return
if row.position_seconds:
row.status = "new"
row.watched_at = None
else:
db.delete(row)
db.flush()
def mark(db: Session, user: User, video_id: str, status: str) -> None:
""" "watched" or "hidden", as one atomic upsert on (user, video). The player marks "watched" at
the end of playback at the instant a checkpoint may delete the in-progress row; a SELECT then
UPDATE raised `StaleDataError` on that race, and an INSERT … ON CONFLICT is immune to it.
"hidden" keeps any earlier `watched_at`."""
now = datetime.now(timezone.utc)
set_: dict = {"status": status, "updated_at": now}
if status == "watched":
set_["watched_at"] = now
db.execute(
pg_insert(VideoState)
.values(
user_id=user.id,
video_id=video_id,
status=status,
watched_at=now if status == "watched" else None,
)
.on_conflict_do_update(constraint="uq_user_video", set_=set_)
)
def clear_state(db: Session, user: User, video_id: str) -> None:
"""The video back to pristine for this user: status, resume point and `watched_at` all go.
Playlist membership lives elsewhere and stays."""
row = queries.video_state(db, user, video_id)
if row is not None:
db.delete(row)
db.flush()
def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
"""The player's resume point. 0 clears it: a row holding nothing else goes, one with a status
(watched, hidden) keeps it with the position zeroed. The status itself is never changed here.
A position is one upsert on (user, video): the first checkpoint and the end-of-playback mark
beside it both read "no row", and a plain add let the second one fail on `uq_user_video`."""
now = datetime.now(timezone.utc)
if position:
db.execute(
pg_insert(VideoState)
.values(
user_id=user.id,
video_id=video_id,
position_seconds=position,
progress_updated_at=now,
)
.on_conflict_do_update(
constraint="uq_user_video",
set_={"position_seconds": position, "progress_updated_at": now, "updated_at": now},
)
)
return
row = queries.video_state(db, user, video_id)
if row is None:
return
if row.status == "new":
db.delete(row)
else:
row.position_seconds = 0
row.progress_updated_at = now
db.flush()
# --- playlists -------------------------------------------------------------------------------
class MoveRefused(Exception):
"""A move between playlists that cannot happen: `reason` is the code the UI translates."""
def __init__(self, reason: str):
super().__init__(reason)
self.reason = reason
def touch(pl: Playlist) -> None:
"""Bump the playlist's `updated_at` for an edit that only writes `playlist_items`.
`UpdatedAtMixin`'s `onupdate` fires on an UPDATE of the PLAYLIST row, and add / remove /
reorder / move write the item rows instead — so without this the rail's "recently changed"
sort would only ever see renames. The assignment is what makes the column mean "last time
this list changed", which is what the rail promises the reader.
The DATABASE's clock, like the mixin's own `onupdate`: one sort key written from two clocks
(this process's and Postgres's) could order a rename and an item edit the wrong way round."""
pl.updated_at = func.now()
def _next_playlist_position(db: Session, user_id: int) -> int:
"""Append position for a new playlist in this user's rail."""
return (
db.execute(
select(func.coalesce(func.max(Playlist.position), -1)).where(
owned_by(Playlist.user_id, user_id)
)
).scalar_one()
+ 1
)
def next_item_position(db: Session, playlist_id: int) -> int:
"""Append position for a new item at the end of a playlist. Locks the playlist row first: two
adds (two tabs, the rail and the page) otherwise both read the same max and land on one
position, and the list's order between them is then arbitrary."""
lock_playlists(db, playlist_id)
return (
db.execute(
select(func.coalesce(func.max(PlaylistItem.position), -1)).where(
PlaylistItem.playlist_id == playlist_id
)
).scalar_one()
+ 1
)
def create_playlist(db: Session, user: User, name: str) -> Playlist:
pl = Playlist(user_id=user.id, name=name, position=_next_playlist_position(db, user.id))
db.add(pl)
db.flush()
return pl
def watch_later(db: Session, user: User) -> Playlist:
"""The user's built-in Watch later playlist, created on first use. Created by the insert itself
(ON CONFLICT DO NOTHING on the one-per-user index) and then read: the first two saves at once
both read "none", and each used to make a list of its own."""
pl = queries.watch_later(db, user)
if pl is None:
db.execute(
pg_insert(Playlist)
.values(
user_id=user.id,
name="Watch later",
kind="watch_later",
position=_next_playlist_position(db, user.id),
)
.on_conflict_do_nothing(
index_elements=["user_id"], index_where=text("kind = 'watch_later'")
)
)
pl = queries.watch_later(db, user)
assert pl is not None # inserted now, or committed by the request that won
return pl
def add_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) -> bool:
"""Append a video to a playlist if not already present (idempotent). Returns whether it
was added. `mark_dirty` recomputes the YouTube-sync dirty flag (skip for Watch later).
Locked BEFORE the presence check: two adds of one video both saw "absent" and the second one's
insert hit uq_playlist_video as a 500."""
lock_playlists(db, pl.id)
if queries.item(db, pl.id, video_id) is not None:
return False
db.add(
PlaylistItem(playlist_id=pl.id, video_id=video_id, position=next_item_position(db, pl.id))
)
if mark_dirty:
recompute_dirty(db, pl)
touch(pl)
db.flush()
return True
def remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) -> bool:
"""Remove a video from a playlist if present (idempotent). Returns whether it was removed.
`mark_dirty` recomputes the YouTube-sync dirty flag (skip for Watch later)."""
lock_playlists(db, pl.id)
found = queries.item(db, pl.id, video_id)
if found is None:
return False
db.delete(found)
if mark_dirty:
recompute_dirty(db, pl)
touch(pl)
db.flush()
return True
def place_item(db: Session, pl: Playlist, video_id: str, position: int) -> None:
"""Put a video already in the list at `position` (clamped to the end), renumbering the whole
list: positions can have gaps after removals, so "index N" is only meaningful over the
ordered rows."""
rows = queries.items(db, pl.id)
new = next(r for r in rows if r.video_id == video_id)
rows.remove(new)
rows.insert(min(position, len(rows)), new)
for pos, row in enumerate(rows):
row.position = pos
db.flush()
def add_to_playlist(db: Session, pl: Playlist, video_id: str, position: int | None) -> None:
"""Add a video to one of the user's lists, at `position` when given (the list page's "Undo" of
a removal puts the row back where it was in ONE request), else at the end. A video already in
the list stays where it is."""
added = add_item(db, pl, video_id, mark_dirty=False)
if added and position is not None:
place_item(db, pl, video_id, position)
recompute_dirty(db, pl)
db.flush()
def rename_playlist(db: Session, pl: Playlist, name: str) -> None:
"""A new name moves `dirty`; the caller holds the playlist's lock (the scheduled mirror refresh
reads `dirty` under it)."""
if name != pl.name:
pl.name = name
recompute_dirty(db, pl)
db.flush()
def delete_playlist(db: Session, pl: Playlist) -> None:
"""The playlist first, as every item write: the cascade deletes the items after the list row
is locked."""
lock_playlists(db, pl.id)
db.delete(pl) # items cascade
db.flush()
def move_item(db: Session, src: Playlist, dst: Playlist, video_id: str) -> None:
"""Move one video from `src` into `dst`, in the caller's one transaction.
The client-side shape — add to the target, then remove from the source — leaves the video in
BOTH lists whenever the second call fails. Idempotent: a video already in the target is simply
dropped from the source (moving into a list that has it is a removal, not a duplicate). Raises
`MoveRefused` for the same list twice, or a video a concurrent edit already took out."""
if src.id == dst.id:
raise MoveRefused("same_playlist")
# Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise.
lock_playlists(db, src.id, dst.id)
row = queries.item(db, src.id, video_id)
if row is None:
raise MoveRefused("not_in_playlist")
dst_changed = False
if queries.item(db, dst.id, video_id) is None:
# Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
# survives the move, and there is no window where the video belongs to neither list. The
# check above holds: both lists are locked, and an insert into the target (another tab's
# add, a YouTube merge) waits for the lock — its foreign key needs the target row.
row.playlist_id = dst.id
row.position = next_item_position(db, dst.id)
dst_changed = True
if not dst_changed:
db.delete(row)
# The source always changed; the target only when the row landed in it. Touching a target
# that already had the video promoted an untouched list in the rail's "recently changed" sort.
recompute_dirty(db, src)
touch(src)
if dst_changed:
recompute_dirty(db, dst)
touch(dst)
db.flush()
def reorder_items(db: Session, pl: Playlist, order: list[str]) -> int:
"""Number the listed videos 0..n in the listed order. Items the list left out (added in another
tab meanwhile) keep their relative order after them, so no two items share a position. The
caller holds the playlist's lock. Returns how many items the list has."""
by_video = {it.video_id: it for it in queries.items(db, pl.id)}
pos = 0
seen: set[str] = set()
for vid in order:
it = by_video.get(vid)
if it is not None:
it.position = pos
pos += 1
seen.add(vid)
for it in sorted(
(it for vid, it in by_video.items() if vid not in seen), key=lambda i: i.position
):
it.position = pos
pos += 1
recompute_dirty(db, pl)
touch(pl)
db.flush()
return pos
def attach_user_tag(db: Session, user: User, channel_id: str, tag_id: int) -> None:
"""The user's own label on a channel (idempotent). The caller has checked the tag is theirs."""
if queries.user_tag_link(db, user, channel_id, tag_id) is None:
db.add(ChannelTag(channel_id=channel_id, tag_id=tag_id, user_id=user.id, source="user"))
db.flush()
def detach_user_tag(db: Session, user: User, channel_id: str, tag_id: int) -> None:
row = queries.user_tag_link(db, user, channel_id, tag_id)
if row is not None:
db.delete(row)
db.flush()
+17
View File
@@ -0,0 +1,17 @@
"""The account filter the activity page's lists share (runs, admin actions, server log, quota): one
filter bar drives them all, so the value it sends means the same on every tab.
The value is an account id, or `none` — the rows no account made (the background) — a value to filter
on, not the absence of a filter.
"""
from sqlalchemy import ColumnElement
NONE = "none"
# The `user` query parameter's shape on every route that takes it.
PATTERN = r"^(none|[0-9]{1,10})$"
def condition(column, value: str) -> ColumnElement[bool]:
"""`column` (an account id column) narrowed to `value`."""
return column.is_(None) if value == NONE else column == int(value)
+6 -4
View File
@@ -16,6 +16,7 @@ from app.db import Base
from app.models import POSITION_MAX_S, User
from app.issues import gitea
from app.refusal import refused
from app.userscope import owned_row
from app.youtube.client import YouTubeError, refusal_reason
M = TypeVar("M", bound=Base)
@@ -88,10 +89,11 @@ def owned_or_404(
) -> M:
"""Fetch a row by primary key that `user` owns, or raise 404 (`lock`: see `get_or_404`). A row owned by someone
else 404s (NOT 403) on purpose — a 403 would confirm the id exists to a stranger.
It replaced the near-identical per-route ownership helpers, and is called directly — a thin
`_own_x` wrapper around it is the ad-hoc shape CLAUDE.md's patterns table rules out."""
obj = get_or_404(db, model, id_, what=what, lock=lock)
if getattr(obj, owner_attr) != user.id:
A core route calls it directly. A unit's route cannot (it may not name a unit's model as a value):
it asks its unit's `queries.own_x`, which goes through the same `owned_row`, and answers
`not_found` itself."""
obj = owned_row(db, model, id_, user, owner_attr=owner_attr, lock=lock)
if obj is None:
raise not_found(what)
return obj
+66
View File
@@ -0,0 +1,66 @@
"""Keyset paging over `(timestamp desc, id desc)`, shared by the admin history lists (server log,
audit log, background runs).
A timestamp alone is not unique (a browser log row's `ts` is clamped into its consent window; a run
and its siblings can start in the same microsecond), and an offset would skip or repeat rows while
writers add new ones at the top — so a page continues from the last row's `(timestamp, id)`.
"""
import base64
import binascii
import json
from datetime import datetime, timezone
from fastapi import HTTPException
from sqlalchemy import and_, or_, select
from sqlalchemy.orm import Session
from app.models import User
PAGE_MAX = 200
def utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the columns are `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def encode_cursor(ts: datetime, row_id: int) -> str:
return base64.urlsafe_b64encode(json.dumps([ts.isoformat(), row_id]).encode()).decode()
def decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
at = datetime.fromisoformat(ts)
return (at if at.tzinfo is not None else at.replace(tzinfo=timezone.utc)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def older_than(ts_col, id_col, cursor: str):
"""The rows after `cursor` in `(ts desc, id desc)` order. The `ts <=` conjunct is implied by the
OR; it is there so the planner can bound the timestamp index scan instead of filtering every
newer row first."""
ts, row_id = decode_cursor(cursor)
return and_(ts_col <= ts, or_(ts_col < ts, and_(ts_col == ts, id_col < row_id)))
def page(rows: list, limit: int, ts_of) -> tuple[list, str | None]:
"""`rows` was read with `limit + 1` (to know whether there is a next page without a count query):
the page itself, and the cursor that continues after it (None on the last page)."""
more = len(rows) > limit
rows = rows[:limit]
return rows, encode_cursor(ts_of(rows[-1]), rows[-1].id) if more else None
def accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the row
was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
+32 -71
View File
@@ -8,33 +8,29 @@ What a row holds was redacted when it was written (`logs.record_fields`, and the
browser's rows), so this serves the columns as they are; a value the redactor took out reads as
`[redacted]` / `[email]`, which is the point — the console shows that something was there.
Paging is a keyset over `(ts desc, id desc)`: a browser row's `ts` is clamped into its consent
window, so `ts` alone is not unique, and an offset would skip or repeat rows while the writer adds
new ones at the top. The live tail re-reads the first page rather than asking for "ids after N":
Paging is a keyset over `(ts desc, id desc)` (`_keyset`). The live tail re-reads the first page rather than asking for "ids after N":
several processes write this table, and a batch's ids are taken when it inserts, not when it
commits, so a slower batch can land below an id the tail already passed — and would never show.
"""
import base64
import binascii
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import Text, and_, cast, func, or_, select
from fastapi import APIRouter, Depends, Query
from sqlalchemy import Text, cast, func, or_, select
from sqlalchemy.orm import Session
from app import logsink
from app.auth import admin_user
from app.db import get_db
from app.models import LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.utils import like_escape
router = APIRouter(prefix="/api/admin/logs", tags=["admin"])
PAGE_MAX = 200
# A facet list is a chip row, not a report: past this many distinct events the rest are reachable
# through the free-text search, and the response stays small whatever a self-host logs.
FACET_EVENTS_MAX = 100
@@ -42,38 +38,9 @@ FACET_EVENTS_MAX = 100
Level = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
# The filters a facet can leave out of its own count (`LogFilters.conditions`).
Facet = Literal["levels", "sources", "user", "event"]
# `none` selects the rows that have no account / no event code (a scheduler line, an anonymous
# request) — a real value to filter on, not the absence of a filter.
NONE = "none"
def _utc(value: datetime | None) -> datetime | None:
"""A bound without a zone is UTC — the column is `timestamptz`, and a naive value would be read
in the database session's zone instead."""
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def _encode_cursor(row: LogEvent) -> str:
return base64.urlsafe_b64encode(json.dumps([row.ts.isoformat(), row.id]).encode()).decode()
def _decode_cursor(cursor: str) -> tuple[datetime, int]:
try:
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
return _utc(datetime.fromisoformat(ts)), int(row_id)
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
def _accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
"""The accounts a page names, by id → address. An id with no account left (erased after the
line was written) is simply absent, and the client shows the bare id."""
if not user_ids:
return {}
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
return {str(uid): email for uid, email in rows}
# `none` selects the rows that have no event code, as the account filter's `none` (app/routes/_account)
# selects the rows of no account — a real value to filter on, not the absence of a filter.
NONE = account.NONE
@dataclass(frozen=True)
@@ -86,6 +53,7 @@ class LogFilters:
event: str | None
request_id: str | None
job_id: int | None
run_id: int | None
q: str | None
since: datetime | None
until: datetime | None
@@ -100,11 +68,7 @@ class LogFilters:
if self.sources and skip != "sources":
out.append(LogEvent.source.in_(self.sources))
if self.user is not None and skip != "user":
out.append(
LogEvent.user_id.is_(None)
if self.user == NONE
else LogEvent.user_id == int(self.user)
)
out.append(account.condition(LogEvent.user_id, self.user))
if self.event is not None and skip != "event":
out.append(
LogEvent.event.is_(None) if self.event == NONE else LogEvent.event == self.event
@@ -113,10 +77,12 @@ class LogFilters:
out.append(LogEvent.request_id == self.request_id)
if self.job_id is not None:
out.append(LogEvent.job_id == self.job_id)
if self.run_id is not None:
out.append(LogEvent.run_id == self.run_id)
if self.since is not None:
out.append(LogEvent.ts >= _utc(self.since))
out.append(LogEvent.ts >= utc(self.since))
if self.until is not None:
out.append(LogEvent.ts < _utc(self.until))
out.append(LogEvent.ts < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
@@ -134,17 +100,20 @@ class LogFilters:
def log_filters(
levels: list[Level] = Query(default=[]),
sources: list[str] = Query(default=[], max_length=8),
user: str | None = Query(default=None, pattern=r"^(none|[0-9]{1,10})$"),
user: str | None = Query(default=None, pattern=account.PATTERN),
event: str | None = Query(
default=None, max_length=64, pattern=r"^(none|[a-z][a-z0-9_]*(\.[a-z0-9_]+)*)$"
),
request_id: str | None = Query(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$"),
job_id: int | None = Query(default=None, ge=1),
run_id: int | None = Query(default=None, ge=1),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> LogFilters:
return LogFilters(list(levels), list(sources), user, event, request_id, job_id, q, since, until)
return LogFilters(
list(levels), list(sources), user, event, request_id, job_id, run_id, q, since, until
)
@router.get("")
@@ -159,28 +128,20 @@ def list_logs(
continues from a previous page's `next_cursor`."""
query = select(LogEvent).where(*filters.conditions())
if before is not None:
ts, row_id = _decode_cursor(before)
# The `ts <=` conjunct is implied by the OR; it is there so the planner can bound the
# `ts` index scan instead of filtering every newer row first.
query = query.where(
and_(
LogEvent.ts <= ts,
or_(LogEvent.ts < ts, and_(LogEvent.ts == ts, LogEvent.id < row_id)),
)
)
# One more than asked, to know whether there is a next page without a second count query.
rows = (
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
query = query.where(older_than(LogEvent.ts, LogEvent.id, before))
rows, cursor = page(
list(
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
.scalars()
.all()
),
limit,
lambda r: r.ts,
)
more = len(rows) > limit
rows = rows[:limit]
return {
"items": [logsink.serialize(r) for r in rows],
"accounts": _accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": _encode_cursor(rows[-1]) if more else None,
"accounts": accounts(db, {r.user_id for r in rows if r.user_id is not None}),
"next_cursor": cursor,
}
@@ -211,6 +172,6 @@ def log_facets(
"sources": counts(LogEvent.source, "sources"),
"events": counts(LogEvent.event, "event", FACET_EVENTS_MAX),
"users": [{**u, "value": str(u["value"])} for u in users],
"accounts": _accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"accounts": accounts(db, {u["value"] for u in users if u["value"] != NONE}),
"total": db.scalar(total) or 0,
}
+106 -29
View File
@@ -1,67 +1,144 @@
"""Admin audit log API: read the append-only who/what/when trail, and clear it.
Admin-only. The client renders these rows in a DataTable (client-side sort/filter/paginate), so
this returns the most recent window of rows (capped) plus the true total, rather than paging
server-side — the log stays small in practice (state-changing runs + admin actions only)."""
Admin-only. The activity page reads it a page at a time, newest first, filtered on the server like
the server log (`_keyset` paging), so the whole trail is reachable however long it grows."""
from fastapi import APIRouter, Depends
from sqlalchemy import delete, func, select
from dataclasses import dataclass
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy import Text, cast, delete, func, or_, select
from sqlalchemy.orm import Session
from app import audit
from app.audit import AuditAction
from app.auth import admin_user
from app.db import get_db
from app.models import AuditLog, User
from app.utils import iso
from app.models import AuditLog, LogEvent, User
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.utils import iso, like_escape
router = APIRouter(prefix="/api/admin/audit", tags=["admin"])
# Cap the client-side window. The audit log is small by design; if it ever outgrows this we'd
# switch the DataTable to server-side paging.
MAX_ROWS = 2000
def _serialize(row: AuditLog, actors: dict[int, str]) -> dict:
def _serialize(row: AuditLog, lines: dict[str, int]) -> dict:
return {
"id": row.id,
"created_at": iso(row.created_at),
# actor_id NULL = the scheduler/background did it (client shows "System"); a set id that
# can't be resolved would mean a deleted user, but SET NULL makes that NULL — so present.
"actor": actors.get(row.actor_id) if row.actor_id is not None else None,
# NULL = the background did it (the client shows "System"); the address is in `accounts`.
"actor_id": row.actor_id,
"action": row.action,
"target_type": row.target_type,
"target_id": row.target_id,
"summary": row.summary,
"before": row.before,
"after": row.after,
"request_id": row.request_id,
# How many of that request's log lines are still kept: they go after days, the row after
# months, and a link to none would open an empty list.
"log_lines": lines.get(row.request_id, 0) if row.request_id else 0,
}
@dataclass(frozen=True)
class AuditFilters:
action: str | None
actor: str | None
q: str | None
since: datetime | None
until: datetime | None
def conditions(self, skip: Literal["action"] | None = None) -> list:
"""The WHERE conditions, ANDed. `skip="action"` leaves the action filter out, so the action
menu counts every action under the other filters (the way back from a pick)."""
out = []
if self.action is not None and skip != "action":
out.append(AuditLog.action == self.action)
if self.actor is not None:
out.append(account.condition(AuditLog.actor_id, self.actor))
if self.since is not None:
out.append(AuditLog.created_at >= utc(self.since))
if self.until is not None:
out.append(AuditLog.created_at < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
or_(
AuditLog.summary.ilike(like),
AuditLog.target_id.ilike(like),
AuditLog.action.ilike(like),
AuditLog.request_id.ilike(like),
cast(AuditLog.before, Text).ilike(like),
cast(AuditLog.after, Text).ilike(like),
)
)
return out
def audit_filters(
action: str | None = Query(default=None, max_length=48, pattern=r"^[a-z][a-z0-9_]*$"),
user: str | None = Query(default=None, pattern=account.PATTERN),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> AuditFilters:
return AuditFilters(action, user, q, since, until)
@router.get("")
def list_audit(
limit: int = 500,
filters: AuditFilters = Depends(audit_filters),
before: str | None = Query(default=None, max_length=200),
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
limit = max(1, min(limit, MAX_ROWS))
total = db.scalar(select(func.count()).select_from(AuditLog)) or 0
rows = (
db.execute(select(AuditLog).order_by(AuditLog.created_at.desc()).limit(limit))
.scalars()
.all()
"""One page of rows, newest first, matching every filter given; `before` continues from a
previous page's `next_cursor`."""
query = select(AuditLog).where(*filters.conditions())
if before is not None:
query = query.where(older_than(AuditLog.created_at, AuditLog.id, before))
order = (AuditLog.created_at.desc(), AuditLog.id.desc())
rows, cursor = page(
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
limit,
lambda r: r.created_at,
)
actor_ids = {r.actor_id for r in rows if r.actor_id is not None}
actors: dict[int, str] = {}
if actor_ids:
actors = dict(db.execute(select(User.id, User.email).where(User.id.in_(actor_ids))).all())
requests = {r.request_id for r in rows if r.request_id}
lines: dict[str, int] = {}
if requests:
counted = db.execute(
select(LogEvent.request_id, func.count())
.where(LogEvent.request_id.in_(requests))
.group_by(LogEvent.request_id)
)
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
return {
"items": [_serialize(r, actors) for r in rows],
"total": total,
"returned": len(rows),
"items": [_serialize(r, lines) for r in rows],
"accounts": accounts(db, {r.actor_id for r in rows if r.actor_id is not None}),
"next_cursor": cursor,
}
@router.get("/facets")
def audit_facets(
filters: AuditFilters = Depends(audit_filters),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""The actions present under the other filters, with counts — what the action menu offers."""
n = func.count().label("n")
stmt = (
select(AuditLog.action, n)
.where(*filters.conditions("action"))
.group_by(AuditLog.action)
.order_by(n.desc(), AuditLog.action)
)
return {"actions": [{"value": a, "count": c} for a, c in db.execute(stmt).all()]}
@router.delete("")
def clear_audit(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
"""Clear the audit log. Leaves ONE row recording who cleared it + how many (tamper-evidence)."""
+2 -1
View File
@@ -16,6 +16,7 @@ from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.userscope import owned_by
from app import apitokens
from app import email as email_mod
from app import logs
@@ -1124,7 +1125,7 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
# Burn any other outstanding reset tokens for this user.
for row in db.execute(
select(AuthToken).where(
AuthToken.user_id == user.id,
owned_by(AuthToken.user_id, user),
AuthToken.kind.in_(_PASSWORD_TOKEN_KINDS),
AuthToken.used_at.is_(None),
)
+86 -270
View File
@@ -3,31 +3,23 @@ hidden, tags) and per-channel sync state. Local reads/writes plus YouTube-side w
(subscribe / unsubscribe) that require the optional write scope."""
from datetime import datetime, timezone
from functools import partial
from typing import TYPE_CHECKING
from fastapi import APIRouter, BackgroundTasks, Depends
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import Select, func, select, update
from sqlalchemy.orm import Session
from app.collation import collated, sorted_words
from app.collation import sorted_words
from app.mailtext import lang_for_user
from app import logs, quota
from app import logs, quota, scheduler
from app.auth import admin_user, current_user, require_write_scope
from app.db import get_db
from app.models import (
LIVE_OR_UPCOMING,
BlockedChannel,
Channel,
ChannelTag,
ExploredChannel,
Playlist,
PlaylistItem,
Subscription,
Tag,
User,
Video,
)
from app.routes._common import get_or_404, not_found, owned_or_404, youtube_refusal
from app.models import User
from app.catalog import queries, service
from app.reader import queries as reader_queries
from app.reader import service as reader_service
from app.routes._common import not_found, youtube_refusal
from app.sync.explore import explore_ingest_page
from app.sync.runner import backfill_channel_now, run_recent_backfill
from app.sync.subscriptions import apply_channel_details, details_stale
@@ -35,35 +27,17 @@ from app.utils import iso
from app.youtube.client import YouTubeClient, YouTubeError
from app.refusal import refused
if TYPE_CHECKING:
from app.catalog.models import Channel, Subscription
router = APIRouter(prefix="/api/channels", tags=["channels"])
log = logs.get_logger(__name__)
def channel_aggregates(channel_ids: list[str]) -> Select:
"""Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
alone; a test holds that (test_channel_list_index_only)."""
return (
select(
Video.channel_id,
func.count(Video.id),
func.max(Video.published_at),
func.coalesce(func.sum(Video.duration_seconds), 0),
)
.where(Video.channel_id.in_(channel_ids))
.group_by(Video.channel_id)
)
@router.get("")
def list_channels(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
rows = db.execute(
select(Subscription, Channel)
.join(Channel, Channel.id == Subscription.channel_id)
.where(Subscription.user_id == user.id)
.order_by(Subscription.priority.desc(), collated(Channel.title, lang_for_user(user)))
).all()
rows = queries.subscriptions(db, user, lang_for_user(user))
# Per-channel aggregates over the shared catalog (channel_aggregates), computed on read rather than
# denormalised: an Index Only Scan over a ~20 MB index, where it used to read every video's heap page
@@ -72,36 +46,21 @@ def list_channels(user: User = Depends(current_user), db: Session = Depends(get_
# computed here — the manager table dropped its N/S/L column.)
channel_ids = [c.id for _, c in rows]
agg: dict[str, dict] = {}
if channel_ids:
for cid, n, last_up, total_dur in db.execute(channel_aggregates(channel_ids)).all():
agg[cid] = {
"stored": n,
"last_video_at": iso(last_up),
"total_duration_seconds": int(total_dur or 0),
}
for cid, n, last_up, total_dur in queries.aggregates(db, channel_ids):
agg[cid] = {
"stored": n,
"last_video_at": iso(last_up),
"total_duration_seconds": int(total_dur or 0),
}
# Channels already in the *global* deep-backfill queue — i.e. at least one user (any
# user) has requested full history and it isn't done yet. Their whole back-catalogue is
# coming for everyone (video data is shared), so the UI shouldn't offer "get full
# history" to a user who simply hasn't personally opted in.
deep_queued: set[str] = set()
if channel_ids:
for (cid,) in db.execute(
select(Subscription.channel_id)
.where(
Subscription.channel_id.in_(channel_ids),
Subscription.deep_requested.is_(True),
)
.distinct()
).all():
deep_queued.add(cid)
deep_queued = queries.deep_queued(db, channel_ids)
# The user's personal tag links, grouped by channel.
tags_by_channel: dict[str, list[int]] = {}
for cid, tag_id in db.execute(
select(ChannelTag.channel_id, ChannelTag.tag_id).where(ChannelTag.user_id == user.id)
).all():
tags_by_channel.setdefault(cid, []).append(tag_id)
tags_by_channel = reader_queries.user_tag_links(db, user)
return [
{
@@ -130,27 +89,7 @@ DISCOVERY_ENRICH_CAP = 200
def _discovery_rows(db: Session, user: User) -> list:
"""The user's playlist channels they don't subscribe to (and that aren't their own),
each with how many playlist videos come from it and across how many playlists."""
subscribed = select(Subscription.channel_id).where(Subscription.user_id == user.id)
q = (
select(
Channel,
func.count(func.distinct(PlaylistItem.video_id)),
func.count(func.distinct(Playlist.id)),
)
.select_from(PlaylistItem)
.join(Playlist, Playlist.id == PlaylistItem.playlist_id)
.join(Video, Video.id == PlaylistItem.video_id)
.join(Channel, Channel.id == Video.channel_id)
.where(Playlist.user_id == user.id, Channel.id.not_in(subscribed))
)
if user.yt_channel_id:
q = q.where(Channel.id != user.yt_channel_id)
return db.execute(
q.group_by(Channel.id).order_by(
func.count(func.distinct(PlaylistItem.video_id)).desc(),
collated(Channel.title, lang_for_user(user)),
)
).all()
return reader_queries.playlist_channels(db, user, user.yt_channel_id, lang_for_user(user))
@router.get("/discovery")
@@ -172,7 +111,7 @@ def discover_channels(
):
own = yt.get_my_channel_id()
if own:
user.yt_channel_id = own
service.remember_own_channel(user, own)
db.commit()
except YouTubeError as exc:
# No traceback anywhere in this file's YouTube handlers: for an upstream answer
@@ -228,7 +167,7 @@ def discover_channels(
]
def _channel_summary(ch: Channel) -> dict:
def _channel_summary(ch: "Channel") -> dict:
"""The channel fields common to the list and discovery projections."""
return {
"id": ch.id,
@@ -242,7 +181,7 @@ def _channel_summary(ch: Channel) -> dict:
def _channel_detail_dict(
channel: Channel,
channel: "Channel",
*,
subscribed: bool,
explored: bool,
@@ -311,7 +250,7 @@ def channel_detail(
needed (a stub) and refreshes it once it is stale (`DETAILS_MAX_AGE`), which is how a channel nobody subscribes to (an explored one, outside the
subscription import) gets a banner that still resolves. The enrich uses the API key (1 unit,
cached after)."""
channel = get_or_404(db, Channel, channel_id, what="channel")
channel = _channel_or_404(db, channel_id)
# No `published_at is None` sentinel: a terminated channel never gets one, so it re-fetched on
# every view; rows enriched before that field existed are long past DETAILS_MAX_AGE anyway.
stale = details_stale(channel, datetime.now(timezone.utc))
@@ -323,7 +262,7 @@ def channel_detail(
):
apply_channel_details(db, yt.get_channels([channel_id]), [channel_id])
db.commit()
channel = db.get(Channel, channel_id)
channel = _channel_or_404(db, channel_id)
except YouTubeError as exc:
log.warning(
"channel detail enrich failed: %s",
@@ -333,41 +272,15 @@ def channel_detail(
),
)
db.rollback()
sub = db.execute(
select(Subscription).where(
Subscription.user_id == user.id, Subscription.channel_id == channel_id
)
).scalar_one_or_none()
sub = queries.subscription(db, user, channel_id)
subscribed = sub is not None
explored = (
db.execute(
select(ExploredChannel.id).where(
ExploredChannel.user_id == user.id, ExploredChannel.channel_id == channel_id
)
).first()
is not None
)
blocked = _is_blocked(db, user, channel_id)
explored = queries.exploration(db, user, channel_id) is not None
blocked = queries.is_blocked(db, user, channel_id)
# Stored total + the Normal/Shorts/Live split in one grouped pass (same filters as the manager
# list), so the channel page can show the per-type breakdown that moved off the table.
n_all, n_short, n_live = db.execute(
select(
func.count(Video.id),
func.count(Video.id).filter(Video.is_short.is_(True)),
func.count(Video.id).filter(Video.live_status.in_(LIVE_OR_UPCOMING)),
).where(Video.channel_id == channel_id)
).one()
stored = int(n_all or 0)
count_short = int(n_short or 0)
count_live = int(n_live or 0)
stored, count_short, count_live = queries.video_counts(db, channel_id)
count_normal = max(0, stored - count_short - count_live)
tag_ids = list(
db.scalars(
select(ChannelTag.tag_id).where(
ChannelTag.user_id == user.id, ChannelTag.channel_id == channel_id
)
).all()
)
tag_ids = reader_queries.user_tag_links(db, user, channel_id).get(channel_id, [])
return _channel_detail_dict(
channel,
subscribed=subscribed,
@@ -397,28 +310,15 @@ def explore_channel(
it, then ingests one page of its uploads (newest-first) marked via_explore + enriched.
Returns the next page token so the page can "load more". Guarded against the shared
backfill quota reserve."""
channel = get_or_404(db, Channel, channel_id, what="channel")
if _is_blocked(db, user, channel_id):
channel = _channel_or_404(db, channel_id)
if queries.is_blocked(db, user, channel_id):
raise refused(403, "channel_blocked")
if quota.backfill_reserve_reached():
raise refused(429, "quota_reserve")
# Record/refresh the exploration (DB only) before ingesting, so the explorer can see the
# videos as soon as they land. Flag the channel ephemeral unless it's organically followed.
existing = db.execute(
select(ExploredChannel).where(
ExploredChannel.user_id == user.id, ExploredChannel.channel_id == channel_id
)
).scalar_one_or_none()
if existing is None:
db.add(ExploredChannel(user_id=user.id, channel_id=channel_id))
else:
existing.created_at = datetime.now(timezone.utc) # refresh the grace clock
followed = db.execute(
select(Subscription.id).where(Subscription.channel_id == channel_id).limit(1)
).first()
if followed is None and not channel.from_explore:
channel.from_explore = True
service.explore(db, user, channel)
db.commit()
try:
@@ -432,7 +332,7 @@ def explore_channel(
if details_stale(channel, datetime.now(timezone.utc)):
apply_channel_details(db, yt.get_channels([channel_id]), [channel_id])
db.commit()
channel = db.get(Channel, channel_id)
channel = _channel_or_404(db, channel_id)
result = explore_ingest_page(db, yt, channel, page_token)
except YouTubeError as exc:
raise youtube_refusal(exc) from exc
@@ -444,30 +344,20 @@ def explore_channel(
}
def _user_subscription(db: Session, user: User, channel_id: str) -> Subscription:
sub = db.execute(
select(Subscription).where(
Subscription.user_id == user.id, Subscription.channel_id == channel_id
)
).scalar_one_or_none()
def _channel_or_404(db: Session, channel_id: str) -> "Channel":
channel = queries.channel(db, channel_id)
if channel is None:
raise not_found("channel")
return channel
def _user_subscription(db: Session, user: User, channel_id: str) -> "Subscription":
sub = queries.subscription(db, user, channel_id)
if sub is None:
raise not_found("subscription")
return sub
def _is_blocked(db: Session, user: User, channel_id: str) -> bool:
"""Whether this user has blocked this channel."""
return (
db.execute(
select(BlockedChannel.id).where(
BlockedChannel.user_id == user.id,
BlockedChannel.channel_id == channel_id,
)
).first()
is not None
)
class ChannelUpdateIn(BaseModel):
# PATCH semantics: a field left unset (None) is untouched. Every field is optional so the
# UI can send just the one control it changed. int/bool coercion + rejection of garbage
@@ -481,24 +371,18 @@ class ChannelUpdateIn(BaseModel):
def update_channel(
channel_id: str,
body: ChannelUpdateIn,
background: BackgroundTasks,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
sub = _user_subscription(db, user, channel_id)
channel = db.get(Channel, channel_id)
deep_turned_on = False
if body.priority is not None:
sub.priority = body.priority
if body.hidden is not None:
sub.hidden = body.hidden
if body.deep_requested is not None:
# Opt this channel into (or out of) full-history backfill. The deep scheduler
# picks the flag up on its next run; turning it off won't undo already-fetched
# videos, it just stops further deep paging if nobody else still wants it.
want = body.deep_requested
deep_turned_on = want and not sub.deep_requested
sub.deep_requested = want
channel = queries.channel(db, channel_id)
deep_turned_on = service.update_subscription(
db,
sub,
priority=body.priority,
hidden=body.hidden,
deep_requested=body.deep_requested,
)
db.commit()
# Opting a channel into full history should give an immediate feed: if we haven't even
@@ -508,12 +392,19 @@ def update_channel(
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_BACKFILL_RECENT):
run_recent_backfill(db, [channel], max_channels=1)
# …and kick the FULL deep backfill for this one channel immediately in the background, so the
# whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks. The
# request returns right away; the channel page's live poll shows the count climb. The scheduler
# still finishes a very large channel's tail on its normal runs (deep_requested stays set).
if deep_turned_on:
background.add_task(backfill_channel_now, channel_id, user.id)
# …and kick the FULL deep backfill for this one channel immediately, as a recorded run of its own,
# so the whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks.
# The request returns right away; the channel page's live poll shows the count climb. The
# scheduler still finishes a very large channel's tail on its normal runs (deep_requested stays set).
# A channel whose history is all in gets no run: it would only say "nothing to do", one kept row
# per flip of the flag.
if deep_turned_on and not (channel is not None and channel.backfill_done):
scheduler.run_ad_hoc(
scheduler.CHANNEL_BACKFILL,
partial(backfill_channel_now, channel_id, user.id),
actor_id=user.id,
target=channel_id,
)
return {
"ok": True,
@@ -535,19 +426,8 @@ def reset_backfill(
state (a "reset" trigger). Clears the channel's backfill markers, opts it back into deep
backfill, and re-runs its recent pull immediately; the deep scheduler re-pages the full
back-catalog on its next run. Idempotent — videos upsert by id, so nothing duplicates."""
channel = get_or_404(db, Channel, channel_id, what="channel")
channel.backfill_done = False
channel.backfill_cursor = None
channel.recent_synced_at = None
# Re-arm the demand-driven deep scheduler for this channel. run_deep_backfill gates on an
# EXISTS over subscriptions with deep_requested=True, so flip it on EVERY subscriber rather
# than requiring the admin to be personally subscribed (they reset from the manager, not
# their own feed). No-op row count if the channel has no subscribers.
db.execute(
update(Subscription)
.where(Subscription.channel_id == channel_id)
.values(deep_requested=True)
)
channel = _channel_or_404(db, channel_id)
service.reset_backfill(db, channel)
db.commit()
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_BACKFILL_RECENT):
run_recent_backfill(db, [channel], max_channels=1)
@@ -565,13 +445,8 @@ def subscribe(
local subscription with YouTube's returned resource id and enriches the channel's metadata
if it's still a stub, so the new row renders properly in the manager. It does NOT pull the
channel's videos — the scheduler picks those up on its next run, like any subscribed channel."""
channel = get_or_404(db, Channel, channel_id, what="channel")
existing = db.execute(
select(Subscription).where(
Subscription.user_id == user.id, Subscription.channel_id == channel_id
)
).scalar_one_or_none()
if existing is not None:
channel = _channel_or_404(db, channel_id)
if queries.subscription(db, user, channel_id) is not None:
raise refused(409, "already_subscribed")
with (
quota.attribute(user.id, quota.QuotaAction.CHANNELS_SUBSCRIBE),
@@ -611,24 +486,7 @@ def subscribe(
reason=exc.reason,
),
)
db.add(
Subscription(
user_id=user.id,
channel_id=channel_id,
yt_subscription_id=yt_sub_id or None,
subscribed_at=datetime.now(timezone.utc),
)
)
# Subscribing is the "keep" signal: a previously-explored channel becomes a normal organic
# one. Drop its ephemeral flags so the cleanup job leaves it alone and its videos stop being
# per-user private (they join the shared Library like any subscribed channel's uploads).
if channel.from_explore:
channel.from_explore = False
db.execute(
update(Video)
.where(Video.channel_id == channel_id, Video.via_explore.is_(True))
.values(via_explore=False)
)
service.record_subscription(db, user, channel, yt_sub_id)
db.commit()
return {"ok": True, "subscribed": channel_id}
@@ -655,7 +513,7 @@ def unsubscribe(
# 422 (a real, explainable error → speaking modal), not 502 which the client treats as
# a transient "connection lost" gateway blip.
raise youtube_refusal(exc) from exc
db.delete(sub)
service.drop_subscription(db, sub)
db.commit()
return {"ok": True, "unsubscribed": channel_id}
@@ -663,16 +521,7 @@ def unsubscribe(
@router.get("/blocked/list")
def list_blocked(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
"""The user's blocked channels (videos from these are excluded from their search/feed/explore)."""
rows = (
db.execute(
select(Channel)
.join(BlockedChannel, BlockedChannel.channel_id == Channel.id)
.where(BlockedChannel.user_id == user.id)
.order_by(collated(Channel.title, lang_for_user(user)))
)
.scalars()
.all()
)
rows = queries.blocked_channels(db, user, lang_for_user(user))
return [
{"id": c.id, "title": c.title, "handle": c.handle, "thumbnail_url": c.thumbnail_url}
for c in rows
@@ -688,16 +537,9 @@ def block_channel(
"""Block a channel for this user: its videos won't appear in their live search (and won't be
ingested for them) and are hidden from their feed / explore. Idempotent. The now-hidden
un-kept search/explore videos are reclaimed by the discovery-cleanup job in due course."""
get_or_404(db, Channel, channel_id, what="channel")
if not _is_blocked(db, user, channel_id):
db.add(BlockedChannel(user_id=user.id, channel_id=channel_id))
# Stop any active exploration of it so it can be cleaned up.
db.execute(
ExploredChannel.__table__.delete().where(
(ExploredChannel.user_id == user.id) & (ExploredChannel.channel_id == channel_id)
)
)
db.commit()
_channel_or_404(db, channel_id)
service.block(db, user, channel_id)
db.commit()
return {"ok": True, "blocked": channel_id}
@@ -707,11 +549,7 @@ def unblock_channel(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
db.execute(
BlockedChannel.__table__.delete().where(
(BlockedChannel.user_id == user.id) & (BlockedChannel.channel_id == channel_id)
)
)
service.unblock(db, user, channel_id)
db.commit()
return {"ok": True, "unblocked": channel_id}
@@ -728,26 +566,12 @@ def attach_tag(
db: Session = Depends(get_db),
) -> dict:
_user_subscription(db, user, channel_id)
# Only the user's own tags may be attached as personal links (owned_or_404 → 404 if the tag
# is missing OR belongs to someone else, without leaking which).
owned_or_404(db, Tag, body.tag_id, user, what="tag")
exists = db.execute(
select(ChannelTag).where(
ChannelTag.channel_id == channel_id,
ChannelTag.tag_id == body.tag_id,
ChannelTag.user_id == user.id,
)
).scalar_one_or_none()
if exists is None:
db.add(
ChannelTag(
channel_id=channel_id,
tag_id=body.tag_id,
user_id=user.id,
source="user",
)
)
db.commit()
# Only the user's own tags may be attached as personal links (a 404 if the tag is missing OR
# belongs to someone else, without leaking which).
if reader_queries.own_tag(db, user, body.tag_id) is None:
raise not_found("tag")
reader_service.attach_user_tag(db, user, channel_id, body.tag_id)
db.commit()
return {"ok": True, "channel_id": channel_id, "tag_id": body.tag_id}
@@ -758,14 +582,6 @@ def detach_tag(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
row = db.execute(
select(ChannelTag).where(
ChannelTag.channel_id == channel_id,
ChannelTag.tag_id == tag_id,
ChannelTag.user_id == user.id,
)
).scalar_one_or_none()
if row is not None:
db.delete(row)
db.commit()
reader_service.detach_user_tag(db, user, channel_id, tag_id)
db.commit()
return {"ok": True, "channel_id": channel_id, "tag_id": tag_id}
+5 -3
View File
@@ -61,7 +61,7 @@ def set_config(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
s = sysconfig.spec(key)
s = sysconfig.config_page_spec(key)
if s is None:
raise not_found("config_key")
if "value" not in body.model_fields_set:
@@ -89,7 +89,9 @@ def set_config(
before=None if s.secret else {"value": _redact(old)},
after=None if s.secret else {"value": _redact(new)},
)
db.commit()
# The override is committed even when the trail skips it: re-saving the value in force still
# pins it as an override, and an uncommitted write would be dropped with the session.
db.commit()
_apply_runtime(db, key)
return sysconfig.describe(db)
@@ -100,7 +102,7 @@ def reset_config(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
s = sysconfig.spec(key)
s = sysconfig.config_page_spec(key)
if s is None:
raise not_found("config_key")
old = None if s.secret else sysconfig.get(db, key)
+3 -1
View File
@@ -18,6 +18,7 @@ from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.userscope import owned_by
from app import audit, debugsession, logs, logsink
from app.audit import AuditAction
from app.auth import current_user
@@ -95,7 +96,7 @@ def stop_session(
def list_sessions(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
sessions = db.scalars(
select(DebugSession)
.where(DebugSession.user_id == user.id)
.where(owned_by(DebugSession.user_id, user))
.order_by(DebugSession.started_at.desc())
.limit(LIST_MAX)
).all()
@@ -219,6 +220,7 @@ def send_issue(sid: str, user: User = Depends(current_user), db: Session = Depen
raise refused(409, "issue_sending")
new = session.issue_url is None
if not new and not debugsession.unfinished(session):
db.commit() # nothing to send: the lock ends here, not when the request's session closes
return {"ok": True, "session": _view(db, session)}
key = str(user.id)
if new:
+9 -19
View File
@@ -13,7 +13,6 @@ from typing import Literal
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -22,9 +21,9 @@ from app.db import get_db
from app.discover import hero as hero_mod
from app.discover import pins, rails
from app.discover.budget import Budget, DEFAULT_BUDGET
from app.models import ComfortPin, User
from app.models import User
from app.plex.cards import norm_lang
from app.routes._common import owned_or_404, not_found
from app.routes._common import not_found
router = APIRouter(prefix="/api/discover", tags=["discover"])
@@ -140,30 +139,19 @@ def add_pin(
raise not_found("item")
column, value = resolved
existing = db.execute(
select(ComfortPin).where(
ComfortPin.user_id == user.id,
getattr(ComfortPin, column) == value,
)
).scalar_one_or_none()
existing = pins.pin_of(db, user, column, value)
if existing is not None:
return {"ok": True, "id": existing.id, "created": False}
pin = ComfortPin(user_id=user.id, position=pins.next_position(db, user), **{column: value})
db.add(pin)
try:
pin = pins.add(db, user, column, value)
db.commit()
except IntegrityError:
# A concurrent pin of the same thing from a second tab. The unique constraint did its job;
# the user's intent ("be on my shelf") is satisfied either way, so report the winner rather
# than a 500 the user can do nothing with.
db.rollback()
winner = db.execute(
select(ComfortPin).where(
ComfortPin.user_id == user.id,
getattr(ComfortPin, column) == value,
)
).scalar_one_or_none()
winner = pins.pin_of(db, user, column, value)
if winner is None:
# Not the unique constraint, then: the only other way in is the target row being pruned
# between `resolve_ref` confirming it and this INSERT (a Plex sync dropping a removed
@@ -180,8 +168,10 @@ def remove_pin(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pin = owned_or_404(db, ComfortPin, pin_id, user, what="pin")
db.delete(pin)
pin = pins.own_pin(db, user, pin_id)
if pin is None:
raise not_found("pin")
pins.remove(db, pin)
db.commit()
return {"deleted": pin_id}
+106 -337
View File
@@ -3,23 +3,22 @@
All routes are behind `current_user`; admin routes add `admin_user`. Live job state is polled by the
frontend (the worker is a separate process); this module never downloads — it enqueues, manages job
state, and shares finished files. The library itself — listing, renaming, moving, deleting and
serving a done file, and uploading one — is `routes.files`.
serving a done file, and uploading one — is `routes.files`. Reads come from `downloads.queries`,
writes go through Capture's service (`service`, `links`, `quota`); this module validates, answers in
HTTP and commits.
"""
import re
from typing import Literal
from typing import TYPE_CHECKING, Literal
from datetime import datetime, timedelta, timezone
from urllib.parse import parse_qs, urlparse
from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import FileResponse
from pydantic import BaseModel, Field
from sqlalchemy import delete, func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.mailtext import lang_for_user
from app.collation import collated
from app import audit, sysconfig
from app.auth import admin_user, current_user
from app.config import settings
@@ -29,45 +28,32 @@ from app.downloads import credentials as dlcreds
from app.downloads import edit as editmod
from app.downloads import folders
from app.downloads import links as linksmod
from app.downloads import quota, service, storage
from app.downloads import queries, quota, service, storage
from app.downloads import urlguard
from app.downloads.folders import FolderKind
from app.downloads.status import FINISHED, NOT_REFETCHABLE, JobStatus
from app.models import (
DownloadJob,
DownloadLink,
FileFolder,
DownloadProfile,
DownloadQuota,
DownloadShare,
LibraryState,
MediaAsset,
User,
display_text,
)
from app.downloads.status import NOT_REFETCHABLE, JobStatus
from app.models import User, display_text
from app.routes._common import (
ProgressIn,
get_or_404,
kept_position,
not_found,
owned_or_404,
)
from app.routes.files import (
MAX_LIST,
accessible_job,
assets_for,
own_folder,
own_job,
own_job_for_update,
positions_for,
quota_detail,
serialize,
serialize_job,
source_keys,
)
from app.security import hash_password
from app.userscope import is_messageable_user, messageable_clauses
from app.utils import iso
from app.userscope import emails_of, is_messageable_user, messageable_others, user_by_email
from app.refusal import refused
if TYPE_CHECKING:
from app.downloads.models import DownloadJob, DownloadLink, DownloadProfile, FileFolder
router = APIRouter(prefix="/api/downloads", tags=["downloads"])
admin_router = APIRouter(prefix="/api/admin/downloads", tags=["admin-downloads"])
@@ -119,7 +105,7 @@ def resolve_source(raw: str) -> tuple[str, str]:
# --- profiles ------------------------------------------------------------------------------
def _serialize_profile(p: DownloadProfile) -> dict:
def _serialize_profile(p: "DownloadProfile") -> dict:
return {
"id": p.id,
"name": p.name,
@@ -131,18 +117,7 @@ def _serialize_profile(p: DownloadProfile) -> dict:
@router.get("/profiles")
def list_profiles(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
rows = (
db.execute(
select(DownloadProfile)
.where((DownloadProfile.is_builtin.is_(True)) | (DownloadProfile.user_id == user.id))
.order_by(
DownloadProfile.is_builtin.desc(), DownloadProfile.sort_order, DownloadProfile.id
)
)
.scalars()
.all()
)
return [_serialize_profile(p) for p in rows]
return [_serialize_profile(p) for p in queries.profiles_for(db, user)]
class CreateProfileIn(BaseModel):
@@ -158,15 +133,7 @@ def create_profile(
spec = body.spec
if not name:
raise refused(400, "profile_name_required")
maxpos = db.execute(
select(func.coalesce(func.max(DownloadProfile.sort_order), 100)).where(
DownloadProfile.user_id == user.id
)
).scalar_one()
p = DownloadProfile(
user_id=user.id, name=name[:80], spec=spec, is_builtin=False, sort_order=maxpos + 1
)
db.add(p)
p = service.create_profile(db, user, name, spec)
db.commit()
return _serialize_profile(p)
@@ -183,31 +150,32 @@ def update_profile(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
p = db.get(DownloadProfile, profile_id)
if p is None or p.is_builtin or p.user_id != user.id:
raise not_found("profile")
p = _own_profile(db, user, profile_id)
sent = body.model_fields_set
name = None
if "name" in sent:
name = (body.name or "").strip()
if not name:
raise refused(400, "profile_name_required")
p.name = name[:80]
if "spec" in sent:
if body.spec is None:
raise refused(400, "spec_not_object")
p.spec = body.spec
if "spec" in sent and body.spec is None:
raise refused(400, "spec_not_object")
service.update_profile(p, name=name, spec=body.spec if "spec" in sent else None)
db.commit()
return _serialize_profile(p)
def _own_profile(db: Session, user: User, profile_id: int) -> "DownloadProfile":
p = queries.own_profile(db, user, profile_id)
if p is None:
raise not_found("profile")
return p
@router.delete("/profiles/{profile_id}")
def delete_profile(
profile_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
p = db.get(DownloadProfile, profile_id)
if p is None or p.is_builtin or p.user_id != user.id:
raise not_found("profile")
db.delete(p)
service.delete_profile(db, _own_profile(db, user, profile_id))
db.commit()
return {"deleted": profile_id}
@@ -277,18 +245,13 @@ def _resolve_spec(
"""Pick the format spec for an enqueue: an explicit profile_id (builtin or the user's own),
an inline spec, or the first builtin as a fallback."""
if profile_id is not None:
p = db.get(DownloadProfile, profile_id)
if p is None or (not p.is_builtin and p.user_id != user.id):
p = queries.usable_profile(db, user, profile_id)
if p is None:
raise not_found("profile")
return p.spec, p.id
if spec is not None:
return spec, None
fallback = db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True))
.order_by(DownloadProfile.sort_order)
.limit(1)
).scalar_one_or_none()
fallback = queries.default_profile(db)
if fallback is None:
raise refused(400, "no_profile")
return fallback.spec, fallback.id
@@ -326,6 +289,7 @@ def enqueue_download(
job = service.enqueue(db, user.id, source_kind, source_ref, spec, profile_id, display_name)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
db.commit()
return serialize_job(db, job)
@@ -355,6 +319,7 @@ def enqueue_edit(
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
except service.EditError as e:
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason}) from e
db.commit()
return serialize_job(db, job)
@@ -398,38 +363,22 @@ def enqueue_clip(
raise HTTPException(status_code=status, detail=detail) from e
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
db.commit()
return serialize_job(db, job)
# How long a finished job stays in the queue: the tray polls it from every page and must see a job
# turn done (its "Done · Open" row and the done notice) — the finished file itself is the library's.
# A done job is timed from when it became done, never from its last write: a rename, a move or the
# GC's expiry warning touches an old file's row, and the tray would announce it as just finished.
QUEUE_RECENT = timedelta(minutes=10)
@router.get("")
def list_downloads(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
"""The queue: what is waiting, running, paused or failed, and what finished a moment ago."""
recent = datetime.now(timezone.utc) - QUEUE_RECENT
jobs = (
db.execute(
select(DownloadJob)
.where(
DownloadJob.user_id == user.id,
DownloadJob.status.not_in(FINISHED)
| ((DownloadJob.status == JobStatus.DONE) & (DownloadJob.ready_at >= recent))
| ((DownloadJob.status == JobStatus.CANCELED) & (DownloadJob.updated_at >= recent)),
)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
)
.scalars()
.all()
)
assets = assets_for(db, jobs)
positions = positions_for(db, user, jobs)
sources = source_keys(db, jobs)
jobs = queries.queue(db, user, datetime.now(timezone.utc) - QUEUE_RECENT)
assets = queries.assets_for(db, jobs)
positions = queries.positions_for(db, user, jobs)
sources = queries.source_keys(db, jobs)
return [
{
**serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None),
@@ -453,14 +402,8 @@ _INDEX_RANK: dict[str, int] = {
def my_download_index(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
"""source_ref -> best active status, for the feed's per-card "downloaded / queued" badge.
Small + cheap so the feed can poll it without loading the full job list."""
rows = db.execute(
select(DownloadJob.source_ref, DownloadJob.status).where(
DownloadJob.user_id == user.id,
DownloadJob.status.in_(tuple(_INDEX_RANK)),
)
).all()
out: dict[str, str] = {}
for ref, status in rows:
for ref, status in queries.statuses_by_source(db, user, tuple(_INDEX_RANK)):
if _INDEX_RANK.get(status, 0) > _INDEX_RANK.get(out.get(ref, ""), 0):
out[ref] = status
return out
@@ -472,6 +415,7 @@ def pause_download(
) -> dict:
job = own_job_for_update(db, user, job_id)
service.pause(db, job)
db.commit()
return serialize_job(db, job)
@@ -481,6 +425,7 @@ def resume_download(
) -> dict:
job = own_job_for_update(db, user, job_id)
service.resume(db, job)
db.commit()
return serialize_job(db, job)
@@ -508,6 +453,7 @@ def redownload_download(
if job.status in (JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED):
raise refused(409, "still_downloading")
service.redownload(db, job)
db.commit()
return serialize_job(db, job)
@@ -517,6 +463,7 @@ def cancel_download(
) -> dict:
job = own_job_for_update(db, user, job_id)
service.cancel(db, job)
db.commit()
return serialize_job(db, job)
@@ -534,24 +481,7 @@ def set_library_progress(
one shared with them). A position too early or too close to the end clears it instead."""
accessible_job(db, user, job_id)
position = kept_position(body)
mine = (LibraryState.user_id == user.id) & (LibraryState.job_id == job_id)
if not position:
# A statement, not a loaded row's delete: the periodic checkpoint and the pagehide beacon of
# one stop may both clear it, and the second then deletes nothing rather than failing.
db.execute(delete(LibraryState).where(mine))
db.commit()
return {"ok": True, "job_id": job_id, "position_seconds": 0}
now = datetime.now(timezone.utc)
# One row per (user, job) even when two checkpoints race (the same pair): an upsert, not a
# read-then-insert that the second would fail on the unique key.
db.execute(
pg_insert(LibraryState)
.values(user_id=user.id, job_id=job_id, position_seconds=position, progress_updated_at=now)
.on_conflict_do_update(
constraint="uq_library_user_job",
set_={"position_seconds": position, "progress_updated_at": now},
)
)
service.set_position(db, user, job_id, position)
db.commit()
return {"ok": True, "job_id": job_id, "position_seconds": position}
@@ -564,8 +494,7 @@ def storyboard_meta(
Best-effort: for a very long/high-res source the sprite may not generate in time — then
`available` is False and the editor shows a plain timeline."""
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
asset = queries.asset_of(db, accessible_job(db, user, job_id))
if asset is None or asset.status != "ready" or not asset.rel_path:
raise refused(409, "download_not_ready")
meta = editmod.ensure_storyboard(asset, settings.files_root)
@@ -585,8 +514,7 @@ def storyboard_meta(
@router.get("/{job_id}/poster.jpg")
def poster_image(job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)):
"""The locally-generated poster frame for a download whose source had no thumbnail."""
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
asset = queries.asset_of(db, accessible_job(db, user, job_id))
if asset is None or not asset.poster_path:
raise not_found("poster")
path = storage.safe_abs_path(settings.files_root, asset.poster_path)
@@ -599,8 +527,7 @@ def poster_image(job_id: int, user: User = Depends(current_user), db: Session =
def storyboard_image(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
):
job = accessible_job(db, user, job_id)
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
asset = queries.asset_of(db, accessible_job(db, user, job_id))
if asset is None or not asset.storyboard_path:
raise not_found("filmstrip")
path = storage.safe_abs_path(settings.files_root, asset.storyboard_path)
@@ -623,45 +550,20 @@ def share_download(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
job = own_job(db, user, job_id)
if job.status != JobStatus.DONE or job.asset_id is None:
raise refused(409, "share_unfinished")
target = body.email.strip().lower()
if not target:
raise refused(400, "recipient_required")
recipient = db.execute(
select(User).where(func.lower(User.email) == target)
).scalar_one_or_none()
recipient = user_by_email(db, target)
# Uniform "no user" for a missing OR non-messageable (suspended/deactivated) target, so
# sharing can't be used to probe which addresses exist or to reach a suspended account.
if not is_messageable_user(recipient):
if recipient is None or not is_messageable_user(recipient):
raise not_found("recipient")
if recipient.id == user.id:
raise refused(400, "share_own")
exists = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == recipient.id
)
).first()
if exists is None:
db.add(DownloadShare(job_id=job_id, from_user_id=user.id, to_user_id=recipient.id))
from app.notifications import create_notification
create_notification(
db,
user_id=recipient.id,
type="download_shared",
title=(job.display_name or "A download") + " was shared with you",
# The client names the notice in the reader's language from `data`; the title is the
# English fallback only.
data={
"kind": "download_shared",
"from": user.email,
"job_id": job_id,
"title": job.display_name,
},
commit=False,
)
if service.share(db, job, user, recipient):
db.commit()
return {"ok": True, "shared_with": recipient.email}
@@ -673,22 +575,10 @@ def unshare_download(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
owned_or_404(
db, DownloadJob, job_id, user, what="download"
) # ownership guard (404s if not the caller's job)
recipient = db.execute(
select(User).where(func.lower(User.email) == email.strip().lower())
).scalar_one_or_none()
if recipient is not None:
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id,
DownloadShare.to_user_id == recipient.id,
)
).scalar_one_or_none()
if row is not None:
db.delete(row)
db.commit()
own_job(db, user, job_id) # ownership guard (404s if not the caller's job)
recipient = user_by_email(db, email)
if recipient is not None and service.unshare(db, job_id, recipient):
db.commit()
return {"ok": True}
@@ -701,54 +591,26 @@ def share_recipients(
) -> list[dict]:
"""Registered users this user can share a download with (excludes self + any non-messageable
account — suspended or deactivated — so the picker never leaks those addresses)."""
rows = (
db.execute(
select(User)
.where(User.id != user.id, *messageable_clauses())
.order_by(collated(User.email, lang_for_user(user)))
)
.scalars()
.all()
)
rows = messageable_others(db, user, lang_for_user(user))
return [{"id": u.id, "email": u.email, "display_name": u.display_name} for u in rows]
# --- public watch links (share by link) ----------------------------------------------------
def _own_link(db: Session, user: User, link_id: int) -> DownloadLink:
link = db.get(DownloadLink, link_id)
def _own_link(db: Session, user: User, link_id: int) -> "DownloadLink":
link = queries.own_link(db, user, link_id)
if link is None:
raise not_found("link")
target = (
db.get(DownloadJob, link.job_id)
if link.job_id is not None
else db.get(FileFolder, link.folder_id)
)
if target is None or target.user_id != user.id:
raise not_found("link")
return link
def _link_expiry(days: int | None) -> datetime | None:
# None/0/negative → no expiry; otherwise N days out, capped at ~10 years. The Pydantic model
# already coerced/validated the type, so a non-number is a 422 before we get here.
if not days or days <= 0:
return None
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
def _links_of(db: Session, *where) -> list[dict]:
rows = db.execute(select(DownloadLink).where(*where).order_by(DownloadLink.id.desc()))
return [linksmod.owner_view(row) for row in rows.scalars()]
@router.get("/{job_id}/links")
def list_links(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
owned_or_404(db, DownloadJob, job_id, user, what="download")
return _links_of(db, DownloadLink.job_id == job_id)
own_job(db, user, job_id)
return [linksmod.owner_view(link) for link in queries.links_to(db, job_id=job_id)]
class CreateLinkIn(BaseModel):
@@ -764,30 +626,18 @@ class CreateLinkIn(BaseModel):
def _new_link(
db: Session, user: User, body: CreateLinkIn, **target: int
) -> tuple[DownloadLink, str]:
"""Make a link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and
the token it is opened by — which is in hand only now."""
pw = body.password.strip()
token = linksmod.new_token()
link = DownloadLink(
**target,
created_by=user.id,
allow_download=body.allow_download,
password_hash=hash_password(pw) if pw else None,
expires_at=_link_expiry(body.expires_days),
)
linksmod.seal(link, token)
db.add(link)
db.flush()
audit.record(
) -> tuple["DownloadLink", str]:
"""A link to `target` (`job_id=` or `folder_id=`, already checked to be the user's) and its
token (`links.create`). The Pydantic model already coerced the expiry, so a non-number is a 422
before this runs."""
return linksmod.create(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_CREATE,
target_type="download_link",
target_id=link.id,
after={**target, **_link_audit(link)},
user,
password=body.password.strip(),
allow_download=body.allow_download,
expires_days=body.expires_days,
**target,
)
return link, token
@router.post("/{job_id}/links")
@@ -805,28 +655,20 @@ def create_link(
return linksmod.owner_view(link, token=token)
def _shareable_job(db: Session, user: User, job_id: int) -> DownloadJob:
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
rather than failing the link's foreign key — or, done first, leaves a 404."""
owned_or_404(db, DownloadJob, job_id, user, what="download")
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update(read=True)
.execution_options(populate_existing=True)
).scalar_one_or_none()
def _shareable_job(db: Session, user: User, job_id: int) -> "DownloadJob":
"""One of the user's files, held for the link's transaction (`service.hold_for_share`)."""
job = service.hold_for_share(db, user, job_id)
if job is None:
raise not_found("download")
return job
def _shareable_folder(db: Session, user: User, folder_id: int) -> FileFolder:
def _shareable_folder(db: Session, user: User, folder_id: int) -> "FileFolder":
"""One of the user's own folders. A source folder lets its files go on its own clock, and the Hub
is the user's channel to a machine of theirs: neither is a shelf to hand out. Held (FOR SHARE)
for the rest of the transaction, so a delete of it at the same moment waits and takes the new
link with it, rather than failing the link's foreign key."""
owned_or_404(db, FileFolder, folder_id, user, what="folder")
own_folder(db, user, folder_id)
try:
folder = folders.own_folder(db, user.id, folder_id)
except folders.FolderError as e:
@@ -840,8 +682,8 @@ def _shareable_folder(db: Session, user: User, folder_id: int) -> FileFolder:
def list_folder_links(
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
owned_or_404(db, FileFolder, folder_id, user, what="folder")
return _links_of(db, DownloadLink.folder_id == folder_id)
own_folder(db, user, folder_id)
return [linksmod.owner_view(link) for link in queries.links_to(db, folder_id=folder_id)]
@router.post("/folders/{folder_id}/links")
@@ -857,15 +699,6 @@ def create_folder_link(
return linksmod.owner_view(link, token=token)
def _link_audit(link: DownloadLink) -> dict:
"""A link's settings as the audit records them: whether it has a password, never which."""
return {
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": iso(link.expires_at),
}
class UpdateLinkIn(BaseModel):
allow_download: bool | None = None
expires_days: int | None = None
@@ -880,29 +713,15 @@ def update_link(
db: Session = Depends(get_db),
) -> dict:
link = _own_link(db, user, link_id)
before = _link_audit(link)
sent = body.model_fields_set
if "allow_download" in sent:
link.allow_download = bool(body.allow_download)
if "expires_days" in sent:
link.expires_at = _link_expiry(body.expires_days)
if "password" in sent:
pw = (body.password or "").strip()
link.password_hash = hash_password(pw) if pw else None
# Bump the version so any grant issued under the previous password stops working now, rather
# than living out its 6h TTL (setting/clearing/changing the password all count as a rotation).
link.password_version += 1
after = _link_audit(link)
if after != before or "password" in sent:
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
target_type="download_link",
target_id=link_id,
before=before,
after=after,
)
linksmod.update(
db,
user,
link,
allow_download=body.allow_download,
expires_days=body.expires_days,
password=body.password,
sent=frozenset(body.model_fields_set),
)
db.commit()
return linksmod.owner_view(link)
@@ -911,16 +730,7 @@ def update_link(
def revoke_link(
link_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
link = _own_link(db, user, link_id)
db.delete(link)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
target_type="download_link",
target_id=link_id,
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
)
linksmod.revoke(db, user, _own_link(db, user, link_id))
db.commit()
return {"ok": True, "deleted": link_id}
@@ -932,20 +742,9 @@ def revoke_link(
def admin_list_downloads(
admin: User = Depends(admin_user), db: Session = Depends(get_db)
) -> list[dict]:
jobs = (
db.execute(
select(DownloadJob)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
)
.scalars()
.all()
)
assets = assets_for(db, jobs)
emails = {
u.id: u.email
for u in db.execute(select(User).where(User.id.in_({j.user_id for j in jobs}))).scalars()
}
jobs = queries.recent_jobs(db)
assets = queries.assets_for(db, jobs)
emails = emails_of(db, {j.user_id for j in jobs})
out = []
for j in jobs:
data = serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None)
@@ -957,27 +756,11 @@ def admin_list_downloads(
@admin_router.get("/storage")
def admin_storage(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
# Every file on disk (`quota.ON_DISK`): ready ones, and uploads waiting for or failed in their
# conversion — the same disk the footprint and the GC cap count.
on_disk = db.execute(
select(func.count(), func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).one()
total_assets = db.execute(select(func.count()).select_from(MediaAsset)).scalar()
files_on_disk, total_bytes, total_assets = queries.storage_totals(db)
# EVERY user, not only those holding a file: a quota is set before anyone downloads.
emails: dict[int, str] = {
uid: email
for uid, email in db.execute(select(User.id, User.email).order_by(User.email)).tuples()
}
emails = emails_of(db)
uids = list(emails)
# What each one's Files lists: their finished files.
files: dict[int, int] = {
uid: n
for uid, n in db.execute(
select(DownloadJob.user_id, func.count())
.where(DownloadJob.status == JobStatus.DONE)
.group_by(DownloadJob.user_id)
).tuples()
}
files = queries.done_files_per_user(db)
def row(uid: int) -> dict:
lim = quota.resolve(db, uid)
@@ -991,8 +774,8 @@ def admin_storage(admin: User = Depends(admin_user), db: Session = Depends(get_d
}
return {
"files_on_disk": on_disk[0],
"total_bytes": int(on_disk[1]),
"files_on_disk": files_on_disk,
"total_bytes": total_bytes,
"total_assets": total_assets,
"total_cap_bytes": sysconfig.get_int(db, "download_total_max_bytes"),
"per_user": [row(uid) for uid in uids],
@@ -1004,10 +787,9 @@ def admin_get_quota(
user_id: int, admin: User = Depends(admin_user), db: Session = Depends(get_db)
) -> dict:
lim = quota.resolve(db, user_id)
row = db.get(DownloadQuota, user_id)
return {
"user_id": user_id,
"custom": row is not None,
"custom": queries.custom_quota(db, user_id) is not None,
"max_bytes": lim.max_bytes,
"max_concurrent": lim.max_concurrent,
"max_jobs": lim.max_jobs,
@@ -1032,25 +814,14 @@ def admin_set_quota(
db: Session = Depends(get_db),
) -> dict:
get_or_404(db, User, user_id, what="user")
cur = quota.resolve(db, user_id)
row = db.get(DownloadQuota, user_id)
if row is None:
row = DownloadQuota(
user_id=user_id,
max_bytes=cur.max_bytes,
max_concurrent=cur.max_concurrent,
max_jobs=cur.max_jobs,
unlimited=cur.unlimited,
)
db.add(row)
if body.max_bytes is not None:
row.max_bytes = max(0, body.max_bytes)
if body.max_concurrent is not None:
row.max_concurrent = max(1, body.max_concurrent)
if body.max_jobs is not None:
row.max_jobs = max(1, body.max_jobs)
if body.unlimited is not None:
row.unlimited = body.unlimited
quota.set_custom(
db,
user_id,
max_bytes=body.max_bytes,
max_concurrent=body.max_concurrent,
max_jobs=body.max_jobs,
unlimited=body.unlimited,
)
db.commit()
return admin_get_quota(user_id, admin, db)
@@ -1059,8 +830,6 @@ def admin_set_quota(
def admin_reset_quota(
user_id: int, admin: User = Depends(admin_user), db: Session = Depends(get_db)
) -> dict:
row = db.get(DownloadQuota, user_id)
if row is not None:
db.delete(row)
if quota.reset_custom(db, user_id):
db.commit()
return admin_get_quota(user_id, admin, db)
+51 -183
View File
@@ -1,9 +1,7 @@
from datetime import date, datetime, timezone
from datetime import date
from fastapi import APIRouter, Depends, Query
from pydantic import BaseModel
from sqlalchemy import and_, func, or_, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from sqlalchemy.orm.exc import StaleDataError
@@ -11,34 +9,10 @@ from app import quota
from app.auth import current_user
from app.db import get_db
from app.mailtext import lang_for_user
from app.feedquery import (
decode_cursor,
encode_cursor,
feed_summary,
feed_years,
filtered_query,
keyset_predicate,
order_by,
rank_window_start,
serialize,
sort_keys,
windowed_rank,
windowed_relevance_rows,
)
from app.models import (
ChannelTag,
Tag,
User,
Video,
VideoState,
)
from app.routes._common import (
ProgressIn,
get_or_404,
kept_position,
not_found,
youtube_refusal,
)
from app.feedquery import BadCursor, facet_counts, feed_count, feed_page, feed_summary, feed_years
from app.models import User
from app.reader import queries, service
from app.routes._common import ProgressIn, kept_position, not_found, youtube_refusal
from app.sync.videos import parse_iso8601_duration
from app.utils import iso
from app.youtube.client import YouTubeClient, YouTubeError
@@ -108,49 +82,20 @@ def get_feed(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
query, rank_expr = filtered_query(db, user, **params)
lang = lang_for_user(user)
keys = sort_keys(sort, seed, rank_expr, lang)
vals, cursor_id, window = None, None, None
if cursor:
# The read-model raises a plain ValueError on a malformed cursor; translate it to the
# HTTP 400 here so feedquery stays web-framework-free.
try:
vals, cursor_id, window = decode_cursor(keys, cursor)
except ValueError:
# `from None`: the ValueError is an expected bad-input signal, not an error to chain.
raise refused(400, "invalid_cursor") from None
ranked = sort == "relevance" and rank_expr is not None
if ranked and not cursor:
window = rank_window_start(db, query)
if ranked and window is not None:
keys = sort_keys(sort, seed, windowed_rank(rank_expr, window), lang)
# Expose each key column on the row so we can build the next cursor from the last item.
query = query.add_columns(*[k["expr"].label(f"_k{i}") for i, k in enumerate(keys)])
if vals is not None:
for k, v in zip(keys, vals, strict=True):
k["value"] = v
if ranked and window is not None:
rows = windowed_relevance_rows(db, query, keys, window, cursor_id, limit + 1)
else:
query = query.order_by(*order_by(keys))
if cursor_id is not None:
query = query.where(keyset_predicate(keys, cursor_id))
rows = db.execute(query.limit(limit + 1)).all()
has_more = len(rows) > limit
page = rows[:limit]
return {
"items": [serialize(r) for r in page],
"next_cursor": (
encode_cursor(keys, page[-1], window if ranked else None)
if (has_more and page)
else None
),
"has_more": has_more,
"limit": limit,
}
try:
return feed_page(
db,
user,
params,
sort=sort,
seed=seed,
limit=limit,
cursor=cursor,
lang=lang_for_user(user),
)
except BadCursor:
# `from None`: a malformed cursor is an expected bad-input signal, not an error to chain.
raise refused(400, "invalid_cursor") from None
@router.get("/feed/count")
@@ -162,9 +107,7 @@ def get_feed_count(
# SUPERSEDED by /feed/summary (PERF-13) — the app no longer calls this. Kept as the reference
# oracle that test_feed_summary_matches_count_plus_facets asserts /feed/summary's `count` against;
# remove both together with that test if this endpoint is ever retired.
query, _rank = filtered_query(db, user, **params)
total = db.scalar(select(func.count()).select_from(query.subquery()))
return {"count": total or 0}
return {"count": feed_count(db, user, **params)}
@router.get("/facets")
@@ -186,31 +129,7 @@ def get_facets(
SUPERSEDED by /feed/summary (PERF-13) — the app no longer calls this. Kept as the reference
oracle for test_feed_summary_matches_count_plus_facets (it asserts /feed/summary's `facets`
against these `counts`); retire the two together."""
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
counts: dict[int, int] = {}
for category in ("language", "topic", "other"):
# Disjunctive (OR) facets drop the category's own selections so its chips keep
# independent counts (you can OR more of them in). Conjunctive (AND, topics only)
# keeps them applied, so each remaining chip narrows to channels that ALSO have all
# already-selected topics — and tags that can't co-occur drop to zero (hidden).
conjunctive = params.get("tag_modes", {}).get(category) == "and"
base, _rank = filtered_query(
db,
user,
**{**params, "exclude_tag_category": None if conjunctive else category},
)
channels = base.with_only_columns(Video.channel_id).distinct().subquery()
rows = db.execute(
select(ChannelTag.tag_id, func.count(func.distinct(ChannelTag.channel_id)))
.select_from(channels)
.join(ChannelTag, ChannelTag.channel_id == channels.c.channel_id)
.join(Tag, and_(Tag.id == ChannelTag.tag_id, Tag.category == category))
.where(visible)
.group_by(ChannelTag.tag_id)
).all()
for tag_id, count in rows:
counts[tag_id] = count
return {"counts": counts}
return {"counts": facet_counts(db, user, **params)}
@router.get("/feed/summary")
@@ -239,6 +158,11 @@ def get_feed_years(
return {"years": feed_years(db, user, **params)}
def _video_or_404(db: Session, video_id: str) -> None:
if queries.video(db, video_id) is None:
raise not_found("video")
class VideoStateIn(BaseModel):
status: str
@@ -253,50 +177,17 @@ def set_video_state(
status = body.status
if status not in VALID_STATES:
raise refused(400, "invalid_status")
get_or_404(db, Video, video_id, what="video")
now = datetime.now(timezone.utc)
_video_or_404(db, video_id)
if status == "new":
# Un-marking: keep the row if it still holds a resume position (un-marking
# "watched" should restore the in-progress state, not wipe where the user left
# off); otherwise drop it. The in-app player fires this alongside a progress
# checkpoint that may DELETE the same row, so tolerate it vanishing under us.
row = db.execute(
select(VideoState).where(VideoState.user_id == user.id, VideoState.video_id == video_id)
).scalar_one_or_none()
if row is not None:
if row.position_seconds:
row.status = "new"
row.watched_at = None
else:
db.delete(row)
try:
db.commit()
# silent-ok: the documented lost race — a concurrent player checkpoint already removed
# or moved this row. The state the caller asked for is the state that holds either way.
except StaleDataError:
db.rollback()
try:
service.unmark(db, user, video_id)
db.commit()
# silent-ok: the documented lost race — a concurrent player checkpoint already removed or
# moved this row (`service.unmark`). The state the caller asked for holds either way.
except StaleDataError:
db.rollback()
return {"ok": True, "video_id": video_id, "status": "new"}
# watched / hidden: atomic upsert. The player marks "watched" at end-of-playback at
# the same instant a progress checkpoint may DELETE the in-progress row; a plain
# SELECT-then-UPDATE then raises StaleDataError ("0 rows matched") on that race. An
# INSERT … ON CONFLICT DO UPDATE keyed on uq_user_video is immune to it.
set_: dict = {"status": status, "updated_at": now}
if status == "watched":
set_["watched_at"] = now # hidden keeps any existing watched_at
stmt = (
pg_insert(VideoState)
.values(
user_id=user.id,
video_id=video_id,
status=status,
watched_at=now if status == "watched" else None,
)
.on_conflict_do_update(constraint="uq_user_video", set_=set_)
)
db.execute(stmt)
service.mark(db, user, video_id, status)
db.commit()
return {"ok": True, "video_id": video_id, "status": status}
@@ -310,16 +201,12 @@ def clear_video_state(
"""Reset a video to pristine for this user — drop the whole VideoState row (status,
watch position and watched_at), as if it had never been opened. Saved/playlist membership
lives elsewhere and is untouched."""
row = db.execute(
select(VideoState).where(VideoState.user_id == user.id, VideoState.video_id == video_id)
).scalar_one_or_none()
if row is not None:
db.delete(row)
try:
db.commit()
# silent-ok: the row went under the delete, which is the outcome the delete wanted.
except StaleDataError:
db.rollback()
try:
service.clear_state(db, user, video_id)
db.commit()
# silent-ok: the row went under the delete, which is the outcome the delete wanted.
except StaleDataError:
db.rollback()
return {"ok": True, "video_id": video_id, "status": "new", "position_seconds": 0}
@@ -335,36 +222,17 @@ def set_video_progress(
partially-watched video can render a progress bar and match the "in progress" filter.
Trivially-early and near-finished positions clear the position rather than store it."""
position = kept_position(body)
get_or_404(db, Video, video_id, what="video")
row = db.execute(
select(VideoState).where(VideoState.user_id == user.id, VideoState.video_id == video_id)
).scalar_one_or_none()
_video_or_404(db, video_id)
if not position:
# Nothing meaningful to store: drop a status-less row entirely, otherwise just
# zero the position (a saved/hidden video keeps its status).
if row is not None:
if row.status == "new":
db.delete(row)
else:
row.position_seconds = 0
row.progress_updated_at = datetime.now(timezone.utc)
# A concurrent state change (e.g. the end-of-playback "watched" mark) may
# have removed/updated this row already — tolerate the lost race.
try:
db.commit()
# silent-ok: exactly the race the comment above describes; the response stays true.
except StaleDataError:
db.rollback()
try:
service.set_progress(db, user, video_id, 0)
db.commit()
# silent-ok: a concurrent state change (the end-of-playback "watched" mark) removed or
# updated this row already — the lost race; the response stays true.
except StaleDataError:
db.rollback()
return {"ok": True, "video_id": video_id, "position_seconds": 0}
if row is None:
row = VideoState(user_id=user.id, video_id=video_id)
db.add(row)
row.position_seconds = position
row.progress_updated_at = datetime.now(timezone.utc)
service.set_progress(db, user, video_id, position)
db.commit()
return {"ok": True, "video_id": video_id, "position_seconds": position}
@@ -382,7 +250,7 @@ def get_video_detail(
our DB (e.g. a YouTube link inside another video's description that the in-app
player navigated to) is resolved via the YouTube API (videos.list, 1 unit,
attributed to the requesting user)."""
v = db.get(Video, video_id)
v = queries.video(db, video_id)
if v is not None:
return {
"id": v.id,
+93 -243
View File
@@ -5,8 +5,10 @@ the per-user layer (name, folder, kept, expiry, shares and links all hang off it
bytes. Folders are rows (`downloads.folders`), so a rename or a move never touches the disk.
In-flight jobs are not here: they are the download tray's (`routes.downloads`). The helpers above
the routes are shared with that module — serialization, the ownership locks, the upload spool —
because a download and a library file are one row at two moments.
the routes are shared with that module — serialization, the ownership checks, the upload spool —
because a download and a library file are one row at two moments. What is read comes from
`downloads.queries`, what is written goes through Capture's service (`service`, `folders`); this
module validates, answers in HTTP and commits.
A machine with an API token (`app.apitokens`) may use the routes that take `files_caller`: list the
folders and files, fetch a file and its hash, upload, delete — each only in a folder its token was
@@ -19,51 +21,37 @@ import hashlib
import os
import tempfile
import threading
from collections.abc import AsyncIterator, Sequence
from collections.abc import AsyncIterator
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import BinaryIO, Literal
from typing import TYPE_CHECKING, BinaryIO, Literal
from urllib.parse import unquote, urlparse
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import FileResponse, Response
from pydantic import BaseModel, Field
from sqlalchemy import func, or_, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import audit, sysconfig
from app.apitokens import FilesCaller, Right, files_caller
from app.audit import AuditAction
from app.auth import current_user
from app.collation import collated
from app.config import settings
from app.db import get_db, release_read_connection
from app.downloads import folders, preview, quota, service, storage
from app.downloads import folders, preview, queries, quota, service, storage
from app.downloads import upload as uploadmod
from app.downloads.status import JobStatus
from app.mailtext import lang_for_user
from app.models import (
DISPLAY_TEXT_MAX,
DownloadJob,
DownloadLink,
DownloadShare,
FileFolder,
LibraryState,
MediaAsset,
User,
display_text,
)
from app.models import DISPLAY_TEXT_MAX, User, display_text
from app.refusal import refused
from app.routes._common import capped_stream_to, not_found, owned_or_404
from app.utils import iso, like_escape
from app.routes._common import capped_stream_to, not_found
from app.utils import iso
if TYPE_CHECKING:
from app.downloads.models import DownloadJob, FileFolder, MediaAsset
router = APIRouter(prefix="/api/files", tags=["files"])
# Cap the listing so a heavy library can't return an unbounded result set (the old Download Center's
# own cap; far beyond any real library).
MAX_LIST = 500
# The longest a folder may keep a file, in days — ten years is "forever" in all but name.
_MAX_RETENTION_DAYS = 3650
_HASH_CHUNK = 1 << 20
@@ -75,7 +63,7 @@ _HASHING = threading.BoundedSemaphore(_HASHERS)
# --- shared with routes.downloads -------------------------------------------------------
def serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
def serialize(job: "DownloadJob", asset: "MediaAsset | None") -> dict:
ready = asset is not None and asset.status == "ready" and bool(asset.rel_path)
return {
"id": job.id,
@@ -135,58 +123,18 @@ def serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
}
def assets_for(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, MediaAsset]:
ids = {j.asset_id for j in jobs if j.asset_id}
if not ids:
return {}
rows = db.execute(select(MediaAsset).where(MediaAsset.id.in_(ids))).scalars()
return {a.id: a for a in rows}
def positions_for(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, int]:
"""`user`'s resume point in each of `jobs` that has one (the library player's `library_states`)."""
ids = [j.id for j in jobs]
if not ids:
return {}
rows = db.execute(
select(LibraryState.job_id, LibraryState.position_seconds).where(
LibraryState.user_id == user.id, LibraryState.job_id.in_(ids)
)
).all()
return {job_id: pos for job_id, pos in rows}
def serialize_job(db: Session, job: DownloadJob) -> dict:
def serialize_job(db: Session, job: "DownloadJob") -> dict:
"""Resolve a job's cache asset (if any) and serialize the pair — the resolve-then-serialize step
every single-job handler shares."""
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
return serialize(job, asset)
return serialize(job, queries.asset_of(db, job))
def own_job_for_update(db: Session, user: User, job_id: int) -> DownloadJob:
"""The ownership check (`owned_or_404`) for a transition that moves its asset's ref count (resume, cancel, delete, redownload)
or writes the status those decide on (pause). The job is re-read under a row lock, so two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`service.finalize_if_running`), so the two can never deadlock."""
while True:
asset_id = owned_or_404(db, DownloadJob, job_id, user, what="download").asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
raise not_found("download")
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
def own_job_for_update(db: Session, user: User, job_id: int) -> "DownloadJob":
"""One of the user's jobs under its row lock (`service.lock_own_job`), or a 404."""
job = service.lock_own_job(db, user, job_id)
if job is None:
raise not_found("download")
return job
def quota_detail(e: quota.QuotaExceeded) -> dict:
@@ -322,30 +270,32 @@ async def spool_upload(
tmp_path.unlink(missing_ok=True)
def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob:
"""A file the caller may read: their own, or one shared with them."""
job = db.get(DownloadJob, job_id)
def accessible_job(db: Session, user: User, job_id: int) -> "DownloadJob":
"""A file the caller may read (`queries.accessible_job`: their own, or shared with them)."""
job = queries.accessible_job(db, user, job_id)
if job is None:
raise not_found("download")
if job.user_id == user.id:
return job
shared = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == user.id
)
).first()
if shared is None:
raise not_found("download")
return job
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob:
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
own in a folder it may read."""
if caller.grants is None:
return accessible_job(db, caller.user, job_id)
job = db.get(DownloadJob, job_id)
if job is None or job.user_id != caller.user.id or not caller.may(job.folder_id, Right.READ):
def own_job(db: Session, user: User, job_id: int) -> "DownloadJob":
job = queries.own_job(db, user, job_id)
if job is None:
raise not_found("download")
return job
def own_folder(db: Session, user: User, folder_id: int) -> "FileFolder":
folder = queries.own_folder(db, user, folder_id)
if folder is None:
raise not_found("folder")
return folder
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> "DownloadJob":
"""A file `caller` may read (`queries.readable_job`), or a 404."""
job = queries.readable_job(db, caller, job_id)
if job is None:
raise not_found("download")
return job
@@ -353,7 +303,7 @@ def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob:
# --- folders -------------------------------------------------------------------------------
def _serialize_folder(folder: FileFolder, count: int, size: int, linked: bool = False) -> dict:
def _serialize_folder(folder: "FileFolder", count: int, size: int, linked: bool = False) -> dict:
return {
"id": folder.id,
"kind": folder.kind,
@@ -369,18 +319,6 @@ def _serialize_folder(folder: FileFolder, count: int, size: int, linked: bool =
}
def _linked(db: Session, folder_ids: list[int]) -> set[int]:
"""Which of these folders a live public link opens (one not past its expiry)."""
now = datetime.now(timezone.utc)
rows = db.execute(
select(DownloadLink.folder_id).where(
DownloadLink.folder_id.in_(folder_ids),
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
)
).scalars()
return {folder_id for folder_id in rows if folder_id is not None}
@router.get("/folders")
def list_folders(
caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
@@ -388,23 +326,9 @@ def list_folders(
"""The caller's folders with how many files each holds and their size (a token: the ones it may
list). A source folder its files have all left stays (count 0): the client decides whether to
show it."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
stmt = select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
).where(FileFolder.user_id == caller.user.id)
listable = caller.folders_with(Right.LIST)
if listable is not None:
stmt = stmt.where(FileFolder.id.in_(listable))
rows = db.execute(
stmt.outerjoin(DownloadJob, done)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.group_by(FileFolder.id)
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
).all()
linked = _linked(db, [f.id for f, _, _ in rows])
return [_serialize_folder(f, count, int(size), f.id in linked) for f, count, size in rows]
rows = queries.folders_with_totals(db, caller.user, caller.folders_with(Right.LIST))
linked = queries.linked_folders(db, [f.id for f, _, _ in rows])
return [_serialize_folder(f, count, size, f.id in linked) for f, count, size in rows]
class FolderIn(BaseModel):
@@ -422,13 +346,10 @@ def _folder_name(raw: str) -> str:
def create_folder(
body: FolderIn, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
folder = FileFolder(user_id=user.id, kind=folders.FolderKind.OWN, name=_folder_name(body.name))
try:
with db.begin_nested():
db.add(folder)
db.flush()
except IntegrityError as e: # the unique (user, lower(name)): one folder per name
raise refused(409, "file_folder_name_taken") from e
folder = folders.create_own(db, user.id, _folder_name(body.name))
except folders.FolderError as e:
raise refused(409, e.reason) from e
audit.record(
db, user.id, AuditAction.FILE_FOLDER_CREATE, target_type="file_folder", target_id=folder.id
)
@@ -451,17 +372,16 @@ def update_folder(
) -> dict:
"""Rename one of the caller's own folders (a source folder's name is its origin's), and/or set
how long any of their folders keeps a file (`folders.set_retention`)."""
folder = owned_or_404(db, FileFolder, folder_id, user, what="folder")
folder = own_folder(db, user, folder_id)
sent = body.model_fields_set
if "name" in sent:
if folder.kind != folders.FolderKind.OWN:
# Which folder may be renamed is answered before what the name is, as it always was.
if not folders.renamable(folder):
raise refused(409, "file_folder_not_own")
folder.name = _folder_name(body.name or "")
try:
with db.begin_nested():
db.flush()
except IntegrityError as e:
raise refused(409, "file_folder_name_taken") from e
folders.rename(db, folder, _folder_name(body.name or ""))
except folders.FolderError as e:
raise refused(409, e.reason) from e
audit.record(
db,
user.id,
@@ -486,18 +406,9 @@ def update_folder(
after={"retention_days": body.retention_days},
)
db.commit()
count, size = _folder_totals(db, folder.id)
return _serialize_folder(folder, count, size, folder.id in _linked(db, [folder.id]))
def _folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
count, size = db.execute(
select(func.count(DownloadJob.id), func.coalesce(func.sum(MediaAsset.size_bytes), 0))
.select_from(DownloadJob)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.folder_id == folder_id, DownloadJob.status == JobStatus.DONE)
).one()
return count, int(size)
count, size = queries.folder_totals(db, folder.id)
linked = folder.id in queries.linked_folders(db, [folder.id])
return _serialize_folder(folder, count, size, linked)
@router.delete("/folders/{folder_id}")
@@ -505,7 +416,7 @@ def delete_folder(
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
"""Delete one of the caller's own folders; its files go back to their source folders."""
folder = owned_or_404(db, FileFolder, folder_id, user, what="folder")
folder = own_folder(db, user, folder_id)
try:
moved = folders.delete_own(db, folder)
except folders.FolderError as e:
@@ -547,13 +458,7 @@ def remove_shared_with_me(
"""Remove a file that was shared WITH this user from their 'Shared with me' list. Deletes
only the recipient's share grant — the owner's job and the physical file are untouched (this is
a per-user dismissal, not a delete)."""
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, DownloadShare.to_user_id == user.id
)
).scalar_one_or_none()
if row is not None:
db.delete(row)
if service.unshare(db, job_id, user):
db.commit()
return {"ok": True, "removed": job_id}
@@ -582,46 +487,22 @@ def list_files(
raise refused(403, "api_token_scope")
if folder_id is not None and folder_id not in listable:
raise not_found("folder")
stmt = (
select(DownloadJob)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.status == JobStatus.DONE)
)
shared = scope == "shared"
if shared:
stmt = stmt.join(DownloadShare, DownloadShare.job_id == DownloadJob.id).where(
DownloadShare.to_user_id == user.id
)
else:
stmt = stmt.where(DownloadJob.user_id == user.id)
if folder_id is not None:
stmt = stmt.where(DownloadJob.folder_id == folder_id)
elif listable is not None:
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
term = q.strip()
if term:
like = f"%{like_escape(term)}%"
stmt = stmt.where(
or_(
name.ilike(like),
func.coalesce(DownloadJob.display_uploader, MediaAsset.uploader, "").ilike(like),
)
)
# A file is added to its owner's list when it is done, not when it was asked for (a download that
# failed for days and then went through is new); one shared with the caller, when it was shared.
key = {
"added": DownloadShare.created_at if shared else DownloadJob.ready_at,
"name": collated(name, lang_for_user(user)),
"size": MediaAsset.size_bytes,
}[sort]
first = key.asc() if order == "asc" else key.desc()
tie = DownloadJob.id.asc() if order == "asc" else DownloadJob.id.desc()
jobs = db.execute(stmt.order_by(first.nulls_last(), tie).limit(MAX_LIST)).scalars().all()
assets = assets_for(db, jobs)
positions = positions_for(db, user, jobs)
shared_at = _shared_at(db, user, jobs) if shared else {}
sources = source_keys(db, jobs)
jobs = queries.library(
db,
user,
folder_id=folder_id,
listable=listable,
shared=shared,
term=q.strip(),
sort=sort,
order=order,
lang=lang_for_user(user),
)
assets = queries.assets_for(db, jobs)
positions = queries.positions_for(db, user, jobs)
shared_at = queries.shared_at(db, user, jobs) if shared else {}
sources = queries.source_keys(db, jobs)
return [
{
**serialize(j, assets.get(j.asset_id) if j.asset_id is not None else None),
@@ -634,36 +515,6 @@ def list_files(
]
def source_keys(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, str]:
"""Where each file came from (its source folder's key), for the list's Source column, the
"back to its source folder" move and the download tray's site: a file in a source folder has its folder's key; one moved into
an own folder (or someone else's, shared) is traced back by `folders.source_key_for`."""
ids = {j.folder_id for j in jobs if j.folder_id is not None}
by_id = (
{f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
if ids
else {}
)
out: dict[int, str] = {}
for j in jobs:
folder = by_id.get(j.folder_id) if j.folder_id is not None else None
if folder is not None and folder.kind == folders.FolderKind.SOURCE and folder.source_key:
out[j.id] = folder.source_key
else:
out[j.id] = folders.source_key_for(db, j)
return out
def _shared_at(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, datetime]:
rows = db.execute(
select(DownloadShare.job_id, DownloadShare.created_at).where(
DownloadShare.to_user_id == user.id,
DownloadShare.job_id.in_([j.id for j in jobs]),
)
).all()
return {job_id: at for job_id, at in rows}
_MAX_EXTRA_LINKS = 8
@@ -711,18 +562,20 @@ def update_file(
"""Rename a file and its channel, set its reference links (all display-only: the file on disk
keeps its own name), move it to another folder (`folders.move`), or keep it until deleted /
hand it back to its folder's retention."""
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
job = own_job(db, user, job_id)
sent = body.model_fields_set
display: dict[str, object] = {}
if "name" in sent:
job.display_name = display_text(body.name)
display["display_name"] = display_text(body.name)
if "uploader" in sent:
job.display_uploader = display_text(body.uploader)
display["display_uploader"] = display_text(body.uploader)
if "uploader_url" in sent:
job.display_uploader_url = _clean_url(body.uploader_url)
display["display_uploader_url"] = _clean_url(body.uploader_url)
if "extra_links" in sent:
raw = body.extra_links or []
cleaned = [u for u in (_clean_url(x) for x in raw) if u][:_MAX_EXTRA_LINKS]
job.extra_links = cleaned or None
display["extra_links"] = cleaned or None
service.set_display(job, **display)
if "folder_id" in sent:
if body.folder_id is None:
raise refused(400, "file_folder_gone")
@@ -768,7 +621,8 @@ def delete_file(
target_id=job_id,
summary=caller.via,
)
service.delete(db, job) # commits, the audit row with it
service.delete(db, job)
db.commit() # the audit row with it
return {"ok": True, "deleted": job_id}
@@ -787,7 +641,7 @@ def upload_file(
return serialize_job(db, ingest_spooled(db, upload))
def ingest_spooled(db: Session, upload: SpooledUpload) -> DownloadJob:
def ingest_spooled(db: Session, upload: SpooledUpload) -> "DownloadJob":
"""The spooled upload into the library (`upload.ingest_upload`, which commits its own rows),
its refusals answered as HTTP."""
if upload.size == 0:
@@ -811,8 +665,8 @@ def ingest_spooled(db: Session, upload: SpooledUpload) -> DownloadJob:
return job
def _ready_file(db: Session, job: DownloadJob) -> tuple[MediaAsset, Path]:
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
def _ready_file(db: Session, job: "DownloadJob") -> tuple["MediaAsset", Path]:
asset = queries.asset_of(db, job)
if asset is None or asset.status != "ready" or not asset.rel_path:
raise refused(409, "download_not_ready")
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
@@ -821,7 +675,7 @@ def _ready_file(db: Session, job: DownloadJob) -> tuple[MediaAsset, Path]:
return asset, path
def serve_file(db: Session, job: DownloadJob) -> FileResponse:
def serve_file(db: Session, job: "DownloadJob") -> FileResponse:
"""The file of `job` (the caller may read it), range-aware, named by the user's display name.
The served Content-Type comes from the ffprobe-verified media_kind + a container allow-list
@@ -839,7 +693,7 @@ def serve_file(db: Session, job: DownloadJob) -> FileResponse:
if asset.sha256:
digest = base64.b64encode(bytes.fromhex(asset.sha256)).decode()
headers["Repr-Digest"] = f"sha-256=:{digest}:"
asset.last_access_at = datetime.now(timezone.utc) # the handler commits it
service.touch(asset) # the handler commits it
media_type = storage.served_media_type(asset.media_kind, asset.container)
if media_type is None:
return FileResponse(
@@ -894,13 +748,13 @@ def preview_file(
text = preview.text_utf8(head)
if text is None:
raise refused(415, "file_no_preview")
asset.last_access_at = datetime.now(timezone.utc)
service.touch(asset)
db.commit()
return Response(text, media_type=preview.media_type(kind, asset.container), headers=headers)
filename = storage.download_filename(
job.display_name or asset.title or asset.source_ref, asset.container, path
)
asset.last_access_at = datetime.now(timezone.utc)
service.touch(asset)
db.commit()
return FileResponse(
path,
@@ -936,10 +790,6 @@ def get_sha256(
finally:
_HASHING.release()
digest = h.hexdigest()
db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id, MediaAsset.sha256.is_(None))
.values(sha256=digest)
)
service.store_sha256(db, asset_id, digest)
db.commit()
return {"sha256": digest}
+7 -6
View File
@@ -5,6 +5,7 @@ from fastapi import APIRouter, Depends, Query
from sqlalchemy import func, select, update
from sqlalchemy.orm import Session
from app.userscope import owned_by
from app.auth import current_user
from app.db import get_db
from app.models import Notification, User
@@ -36,7 +37,7 @@ def list_notifications(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
where = [Notification.user_id == user.id]
where = [owned_by(Notification.user_id, user)]
if not include_dismissed:
where.append(Notification.dismissed.is_(False))
total = db.scalar(select(func.count()).select_from(Notification).where(*where)) or 0
@@ -61,7 +62,7 @@ def unread_count(user: User = Depends(current_user), db: Session = Depends(get_d
select(func.count())
.select_from(Notification)
.where(
Notification.user_id == user.id,
owned_by(Notification.user_id, user),
Notification.read.is_(False),
Notification.dismissed.is_(False),
)
@@ -77,8 +78,8 @@ def mark_read(
) -> dict:
n = owned_or_404(db, Notification, notification_id, user, what="notification")
n.read = True
db.commit()
trim_read(db, user.id)
db.commit()
return {"ok": True}
@@ -86,11 +87,11 @@ def mark_read(
def mark_all_read(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
db.execute(
update(Notification)
.where(Notification.user_id == user.id, Notification.read.is_(False))
.where(owned_by(Notification.user_id, user), Notification.read.is_(False))
.values(read=True)
)
db.commit()
trim_read(db, user.id)
db.commit()
return {"ok": True}
@@ -113,7 +114,7 @@ def clear_all(user: User = Depends(current_user), db: Session = Depends(get_db))
"""Dismiss every (still-visible) notification for this user."""
db.execute(
update(Notification)
.where(Notification.user_id == user.id, Notification.dismissed.is_(False))
.where(owned_by(Notification.user_id, user), Notification.dismissed.is_(False))
.values(dismissed=True, read=True)
)
db.commit()
+61 -304
View File
@@ -3,31 +3,34 @@ CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-w
mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from sqlalchemy import func, select, and_
from sqlalchemy.orm import Session, aliased
from sqlalchemy.orm import Session
from app import quota
from app.auth import current_user, has_read_scope, has_write_scope, require_write_scope
from app.db import get_db
from app.feedquery import feed_columns, serialize
from app.models import Channel, Playlist, PlaylistItem, User, Video, VideoState
from app.feedquery import serialize
from app.models import User
from app.reader import queries, service
from app.sync.playlists import (
PlaylistGone,
lock_playlists,
merge_playlist,
plan_merge,
recompute_dirty,
pull_user_playlists,
repull_playlist,
sync_claim,
user_sync_claim,
)
from app.ratelimit import RateLimiter
from app.routes._common import get_or_404, not_found, owned_or_404, youtube_refusal
from app.routes._common import not_found, youtube_refusal
from app.youtube.client import YouTubeClient, YouTubeError
if TYPE_CHECKING:
from app.reader.models import Playlist
router = APIRouter(prefix="/api/playlists", tags=["playlists"])
# SEC-14, same reasoning as the sync module's limiter: a playlist pull lists every playlist the
@@ -56,90 +59,20 @@ def _refused(status: int, reason: str) -> HTTPException:
return HTTPException(status_code=status, detail={"code": "playlist", "reason": reason})
def _touch(pl: Playlist) -> None:
"""Bump the playlist's `updated_at` for an edit that only writes `playlist_items`.
`UpdatedAtMixin`'s `onupdate` fires on an UPDATE of the PLAYLIST row, and add / remove /
reorder / move write the item rows instead — so without this the rail's "recently changed"
sort would only ever see renames. The assignment is what makes the column mean "last time
this list changed", which is what the rail promises the reader.
The DATABASE's clock, like the mixin's own `onupdate`: one sort key written from two clocks
(this process's and Postgres's) could order a rename and an item edit the wrong way round."""
pl.updated_at = func.now()
def _own_playlist(db: Session, user: User, playlist_id: int, *, lock: bool = False) -> "Playlist":
pl = queries.own_playlist(db, user, playlist_id, lock=lock)
if pl is None:
raise not_found("playlist")
return pl
def _next_playlist_position(db: Session, user_id: int) -> int:
"""Append position for a new playlist in this user's rail."""
return (
db.scalar(
select(func.coalesce(func.max(Playlist.position), -1)).where(
Playlist.user_id == user_id
)
)
+ 1
)
def _next_item_position(db: Session, playlist_id: int) -> int:
"""Append position for a new item at the end of a playlist. Locks the playlist row first: two
adds (two tabs, the rail and the page) otherwise both read the same max and land on one
position, and the list's order between them is then arbitrary."""
lock_playlists(db, playlist_id)
return (
db.scalar(
select(func.coalesce(func.max(PlaylistItem.position), -1)).where(
PlaylistItem.playlist_id == playlist_id
)
)
+ 1
)
def _add_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) -> bool:
"""Append a video to a playlist if not already present (idempotent). Returns whether it
was added. `mark_dirty` recomputes the YouTube-sync dirty flag (skip for Watch later).
Locked BEFORE the presence check: two adds of one video both saw "absent" and the second one's
insert hit uq_playlist_video as a 500."""
lock_playlists(db, pl.id)
exists = db.scalar(
select(PlaylistItem).where(
PlaylistItem.playlist_id == pl.id, PlaylistItem.video_id == video_id
)
)
if exists is not None:
return False
db.add(
PlaylistItem(playlist_id=pl.id, video_id=video_id, position=_next_item_position(db, pl.id))
)
if mark_dirty:
recompute_dirty(db, pl)
_touch(pl)
db.flush() # the handler commits (CLAUDE.md: helpers flush)
return True
def _remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) -> bool:
"""Remove a video from a playlist if present (idempotent). Returns whether it was removed.
`mark_dirty` recomputes the YouTube-sync dirty flag (skip for Watch later)."""
lock_playlists(db, pl.id)
item = db.scalar(
select(PlaylistItem).where(
PlaylistItem.playlist_id == pl.id, PlaylistItem.video_id == video_id
)
)
if item is None:
return False
db.delete(item)
if mark_dirty:
recompute_dirty(db, pl)
_touch(pl)
db.flush() # the handler commits (CLAUDE.md: helpers flush)
return True
def _video_or_404(db: Session, video_id: str) -> None:
if queries.video(db, video_id) is None:
raise not_found("video")
def _summary(
pl: Playlist,
pl: "Playlist",
count: int,
total_duration: int = 0,
has_video: bool | None = None,
@@ -156,7 +89,7 @@ def _summary(
"total_duration_seconds": total_duration,
"cover_thumbnail": cover,
# When this list last CHANGED (rename, add/remove/move/reorder) — the rail's
# "recently changed first" sort reads it. See _touch for why it isn't automatic.
# "recently changed first" sort reads it. See `service.touch` for why it isn't automatic.
"updated_at": pl.updated_at,
}
if has_video is not None:
@@ -172,73 +105,14 @@ def list_playlists(
) -> list[dict]:
"""The user's playlists (newest custom order). With `contains=<video_id>`, each entry
also reports whether that video is already in it — used by the add-to-playlist popover."""
pls = (
db.execute(
select(Playlist)
.where(Playlist.user_id == user.id)
.order_by(Playlist.position, Playlist.created_at)
)
.scalars()
.all()
)
pls = queries.playlists_for(db, user)
if not pls:
return []
ids = [p.id for p in pls]
counts = dict(
db.execute(
select(PlaylistItem.playlist_id, func.count())
.where(PlaylistItem.playlist_id.in_(ids))
.group_by(PlaylistItem.playlist_id)
).all()
)
durations = dict(
db.execute(
select(
PlaylistItem.playlist_id,
func.coalesce(func.sum(Video.duration_seconds), 0),
)
.join(Video, Video.id == PlaylistItem.video_id)
.where(PlaylistItem.playlist_id.in_(ids))
.group_by(PlaylistItem.playlist_id)
).all()
)
member: set[int] = set()
if contains:
member = set(
db.execute(
select(PlaylistItem.playlist_id).where(
PlaylistItem.playlist_id.in_(ids),
PlaylistItem.video_id == contains,
)
)
.scalars()
.all()
)
# Batch the cover thumbnails (first item per playlist by position) in ONE window query
# instead of a per-playlist SELECT inside _summary (the old N+1).
rn = (
func.row_number()
.over(
partition_by=PlaylistItem.playlist_id,
order_by=PlaylistItem.position,
)
.label("rn")
)
first_items = (
select(
PlaylistItem.playlist_id.label("pid"),
Video.thumbnail_url.label("thumb"),
rn,
)
.join(Video, Video.id == PlaylistItem.video_id)
.where(PlaylistItem.playlist_id.in_(ids))
.subquery()
)
covers = dict(
db.execute(
select(first_items.c.pid, first_items.c.thumb).where(first_items.c.rn == 1)
).all()
)
counts = queries.item_counts(db, ids)
durations = queries.durations(db, ids)
member = queries.containing(db, ids, contains) if contains else set()
covers = queries.covers(db, ids)
return [
_summary(
p,
@@ -269,29 +143,11 @@ def create_playlist(
name = body.name.strip()
if not name:
raise _refused(400, "name_required")
pl = Playlist(user_id=user.id, name=name, position=_next_playlist_position(db, user.id))
db.add(pl)
pl = service.create_playlist(db, user, name)
db.commit()
return _summary(pl, 0)
def _watch_later(db: Session, user: User) -> Playlist:
"""The user's built-in Watch later playlist, created on first use."""
pl = db.scalar(
select(Playlist).where(Playlist.user_id == user.id, Playlist.kind == "watch_later")
)
if pl is None:
pl = Playlist(
user_id=user.id,
name="Watch later",
kind="watch_later",
position=_next_playlist_position(db, user.id),
)
db.add(pl)
db.flush() # the id, for the item the caller adds next; the handler commits
return pl
@router.post("/watch-later")
def add_watch_later(
body: VideoRefIn,
@@ -300,9 +156,9 @@ def add_watch_later(
) -> dict:
"""Bookmark shortcut: add a video to the built-in Watch later playlist."""
video_id = body.video_id
get_or_404(db, Video, video_id, what="video")
pl = _watch_later(db, user)
_add_item(db, pl, video_id, mark_dirty=False)
_video_or_404(db, video_id)
pl = service.watch_later(db, user)
service.add_item(db, pl, video_id, mark_dirty=False)
db.commit()
return {"ok": True, "saved": True, "playlist_id": pl.id}
@@ -313,11 +169,9 @@ def remove_watch_later(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = db.scalar(
select(Playlist).where(Playlist.user_id == user.id, Playlist.kind == "watch_later")
)
pl = queries.watch_later(db, user)
if pl is not None:
_remove_item(db, pl, video_id, mark_dirty=False)
service.remove_item(db, pl, video_id, mark_dirty=False)
db.commit()
return {"ok": True, "saved": False}
@@ -354,7 +208,7 @@ def revert_youtube(
"""Reset one YouTube-linked playlist to its current YouTube state, discarding local
edits (and clearing the dirty flag). The user-facing 'undo all my changes' for a list
once the in-session undo history is gone."""
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
pl = _own_playlist(db, user, playlist_id)
if not pl.yt_playlist_id:
raise _refused(400, "not_on_youtube")
if not has_read_scope(user):
@@ -375,10 +229,10 @@ def revert_youtube(
raise youtube_refusal(exc, playlist=True) from exc
def _pushable(db: Session, user: User, playlist_id: int) -> Playlist:
def _pushable(db: Session, user: User, playlist_id: int) -> "Playlist":
"""A local playlist the user may push to YouTube (create or update). The built-in
Watch later and read-only YouTube mirrors are never pushable."""
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
pl = _own_playlist(db, user, playlist_id)
if pl.kind == "watch_later":
raise _refused(400, "watch_later_not_pushable")
return pl
@@ -448,17 +302,8 @@ def get_playlist(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
state = aliased(VideoState)
rows = db.execute(
select(*feed_columns(user, state))
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(state, and_(state.video_id == Video.id, state.user_id == user.id))
.where(PlaylistItem.playlist_id == pl.id)
# `id` breaks a position tie the same way on every read (the renumbering add does too).
.order_by(PlaylistItem.position, PlaylistItem.id)
).all()
pl = _own_playlist(db, user, playlist_id)
rows = queries.playlist_rows(db, user, pl.id)
return {
"id": pl.id,
"name": pl.name,
@@ -482,34 +327,15 @@ def rename_playlist(
db: Session = Depends(get_db),
) -> dict:
# Locked: the new name moves `dirty`, which the scheduled mirror refresh reads under this lock.
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist", lock=True)
pl = _own_playlist(db, user, playlist_id, lock=True)
if "name" in body.model_fields_set:
name = (body.name or "").strip()
if not name:
raise _refused(400, "name_required")
if name != pl.name:
pl.name = name
recompute_dirty(db, pl)
service.rename_playlist(db, pl, name)
db.commit()
# One aggregate for count + duration (outer join so an empty playlist still returns 0),
# plus a single cover lookup — was 3 separate round-trips.
count, total = db.execute(
select(
func.count(PlaylistItem.id),
func.coalesce(func.sum(Video.duration_seconds), 0),
)
.select_from(PlaylistItem)
.outerjoin(Video, Video.id == PlaylistItem.video_id)
.where(PlaylistItem.playlist_id == pl.id)
).one()
cover = db.scalar(
select(Video.thumbnail_url)
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.where(PlaylistItem.playlist_id == pl.id)
.order_by(PlaylistItem.position)
.limit(1)
)
return _summary(pl, count or 0, total or 0, cover=cover)
count, total, cover = queries.playlist_totals(db, pl.id)
return _summary(pl, count, total, cover=cover)
@router.delete("/{playlist_id}")
@@ -519,7 +345,7 @@ def delete_playlist(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
pl = _own_playlist(db, user, playlist_id)
if pl.kind == "watch_later":
raise _refused(400, "watch_later_undeletable")
# Not in the middle of a sync of it (`sync_claim`): a push that read the list before the delete
@@ -540,9 +366,7 @@ def delete_playlist(
yt.delete_playlist(pl.yt_playlist_id)
except YouTubeError as exc:
raise youtube_refusal(exc, playlist=True) from exc
# The playlist first, as every item write: the cascade deletes the items before the list row.
lock_playlists(db, pl.id)
db.delete(pl) # items cascade
service.delete_playlist(db, pl)
db.commit()
return {"deleted": playlist_id}
@@ -561,26 +385,10 @@ def add_item(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
pl = _own_playlist(db, user, playlist_id)
video_id = body.video_id
get_or_404(db, Video, video_id, what="video")
added = _add_item(db, pl, video_id, mark_dirty=False)
if added and body.position is not None:
# Renumber the whole list with the new row at `position` (clamped to the end): positions
# can have gaps after removals, so "index N" is only meaningful over the ordered rows.
rows = list(
db.execute(
select(PlaylistItem)
.where(PlaylistItem.playlist_id == pl.id)
.order_by(PlaylistItem.position, PlaylistItem.id)
).scalars()
)
new = next(r for r in rows if r.video_id == video_id)
rows.remove(new)
rows.insert(min(body.position, len(rows)), new)
for pos, row in enumerate(rows):
row.position = pos
recompute_dirty(db, pl)
_video_or_404(db, video_id)
service.add_to_playlist(db, pl, video_id, body.position)
db.commit()
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
@@ -592,8 +400,8 @@ def remove_item(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
_remove_item(db, pl, video_id, mark_dirty=True)
pl = _own_playlist(db, user, playlist_id)
service.remove_item(db, pl, video_id, mark_dirty=True)
db.commit()
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
@@ -617,49 +425,21 @@ def move_item(
exactly the kind of half-applied state this codebase treats as a bug rather than an edge
case. Idempotent: if the video is already in the target, the source row is simply dropped
(moving into a list that already has it is a removal, not a duplicate)."""
src = owned_or_404(db, Playlist, playlist_id, user, what="playlist")
src = _own_playlist(db, user, playlist_id)
# Its own reason: "this playlist doesn't exist any more" about a TARGET that is gone was read as
# the list on screen — which is still there.
dst = owned_or_404(db, Playlist, body.target_playlist_id, user, what="move_target")
if src.id == dst.id:
dst = queries.own_playlist(db, user, body.target_playlist_id)
if dst is None:
raise not_found("move_target")
try:
service.move_item(db, src, dst, video_id)
except service.MoveRefused as e:
# A same-list move is a bad request; a video a concurrent edit (another tab, a YouTube
# re-pull) already took out is gone.
status = 400 if e.reason == "same_playlist" else 404
raise HTTPException(
status_code=400, detail={"code": "playlist_move", "reason": "same_playlist"}
)
# Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise.
lock_playlists(db, src.id, dst.id)
item = db.scalar(
select(PlaylistItem).where(
PlaylistItem.playlist_id == src.id, PlaylistItem.video_id == video_id
)
)
if item is None:
# A concurrent edit (another tab, or a YouTube re-pull) already took it out.
raise HTTPException(
status_code=404, detail={"code": "playlist_move", "reason": "not_in_playlist"}
)
already = db.scalar(
select(PlaylistItem).where(
PlaylistItem.playlist_id == dst.id, PlaylistItem.video_id == video_id
)
)
dst_changed = False
if already is None:
# Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
# survives the move, and there is no window where the video belongs to neither list. The
# check above holds: both lists are locked, and an insert into the target (another tab's
# add, a YouTube merge) waits for the lock — its foreign key needs the target row.
item.playlist_id = dst.id
item.position = _next_item_position(db, dst.id)
dst_changed = True
if not dst_changed:
db.delete(item)
# The source always changed; the target only when the row landed in it. `_touch` on a target
# that already had the video promoted an untouched list in the rail's "recently changed" sort.
recompute_dirty(db, src)
_touch(src)
if dst_changed:
recompute_dirty(db, dst)
_touch(dst)
status_code=status, detail={"code": "playlist_move", "reason": e.reason}
) from None
db.commit()
return {"ok": True, "from_playlist_id": src.id, "to_playlist_id": dst.id, "video_id": video_id}
@@ -676,30 +456,7 @@ def reorder_items(
db: Session = Depends(get_db),
) -> dict:
# Locked: a concurrent add between reading the items and writing the order would collide.
pl = owned_or_404(db, Playlist, playlist_id, user, what="playlist", lock=True)
order = body.video_ids
items = {
it.video_id: it
for it in db.execute(
select(PlaylistItem).where(PlaylistItem.playlist_id == pl.id)
).scalars()
}
pos = 0
seen: set[str] = set()
for vid in order:
it = items.get(vid)
if it is not None:
it.position = pos
pos += 1
seen.add(vid)
# Any items the payload omitted (e.g. added concurrently in another tab) keep their relative
# order but get fresh trailing positions, so no two items collide on the same position.
for it in sorted(
(it for vid, it in items.items() if vid not in seen), key=lambda i: i.position
):
it.position = pos
pos += 1
recompute_dirty(db, pl)
_touch(pl)
pl = _own_playlist(db, user, playlist_id, lock=True)
pos = service.reorder_items(db, pl, body.video_ids)
db.commit()
return {"ok": True, "playlist_id": pl.id, "count": pos}
+8 -4
View File
@@ -1,18 +1,22 @@
"""Shared helpers, serializers and constants for the plex route package."""
from typing import TYPE_CHECKING
from sqlalchemy.orm import Session
from app import logs
from app.models import PlexItem
from app.plex import queries
from app.plex.cards import norm_lang as _norm_lang # noqa: F401 — the package's historic name
from app.plex.lookups import by_rating_key
from app.routes._common import not_found
if TYPE_CHECKING:
from app.plex.models import PlexItem
log = logs.get_logger(__name__)
def item_or_404(db: Session, rating_key: str) -> PlexItem:
it = by_rating_key(db, PlexItem, rating_key)
def item_or_404(db: Session, rating_key: str) -> "PlexItem":
it = queries.item(db, rating_key)
if it is None:
raise not_found("plex_item")
return it
+11 -17
View File
@@ -1,27 +1,26 @@
"""Plex admin: connection test, catalog sync, and owner watch-link config."""
from datetime import datetime, timezone
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.auth import admin_user, current_user
from app.db import get_db
from app.models import (
PlexItem,
PlexLink,
User,
)
from app.models import User
from app.plex import paths as plex_paths
from app.plex import queries, service
from app.plex import sync as plex_sync
from app.plex import watch_sync as plex_watch
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.utils import iso
from app.refusal import refused
if TYPE_CHECKING:
from app.plex.models import PlexLink
log = logs.get_logger(__name__)
router = APIRouter(prefix="/api/plex", tags=["plex"])
@@ -69,7 +68,7 @@ def _sample_plex_path(db: Session, plex: PlexClient, sections: list[dict]) -> st
"""A real Plex-side media file path to validate the mount + path-map against. Prefers the
already-mirrored catalog (cheap); before the first sync, pulls one movie leaf from an enabled
section straight from Plex."""
fp = db.scalar(select(PlexItem.file_path).where(PlexItem.file_path.isnot(None)))
fp = queries.sample_file_path(db)
if fp:
return fp
wanted = plex_sync._enabled_section_keys(db)
@@ -106,7 +105,7 @@ def trigger_sync(_: User = Depends(admin_user), db: Session = Depends(get_db)) -
return {"ok": True, **plex_sync.sync(db)}
def _link_dict(link: PlexLink | None) -> dict:
def _link_dict(link: "PlexLink | None") -> dict:
return {
"linked": link is not None,
"sync_enabled": bool(link and link.sync_enabled),
@@ -119,8 +118,7 @@ def _link_dict(link: PlexLink | None) -> dict:
@router.get("/watch/link")
def watch_link_status(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
"""This user's Plex watch-sync link status (drives the Settings toggle)."""
link = db.execute(select(PlexLink).where(PlexLink.user_id == user.id)).scalars().first()
return _link_dict(link)
return _link_dict(queries.watch_link(db, user))
class WatchLinkIn(BaseModel):
@@ -139,11 +137,7 @@ def watch_link_set(
if not sysconfig.get_bool(db, "plex_enabled"):
raise refused(409, "plex_disabled")
enabled = body.enabled
link = db.execute(select(PlexLink).where(PlexLink.user_id == user.id)).scalars().first()
if link is None:
link = PlexLink(user_id=user.id, uses_admin=True, linked_at=datetime.now(timezone.utc))
db.add(link)
link.sync_enabled = enabled
link = service.set_watch_link(db, user, enabled)
db.commit()
result: dict = {}
@@ -156,7 +150,7 @@ def watch_link_set(
def watch_import_now(user: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
"""Admin: (re-)run the "Plex is master" import for this account — Plex re-wins on the
intersection; Siftlode-only states are preserved. Idempotent."""
link = db.execute(select(PlexLink).where(PlexLink.user_id == user.id)).scalars().first()
link = queries.watch_link(db, user)
if link is None or not link.sync_enabled:
raise refused(409, "watch_sync_off")
result = plex_watch.import_owner_watch_state(db, link)
+63 -509
View File
@@ -1,110 +1,30 @@
"""Plex browse: the library list, the unified cross-library browse feed, and its faceted filters."""
from fastapi import APIRouter, Depends, Query
from sqlalchemy import (
Boolean,
Integer,
String,
and_,
cast,
func,
literal,
null,
or_,
select,
)
from sqlalchemy.orm import Session, aliased
from sqlalchemy.orm import Session
from app.mailtext import lang_for_user
from app.collation import collated, sorted_words
from app import sysconfig
from app.auth import current_user
from app.db import get_db
from app.models import (
PlexItem,
PlexLibrary,
PlexShow,
PlexState,
User,
)
from app.plex import rollup
from app.mailtext import lang_for_user
from app.models import User
from app.plex import library as plex_library
from app.plex import queries
from app.plex.cards import display_title, episode_card
from app.plex.query import (
FIELD_COLUMNS,
PLAYBACK_CLASSES,
TS_CONFIG,
LibraryFilters,
apply_meta_filters,
apply_movie_state_filter,
apply_show_state_filter,
content_libs,
field_recheck,
genres_col,
meta_filter_conds,
movie_status_case,
playable_conds,
playback_class_expr,
show_status_case,
split_csv,
summary_col,
title_col,
vector_col,
vector_match,
)
from app.plex.query import LibraryFilters
from app.routes.plex._shared import _norm_lang
from app.utils import to_tsquery_str
router = APIRouter(prefix="/api/plex", tags=["plex"])
def sort_column(u, sort: str, lang: str = "en"):
"""The browse grid's sort key -> orderable column on the union subquery. Module-level (not inline
in the handler) so a test can pin its vocabulary to `PLEX_SORT_KEYS` — a key the Sort control
offers but this cannot order would quietly fall back to `added_at` and look like a broken control.
An unknown key (a stale bookmark, a hand-edited URL) deliberately keeps that fallback. The title sorts
in the reader's alphabet (`collation`)."""
return {
"added": u.c.added_at,
"title": collated(u.c.title, lang),
"year": u.c.year,
"rating": u.c.rating,
"duration": u.c.duration_s,
"release": u.c.rel,
"lastwatched": u.c.last_watched,
}.get(sort, u.c.added_at)
@router.get("/libraries")
def list_libraries(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
"""The mirrored libraries the user can browse (drives the Plex scope selector). HU overlay
libraries are excluded — they're not browsable, they only feed the EN twin's Hungarian metadata."""
libs = (
db.execute(
select(PlexLibrary)
.where(PlexLibrary.overlay_of.is_(None))
.order_by(PlexLibrary.kind.desc(), collated(PlexLibrary.title, lang_for_user(user)))
)
.scalars()
.all()
)
out = []
for lib in libs:
if lib.kind == "movie":
count = db.scalar(
select(func.count(PlexItem.id)).where(
PlexItem.library_id == lib.id, PlexItem.kind == "movie"
)
)
else:
count = db.scalar(select(func.count(PlexShow.id)).where(PlexShow.library_id == lib.id))
out.append(
{
"key": lib.plex_key,
"title": lib.title,
"kind": lib.kind,
"count": count or 0,
}
)
out = [
{"key": lib.plex_key, "title": lib.title, "kind": lib.kind, "count": count}
for lib, count in queries.browsable_libraries(db, lang_for_user(user))
]
return {"enabled": sysconfig.get_bool(db, "plex_enabled"), "libraries": out}
@@ -129,193 +49,60 @@ def unified_library(
episodes come back in a separate `episodes` list (grouped result — the 'Episodes' section)."""
offset = max(0, offset)
lang = _norm_lang(lang)
p = f.as_dict()
movie_lib_ids, show_lib_ids = content_libs(db)
want_movies = scope in ("movie", "both") and bool(movie_lib_ids)
want_shows = scope in ("show", "both") and bool(show_lib_ids)
ts = to_tsquery_str(q) if q else None
tsq = func.to_tsquery(TS_CONFIG, ts) if ts else None
if field not in FIELD_COLUMNS and field != "all":
field = "all"
# `episode` scope targets episode rows only — the movie/show grid stays empty; the recheck for
# the specific title/people/summary fields is a strict narrowing on top of the GIN prefilter.
grid_field = field != "episode"
recheck = field_recheck if tsq is not None and field not in ("all", "episode") else None
selects = []
if want_movies and grid_field:
st = aliased(PlexState)
m_status = movie_status_case(st)
mq = (
select(
PlexItem.rating_key.label("rk"),
literal("movie").label("kind"),
title_col(PlexItem, lang).label("title"),
PlexItem.year.label("year"),
PlexItem.rating.label("rating"),
# Truncated so the card hover-overview costs a few hundred bytes, not the full synopsis.
func.left(summary_col(PlexItem, lang), 300).label("summary"),
PlexItem.added_at.label("added_at"),
PlexItem.originally_available_at.label("rel"),
PlexItem.duration_s.label("duration_s"),
PlexItem.playable.label("playable"),
PlexItem.transcode_hw.label("transcode_hw"),
cast(null(), Integer).label("season_count"),
func.coalesce(st.position_seconds, 0).label("position_seconds"),
m_status.label("status"),
literal(False).label("resumable"),
# "last watched" sort key: when the movie was last resumed, else when it was marked
# watched. NULL for never-touched movies → nullslast sinks them (see the sort dict).
func.coalesce(st.progress_updated_at, st.watched_at).label("last_watched"),
(
func.ts_rank(vector_col(PlexItem, lang), tsq)
if tsq is not None
else literal(0.0)
).label("rank"),
)
.outerjoin(st, and_(st.item_id == PlexItem.id, st.user_id == user.id))
.where(PlexItem.kind == "movie", PlexItem.library_id.in_(movie_lib_ids))
)
mq = apply_movie_state_filter(mq, st, show)
mq = apply_meta_filters(mq, PlexItem, p, lang)
if f.duration_min is not None:
mq = mq.where(PlexItem.duration_s >= f.duration_min)
if f.duration_max is not None:
mq = mq.where(PlexItem.duration_s <= f.duration_max)
# Playback-class filter (movie-only — a show is an aggregate with no single class).
mq = mq.where(*playable_conds(split_csv(f.playable)))
if tsq is not None:
mq = mq.where(vector_match(PlexItem, lang, tsq))
if recheck is not None:
mq = mq.where(recheck(PlexItem, field, tsq, lang))
selects.append(mq)
if want_shows and grid_field:
agg = rollup.agg_subquery(db, user.id, include_resumed=True).subquery()
tot = func.coalesce(agg.c.total, 0)
wat = func.coalesce(agg.c.watched, 0)
inp = func.coalesce(agg.c.inprog, 0)
s_status = show_status_case(tot, wat, inp)
sq = (
select(
PlexShow.rating_key.label("rk"),
literal("show").label("kind"),
title_col(PlexShow, lang).label("title"),
PlexShow.year.label("year"),
PlexShow.rating.label("rating"),
func.left(summary_col(PlexShow, lang), 300).label("summary"),
PlexShow.added_at.label("added_at"),
PlexShow.originally_available_at.label("rel"),
cast(null(), Integer).label("duration_s"),
cast(null(), String).label("playable"),
cast(null(), Boolean).label("transcode_hw"),
PlexShow.child_count.label("season_count"),
literal(0).label("position_seconds"),
s_status.label("status"),
# An episode holds a resume point (`ShowState.resumable`, the card's Continue + Restart).
(inp > 0).label("resumable"),
# Show leg's "last watched": max(resume/watched time) across the show's episodes (see
# rollup.agg_subquery include_resumed). NULL for a never-watched show → nullslast.
agg.c.resumed_at.label("last_watched"),
(
func.ts_rank(vector_col(PlexShow, lang), tsq)
if tsq is not None
else literal(0.0)
).label("rank"),
)
.outerjoin(agg, agg.c.show_id == PlexShow.id)
.where(PlexShow.library_id.in_(show_lib_ids))
)
sq = apply_show_state_filter(sq, tot, wat, inp, show)
sq = apply_meta_filters(sq, PlexShow, p, lang)
if tsq is not None:
sq = sq.where(vector_match(PlexShow, lang, tsq))
if recheck is not None:
sq = sq.where(recheck(PlexShow, field, tsq, lang))
selects.append(sq)
total, rows, eps = plex_library.library_page(
db,
user,
scope=scope,
q=q,
field=field,
sort=sort,
sort_dir=sort_dir,
show=show,
lang=lang,
offset=offset,
limit=limit,
f=f,
)
items = []
total = 0
if selects:
union = selects[0] if len(selects) == 1 else selects[0].union_all(*selects[1:])
u = union.subquery()
total = db.scalar(select(func.count()).select_from(u)) or 0
if tsq is not None:
order = [u.c.rank.desc()]
else:
col = sort_column(u, sort, lang)
order = [(col.asc() if sort_dir == "asc" else col.desc()).nullslast()]
order.append(u.c.rk.desc())
rows = db.execute(select(u).order_by(*order).offset(offset).limit(limit)).all()
for r in rows:
if r.kind == "movie":
items.append(
{
"id": r.rk,
"type": "movie",
"title": r.title,
"year": r.year,
"rating": r.rating,
"summary": r.summary,
"duration_seconds": r.duration_s,
"thumb": f"/api/plex/image/{r.rk}",
"playable": r.playable,
"transcode_hw": r.transcode_hw,
"status": r.status,
"position_seconds": r.position_seconds,
}
)
else:
items.append(
{
"id": r.rk,
"type": "show",
"title": r.title,
"year": r.year,
"rating": r.rating,
"summary": r.summary,
"thumb": f"/api/plex/image/{r.rk}",
"season_count": r.season_count,
"status": r.status,
"resumable": r.resumable,
}
)
# Episode "grouped result" section: shown for Everywhere (full-vector match, today's behaviour)
# and for the Episode scope (title-only match, empty item grid). The title/people/summary scopes
# target the movie/show grid, so they suppress episodes.
episodes = []
if tsq is not None and want_shows and field in ("all", "episode"):
ep_st = aliased(PlexState)
# In HU an episode is found by its Hungarian title too (the overlay fills it) — and
# still by its English one, like `vector_match` does for the grid.
if field == "episode":
ep_vec = func.to_tsvector(TS_CONFIG, func.coalesce(PlexItem.title, ""))
ep_vec_hu = func.to_tsvector(TS_CONFIG, func.coalesce(PlexItem.title_hu, ""))
else:
ep_vec, ep_vec_hu = PlexItem.search_vector, PlexItem.search_vector_hu
hit = ep_vec.op("@@")(tsq)
if lang == "hu":
hit = or_(ep_vec_hu.op("@@")(tsq), hit)
ep_vec = ep_vec_hu
eps = db.execute(
select(PlexItem, ep_st, PlexShow)
.outerjoin(ep_st, and_(ep_st.item_id == PlexItem.id, ep_st.user_id == user.id))
.outerjoin(PlexShow, PlexShow.id == PlexItem.show_id)
.where(
PlexItem.kind == "episode",
PlexItem.library_id.in_(show_lib_ids),
hit,
for r in rows:
if r.kind == "movie":
items.append(
{
"id": r.rk,
"type": "movie",
"title": r.title,
"year": r.year,
"rating": r.rating,
"summary": r.summary,
"duration_seconds": r.duration_s,
"thumb": f"/api/plex/image/{r.rk}",
"playable": r.playable,
"transcode_hw": r.transcode_hw,
"status": r.status,
"position_seconds": r.position_seconds,
}
)
.order_by(func.ts_rank(ep_vec, tsq).desc())
.limit(24)
).all()
for e, s, sh in eps:
card = episode_card(e, s, lang=lang)
card["show_title"] = display_title(sh, lang) if sh is not None else None
episodes.append(card)
else:
items.append(
{
"id": r.rk,
"type": "show",
"title": r.title,
"year": r.year,
"rating": r.rating,
"summary": r.summary,
"thumb": f"/api/plex/image/{r.rk}",
"season_count": r.season_count,
"status": r.status,
"resumable": r.resumable,
}
)
episodes = []
for e, s, sh in eps:
card = episode_card(e, s, lang=lang)
card["show_title"] = display_title(sh, lang) if sh is not None else None
episodes.append(card)
return {
"scope": scope,
@@ -341,237 +128,4 @@ def facets(
filters applied but NOT its own — so selecting a filter narrows what the other groups offer, while
a multi-select group still lets you add more. Merged across the scope's libraries; duration is
movie-only. (Search text is not factored in — facets track the sidebar filters.)"""
empty = {
"genres": [],
"content_ratings": [],
"studios": [],
"playable": [],
"year_min": None,
"year_max": None,
"rating_max": None,
"duration_min": None,
"duration_max": None,
}
all_movie_ids, all_show_ids = content_libs(db)
movie_ids = all_movie_ids if scope in ("movie", "both") else []
show_ids = all_show_ids if scope in ("show", "both") else []
if not movie_ids and not show_ids:
return empty
lang = _norm_lang(lang)
p = f.as_dict()
uid = user.id
# The shows that pass the watch-state scope, asked ONCE. The per-user episode rollup behind it is a
# full pass over the user's episodes (~10 ms on a prod-sized library), and each of the show groups
# (genres, ratings, studios, years, rating max) asked it again: half of a filtered facets call.
# The same filter on the same rows, so the counts do not change
# (test_plex_facets_grid_parity).
shows_in_scope: list[int] | None = None
if show_ids and show in ("watched", "in_progress", "unwatched"):
agg = rollup.agg_subquery(db, uid).subquery()
in_scope = select(PlexShow.id).outerjoin(agg, agg.c.show_id == PlexShow.id)
in_scope = apply_show_state_filter(
in_scope,
func.coalesce(agg.c.total, 0),
func.coalesce(agg.c.watched, 0),
func.coalesce(agg.c.inprog, 0),
show,
)
shows_in_scope = list(db.scalars(in_scope.where(PlexShow.library_id.in_(show_ids))))
def wsq(query, model, ids, pp, kind_movie, with_dur):
"""Apply the (self-excluded) metadata filters + library scope + the per-user watch-state to a
facet aggregation query, so the facets reflect exactly the visible result set."""
query = query.where(model.library_id.in_(ids), *meta_filter_conds(model, pp, lang))
if kind_movie:
query = query.where(model.kind == "movie")
if with_dur and pp.get("duration_min") is not None:
query = query.where(model.duration_s >= pp["duration_min"])
if with_dur and pp.get("duration_max") is not None:
query = query.where(model.duration_s <= pp["duration_max"])
# Playback-class filter (movie-only) — applied for the OTHER groups so an active class
# narrows them; the playable facet itself passes pp with playable=None (self-exclusion).
query = query.where(*playable_conds(split_csv(pp.get("playable"))))
# Join watch-state and reuse the grid's exact movie state filter (unified_library) —
# hidden movies never appear in the grid, so they must not inflate facet counts/bounds either.
st = aliased(PlexState)
query = query.outerjoin(st, and_(st.item_id == model.id, st.user_id == uid))
query = apply_movie_state_filter(query, st, show)
elif shows_in_scope is not None:
query = query.where(model.id.in_(shows_in_scope))
return query
genre_counts: dict[str, int] = {}
cr_counts: dict[str, int] = {}
studio_counts: dict[str, int] = {}
playable_counts: dict[str, int] = {}
years: list[int] = []
ratings: list[float] = []
durs: list[int] = []
def per_table(model, ids, kind_movie: bool) -> None:
if not ids:
return
# Genres — in "any"/OR mode, exclude the genre filter itself so each offered genre is a valid
# OR-addition that only widens (keep adding). In "all"/AND mode, KEEP the filter applied so we
# offer only genres that CO-OCCUR on the current result set — a non-co-occurring genre would AND
# the result down to zero, i.e. a dead-end chip. (The already-selected genres still appear, since
# every matching title carries them, so they stay togglable.)
genre_pp = p if p.get("genre_mode") == "all" else {**p, "genres": None}
# Only rows whose `genres` is a JSON array can be unnested. A genre-less item stores JSON
# `null` (SQLAlchemy's JSONB persists a Python None as JSON null, not SQL NULL), and
# jsonb_array_elements_text() raises on a scalar ("cannot extract elements from a scalar") —
# so a single metadata-poor library (e.g. one bare Shows library with no genres) 500'd the
# whole facets endpoint. Mirror the exact guard people.py already applies to the same columns.
# In HU the vocabulary comes from the active-language genre column (coalesce(genres_hu, genres)).
gcol = genres_col(model, lang)
expanded = (
wsq(
select(func.jsonb_array_elements_text(gcol).label("g")),
model,
ids,
genre_pp,
kind_movie,
True,
)
.where(func.jsonb_typeof(gcol) == "array")
.subquery()
)
for g, c in db.execute(select(expanded.c.g, func.count()).group_by(expanded.c.g)).all():
genre_counts[g] = genre_counts.get(g, 0) + c
# Content ratings — exclude its own.
crq = (
wsq(
select(model.content_rating, func.count()),
model,
ids,
{**p, "content_ratings": None},
kind_movie,
True,
)
.where(model.content_rating.isnot(None))
.group_by(model.content_rating)
)
for cr, c in db.execute(crq).all():
cr_counts[cr] = cr_counts.get(cr, 0) + c
# Studios — exclude its own (each offered studio is a valid OR-addition that only widens).
stq = (
wsq(
select(model.studio, func.count()),
model,
ids,
{**p, "studios": None},
kind_movie,
True,
)
.where(model.studio.isnot(None))
.group_by(model.studio)
)
for s, c in db.execute(stq).all():
studio_counts[s] = studio_counts.get(s, 0) + c
# Playback class (movie-only) — exclude its own so each class stays an addable OR option.
if kind_movie:
pcls = playback_class_expr()
pcq = (
wsq(
select(pcls.label("cls"), func.count()),
model,
ids,
{**p, "playable": None},
kind_movie,
True,
)
.where(pcls.isnot(None))
.group_by(pcls)
)
for cls, c in db.execute(pcq).all():
playable_counts[cls] = playable_counts.get(cls, 0) + c
# Year bounds — exclude the year filter.
by = db.execute(
wsq(
select(func.min(model.year), func.max(model.year)),
model,
ids,
{**p, "year_min": None, "year_max": None},
kind_movie,
True,
)
).first()
if by[0] is not None:
years.append(by[0])
if by[1] is not None:
years.append(by[1])
# Rating max — exclude the rating filter. Coalesce(imdb, audience) to match the score the
# rating filter and the UI badge use (IMDb preferred, audience fallback).
br = db.scalar(
wsq(
select(func.max(func.coalesce(model.imdb_rating, model.rating))),
model,
ids,
{**p, "rating_min": None},
kind_movie,
True,
)
)
if br is not None:
ratings.append(float(br))
# Duration bounds (movies) — exclude the duration filter.
if kind_movie:
bd = db.execute(
wsq(
select(func.min(model.duration_s), func.max(model.duration_s)),
model,
ids,
p,
kind_movie,
False,
)
).first()
if bd[0] is not None:
durs.append(bd[0])
if bd[1] is not None:
durs.append(bd[1])
per_table(PlexItem, movie_ids, True)
per_table(PlexShow, show_ids, False)
# Always surface the actively-selected genres (even a zero-match AND combo, where they'd otherwise
# be absent): the sidebar renders genres solely from this list and hides the whole genre section —
# chips AND the Any/All toggle — when it's empty, which would trap the user with no per-chip way to
# undo a zero-result selection. Count 0 is fine (genre chips don't display counts).
for g in split_csv(f.genres):
genre_counts.setdefault(g, 0)
# Same for the actively-selected studios, so a selected studio stays removable even at zero matches.
for s in split_csv(f.studios):
studio_counts.setdefault(s, 0)
# Keep an actively-selected playback class visible even at zero matches (the sidebar hides the whole
# section when its facet list is empty, which would trap a zero-result selection with no way to undo).
for c in split_csv(f.playable):
if c in PLAYBACK_CLASSES:
playable_counts.setdefault(c, 0)
# Names in the reader's alphabet: the genres are Hungarian words for a Hungarian reader.
alpha = {w: i for i, w in enumerate(sorted_words(db, [*genre_counts, *studio_counts], lang))}
return {
"genres": [
{"value": g, "count": c}
for g, c in sorted(genre_counts.items(), key=lambda kv: alpha[kv[0]])
],
"content_ratings": [
{"value": cr, "count": c} for cr, c in sorted(cr_counts.items(), key=lambda kv: -kv[1])
],
"studios": [
{"value": s, "count": c}
for s, c in sorted(studio_counts.items(), key=lambda kv: (-kv[1], alpha[kv[0]]))
],
# Fixed good→heavy order (not by count), so the class list reads as a severity scale.
"playable": [
{"value": c, "count": playable_counts[c]}
for c in PLAYBACK_CLASSES
if c in playable_counts
],
"year_min": min(years) if years else None,
"year_max": max(years) if years else None,
"rating_max": max(ratings) if ratings else None,
"duration_min": min(durs) if durs else None,
"duration_max": max(durs) if durs else None,
}
return plex_library.facets(db, user, scope=scope, show=show, lang=lang, f=f)
+56 -269
View File
@@ -1,96 +1,41 @@
"""Plex collections: browse mirrored collections + the admin manual-collection editor."""
from collections.abc import Callable
from datetime import date, datetime
from typing import Any, NamedTuple
from datetime import datetime
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.mailtext import lang_for_user
from app.collation import collated, sorted_words
from app.collation import sorted_words
from app import logs
from app.auth import admin_user, current_user
from app.db import get_db
from app.models import (
PlexCollection,
PlexItem,
PlexLibrary,
PlexShow,
PlexState,
User,
)
from app.plex import rollup
from app.models import User
from app.plex import queries, rollup, service
from app.plex import sync as plex_sync
from app.plex.cards import SHOW_NEW, display_title, movie_card, norm_lang, show_card
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.lookups import by_rating_key
from app.plex.query import content_libs, runtime_by_key, title_col, top_art_keys
from app.utils import like_escape
from app.plex.collections import (
DEFAULT_MEMBER_SORT,
Member,
collection_card,
collection_cert,
collection_editable,
collection_source,
sort_members,
)
from app.refusal import refused
from app.routes._common import not_found
if TYPE_CHECKING:
from app.plex.models import PlexCollection, PlexState
log = logs.get_logger(__name__)
# The most-restrictive → least ordering used to pick a collection's effective content rating from its
# members (Plex shows one cert on the collection; we don't store it, so derive it).
_CERT_RANK = {
"G": 1,
"TV-Y": 1,
"TV-Y7": 1,
"TV-G": 1,
"PG": 2,
"TV-PG": 2,
"PG-13": 3,
"TV-14": 3,
"R": 4,
"TV-MA": 4,
"NC-17": 5,
}
def _collection_cert(certs: list[str | None]) -> str | None:
ranked = [(c, _CERT_RANK.get(c, 0)) for c in certs if c]
return max(ranked, key=lambda x: x[1])[0] if ranked else None
router = APIRouter(prefix="/api/plex", tags=["plex"])
_STRIP_PROVIDERS = ("imdb", "tmdb", "tvdb", "trakt")
def _collection_source(col: PlexCollection) -> str:
first = (col.title or "").strip().split(" ", 1)[0].lower()
if first in _STRIP_PROVIDERS:
return first
if col.smart:
return "smart"
return "collection"
def _collection_editable(c: PlexCollection) -> bool:
"""Effective editability: the admin marked it editable AND it's a plain manual collection — never
smart, never an external auto-list (IMDb/TMDb/…), which are managed outside Plex."""
return bool(c.editable) and not c.smart and _collection_source(c) == "collection"
def _collection_card(c: PlexCollection) -> dict:
return {
"id": c.rating_key,
"title": c.title,
"summary": c.summary,
"thumb": f"/api/plex/image/{c.rating_key}" if c.thumb_key else None,
"child_count": c.child_count,
"smart": c.smart,
"source": _collection_source(c),
"editable": c.editable,
"can_edit": _collection_editable(
c
), # whether the edit controls should be offered (admin UI)
}
@router.get("/collections")
def collections(
@@ -110,64 +55,22 @@ def collections(
sidebar token and the applied-filter chip had nothing to render but the raw id. Resolving them
through this endpoint costs one small row each; pulling the whole list to find two titles is
~200 KB."""
if library is not None:
lib = (
db.execute(select(PlexLibrary).where(PlexLibrary.plex_key == str(library)))
.scalars()
.first()
)
if lib is None:
return {"collections": []}
query = select(PlexCollection).where(PlexCollection.library_id == lib.id)
else:
enabled_ids = list(
db.execute(
select(PlexLibrary.id).where(
PlexLibrary.enabled.is_(True), PlexLibrary.overlay_of.is_(None)
)
).scalars()
)
if not enabled_ids:
return {"collections": []}
query = select(PlexCollection).where(PlexCollection.library_id.in_(enabled_ids))
term = (q or "").strip()
if term:
query = query.where(PlexCollection.title.ilike(f"%{like_escape(term)}%"))
if keys is not None:
wanted = [k for k in (keys or "").split(",") if k]
# An explicit but empty `keys=` asks for nothing — answer nothing, rather than falling through
# to "every collection" (which is what an unguarded empty IN () would have to become).
if not wanted:
return {"collections": []}
query = query.where(PlexCollection.rating_key.in_(wanted))
rows = (
db.execute(
query.order_by(
PlexCollection.child_count.desc().nullslast(),
collated(PlexCollection.title, lang_for_user(user)),
)
)
.scalars()
.all()
)
wanted = [k for k in keys.split(",") if k] if keys is not None else None
rows = queries.collections_page(db, library, q, wanted, lang_for_user(user))
# A montage of the top titles per tile + the collection's total runtime. Art unions MOVIE and SHOW
# members (a collection lives in one library, so exactly one side has art) — a show collection would
# otherwise render blank; the collection's own poster is the next fallback, then the colour hash.
# Runtime is movie-only (shows have no single duration).
movie_ids, show_ids = content_libs(db)
keys = [c.rating_key for c in rows]
art_m = top_art_keys(db, PlexItem, movie_ids, PlexItem.collection_keys, keys, per=2)
art_s = top_art_keys(db, PlexShow, show_ids, PlexShow.collection_keys, keys, per=2)
runtime = runtime_by_key(db, PlexItem, movie_ids, PlexItem.collection_keys, keys)
art_m, art_s, runtime = queries.collection_tiles(db, [c.rating_key for c in rows])
def _art(c: PlexCollection) -> list[str]:
def _art(c: "PlexCollection") -> list[str]:
member = art_m.get(c.rating_key) or art_s.get(c.rating_key)
return member or ([c.rating_key] if c.thumb_key else [])
return {
"collections": [
{
**_collection_card(c),
**collection_card(c),
"art": _art(c),
"total_runtime_s": runtime.get(c.rating_key, 0),
}
@@ -185,89 +88,21 @@ def collections_for_item(
"""What a movie or show card's "Add to collection" lists: the editable collections of the title's
library (a collection lives in one library, and only a manual one can take a title — the same
`can_edit` the editor offers), each saying whether the title is in it already, by name."""
movie = by_rating_key(db, PlexItem, item_rating_key)
title: PlexItem | PlexShow | None = (
movie if movie is not None and movie.kind == "movie" else None
) or by_rating_key(db, PlexShow, item_rating_key)
movie = queries.movie(db, item_rating_key)
title = movie or queries.show(db, item_rating_key)
if title is None:
raise not_found("item")
member_of = set(title.collection_keys or [])
rows = (
db.execute(
select(PlexCollection)
.where(PlexCollection.library_id == title.library_id)
.order_by(collated(PlexCollection.title, lang_for_user(user)))
)
.scalars()
.all()
)
rows = queries.library_collections(db, title.library_id, lang_for_user(user))
return {
"collections": [
{**_collection_card(c), "contains": c.rating_key in member_of}
{**collection_card(c), "contains": c.rating_key in member_of}
for c in rows
if _collection_editable(c)
if collection_editable(c)
]
}
class Member(NamedTuple):
"""One collection member flattened to everything the Sort control can order by. The page's two
legs are different tables (movies and shows), so there is no single query to ORDER BY — the
ordering is a pure list sort over these, which is also what makes it unit-testable."""
kind: str # "movie" | "show"
obj: Any # PlexItem | PlexShow, carried through to the card serializer
title: str # lower-cased; never NULL (the column is not nullable)
year: int | None
rating: float | None
added: datetime | None
release: date | None
duration: int | None # movies only — a show has no single runtime
last_watched: datetime | None
# Sort key -> what to read off a Member. Pinned to `PLEX_SORT_KEYS` by a test: the collection page
# shows the SAME Sort control as the browse grid, so a key this map is missing would be an option
# that silently does nothing here.
MEMBER_SORT_KEYS: dict[str, Callable[[Member], Any]] = {
"added": lambda m: m.added,
"lastwatched": lambda m: m.last_watched,
"release": lambda m: m.release,
"year": lambda m: m.year,
"rating": lambda m: m.rating,
"duration": lambda m: m.duration,
"title": lambda m: m.title,
}
# What the page ordered by before it had a control, and still its default: a film series wants to be
# read in release order, not by whenever Plex happened to ingest it.
DEFAULT_MEMBER_SORT = "year"
def sort_members(
members: list[Member], sort: str, sort_dir: str, alpha: dict[str, int] | None = None
) -> list[Member]:
"""Order a collection's members by one of `PLEX_SORT_KEYS`.
Two properties worth stating because they are easy to get wrong with a plain `reverse=True`:
members with no value for the key sink to the END in BOTH directions (matching the browse grid's
`nullslast()` — a movie with no rating is not "the worst rated"), and ties always resolve
title-ascending, so flipping the direction reverses the ranking rather than the alphabet inside
it. An unknown key falls back to the page's default instead of returning an arbitrary order.
`alpha` ranks each title in the reader's alphabet (`collation.sorted_words`); without it, titles
compare as Python strings, which puts every accented initial after "z"."""
by_title = (lambda m: alpha[m.title]) if alpha is not None else (lambda m: m.title)
key = MEMBER_SORT_KEYS.get(sort, MEMBER_SORT_KEYS[DEFAULT_MEMBER_SORT])
if key is MEMBER_SORT_KEYS["title"]:
key = by_title
ordered = sorted(members, key=by_title) # stable secondary, applied first
have = [m for m in ordered if key(m) is not None]
missing = [m for m in ordered if key(m) is None]
have.sort(key=key, reverse=sort_dir != "asc")
return have + missing
@router.get("/collections/{rating_key}")
def collection_detail(
rating_key: str,
@@ -283,45 +118,20 @@ def collection_detail(
what this page did before it had a control. Read-only here; edits go through the admin mutation
endpoints below."""
lang = norm_lang(lang)
col = by_rating_key(db, PlexCollection, rating_key)
col = queries.collection(db, rating_key)
if col is None:
raise not_found("collection")
movies = (
db.execute(
select(PlexItem)
.where(PlexItem.kind == "movie", PlexItem.collection_keys.contains([rating_key]))
.order_by(PlexItem.year.asc().nullslast(), collated(title_col(PlexItem, lang), lang))
)
.scalars()
.all()
)
shows = (
db.execute(
select(PlexShow)
.where(PlexShow.collection_keys.contains([rating_key]))
.order_by(PlexShow.year.asc().nullslast(), collated(title_col(PlexShow, lang), lang))
)
.scalars()
.all()
)
movies, shows = queries.collection_members(db, rating_key, lang)
mstate = rollup.item_state_map(db, user.id, [m.id for m in movies])
# ONE rollup pass over the show members, for both the aggregate badge status and `resumed_at` —
# the browse grid's "last watched" for a show (the newest resume/watched across its episodes). The
# cheaper `rollup.state_map` gives only the status, which would leave `lastwatched` unorderable on
# a show collection while the control still offered it.
show_rows = (
db.execute(
rollup.agg_subquery(db, user.id, include_resumed=True).where(
PlexItem.show_id.in_([s.id for s in shows])
)
).all()
if shows
else []
)
show_rows = queries.show_rollups(db, user.id, [s.id for s in shows])
sstate = {r.show_id: rollup.show_state(r.total, r.watched, r.inprog) for r in show_rows}
sresumed = {r.show_id: r.resumed_at for r in show_rows}
def _movie_last_watched(st: PlexState | None) -> datetime | None:
def _movie_last_watched(st: "PlexState | None") -> datetime | None:
# Same definition as the browse grid's: when it was last resumed, else when it was marked
# watched (coalesce(progress_updated_at, watched_at)).
return (st.progress_updated_at or st.watched_at) if st else None
@@ -365,8 +175,8 @@ def collection_detail(
years = [m.year for m in members if m.year is not None]
certs = [m.content_rating for m in movies] + [s.content_rating for s in shows]
return {
**_collection_card(col),
"content_rating": _collection_cert(certs),
**collection_card(col),
"content_rating": collection_cert(certs),
"year_min": min(years) if years else None,
"year_max": max(years) if years else None,
"total_runtime_s": sum(m.duration_s or 0 for m in movies),
@@ -374,11 +184,11 @@ def collection_detail(
}
def _editable_collection_or_403(db: Session, rating_key: str) -> PlexCollection:
col = by_rating_key(db, PlexCollection, rating_key)
def _editable_collection_or_403(db: Session, rating_key: str) -> "PlexCollection":
col = queries.collection(db, rating_key)
if col is None:
raise not_found("collection")
if not _collection_editable(col):
if not collection_editable(col):
raise refused(403, "collection_read_only")
return col
@@ -389,16 +199,7 @@ def _reject_duplicate_name(
"""No two collections in a section may share a name: Plex CONFLATES same-named collections'
membership (both rating_keys resolve to one merged member list — a real footgun), so forbid it up
front on create + rename rather than mirror the mess."""
dup = (
db.execute(
select(PlexCollection).where(
PlexCollection.library_id == library_id,
func.lower(PlexCollection.title) == title.lower(),
)
)
.scalars()
.first()
)
dup = queries.collection_named(db, library_id, title)
if dup is not None and dup.rating_key != exclude_rk:
# Structured, so the bilingual UI says it in the reader's language (it was toasted in
# Hungarian beside an English error dialog saying the same thing).
@@ -441,15 +242,13 @@ def create_collection(
seed = body.item_rating_key
if not title or not seed:
raise refused(400, "collection_fields_required")
seed_show = by_rating_key(db, PlexShow, seed)
seed_show = queries.show(db, seed)
if library:
lib = (
db.execute(select(PlexLibrary).where(PlexLibrary.plex_key == library)).scalars().first()
)
lib = queries.library_by_key(db, library)
else:
# The "New collection" flow only knows the seed title → derive its library.
seed_item = by_rating_key(db, PlexItem, seed) or seed_show
lib = db.get(PlexLibrary, seed_item.library_id) if seed_item else None
seed_item = queries.item(db, seed) or seed_show
lib = queries.library(db, seed_item.library_id) if seed_item else None
if lib is None:
raise not_found("library")
_reject_duplicate_name(db, lib.id, title)
@@ -462,28 +261,16 @@ def create_collection(
wrote = True
new_rk = str(new.get("ratingKey") or "")
col = plex_sync.resync_collection(db, plex, lib, new_rk)
col.editable = True
service.set_collection_editable(col, True)
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
if wrote and new_rk:
# It is on Plex now, and only the follow-up read failed. Its row is kept, editable: the
# full sync preserves `editable` but creates a missing row read-only, so the collection
# would drop out of every "add to" list; and the duplicate-name check reads these rows,
# so a retry of the same name would make a second one on Plex. Plex made it with exactly
# the seed in it, so the seed's row says so too — otherwise the title showed unticked in
# the collection it was just added to, and the collection empty, until that sync.
db.add(
PlexCollection(
rating_key=new_rk, library_id=lib.id, title=title, editable=True, child_count=1
)
)
Model = PlexItem if lib.kind == "movie" else PlexShow
if row := by_rating_key(db, Model, seed):
row.collection_keys = list(dict.fromkeys([*(row.collection_keys or []), new_rk]))
# It is on Plex now, and only the follow-up read failed: keep its row (see the service).
service.keep_created_collection(db, lib, new_rk, title, seed)
db.commit()
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
return collection_card(col)
@router.post("/collections/{rating_key}/items/{item_rating_key}")
@@ -494,7 +281,7 @@ def collection_add_item(
db: Session = Depends(get_db),
) -> dict:
col = _editable_collection_or_403(db, rating_key)
lib = db.get(PlexLibrary, col.library_id)
lib = queries.library(db, col.library_id)
wrote = False
try:
with PlexClient(db) as plex:
@@ -505,7 +292,7 @@ def collection_add_item(
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
return collection_card(col)
@router.delete("/collections/{rating_key}/items/{item_rating_key}")
@@ -516,7 +303,7 @@ def collection_remove_item(
db: Session = Depends(get_db),
) -> dict:
col = _editable_collection_or_403(db, rating_key)
lib = db.get(PlexLibrary, col.library_id)
lib = queries.library(db, col.library_id)
wrote = False
try:
with PlexClient(db) as plex:
@@ -527,7 +314,7 @@ def collection_remove_item(
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
return collection_card(col)
class CollectionRenameIn(BaseModel):
@@ -546,7 +333,7 @@ def collection_rename(
if not title:
raise refused(400, "title_required")
_reject_duplicate_name(db, col.library_id, title, exclude_rk=rating_key)
lib = db.get(PlexLibrary, col.library_id)
lib = queries.library(db, col.library_id)
wrote = False
try:
with PlexClient(db) as plex:
@@ -557,7 +344,7 @@ def collection_rename(
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
return collection_card(col)
@router.delete("/collections/{rating_key}")
@@ -565,7 +352,7 @@ def collection_delete(
rating_key: str, _: User = Depends(admin_user), db: Session = Depends(get_db)
) -> dict:
col = _editable_collection_or_403(db, rating_key)
lib = db.get(PlexLibrary, col.library_id)
lib = queries.library(db, col.library_id)
try:
with PlexClient(db) as plex:
plex.delete_collection(rating_key)
@@ -590,12 +377,12 @@ def collection_set_editable(
) -> dict:
"""Mark an existing Plex collection as 'mine to edit' (or unmark). Smart / auto-list collections
(IMDb/TMDb/…) can never be made editable — they're regenerated outside Plex."""
col = by_rating_key(db, PlexCollection, rating_key)
col = queries.collection(db, rating_key)
if col is None:
raise not_found("collection")
want = body.editable
if want and (col.smart or _collection_source(col) != "collection"):
if want and (col.smart or collection_source(col) != "collection"):
raise refused(400, "collection_smart")
col.editable = want
service.set_collection_editable(col, want)
db.commit()
return _collection_card(col)
return collection_card(col)
+31 -130
View File
@@ -4,52 +4,34 @@ import re
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.collation import collated, sorted_words
from app.collation import sorted_words
from app import logs
from app.auth import current_user
from app.db import get_db
from app.discover import queries
from app.models import (
PlexCollection,
PlexItem,
PlexLibrary,
PlexSeason,
PlexShow,
PlexState,
User,
)
from app.discover import queries as discover_queries
from app.models import User
from app.plex import detail as plex_detail
from app.plex import people as plex_people
from app.plex import rollup, subtitles
from app.plex.query import title_col
from app.plex import queries, rollup, service, subtitles
from app.plex import tmdb as plex_tmdb
from app.plex.collections import collection_source
from app.plex.cards import SHOW_NEW, episode_card, movie_card, show_card
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.lookups import by_rating_key
from app.routes.plex._shared import _norm_lang, item_or_404
from app.routes.plex.collections import _collection_source
from app.routes._common import not_found
if TYPE_CHECKING:
from app.plex.models import PlexItem, PlexSeason
log = logs.get_logger(__name__)
router = APIRouter(prefix="/api/plex", tags=["plex"])
def _lib_key(db: Session, library_id: int | None) -> str | None:
"""The Plex section key for a library id — the info/show page passes it to the admin collection
editor (in the unified cross-library view there's no single library prop otherwise)."""
if library_id is None:
return None
lib = db.get(PlexLibrary, library_id)
return lib.plex_key if lib else None
_SHOW_ENRICH_TTL_S = 300
_show_enrich_cache: dict[str, tuple[float, dict, list]] = {}
@@ -113,39 +95,12 @@ def show_detail(
"""A show's seasons + episodes (the drill-down view) with per-user watch state."""
lang = _norm_lang(lang)
hu = lang == "hu"
sh = by_rating_key(db, PlexShow, rating_key)
sh = queries.show(db, rating_key)
if sh is None:
raise not_found("show")
seasons = (
db.execute(
select(PlexSeason)
.where(PlexSeason.show_id == sh.id)
.order_by(PlexSeason.season_number.asc().nullsfirst())
)
.scalars()
.all()
)
eps = (
db.execute(
select(PlexItem)
.where(PlexItem.show_id == sh.id, PlexItem.kind == "episode")
.order_by(
PlexItem.season_number.asc().nullsfirst(),
PlexItem.episode_number.asc().nullsfirst(),
)
)
.scalars()
.all()
)
states = {
s.item_id: s
for s in db.execute(
select(PlexState).where(
PlexState.user_id == user.id,
PlexState.item_id.in_([e.id for e in eps] or [0]),
)
).scalars()
}
seasons = queries.seasons(db, sh)
eps = queries.episodes(db, sh)
states = rollup.item_state_map(db, user, [e.id for e in eps])
by_season: dict[int | None, list] = {}
all_cards: list[dict] = [] # every episode, in season/episode order → show-level rollup
for e in eps:
@@ -163,9 +118,7 @@ def show_detail(
meta, related_raw = _show_enrichment(db, sh.rating_key)
related = _relatedshow_cards(db, user.id, related_raw, exclude=sh.id, lang=lang)
if sh.people_enriched_at is None and meta:
plex_people.enrich_row_people(db, sh, plex_people.cast_from_meta(meta))
plex_detail.apply_rich_detail(sh, meta) # imdb/tagline/full_cast for future views
sh.people_enriched_at = datetime.now(timezone.utc)
service.enrich_show(db, sh, meta) # imdb/tagline/full_cast for future views
try:
db.commit()
except Exception: # noqa: BLE001 — a failed enrich must not fail the page
@@ -198,7 +151,7 @@ def show_detail(
"year": sh.year,
"thumb": f"/api/plex/image/{sh.rating_key}",
"art": f"/api/plex/image/{sh.rating_key}?variant=art",
"library": _lib_key(db, sh.library_id),
"library": queries.library_key(db, sh.library_id),
"content_rating": sh.content_rating,
"rating": sh.rating,
"genres": (sh.genres_hu or sh.genres if hu else sh.genres) or [],
@@ -234,13 +187,13 @@ def show_next(
Asked one show at a time, when a card is hovered or played, rather than carried on every card of a
grid: three probes a show would cost a 60-card page 180 queries for the few cards anyone plays.
`next` is null for a show with no episodes."""
sh = by_rating_key(db, PlexShow, rating_key)
sh = queries.show(db, rating_key)
if sh is None:
raise not_found("show")
return {"next": queries.next_episode(db, user.id, sh.id, lang=_norm_lang(lang))}
return {"next": discover_queries.next_episode(db, user.id, sh.id, lang=_norm_lang(lang))}
def _season_block(se: PlexSeason, cards: list[dict]) -> dict:
def _season_block(se: "PlexSeason", cards: list[dict]) -> dict:
"""A season's card for the show page (with its aggregate rollup) plus its episodes (used by the
season subpage, read from the same cached show-detail payload)."""
roll = rollup.rollup(cards)
@@ -265,12 +218,7 @@ def _relatedshow_cards(
rks = [str(m.get("ratingKey")) for m in related if m.get("ratingKey")]
if not rks:
return []
shows = {
s.rating_key: s
for s in db.execute(
select(PlexShow).where(PlexShow.rating_key.in_(rks[:60]), PlexShow.id != exclude)
).scalars()
}
shows = queries.shows_by_keys(db, rks[:60], exclude)
statuses = rollup.state_map(db, user_id, [s.id for s in shows.values()])
out: list[dict] = []
seen: set[str] = set()
@@ -285,73 +233,36 @@ def _relatedshow_cards(
return out
def _collection_strips(db: Session, it: PlexItem, user_id: int, lang: str) -> list[dict]:
def _collection_strips(db: Session, it: "PlexItem", user_id: int, lang: str) -> list[dict]:
"""The collections this movie belongs to, each with its member titles (playable cards) — the
'Avatar Collection' strip on the info page. Pure local lookup (no Plex call)."""
keys = it.collection_keys or []
if not keys or it.kind != "movie":
return []
cols = {
c.rating_key: c
for c in db.execute(
select(PlexCollection).where(PlexCollection.rating_key.in_(keys))
).scalars()
}
cols = {c.rating_key: c for c in queries.collections_by_keys(db, keys)}
# Smallest (most specific — e.g. a franchise) first; the big auto-lists (IMDb/TMDb Popular) last.
alpha = {w: i for i, w in enumerate(sorted_words(db, [c.title for c in cols.values()], lang))}
ordered = sorted(cols.values(), key=lambda c: (c.child_count or 0, alpha[c.title]))
out = []
for col in ordered:
rk = col.rating_key
members = (
db.execute(
select(PlexItem)
.where(
PlexItem.library_id == it.library_id,
PlexItem.kind == "movie",
PlexItem.collection_keys.contains([rk]),
)
.order_by(
PlexItem.year.asc().nullslast(), collated(title_col(PlexItem, lang), lang)
)
.limit(60)
)
.scalars()
.all()
)
ids = [m.id for m in members]
states = {
s.item_id: s
for s in db.execute(
select(PlexState).where(
PlexState.user_id == user_id, PlexState.item_id.in_(ids or [0])
)
).scalars()
}
members = queries.library_movies_in(db, it.library_id, rk, lang, limit=60)
states = rollup.item_state_map(db, user_id, [m.id for m in members])
out.append(
{
"id": rk,
"title": col.title,
"source": _collection_source(col),
"source": collection_source(col),
"items": [movie_card(m, states.get(m.id), lang=lang) for m in members],
}
)
return out
def _episode_nav(db: Session, it: PlexItem) -> tuple[str | None, str | None]:
def _episode_nav(db: Session, it: "PlexItem") -> tuple[str | None, str | None]:
if it.kind != "episode" or it.show_id is None:
return None, None
keys = list(
db.execute(
select(PlexItem.rating_key)
.where(PlexItem.show_id == it.show_id, PlexItem.kind == "episode")
.order_by(
PlexItem.season_number.asc().nullsfirst(),
PlexItem.episode_number.asc().nullsfirst(),
)
).scalars()
)
keys = queries.episode_keys(db, it.show_id)
try:
i = keys.index(it.rating_key)
except ValueError:
@@ -398,13 +309,7 @@ def item_detail(
prev/next, and the per-user resume position + watch status."""
lang = _norm_lang(lang)
it = item_or_404(db, rating_key)
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user.id, PlexState.item_id == it.id)
)
.scalars()
.first()
)
st = queries.state(db, user, it.id)
# Markers + audio/subtitle streams are file-specific, so they stay LIVE; the rich detail (cast/IMDb/
# tagline/genres) is served from the DB row for movies (persisted below / by the backfill) and only
@@ -485,11 +390,7 @@ def item_detail(
# only stored the listing's top ~3), so fill a not-yet-enriched movie — its cast becomes filterable
# and future views serve from the row. Episodes aren't enriched; a show's cast is filled from /show.
if it.kind == "movie" and it.people_enriched_at is None and meta:
cast = plex_people.cast_from_meta(meta)
if cast:
plex_people.enrich_row_people(db, it, cast)
plex_detail.apply_rich_detail(it, meta)
it.people_enriched_at = datetime.now(timezone.utc)
service.enrich_movie(db, it, meta)
try:
db.commit()
except Exception: # noqa: BLE001 — a failed enrich must not fail the page
@@ -522,8 +423,8 @@ def item_detail(
rich = plex_detail.rich_detail(meta)
prev_id, next_id = _episode_nav(db, it)
show = db.get(PlexShow, it.show_id) if it.kind == "episode" and it.show_id else None
season = db.get(PlexSeason, it.season_id) if show and it.season_id else None
show = queries.show_by_id(db, it.show_id) if it.kind == "episode" and it.show_id else None
season = queries.season_by_id(db, it.season_id) if show and it.season_id else None
return {
"id": it.rating_key,
"kind": it.kind,
@@ -538,7 +439,7 @@ def item_detail(
"video_bit_depth": it.video_bit_depth,
"thumb": f"/api/plex/image/{it.rating_key}",
"art": f"/api/plex/image/{it.rating_key}?variant=art",
"library": _lib_key(db, it.library_id),
"library": queries.library_key(db, it.library_id),
"cast": rich.get("cast", []),
"imdb_rating": rich.get("imdb_rating"),
"imdb_id": rich.get("imdb_id"),
+7 -510
View File
@@ -1,391 +1,18 @@
"""Plex Discover: the rail-surface read-model.
A consolidated payload for the dedicated Discover landing — a stack of horizontal rails:
Continue watching · Recently added · Browse-by-genre tiles · Because you watched X — plus a
separate, reshuffling Surprise-me sample. Serve-from-DB only: the recommendation rail reads the
recs the TMDB engine already persisted (`PlexItem.recommendations`), never a live TMDB call, and
every card comes from the same `movie_card`/`show_card` serializers the grid uses (so rails inherit
their `art` + rating/summary for free). The rails carry EN metadata (like the other non-grid
card paths); the HU overlay coalesces only in the main grid.
The sidebar metadata filters (genre / rating / year / collection / …) narrow EVERY rail too, so the
Discover surface behaves like the Browse grid — the same `LibraryFilters` / `meta_filter_conds` the
grid uses, applied to each rail's movie + show query (a rail may legitimately come back empty, which
the client hides). Genre-language matches the grid; duration is movie-only.
"""
"""Plex Discover: the rail-surface landing and its Surprise-me draw (`plex.discover`)."""
from fastapi import APIRouter, Depends, Query
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
from app.collation import sorted_words
from app.auth import current_user
from app.db import get_db
from app.models import PlexItem, PlexShow, PlexState, User
from app.plex import rollup
from app.plex.cards import (
SHOW_NEW,
display_title,
leaf_card,
movie_card,
show_card,
)
from app.plex.query import (
LibraryFilters,
apply_meta_filters,
apply_show_state_filter,
content_libs,
genres_col,
meta_filter_conds,
top_art_keys,
)
from app.discover.queries import RAIL_LIMIT
from app.plex.tmdb import resolve_recs
from app.models import User
from app.plex import discover as plex_discover
from app.plex.discover import SURPRISE_PER_KIND
from app.plex.query import LibraryFilters
from app.routes.plex._shared import _norm_lang
router = APIRouter(prefix="/api/plex", tags=["plex"])
GENRE_TILES = 12
SURPRISE_PER_KIND = 3 # Surprise-me: at most this many movies + this many shows (6 total)
def _scope_libs(db: Session, scope: str) -> tuple[list[int], list[int]]:
"""The movie/show library ids visible for this scope (empty list = out of scope)."""
movie_ids, show_ids = content_libs(db)
return (
movie_ids if scope in ("movie", "both") else [],
show_ids if scope in ("show", "both") else [],
)
def _movie_meta(q, p: dict, lang: str, f: LibraryFilters):
"""Apply the shared metadata filters + movie-only duration to a movie query (same as the grid)."""
q = apply_meta_filters(q, PlexItem, p, lang)
if f.duration_min is not None:
q = q.where(PlexItem.duration_s >= f.duration_min)
if f.duration_max is not None:
q = q.where(PlexItem.duration_s <= f.duration_max)
return q
def _movie_cards(db: Session, uid: int, items: list[PlexItem], lang: str) -> list[dict]:
sts = rollup.item_state_map(db, uid, [it.id for it in items])
return [movie_card(it, sts.get(it.id), lang=lang) for it in items]
def _show_cards(db: Session, uid: int, shows: list[PlexShow], lang: str) -> list[dict]:
statuses = rollup.state_map(db, uid, [s.id for s in shows])
return [show_card(s, statuses.get(s.id, SHOW_NEW), lang=lang) for s in shows]
def _recently_added(
db: Session,
uid: int,
movie_ids: list[int],
show_ids: list[int],
p: dict,
lang: str,
f: LibraryFilters,
) -> list[dict]:
"""Newest movies + shows, interleaved by `added_at` desc into one rail."""
movies = (
db.execute(
_movie_meta(
select(PlexItem).where(
PlexItem.kind == "movie", PlexItem.library_id.in_(movie_ids)
),
p,
lang,
f,
)
.order_by(PlexItem.added_at.desc().nullslast())
.limit(RAIL_LIMIT)
)
.scalars()
.all()
if movie_ids
else []
)
shows = (
db.execute(
apply_meta_filters(
select(PlexShow).where(PlexShow.library_id.in_(show_ids)), PlexShow, p, lang
)
.order_by(PlexShow.added_at.desc().nullslast())
.limit(RAIL_LIMIT)
)
.scalars()
.all()
if show_ids
else []
)
# Merge the two type-homogeneous lists by added_at, then serialize each side in one batch so the
# per-card state lookups stay batched (no N+1). Same `or NO_TS` key as the two rails below — a
# review found this one still on the older `(is not None, value)` form, i.e. one rule in two
# spellings, with the two comments explaining it differently.
tagged = [(m.added_at, "movie", m) for m in movies] + [(s.added_at, "show", s) for s in shows]
tagged.sort(key=lambda t: t[0] or rollup.NO_TS, reverse=True)
top = tagged[:RAIL_LIMIT]
# Key the serialized cards back by rating_key (globally unique in Plex — a movie and a show never
# share one), so the merge doesn't depend on object identity staying alive.
top_movies = [o for _, kind, o in top if kind == "movie"]
top_shows = [o for _, kind, o in top if kind == "show"]
mcards = {
m.rating_key: c
for m, c in zip(top_movies, _movie_cards(db, uid, top_movies, lang), strict=True)
}
scards = {
s.rating_key: c
for s, c in zip(top_shows, _show_cards(db, uid, top_shows, lang), strict=True)
}
return [mcards[o.rating_key] if kind == "movie" else scards[o.rating_key] for _, kind, o in top]
def _recently_watched(
db: Session,
uid: int,
movie_ids: list[int],
show_ids: list[int],
p: dict,
lang: str,
f: LibraryFilters,
) -> list[dict]:
"""Recently watched rail: the titles the user last FULLY watched, newest first. A movie is its own
card; a show contributes the specific EPISODE watched (carrying its show title). Movies honour the
sidebar filters (like the other rails); episodes carry no filterable metadata, so they don't."""
rows: list[tuple[object, dict]] = [] # (watched_at, card)
if movie_ids:
mq = (
select(PlexItem, PlexState)
.join(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.status == "watched",
)
)
for it, st in db.execute(
_movie_meta(mq, p, lang, f)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all():
rows.append((st.watched_at, leaf_card(it, st, lang=lang)))
if show_ids:
eq = (
select(PlexItem, PlexState, PlexShow)
.join(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.join(PlexShow, PlexShow.id == PlexItem.show_id)
.where(
PlexItem.kind == "episode",
PlexItem.library_id.in_(show_ids),
PlexState.status == "watched",
)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(RAIL_LIMIT)
)
for ep, st, sh in db.execute(eq).all():
card = leaf_card(
ep, st, show_title=display_title(sh, lang), show_rk=sh.rating_key, lang=lang
)
rows.append((st.watched_at, card))
rows.sort(key=lambda r: r[0] or rollup.NO_TS, reverse=True)
return [card for _, card in rows[:RAIL_LIMIT]]
def _continue_watching(
db: Session,
uid: int,
movie_ids: list[int],
show_ids: list[int],
p: dict,
lang: str,
f: LibraryFilters,
) -> list[dict]:
"""Resume rail: in-progress movies + partially-watched shows, most-recently-progressed first.
A movie's resume time is its own `progress_updated_at`; a show's is the max across its episodes."""
rows: list[tuple[object, dict]] = [] # (resumed_at, card)
if movie_ids:
mq = (
select(PlexItem, PlexState)
.join(
PlexState,
and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid),
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.position_seconds > 0,
# A NULL status is still "started" (mirrors apply_movie_state_filter's in_progress) —
# `notin_` alone would drop it, since `NULL NOT IN (...)` is NULL, not TRUE.
or_(
PlexState.status.is_(None),
PlexState.status.notin_(["watched", "hidden"]),
),
)
)
mrows = db.execute(
_movie_meta(mq, p, lang, f)
.order_by(PlexState.progress_updated_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all()
for it, st in mrows:
rows.append((st.progress_updated_at, movie_card(it, st, lang=lang)))
if show_ids:
# Reuse the shared per-show episode rollup (total/watched/inprog) rather than re-inlining it, plus
# its `resumed_at` = max(progress_updated_at); the in-progress predicate comes from the same shared
# `apply_show_state_filter` the grid uses, so the rail can't drift from the grid's definition.
prog = rollup.agg_subquery(db, uid, include_resumed=True).subquery()
sq = apply_show_state_filter(
select(PlexShow, prog.c.resumed_at, prog.c.total, prog.c.watched, prog.c.inprog).join(
prog, prog.c.show_id == PlexShow.id
),
prog.c.total,
prog.c.watched,
prog.c.inprog,
"in_progress",
).where(PlexShow.library_id.in_(show_ids))
srows = db.execute(
apply_meta_filters(sq, PlexShow, p, lang)
.order_by(prog.c.resumed_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all()
for sh, resumed_at, total, watched, inprog in srows:
state = rollup.show_state(total, watched, inprog)
rows.append((resumed_at, show_card(sh, state, lang=lang)))
rows.sort(key=lambda r: r[0] or rollup.NO_TS, reverse=True)
return [card for _, card in rows[:RAIL_LIMIT]]
def _genre_tiles(
db: Session, movie_ids: list[int], show_ids: list[int], p: dict, lang: str
) -> list[dict]:
"""The most-common genres across the scope, as {value, count} tiles → tap to filter the grid.
The genre vocabulary is language-specific (coalesce(genres_hu, genres) in HU), matching the grid.
The active metadata filters narrow the counts too, so the tiles agree with the filtered rails."""
counts: dict[str, int] = {}
for model, ids in ((PlexItem, movie_ids), (PlexShow, show_ids)):
if not ids:
continue
gcol = genres_col(model, lang)
base = select(func.jsonb_array_elements_text(gcol).label("g")).where(
model.library_id.in_(ids),
func.jsonb_typeof(gcol) == "array",
*meta_filter_conds(model, p, lang),
)
if model is PlexItem:
base = base.where(model.kind == "movie")
expanded = base.subquery()
for g, c in db.execute(select(expanded.c.g, func.count()).group_by(expanded.c.g)).all():
counts[g] = counts.get(g, 0) + c
alpha = {w: i for i, w in enumerate(sorted_words(db, list(counts), lang))}
top = sorted(counts.items(), key=lambda kv: (-kv[1], alpha[kv[0]]))[:GENRE_TILES]
# A 2×2 poster collage per tile: the top movies in that genre (color hash is the client fallback).
art = top_art_keys(
db,
PlexItem,
movie_ids,
genres_col(PlexItem, lang),
[g for g, _ in top],
extra_conds=(PlexItem.kind == "movie", *meta_filter_conds(PlexItem, p, lang)),
)
return [{"value": g, "count": c, "art": art.get(g, [])} for g, c in top]
def _because_you_watched(
db: Session,
uid: int,
movie_ids: list[int],
show_ids: list[int],
p: dict,
lang: str,
f: LibraryFilters,
) -> dict | None:
"""Recommendations anchored on the user's most-recently-watched, TMDB-enriched movie. Only the
recs that resolve to OUR library (by guid) become cards — the rest aren't actionable here. Returns
None (rail hidden) when there's no enriched watch history or none of its recs are in the library.
The metadata filters narrow the resolved recs (not the anchor pick) so the rail matches the grid."""
if not movie_ids:
return None
anchor = (
db.execute(
select(PlexItem)
.join(
PlexState,
and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid),
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.status == "watched",
PlexItem.recommendations.isnot(None),
)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(1)
)
.scalars()
.first()
)
if anchor is None:
return None
resolved = resolve_recs(db, anchor.recommendations, lang=lang)
# Preserve TMDB's recommendation order, keep only in-library hits, and drop the anchor itself.
rks = [
r["rating_key"]
for r in resolved
if r.get("in_library") and r.get("rating_key") != anchor.rating_key
]
if not rks:
return None
movies = {
m.rating_key: m
for m in db.execute(
_movie_meta(
select(PlexItem).where(PlexItem.kind == "movie", PlexItem.rating_key.in_(rks)),
p,
lang,
f,
)
)
.scalars()
.all()
}
shows = {
s.rating_key: s
for s in db.execute(
apply_meta_filters(
select(PlexShow).where(PlexShow.rating_key.in_(rks)), PlexShow, p, lang
)
)
.scalars()
.all()
}
ordered_movies = [movies[rk] for rk in rks if rk in movies]
ordered_shows = [shows[rk] for rk in rks if rk in shows]
# rating_key is globally unique in Plex, so it keys the serialized cards back safely (no id()).
mcards = {
m.rating_key: c
for m, c in zip(ordered_movies, _movie_cards(db, uid, ordered_movies, lang), strict=True)
}
scards = {
s.rating_key: c
for s, c in zip(ordered_shows, _show_cards(db, uid, ordered_shows, lang), strict=True)
}
items: list[dict] = []
for rk in rks:
if rk in mcards:
items.append(mcards[rk])
elif rk in scards:
items.append(scards[rk])
if not items:
return None
return {
"anchor": {"id": anchor.rating_key, "title": display_title(anchor, lang)},
"items": items,
}
@router.get("/discover")
def discover(
@@ -398,72 +25,7 @@ def discover(
"""The Discover landing payload — every rail in one pass (serve-from-DB), narrowed by the active
sidebar filters. Empty rails come back as empty lists (the client hides them); `because_you_watched`
is null when there's no anchor."""
lang = _norm_lang(lang)
p = f.as_dict()
movie_ids, show_ids = _scope_libs(db, scope)
return {
"scope": scope,
"continue_watching": _continue_watching(db, user.id, movie_ids, show_ids, p, lang, f),
"recently_watched": _recently_watched(db, user.id, movie_ids, show_ids, p, lang, f),
"recently_added": _recently_added(db, user.id, movie_ids, show_ids, p, lang, f),
"genres": _genre_tiles(db, movie_ids, show_ids, p, lang),
"because_you_watched": _because_you_watched(db, user.id, movie_ids, show_ids, p, lang, f),
}
def _draw(db: Session, stmt, key_col, per: int, exclude: list[str]) -> list:
"""`per` random rows of `stmt`, the previous draw's (`exclude`) left out while the pool allows: a
reshuffle over a small filtered pool kept dealing the same few titles. What the fresh rows leave
short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail."""
fresh = (
db.execute(stmt.where(key_col.not_in(exclude)).order_by(func.random()).limit(per))
.scalars()
.all()
if exclude
else []
)
if len(fresh) == per:
return list(fresh)
taken = [key_col.not_in([r.rating_key for r in fresh])] if fresh else []
rest = db.execute(stmt.where(*taken).order_by(func.random()).limit(per - len(fresh))).scalars()
return [*fresh, *rest.all()]
def _kept(
db: Session, uid: int, movie_ids: list[int], show_ids: list[int], keep: list[str], lang: str
) -> list[dict]:
"""The `keep` titles' cards in their given order, as they stand now: a title watched since the draw
stays in it, marked watched, and one gone from the scope's libraries drops out."""
movies = (
db.execute(
select(PlexItem).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexItem.rating_key.in_(keep),
)
)
.scalars()
.all()
if movie_ids
else []
)
shows = (
db.execute(
select(PlexShow).where(PlexShow.library_id.in_(show_ids), PlexShow.rating_key.in_(keep))
)
.scalars()
.all()
if show_ids
else []
)
cards = {
c["id"]: c
for c in [
*_movie_cards(db, uid, list(movies), lang),
*_show_cards(db, uid, list(shows), lang),
]
}
return [cards[k] for k in keep if k in cards]
return plex_discover.discover_page(db, user, scope, _norm_lang(lang), f)
@router.get("/surprise")
@@ -485,69 +47,4 @@ def surprise(
excluded (the point is to surface something to watch). Capped at SURPRISE_PER_KIND movies +
SURPRISE_PER_KIND shows, drawn separately then interleaved so a lopsided library doesn't crowd out
one type."""
lang = _norm_lang(lang)
p = f.as_dict()
movie_ids, show_ids = _scope_libs(db, scope)
if keep:
return {"scope": scope, "items": _kept(db, user.id, movie_ids, show_ids, keep, lang)}
per = min(limit, SURPRISE_PER_KIND)
movies = []
if movie_ids:
# Exclude movies this user has watched.
watched_movie = (
select(PlexState.id)
.where(
PlexState.item_id == PlexItem.id,
PlexState.user_id == user.id,
PlexState.status == "watched",
)
.exists()
)
movies = _draw(
db,
_movie_meta(
select(PlexItem).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
~watched_movie,
),
p,
lang,
f,
),
PlexItem.rating_key,
per,
exclude,
)
shows = []
if show_ids:
# Exclude fully-watched shows (LEFT JOIN so a never-started show — no rollup row — still counts
# as unseen). `total IS NULL / == 0 / watched < total` == "not fully watched".
prog = rollup.agg_subquery(db, user.id).subquery()
not_fully_watched = or_(
prog.c.total.is_(None), prog.c.total == 0, prog.c.watched < prog.c.total
)
shows = _draw(
db,
apply_meta_filters(
select(PlexShow)
.outerjoin(prog, prog.c.show_id == PlexShow.id)
.where(PlexShow.library_id.in_(show_ids), not_fully_watched),
PlexShow,
p,
lang,
),
PlexShow.rating_key,
per,
exclude,
)
mcards = _movie_cards(db, user.id, movies, lang)
scards = _show_cards(db, user.id, shows, lang)
# Interleave movie/show cards, then trim to `limit`.
items: list[dict] = []
for i in range(max(len(mcards), len(scards))):
if i < len(mcards):
items.append(mcards[i])
if i < len(scards):
items.append(scards[i])
return {"scope": scope, "items": items[:limit]}
return plex_discover.surprise(db, user, scope, _norm_lang(lang), limit, exclude, keep, f)
+16 -99
View File
@@ -4,26 +4,18 @@ import threading
import time
from fastapi import APIRouter, BackgroundTasks, Depends
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app.collation import collated
from app import logs, sysconfig
from app import sysconfig
from app.auth import current_user
from app.db import SessionLocal, get_db
from app.models import (
PlexItem,
PlexPerson,
PlexShow,
User,
)
from app.plex import rollup
from app.db import get_db
from app.models import User
from app.plex import queries, rollup, service
from app.plex import tmdb as plex_tmdb
from app.plex.cards import SHOW_NEW, movie_card, norm_lang, show_card
from app.plex.query import content_libs, split_csv, title_col
from app.plex.query import content_libs, split_csv
router = APIRouter(prefix="/api/plex", tags=["plex"])
log = logs.get_logger(__name__)
# How many "Known for" titles the person page shows (the stored pool is deeper; see plex.tmdb).
@@ -80,35 +72,13 @@ def people_cards(
movie_lib_ids, show_lib_ids = content_libs(db)
want_movies = scope in ("movie", "both") and bool(movie_lib_ids)
want_shows = scope in ("show", "both") and bool(show_lib_ids)
photos = {
p.name: p.photo_thumb
for p in db.execute(select(PlexPerson).where(PlexPerson.name.in_(wanted))).scalars()
}
out = []
for nm in wanted:
count = 0
if want_movies:
count += (
db.scalar(
select(func.count(PlexItem.id)).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_lib_ids),
PlexItem.cast_names.contains([nm]),
)
)
or 0
)
if want_shows:
count += (
db.scalar(
select(func.count(PlexShow.id)).where(
PlexShow.library_id.in_(show_lib_ids),
PlexShow.cast_names.contains([nm]),
)
)
or 0
)
out.append({"name": nm, "photo": photos.get(nm), "count": count})
photos = queries.person_photos(db, wanted)
movies = movie_lib_ids if want_movies else []
shows = show_lib_ids if want_shows else []
out = [
{"name": nm, "photo": photos.get(nm), "count": queries.cast_count(db, nm, movies, shows)}
for nm in wanted
]
return {"people": out}
@@ -144,24 +114,6 @@ def _should_schedule_enrich(person_id: int) -> bool:
return True
def _enrich_person_bg(person_id: int) -> None:
"""Fetch + persist one person's TMDB bio OFF the request path — a FastAPI background task that runs
AFTER the response is sent, so the outbound TMDB call never sits on the serve latency (the module's
"no live TMDB call at serve time" invariant). The request's session is already closed by then, so
open a fresh one. Best-effort: a TMDB failure just leaves `tmdb_enriched_at` NULL to retry on the
next open, and the bio lands from the row by then."""
db = SessionLocal()
try:
person = db.get(PlexPerson, person_id)
if person is not None:
plex_tmdb.enrich_person(db, person)
except Exception: # noqa: BLE001 — background best-effort; never surfaces to a user
db.rollback()
log.warning("background person bio enrich failed for id=%s", person_id, exc_info=True)
finally:
db.close()
@router.get("/person/{name}")
def person_detail(
name: str,
@@ -183,7 +135,7 @@ def person_detail(
fetches everyone ahead of time, so this lazy path is only for a person the job hasn't reached yet.
"""
lang = norm_lang(lang)
person = db.execute(select(PlexPerson).where(PlexPerson.name == name)).scalars().first()
person = queries.person(db, name)
# Lazy one-time bio enrich, scheduled AFTER the response so the TMDB call is never on the serve path.
# Gated by a per-person cooldown so concurrent/rapid opens (or repeated opens while TMDB is degraded)
# can't pile background tasks onto the shared threadpool.
@@ -197,53 +149,18 @@ def person_detail(
and bool(sysconfig.get_str(db, "tmdb_api_key"))
)
if enriching and person is not None and _should_schedule_enrich(person.id):
background_tasks.add_task(_enrich_person_bg, person.id)
background_tasks.add_task(service.enrich_person_bg, person.id)
movie_lib_ids, show_lib_ids = content_libs(db)
movies: list[dict] = []
if movie_lib_ids:
items = (
db.execute(
select(PlexItem)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_lib_ids),
or_(
PlexItem.cast_names.contains([name]),
PlexItem.directors.contains([name]),
),
)
.order_by(
PlexItem.year.desc().nullslast(), collated(title_col(PlexItem, lang), lang)
)
.limit(_PERSON_SECTION_MAX)
)
.scalars()
.all()
)
items = queries.filmography(db, "movie", name, movie_lib_ids, lang, _PERSON_SECTION_MAX)
states = rollup.item_state_map(db, user.id, [it.id for it in items])
movies = [movie_card(it, states.get(it.id), lang=lang) for it in items]
shows: list[dict] = []
if show_lib_ids:
show_rows = (
db.execute(
select(PlexShow)
.where(
PlexShow.library_id.in_(show_lib_ids),
or_(
PlexShow.cast_names.contains([name]),
PlexShow.directors.contains([name]),
),
)
.order_by(
PlexShow.year.desc().nullslast(), collated(title_col(PlexShow, lang), lang)
)
.limit(_PERSON_SECTION_MAX)
)
.scalars()
.all()
)
show_rows = queries.filmography(db, "show", name, show_lib_ids, lang, _PERSON_SECTION_MAX)
statuses = rollup.state_map(db, user.id, [s.id for s in show_rows])
shows = [show_card(sh, statuses.get(sh.id, SHOW_NEW), lang=lang) for sh in show_rows]
+25 -134
View File
@@ -1,6 +1,7 @@
"""Plex playback: image/subtitle proxies, HLS streaming, and watch-state writes."""
import threading
from typing import TYPE_CHECKING
from datetime import datetime, timedelta, timezone
import httpx
@@ -16,22 +17,13 @@ from fastapi import (
)
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.auth import current_user, resolved_user_id
from app.db import SessionLocal, get_db
from app.models import (
PlexItem,
PlexSeason,
PlexShow,
PlexState,
User,
)
from app.plex import imgcache, subtitles
from app.models import User
from app.plex import imgcache, queries, service, subtitles
from app.plex import paths as plex_paths
from app.plex.lookups import by_rating_key
from app.plex import people as plex_people
from app.plex import stream as plex_stream
from app.plex import watch_sync as plex_watch
@@ -40,6 +32,9 @@ from app.routes.plex._shared import item_or_404
from app.refusal import refused
from app.routes._common import ProgressIn, not_found
if TYPE_CHECKING:
from app.plex.models import PlexItem
router = APIRouter(prefix="/api/plex", tags=["plex"])
@@ -280,14 +275,7 @@ def _push_watch(
return
# Capture the row's freshest timestamp NOW so the background push can tell if a newer local edit
# lands before it settles synced_to_plex (which would otherwise strand that newer value).
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user_id, PlexState.item_id == item_id)
)
.scalars()
.first()
)
marker = plex_watch._state_marker(st)
marker = plex_watch._state_marker(queries.state(db, user_id, item_id))
if action == "resume" and intent is not None:
background.add_task(
_push_resume_guarded, user_id, item_id, rating_key, time_ms, duration_ms, marker, intent
@@ -305,37 +293,6 @@ def _push_watch(
)
def _mark_finished(db: Session, user_id: int, item_id: int, now: datetime) -> bool:
"""Mark the item watched (position cleared) and say whether THIS call made it so. Two near-end
checkpoints arrive together routinely (the 10 s tick and the debounced `seeked` flush), and a
read-then-write let both see "not watched" and both scrobble. The transition is decided by the
write itself: an insert that did not conflict, or an update conditional on the old status —
under READ COMMITTED the second writer re-reads the committed row and matches nothing."""
values = {
"status": "watched",
"watched_at": now,
"position_seconds": 0,
"progress_updated_at": now,
"synced_to_plex": False,
}
inserted = db.execute(
pg_insert(PlexState)
.values(user_id=user_id, item_id=item_id, **values)
.on_conflict_do_nothing(constraint="uq_plex_user_item")
.returning(PlexState.id)
).first()
if inserted is not None:
return True
mine = (PlexState.user_id == user_id) & (PlexState.item_id == item_id)
if db.execute(
update(PlexState).where(mine, PlexState.status != "watched").values(values)
).rowcount:
return True
# Already watched: a re-finish still restamps the row, as it always did — just no scrobble.
db.execute(update(PlexState).where(mine).values(values))
return False
class ItemProgressIn(ProgressIn):
"""The shared checkpoint (its bounds included), plus whether the player is leaving the item."""
@@ -371,7 +328,7 @@ def item_progress(
now = datetime.now(timezone.utc)
near_end = duration > 0 and position > duration - _FINISH_MARGIN_S
if near_end:
flipped = _mark_finished(db, user.id, it.id, now)
flipped = service.mark_finished(db, user.id, it.id, now)
# The finish supersedes every resume push still in flight for this item: stamping it as the
# latest push makes `_resume_push_superseded` drop them. Popping the entry instead (to keep
# the map small) removed the very guard, and an older resume push could then land at Plex
@@ -385,33 +342,17 @@ def item_progress(
_push_watch(background, db, user.id, it.id, rating_key, "watched")
return {"ok": True, "status": "watched", "position_seconds": 0}
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user.id, PlexState.item_id == it.id)
)
.scalars()
.first()
)
st = queries.state(db, user, it.id)
if position < _PROGRESS_MIN_S:
if st is not None and st.status == "new":
db.delete(st)
elif st is not None:
st.position_seconds = 0
st.progress_updated_at = now
service.clear_progress(db, st, now)
db.commit()
return {"ok": True, "position_seconds": 0}
if st is None:
st = PlexState(user_id=user.id, item_id=it.id)
db.add(st)
st.position_seconds = position
st.progress_updated_at = now
push_now = _should_push_resume(user.id, it.id, now, final)
if push_now:
# Mark dirty before the (background) push so the reconcile knows to re-push if it fails, and so
# `push_state_to_plex` can settle synced_to_plex once Plex confirms.
st.synced_to_plex = False
# Marked dirty before the (background) push so the reconcile knows to re-push if it fails, and so
# `push_state_to_plex` can settle synced_to_plex once Plex confirms.
service.checkpoint(db, user, it.id, st, position, now, dirty=push_now)
db.commit()
if push_now:
_push_watch(
@@ -447,13 +388,7 @@ def item_state(
status = body.status
if status not in ("new", "watched", "hidden"):
raise refused(400, "invalid_status")
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user.id, PlexState.item_id == it.id)
)
.scalars()
.first()
)
st = queries.state(db, user, it.id)
prev_status = st.status if st is not None else "new"
if status == "new":
# Does Plex hold anything for this item that the un-mark has to clear? Two things qualify: a
@@ -467,19 +402,12 @@ def item_state(
st.status == "watched" or (st.position_seconds or 0) > 0
)
if st is not None:
db.delete(st)
service.forget_state(db, st)
db.commit()
if mirrored_on_plex:
_push_watch(background, db, user.id, it.id, rating_key, "unwatched")
return {"ok": True, "status": "new"}
if st is None:
st = PlexState(user_id=user.id, item_id=it.id)
db.add(st)
st.status = status
if status == "watched":
st.watched_at = datetime.now(timezone.utc)
st.position_seconds = 0
st.synced_to_plex = False
service.set_status(db, user, it.id, st, status)
db.commit()
if status == "watched" and prev_status != "watched":
_push_watch(background, db, user.id, it.id, rating_key, "watched")
@@ -490,44 +418,15 @@ def _bulk_state(
background: BackgroundTasks,
db: Session,
user_id: int,
eps: list[PlexItem],
eps: "list[PlexItem]",
watched: bool,
) -> int:
"""Mark every episode in `eps` watched (or unwatched) for a user, mirroring each change to a
linked Plex account in the background (best-effort). Returns how many rows actually changed."""
now = datetime.now(timezone.utc)
existing = {
s.item_id: s
for s in db.execute(
select(PlexState).where(
PlexState.user_id == user_id,
PlexState.item_id.in_([e.id for e in eps] or [0]),
)
).scalars()
}
linked Plex account in the background (best-effort). Returns how many rows actually changed.
The handler commits before the response — which is when the background push runs, so the push
always reads the committed marks."""
pushable = plex_watch.link_for_push(db, user_id) is not None
to_push: list[tuple[int, str, str]] = []
for e in eps:
st = existing.get(e.id)
prev = st.status if st is not None else "new"
if watched:
if st is None:
st = PlexState(user_id=user_id, item_id=e.id)
db.add(st)
st.status = "watched"
st.watched_at = now
st.position_seconds = 0
st.synced_to_plex = False
if prev != "watched":
to_push.append((e.id, e.rating_key, "watched"))
else: # → new: drop any state (but keep the user's private "hidden" marks)
if st is not None and prev != "hidden":
db.delete(st)
if prev != "new":
to_push.append((e.id, e.rating_key, "unwatched"))
# Flush, not commit: the handler commits, and it does so before the response — which is when
# the background push below runs, so the push always reads the committed marks.
db.flush()
to_push = service.bulk_state(db, user_id, eps, watched)
if pushable and to_push:
# One coalesced background task for the whole batch (reuses a single session + Plex client),
# not one task per episode — a 200-ep show mark is 1 task, not 200.
@@ -548,15 +447,11 @@ def show_state(
db: Session = Depends(get_db),
) -> dict:
"""Mark a whole show watched/unwatched (all its episodes) for the current user + push to Plex."""
sh = by_rating_key(db, PlexShow, rating_key)
sh = queries.show(db, rating_key)
if sh is None:
raise not_found("show")
watched = body.watched
eps = (
db.execute(select(PlexItem).where(PlexItem.show_id == sh.id, PlexItem.kind == "episode"))
.scalars()
.all()
)
eps = queries.episodes(db, sh)
changed = _bulk_state(background, db, user.id, eps, watched)
db.commit()
return {"ok": True, "changed": changed, "watched": watched}
@@ -571,15 +466,11 @@ def season_state(
db: Session = Depends(get_db),
) -> dict:
"""Mark a whole season watched/unwatched (all its episodes) for the current user + push to Plex."""
se = by_rating_key(db, PlexSeason, rating_key)
se = queries.season(db, rating_key)
if se is None:
raise not_found("season")
watched = body.watched
eps = (
db.execute(select(PlexItem).where(PlexItem.season_id == se.id, PlexItem.kind == "episode"))
.scalars()
.all()
)
eps = queries.season_episodes(db, se)
changed = _bulk_state(background, db, user.id, eps, watched)
db.commit()
return {"ok": True, "changed": changed, "watched": watched}
+54 -268
View File
@@ -1,33 +1,26 @@
"""Plex playlists: the current user's personal, orderable playlists of playable items."""
from datetime import datetime, timezone
from typing import TYPE_CHECKING, Annotated
from fastapi import APIRouter, Depends
from typing import Annotated
from pydantic import BaseModel, Field, model_validator
from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from app.auth import current_user
from app.db import get_db
from app.models import (
PlexItem,
PlexPlaylist,
PlexPlaylistItem,
PlexShow,
PlexState,
User,
)
from app.models import User
from app.plex import queries, rollup, service
from app.plex.cards import display_title, leaf_card, norm_lang
from app.plex.lookups import by_rating_key
from app.routes._common import owned_or_404, not_found
from app.refusal import refused
from app.routes._common import not_found
if TYPE_CHECKING:
from app.plex.models import PlexPlaylist
router = APIRouter(prefix="/api/plex", tags=["plex"])
def _playlist_card(pl: PlexPlaylist, count: int, thumb_rk: str | None) -> dict:
def _playlist_card(pl: "PlexPlaylist", count: int, thumb_rk: str | None) -> dict:
return {
"id": pl.id,
"title": pl.title,
@@ -36,13 +29,15 @@ def _playlist_card(pl: PlexPlaylist, count: int, thumb_rk: str | None) -> dict:
}
def _playlist_item_query(pid: int):
return (
select(PlexPlaylistItem, PlexItem)
.join(PlexItem, PlexItem.id == PlexPlaylistItem.item_id)
.where(PlexPlaylistItem.playlist_id == pid)
.order_by(PlexPlaylistItem.position, PlexPlaylistItem.id)
)
def _card_of(db: Session, pl: "PlexPlaylist") -> dict:
return _playlist_card(pl, queries.playlist_count(db, pl.id), queries.playlist_cover(db, pl.id))
def _own_playlist(db: Session, user: User, pid: int, *, lock: bool = False) -> "PlexPlaylist":
pl = queries.own_playlist(db, user, pid, lock=lock)
if pl is None:
raise not_found("playlist")
return pl
@router.get("/playlists")
@@ -56,49 +51,17 @@ def playlists(
`contains`=an item rating_key adds `has_item` per playlist (for the single 'add to playlist' dialog).
`contains_group`=a comma-separated list of rating_keys adds `group_in` per playlist (how many of the
group it already holds) + a top-level `group_size`, for the 'add a whole season/show' dialog."""
contains_id = None
if contains:
contains_id = db.scalar(select(PlexItem.id).where(PlexItem.rating_key == str(contains)))
contains_id = queries.item_id(db, contains) if contains else None
group_ids: set[int] | None = None
if contains_group is not None:
rks = [s for s in (contains_group or "").split(",") if s]
group_ids = set(
db.execute(select(PlexItem.id).where(PlexItem.rating_key.in_(rks or [""]))).scalars()
)
group_ids = queries.item_ids(db, [s for s in (contains_group or "").split(",") if s])
out = []
for pl in db.execute(
select(PlexPlaylist)
.where(PlexPlaylist.user_id == user.id)
.order_by(PlexPlaylist.updated_at.desc())
).scalars():
count = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(PlexPlaylistItem.playlist_id == pl.id)
)
or 0
)
first = db.execute(_playlist_item_query(pl.id)).first()
card = _playlist_card(pl, count, first[1].rating_key if first else None)
for pl in queries.playlists_of(db, user):
card = _card_of(db, pl)
if contains_id is not None:
card["has_item"] = (
db.execute(
select(PlexPlaylistItem.id).where(
PlexPlaylistItem.playlist_id == pl.id,
PlexPlaylistItem.item_id == contains_id,
)
).first()
is not None
)
card["has_item"] = queries.playlist_holds(db, pl.id, {contains_id}) > 0
if group_ids is not None:
card["group_in"] = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(
PlexPlaylistItem.playlist_id == pl.id,
PlexPlaylistItem.item_id.in_(group_ids or [0]),
)
)
or 0
)
card["group_in"] = queries.playlist_holds(db, pl.id, group_ids)
out.append(card)
resp: dict = {"playlists": out}
if group_ids is not None:
@@ -120,19 +83,10 @@ def create_playlist(
title = body.title.strip()
if not title:
raise refused(400, "title_required")
pl = PlexPlaylist(user_id=user.id, title=title)
db.add(pl)
db.flush()
seed = body.item_rating_key
count = 0
thumb_rk = None
if seed:
it = by_rating_key(db, PlexItem, seed)
if it is not None:
db.add(PlexPlaylistItem(playlist_id=pl.id, item_id=it.id, position=0))
count, thumb_rk = 1, it.rating_key
seed = queries.item(db, body.item_rating_key) if body.item_rating_key else None
pl = service.create_playlist(db, user, title, seed)
db.commit()
return _playlist_card(pl, count, thumb_rk)
return _playlist_card(pl, 1 if seed else 0, seed.rating_key if seed else None)
@router.get("/playlists/{pid}")
@@ -143,26 +97,10 @@ def playlist_detail(
db: Session = Depends(get_db),
) -> dict:
lang = norm_lang(lang)
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
rows = db.execute(_playlist_item_query(pid)).all()
items = [it for (_pi, it) in rows]
states = {
s.item_id: s
for s in db.execute(
select(PlexState).where(
PlexState.user_id == user.id,
PlexState.item_id.in_([it.id for it in items] or [0]),
)
).scalars()
}
shows = {
sh.id: sh
for sh in db.execute(
select(PlexShow).where(
PlexShow.id.in_([it.show_id for it in items if it.show_id] or [0])
)
).scalars()
}
pl = _own_playlist(db, user, pid)
items = queries.playlist_items(db, pl.id)
states = rollup.item_state_map(db, user, [it.id for it in items])
shows = queries.shows_by_ids(db, {it.show_id for it in items if it.show_id})
cards = [
leaf_card(
it,
@@ -187,28 +125,21 @@ def rename_playlist(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
pl = _own_playlist(db, user, pid)
title = body.title.strip()
if not title:
raise refused(400, "title_required")
pl.title = title
service.rename_playlist(pl, title)
db.commit()
count = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(PlexPlaylistItem.playlist_id == pl.id)
)
or 0
)
first = db.execute(_playlist_item_query(pl.id)).first()
return _playlist_card(pl, count, first[1].rating_key if first else None)
return _playlist_card(pl, queries.playlist_count(db, pl.id), queries.playlist_cover(db, pl.id))
@router.delete("/playlists/{pid}")
def delete_playlist(
pid: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
db.delete(pl)
pl = _own_playlist(db, user, pid)
service.delete_playlist(db, pl)
db.commit()
return {"deleted": pid}
@@ -225,27 +156,11 @@ def playlist_add_item(
db: Session = Depends(get_db),
) -> dict:
# Locked (like every add / reorder here): two adds otherwise read the same max position.
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist", lock=True)
it = by_rating_key(db, PlexItem, body.item_rating_key)
pl = _own_playlist(db, user, pid, lock=True)
it = queries.item(db, body.item_rating_key)
if it is None:
raise not_found("item")
exists = (
db.execute(
select(PlexPlaylistItem).where(
PlexPlaylistItem.playlist_id == pid, PlexPlaylistItem.item_id == it.id
)
)
.scalars()
.first()
)
if exists is None:
maxpos = db.scalar(
select(func.coalesce(func.max(PlexPlaylistItem.position), -1)).where(
PlexPlaylistItem.playlist_id == pid
)
)
db.add(PlexPlaylistItem(playlist_id=pid, item_id=it.id, position=int(maxpos) + 1))
pl.updated_at = datetime.now(timezone.utc)
service.add_item(db, pl, it)
db.commit()
return {"ok": True}
@@ -259,15 +174,10 @@ def playlist_remove_item(
) -> dict:
# Locked like the adds and the reorder: the list row first, then its items — a remove that
# locked an item row before a reorder locked the list deadlocked against it.
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist", lock=True)
it = by_rating_key(db, PlexItem, item_rating_key)
pl = _own_playlist(db, user, pid, lock=True)
it = queries.item(db, item_rating_key)
if it is not None:
db.execute(
delete(PlexPlaylistItem).where(
PlexPlaylistItem.playlist_id == pid, PlexPlaylistItem.item_id == it.id
)
)
pl.updated_at = datetime.now(timezone.utc)
service.remove_item(db, pl, it)
db.commit()
return {"ok": True}
@@ -276,25 +186,6 @@ class PlexItemKeysIn(BaseModel):
item_rating_keys: list[str] = []
def _items_by_rks(db: Session, rks: list) -> list[PlexItem]:
"""Resolve rating_keys to PlexItems, preserving the input order (drops unknown keys)."""
order = [str(k) for k in rks if str(k)]
if not order:
return []
by_rk = {
it.rating_key: it
for it in db.execute(select(PlexItem).where(PlexItem.rating_key.in_(order))).scalars()
}
seen: set[str] = set()
out: list[PlexItem] = []
for rk in order:
it = by_rk.get(rk)
if it is not None and rk not in seen:
seen.add(rk)
out.append(it)
return out
class PlexItemAddBulkIn(PlexItemKeysIn):
# Where each key goes back (the list page's "Undo" of a removal): an index over the
# ORDERED rows, one per key. The two-call shape (append, then send the whole order) leaves the rows
@@ -309,32 +200,6 @@ class PlexItemAddBulkIn(PlexItemKeysIn):
return self
def _insert_at(db: Session, pid: int, placed: list[tuple[int, PlexItem]]) -> int:
"""Insert new rows at their indices (ascending, so each index is where the row stood when the
rows before it were back), renumber the whole list 0..n-1, and say how many were added. An item
already in the list does not move: the add stays idempotent."""
rows = list(
db.execute(
select(PlexPlaylistItem)
.where(PlexPlaylistItem.playlist_id == pid)
.order_by(PlexPlaylistItem.position, PlexPlaylistItem.id)
).scalars()
)
present = {r.item_id for r in rows}
added = 0
for at, it in sorted(placed, key=lambda p: p[0]):
if it.id in present:
continue
row = PlexPlaylistItem(playlist_id=pid, item_id=it.id, position=0)
db.add(row)
rows.insert(min(at, len(rows)), row)
present.add(it.id)
added += 1
for i, r in enumerate(rows):
r.position = i
return added
@router.post("/playlists/{pid}/items/bulk")
def playlist_add_bulk(
pid: int,
@@ -345,56 +210,20 @@ def playlist_add_bulk(
"""Append many items at once (a whole season/show). Already-present items are skipped; the input
order is preserved for the newly-added ones. With `positions`, each key is put back at its index
instead of appended. Returns how many were added + the new total."""
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist", lock=True)
pl = _own_playlist(db, user, pid, lock=True)
items = queries.items_by_keys(db, body.item_rating_keys)
if body.positions is not None:
by_rk = {it.rating_key: it for it in _items_by_rks(db, body.item_rating_keys)}
by_rk = {it.rating_key: it for it in items}
placed = [
(at, by_rk[str(rk)])
for rk, at in zip(body.item_rating_keys, body.positions, strict=True)
if str(rk) in by_rk
]
added = _insert_at(db, pid, placed)
if added:
pl.updated_at = datetime.now(timezone.utc)
db.commit()
total = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(PlexPlaylistItem.playlist_id == pid)
)
or 0
)
return {"ok": True, "added": added, "item_count": total}
items = _items_by_rks(db, body.item_rating_keys)
existing = set(
db.execute(
select(PlexPlaylistItem.item_id).where(PlexPlaylistItem.playlist_id == pid)
).scalars()
)
pos = int(
db.scalar(
select(func.coalesce(func.max(PlexPlaylistItem.position), -1)).where(
PlexPlaylistItem.playlist_id == pid
)
)
)
added = 0
for it in items:
if it.id in existing:
continue
pos += 1
db.add(PlexPlaylistItem(playlist_id=pid, item_id=it.id, position=pos))
existing.add(it.id)
added += 1
if added:
pl.updated_at = datetime.now(timezone.utc)
added = service.insert_items_at(db, pl, placed)
else:
added = service.add_items(db, pl, items)
db.commit()
total = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(PlexPlaylistItem.playlist_id == pid)
)
or 0
)
return {"ok": True, "added": added, "item_count": total}
return {"ok": True, "added": added, "item_count": queries.playlist_count(db, pid)}
@router.post("/playlists/{pid}/items/remove-bulk")
@@ -406,26 +235,11 @@ def playlist_remove_bulk(
) -> dict:
"""Remove many items at once (a whole season/show, or a multi-select). Returns how many were
removed + the new total."""
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist", lock=True)
ids = [it.id for it in _items_by_rks(db, body.item_rating_keys)]
removed = 0
if ids:
removed = db.execute(
delete(PlexPlaylistItem).where(
PlexPlaylistItem.playlist_id == pid, PlexPlaylistItem.item_id.in_(ids)
),
execution_options={"synchronize_session": False},
).rowcount
if removed:
pl.updated_at = datetime.now(timezone.utc)
pl = _own_playlist(db, user, pid, lock=True)
ids = [it.id for it in queries.items_by_keys(db, body.item_rating_keys)]
removed = service.remove_items(db, pl, ids)
db.commit()
total = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(PlexPlaylistItem.playlist_id == pid)
)
or 0
)
return {"ok": True, "removed": removed, "item_count": total}
return {"ok": True, "removed": removed, "item_count": queries.playlist_count(db, pid)}
@router.put("/playlists/{pid}/order")
@@ -436,35 +250,7 @@ def reorder_playlist(
db: Session = Depends(get_db),
) -> dict:
"""Set the playlist order from an explicit list of item rating_keys (0-based positions)."""
pl = owned_or_404(db, PlexPlaylist, pid, user, what="playlist", lock=True)
order = body.item_rating_keys
id_by_rk = {
it.rating_key: it.id
for it in db.execute(
select(PlexItem).where(PlexItem.rating_key.in_(order or [""]))
).scalars()
}
rows = {
r.item_id: r
for r in db.execute(
select(PlexPlaylistItem).where(PlexPlaylistItem.playlist_id == pid)
).scalars()
}
pos = 0
seen: set[int] = set()
for rk in order:
row = rows.get(id_by_rk.get(rk, -1))
if row is not None and row.item_id not in seen:
row.position = pos
pos += 1
seen.add(row.item_id)
# Rows the payload omitted (added in another tab meanwhile) keep their relative order after
# the given ones, so no two rows share a position (the YouTube reorder does the same).
for row in sorted(
(r for iid, r in rows.items() if iid not in seen), key=lambda r: (r.position, r.id)
):
row.position = pos
pos += 1
pl.updated_at = datetime.now(timezone.utc)
pl = _own_playlist(db, user, pid, lock=True)
service.reorder_playlist(db, pl, body.item_rating_keys)
db.commit()
return {"ok": True}
+27 -32
View File
@@ -6,23 +6,26 @@ the app or any account data. Password-protected links exchange the password once
a short-lived signed grant that the media URL carries (a `<video src>` can't send a header).
"""
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends, Request
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import select, update
from sqlalchemy.orm import Session
from app.config import settings
from app.db import get_db
from app.downloads import links as linksmod
from app.downloads import storage
from app.downloads import queries, storage
from app.downloads.status import JobStatus
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset, User
from app.ratelimit import RateLimiter, client_key
from app.security import verify_password
from app.refusal import refused
from app.routes._common import not_found
if TYPE_CHECKING:
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
router = APIRouter(prefix="/api/public", tags=["public"])
# Slow password guessing on a link (the token itself is 192-bit, so it needs no throttle). Counted per
@@ -37,26 +40,25 @@ _unlock_limit = RateLimiter(max_events=10, window_seconds=300)
_unlock_failures = RateLimiter(max_events=50, window_seconds=300)
def _link_or_404(db: Session, token: str) -> DownloadLink:
def _link_or_404(db: Session, token: str) -> "DownloadLink":
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
raise refused(404, "link_gone")
return link
def _being_made(db: Session, job: DownloadJob) -> bool:
def _being_made(db: Session, job: "DownloadJob") -> bool:
"""The job is running, or queued for a worker that will take it: the worker leaves a suspended
owner's queued jobs where they are (`worker._claim_job`), as long as the suspension lasts."""
if job.status == JobStatus.RUNNING:
return True
if job.status != JobStatus.QUEUED:
return False
owner = db.get(User, job.user_id)
return owner is not None and not owner.is_suspended
return not queries.owner_suspended(db, job)
def _ready_job(db: Session, job: DownloadJob | None) -> tuple[DownloadJob, MediaAsset]:
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
def _ready_job(db: Session, job: "DownloadJob | None") -> tuple["DownloadJob", "MediaAsset"]:
asset = queries.asset_of(db, job)
if (
job is not None
and asset is not None
@@ -78,25 +80,24 @@ def _ready_job(db: Session, job: DownloadJob | None) -> tuple[DownloadJob, Media
return job, asset
def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAsset]:
def _ready_target(db: Session, link: "DownloadLink") -> tuple["DownloadJob", "MediaAsset"]:
"""A file link's file. A folder link names no one file: its files are asked for by id."""
if link.job_id is None:
raise refused(404, "video_gone")
return _ready_job(db, db.get(DownloadJob, link.job_id))
return _ready_job(db, queries.job(db, link.job_id))
def _link_folder(db: Session, link: DownloadLink) -> FileFolder:
folder = db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
def _link_folder(db: Session, link: "DownloadLink") -> "FileFolder":
folder = queries.link_folder(db, link)
if folder is None:
raise refused(404, "link_gone")
return folder
def _folder_job(db: Session, link: DownloadLink, job_id: int) -> tuple[DownloadJob, MediaAsset]:
folder = _link_folder(db, link)
job = db.execute(
select(DownloadJob).where(DownloadJob.id == job_id, *linksmod.in_folder(folder))
).scalar_one_or_none()
def _folder_job(
db: Session, link: "DownloadLink", job_id: int
) -> tuple["DownloadJob", "MediaAsset"]:
job = queries.folder_job(db, _link_folder(db, link), job_id)
if job is None:
raise refused(404, "video_gone")
return _ready_job(db, job)
@@ -113,7 +114,7 @@ def _file_url(token: str, grant: str | None = None, job_id: int | None = None) -
def _poster_url(
token: str, asset: MediaAsset, grant: str | None = None, job_id: int | None = None
token: str, asset: "MediaAsset", grant: str | None = None, job_id: int | None = None
) -> str | None:
if not asset.poster_path:
return None
@@ -121,7 +122,7 @@ def _poster_url(
return f"{url}?g={grant}" if grant else url
def _folder_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
def _folder_meta(db: Session, link: "DownloadLink", token: str, grant: str | None) -> dict:
"""A folder link's page: the folder's name and each file's own watch metadata
(`linksmod.folder_files`: ready, on disk, newest first)."""
folder = _link_folder(db, link)
@@ -147,7 +148,7 @@ def _folder_meta(db: Session, link: DownloadLink, token: str, grant: str | None)
}
def _file_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
def _file_meta(db: Session, link: "DownloadLink", token: str, grant: str | None) -> dict:
job, asset = _ready_target(db, link)
return {
"needs_password": False,
@@ -158,20 +159,14 @@ def _file_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -
}
def _opened(db: Session, link: DownloadLink, token: str, grant: str | None = None) -> dict:
def _opened(db: Session, link: "DownloadLink", token: str, grant: str | None = None) -> dict:
"""The page an open link shows, counted as one view (the handler commits the count)."""
meta = (
_folder_meta(db, link, token, grant)
if link.folder_id is not None
else _file_meta(db, link, token, grant)
)
# In SQL, not `+= 1` on the loaded row: two opens at once would both write the same count.
db.execute(
update(DownloadLink)
.where(DownloadLink.id == link.id)
.values(view_count=DownloadLink.view_count + 1)
.execution_options(synchronize_session=False)
)
linksmod.count_view(db, link)
return meta
@@ -217,13 +212,13 @@ def watch_unlock(
return {"ok": True, **meta}
def _granted(link: DownloadLink, token: str, g: str | None) -> None:
def _granted(link: "DownloadLink", token: str, g: str | None) -> None:
"""A password link's files want the grant its unlock handed out."""
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
raise refused(403, "share_password_required")
def _serve(link: DownloadLink, asset: MediaAsset) -> FileResponse:
def _serve(link: "DownloadLink", asset: "MediaAsset") -> FileResponse:
"""Range-aware stream of a shared file. Inline by default (plays in the page); an
`allow_download` link serves it as an attachment."""
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
@@ -252,7 +247,7 @@ def _serve(link: DownloadLink, asset: MediaAsset) -> FileResponse:
)
def _poster(asset: MediaAsset) -> FileResponse:
def _poster(asset: "MediaAsset") -> FileResponse:
if not asset.poster_path:
raise not_found("poster")
path = storage.safe_abs_path(settings.files_root, asset.poster_path)
+22 -10
View File
@@ -6,14 +6,17 @@ The aggregate daily budget lives in ApiQuotaUsage; here we read the per-spend au
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, Query
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.userscope import owned_by
from app import quota
from app.auth import admin_user, current_user
from app.db import get_db
from app.models import QuotaEvent, User
from app.routes import _account as account
from app.routes._keyset import utc
router = APIRouter(prefix="/api/quota", tags=["quota"])
@@ -25,7 +28,7 @@ def _sum(db: Session, *conds) -> int:
@router.get("/my-usage")
def my_usage(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
now = datetime.now(timezone.utc)
mine = QuotaEvent.user_id == user.id
mine = owned_by(QuotaEvent.user_id, user)
by_action = {
action: int(units)
for action, units in db.execute(
@@ -45,23 +48,32 @@ def my_usage(user: User = Depends(current_user), db: Session = Depends(get_db))
@router.get("/admin")
def admin_usage(
days: int = 30,
since: datetime | None = None,
until: datetime | None = None,
user: str | None = Query(default=None, pattern=account.PATTERN),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
days = max(1, min(days, 365))
since = datetime.now(timezone.utc) - timedelta(days=days)
"""Spend per account (and per action) and per day, between `since` (default: 30 days back) and
`until` (default: now), optionally for one account (`none` = the background)."""
now = datetime.now(timezone.utc)
lo = utc(since) or now - timedelta(days=30)
window = [QuotaEvent.created_at >= lo]
if until is not None:
window.append(QuotaEvent.created_at < utc(until))
if user is not None:
window.append(account.condition(QuotaEvent.user_id, user))
totals = db.execute(
select(QuotaEvent.user_id, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(QuotaEvent.user_id)
).all()
per_action: dict[int | None, dict[str, int]] = {}
for uid, action, units in db.execute(
select(QuotaEvent.user_id, QuotaEvent.action, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(QuotaEvent.user_id, QuotaEvent.action)
).all():
per_action.setdefault(uid, {})[action] = int(units)
@@ -84,16 +96,16 @@ def admin_usage(
]
rows.sort(key=lambda r: r["total"], reverse=True)
# Instance-wide daily series in Pacific days (matches the budget reset).
# The daily series in Pacific days (matches the budget reset), of the one account when filtered.
day_expr = func.date(func.timezone("America/Los_Angeles", QuotaEvent.created_at))
daily = [
{"day": str(day), "total": int(units)}
for day, units in db.execute(
select(day_expr, func.sum(QuotaEvent.units))
.where(QuotaEvent.created_at >= since)
.where(*window)
.group_by(day_expr)
.order_by(day_expr)
).all()
]
return {"range_days": days, "rows": rows, "daily": daily}
return {"rows": rows, "daily": daily}
+154
View File
@@ -0,0 +1,154 @@
"""Admin background-run history: read `job_runs`, a page at a time.
Admin-only and read-only — rows leave the table through the retention job
(`job_run_retention_days`). Filtered on the server and paged by `_keyset`, like the server log and
the audit log beside it on the activity page. A row names what a reader needs without a second
request: the channel a user's ad-hoc run worked on, and how many log lines the run left (only
WARNING and up are stored, so most runs have none — the count says so before the click).
"""
from dataclasses import dataclass
from datetime import datetime
from typing import Literal
from fastapi import APIRouter, Depends, Query
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app import scheduler
from app.auth import admin_user
from app.db import get_db
from app.models import JobRun, LogEvent, User
from app.catalog import queries as catalog_queries
from app.routes import _account as account
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
from app.runstatus import RunStatus
from app.utils import iso, like_escape
router = APIRouter(prefix="/api/admin/runs", tags=["admin"])
Facet = Literal["job", "status"]
def _serialize(run: JobRun, titles: dict[str, str], lines: dict[int, int]) -> dict:
return {
"id": run.id,
"job": run.job,
"trigger": run.trigger,
"actor_id": run.actor_id,
"target": run.target,
"target_title": titles.get(run.target) if run.target is not None else None,
"status": run.status,
"started_at": iso(run.started_at),
"finished_at": iso(run.finished_at),
"summary": run.summary,
"result": run.result,
"error": run.error,
"failure": run.failure,
"quota_units": run.quota_units,
"log_lines": lines.get(run.id, 0),
}
@dataclass(frozen=True)
class RunFilters:
job: str | None
status: RunStatus | None
actor: str | None
q: str | None
since: datetime | None
until: datetime | None
def conditions(self, skip: Facet | None = None) -> list:
"""The WHERE conditions, ANDed. `skip` leaves one menu's own filter out, so that menu counts
its values under every other filter (the way back from a pick)."""
out = []
if self.job is not None and skip != "job":
out.append(JobRun.job == self.job)
if self.status is not None and skip != "status":
out.append(JobRun.status == self.status)
if self.actor is not None:
out.append(account.condition(JobRun.actor_id, self.actor))
if self.since is not None:
out.append(JobRun.started_at >= utc(self.since))
if self.until is not None:
out.append(JobRun.started_at < utc(self.until))
if self.q and self.q.strip():
like = f"%{like_escape(self.q.strip())}%"
out.append(
or_(
JobRun.job.ilike(like),
JobRun.target.ilike(like),
JobRun.summary.ilike(like),
JobRun.error.ilike(like),
)
)
return out
def run_filters(
job: str | None = Query(default=None, max_length=40, pattern=r"^[a-z][a-z0-9_]*$"),
status: RunStatus | None = None,
user: str | None = Query(default=None, pattern=account.PATTERN),
q: str | None = Query(default=None, max_length=200),
since: datetime | None = None,
until: datetime | None = None,
) -> RunFilters:
return RunFilters(job, status, user, q, since, until)
@router.get("")
def list_runs(
filters: RunFilters = Depends(run_filters),
before: str | None = Query(default=None, max_length=200),
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""One page of runs, newest first, matching every filter given; `before` continues from a
previous page's `next_cursor`."""
query = select(JobRun).where(*filters.conditions())
if before is not None:
query = query.where(older_than(JobRun.started_at, JobRun.id, before))
order = (JobRun.started_at.desc(), JobRun.id.desc())
runs, cursor = page(
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
limit,
lambda r: r.started_at,
)
channels = {r.target for r in runs if r.job in scheduler.AD_HOC_JOBS and r.target}
titles = catalog_queries.channel_titles(db, channels)
lines: dict[int, int] = {}
if runs:
counted = db.execute(
select(LogEvent.run_id, func.count())
.where(LogEvent.run_id.in_([r.id for r in runs]))
.group_by(LogEvent.run_id)
)
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
return {
"items": [_serialize(r, titles, lines) for r in runs],
"accounts": accounts(db, {r.actor_id for r in runs if r.actor_id is not None}),
"next_cursor": cursor,
}
@router.get("/facets")
def run_facets(
filters: RunFilters = Depends(run_filters),
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""What the job menu and the status chips offer, each counted under every filter but its own."""
def counts(col, skip: Facet) -> list[dict]:
n = func.count().label("n")
stmt = select(col, n).where(*filters.conditions(skip)).group_by(col).order_by(n.desc(), col)
return [{"value": v, "count": c} for v, c in db.execute(stmt).all()]
total = select(func.count()).select_from(JobRun).where(*filters.conditions("status"))
return {
"jobs": counts(JobRun.job, "job"),
"statuses": counts(JobRun.status, "status"),
"total": db.scalar(total) or 0,
}
+29 -65
View File
@@ -1,15 +1,20 @@
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends
from pydantic import BaseModel, field_validator
from sqlalchemy import func, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.auth import current_user
from app.db import get_db
from app.models import SavedView, User
from app.routes._common import owned_or_404
from app.models import User
from app.reader import queries, service
from app.routes._common import not_found
from app.refusal import refused
if TYPE_CHECKING:
from app.reader.models import SavedView
# Saved "smart views": per-user named snapshots of a module's filters (feed or plex, keyed by
# `source`).
router = APIRouter(prefix="/api/saved-views", tags=["saved-views"])
@@ -23,7 +28,7 @@ def _norm_source(source: str) -> str:
return source if source in _SOURCES else "feed"
def _serialize(v: SavedView) -> dict:
def _serialize(v: "SavedView") -> dict:
return {
"id": v.id,
"name": v.name,
@@ -33,23 +38,20 @@ def _serialize(v: SavedView) -> dict:
}
def _own_view(db: Session, user: User, view_id: int) -> "SavedView":
view = queries.own_view(db, user, view_id)
if view is None:
raise not_found("view")
return view
@router.get("")
def list_views(
source: str = "feed",
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> list[dict]:
source = _norm_source(source)
rows = (
db.execute(
select(SavedView)
.where(SavedView.user_id == user.id, SavedView.source == source)
.order_by(SavedView.position, SavedView.id)
)
.scalars()
.all()
)
return [_serialize(v) for v in rows]
return [_serialize(v) for v in queries.views(db, user, _norm_source(source))]
# A feed view's tag pick is read by more than the feed: the autotag job keeps every tag a view names
@@ -86,22 +88,8 @@ def create_view(
name = body.name.strip()
if not name:
raise refused(400, "name_required")
source = _norm_source(body.source)
# Position is per (user, source), so each module's views number 0..N independently.
maxpos = db.execute(
select(func.coalesce(func.max(SavedView.position), -1)).where(
SavedView.user_id == user.id, SavedView.source == source
)
).scalar_one()
view = SavedView(
user_id=user.id,
source=source,
name=name[:80],
filters=body.filters,
position=maxpos + 1,
)
db.add(view)
try:
view = service.create_view(db, user, _norm_source(body.source), name, body.filters)
db.commit()
except IntegrityError:
db.rollback()
@@ -126,32 +114,17 @@ def update_view(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
view = owned_or_404(db, SavedView, view_id, user, what="view")
view = _own_view(db, user, view_id)
sent = body.model_fields_set
if "name" in sent:
name = (body.name or "").strip()
if not name:
raise refused(400, "name_required")
view.name = name[:80]
if "filters" in sent:
if body.filters is None:
raise refused(400, "filters_not_object")
view.filters = body.filters
if "is_default" in sent:
make_default = bool(body.is_default)
if make_default:
# At most one default per (user, source): clear the flag on this module's others first.
db.execute(
update(SavedView)
.where(
SavedView.user_id == user.id,
SavedView.source == view.source,
SavedView.id != view.id,
)
.values(is_default=False)
)
view.is_default = make_default
name = (body.name or "").strip()
if "name" in sent and not name:
raise refused(400, "name_required")
if "filters" in sent and body.filters is None:
raise refused(400, "filters_not_object")
try:
service.update_view(
db, view, sent=sent, name=name, filters=body.filters, is_default=body.is_default
)
db.commit()
except IntegrityError:
db.rollback()
@@ -165,8 +138,7 @@ def delete_view(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
view = owned_or_404(db, SavedView, view_id, user, what="view")
db.delete(view)
service.delete_view(db, _own_view(db, user, view_id))
db.commit()
return {"deleted": view_id}
@@ -181,14 +153,6 @@ def reorder_views(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
ids = body.ids
rows = db.execute(select(SavedView).where(SavedView.user_id == user.id)).scalars().all()
by_id = {v.id: v for v in rows}
pos = 0
for vid in ids:
v = by_id.get(vid)
if v is not None:
v.position = pos
pos += 1
service.reorder_views(db, user, body.ids)
db.commit()
return {"ok": True}
+17 -60
View File
@@ -4,7 +4,6 @@ and the shared quota picture."""
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
from app import audit, quota, state, sysconfig
@@ -12,17 +11,18 @@ from app.audit import AuditAction
from app.auth import admin_user
from app.config import settings
from app.db import get_db
from app.models import Channel, Subscription, User, Video
from app.models import User
from app.catalog import queries as catalog_queries
from app.scheduler import (
JOB_INTERVALS,
MAX_INTERVAL,
MIN_INTERVAL,
apply_interval,
reschedule,
scheduler_snapshot,
stage_interval,
trigger_all,
trigger_job,
)
from app.sync.explore import followed_channel
from app.sync.runner import estimate_deep_backfill
from app.routes._common import not_found
from app.refusal import refused
@@ -42,14 +42,14 @@ def set_job_interval(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Change how often a scheduler job runs (minutes). Persisted as an override and applied
to the live scheduler immediately."""
"""Change how often a scheduler job runs (minutes). Stored as an override together with its
audit row, then applied to the live scheduler."""
if job_id not in JOB_INTERVALS:
raise not_found("job")
minutes = body.interval_minutes
if not (MIN_INTERVAL <= minutes <= MAX_INTERVAL):
raise refused(400, "invalid_interval", min=MIN_INTERVAL, max=MAX_INTERVAL)
applied = apply_interval(db, job_id, minutes)
applied = stage_interval(db, job_id, minutes)
audit.record(
db,
admin.id,
@@ -60,6 +60,7 @@ def set_job_interval(
after={"interval_minutes": applied},
)
db.commit()
reschedule(job_id, applied)
return {"ok": True, "id": job_id, "interval_minutes": applied}
@@ -117,61 +118,17 @@ def set_maintenance_settings(
@router.get("")
def get_scheduler(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
snap = scheduler_snapshot()
snap = scheduler_snapshot(db)
# Work still queued (shared catalog, so these are instance-wide, not per-user).
deep_requested = (
select(Subscription.id)
.where(
Subscription.channel_id == Channel.id,
Subscription.deep_requested.is_(True),
)
.exists()
)
deep_channels = (
db.execute(select(Channel).where(Channel.backfill_done.is_(False), deep_requested))
.scalars()
.all()
)
deep_eta = estimate_deep_backfill(db, deep_channels)
queue = {
"channels_recent_pending": db.scalar(
# The same selection run_recent_backfill works through, or the card counts channels
# the job will never touch and never reaches zero.
select(func.count())
.select_from(Channel)
.where(Channel.recent_synced_at.is_(None), followed_channel())
)
or 0,
# The whole opted-in BACKLOG, deliberately not `run_deep_backfill`'s next-pass selection
# (which also wants `recent_synced_at`): the same figure /api/sync/my-status shows the user,
# and a channel missing its recent pull is one the same job tick fetches first, not one the
# job will never touch. Narrowing it here would under-report work that is genuinely queued.
"channels_deep_pending": deep_eta["channels_pending"],
"deep_eta_seconds": deep_eta["eta_seconds"],
"videos_pending_enrich": db.scalar(
select(func.count()).select_from(Video).where(Video.enriched_at.is_(None))
)
or 0,
"videos_pending_shorts": db.scalar(
select(func.count())
.select_from(Video)
.where(Video.shorts_probed.is_(False), Video.enriched_at.is_not(None))
)
or 0,
"videos_live_refresh": db.scalar(
select(func.count())
.select_from(Video)
.where(
or_(
Video.live_status.in_(("live", "upcoming")),
and_(Video.live_status == "was_live", Video.duration_seconds.is_(None)),
)
)
)
or 0,
}
deep_eta = estimate_deep_backfill(db, catalog_queries.deep_backlog(db))
queue = catalog_queries.work_queue(db)
# The whole opted-in BACKLOG, deliberately not `run_deep_backfill`'s next-pass selection
# (which also wants `recent_synced_at`): the same figure /api/sync/my-status shows the user,
# and a channel missing its recent pull is one the same job tick fetches first, not one the
# job will never touch. Narrowing it here would under-report work that is genuinely queued.
queue["channels_deep_pending"] = deep_eta["channels_pending"]
queue["deep_eta_seconds"] = deep_eta["eta_seconds"]
used = quota.units_used_today(db)
quota_info = {
+14 -245
View File
@@ -11,39 +11,24 @@ regardless of source.
Ingested results are flagged (videos.via_search / channels.from_search) so the Library can hide
this search-discovered "noise" by default. Shorts and currently-live/upcoming videos are never
ingested this way (Shorts are confirmed by the youtube.com/shorts probe; live/upcoming are
dropped from the search response up front)."""
dropped from the search response up front).
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
This is the HTTP side — the paging, the caps and the refusals; what a page of results leaves in the
catalog is `catalog/search`."""
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from sqlalchemy import and_, delete, func, or_, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session, aliased
from sqlalchemy.orm import Session
from app import logs, quota, sysconfig
from app.auth import current_user
from app.catalog import queries
from app.catalog import search as catalog_search
from app.db import get_db
from app.models import (
LIVE_OR_UPCOMING,
BlockedChannel,
Channel,
Playlist,
PlaylistItem,
SearchFind,
Subscription,
User,
Video,
VideoState,
)
from app.feedquery import feed_columns, serialize
from app.utils import like_escape
from app.sync.subscriptions import apply_channel_details
from app.sync.videos import insert_stubs, apply_video_details, parse_dt
from app.models import User
from app.reader import queries as reader_queries
from app.youtube import search_scrape
from app.youtube.client import YouTubeClient, YouTubeError
from app.youtube.shorts import make_client, probe_is_short
from app.refusal import refused
from app.routes._common import youtube_refusal
@@ -54,198 +39,6 @@ router = APIRouter(prefix="/api/search", tags=["search"])
SEARCH_COST = 100
def _classify_shorts(db: Session, videos: list[Video]) -> None:
"""Confirm/deny Shorts for the freshly-ingested videos so none slip into the feed.
Mirrors the scheduler's run_shorts_classification, but scoped to just these rows and run
synchronously (the search is an explicit, awaited action). Non-candidates (too long, no
duration, or live) are finalised without a probe; the rest hit youtube.com/shorts (no
quota). Undetermined probes are left for the scheduler to retry."""
probe_max = sysconfig.get_int(db, "shorts_probe_max_seconds")
to_probe: list[Video] = []
for v in videos:
if v.shorts_probed:
continue
if v.enriched_at is None:
# Enrichment failed (proxy blip) or YouTube omitted the row — duration is unknown,
# so DON'T finalise shorts_probed off a null duration; leave it for the scheduler
# (which only classifies enriched rows) to retry. Otherwise a real Short would leak.
continue
if v.live_status != "none" or v.duration_seconds is None or v.duration_seconds > probe_max:
v.is_short = False
v.shorts_probed = True
else:
to_probe.append(v)
if to_probe:
with make_client() as client:
def work(v: Video):
return v, probe_is_short(client, v.id)
with ThreadPoolExecutor(max_workers=16) as pool:
for v, result in pool.map(work, to_probe):
if result is None:
continue
v.is_short = result
v.shorts_probed = True
db.commit()
def _serialize_ids(db: Session, user: User, ids: list[str]) -> list[dict]:
"""Serialize the given video ids with the shared feed projection (joined channel + this
user's watch state + saved), preserving the input (search relevance) order."""
if not ids:
return []
state = aliased(VideoState)
rows = db.execute(
select(*feed_columns(user, state))
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(state, and_(state.video_id == Video.id, state.user_id == user.id))
.where(Video.id.in_(ids))
).all()
by_id = {r.id: serialize(r) for r in rows}
return [by_id[i] for i in ids if i in by_id]
def _ingest_candidates(
db: Session, yt: YouTubeClient, user: User, candidates: list[dict]
) -> list[str]:
"""Materialise one page of search candidates into the catalog (channel stubs + enrich,
video stubs flagged via_search, enrich, Shorts classification) and return the KEEPABLE
video ids — non-Short, non-live — in the input relevance order."""
ids = [it["id"] for it in candidates]
if not ids:
return []
# 1) Channels are the FK target — create stubs for unknown ones (flagged from_search),
# then enrich the new ones (channels.list, 1 unit). Stubs survive an enrich failure.
channel_ids = list({it["channel_id"] for it in candidates})
existing = set(db.scalars(select(Channel.id).where(Channel.id.in_(channel_ids))).all())
new_channel_ids = [c for c in channel_ids if c not in existing]
if new_channel_ids:
titles = {it["channel_id"]: it["channel_title"] for it in candidates}
for cid in new_channel_ids:
db.add(Channel(id=cid, title=titles.get(cid), from_search=True))
db.commit()
try:
with quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER):
details = yt.get_channels(new_channel_ids)
apply_channel_details(db, details, new_channel_ids)
db.commit()
except YouTubeError:
# The stubs remain and the details can be backfilled later — but until they are, those
# channels sit in the results with no thumbnail and no subscriber count, looking like
# channels that have neither.
log.warning(
"the channels found by a search could not be enriched",
extra=logs.event("search.channel_enrich_failed", channels=len(new_channel_ids)),
)
db.rollback() # details can be backfilled later; the stubs remain
# 2) Video stubs, flagged via_search. Existing rows keep their flags (ON CONFLICT DO
# NOTHING), so an organically-synced video stays organic.
# published_at is NOT NULL (migration 0068); the scrape source exposes no publish date
# (only a relative "3 years ago"), so fall back to now() to satisfy the constraint — the
# enrich in step 3 overwrites it with the video's real snippet date.
now = datetime.now(timezone.utc)
insert_stubs(
db,
[
{
"id": it["id"],
"channel_id": it["channel_id"],
"title": it["title"],
"published_at": parse_dt(it["published_at"]) or now,
"thumbnail_url": it["thumbnail_url"],
"via_search": True,
}
for it in candidates
],
)
# 3) Enrich (videos.list, 1 unit/50) so cards have duration/stats and we can judge live.
videos = db.scalars(select(Video).where(Video.id.in_(ids))).all()
try:
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_ENRICH):
detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
except YouTubeError:
# Every candidate then stays unenriched, and an unenriched video cannot be judged live or
# Short — so it is dropped. A working search returns nothing and reads as "no results".
log.warning(
"the videos found by a search could not be enriched",
extra=logs.event("search.video_enrich_failed", candidates=len(ids)),
)
detail_by_id = {}
enriched_at = datetime.now(
timezone.utc
) # when THIS enrich ran (distinct from the stub placeholder)
for v in videos:
item = detail_by_id.get(v.id)
if item is None:
continue
apply_video_details(v, item)
v.enriched_at = enriched_at
db.commit()
# 4) Confirm/deny Shorts (no quota) so none enter via search.
_classify_shorts(db, videos)
keep = {
v.id
for v in videos
if v.enriched_at is not None and not v.is_short and v.live_status not in LIVE_OR_UPCOMING
}
return [vid for vid in ids if vid in keep]
def _new_first(db: Session, user: User, ids: list[str]) -> list[str]:
"""Reorder results so genuinely-new discoveries come first and things the user already has
(a subscribed channel, or a video they've watched / in-progress / saved / playlisted) sink
to the back — keeping the relevance order within each group."""
if not ids:
return ids
sub_ch = select(Subscription.channel_id).where(Subscription.user_id == user.id)
has_state = (
select(VideoState.id)
.where(VideoState.video_id == Video.id, VideoState.user_id == user.id)
.exists()
)
in_playlist = (
select(PlaylistItem.id)
.join(Playlist, Playlist.id == PlaylistItem.playlist_id)
.where(PlaylistItem.video_id == Video.id, Playlist.user_id == user.id)
.exists()
)
have = set(
db.scalars(
select(Video.id).where(
Video.id.in_(ids),
or_(Video.channel_id.in_(sub_ch), has_state, in_playlist),
)
).all()
)
return [i for i in ids if i not in have] + [i for i in ids if i in have]
def _fold_term(db: Session, video_ids: list[str], term: str) -> None:
"""Append `term` to each video's space-joined `search_terms`, unless it is already one of them.
"Already one of them" is a whole-term match, as a literal: a substring test skipped `cat` on a
video already found for `category`, and an unescaped one read the `_` in `my_channel` as a
wildcard — either way the term was never folded in, and local search lost that relevance."""
padded = func.concat(" ", func.coalesce(Video.search_terms, ""), " ")
db.execute(
update(Video)
.where(Video.id.in_(video_ids))
.where(padded.notilike(f"% {like_escape(term)} %"))
.values(
search_terms=func.trim(
func.coalesce(Video.search_terms, "").op("||")(" ").op("||")(term)
)
)
)
@router.get("/youtube")
def search_youtube(
q: str | None = None,
@@ -287,9 +80,7 @@ def search_youtube(
raise refused(429, "quota_spent")
# Channels this user has blocked are dropped before ingestion — never shown, never stored.
blocked = set(
db.scalars(select(BlockedChannel.channel_id).where(BlockedChannel.user_id == user.id)).all()
)
blocked = queries.blocked_ids(db, user)
collected: list[str] = []
next_cursor = cursor
@@ -330,7 +121,7 @@ def search_youtube(
# write/shape failure degrades to a 422 with a message on this JSON endpoint instead of a
# bare 500 hidden behind the FE's generic "search failed" (same rationale as B-core-1).
try:
collected.extend(_ingest_candidates(db, yt, user, candidates))
collected.extend(catalog_search.ingest_candidates(db, yt, user, candidates))
except Exception as exc: # any ingest failure, not just YouTubeError
db.rollback()
# The term is NOT in this line. A search string is exactly what the privacy
@@ -378,17 +169,12 @@ def search_youtube(
# Record these as THIS user's search finds (Mine feed / Source filter), and fold the
# query into each result's search_terms so local full-text search inherits YouTube's
# relevance. Idempotent; the generated search_vector updates automatically.
db.execute(
pg_insert(SearchFind)
.values([{"user_id": user.id, "video_id": vid} for vid in ordered])
.on_conflict_do_nothing(index_elements=["user_id", "video_id"])
)
_fold_term(db, ordered, term)
catalog_search.record_finds(db, user, ordered, term)
db.commit()
ordered = _new_first(db, user, ordered)
ordered = catalog_search.new_first(db, user, ordered)
return {
"items": _serialize_ids(db, user, ordered),
"items": reader_queries.feed_rows_by_ids(db, user, ordered),
"next_cursor": next_cursor,
"has_more": bool(next_cursor),
"source": source,
@@ -412,23 +198,6 @@ def clear_search(
ids = body.video_ids
if not ids:
return {"ok": True, "removed": 0}
db.execute(
delete(SearchFind).where(SearchFind.user_id == user.id, SearchFind.video_id.in_(ids))
)
db.commit()
has_state = select(VideoState.id).where(VideoState.video_id == Video.id).exists()
in_playlist = select(PlaylistItem.id).where(PlaylistItem.video_id == Video.id).exists()
subscribed = select(Subscription.id).where(Subscription.channel_id == Video.channel_id).exists()
other_find = select(SearchFind.id).where(SearchFind.video_id == Video.id).exists()
removed = db.execute(
delete(Video).where(
Video.id.in_(ids),
Video.via_search.is_(True),
~has_state,
~in_playlist,
~subscribed,
~other_find,
)
).rowcount
removed = catalog_search.clear(db, user, ids)
db.commit()
return {"ok": True, "removed": removed}
+3 -3
View File
@@ -91,12 +91,12 @@ def setup_admin(body: SetupAdminIn, db: Session = Depends(require_setup)) -> dic
@router.post("/config")
def setup_config(settings_map: dict[str, Any], db: Session = Depends(require_setup)) -> dict:
"""Persist provided DB-backed settings (Google creds, SMTP, quota…). Keys must be in the
sysconfig registry; empty values are skipped (left at the default). Genuinely dynamic
"""Persist provided DB-backed settings (Google creds, SMTP, quota…). Keys must be the
Configuration page's; empty values are skipped (left at the default). Genuinely dynamic
key→value map, so it stays a typed dict rather than a fixed model."""
saved: list[str] = []
for key, value in settings_map.items():
if sysconfig.spec(key) is None:
if sysconfig.config_page_spec(key) is None:
raise refused(400, "unknown_setting", key=key)
if value is None or value == "":
continue

Some files were not shown because too many files have changed in this diff Show More