Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
77f8170153 | ||
|
|
0fd0f1f637 | ||
|
|
7371cdeb94 | ||
|
|
abb18d4d8c | ||
|
|
5bf13c8701 | ||
|
|
fe6af1445f | ||
|
|
8845d1aa49 | ||
|
|
0b92b40db5 | ||
|
|
27ccf253f5 | ||
|
|
c221452a00 | ||
|
|
fe38c07e7e | ||
|
|
45b5e19fb8 | ||
|
|
e08278fae2 | ||
|
|
350913b6c2 | ||
|
|
470caacfb0 | ||
|
|
e8b9bdd7c1 | ||
|
|
55cbbfb6e4 | ||
|
|
4d058cba81 | ||
|
|
872b583f06 | ||
|
|
90931b4750 | ||
|
|
a6f1a26658 | ||
|
|
4ed07388e8 | ||
|
|
bb91d0a22d | ||
|
|
e7e8d8236e | ||
|
|
fc65b8ab83 | ||
|
|
26ed8855c9 | ||
|
|
9778bdf351 | ||
|
|
2b7620ded9 | ||
|
|
c0682aa572 | ||
|
|
25aed6635f | ||
|
|
9310ff85bf | ||
|
|
5b846136ad | ||
|
|
1bf3d6a592 | ||
|
|
4e4fff87d4 | ||
|
|
995a493bd3 | ||
|
|
5a93a9a8e7 | ||
|
|
911fea3243 | ||
|
|
4aece8aff7 | ||
|
|
adb5e94511 | ||
|
|
673decf4d4 | ||
|
|
4d1cf66d96 | ||
|
|
2baad2ebae | ||
|
|
8383459f70 | ||
|
|
c34f6f3297 | ||
|
|
e4b6d918e4 | ||
|
|
bf5cf011ce | ||
|
|
6e6e47652a | ||
|
|
4c09a22add | ||
|
|
2bdf1ec76f | ||
|
|
f7e036abef | ||
|
|
f2de8f73aa | ||
|
|
643f305814 | ||
|
|
cbe1b3bd63 | ||
|
|
ae775b383c | ||
|
|
358d5af70b | ||
|
|
c6831090b5 | ||
|
|
38fc95d772 | ||
|
|
3f6638bf96 | ||
|
|
469c8aa481 | ||
|
|
df2997e9f3 | ||
|
|
f69725503f | ||
|
|
d5bc7e8031 | ||
|
|
efdffc82ae | ||
|
|
e23544a46b | ||
|
|
035ee937a0 | ||
|
|
bbb4063362 | ||
|
|
b4ba9d95b1 | ||
|
|
c18a2063c8 | ||
|
|
b636ef38bc | ||
|
|
195b5f3917 | ||
|
|
1a1e9d9542 | ||
|
|
d09b5d6abb | ||
|
|
439f5cd67b | ||
|
|
d79cb528fd | ||
|
|
9b76d2f96c | ||
|
|
bf54a3ce5f | ||
|
|
efbbb23525 | ||
|
|
3b85881af4 | ||
|
|
8cf8c4139c | ||
|
|
cd773e8f33 | ||
|
|
242f00671d | ||
|
|
7f8d7cbe20 | ||
|
|
8dd2f0c015 | ||
|
|
b3402d158f | ||
|
|
c910cf2f20 | ||
|
|
245c0a2019 | ||
|
|
add7f7d167 | ||
|
|
a29afd916d | ||
|
|
e9ce989998 | ||
|
|
7299e82c22 | ||
|
|
41052eb4ea | ||
|
|
85fd9b6704 | ||
|
|
e06c0688d9 | ||
|
|
15a3603513 | ||
|
|
c430558ab3 | ||
|
|
be38d652a5 | ||
|
|
31a75f4985 | ||
|
|
f8a8e905da | ||
|
|
5e8589bbf8 | ||
|
|
ceadf69e43 | ||
|
|
4379f76747 | ||
|
|
b3be568e36 | ||
|
|
a8cbc7b338 | ||
|
|
cf8dc80428 | ||
|
|
da74b8b9bf | ||
|
|
6944622085 | ||
|
|
2bdcebdb34 | ||
|
|
046dde44a4 | ||
|
|
74f58ea001 | ||
|
|
b515c16117 | ||
|
|
4a0d4b0964 | ||
|
|
cb8b99e0e7 | ||
|
|
e5c75702e8 | ||
|
|
4ef772d37b | ||
|
|
6986eb1937 | ||
|
|
41ad538edc | ||
|
|
7489cab3bc | ||
|
|
91256b10b0 | ||
|
|
b22e903174 | ||
|
|
d6e72d7576 | ||
|
|
e3a4593acf | ||
|
|
fa271a31b9 | ||
|
|
947efaba6e | ||
|
|
22d873ed49 | ||
|
|
d825aff30d | ||
|
|
9dd56e94d4 | ||
|
|
25ad2ee1ba | ||
|
|
c86ec7f7a5 | ||
|
|
0469812499 | ||
|
|
ee658b8bfc | ||
|
|
52b93d0014 | ||
|
|
c3ad7684ea | ||
|
|
a5666bfde9 | ||
|
|
f49dc6532f | ||
|
|
0d410592e2 | ||
|
|
7e5d4a7d93 | ||
|
|
24190e4c16 | ||
|
|
e96a562f14 |
+106
@@ -7,6 +7,112 @@ edit the release notes and re-run the generator.
|
||||
|
||||
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
|
||||
|
||||
## 0.119.0 — 2026-11-13
|
||||
|
||||
### New
|
||||
|
||||
- Make a GIF of up to 30 seconds: in a player's Clip panel, or in the Downloads editor, where each kept segment becomes a GIF of its own. Both say the expected size before it is made.
|
||||
- A shared GIF's link opens the moving picture itself, with Download beside it. In Teams its preview moves when the GIF is 15 MB or less.
|
||||
|
||||
## 0.118.0 — 2026-11-08
|
||||
|
||||
### New
|
||||
|
||||
- Surprise me now leans on the channels and genres you watch, and one pick in three stays a blind draw, so it remains a surprise.
|
||||
- On a connection too slow for a Plex video, playback waits and goes on where it was, and says how fast the video needs and how fast it arrives.
|
||||
|
||||
### Fixed
|
||||
|
||||
- A 24/7 stream's archive no longer adds years to a list's total: a total names the videos it could not count as "+N without length". The Feed sorted by Surprise me opens faster.
|
||||
- Plex films the server's GPU cannot decode (VC-1, older DivX or XviD) start in a few seconds instead of about 20. Text over the Plex picture is readable on any frame.
|
||||
|
||||
## 0.117.0 — 2026-11-06
|
||||
|
||||
### New
|
||||
|
||||
- A share link's address now says what it opens: /file/ for a file, /folder/ for a folder, /watch/ for a video. Links you already sent keep working.
|
||||
- The upload dialog shows which folder the file goes to, and you can pick another one or make a new folder there. Opening Share copies a link at once.
|
||||
- The Channels tag chips show how many followed channels each holds. Continue watching in Plex has a See all, and a Plex item's Info shows the file size.
|
||||
|
||||
### Fixed
|
||||
|
||||
- An expired share link is marked as expired and is never copied. A crop of a video with wide pixels keeps exactly what its frame showed.
|
||||
- A Plex auto-skip waits 3 seconds into the marker before its countdown starts.
|
||||
|
||||
## 0.116.3 — 2026-11-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- If a stylesheet is lost on the way, the app now asks for it again, so a poor connection no longer leaves the page unstyled until a reload.
|
||||
|
||||
## 0.116.2 — 2026-11-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- A Plex video paused or muted in a background tab no longer loses its session after a minute, and if a session is lost anyway, playback comes back at once instead of after half a minute.
|
||||
|
||||
## 0.116.1 — 2026-11-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- A playlist now shows the order the server holds after a revert to YouTube, a sync or a change on another device, and the next drag no longer brings back the old one.
|
||||
- An Undo on a removed or moved row puts it back in the right place while another change is still saving.
|
||||
|
||||
## 0.116.0 — 2026-11-03
|
||||
|
||||
### New
|
||||
|
||||
- In a playlist, tick rows with their box, Shift+click or x, then move or remove them together. An Undo puts them back at their places, and a single row's move has one too.
|
||||
- Drag rows onto another list in the rail: one row, or all the selected ones, also under a sort or a filter.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Removing a video from inside the player no longer loses your place in the list when the player closes.
|
||||
|
||||
## 0.115.0 — 2026-11-01
|
||||
|
||||
### New
|
||||
|
||||
- Clear watch progress is on every YouTube video's menu, on Home, in playlists and in the player's related videos. It forgets where you left a started video and changes nothing else.
|
||||
- Channels have an Added column, and an About card opens only when the pointer rests on it.
|
||||
- In Files, a press anywhere on a row plays the file.
|
||||
- The feed's toolbar pills are named by a glyph, with Sort and View on the filters' row.
|
||||
|
||||
### Fixed
|
||||
|
||||
- In a play queue, Previous and Next pass over a video that can't play, in the direction you were going, and say why when only such videos are left.
|
||||
- A Facebook download titled only “Facebook” now takes the post's text as its name.
|
||||
- The player's big view keeps its way out in sight on a tablet, and Tab no longer lands on controls hidden under it.
|
||||
|
||||
## 0.114.3 — 2026-11-01
|
||||
|
||||
### Fixed
|
||||
|
||||
- Editing a file that is neither video nor audio now says why it can't be done, instead of a generic error.
|
||||
|
||||
## 0.114.2 — 2026-10-30
|
||||
|
||||
### Fixed
|
||||
|
||||
- Security: a library the sign-in relies on is updated, closing two published vulnerabilities.
|
||||
|
||||
## 0.114.1 — 2026-10-30
|
||||
|
||||
### Fixed
|
||||
|
||||
- Notifications follow a language switch: a notice on screen and every row in the notification center are said again in the new language, their buttons too.
|
||||
|
||||
## 0.114.0 — 2026-10-29
|
||||
|
||||
### New
|
||||
|
||||
- Notifications: what asks for you comes first, then everything else as one stream, newest first. Read notices older than a week fold under Earlier, and Back returns to the row you left.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Plex: a progress save sent just before an episode ends no longer leaves it unwatched, and a refused finish no longer sticks.
|
||||
- After a dialog closes, the keyboard focus returns to where it was, also inside a menu that stays open.
|
||||
|
||||
## 0.113.0 — 2026-10-28
|
||||
|
||||
### New
|
||||
|
||||
@@ -65,7 +65,7 @@ how it gets built. Concretely:
|
||||
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
|
||||
`downloads/`, `reader/`, `discover/`, `catalog/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
|
||||
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
|
||||
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
|
||||
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`, `draw`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
|
||||
The domain is four **units**, each owning its tables: Capture (`downloads/`), Reader (`reader/`, `discover/`, the playlist
|
||||
sync), Catalog (`catalog/`, the `sync/` ingests) and Plex (`plex/`). A unit's route does no database work: it reads through
|
||||
the unit's `queries`, writes through its `service` (which flushes) and commits. A unit's code reads another unit's tables
|
||||
@@ -74,7 +74,7 @@ only along a declared edge and writes them only through that unit's service; cor
|
||||
cannot (an ORM write on another unit's loaded row, SQL in `text()`, a commit by `begin()`) is review's to catch, listed in
|
||||
[`backend/app/README.md`](backend/app/README.md) with the conventions.
|
||||
Frontend, by folder under `src/`: `main.tsx` provider tree → `shell/` (`App`, header, nav rail, palette,
|
||||
boot; a shared `/watch` link boots from its own entry, `watch.html` → `watchMain.tsx` → `watch/`, held
|
||||
boot; a share link (`/watch`, `/file`, `/folder`) boots from its own entry, `watch.html` → `watchMain.tsx` → `watch/`, held
|
||||
app-free by a contract test) → **`modules/<page>/`** (one folder per page: the page, its rail, its store, its tests,
|
||||
and its `module.ts` definition — label, icon, gate, the lazy page and rail, its store — collected in nav order by
|
||||
`features/navigation/moduleRegistry`, which also mounts every store at the root; a new page is its folder, its id in
|
||||
@@ -105,7 +105,7 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
|
||||
| Ownership check | `userscope.owned_by` (a filter) / `owned_row` (a row by id): a unit's `queries.own_x` returns None and its route answers `not_found`; a core route `owned_or_404`. A composite rule (own or shared, built-in or own) says so in its `queries` | an inline owner filter — `Model.user_id == …`, an alias's, `.in_()`, `filter_by(user_id=…)` (fence-held, every module) — or a `db.get` and a compare where `owned_row` fits |
|
||||
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
|
||||
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
|
||||
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
|
||||
| User-facing error | structured `detail` (`{code, reason}`), its sentence named in `lib/api/refusalCodes.ts` (contract-held) | a bare English string under a bilingual UI |
|
||||
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
|
||||
| Transaction commit | the handler commits; a unit's `service` and every helper `flush` (fence-held, every module) | a helper that commits — only a listed one (what owns its session, or a core step that must land whatever its caller does next), each with its reason: the fence's `UNIT_COMMITS` / `CORE_COMMITS`, a route's `COMMITTING_HELPERS` (`test_route_conventions`) |
|
||||
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
|
||||
|
||||
@@ -185,118 +185,6 @@
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
@@ -364,25 +252,7 @@
|
||||
}
|
||||
}
|
||||
],
|
||||
"./app/downloads/adapters.py": [
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 30,
|
||||
"endColumn": 38,
|
||||
"lineCount": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"./app/downloads/backfill.py": [
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 34,
|
||||
"endColumn": 38,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportReturnType",
|
||||
"range": {
|
||||
@@ -413,14 +283,6 @@
|
||||
}
|
||||
],
|
||||
"./app/downloads/pipeline.py": [
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 34,
|
||||
"endColumn": 38,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportAttributeAccessIssue",
|
||||
"range": {
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
"""videos: the duration sorts ride an index on the video's length, not on the raw column
|
||||
|
||||
Revision ID: 0117_videos_length_sort_indexes
|
||||
Revises: 0116_followed_explored_channels
|
||||
Create Date: 2026-11-07
|
||||
|
||||
A YouTube length past a day is not a length (`catalog.models.LENGTH_MAX_S`): a 24/7 stream's archive
|
||||
reports the years it has run. The feed's duration sorts now order by `Video.length_seconds`, which
|
||||
is NULL for such a video, so "longest first" no longer opens on a stream archive. Migration 0096's
|
||||
two duration indexes are on the raw column and no longer match that ORDER BY; they are replaced by
|
||||
the same indexes over the length expression. The expression is spelled out here as SQLAlchemy emits
|
||||
it, with the limit as a literal (an index matches a query's constant, never a bound parameter);
|
||||
tests/test_feed_sort_indexes.py fails if the two drift apart.
|
||||
|
||||
CONCURRENTLY (autocommit_block), re-runnable, as in 0096.
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import text
|
||||
|
||||
revision: str = "0117_videos_length_sort_indexes"
|
||||
down_revision: str | None = "0116_followed_explored_channels"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
# Frozen with this migration: `catalog.models.LENGTH_MAX_S` is 24 h.
|
||||
LENGTH = "(CASE WHEN (duration_seconds > 86400) THEN NULL ELSE duration_seconds END)"
|
||||
|
||||
NEW = {
|
||||
"ix_videos_length_desc": f"{LENGTH} DESC NULLS LAST, id",
|
||||
"ix_videos_length_asc": f"{LENGTH} ASC NULLS LAST, id",
|
||||
}
|
||||
OLD = {
|
||||
"ix_videos_duration_desc": "duration_seconds DESC NULLS LAST, id",
|
||||
"ix_videos_duration_asc": "duration_seconds ASC NULLS LAST, id",
|
||||
}
|
||||
|
||||
|
||||
def _valid(name: str) -> bool | None:
|
||||
"""The index's `indisvalid`, or None when there is no index of that name."""
|
||||
row = (
|
||||
op.get_bind()
|
||||
.execute(
|
||||
text(
|
||||
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
|
||||
"WHERE c.relname = :name"
|
||||
),
|
||||
{"name": name},
|
||||
)
|
||||
.first()
|
||||
)
|
||||
return None if row is None else bool(row[0])
|
||||
|
||||
|
||||
def _build(name: str, columns: str) -> None:
|
||||
"""CREATE INDEX CONCURRENTLY, re-runnable: an INVALID leftover of a failed build is dropped first
|
||||
(see 0096 `_build`)."""
|
||||
if _valid(name) is False:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
|
||||
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} ON videos ({columns})")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
for name, columns in NEW.items():
|
||||
_build(name, columns)
|
||||
for name in OLD:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
for name, columns in OLD.items():
|
||||
_build(name, columns)
|
||||
for name in NEW:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
|
||||
@@ -0,0 +1,40 @@
|
||||
"""videos: a random key per video, so a seeded shuffle walks an index instead of sorting the scope
|
||||
|
||||
Revision ID: 0118_videos_shuffle_key
|
||||
Revises: 0117_videos_length_sort_indexes
|
||||
Create Date: 2026-11-07
|
||||
|
||||
The feed's Shuffle sort and the Home surprise ordered the whole feed scope by md5(id || seed) on every
|
||||
call: ~350 ms on prod for a scope of ~213k videos, and a Shuffle page deeper in the list cost more
|
||||
(767 ms on page 7 on localdev, against 4 ms for Newest). Each video now carries a random number drawn
|
||||
once (`Video.shuffle_key`); a seed picks a point on that ring and the draw reads the index from there
|
||||
(`feedquery.shuffle_walk`).
|
||||
|
||||
One ADD COLUMN with a volatile default rewrites the table and fills every row with its own value. On
|
||||
localdev (278k videos) that took 28 s, against 84 s for an UPDATE of a nullable column, so the rewrite
|
||||
it is: the table is locked for that time, and a deploy has the app down while it migrates anyway.
|
||||
The index is (shuffle_key, id): the walk's keyset cursor compares the pair.
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0118_videos_shuffle_key"
|
||||
down_revision: str | None = "0117_videos_length_sort_indexes"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
"ALTER TABLE videos ADD COLUMN IF NOT EXISTS shuffle_key double precision "
|
||||
"NOT NULL DEFAULT random()"
|
||||
)
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
|
||||
op.execute("CREATE INDEX CONCURRENTLY ix_videos_shuffle_key ON videos (shuffle_key, id)")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
|
||||
op.execute("ALTER TABLE videos DROP COLUMN IF EXISTS shuffle_key")
|
||||
@@ -0,0 +1,65 @@
|
||||
"""plex_items.transcode_hw: the facet says whether the GPU ENCODES, under the measured codec sets
|
||||
|
||||
Measured on prod (2026-11-07): the iGPU has no VC-1 Advanced decode, so a VC-1 film never ran on the
|
||||
GPU although the facet said it did; VC-1, XviD and DivX now take the CPU-decode → GPU-encode path; and an
|
||||
HDR film runs on the GPU through the OpenCL tonemap although the facet said software. The rule is
|
||||
`plex.sync.transcode_hw_capable`; the two codec sets below are pinned to `plex.sync` by
|
||||
tests/test_plex_playback_class.py.
|
||||
|
||||
Rewritten from the stored codec and bit depth. HDR is not stored, so a 10-bit HW-decodable item is set to
|
||||
software here and its probe watermark cleared: the bit-depth backfill (`plex.sync._probe_media_batch`,
|
||||
a bounded batch per Plex sync) reads its HDR flag again and moves an HDR one back to the GPU.
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0119_plex_transcode_gpu"
|
||||
down_revision: Union[str, None] = "0118_videos_shuffle_key"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
HW_DECODABLE_CODECS = ("h264", "hevc", "h265", "vp9", "mpeg2video")
|
||||
CPU_DECODE_CODECS = ("vc1", "mpeg4", "msmpeg4v3")
|
||||
|
||||
|
||||
def _in(codecs: tuple[str, ...]) -> str:
|
||||
return ", ".join(f"'{c}'" for c in codecs)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
f"""
|
||||
UPDATE plex_items
|
||||
SET transcode_hw = CASE
|
||||
WHEN lower(codec_video) IN ({_in(HW_DECODABLE_CODECS)})
|
||||
THEN coalesce(video_bit_depth, 8) <= 8
|
||||
WHEN lower(codec_video) IN ({_in(CPU_DECODE_CODECS)})
|
||||
THEN coalesce(video_bit_depth, 8) <= 8
|
||||
ELSE false
|
||||
END
|
||||
WHERE playable = 'transcode'
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
f"""
|
||||
UPDATE plex_items
|
||||
SET media_probed_at = NULL
|
||||
WHERE playable = 'transcode'
|
||||
AND lower(codec_video) IN ({_in(HW_DECODABLE_CODECS)})
|
||||
AND video_bit_depth > 8
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# The old rule: a decodable codec (VC-1 counted) at 8 bits; HDR and every other codec software.
|
||||
op.execute(
|
||||
f"""
|
||||
UPDATE plex_items
|
||||
SET transcode_hw = lower(codec_video) IN ({_in(HW_DECODABLE_CODECS + ("vc1",))})
|
||||
AND coalesce(video_bit_depth, 8) <= 8
|
||||
WHERE playable = 'transcode'
|
||||
"""
|
||||
)
|
||||
@@ -25,8 +25,8 @@ Three layers, by *who* runs the code — not by feature:
|
||||
background run's durable row, its words in `runstatus`),
|
||||
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
|
||||
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
|
||||
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
|
||||
reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core
|
||||
level), `debugsession` (a recording that tags its user's lines, E16), `collation` (A→Z in the
|
||||
reader's alphabet, E30) and `draw` (the weighted, seeded draw "Surprise me" deals from), … are single-file primitives many layers import. They depend on nothing above them (a core
|
||||
module must never import a `routes/…` module).
|
||||
|
||||
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
|
||||
|
||||
@@ -10,16 +10,22 @@ from sqlalchemy import (
|
||||
Boolean,
|
||||
Computed,
|
||||
DateTime,
|
||||
Float,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
JSON,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
case,
|
||||
func,
|
||||
literal_column,
|
||||
null,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql import TSVECTOR
|
||||
from sqlalchemy.ext.hybrid import hybrid_property
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
from sqlalchemy.sql.elements import ColumnElement
|
||||
|
||||
from app.db import Base
|
||||
from app.models import TimestampMixin
|
||||
@@ -114,6 +120,20 @@ class Subscription(Base, TimestampMixin):
|
||||
# hidden from feeds/search and revisited by the refresh job until they settle into a VOD.
|
||||
LIVE_OR_UPCOMING = ("live", "upcoming")
|
||||
|
||||
# A YouTube length past a day is not a length. A 24/7 stream's archive (`was_live`) reports the years
|
||||
# the stream has run (measured on prod, 2026-11-06: 122,962,914 s), and one such video made a Watch
|
||||
# later total read 44,073 hours. YouTube caps an upload at 12 hours, so nothing real is lost. Such a
|
||||
# length is unknown wherever a length is used: the API sends none (`Video.length_seconds`), sums leave
|
||||
# it out (a playlist's counts it as `without_length`), and the duration filter, sort and Home budget never
|
||||
# match it. The raw `duration_seconds` stays for what it does prove — that the video is not a Short —
|
||||
# and for the jobs that ask whether YouTube has answered yet (test_video_length holds who reads it).
|
||||
LENGTH_MAX_S = 24 * 3600
|
||||
|
||||
|
||||
def known_length(seconds: int | None) -> int | None:
|
||||
"""`seconds` as a video's length, or None when it is not one (see LENGTH_MAX_S)."""
|
||||
return seconds if seconds is not None and seconds <= LENGTH_MAX_S else None
|
||||
|
||||
|
||||
class Video(Base, TimestampMixin):
|
||||
"""A YouTube video. Shared across users; per-user state lives elsewhere."""
|
||||
@@ -141,8 +161,8 @@ class Video(Base, TimestampMixin):
|
||||
)
|
||||
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
|
||||
# The feed's views / duration / title sorts each ride a migration-defined index in their exact
|
||||
# ORDER BY, one per direction (and per ICU collation for title): migration 0096, held to
|
||||
# `feedquery.order_by` by test_feed_sort_indexes.
|
||||
# ORDER BY, one per direction (and per ICU collation for title): migration 0096 (the duration
|
||||
# pair since 0117, on `length_seconds`), held to `feedquery.order_by` by test_feed_sort_indexes.
|
||||
duration_seconds: Mapped[int | None] = mapped_column(Integer)
|
||||
view_count: Mapped[int | None] = mapped_column(BigInteger)
|
||||
like_count: Mapped[int | None] = mapped_column(BigInteger)
|
||||
@@ -197,6 +217,37 @@ class Video(Base, TimestampMixin):
|
||||
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
unavailable_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
unavailable_reason: Mapped[str | None] = mapped_column(String(24))
|
||||
# A random number in [0, 1), drawn once when the row is written: the ring a seeded shuffle walks
|
||||
# (`feedquery.shuffle_walk`, index ix_videos_shuffle_key on (shuffle_key, id), migration 0118).
|
||||
shuffle_key: Mapped[float] = mapped_column(Float, nullable=False, server_default=func.random())
|
||||
|
||||
@hybrid_property
|
||||
def length_seconds(self) -> int | None:
|
||||
"""The video's length, or None when it is unknown — not fetched yet, or not a length at all
|
||||
(`LENGTH_MAX_S`). What every reader of a length reads."""
|
||||
return known_length(self.duration_seconds)
|
||||
|
||||
@length_seconds.inplace.expression
|
||||
@classmethod
|
||||
def _length_seconds_expression(cls) -> ColumnElement[int | None]:
|
||||
# The limit is a literal, not a bound parameter: the duration sort indexes (migration 0117)
|
||||
# are built on this very expression, and Postgres matches an index to a query's constant only.
|
||||
return case(
|
||||
(cls.duration_seconds > literal_column(str(LENGTH_MAX_S)), null()),
|
||||
else_=cls.duration_seconds,
|
||||
)
|
||||
|
||||
@hybrid_property
|
||||
def without_length(self) -> bool:
|
||||
"""A video whose length is unknown — live, an archive past a day, or not fetched yet: what a
|
||||
sum of lengths leaves out and names as "+N without length". Every such video, so the count
|
||||
and the sum never disagree about what was left out."""
|
||||
return self.length_seconds is None
|
||||
|
||||
@without_length.inplace.expression
|
||||
@classmethod
|
||||
def _without_length_expression(cls) -> ColumnElement[bool]:
|
||||
return cls.length_seconds.is_(None)
|
||||
|
||||
# Weighted full-text search document (DB-generated, never written by the app): title (A) >
|
||||
# creator keywords + search queries (B) > truncated description (C). ts_rank honours the
|
||||
|
||||
@@ -44,7 +44,8 @@ def channel_titles(db: Session, ids: set[str]) -> dict[str, str]:
|
||||
|
||||
|
||||
def channel_aggregates(channel_ids: list[str]) -> Select:
|
||||
"""Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
|
||||
"""Per-channel stored count, last upload and total duration (the lengths it knows: a video with
|
||||
none is left out, `Video.length_seconds`), in one grouped pass. Every column it
|
||||
reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
|
||||
alone; a test holds that (test_channel_list_index_only)."""
|
||||
return (
|
||||
@@ -52,7 +53,7 @@ def channel_aggregates(channel_ids: list[str]) -> Select:
|
||||
Video.channel_id,
|
||||
func.count(Video.id),
|
||||
func.max(Video.published_at),
|
||||
func.coalesce(func.sum(Video.duration_seconds), 0),
|
||||
func.coalesce(func.sum(Video.length_seconds), 0),
|
||||
)
|
||||
.where(Video.channel_id.in_(channel_ids))
|
||||
.group_by(Video.channel_id)
|
||||
|
||||
+101
-29
@@ -10,16 +10,25 @@ explore-visible and in-progress for free, and cannot drift from them. The Plex h
|
||||
narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
|
||||
sidebar and wrong for a landing page that does not.
|
||||
|
||||
**One seed drives both halves.** The feed orders a shuffle by `md5(id || seed)` so a keyset cursor
|
||||
stays valid across pages; the Plex surprise rail uses `ORDER BY random()`, which is fine for a single
|
||||
un-paged draw but cannot be reproduced. A cross-source reroll needs both halves to move together —
|
||||
otherwise "Mást kérek" re-draws the Plex picks while the YouTube ones stand still — so the Plex side
|
||||
here takes the feed's seeded-md5 idiom instead of `random()`.
|
||||
**One seed drives both halves.** The YouTube half starts on the ring of stored random keys at the
|
||||
seed's point and shuffles that window by `md5(id || seed)`; the Plex surprise rail uses
|
||||
`ORDER BY random()`, which is fine for a single un-paged draw but cannot be reproduced. A
|
||||
cross-source reroll needs both halves to move together — otherwise "Mást kérek" re-draws the Plex
|
||||
picks while the YouTube ones stand still — so the Plex side here orders by the same seeded md5
|
||||
instead of `random()`.
|
||||
"""
|
||||
|
||||
from sqlalchemy import and_, func, or_, select
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import (
|
||||
and_,
|
||||
func,
|
||||
or_,
|
||||
select,
|
||||
)
|
||||
from sqlalchemy.orm import Session, aliased
|
||||
|
||||
from app import draw
|
||||
from app.discover.budget import band
|
||||
from app.feedquery import (
|
||||
feed_columns,
|
||||
@@ -27,13 +36,14 @@ from app.feedquery import (
|
||||
not_blocked_cond,
|
||||
order_by,
|
||||
serialize,
|
||||
shuffle_walk,
|
||||
sort_keys,
|
||||
)
|
||||
from app.models import User
|
||||
from app.plex.models import PlexItem, PlexShow, PlexState
|
||||
from app.catalog.models import Channel, Video
|
||||
from app.reader.models import PlaylistItem, VideoState
|
||||
from app.plex import rollup
|
||||
from app.plex import affinity, rollup
|
||||
from app.plex.cards import display_title, leaf_card, movie_card, show_card
|
||||
from app.plex.query import apply_show_state_filter, content_libs
|
||||
from app.userscope import owned_by
|
||||
@@ -67,7 +77,8 @@ _FEED_BASE: dict = {
|
||||
|
||||
|
||||
def _seeded(col, seed: int):
|
||||
"""The feed's deterministic shuffle expression, reused for Plex so one seed moves both halves."""
|
||||
"""The Home surprise's seeded order (md5 of the key and the seed), the same for Plex so one
|
||||
seed moves both halves."""
|
||||
return func.md5(func.concat(col, str(seed)))
|
||||
|
||||
|
||||
@@ -112,17 +123,75 @@ def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) ->
|
||||
Shorts are opened up only for the shortest band: "I have ten minutes" is the one answer a Short is
|
||||
actually a good response to, and the feed's own default keeps them out otherwise."""
|
||||
lo, hi = band(budget)
|
||||
params = dict(_FEED_BASE, min_duration=lo, max_duration=hi, include_shorts=budget == "short")
|
||||
query, rank_expr = filtered_query(db, user, **params)
|
||||
keys = sort_keys("shuffle", seed, rank_expr)
|
||||
# Over-fetch, then drop anything already started. The feed's `unwatched` scope keeps watched and
|
||||
# hidden out but deliberately keeps in-progress rows in (they belong in the feed). Here they must
|
||||
# go: the resume tier has already had its chance at them, and it may have passed precisely because
|
||||
# the REMAINING time did not fit the budget — offering the same item back as a surprise, whose
|
||||
# runtime does fit, would label a half-finished thing "Meglepetés" and contradict the question.
|
||||
rows = db.execute(query.order_by(*order_by(keys)).limit(limit * 4)).all()
|
||||
out = [serialize(r) for r in rows]
|
||||
return [r for r in out if not r["position_seconds"]][:limit]
|
||||
params: dict = {
|
||||
**_FEED_BASE,
|
||||
"min_duration": lo,
|
||||
"max_duration": hi,
|
||||
"include_shorts": budget == "short",
|
||||
}
|
||||
query, _rank = filtered_query(db, user, **params)
|
||||
|
||||
# A window of the ring, not the whole scope: the seed's SURPRISE_WINDOW videos read along the
|
||||
# stored random key (`shuffle_walk`, a millisecond), then shuffled by the seed. Dealing straight
|
||||
# from the ring would favour for good every video that sits after a wide gap of it (the first
|
||||
# video past a random point is the one whose gap the point fell into); shuffling the window makes
|
||||
# each video's chance the window's share, even to within a few percent. A scope no larger than
|
||||
# the window is the whole scope, in exactly the md5 order the draw always had.
|
||||
#
|
||||
# Anything already started is dropped. The feed's `unwatched` scope keeps watched and hidden out
|
||||
# but deliberately keeps in-progress rows in (they belong in the feed). Here they must go: the
|
||||
# resume tier has already had its chance at them, and it may have passed precisely because the
|
||||
# REMAINING time did not fit the budget — offering the same item back as a surprise, whose runtime
|
||||
# does fit, would label a half-finished thing "Meglepetés" and contradict the question.
|
||||
window = [
|
||||
r
|
||||
for r in map(serialize, shuffle_walk(db, query, seed, SURPRISE_WINDOW))
|
||||
if not r["position_seconds"]
|
||||
]
|
||||
blind = draw.shuffled(window, seed, key=_video_id)
|
||||
channels = _channel_affinity(db, user)
|
||||
if not channels:
|
||||
return blind[:limit]
|
||||
|
||||
def weight(row: dict) -> float:
|
||||
return 1.0 + CHANNEL_WEIGHT * channels.get(row["channel_id"], 0.0)
|
||||
|
||||
leaning = draw.leaning(window, seed, key=_video_id, weight=weight)
|
||||
return draw.blend(leaning, blind, seed, limit, key=_video_id)
|
||||
|
||||
|
||||
# How many videos of the ring a surprise draws from: wide enough that a channel the user watches has
|
||||
# a few in it, small enough to read in a millisecond or two.
|
||||
SURPRISE_WINDOW = 500
|
||||
|
||||
|
||||
def _video_id(row: dict) -> str:
|
||||
return row["id"]
|
||||
|
||||
|
||||
# How much a channel the user watches most outweighs one they never watch (1 + this), and how far back
|
||||
# the watched history counts: the latest WATCHED_RECENT watches, the recent ones weighing more.
|
||||
CHANNEL_WEIGHT = 3.0
|
||||
WATCHED_RECENT = 300
|
||||
WATCHED_HALF_LIFE_DAYS = 90
|
||||
|
||||
|
||||
def _channel_affinity(db: Session, user: User) -> dict[str, float]:
|
||||
"""Each channel the user watched lately, scored 0..1 against the most watched: the sum of its
|
||||
watches, each weighing half as much every WATCHED_HALF_LIFE_DAYS. Empty with no history."""
|
||||
rows = db.execute(
|
||||
select(Video.channel_id, VideoState.watched_at)
|
||||
.join(Video, Video.id == VideoState.video_id)
|
||||
.where(owned_by(VideoState.user_id, user), VideoState.status == "watched")
|
||||
.order_by(VideoState.watched_at.desc().nullslast())
|
||||
.limit(WATCHED_RECENT)
|
||||
).all()
|
||||
now = datetime.now(timezone.utc)
|
||||
scores: dict[str, float] = {}
|
||||
for channel_id, watched_at in rows:
|
||||
r = draw.recency(watched_at, now, WATCHED_HALF_LIFE_DAYS)
|
||||
scores[channel_id] = scores.get(channel_id, 0.0) + r
|
||||
return draw.scaled(scores)
|
||||
|
||||
|
||||
def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
|
||||
@@ -289,21 +358,24 @@ def plex_surprise(
|
||||
if not movie_libs:
|
||||
return []
|
||||
lo, hi = band(budget)
|
||||
q = (
|
||||
select(PlexItem)
|
||||
.where(
|
||||
PlexItem.kind == "movie",
|
||||
PlexItem.library_id.in_(movie_libs),
|
||||
_unseen_movie_conds(uid),
|
||||
)
|
||||
.order_by(_seeded(PlexItem.rating_key, seed))
|
||||
.limit(limit)
|
||||
q = select(PlexItem).where(
|
||||
PlexItem.kind == "movie",
|
||||
PlexItem.library_id.in_(movie_libs),
|
||||
_unseen_movie_conds(uid),
|
||||
)
|
||||
if lo is not None:
|
||||
q = q.where(PlexItem.duration_s >= lo)
|
||||
if hi is not None:
|
||||
q = q.where(PlexItem.duration_s <= hi)
|
||||
items = db.execute(q).scalars().all()
|
||||
shuffled = _seeded(PlexItem.rating_key, seed)
|
||||
blind = db.execute(q.order_by(shuffled, PlexItem.id).limit(limit)).scalars().all()
|
||||
leaning = affinity.for_user(db, uid)
|
||||
if leaning:
|
||||
key = draw.weighted_key(draw.seeded_u(PlexItem.rating_key, seed), leaning.weight(PlexItem))
|
||||
weighted = db.execute(q.order_by(key, shuffled, PlexItem.id).limit(limit)).scalars().all()
|
||||
items = draw.blend(list(weighted), list(blind), seed, limit, key=lambda it: it.id)
|
||||
else:
|
||||
items = list(blind)
|
||||
states = rollup.item_state_map(db, uid, [it.id for it in items])
|
||||
return [movie_card(it, states.get(it.id), lang=lang) for it in items]
|
||||
|
||||
|
||||
@@ -14,10 +14,10 @@ Resolution works both before extraction (from source_kind / URL host) and after
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import yt_dlp
|
||||
from yt_dlp.networking.impersonate import ImpersonateTarget
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import egress
|
||||
from app.downloads.stated import StatedFailure
|
||||
|
||||
# Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
|
||||
@@ -33,6 +33,9 @@ class SourceAdapter:
|
||||
"""Generic source: yt-dlp handles everything and the ref is already a fetchable URL."""
|
||||
|
||||
key = "generic"
|
||||
# SEC-17: every fetch of this source's download goes through the egress gate (`egress`). A
|
||||
# source's own host list is no guarantee — its pages, redirects and formats can point anywhere.
|
||||
egress_guarded = True
|
||||
|
||||
# --- extraction (worker._extract_with_fallback / backfill) ---------------------------------
|
||||
def resolve_url(self, url: str, ydl_opts: dict) -> str:
|
||||
@@ -66,6 +69,11 @@ class SourceAdapter:
|
||||
"""The download's raw title, before `titles.normalize_title` cleans it. Generic: yt-dlp's."""
|
||||
return info.get("title")
|
||||
|
||||
def is_placeholder_title(self, title: str | None) -> bool:
|
||||
"""Whether `title` is a stand-in the source gives in place of the real one (worth one more,
|
||||
signed-out read of the page). Generic: never."""
|
||||
return False
|
||||
|
||||
def uploader_url(self, info: dict) -> str | None:
|
||||
"""The channel/uploader page URL. Generic: whatever yt-dlp exposes directly (YouTube gives
|
||||
`channel_url`), else None (reddit / a raw media URL have no derivable channel page)."""
|
||||
@@ -87,6 +95,9 @@ class SourceAdapter:
|
||||
|
||||
class YouTubeAdapter(SourceAdapter):
|
||||
key = "youtube"
|
||||
# Not through the gate: YouTube's fetches go to Google's fixed hosts, and the PO-token provider
|
||||
# it needs is the sidecar on our internal network, which the gate exists to refuse.
|
||||
egress_guarded = False
|
||||
# Errors worth retrying with a different player-client set (YouTube per-client flakiness).
|
||||
_RETRIABLE = (
|
||||
"not a bot",
|
||||
@@ -174,7 +185,7 @@ class FacebookAdapter(SourceAdapter):
|
||||
post_id = facebook_group_post_id(url)
|
||||
if post_id is None or not ydl_opts.get("cookiefile"):
|
||||
return url
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
with egress.open_ydl(ydl_opts) as ydl:
|
||||
raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
|
||||
page = raw.decode("utf-8", "replace")
|
||||
# Measured: a refused or expired cookie gets the login page (~450 KB, a login
|
||||
@@ -201,11 +212,17 @@ class FacebookAdapter(SourceAdapter):
|
||||
# Measured (2026-09-14): a SIGNED-IN read gives the placeholder "Video" (plain and as Chrome), and an
|
||||
# 18+ reel only readable as Chrome gives "Facebook"; the post's text is still in `description`. A
|
||||
# signed-out read titles the page with that same text behind the engagement counts, cut short.
|
||||
# Measured (2026-11-01, yt-dlp 2026.08.19): since 2026-10-28 every signed-in reel read gives
|
||||
# "Facebook" and NO description, while a signed-out read of the same reel still carries the text —
|
||||
# so a placeholder left with nothing behind it gets one signed-out read (`backfill.public_title_info`).
|
||||
_PLACEHOLDER_TITLES = frozenset({"video", "facebook"})
|
||||
|
||||
def is_placeholder_title(self, title: str | None) -> bool:
|
||||
return (title or "").strip().lower() in self._PLACEHOLDER_TITLES
|
||||
|
||||
def title(self, info: dict) -> str | None:
|
||||
raw = info.get("title")
|
||||
if (raw or "").strip().lower() not in self._PLACEHOLDER_TITLES:
|
||||
if not self.is_placeholder_title(raw):
|
||||
return raw
|
||||
line = next(
|
||||
(ln.strip() for ln in (info.get("description") or "").splitlines() if ln.strip()), ""
|
||||
|
||||
@@ -5,7 +5,10 @@ auto-detected channel becomes a clickable link) and `poster_path` (an ffmpeg fra
|
||||
thumbnail-less sources, so the card / watch page / link preview still shows an image). Best-effort
|
||||
and safe to re-run: it only touches assets still missing the field, and only re-fetches YouTube /
|
||||
Facebook sources (a direct media URL like a reddit HLS manifest carries no channel and its signed
|
||||
URL may have expired — those just get a locally-generated poster instead).
|
||||
URL may have expired — those just get a locally-generated poster instead). A Facebook download
|
||||
titled with the source's placeholder ("Facebook", "Video") is retitled from a signed-out read of its
|
||||
page, and so is each download name that only repeated the placeholder. That page read
|
||||
(`public_title_info`) also titles every new download, from the worker.
|
||||
|
||||
Run once per environment:
|
||||
docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
|
||||
@@ -13,21 +16,23 @@ Run once per environment:
|
||||
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import yt_dlp
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import select, update
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import logs
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.downloads import adapters
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import formats, service
|
||||
from app.downloads.models import MediaAsset
|
||||
from app.downloads import egress, formats, service
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
from app.models import DISPLAY_TEXT_MAX
|
||||
from app.titles import normalize_title
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
|
||||
def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
|
||||
def extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
|
||||
"""Metadata-only yt-dlp extraction, using the source adapter's client sets + options (YouTube
|
||||
needs the POT/player-client strategy even just to extract)."""
|
||||
with formats.hiding(url):
|
||||
@@ -37,11 +42,15 @@ def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> d
|
||||
def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
|
||||
for clients in adapter.client_sets():
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "%(id)s.%(ext)s", lambda d: None, extra=adapter.ydl_opts_extra(clients)
|
||||
{},
|
||||
"%(id)s.%(ext)s",
|
||||
lambda d: None,
|
||||
extra=adapter.ydl_opts_extra(clients),
|
||||
egress=adapter.egress_guarded,
|
||||
)
|
||||
opts["skip_download"] = True
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(opts) as ydl:
|
||||
with egress.open_ydl(opts) as ydl:
|
||||
info = ydl.extract_info(url, download=False)
|
||||
if info:
|
||||
return info
|
||||
@@ -62,8 +71,43 @@ def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: in
|
||||
return None
|
||||
|
||||
|
||||
def public_title_info(info: dict, url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict:
|
||||
"""`info` to title a download from: itself, unless its title is a source's placeholder with
|
||||
nothing behind it (a signed-in Facebook read: "Facebook", no description). Then one signed-out,
|
||||
metadata-only read of the page lends its text — the description, else its title — as the
|
||||
description the adapter takes a placeholder's title from. Nothing better there: `info` as it was."""
|
||||
if not adapter.is_placeholder_title(adapter.title(info)):
|
||||
return info
|
||||
public = extract_meta(url, adapter, asset_id)
|
||||
text = public and (public.get("description") or public.get("title"))
|
||||
return {**info, "description": text} if text else info
|
||||
|
||||
|
||||
def _retitle(db: Session, a: MediaAsset, adapter: adapters.SourceAdapter, url: str) -> bool:
|
||||
"""Swap a placeholder title kept from an earlier download for the page's text, and the download
|
||||
names that only repeated it. A name its owner gave (anything else) stays."""
|
||||
if not adapter.is_placeholder_title(a.title):
|
||||
return False
|
||||
info = public_title_info({"title": a.title, "description": None}, url, adapter, a.id)
|
||||
named = normalize_title(adapter.title(info))
|
||||
if not named or adapter.is_placeholder_title(named):
|
||||
return False
|
||||
# One conditional write, not a read and a compare: a name its owner gives while the page is read
|
||||
# is not overwritten by what was read before it. The asset first, as every other writer of the
|
||||
# two locks them (`service.lock_own_job`): the session does not flush by itself.
|
||||
placeholder = a.title
|
||||
a.title = named
|
||||
db.flush()
|
||||
db.execute(
|
||||
update(DownloadJob)
|
||||
.where(DownloadJob.asset_id == a.id, DownloadJob.display_name == placeholder)
|
||||
.values(display_name=named[:DISPLAY_TEXT_MAX])
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def run() -> dict:
|
||||
filled_url = filled_poster = 0
|
||||
filled_url = filled_poster = retitled = 0
|
||||
with SessionLocal() as db:
|
||||
assets = (
|
||||
db.execute(
|
||||
@@ -78,7 +122,7 @@ def run() -> dict:
|
||||
adapter = adapters.resolve_adapter(a.source_kind, a.source_ref)
|
||||
if a.uploader_url is None and adapter.wants_metadata_backfill():
|
||||
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
|
||||
info = _extract_meta(url, adapter, a.id)
|
||||
info = extract_meta(url, adapter, a.id)
|
||||
uu = adapter.uploader_url(info) if info else None
|
||||
if uu:
|
||||
a.uploader_url = uu
|
||||
@@ -86,6 +130,14 @@ def run() -> dict:
|
||||
log.info(
|
||||
"asset %s uploader_url filled", a.id
|
||||
) # the URL names what they downloaded
|
||||
if adapter.wants_metadata_backfill():
|
||||
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
|
||||
if _retitle(db, a, adapter, url):
|
||||
# Before the poster step: it can cut a frame for 30 s, and these locks held
|
||||
# across it block a rename and deadlock a delete.
|
||||
db.commit()
|
||||
retitled += 1
|
||||
log.info("asset %s retitled", a.id) # the title names what they downloaded
|
||||
if a.poster_path is None:
|
||||
# Record a self-hosted poster for every download (the reliable og:image source):
|
||||
# ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
|
||||
@@ -99,6 +151,7 @@ def run() -> dict:
|
||||
"assets": len(assets),
|
||||
"filled_uploader_url": filled_url,
|
||||
"filled_poster": filled_poster,
|
||||
"retitled": retitled,
|
||||
}
|
||||
log.info("backfill done: %s", result)
|
||||
return result
|
||||
|
||||
@@ -12,6 +12,9 @@ given in fractions of the box it was drawn in, with that box's aspect: the YouTu
|
||||
tells its picture's size, so the frame there is drawn over the player's 16:9 box (a short's 9:16),
|
||||
and the cut maps it onto the real picture (`crop_filter`) — what lies over a letterbox bar is dropped.
|
||||
|
||||
A clip may be a GIF instead (`spec["gif"]`, downloads/gif): the same range, crop and tonemap, then a
|
||||
silent animated picture of at most 30 s rather than a video.
|
||||
|
||||
The spec is stored on the job (`DownloadJob.edit_spec`) and is the clip's whole identity: its
|
||||
signature is the asset's cache key, so marking the same range again is an instant cache hit, and a
|
||||
different range or audio track is a different clip.
|
||||
@@ -22,7 +25,9 @@ import json
|
||||
import math
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads.edit import AAC_ARGS, EVEN_FRAME, INPUT_GUARD, X264_ARGS
|
||||
from app.plex.sync import HDR_TRANSFERS
|
||||
|
||||
@@ -57,12 +62,29 @@ _CROP_ASPECTS = (0.25, 4.0)
|
||||
_EDGE_SLACK = 1e-3
|
||||
|
||||
|
||||
# Every reason a clip is refused; the UI names each one (`clip.errors.<reason>`, held by the frontend's
|
||||
# backendIds contract, which reads this Literal).
|
||||
ClipReason = Literal[
|
||||
"bad_source",
|
||||
"bad_range",
|
||||
"too_short",
|
||||
"too_long",
|
||||
"bad_audio",
|
||||
"bad_crop",
|
||||
"not_found",
|
||||
"media_missing",
|
||||
"past_end",
|
||||
"bad_gif",
|
||||
"gif_too_long",
|
||||
]
|
||||
|
||||
|
||||
class ClipError(Exception):
|
||||
"""A clip request that cannot be cut; `reason` is an i18n-mappable key, `params` the numbers its
|
||||
message names (the cap itself, so the client never says a stale one)."""
|
||||
|
||||
def __init__(self, reason: str, **params: float):
|
||||
self.reason = reason
|
||||
def __init__(self, reason: ClipReason, **params: float):
|
||||
self.reason: ClipReason = reason
|
||||
self.params = params
|
||||
super().__init__(reason)
|
||||
|
||||
@@ -74,9 +96,12 @@ def normalize(
|
||||
end_s: float,
|
||||
audio_track: int | None = None,
|
||||
crop: dict | None = None,
|
||||
gif_side: object = None,
|
||||
) -> dict:
|
||||
"""The validated, canonical clip spec. Times are rounded to the millisecond and a crop to four
|
||||
decimals, so the same marks always give the same cache key. Raises ClipError."""
|
||||
decimals, so the same marks always give the same cache key. A `gif_side` makes it a GIF of that
|
||||
size (`downloads/gif`), held to the GIF's own length cap; without one it is the video clip it
|
||||
always was, spec and cache key alike. Raises ClipError."""
|
||||
if source not in SOURCES:
|
||||
raise ClipError("bad_source")
|
||||
pattern = _YOUTUBE_ID if source == "youtube" else _RATING_KEY
|
||||
@@ -85,12 +110,27 @@ def normalize(
|
||||
if not (math.isfinite(start_s) and math.isfinite(end_s)) or start_s < 0:
|
||||
raise ClipError("bad_range")
|
||||
start, end = round(start_s, 3), round(end_s, 3)
|
||||
if end - start < MIN_CLIP_S:
|
||||
# Judged in whole milliseconds, as the ends are kept: a float difference lands a hair off
|
||||
# (60.529 - 30.529 = 30.000000000000004), which would refuse a range exactly at a cap.
|
||||
span_ms = round((end - start) * 1000)
|
||||
if span_ms < MIN_CLIP_S * 1000:
|
||||
raise ClipError("too_short")
|
||||
if end - start > MAX_CLIP_S:
|
||||
side = None
|
||||
if gif_side is not None:
|
||||
try:
|
||||
side = gifmod.normalize_side(gif_side)
|
||||
except ValueError as e:
|
||||
raise ClipError("bad_gif") from e
|
||||
if side is not None and gifmod.too_long(start, end):
|
||||
# Its own reason: a GIF's cap is said in seconds, a clip's in minutes.
|
||||
raise ClipError("gif_too_long", limit_s=gifmod.MAX_S)
|
||||
if span_ms > MAX_CLIP_S * 1000:
|
||||
raise ClipError("too_long", limit_s=MAX_CLIP_S)
|
||||
spec: dict = {"source": source, "ref": ref, "start_s": start, "end_s": end}
|
||||
if source == "plex" and audio_track is not None:
|
||||
if side is not None:
|
||||
spec["gif"] = {"side": side}
|
||||
# A GIF has no sound: the track the player was on is not part of what it is (its cache key).
|
||||
if source == "plex" and audio_track is not None and side is None:
|
||||
if not 0 <= audio_track <= _MAX_AUDIO_TRACK:
|
||||
raise ClipError("bad_audio")
|
||||
spec["audio_track"] = audio_track
|
||||
@@ -185,6 +225,16 @@ def _ratio(v: object) -> float | None:
|
||||
return n / d if n > 0 and d > 0 else None
|
||||
|
||||
|
||||
def quarter_turned(probe: dict) -> bool | None:
|
||||
"""Whether the first video stream of an ffprobe `-show_streams` JSON is stored turned a quarter
|
||||
(ffmpeg stands it upright before any filter, so its filters see width and height swapped). None
|
||||
when there is no video stream."""
|
||||
for s in probe.get("streams") or []:
|
||||
if s.get("codec_type") == "video":
|
||||
return _quarter_turned(s)
|
||||
return None
|
||||
|
||||
|
||||
def _quarter_turned(stream: dict) -> bool:
|
||||
rotation: object = (stream.get("tags") or {}).get("rotate")
|
||||
for sd in stream.get("side_data_list") or []:
|
||||
@@ -231,6 +281,12 @@ def audio_track_count(probe: dict) -> int:
|
||||
return sum(1 for s in probe.get("streams") or [] if s.get("codec_type") == "audio")
|
||||
|
||||
|
||||
def gif_pre(crop: str, hdr: bool) -> str:
|
||||
"""What a GIF clip's frames go through before the GIF's own size and palette: the crop, then an HDR
|
||||
source's tonemap (a palette made from PQ frames is as washed out as the frames are)."""
|
||||
return ",".join([*([crop] if crop else []), *([_TONEMAP] if hdr else [])])
|
||||
|
||||
|
||||
def build_cmd(
|
||||
src: Path,
|
||||
dest: Path,
|
||||
|
||||
@@ -25,6 +25,7 @@ Security controls enforced here:
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -163,9 +164,18 @@ def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredenti
|
||||
).scalar_one_or_none()
|
||||
|
||||
|
||||
# Every reason a pasted cookies.txt is refused; the UI names each one (`downloads.errors.cookies.<reason>`,
|
||||
# held by the frontend's backendIds contract, which reads this Literal).
|
||||
CookieReason = Literal["empty", "format"]
|
||||
|
||||
|
||||
class InvalidCookies(ValueError):
|
||||
"""The pasted cookies aren't a usable Netscape cookies.txt — surfaced as a 400 by the route."""
|
||||
|
||||
def __init__(self, reason: CookieReason):
|
||||
super().__init__(reason)
|
||||
self.reason: CookieReason = reason
|
||||
|
||||
|
||||
def validate_cookies(raw: str) -> str:
|
||||
"""Sanity-check a pasted cookies.txt (Netscape format) and return the cleaned text. yt-dlp reads
|
||||
|
||||
@@ -11,7 +11,9 @@ with the worker branching on the asset's `source_kind` to run ffmpeg instead of
|
||||
(the `-ss` seeks first), so a short clip is cheap even from a long source
|
||||
* crop (± trim) → always re-encode (a filter can't be stream-copied), frame-accurate
|
||||
|
||||
`edit_spec = {trim?: {start_s, end_s}, crop?: {x, y, w, h}, accurate?: bool}`. `accurate` is the
|
||||
`edit_spec = {trim?: {start_s, end_s}, crop?: {x, y, w, h, frame_w?, frame_h?}, accurate?: bool}`. A
|
||||
crop with `frame_w`/`frame_h` counts the picture as it is SHOWN, measured on a frame of that size
|
||||
(`crop_filter`); one without them is in the file's stored pixels (its older form). `accurate` is the
|
||||
per-edit user choice for a trim-only cut (ignored when a crop forces a re-encode anyway). `split`
|
||||
is a frontend concept: the
|
||||
UI fans a split out into N trim jobs, so there is one output file per job here.
|
||||
@@ -60,6 +62,46 @@ class EditAborted(Exception):
|
||||
_MAX_SEGMENTS = 100
|
||||
|
||||
|
||||
def _crop_frame(crop: dict) -> dict | None:
|
||||
"""The picture size a crop was measured on, or None when it has none it could be scaled by (a
|
||||
spec from before it was sent, which `crop_filter` cuts in the file's own pixels)."""
|
||||
try:
|
||||
w, h = int(crop["frame_w"]), int(crop["frame_h"])
|
||||
# silent-ok: no frame is a crop's older form, and it is still the crop that was asked for. An
|
||||
# infinite number (JSON `1e400`) is OverflowError, and it is no more a frame than "x" is.
|
||||
except (KeyError, TypeError, ValueError, OverflowError):
|
||||
return None
|
||||
return {"frame_w": w, "frame_h": h} if w > 0 and h > 0 else None
|
||||
|
||||
|
||||
def crop_filter(crop: dict) -> str:
|
||||
"""The ffmpeg `crop` of an edit. Its numbers count the picture as the browser reports it, which
|
||||
is its SHOWN size: a video with wide pixels (320 stored, 640 shown) would be cut off to one side
|
||||
in ffmpeg's stored ones, so they are scaled to `iw` / `ih` by the size they were measured on. A
|
||||
rotated file needs nothing more: both the browser and ffmpeg's filters see it upright."""
|
||||
frame_w, frame_h = crop.get("frame_w"), crop.get("frame_h")
|
||||
if not frame_w or not frame_h:
|
||||
return f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}"
|
||||
return (
|
||||
f"crop=iw*{crop['w']}/{frame_w}:ih*{crop['h']}/{frame_h}"
|
||||
f":iw*{crop['x']}/{frame_w}:ih*{crop['y']}/{frame_h}"
|
||||
)
|
||||
|
||||
|
||||
def crop_stored_size(crop: dict, src_w: int | None, src_h: int | None) -> tuple[int, int] | None:
|
||||
"""The size `crop_filter` cuts out of a `src_w` x `src_h` frame, evened as `EVEN_FRAME` evens it —
|
||||
the clip's size when ffprobe cannot tell. The frame is the one ffmpeg's filters see: a rotated
|
||||
file's stood upright, its stored size turned (the caller's to turn). A crop counted in the shown
|
||||
picture is a fraction of that frame, not its own pixel count; None when the size is unknown."""
|
||||
if crop.get("frame_w") and crop.get("frame_h"):
|
||||
if not src_w or not src_h:
|
||||
return None
|
||||
w, h = src_w * crop["w"] // crop["frame_w"], src_h * crop["h"] // crop["frame_h"]
|
||||
else:
|
||||
w, h = crop["w"], crop["h"]
|
||||
return w // 2 * 2, h // 2 * 2
|
||||
|
||||
|
||||
def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
"""Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
|
||||
|
||||
@@ -67,12 +109,18 @@ def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
spec = spec or {}
|
||||
out: dict = {}
|
||||
|
||||
# A GIF output (downloads/gif) keeps its size; one it cannot name is dropped here and refused by
|
||||
# the service, which reads the raw spec for exactly that (`gif_side`).
|
||||
side = gif_side(spec)
|
||||
if side is not None:
|
||||
out["gif"] = {"side": side}
|
||||
|
||||
# Crop applies to both a single trim and a multi-segment join.
|
||||
crop = spec.get("crop") or {}
|
||||
if crop:
|
||||
try:
|
||||
c = {k: int(round(float(crop[k]))) for k in ("x", "y", "w", "h")}
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
except (KeyError, TypeError, ValueError, OverflowError) as exc:
|
||||
# The clip still renders — uncropped. The user asked for a crop, got a whole frame,
|
||||
# and nothing anywhere called it a failure; this line is the only trace there is.
|
||||
log.warning(
|
||||
@@ -81,7 +129,8 @@ def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
)
|
||||
c = None
|
||||
if c and c["w"] > 0 and c["h"] > 0 and c["x"] >= 0 and c["y"] >= 0:
|
||||
out["crop"] = c
|
||||
frame = _crop_frame(crop)
|
||||
out["crop"] = {**c, **frame} if frame else c
|
||||
|
||||
# Multi-segment JOIN (cut-list → one concatenated file). Takes precedence over `trim`.
|
||||
segs = spec.get("segments")
|
||||
@@ -154,6 +203,21 @@ def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
return out
|
||||
|
||||
|
||||
def gif_side(spec: dict | None) -> int | None:
|
||||
"""The GIF size a raw edit spec asks for, or None when it asks for none — or for one that is not
|
||||
offered (the service tells the two apart by `"gif" in spec`)."""
|
||||
# Imported here: `gif` builds its command on this module's input guard.
|
||||
from app.downloads import gif as gifmod
|
||||
|
||||
raw = (spec or {}).get("gif")
|
||||
if not isinstance(raw, dict):
|
||||
return None
|
||||
try:
|
||||
return gifmod.normalize_side(raw.get("side"))
|
||||
except ValueError: # silent-ok: an unoffered size is refused by the service (`bad_gif`)
|
||||
return None
|
||||
|
||||
|
||||
def edit_sig(spec: dict) -> str:
|
||||
"""Stable 24-char signature of a (normalized) edit spec — the per-user cache identity."""
|
||||
payload = json.dumps(normalize_edit_spec(spec), sort_keys=True, separators=(",", ":"))
|
||||
@@ -165,8 +229,8 @@ def has_crop(spec: dict) -> bool:
|
||||
|
||||
|
||||
def needs_reencode(spec: dict) -> bool:
|
||||
"""A crop always re-encodes; a trim re-encodes only when the user asked for an accurate cut."""
|
||||
return bool(spec.get("crop")) or bool(spec.get("accurate"))
|
||||
"""A crop or a GIF always re-encodes; a trim only when the user asked for an accurate cut."""
|
||||
return bool(spec.get("crop")) or bool(spec.get("gif")) or bool(spec.get("accurate"))
|
||||
|
||||
|
||||
def clip_duration(spec: dict, source_duration: int | None) -> int | None:
|
||||
@@ -268,7 +332,7 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
|
||||
if reencode:
|
||||
# Input -ss + re-encode is frame-accurate (decodes from the prior keyframe, discards up to
|
||||
# the exact cut). A crop adds the filter; an accurate trim only evens the frame.
|
||||
cropf = f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}," if crop else ""
|
||||
cropf = f"{crop_filter(crop)}," if crop else ""
|
||||
cmd += ["-vf", cropf + EVEN_FRAME, *X264_ARGS, *AAC_ARGS]
|
||||
else:
|
||||
cmd += ["-c", "copy", "-avoid_negative_ts", "make_zero"]
|
||||
@@ -313,7 +377,7 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
|
||||
]
|
||||
|
||||
if needs_reencode(spec):
|
||||
cropf = f",crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}" if crop else ""
|
||||
cropf = f",{crop_filter(crop)}" if crop else ""
|
||||
parts, labels = [], []
|
||||
for i, s in enumerate(segs):
|
||||
st, en = s["start_s"], s["end_s"]
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
"""The egress gate: every fetch of a user-supplied download goes through here, and this is where the
|
||||
address is judged — at the moment of the connection, not when the URL was typed (SEC-17).
|
||||
|
||||
`urlguard` judges the URL the user typed, at enqueue and again in the worker. What that cannot see is
|
||||
every request yt-dlp makes after it: a public host answering 3xx with an internal `Location`, a page
|
||||
whose formats point inside, a name that resolves public for the check and private for the fetch (DNS
|
||||
rebinding). So a guarded download runs with yt-dlp's `proxy` set to this gate, a small HTTP proxy on
|
||||
the worker's loopback. For each request it resolves the host ONCE, refuses the connection if any
|
||||
address the name has is one the server must not fetch from (`urlguard.is_forbidden`), and otherwise
|
||||
connects to the very address it vetted — there is no second lookup for a rebinding answer to slip into.
|
||||
|
||||
Measured (yt-dlp 2026.8.19, 2026-11-06) on the worker image, with a logging proxy: the `proxy` option
|
||||
reaches every path a download takes — the Requests handler, the CurlCFFI one an impersonated request
|
||||
uses, the native HLS downloader and an ffmpeg HLS download (segments included). ffmpeg itself also
|
||||
speaks rtmp, rtsp, ftp, tcp and more, which no HTTP proxy covers, so a guarded download keeps only the
|
||||
formats it fetches over http(s) (`HttpOnlyFormats`, added by `open_ydl`) and fetches HLS natively.
|
||||
|
||||
A refused connection is answered `403` before anything is dialled, so a live internal port and a dead
|
||||
one read the same; `errors.user_error_text` then stores the one redacted sentence for both.
|
||||
|
||||
Not used for YouTube: those downloads talk to Google's fixed hosts and to the PO-token sidecar, which
|
||||
lives on the internal network on purpose (`adapters.YouTubeAdapter.egress_guarded`).
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import select
|
||||
import socket
|
||||
import socketserver
|
||||
import threading
|
||||
from collections.abc import Callable
|
||||
from typing import Any, cast
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import yt_dlp
|
||||
from yt_dlp.postprocessor.common import PostProcessor
|
||||
|
||||
from app import logs
|
||||
from app.downloads import urlguard
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
Resolver = Callable[..., list]
|
||||
Forbidden = Callable[[ipaddress.IPv4Address | ipaddress.IPv6Address], bool]
|
||||
|
||||
_BLOCKED = b"HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
_BAD_GATEWAY = b"HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
_BAD_REQUEST = b"HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
_MAX_HEAD = 64 * 1024
|
||||
# A tunnel with no traffic either way for this long is closed. Above yt-dlp's own socket_timeout
|
||||
# (30 s, `formats.build_ydl_opts`), so the gate never ends a transfer yt-dlp would still wait for.
|
||||
_IDLE_S = 120
|
||||
# Hop-by-hop headers a proxy must not pass on; `Connection: close` replaces them, so one connection
|
||||
# carries one request and a reused client connection can never be steered to a second host unvetted.
|
||||
_HOP = (b"proxy-", b"connection:", b"keep-alive:")
|
||||
|
||||
|
||||
def vetted_address(
|
||||
host: str, port: int, *, resolve: Resolver = socket.getaddrinfo, forbidden: Forbidden
|
||||
) -> tuple[str, int] | None:
|
||||
"""The address to connect to for `host:port`, or None when it must not be fetched: the name has
|
||||
no answer, or ANY of its addresses is forbidden (a round-robin answer is not a coin flip). IPv4
|
||||
only, as every download is (`source_address` 0.0.0.0 in `formats.build_ydl_opts`)."""
|
||||
try:
|
||||
infos = resolve(host, port, socket.AF_INET, socket.SOCK_STREAM)
|
||||
except (socket.gaierror, UnicodeError, OSError):
|
||||
# silent-ok: "no answer" is a refusal, and the caller's refusal is the line that is logged.
|
||||
return None
|
||||
addrs = [info[4] for info in infos]
|
||||
if not addrs:
|
||||
return None
|
||||
for addr in addrs:
|
||||
try:
|
||||
ip = ipaddress.ip_address(addr[0])
|
||||
except ValueError:
|
||||
# silent-ok: an answer that is not an address is refused like a forbidden one, and the
|
||||
# caller logs the refusal.
|
||||
return None
|
||||
if forbidden(ip):
|
||||
return None
|
||||
return addrs[0][0], addrs[0][1]
|
||||
|
||||
|
||||
def _read_head(sock: socket.socket) -> bytes | None:
|
||||
buf = b""
|
||||
while b"\r\n\r\n" not in buf:
|
||||
if len(buf) > _MAX_HEAD:
|
||||
return None
|
||||
chunk = sock.recv(8192)
|
||||
if not chunk:
|
||||
return None
|
||||
buf += chunk
|
||||
return buf
|
||||
|
||||
|
||||
def _pipe(a: socket.socket, b: socket.socket) -> None:
|
||||
try:
|
||||
_relay(a, b)
|
||||
except OSError:
|
||||
# silent-ok: one side hung up mid-transfer (yt-dlp closes a connection once it has read a
|
||||
# file's size, or gives up on a slow host); whichever side cared reports it itself.
|
||||
return
|
||||
|
||||
|
||||
def _relay(a: socket.socket, b: socket.socket) -> None:
|
||||
socks = [a, b]
|
||||
while True:
|
||||
ready, _, _ = select.select(socks, [], [], _IDLE_S)
|
||||
if not ready:
|
||||
return
|
||||
for s in ready:
|
||||
data = s.recv(65536)
|
||||
if not data:
|
||||
return
|
||||
(b if s is a else a).sendall(data)
|
||||
|
||||
|
||||
class _Handler(socketserver.BaseRequestHandler):
|
||||
def handle(self) -> None:
|
||||
client: socket.socket = self.request
|
||||
head = _read_head(client)
|
||||
if head is None:
|
||||
return
|
||||
head_part, _, body = head.partition(b"\r\n\r\n")
|
||||
lines = head_part.split(b"\r\n")
|
||||
try:
|
||||
method, target, version = lines[0].decode("latin1").split(" ", 2)
|
||||
except ValueError:
|
||||
# silent-ok: the only client is yt-dlp on this loopback; a malformed request is answered
|
||||
# 400 and reaches its own error, which the job records.
|
||||
client.sendall(_BAD_REQUEST)
|
||||
return
|
||||
if method == "CONNECT":
|
||||
host, _, port_s = target.rpartition(":")
|
||||
host = host.strip("[]")
|
||||
upstream = self._dial(host, port_s)
|
||||
if upstream is None:
|
||||
return
|
||||
with upstream:
|
||||
client.sendall(b"HTTP/1.1 200 Connection established\r\n\r\n")
|
||||
if body:
|
||||
upstream.sendall(body)
|
||||
_pipe(client, upstream)
|
||||
return
|
||||
u = urlsplit(target)
|
||||
if u.scheme != "http" or not u.hostname:
|
||||
client.sendall(_BLOCKED)
|
||||
return
|
||||
try:
|
||||
port = u.port or 80
|
||||
except ValueError:
|
||||
# silent-ok: as for a malformed request line above — answered 400, recorded by the job.
|
||||
client.sendall(_BAD_REQUEST)
|
||||
return
|
||||
upstream = self._dial(u.hostname, str(port))
|
||||
if upstream is None:
|
||||
return
|
||||
path = (u.path or "/") + (f"?{u.query}" if u.query else "")
|
||||
headers = [h for h in lines[1:] if not h.lower().startswith(_HOP)]
|
||||
request = b"\r\n".join(
|
||||
[f"{method} {path} {version}".encode("latin1"), *headers, b"Connection: close"]
|
||||
)
|
||||
with upstream:
|
||||
upstream.sendall(request + b"\r\n\r\n" + body)
|
||||
_pipe(client, upstream)
|
||||
|
||||
def _dial(self, host: str, port_s: str) -> socket.socket | None:
|
||||
"""Connect to the address `host` was vetted at, or answer the client and return None."""
|
||||
client: socket.socket = self.request
|
||||
try:
|
||||
port = int(port_s)
|
||||
except ValueError:
|
||||
# silent-ok: as for a malformed request line above — answered 400, recorded by the job.
|
||||
client.sendall(_BAD_REQUEST)
|
||||
return None
|
||||
gate = cast("_Gate", self.server)
|
||||
addr = vetted_address(host, port, resolve=gate.resolve, forbidden=gate.forbidden)
|
||||
if addr is None:
|
||||
log.warning(
|
||||
"egress refused a connection to %s:%s",
|
||||
host, # log-pii-ok: the host a download tried to reach, never its path or query
|
||||
port,
|
||||
extra=logs.event("download.egress.refused", port=port),
|
||||
)
|
||||
client.sendall(_BLOCKED)
|
||||
return None
|
||||
try:
|
||||
return socket.create_connection(addr, timeout=_IDLE_S)
|
||||
except OSError:
|
||||
# silent-ok: a public host that does not answer; yt-dlp reports it as the job's own
|
||||
# network failure, which the job records.
|
||||
client.sendall(_BAD_GATEWAY)
|
||||
return None
|
||||
|
||||
|
||||
class _Gate(socketserver.ThreadingTCPServer):
|
||||
daemon_threads = True
|
||||
allow_reuse_address = True
|
||||
|
||||
def __init__(self, resolve: Resolver, forbidden: Forbidden):
|
||||
self.resolve = resolve
|
||||
self.forbidden = forbidden
|
||||
super().__init__(("127.0.0.1", 0), _Handler)
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
return f"http://127.0.0.1:{self.server_address[1]}"
|
||||
|
||||
def handle_error(self, request, client_address) -> None:
|
||||
# socketserver's default prints a bare traceback to stderr, outside the log; a connection the
|
||||
# gate could not finish is a line with its code instead.
|
||||
log.exception("egress gate dropped a connection", extra=logs.event("download.egress.error"))
|
||||
|
||||
|
||||
def start_gate(
|
||||
*, resolve: Resolver = socket.getaddrinfo, forbidden: Forbidden = urlguard.is_forbidden
|
||||
) -> _Gate:
|
||||
"""A gate serving on its own thread. The worker uses the one `proxy_url` starts; a test starts
|
||||
its own, with a resolver and a judgement it controls."""
|
||||
gate = _Gate(resolve, forbidden)
|
||||
threading.Thread(target=gate.serve_forever, name="egress-gate", daemon=True).start()
|
||||
return gate
|
||||
|
||||
|
||||
# The options key `formats.build_ydl_opts` marks a guarded download with; `open_ydl` reads it. yt-dlp
|
||||
# keeps unknown keys in its params and never acts on them.
|
||||
GUARDED = "siftlode_egress_guarded"
|
||||
|
||||
|
||||
def open_ydl(opts: dict) -> yt_dlp.YoutubeDL:
|
||||
"""The one way this package opens yt-dlp (a contract test holds it): a guarded download also
|
||||
gets `HttpOnlyFormats`, which yt-dlp takes only as an instance, not through the options dict."""
|
||||
# yt-dlp's stub types its params as a TypedDict; these are the plain dict `formats.build_ydl_opts`
|
||||
# assembles, the same every caller handed yt-dlp before.
|
||||
ydl = yt_dlp.YoutubeDL(opts) # type: ignore[arg-type]
|
||||
if opts.get(GUARDED):
|
||||
ydl.add_post_processor(HttpOnlyFormats(), when="pre_process")
|
||||
return ydl
|
||||
|
||||
|
||||
_gate: _Gate | None = None
|
||||
_gate_lock = threading.Lock()
|
||||
|
||||
|
||||
def proxy_url() -> str:
|
||||
"""This process's gate, started on first use (the worker's jobs share it)."""
|
||||
global _gate
|
||||
with _gate_lock:
|
||||
if _gate is None:
|
||||
_gate = start_gate()
|
||||
return _gate.url
|
||||
|
||||
|
||||
# The protocols fetched over http(s) through the proxy option — the only ones the gate sees: yt-dlp's
|
||||
# own fetchers (http, f4m, ism, the DASH segments and their generator, native HLS — which a bare m3u8
|
||||
# also gets, as a guarded download sets hls_prefer_native) and ffmpeg's http input (a live stream or
|
||||
# a clip), which takes the proxy too. Everything else (rtmp, rtsp, mms, ws) ffmpeg or another tool
|
||||
# would dial directly.
|
||||
_HTTP_PROTOCOLS = frozenset(
|
||||
{
|
||||
"http",
|
||||
"https",
|
||||
"f4m",
|
||||
"ism",
|
||||
"m3u8",
|
||||
"m3u8_native",
|
||||
"http_dash_segments",
|
||||
"http_dash_segments_generator",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class HttpOnlyFormats(PostProcessor):
|
||||
"""Drops every format a guarded download would fetch past the gate. Runs as `pre_process`, which
|
||||
yt-dlp re-reads its formats after ("The pre-processors may have modified the formats"), so the
|
||||
selector never sees them. With nothing left, yt-dlp reports no formats — a refusal, not a leak."""
|
||||
|
||||
def set_downloader(self, downloader):
|
||||
# Not the job's postprocessor hooks: they would report this filter as a "processing" step
|
||||
# before a byte is fetched (yt-dlp attaches them here, once per call).
|
||||
self._downloader = downloader
|
||||
|
||||
def run(self, information):
|
||||
# yt-dlp's stub types the info dict as a TypedDict with optional keys; it is a dict here.
|
||||
info = cast("dict[str, Any]", information)
|
||||
if info.get("formats"):
|
||||
info["formats"] = [f for f in info["formats"] if _over_http(f)]
|
||||
elif info.get("url") and not _over_http(info):
|
||||
info["url"] = None
|
||||
return [], information
|
||||
|
||||
|
||||
def _over_http(fmt: dict) -> bool:
|
||||
url = fmt.get("manifest_url") or fmt.get("url") or ""
|
||||
if urlsplit(url).scheme not in ("http", "https"):
|
||||
return False
|
||||
protocol = fmt.get("protocol") or urlsplit(fmt.get("url") or "").scheme
|
||||
return all(p in _HTTP_PROTOCOLS for p in protocol.split("+"))
|
||||
@@ -68,6 +68,10 @@ CONVERTED_TOO_BIG_MESSAGE = (
|
||||
"free up space, then try again."
|
||||
)
|
||||
UPLOAD_MISSING_MESSAGE = "The uploaded file is missing."
|
||||
# A GIF over `gif.MAX_BYTES`: not the storage left, a cap of the GIF's own (its own category).
|
||||
GIF_TOO_BIG_MESSAGE = (
|
||||
"The GIF came out larger than a GIF may be — mark a shorter range or choose a smaller size."
|
||||
)
|
||||
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
|
||||
|
||||
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
|
||||
@@ -77,6 +81,8 @@ _RULES: tuple[tuple[str, str], ...] = (
|
||||
# first — it is our wording, so it can never be a false match on an upstream message.
|
||||
("download storage is full", "quota_exceeded"),
|
||||
("larger than the download storage you have left", "quota_exceeded"),
|
||||
# Our own refusal of a GIF over its size cap — the range or the size, not the storage left.
|
||||
("larger than a gif may be", "gif_too_big"),
|
||||
# Our own redaction of a transport failure (see NETWORK_REDACTED_MESSAGE) — our wording too.
|
||||
(NETWORK_REDACTED_MESSAGE.lower(), "network"),
|
||||
# Our own verdicts on a post link (adapters.py) — our wording too: its page holds no single video,
|
||||
|
||||
@@ -24,6 +24,7 @@ kept it — and its `ready_at`, the moment it finished (`stamp_done`).
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from enum import StrEnum
|
||||
from typing import Literal
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from sqlalchemy import select
|
||||
@@ -181,12 +182,22 @@ def stamp_done(db: Session, job: DownloadJob) -> None:
|
||||
job.expiry_warned = False
|
||||
|
||||
|
||||
# Every folder change the rules refuse, answered as a generic refusal; the UI names each one
|
||||
# (`errors.refused.<reason>`, held by the frontend's backendIds contract, which reads this Literal).
|
||||
FolderReason = Literal[
|
||||
"file_folder_gone",
|
||||
"file_folder_name_taken",
|
||||
"file_folder_not_its_source",
|
||||
"file_folder_not_own",
|
||||
]
|
||||
|
||||
|
||||
class FolderError(Exception):
|
||||
"""A folder change the rules refuse; `reason` is the refusal the route answers with."""
|
||||
|
||||
def __init__(self, reason: str) -> None:
|
||||
def __init__(self, reason: FolderReason) -> None:
|
||||
super().__init__(reason)
|
||||
self.reason = reason
|
||||
self.reason: FolderReason = reason
|
||||
|
||||
|
||||
def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
|
||||
|
||||
@@ -28,6 +28,7 @@ import urllib.parse
|
||||
from collections.abc import Iterator
|
||||
|
||||
from app import logs
|
||||
from app.downloads import egress as egress_gate
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
@@ -294,6 +295,8 @@ def build_ydl_opts(
|
||||
extra: dict | None = None,
|
||||
cookiefile: str | None = None,
|
||||
max_bytes: int | None = None,
|
||||
*,
|
||||
egress: bool,
|
||||
) -> dict:
|
||||
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
|
||||
|
||||
@@ -309,7 +312,11 @@ def build_ydl_opts(
|
||||
a file announced larger than it rather than filling the disk and only then being caught by the
|
||||
quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
|
||||
a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
|
||||
pipeline also holds every produced file to what is left by its real size."""
|
||||
pipeline also holds every produced file to what is left by its real size.
|
||||
|
||||
`egress` (SEC-17) routes every fetch through the worker's egress gate, which refuses a connection
|
||||
into our own network, and keeps only the formats fetched over http(s). Required, with no default:
|
||||
a new caller has to say which kind of download it builds (`SourceAdapter.egress_guarded`)."""
|
||||
s = normalize(spec)
|
||||
h = s["max_height"]
|
||||
opts: dict = {
|
||||
@@ -364,6 +371,13 @@ def build_ydl_opts(
|
||||
if cookiefile:
|
||||
opts["cookiefile"] = cookiefile
|
||||
|
||||
# SEC-17. AFTER the adapter merge, like the cookie: an adapter must not be able to point a
|
||||
# guarded download past the gate.
|
||||
if egress:
|
||||
opts["proxy"] = egress_gate.proxy_url()
|
||||
opts["hls_prefer_native"] = True
|
||||
opts[egress_gate.GUARDED] = True
|
||||
|
||||
if s["mode"] == "a":
|
||||
opts["format"] = "bestaudio/best"
|
||||
opts["postprocessors"].append(
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""The GIF output of a clip or an edit: its limits and the one ffmpeg command every GIF goes through.
|
||||
|
||||
A GIF is a picture, not a video: no sound, a library `file` that previews as an image (ffprobe's `gif`
|
||||
format is one of `probe._IMAGE_FORMATS`), and never the source of another edit. The numbers were
|
||||
measured on real sources (2026-11-09/11):
|
||||
|
||||
- 12 frames a second, the long side capped (320 / 480 / 640): a width cap made a phone's portrait
|
||||
video three times the size of a landscape one (44 MB for 21 s at "480").
|
||||
- One palette per GIF, made from the GIF's own frames (`palettegen` → `paletteuse`, light ordered
|
||||
dither): the generic 256-colour palette bands every gradient.
|
||||
- 30 s at most. 480 px fills 12–29 MB in 30 s, depending on the picture; a GIF over `MAX_BYTES` is
|
||||
not kept, and the owner is told to cut less or smaller.
|
||||
- A chat link preview: Teams moves a GIF `og:image` up to 15 MB (22 MB showed nothing), so
|
||||
`PREVIEW_MAX_BYTES` decides whether the share page offers the GIF itself or its still (`ensure_still`).
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from app.downloads.edit import INPUT_GUARD
|
||||
|
||||
SIDES = (320, 480, 640)
|
||||
DEFAULT_SIDE = 480
|
||||
FPS = 12
|
||||
MAX_S = 30
|
||||
PREVIEW_MAX_BYTES = 15 * 1024 * 1024
|
||||
MAX_BYTES = 30 * 1024 * 1024
|
||||
|
||||
# The still is cut from the GIF we made, read with the GIF demuxer alone: a library file named .gif
|
||||
# whose bytes are a playlist must not open the paths inside it (see `edit.MEDIA_DEMUXERS`).
|
||||
_GIF_GUARD = ["-format_whitelist", "gif"]
|
||||
_STILL_TIMEOUT_S = 30
|
||||
|
||||
|
||||
def normalize_side(side: object) -> int:
|
||||
"""One of `SIDES`, or ValueError. A bool or a float is not a size, even when it equals one."""
|
||||
if type(side) is not int or side not in SIDES:
|
||||
raise ValueError(side)
|
||||
return side
|
||||
|
||||
|
||||
def too_long(start_s: float, end_s: float) -> bool:
|
||||
"""Over `MAX_S`, counted in whole milliseconds: the ends are kept to the millisecond, and their
|
||||
float difference may land a hair over (60.529 - 30.529 = 30.000000000000004), which would refuse
|
||||
the very 30 s the panels allow."""
|
||||
return round((end_s - start_s) * 1000) > MAX_S * 1000
|
||||
|
||||
|
||||
# A GIF has no pixel shape a viewer honours (browsers ignore its aspect byte), so a source stored
|
||||
# with wide or narrow pixels (an anamorphic DVD: 720x480 shown at 16:9) is first resampled to square
|
||||
# ones, or it shows squeezed. `sar` is 1 for a source that names none; even, as a 4:2:0 frame wants.
|
||||
_SQUARE_PIXELS = "scale=w='trunc(iw*sar/2)*2':h=ih,setsar=1"
|
||||
|
||||
|
||||
def _scale(side: int) -> str:
|
||||
"""Fit the long side into `side`, never upscaling a smaller picture; the short side keeps the
|
||||
aspect (-1: a GIF has no chroma grid that wants it even)."""
|
||||
return (
|
||||
f"scale=w='if(gte(iw,ih),min({side},iw),-1)'"
|
||||
f":h='if(gte(iw,ih),-1,min({side},ih))':flags=lanczos"
|
||||
)
|
||||
|
||||
|
||||
def filter_graph(pre: str, side: int) -> str:
|
||||
"""The GIF's filters: `pre` (a crop, an HDR tonemap) on the source's own frame first, then the
|
||||
frame rate, square pixels and the size, then a palette made from these frames and applied to them.
|
||||
A crop counts the stored frame (both callers' crops are fractions of `iw`/`ih`), so it comes
|
||||
before the pixels are squared."""
|
||||
head = ",".join([*([pre] if pre else []), f"fps={FPS}", _SQUARE_PIXELS, _scale(side)])
|
||||
return (
|
||||
f"[0:v:0]{head},split[a][b];"
|
||||
"[a]palettegen=stats_mode=diff[p];"
|
||||
"[b][p]paletteuse=dither=bayer:bayer_scale=5:diff_mode=rectangle"
|
||||
)
|
||||
|
||||
|
||||
def build_cmd(
|
||||
src: Path, dest: Path, start_s: float, dur_s: float, *, pre: str, side: int
|
||||
) -> list[str]:
|
||||
"""ffmpeg for one GIF: an input seek (the GIF starts on the marked frame), the first video stream
|
||||
through `filter_graph`, nothing else — no sound, subtitles, data or chapters."""
|
||||
return [
|
||||
"ffmpeg",
|
||||
"-y",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-nostdin",
|
||||
"-progress",
|
||||
"pipe:1",
|
||||
"-nostats",
|
||||
"-ss",
|
||||
f"{start_s:.3f}",
|
||||
*INPUT_GUARD,
|
||||
"-i",
|
||||
str(src),
|
||||
"-t",
|
||||
f"{dur_s:.3f}",
|
||||
"-filter_complex",
|
||||
filter_graph(pre, side),
|
||||
"-an",
|
||||
"-sn",
|
||||
"-dn",
|
||||
"-map_chapters",
|
||||
"-1",
|
||||
str(dest),
|
||||
]
|
||||
|
||||
|
||||
def size(path: Path) -> tuple[int, int] | None:
|
||||
"""(width, height) from a GIF's header — ffprobe reports none for an image format, and the share
|
||||
page's preview card wants them. None for anything that is not a GIF."""
|
||||
try:
|
||||
with path.open("rb") as f:
|
||||
head = f.read(10)
|
||||
# silent-ok: no size is what an unreadable file has; the card is then built without one.
|
||||
except OSError:
|
||||
return None
|
||||
if len(head) < 10 or head[:6] not in (b"GIF87a", b"GIF89a"):
|
||||
return None
|
||||
return int.from_bytes(head[6:8], "little"), int.from_bytes(head[8:10], "little")
|
||||
|
||||
|
||||
def ensure_still(path: Path) -> Path | None:
|
||||
"""A JPEG still of the GIF's first frame beside it (`<name>.jpg`, the poster sidecar every library
|
||||
file uses): the link preview for a chat that shows no GIF (Messenger) or a GIF too big to move.
|
||||
None when it cannot be cut — the card is then built without a picture."""
|
||||
dest = path.with_suffix(".jpg")
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
|
||||
*_GIF_GUARD, "-i", str(path), "-frames:v", "1", "-q:v", "3", str(dest),
|
||||
] # fmt: skip
|
||||
try:
|
||||
subprocess.run(cmd, capture_output=True, timeout=_STILL_TIMEOUT_S, check=True)
|
||||
# silent-ok: the still is best-effort; without it the card has no picture, which is true.
|
||||
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError):
|
||||
dest.unlink(missing_ok=True)
|
||||
return None
|
||||
return dest if dest.exists() else None
|
||||
@@ -127,14 +127,35 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
|
||||
# --- serialization -------------------------------------------------------------------------
|
||||
|
||||
|
||||
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
|
||||
def opens(db: Session, link: DownloadLink) -> str:
|
||||
"""What a link opens, in the words its address and its preview use: "folder", "video" or
|
||||
"file". A file still being made reads as the video it mostly is."""
|
||||
from app.downloads import mediakind
|
||||
|
||||
if link.folder_id is not None:
|
||||
return "folder"
|
||||
job = db.get(DownloadJob, link.job_id)
|
||||
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
|
||||
kind = asset.media_kind if asset else None
|
||||
return "file" if kind and kind != mediakind.VIDEO else "video"
|
||||
|
||||
|
||||
def share_path(db: Session, link: DownloadLink, token: str) -> str:
|
||||
"""The address a link is given out on: its path says what it opens (`opens`). Only a word for the
|
||||
reader, the token is the whole key: every prefix of `main.SHARE_PREFIXES` opens every link, so
|
||||
the links already sent on /watch/ work for good."""
|
||||
prefix = {"folder": "folder", "file": "file", "video": "watch"}[opens(db, link)]
|
||||
return f"/{prefix}/{token}"
|
||||
|
||||
|
||||
def owner_view(db: Session, link: DownloadLink, token: str | None = None) -> dict:
|
||||
"""What the file owner sees when managing their links. `token` is the one just made (it is in
|
||||
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
|
||||
token = token or recover(link)
|
||||
return {
|
||||
"id": link.id,
|
||||
"token": token,
|
||||
"url": f"{base_url}/watch/{token}" if token else None,
|
||||
"url": share_path(db, link, token) if token else None,
|
||||
"allow_download": link.allow_download,
|
||||
"has_password": link.password_hash is not None,
|
||||
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
|
||||
@@ -150,7 +171,7 @@ def public_meta(
|
||||
|
||||
Resolves the owner's per-download display overrides (custom title/channel) over the asset's
|
||||
auto-extracted values, and surfaces the source + any extra reference links as clickable URLs."""
|
||||
from app.downloads import mediakind, service
|
||||
from app.downloads import mediakind, preview, service
|
||||
|
||||
source_url = service.reference_url(job, asset)
|
||||
# Audio-only assets (mp3 rips, extracted audio) can't play inside a <video> element on WebKit
|
||||
@@ -168,6 +189,8 @@ def public_meta(
|
||||
"height": asset.height,
|
||||
"audio": asset.media_kind == mediakind.AUDIO,
|
||||
"media_kind": asset.media_kind,
|
||||
# A picture is shown as itself (the file route serves it inline); other files download.
|
||||
"preview": "image" if preview.kind(asset.media_kind, asset.container) == "image" else None,
|
||||
"size_bytes": asset.size_bytes,
|
||||
"container": asset.container,
|
||||
"allow_download": link.allow_download,
|
||||
|
||||
@@ -234,7 +234,9 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
|
||||
source_asset_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("media_assets.id", ondelete="SET NULL")
|
||||
)
|
||||
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
|
||||
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h,frame_w?,frame_h?}}, its
|
||||
# crop in the SHOWN picture's pixels when it carries the frame it was measured on (`edit.crop_filter`),
|
||||
# else in the file's stored ones. For clip
|
||||
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
|
||||
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
|
||||
edit_spec: Mapped[dict | None] = mapped_column(JSON)
|
||||
|
||||
+56
-18
@@ -1,4 +1,4 @@
|
||||
"""Server-rendered Open Graph tags for the public /watch/{token} page.
|
||||
"""Server-rendered Open Graph tags for the public page of a share link (/watch/, /file/, /folder/).
|
||||
|
||||
The watch page is a client-side SPA, so a social crawler (facebookexternalhit, Twitterbot, …) that
|
||||
fetches /watch/{token} would otherwise only see the generic index.html — a blank/greyed link card.
|
||||
@@ -14,8 +14,9 @@ from pathlib import Path
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import links as linksmod
|
||||
from app.downloads import storage
|
||||
from app.downloads import preview, storage
|
||||
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
|
||||
|
||||
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
|
||||
@@ -71,21 +72,51 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
|
||||
return None
|
||||
|
||||
|
||||
def _watch_tags(token: str, db: Session) -> str | None:
|
||||
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
|
||||
# Facebook's crawler (Messenger's previews): it never showed a GIF `og:image` and did show a JPEG one
|
||||
# (measured 2026-11-11), so it is given the still. Matched on its documented agent names.
|
||||
_FACEBOOK_AGENTS = ("facebookexternalhit", "facebot")
|
||||
|
||||
|
||||
def _is_facebook(user_agent: str | None) -> bool:
|
||||
agent = (user_agent or "").lower()
|
||||
return any(name in agent for name in _FACEBOOK_AGENTS)
|
||||
|
||||
|
||||
def _moving_image(
|
||||
token: str, asset: MediaAsset, user_agent: str | None
|
||||
) -> tuple[str, tuple[int, int] | None] | None:
|
||||
"""The shared GIF itself as the card's picture, and its size from its header — when a chat can
|
||||
move it: not for Facebook's crawler, and not over `gif.PREVIEW_MAX_BYTES` (Teams moved 15 MB and
|
||||
showed nothing at 22). None: the card takes the still, as any other file's does."""
|
||||
if preview.kind(asset.media_kind, asset.container) != "image" or asset.container != "gif":
|
||||
return None
|
||||
if _is_facebook(user_agent) or (asset.size_bytes or 0) > gifmod.PREVIEW_MAX_BYTES:
|
||||
return None
|
||||
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
|
||||
return f"{settings.app_base}/api/public/watch/{token}/file", gifmod.size(path) if path else None
|
||||
|
||||
|
||||
def _watch_tags(token: str, db: Session, user_agent: str | None = None) -> str | None:
|
||||
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card.
|
||||
`user_agent` is the crawler's: a GIF's card is the GIF for a chat that moves it (`_moving_image`)."""
|
||||
link = linksmod.find(db, token)
|
||||
if link is None or linksmod.is_expired(link):
|
||||
return None
|
||||
url = f"{settings.app_base}/watch/{token}"
|
||||
folder = link.folder_id is not None
|
||||
url = f"{settings.app_base}{linksmod.share_path(db, link, token)}"
|
||||
# What the card calls it, as its address does.
|
||||
kind = linksmod.opens(db, link)
|
||||
folder, video = kind == "folder", kind == "video"
|
||||
if link.password_hash:
|
||||
# Password-gated: don't leak the title/thumbnail in the unfurl.
|
||||
return "\n ".join(
|
||||
[
|
||||
"<title>Siftlode</title>",
|
||||
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
|
||||
_tag(
|
||||
"og:title",
|
||||
"Password-protected " + ("folder" if folder else "video" if video else "file"),
|
||||
),
|
||||
_tag("og:description", "Shared via Siftlode"),
|
||||
_tag("og:type", "website" if folder else "video.other"),
|
||||
_tag("og:type", "video.other" if video else "website"),
|
||||
_tag("og:url", url),
|
||||
_tag("og:site_name", "Siftlode"),
|
||||
]
|
||||
@@ -96,7 +127,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
|
||||
if job is None or asset is None or asset.status != "ready":
|
||||
return None
|
||||
title = (job.display_name or asset.title or "Video").strip()
|
||||
title = (job.display_name or asset.title or ("Video" if video else "File")).strip()
|
||||
channel = (job.display_uploader or asset.uploader or "").strip()
|
||||
# Pick the link-preview image. Facebook's scraper REJECTS our own poster ("Corrupted Image" in the
|
||||
# Sharing Debugger): the poster is an ffmpeg-produced JPEG whose non-standard marker order FB's
|
||||
@@ -107,7 +138,12 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
# non-password links, so the poster needs no grant.
|
||||
yt_id_match = re.search(r"i\.ytimg\.com/vi/([\w-]+)/", asset.thumbnail_url or "")
|
||||
image_dims: tuple[int, int] | None = None
|
||||
if yt_id_match:
|
||||
image_type = "image/jpeg"
|
||||
moving = _moving_image(token, asset, user_agent)
|
||||
if moving:
|
||||
image, image_dims = moving
|
||||
image_type = "image/gif"
|
||||
elif yt_id_match:
|
||||
image: str | None = f"https://i.ytimg.com/vi/{yt_id_match.group(1)}/hqdefault.jpg"
|
||||
image_dims = (480, 360)
|
||||
elif asset.poster_path:
|
||||
@@ -118,7 +154,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
f"<title>{html.escape(title)}</title>",
|
||||
_tag("og:title", title),
|
||||
_tag("og:description", channel or "Shared via Siftlode"),
|
||||
_tag("og:type", "video.other"),
|
||||
_tag("og:type", "video.other" if video else "website"),
|
||||
_tag("og:url", url),
|
||||
_tag("og:site_name", "Siftlode"),
|
||||
_tag("twitter:title", title, attr="name"),
|
||||
@@ -128,12 +164,12 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
if image:
|
||||
# A real thumbnail → a large image card; without one, a plain (text) card is served.
|
||||
tags.append(_tag("og:image", image))
|
||||
# Declare the image is a public HTTPS JPEG with known dimensions, so FB (which fetches
|
||||
# og:image asynchronously on first scrape) includes it instead of unfurling title-only. For a
|
||||
# poster we read the real dimensions from its JPEG header; the ytimg hqdefault is a fixed 480×360.
|
||||
# Declare the image's type and known dimensions, so FB (which fetches og:image
|
||||
# asynchronously on first scrape) includes it instead of unfurling title-only. For a poster
|
||||
# we read the real dimensions from its JPEG header; the ytimg hqdefault is a fixed 480×360.
|
||||
if image.startswith("https://"):
|
||||
tags.append(_tag("og:image:secure_url", image))
|
||||
tags.append(_tag("og:image:type", "image/jpeg"))
|
||||
tags.append(_tag("og:image:type", image_type))
|
||||
if image_dims is None and image.endswith("poster.jpg") and asset.poster_path:
|
||||
# safe_abs_path returns None if the poster file is missing (gc'd) or escapes the root —
|
||||
# guard it, else _jpeg_size(None) would raise an uncaught AttributeError on this public page.
|
||||
@@ -173,11 +209,13 @@ def _folder_tags(db: Session, link: DownloadLink, url: str) -> str | None:
|
||||
)
|
||||
|
||||
|
||||
def render_watch_html(token: str, page_html: Path, db: Session) -> str | None:
|
||||
def render_watch_html(
|
||||
token: str, page_html: Path, db: Session, user_agent: str | None = None
|
||||
) -> str | None:
|
||||
"""Return the watch page (watch.html) with the OG block replaced for this token, or None to
|
||||
serve the generic page unchanged (invalid/expired/not-ready link, or a template without the
|
||||
markers)."""
|
||||
tags = _watch_tags(token, db)
|
||||
markers). `user_agent` is the fetcher's, which picks a GIF's card picture."""
|
||||
tags = _watch_tags(token, db, user_agent)
|
||||
if tags is None:
|
||||
return None
|
||||
text = page_html.read_text(encoding="utf-8")
|
||||
|
||||
@@ -7,6 +7,7 @@ from datetime import date, datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import yt_dlp
|
||||
from sqlalchemy import update
|
||||
from yt_dlp.utils import download_range_func
|
||||
|
||||
from app import logs, sysconfig
|
||||
@@ -15,7 +16,9 @@ from app.db import SessionLocal
|
||||
from app.downloads import clip as clipmod
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import errors as dlerrors
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import adapters, credentials, formats, mediakind, playable, probe, quota
|
||||
from app.downloads import backfill, egress
|
||||
from app.downloads import service, storage
|
||||
from app.downloads import urlguard
|
||||
from app.downloads.jobdb import (
|
||||
@@ -392,6 +395,7 @@ def _extract_with_fallback(
|
||||
_make_progress_hook(job_id, asset_id),
|
||||
cookiefile=cookiefile,
|
||||
max_bytes=max_bytes,
|
||||
egress=adapter.egress_guarded,
|
||||
),
|
||||
)
|
||||
last_exc: Exception | None = None
|
||||
@@ -403,6 +407,7 @@ def _extract_with_fallback(
|
||||
extra=adapter.ydl_opts_extra(clients),
|
||||
cookiefile=cookiefile,
|
||||
max_bytes=max_bytes,
|
||||
egress=adapter.egress_guarded,
|
||||
)
|
||||
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
|
||||
if section is not None:
|
||||
@@ -413,7 +418,7 @@ def _extract_with_fallback(
|
||||
# (Per-ATTEMPT only; a single extract_info that itself runs past the stall timeout is still
|
||||
# killed — acceptable, as socket_timeout + extractor_retries bound one attempt well under it.)
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(opts) as ydl:
|
||||
with egress.open_ydl(opts) as ydl:
|
||||
return ydl.extract_info(target, download=True)
|
||||
except yt_dlp.utils.DownloadError as exc:
|
||||
last_exc = exc
|
||||
@@ -585,9 +590,14 @@ def _download(
|
||||
# missing duration hid the trim action. Probed before the sidecars so the .nfo runtime has it
|
||||
# too. Same probe the upload ingest uses.
|
||||
vcodec, acodec, width, height, duration_s = probe.resolve_media_facts(produced, info)
|
||||
# A placeholder title with nothing behind it ("Facebook" from a signed-in read) borrows the
|
||||
# page's text from one signed-out read; the download itself stays the signed-in one. The page
|
||||
# yt-dlp read, not the link queued: a group post's link is rewritten to its video, and
|
||||
# signed out the post itself is the login wall.
|
||||
titled = backfill.public_title_info(info, info.get("webpage_url") or url, adapter, asset_id)
|
||||
meta = storage.MediaMeta(
|
||||
video_id=info.get("id") or source_ref,
|
||||
title=normalize_title(adapter.title(info)) or source_ref,
|
||||
title=normalize_title(adapter.title(titled)) or source_ref,
|
||||
uploader=info.get("uploader") or info.get("channel") or "Unknown Channel",
|
||||
upload_date=_parse_upload_date(info.get("upload_date")),
|
||||
uniqueid_type=adapter.nfo_uniqueid_type(),
|
||||
@@ -830,6 +840,18 @@ def _fail_edit(
|
||||
)
|
||||
|
||||
|
||||
def _filter_frame(
|
||||
path: Path, width: int | None, height: int | None
|
||||
) -> tuple[int | None, int | None]:
|
||||
"""A file's frame as ffmpeg's filters see it: the stored size, turned when the stream is stored
|
||||
turned a quarter (ffmpeg stands it upright first). Unknown when the file cannot be read."""
|
||||
data = probe.probe_json(path)
|
||||
turned = clipmod.quarter_turned(data) if data else None
|
||||
if turned is None:
|
||||
return None, None
|
||||
return (height, width) if turned else (width, height)
|
||||
|
||||
|
||||
def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
"""Produce a per-user trim/crop clip from the source asset's downloaded file via ffmpeg."""
|
||||
with SessionLocal() as db:
|
||||
@@ -853,7 +875,8 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
|
||||
src_ext = src_path.suffix.lstrip(".").lower() or "mp4"
|
||||
crop = editmod.has_crop(edit_spec)
|
||||
out_ext = "mp4" if editmod.needs_reencode(edit_spec) else src_ext
|
||||
gif = edit_spec.get("gif")
|
||||
out_ext = "gif" if gif else "mp4" if editmod.needs_reencode(edit_spec) else src_ext
|
||||
total_s = editmod.clip_duration(edit_spec, src_dur) or src_dur or 0
|
||||
staging = _edit_staging_dir(asset_id)
|
||||
|
||||
@@ -864,7 +887,25 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
dest_staging = staging / f"out.{out_ext}"
|
||||
|
||||
_set_job(job_id, phase="editing", progress=0, speed_bps=None, eta_s=None)
|
||||
if edit_spec.get("segments"):
|
||||
if gif:
|
||||
# One range with an end (the service's check), cropped first if a frame was drawn, then
|
||||
# tonemapped as a clip GIF is: an HDR upload keeps its HLG/PQ tags, and a GIF has none,
|
||||
# so a palette made from those frames would be washed out.
|
||||
trim = edit_spec["trim"]
|
||||
start = trim.get("start_s") or 0.0
|
||||
data = probe.probe_json(src_path)
|
||||
cmd = gifmod.build_cmd(
|
||||
src_path,
|
||||
dest_staging,
|
||||
start,
|
||||
trim["end_s"] - start,
|
||||
pre=clipmod.gif_pre(
|
||||
editmod.crop_filter(edit_spec["crop"]) if crop else "",
|
||||
bool(data) and clipmod.is_hdr(data),
|
||||
),
|
||||
side=gif["side"],
|
||||
)
|
||||
elif edit_spec.get("segments"):
|
||||
cmd, prep = editmod.build_concat_plan(
|
||||
src_path, dest_staging, edit_spec, out_ext, staging
|
||||
)
|
||||
@@ -880,9 +921,12 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
)
|
||||
if not dest_staging.exists():
|
||||
raise StatedFailure("ffmpeg produced no output file")
|
||||
# A fast trim stream-copies the source's codecs and container, so the clip is only as
|
||||
# playable as its source: hold it to the same rule as every other ready file.
|
||||
dest_staging = _ensure_phone_safe(job_id, dest_staging, staging, "editing", total_s)
|
||||
if gif:
|
||||
_hold_gif_cap(dest_staging)
|
||||
else:
|
||||
# A fast trim stream-copies the source's codecs and container, so the clip is only as
|
||||
# playable as its source: hold it to the same rule as every other ready file.
|
||||
dest_staging = _ensure_phone_safe(job_id, dest_staging, staging, "editing", total_s)
|
||||
out_ext = dest_staging.suffix.lstrip(".").lower() or out_ext
|
||||
# New storage on the owner's footprint, held to what is left by its real size, as a
|
||||
# download and a player clip are.
|
||||
@@ -896,8 +940,18 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
storage.place_file(dest_staging, settings.files_root, rel)
|
||||
final = storage.abs_path(settings.files_root, rel)
|
||||
size = final.stat().st_size if final.exists() else None
|
||||
# The clip's real frame: a re-encode evens an odd crop or source (`editmod.EVEN_FRAME`).
|
||||
# The clip's real frame: a re-encode evens an odd crop or source (`editmod.EVEN_FRAME`). A
|
||||
# GIF's is in its header (ffprobe reports no stream size for an image format).
|
||||
_, _, out_w, out_h, _ = probe.probe_media(final)
|
||||
if gif:
|
||||
out_w, out_h = gifmod.size(final) or (None, None)
|
||||
poster = _gif_still(rel) if gif else None
|
||||
# Unsized, a crop's own size stands in, cut from the frame the filters saw.
|
||||
cut = (
|
||||
editmod.crop_stored_size(edit_spec["crop"], *_filter_frame(src_path, src_w, src_h))
|
||||
if crop and not (out_w and out_h)
|
||||
else None
|
||||
)
|
||||
|
||||
with SessionLocal() as db:
|
||||
asset = db.get(MediaAsset, asset_id)
|
||||
@@ -909,11 +963,12 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
asset.duration_s = total_s or asset.duration_s
|
||||
if out_w and out_h:
|
||||
asset.width, asset.height = out_w, out_h
|
||||
elif crop:
|
||||
asset.width = edit_spec["crop"]["w"]
|
||||
asset.height = edit_spec["crop"]["h"]
|
||||
elif cut:
|
||||
asset.width, asset.height = cut
|
||||
else:
|
||||
asset.width, asset.height = src_w, src_h
|
||||
asset.width, asset.height = (None, None) if crop else (src_w, src_h)
|
||||
if poster:
|
||||
asset.poster_path = poster
|
||||
asset.error = None
|
||||
asset.last_access_at = datetime.now(timezone.utc)
|
||||
job = db.get(DownloadJob, job_id)
|
||||
@@ -1031,6 +1086,19 @@ def _clip_crop(spec: dict, data: dict) -> str:
|
||||
return crop
|
||||
|
||||
|
||||
def _hold_gif_cap(out: Path) -> None:
|
||||
"""A GIF over its own size cap is not kept (`gif.MAX_BYTES`): a stated failure that names the
|
||||
range and the size, the two things that make it smaller."""
|
||||
if out.stat().st_size > gifmod.MAX_BYTES:
|
||||
raise StatedFailure(dlerrors.GIF_TOO_BIG_MESSAGE)
|
||||
|
||||
|
||||
def _gif_still(rel: str) -> str | None:
|
||||
"""The placed GIF's still (`gif.ensure_still`), as the poster's rel path beside it."""
|
||||
still = gifmod.ensure_still(storage.abs_path(settings.files_root, rel))
|
||||
return str(Path(rel).with_suffix(".jpg")) if still else None
|
||||
|
||||
|
||||
def _process_clip(job_id: int, asset_id: int) -> None:
|
||||
"""Cut an ad-hoc clip (downloads/clip): fetch or open its source, ONE re-encode to the clip
|
||||
shape, then into the user's library beside their edits."""
|
||||
@@ -1053,34 +1121,49 @@ def _process_clip(job_id: int, asset_id: int) -> None:
|
||||
raise StatedFailure("The source file could not be read.")
|
||||
audio_track = spec.get("audio_track")
|
||||
tracks = clipmod.audio_track_count(data)
|
||||
if tracks == 0:
|
||||
gif = spec.get("gif")
|
||||
if gif:
|
||||
# A GIF has no sound: no track is chosen, so none can be missing.
|
||||
audio_track = None
|
||||
elif tracks == 0:
|
||||
# A silent source: the player still names its track 0, and the clip is silent too.
|
||||
audio_track = None
|
||||
elif audio_track is not None and audio_track >= tracks:
|
||||
raise StatedFailure("The chosen audio track is not in the file.")
|
||||
crop = _clip_crop(spec, data)
|
||||
total_s = clipmod.duration(spec)
|
||||
out = staging / "clip.mp4"
|
||||
hdr = clipmod.is_hdr(data)
|
||||
out = staging / ("clip.gif" if gif else "clip.mp4")
|
||||
_set_job(job_id, phase="editing", progress=0, speed_bps=None, eta_s=None)
|
||||
editmod.run_ffmpeg(
|
||||
clipmod.build_cmd(
|
||||
cmd = (
|
||||
gifmod.build_cmd(
|
||||
src, out, start, total_s, pre=clipmod.gif_pre(crop, hdr), side=gif["side"]
|
||||
)
|
||||
if gif
|
||||
else clipmod.build_cmd(
|
||||
src,
|
||||
out,
|
||||
start,
|
||||
total_s,
|
||||
audio_track=audio_track,
|
||||
hdr=clipmod.is_hdr(data),
|
||||
hdr=hdr,
|
||||
threads=editmod.encode_thread_cap(),
|
||||
crop=crop,
|
||||
),
|
||||
)
|
||||
)
|
||||
editmod.run_ffmpeg(
|
||||
cmd,
|
||||
total_s,
|
||||
on_progress=_ffmpeg_progress_writer(job_id, "editing"),
|
||||
should_cancel=_cancel_poll(job_id),
|
||||
)
|
||||
if not out.exists():
|
||||
raise StatedFailure("ffmpeg produced no output file")
|
||||
# The phone-safe shape by construction; still held to the rule every ready file meets.
|
||||
out = _ensure_phone_safe(job_id, out, staging, "editing", round(total_s))
|
||||
if gif:
|
||||
_hold_gif_cap(out)
|
||||
else:
|
||||
# The phone-safe shape by construction; still held to the rule every ready file meets.
|
||||
out = _ensure_phone_safe(job_id, out, staging, "editing", round(total_s))
|
||||
# Held to the cap by its real size: a range fetch passes yt-dlp's `max_filesize` to its
|
||||
# ffmpeg downloader, which never reads it, and a Plex clip fetches nothing at all. Read
|
||||
# again, since other jobs of this owner may have landed during the encode.
|
||||
@@ -1092,6 +1175,8 @@ def _process_clip(job_id: int, asset_id: int) -> None:
|
||||
storage.place_file(out, settings.files_root, rel)
|
||||
final = storage.abs_path(settings.files_root, rel)
|
||||
vcodec, acodec, width, height, _ = probe.probe_media(final)
|
||||
if gif:
|
||||
width, height = gifmod.size(final) or (None, None)
|
||||
|
||||
with SessionLocal() as db:
|
||||
asset = db.get(MediaAsset, asset_id)
|
||||
@@ -1109,21 +1194,29 @@ def _process_clip(job_id: int, asset_id: int) -> None:
|
||||
asset.duration_s = round(total_s)
|
||||
asset.media_kind = mediakind.derive(vcodec, acodec)
|
||||
# The clip's own frame as its poster: the link preview shows the moment, and a Plex
|
||||
# item has no public thumbnail at all.
|
||||
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
|
||||
# item has no public thumbnail at all. A GIF's is its first frame, cut from the GIF.
|
||||
asset.poster_path = (
|
||||
_gif_still(rel) if gif else editmod.ensure_poster(asset, settings.files_root)
|
||||
)
|
||||
asset.error = None
|
||||
asset.last_access_at = datetime.now(timezone.utc)
|
||||
fields: dict = {"status": JobStatus.DONE, "progress": 100, "phase": None}
|
||||
# A video outside the catalog was queued as "Clip (clip)": the fetch has its title now.
|
||||
placeholder = service.clip_title(None, None)
|
||||
named = None
|
||||
if asset and fetched_title and asset.title == placeholder:
|
||||
named = service.clip_title(fetched_title, None)
|
||||
asset.title = named
|
||||
if job_row := db.get(DownloadJob, job_id):
|
||||
if job_row.display_name == placeholder:
|
||||
fields["display_name"] = named[:DISPLAY_TEXT_MAX]
|
||||
title = asset.title if asset else None
|
||||
finalized = service.finalize_if_running(db, job_id, **fields)
|
||||
if finalized and named:
|
||||
# Only a name that still repeats the placeholder, in one conditional write: one its
|
||||
# owner gave meanwhile stays.
|
||||
db.execute(
|
||||
update(DownloadJob)
|
||||
.where(DownloadJob.id == job_id, DownloadJob.display_name == placeholder)
|
||||
.values(display_name=named[:DISPLAY_TEXT_MAX])
|
||||
)
|
||||
db.commit()
|
||||
if finalized:
|
||||
_notify_download_done(db, user_id, title, job_id, kind="edit")
|
||||
|
||||
@@ -45,6 +45,16 @@ TEXT_EXTS: frozenset[str] = frozenset(
|
||||
|
||||
TEXT_TYPE = "text/plain; charset=utf-8"
|
||||
|
||||
# Every preview served inline — the library's viewer and a share link's picture — goes with these.
|
||||
# `sandbox` leaves the answer no script and an opaque origin, whatever the file turns out to hold (a
|
||||
# "gif" that is really a page); `frame-ancestors 'self'` lets the app's viewer frame a PDF (every
|
||||
# other answer keeps the app-wide 'none', main.py), with X-Frame-Options saying the same for old
|
||||
# engines.
|
||||
SANDBOX_HEADERS: dict[str, str] = {
|
||||
"Content-Security-Policy": "sandbox; frame-ancestors 'self'",
|
||||
"X-Frame-Options": "SAMEORIGIN",
|
||||
}
|
||||
|
||||
# A text preview is the file's head: enough to read, never a whole log loaded into a tab.
|
||||
TEXT_PREVIEW_BYTES = 1 << 20
|
||||
# A NUL byte in the first this-many bytes means the "text" is not text.
|
||||
|
||||
@@ -16,10 +16,12 @@ That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user
|
||||
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
|
||||
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
|
||||
is unknown until it finishes. CLIP jobs consult it before their fetch or encode and again with
|
||||
their output's real size before it is placed. Edit jobs do not consult it at all (tracked as SEC-19).
|
||||
their output's real size before it is placed. Edit jobs consult it with their output's real size
|
||||
before it is placed (`test_edit_cap.py`).
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
|
||||
from sqlalchemy import and_, func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -50,11 +52,16 @@ class QuotaLimits:
|
||||
unlimited: bool
|
||||
|
||||
|
||||
class QuotaExceeded(Exception):
|
||||
"""Raised by check_enqueue; `reason` is one of max_jobs|max_bytes (i18n key on the client)."""
|
||||
# Every cap a download can hit; the UI names each one (`downloads.errors.quota.<reason>`, held by the
|
||||
# frontend's backendIds contract, which reads this Literal).
|
||||
QuotaReason = Literal["max_jobs", "max_bytes"]
|
||||
|
||||
def __init__(self, reason: str, limit: int, current: int):
|
||||
self.reason = reason
|
||||
|
||||
class QuotaExceeded(Exception):
|
||||
"""Raised by check_enqueue; `reason` names the cap it hit (an i18n key on the client)."""
|
||||
|
||||
def __init__(self, reason: QuotaReason, limit: int, current: int):
|
||||
self.reason: QuotaReason = reason
|
||||
self.limit = limit
|
||||
self.current = current
|
||||
super().__init__(f"download quota exceeded: {reason} ({current}/{limit})")
|
||||
|
||||
@@ -10,6 +10,7 @@ handler); the folder, link and quota writes are `folders`, `links` and `quota`.
|
||||
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from sqlalchemy import delete as delete_rows
|
||||
from sqlalchemy import func, select, update
|
||||
@@ -22,6 +23,7 @@ from app.downloads import adapters
|
||||
from app.downloads import clip as clipmod
|
||||
from app.downloads import credentials
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import folders, formats, mediakind, quota, storage
|
||||
from app.downloads.status import HOLDING, JobKind, JobStatus
|
||||
from app.models import DISPLAY_TEXT_MAX, User
|
||||
@@ -39,11 +41,26 @@ from app.plex import paths as plex_paths
|
||||
from app.userscope import owned_by, owned_row
|
||||
|
||||
|
||||
class EditError(Exception):
|
||||
"""Bad edit request (empty spec / source not ready); `reason` is an i18n-mappable key."""
|
||||
# Every reason an edit is refused; the UI names each one (`downloads.errors.edit.<reason>`, held by
|
||||
# the frontend's backendIds contract, which reads this Literal).
|
||||
EditReason = Literal[
|
||||
"source_not_ready",
|
||||
"not_media",
|
||||
"empty_edit",
|
||||
"bad_gif",
|
||||
"gif_range",
|
||||
"gif_too_long",
|
||||
"gif_not_video",
|
||||
]
|
||||
|
||||
def __init__(self, reason: str):
|
||||
self.reason = reason
|
||||
|
||||
class EditError(Exception):
|
||||
"""Bad edit request (empty spec / source not ready / not media / a GIF it cannot make); `reason`
|
||||
names the sentence, `params` the numbers it names (a cap travels with its refusal)."""
|
||||
|
||||
def __init__(self, reason: EditReason, **params: float):
|
||||
self.reason: EditReason = reason
|
||||
self.params = params
|
||||
super().__init__(reason)
|
||||
|
||||
|
||||
@@ -147,7 +164,7 @@ def populate_from_catalog(db: Session, asset: MediaAsset) -> bool:
|
||||
return False
|
||||
asset.title = video.title
|
||||
asset.thumbnail_url = video.thumbnail_url
|
||||
asset.duration_s = video.duration_seconds
|
||||
asset.duration_s = video.length_seconds
|
||||
if video.published_at:
|
||||
asset.upload_date = video.published_at.date()
|
||||
channel = db.get(Channel, video.channel_id)
|
||||
@@ -249,6 +266,18 @@ def clip_title(source_title: str | None, display_name: str | None) -> str:
|
||||
return f"{base} (clip)"[:DISPLAY_TEXT_MAX]
|
||||
|
||||
|
||||
def _check_gif_edit(spec: dict) -> None:
|
||||
"""A GIF edit is one marked range with an end, at most `gif.MAX_S` long, of a size `gif` offers
|
||||
(a size it does not offer was dropped by `normalize_edit_spec`). Raises EditError."""
|
||||
if not spec.get("gif"):
|
||||
raise EditError("bad_gif")
|
||||
trim = spec.get("trim") or {}
|
||||
if spec.get("segments") or trim.get("end_s") is None:
|
||||
raise EditError("gif_range")
|
||||
if gifmod.too_long(trim.get("start_s", 0.0), trim["end_s"]):
|
||||
raise EditError("gif_too_long", limit_s=gifmod.MAX_S)
|
||||
|
||||
|
||||
def enqueue_edit(
|
||||
db: Session,
|
||||
user_id: int,
|
||||
@@ -274,6 +303,11 @@ def enqueue_edit(
|
||||
spec = editmod.normalize_edit_spec(edit_spec)
|
||||
if not spec:
|
||||
raise EditError("empty_edit")
|
||||
if "gif" in (edit_spec or {}):
|
||||
# A GIF is made of the first video stream: audio has none, or only its cover picture.
|
||||
if src_asset.media_kind != mediakind.VIDEO:
|
||||
raise EditError("gif_not_video")
|
||||
_check_gif_edit(spec)
|
||||
|
||||
quota.check_enqueue(db, user_id)
|
||||
|
||||
@@ -288,7 +322,10 @@ def enqueue_edit(
|
||||
asset.duration_s = editmod.clip_duration(spec, src_asset.duration_s)
|
||||
# A clip inherits its source's kind (a video edit stays video, an audio trim stays audio);
|
||||
# the ffmpeg edit worker doesn't re-probe codecs, so set it here where the source is known.
|
||||
asset.media_kind = src_asset.media_kind or mediakind.VIDEO
|
||||
# A GIF is a picture whatever it was cut from (downloads/gif).
|
||||
asset.media_kind = (
|
||||
mediakind.FILE if spec.get("gif") else src_asset.media_kind or mediakind.VIDEO
|
||||
)
|
||||
|
||||
job = DownloadJob(
|
||||
user_id=user_id,
|
||||
@@ -331,7 +368,7 @@ def _clip_source_facts(db: Session, spec: dict) -> dict:
|
||||
"title": video.title,
|
||||
"uploader": channel.title if channel else None,
|
||||
"thumbnail_url": video.thumbnail_url,
|
||||
"length_s": video.duration_seconds,
|
||||
"length_s": video.length_seconds,
|
||||
}
|
||||
item = plex_lookups.by_rating_key(db, PlexItem, spec["ref"])
|
||||
if item is None:
|
||||
@@ -378,7 +415,8 @@ def enqueue_clip(
|
||||
asset.uploader = facts["uploader"]
|
||||
asset.thumbnail_url = facts["thumbnail_url"]
|
||||
asset.duration_s = round(clipmod.duration(spec))
|
||||
asset.media_kind = mediakind.VIDEO
|
||||
# A GIF is a picture: no player and no editor, an image preview (downloads/gif).
|
||||
asset.media_kind = mediakind.FILE if spec.get("gif") else mediakind.VIDEO
|
||||
|
||||
job = DownloadJob(
|
||||
user_id=user_id,
|
||||
|
||||
@@ -10,22 +10,12 @@ What this closes: a URL whose host resolves to a loopback / private / link-local
|
||||
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
|
||||
differently between the two moments, and the worker is where the socket is actually opened.
|
||||
|
||||
What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
|
||||
the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
|
||||
|
||||
* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
|
||||
with an internal address.
|
||||
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
|
||||
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
|
||||
open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
|
||||
response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
|
||||
the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
|
||||
class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
|
||||
one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
|
||||
kept). Tracked as SEC-17.
|
||||
|
||||
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
|
||||
hostname into the box), which is the reachable one.
|
||||
What this alone cannot see is every request yt-dlp makes AFTER the typed URL: a public host
|
||||
answering 3xx with an internal ``Location``, and DNS rebinding (a name that passes this check and
|
||||
then resolves inside for yt-dlp's own lookup). Those are closed one layer down, by `egress`: a guarded
|
||||
download's every connection goes through a gate that judges the address it actually dials, with
|
||||
this module's `is_forbidden`. The checks here stay as the early, structured refusal — a user typing
|
||||
``http://127.0.0.1:6379`` hears why at once instead of from a failed job.
|
||||
|
||||
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
|
||||
user-controlled destination to guard.
|
||||
@@ -33,27 +23,36 @@ user-controlled destination to guard.
|
||||
|
||||
import ipaddress
|
||||
import socket
|
||||
from typing import Literal
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# Reason codes travel to the client inside the structured 422 detail, so they are part of the API
|
||||
# vocabulary (frontend `localizeDetail`) — keep them stable.
|
||||
REASON_SCHEME = "scheme" # not http(s)
|
||||
REASON_NO_HOST = "no_host" # URL carries no hostname
|
||||
REASON_MALFORMED = "malformed" # unparseable authority or port; urlparse raises late, so guard it
|
||||
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
|
||||
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
|
||||
# vocabulary — keep them stable. The UI names each one (`downloads.errors.urlBlocked.<reason>`, held by
|
||||
# the frontend's backendIds contract, which reads this Literal).
|
||||
UrlBlockedReason = Literal["scheme", "no_host", "malformed", "unresolvable", "private"]
|
||||
REASON_SCHEME: UrlBlockedReason = "scheme" # not http(s)
|
||||
REASON_NO_HOST: UrlBlockedReason = "no_host" # URL carries no hostname
|
||||
REASON_MALFORMED: UrlBlockedReason = (
|
||||
"malformed" # unparseable authority or port; urlparse raises late
|
||||
)
|
||||
REASON_UNRESOLVABLE: UrlBlockedReason = (
|
||||
"unresolvable" # DNS said nothing (not the same as "blocked")
|
||||
)
|
||||
REASON_PRIVATE: UrlBlockedReason = (
|
||||
"private" # resolves into a network the server must not fetch from
|
||||
)
|
||||
|
||||
|
||||
class BlockedUrl(Exception):
|
||||
"""A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
|
||||
REASON_* codes above."""
|
||||
|
||||
def __init__(self, reason: str):
|
||||
def __init__(self, reason: UrlBlockedReason):
|
||||
super().__init__(reason)
|
||||
self.reason = reason
|
||||
self.reason: UrlBlockedReason = reason
|
||||
|
||||
|
||||
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
def is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
"""Every address family we must not open a connection to. Checked explicitly rather than
|
||||
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
|
||||
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
|
||||
@@ -104,5 +103,5 @@ def check_url(raw: str) -> None:
|
||||
ip = ipaddress.ip_address(addr)
|
||||
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
|
||||
raise BlockedUrl(REASON_PRIVATE) from None
|
||||
if _is_forbidden(ip):
|
||||
if is_forbidden(ip):
|
||||
raise BlockedUrl(REASON_PRIVATE)
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
"""A weighted draw that stays a surprise: what "Surprise me" deals from.
|
||||
|
||||
A title's chance to come first is its weight over the pool's (Efraimidis–Spirakis: order by
|
||||
-ln(1-u)/w, u uniform per row). The weight is 1 + the title's affinity to what the user watches
|
||||
(`plex.affinity`, `discover.queries`), so a title nobody leans towards still comes up — only less
|
||||
often. Two things keep it a surprise rather than a ranking:
|
||||
|
||||
- **u is the seed's md5 of the row's key**, the Home surprise's seeded order. With every weight 1
|
||||
the order is exactly that md5 order, so a user with no history sees the unweighted draw, and one
|
||||
seed still moves both sources of the Home hero together. A draw that is not seeded (the Plex
|
||||
Discover rail) takes u from random().
|
||||
- **One slot of three is blind** (`blend`): dealt from the unweighted order, so the same few
|
||||
favourites cannot fill the rail.
|
||||
|
||||
The YouTube surprise draws over a window of the feed scope rather than all of it (`yt_surprise`):
|
||||
the window is read in SQL, and `shuffled` / `leaning` order it here with the same u.
|
||||
|
||||
Both affinities weigh a watch by `recency` and score against the strongest with `scaled`, so the two
|
||||
sources lean by one rule.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import math
|
||||
from collections.abc import Callable, Hashable
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import BigInteger, Float, cast, func, literal
|
||||
from sqlalchemy.dialects.postgresql import BIT
|
||||
from sqlalchemy.orm import InstrumentedAttribute
|
||||
from sqlalchemy.sql.elements import ColumnElement
|
||||
|
||||
# 2**32: the first 8 hex digits of an md5 as a fraction of one.
|
||||
_SPAN = 4294967296.0
|
||||
|
||||
# A watch with no date (an old import) counts as one this many half-lives ago.
|
||||
UNDATED_HALF_LIVES = 2
|
||||
|
||||
|
||||
def recency(watched_at: datetime | None, now: datetime, half_life_days: float) -> float:
|
||||
"""A watch's weight in an affinity: 1 now, half as much every `half_life_days`."""
|
||||
if watched_at is None:
|
||||
return 0.5**UNDATED_HALF_LIVES
|
||||
return 0.5 ** (max(0.0, (now - watched_at).total_seconds() / 86400) / half_life_days)
|
||||
|
||||
|
||||
def scaled(scores: dict[str, float]) -> dict[str, float]:
|
||||
"""Each score over the strongest, so the strongest is 1; empty when nothing scored."""
|
||||
top = max(scores.values(), default=0.0)
|
||||
return {k: v / top for k, v in scores.items()} if top else {}
|
||||
|
||||
|
||||
def seeded_u(
|
||||
key: ColumnElement[Any] | InstrumentedAttribute[Any], seed: int
|
||||
) -> ColumnElement[float]:
|
||||
"""A uniform u in [0, 1) per row, from md5(key || seed) — the order `md5(...)` ascending gives."""
|
||||
digest = func.md5(func.concat(key, str(seed)))
|
||||
head = cast(cast(literal("x") + func.substr(digest, 1, 8), BIT(32)), BigInteger)
|
||||
return cast(head, Float) / _SPAN
|
||||
|
||||
|
||||
def weighted_key(u: ColumnElement[float], weight: ColumnElement[float]) -> ColumnElement[float]:
|
||||
"""The row's place in a weighted draw, ascending: -ln(1-u)/w. Monotonic in u when every w is 1,
|
||||
so the unweighted order is unchanged."""
|
||||
return -func.ln(1.0 - u) / weight
|
||||
|
||||
|
||||
def random_u() -> ColumnElement[float]:
|
||||
return func.random()
|
||||
|
||||
|
||||
def u_of(key: str, seed: int) -> float:
|
||||
"""`seeded_u` in Python: the same u for the same key and seed."""
|
||||
return int(hashlib.md5(f"{key}{seed}".encode()).hexdigest()[:8], 16) / _SPAN
|
||||
|
||||
|
||||
def shuffled(items: list, seed: int, key: Callable[[Any], str]) -> list:
|
||||
"""`items` in the seed's md5 order — the order `ORDER BY md5(key || seed)` gives."""
|
||||
return sorted(items, key=lambda it: hashlib.md5(f"{key(it)}{seed}".encode()).hexdigest())
|
||||
|
||||
|
||||
def leaning(
|
||||
items: list, seed: int, key: Callable[[Any], str], weight: Callable[[Any], float]
|
||||
) -> list:
|
||||
"""`items` in the weighted draw's order (`weighted_key` in Python), ties in the md5 order."""
|
||||
order = {key(it): i for i, it in enumerate(shuffled(items, seed, key))}
|
||||
return sorted(
|
||||
items,
|
||||
key=lambda it: (-math.log(1.0 - u_of(key(it), seed)) / weight(it), order[key(it)]),
|
||||
)
|
||||
|
||||
|
||||
def blend(
|
||||
weighted: list, blind: list, offset: int, n: int, key: Callable[[Any], Hashable] = lambda x: x
|
||||
) -> list:
|
||||
"""`n` picks: slot i comes from `blind` when (i + offset) % 3 == 0, else from `weighted`, each
|
||||
skipping what an earlier slot took (`key` tells two picks apart). A side that runs out leaves the
|
||||
rest to the other."""
|
||||
out: list = []
|
||||
taken: set = set()
|
||||
sources = (iter(weighted), iter(blind))
|
||||
|
||||
def nxt(source):
|
||||
for item in source:
|
||||
if key(item) not in taken:
|
||||
return item
|
||||
return None
|
||||
|
||||
for i in range(n):
|
||||
first, second = (sources[1], sources[0]) if (i + offset) % 3 == 0 else sources
|
||||
item = nxt(first)
|
||||
if item is None:
|
||||
item = nxt(second)
|
||||
if item is None:
|
||||
break
|
||||
taken.add(key(item))
|
||||
out.append(item)
|
||||
return out
|
||||
+71
-12
@@ -13,7 +13,19 @@ import binascii
|
||||
import json
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import BigInteger, Select, and_, case, cast, false, func, or_, select
|
||||
from sqlalchemy import (
|
||||
BigInteger,
|
||||
Select,
|
||||
and_,
|
||||
case,
|
||||
cast,
|
||||
false,
|
||||
func,
|
||||
literal,
|
||||
or_,
|
||||
select,
|
||||
tuple_,
|
||||
)
|
||||
from sqlalchemy.orm import Session, aliased
|
||||
|
||||
from app.collation import collated
|
||||
@@ -68,7 +80,7 @@ def feed_columns(user: User, state) -> list:
|
||||
Channel.handle.label("channel_handle"),
|
||||
Video.published_at,
|
||||
Video.thumbnail_url,
|
||||
Video.duration_seconds,
|
||||
Video.length_seconds.label("length_seconds"),
|
||||
Video.view_count,
|
||||
Video.is_short,
|
||||
Video.live_status,
|
||||
@@ -88,7 +100,7 @@ def serialize(row) -> dict:
|
||||
"channel_url": _channel_url(row.channel_id, row.channel_handle),
|
||||
"published_at": iso(row.published_at),
|
||||
"thumbnail_url": row.thumbnail_url,
|
||||
"duration_seconds": row.duration_seconds,
|
||||
"duration_seconds": row.length_seconds,
|
||||
"view_count": row.view_count,
|
||||
"is_short": row.is_short,
|
||||
"live_status": row.live_status,
|
||||
@@ -260,9 +272,9 @@ def filtered_query(
|
||||
# OR across the picked channels; every other filter still applies on top.
|
||||
query = query.where(Video.channel_id.in_(channel_ids))
|
||||
if min_duration is not None:
|
||||
query = query.where(Video.duration_seconds >= min_duration)
|
||||
query = query.where(Video.length_seconds >= min_duration)
|
||||
if max_duration is not None:
|
||||
query = query.where(Video.duration_seconds <= max_duration)
|
||||
query = query.where(Video.length_seconds <= max_duration)
|
||||
if max_age_days is not None:
|
||||
cutoff = datetime.now(timezone.utc).timestamp() - max_age_days * 86400
|
||||
query = query.where(Video.published_at >= datetime.fromtimestamp(cutoff, tz=timezone.utc))
|
||||
@@ -503,10 +515,10 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
|
||||
"views": [{"expr": Video.view_count, "asc": False, "nullable": True, "kind": "num"}],
|
||||
"views_asc": [{"expr": Video.view_count, "asc": True, "nullable": True, "kind": "num"}],
|
||||
"duration_desc": [
|
||||
{"expr": Video.duration_seconds, "asc": False, "nullable": True, "kind": "num"}
|
||||
{"expr": Video.length_seconds, "asc": False, "nullable": True, "kind": "num"}
|
||||
],
|
||||
"duration_asc": [
|
||||
{"expr": Video.duration_seconds, "asc": True, "nullable": True, "kind": "num"}
|
||||
{"expr": Video.length_seconds, "asc": True, "nullable": True, "kind": "num"}
|
||||
],
|
||||
"title": [{"expr": title, "asc": True, "nullable": True, "kind": "str"}],
|
||||
"title_desc": [{"expr": title, "asc": False, "nullable": True, "kind": "str"}],
|
||||
@@ -536,15 +548,17 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
|
||||
},
|
||||
pub,
|
||||
],
|
||||
# Deterministic per-seed shuffle: the same seed yields the same total order, so
|
||||
# the cursor stays valid across pages of one shuffled run.
|
||||
# The ring of stored random keys from the seed's point (`shuffle_walk`): the keys at or after
|
||||
# it, then the ones before. The same seed yields the same total order, so the cursor stays
|
||||
# valid across pages; `feed_page` reads it as two index ranges, never as this sort.
|
||||
"shuffle": [
|
||||
{
|
||||
"expr": func.md5(func.concat(Video.id, str(seed))),
|
||||
"expr": case((Video.shuffle_key < shuffle_start(seed), 1), else_=0),
|
||||
"asc": True,
|
||||
"nullable": False,
|
||||
"kind": "str",
|
||||
}
|
||||
"kind": "num",
|
||||
},
|
||||
{"expr": Video.shuffle_key, "asc": True, "nullable": False, "kind": "num"},
|
||||
],
|
||||
}
|
||||
# Relevance (search-result ranking) only exists when there's a query to rank against;
|
||||
@@ -700,6 +714,43 @@ class BadCursor(ValueError):
|
||||
"""A `cursor` this module did not write (or one from another sort): the caller's bad input."""
|
||||
|
||||
|
||||
# The golden ratio's fraction: consecutive seeds land far apart on the ring, never next to each other.
|
||||
_GOLDEN = 0.6180339887498949
|
||||
|
||||
|
||||
def shuffle_start(seed: int) -> float:
|
||||
"""The point in [0, 1) where a seed's shuffle starts on the ring of `Video.shuffle_key`.
|
||||
|
||||
Folded to 32 bits first: an int past a float's range overflowed the product, and one past 2**53
|
||||
left the product no fraction to land on. A seed the client draws (under a billion) is unchanged."""
|
||||
return ((seed % 2**32) * _GOLDEN) % 1.0
|
||||
|
||||
|
||||
def shuffle_walk(
|
||||
db: Session, query: Select, seed: int, n: int, after: tuple[int, float, str] | None = None
|
||||
) -> list:
|
||||
"""Up to `n` rows of `query` in the seed's ring order: shuffle_key from the seed's point upwards,
|
||||
then from 0 up to that point. `after` is a cursor's (phase, key, id): the walk resumes past it.
|
||||
|
||||
Each arc is an ascending range read along ix_videos_shuffle_key with the scope's filters checked
|
||||
row by row, so a draw costs the rows it reads, not a sort of the scope — the md5(id || seed) order
|
||||
it replaced read and sorted every row the filters let through (~350 ms on prod for ~213k)."""
|
||||
start = shuffle_start(seed)
|
||||
phase_from, key, last_id = after if after is not None else (0, None, None)
|
||||
rows: list = []
|
||||
for phase in (0, 1):
|
||||
if phase < phase_from or len(rows) >= n:
|
||||
continue
|
||||
arc = Video.shuffle_key >= start if phase == 0 else Video.shuffle_key < start
|
||||
q = query.where(arc)
|
||||
if key is not None and phase == phase_from:
|
||||
q = q.where(
|
||||
tuple_(Video.shuffle_key, Video.id) > tuple_(literal(key), literal(last_id))
|
||||
)
|
||||
rows += db.execute(q.order_by(Video.shuffle_key, Video.id).limit(n - len(rows))).all()
|
||||
return rows
|
||||
|
||||
|
||||
def feed_page(
|
||||
db: Session,
|
||||
user: User,
|
||||
@@ -734,6 +785,14 @@ def feed_page(
|
||||
k["value"] = v
|
||||
if ranked and window is not None:
|
||||
rows = windowed_relevance_rows(db, query, keys, window, cursor_id, limit + 1)
|
||||
elif sort == "shuffle":
|
||||
after = None
|
||||
if vals is not None and cursor_id is not None:
|
||||
try:
|
||||
after = (int(vals[0]), float(vals[1]), str(cursor_id))
|
||||
except (TypeError, ValueError, OverflowError) as e: # Infinity, a 400-digit key
|
||||
raise BadCursor("Invalid feed cursor") from e
|
||||
rows = shuffle_walk(db, query, seed, limit + 1, after)
|
||||
else:
|
||||
query = query.order_by(*order_by(keys))
|
||||
if cursor_id is not None:
|
||||
|
||||
+15
-8
@@ -151,7 +151,7 @@ async def _reap_hls_sessions() -> None:
|
||||
somebody pressed play again or the process restarted.
|
||||
|
||||
A FINISHED session is reaped on the short rule here, not kept for the full idle window: an open
|
||||
player checkpoints every 10 s and that route touches the session, so a paused viewer is no longer
|
||||
player keeps checkpointing and that route touches the session, so a paused viewer is no longer
|
||||
invisible to this pass (see `reap_idle`). Keeping every finished session for ten minutes instead
|
||||
held one whole segmented copy of a film per item on the download volume."""
|
||||
while True:
|
||||
@@ -368,6 +368,10 @@ INDEX_HTML = STATIC_DIR / "index.html"
|
||||
# A public share link's own page: the player alone, without the app's bundle (frontend/watch.html).
|
||||
WATCH_HTML = STATIC_DIR / "watch.html"
|
||||
INDEX_HEADERS = {"Cache-Control": "no-cache"}
|
||||
# The paths a share link opens on, by what it opens (`links.share_path`). Any of them opens any link.
|
||||
# A new one also goes in frontend/public/robots.txt, the Vite dev entry and WatchPage's token parsing
|
||||
# (watchEntry.contract.test.ts).
|
||||
SHARE_PREFIXES = ("watch/", "file/", "folder/")
|
||||
app.mount(
|
||||
"/assets",
|
||||
StaticFiles(directory=STATIC_DIR / "assets", check_dir=False),
|
||||
@@ -381,18 +385,21 @@ async def index() -> FileResponse:
|
||||
|
||||
|
||||
@app.get("/watch/{token}")
|
||||
async def watch_page(token: str):
|
||||
"""Serve the public player page for a share link, with per-video Open Graph tags injected so the
|
||||
@app.get("/file/{token}")
|
||||
@app.get("/folder/{token}")
|
||||
async def share_page(token: str, request: Request):
|
||||
"""Serve the public page of a share link, with per-link Open Graph tags injected so the
|
||||
link unfurls richly (title/channel/thumbnail) in Messenger/social crawlers. A real browser
|
||||
ignores the extra tags and boots WatchPage; invalid/expired/password links fall back to the
|
||||
generic card."""
|
||||
generic card. A GIF's card picture depends on which crawler asks (`og._moving_image`), so the
|
||||
answer varies by User-Agent."""
|
||||
from app.downloads import og
|
||||
|
||||
with SessionLocal() as db:
|
||||
rendered = og.render_watch_html(token, WATCH_HTML, db)
|
||||
rendered = og.render_watch_html(token, WATCH_HTML, db, request.headers.get("user-agent"))
|
||||
if rendered is None:
|
||||
return FileResponse(WATCH_HTML, headers=INDEX_HEADERS)
|
||||
return HTMLResponse(rendered, headers=INDEX_HEADERS)
|
||||
return HTMLResponse(rendered, headers={**INDEX_HEADERS, "Vary": "User-Agent"})
|
||||
|
||||
|
||||
@app.get("/{full_path:path}")
|
||||
@@ -409,8 +416,8 @@ async def spa_fallback(full_path: str) -> FileResponse:
|
||||
# Real SPA-root assets (welcome images, favicon, robots.txt) rarely change and have
|
||||
# stable names, so a moderate cache is safe and satisfies the cache-lifetime audit.
|
||||
return FileResponse(candidate, headers={"Cache-Control": "public, max-age=604800"})
|
||||
# A /watch/ path the route above does not match (a trailing segment, no token) is still the
|
||||
# A share path the route above does not match (a trailing segment, no token) is still the
|
||||
# player's: WatchPage reads the token off the path and says when a link is gone.
|
||||
if full_path.startswith("watch/"):
|
||||
if full_path.startswith(SHARE_PREFIXES):
|
||||
return FileResponse(WATCH_HTML, headers=INDEX_HEADERS)
|
||||
return FileResponse(INDEX_HTML, headers=INDEX_HEADERS)
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
"""What a user's Plex history leans towards, as a weight for a surprise draw (`app.draw`).
|
||||
|
||||
Two signals, both already in the database (no TMDB call):
|
||||
- **genres** of the titles the user watched, the recent ones weighing more (a half-life of
|
||||
HALF_LIFE_DAYS), each scored 0..1 against the strongest;
|
||||
- the **TMDB recommendations** stored on those titles (`recommendations`), each scored by the most
|
||||
recent title that recommends it.
|
||||
|
||||
A title counts once however many of its episodes were watched — a binge of one show would otherwise
|
||||
outvote every film — and only the RECENT_TITLES latest titles count. A candidate's weight is
|
||||
1 + GENRE_WEIGHT × (the mean score of its genres) + REC_WEIGHT × (its recommendation score), so a
|
||||
title with no affinity keeps weight 1 and still comes up.
|
||||
"""
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import Float, cast, func, literal, literal_column, select
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.sql.elements import ColumnElement
|
||||
|
||||
from app import draw
|
||||
from app.plex.models import PlexItem, PlexShow, PlexState
|
||||
from app.userscope import owned_by
|
||||
|
||||
HALF_LIFE_DAYS = 180
|
||||
RECENT_TITLES = 200
|
||||
GENRE_WEIGHT = 2.0
|
||||
REC_WEIGHT = 3.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Affinity:
|
||||
genres: dict[str, float] = field(default_factory=dict)
|
||||
movie_recs: dict[str, float] = field(default_factory=dict)
|
||||
show_recs: dict[str, float] = field(default_factory=dict)
|
||||
|
||||
def __bool__(self) -> bool:
|
||||
return bool(self.genres or self.movie_recs or self.show_recs)
|
||||
|
||||
def weight(self, model: type[PlexItem] | type[PlexShow]) -> ColumnElement[float]:
|
||||
"""The draw weight of a `model` row (a movie or a show)."""
|
||||
scores = cast(literal(json.dumps(self.genres)), JSONB)
|
||||
recs = self.show_recs if model is PlexShow else self.movie_recs
|
||||
genre = func.jsonb_array_elements_text(model.genres).table_valued("value").alias("g")
|
||||
genre_aff = (
|
||||
select(
|
||||
func.coalesce(
|
||||
func.avg(func.coalesce(cast(scores.op("->>")(genre.c.value), Float), 0.0)),
|
||||
0.0,
|
||||
)
|
||||
)
|
||||
.select_from(genre)
|
||||
.scalar_subquery()
|
||||
)
|
||||
rec_aff = func.coalesce(
|
||||
cast(cast(literal(json.dumps(recs)), JSONB).op("->>")(model.guid), Float), 0.0
|
||||
)
|
||||
return literal_column("1.0") + GENRE_WEIGHT * genre_aff + REC_WEIGHT * rec_aff
|
||||
|
||||
|
||||
def for_user(db: Session, uid: int) -> Affinity:
|
||||
"""The user's affinity from their latest watched titles; empty (`not aff`) with no history."""
|
||||
# One key per title: a show for an episode, the item for a film. concat() of a NULL show_id is
|
||||
# just "s", which nullif turns back into NULL so the film's key takes over.
|
||||
title = func.coalesce(
|
||||
func.nullif(func.concat("s", PlexItem.show_id), "s"), func.concat("m", PlexItem.id)
|
||||
).label("title")
|
||||
latest = (
|
||||
select(title, PlexState.watched_at, PlexItem.id.label("item_id"), PlexItem.show_id)
|
||||
.join(PlexItem, PlexItem.id == PlexState.item_id)
|
||||
.where(owned_by(PlexState.user_id, uid), PlexState.status == "watched")
|
||||
.distinct(title)
|
||||
.order_by(title, PlexState.watched_at.desc().nullslast())
|
||||
.subquery()
|
||||
)
|
||||
rows = db.execute(
|
||||
select(
|
||||
latest.c.watched_at,
|
||||
PlexItem.genres,
|
||||
PlexItem.recommendations,
|
||||
PlexShow.genres,
|
||||
PlexShow.recommendations,
|
||||
)
|
||||
.select_from(latest)
|
||||
.join(PlexItem, PlexItem.id == latest.c.item_id)
|
||||
.outerjoin(PlexShow, PlexShow.id == latest.c.show_id)
|
||||
.order_by(latest.c.watched_at.desc().nullslast())
|
||||
.limit(RECENT_TITLES)
|
||||
).all()
|
||||
if not rows:
|
||||
return Affinity()
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
genres: dict[str, float] = {}
|
||||
movie_recs: dict[str, float] = {}
|
||||
show_recs: dict[str, float] = {}
|
||||
for watched_at, item_genres, item_recs, show_genres, show_recs_json in rows:
|
||||
r = draw.recency(watched_at, now, HALF_LIFE_DAYS)
|
||||
for g in (show_genres if show_genres is not None else item_genres) or []:
|
||||
genres[g] = genres.get(g, 0.0) + r
|
||||
for rec in (show_recs_json if show_recs_json is not None else item_recs) or []:
|
||||
if not rec.get("tmdb_id"):
|
||||
continue
|
||||
bucket = show_recs if rec.get("media_type") == "tv" else movie_recs
|
||||
guid = f"tmdb:{rec['tmdb_id']}"
|
||||
bucket[guid] = max(bucket.get(guid, 0.0), r)
|
||||
return Affinity(draw.scaled(genres), draw.scaled(movie_recs), draw.scaled(show_recs))
|
||||
@@ -22,7 +22,8 @@ from sqlalchemy.orm import Session
|
||||
from app.collation import sorted_words
|
||||
from app.discover.queries import RAIL_LIMIT
|
||||
from app.models import User
|
||||
from app.plex import rollup
|
||||
from app import draw
|
||||
from app.plex import affinity, rollup
|
||||
from app.plex.cards import SHOW_NEW, display_title, leaf_card, movie_card, show_card
|
||||
from app.plex.models import PlexItem, PlexShow, PlexState
|
||||
from app.plex.query import (
|
||||
@@ -399,22 +400,25 @@ def discover_page(db: Session, user: User, scope: str, lang: str, f: LibraryFilt
|
||||
}
|
||||
|
||||
|
||||
def _draw(db: Session, stmt, key_col, per: int, exclude: list[str]) -> list:
|
||||
def _draw(db: Session, stmt, key_col, per: int, exclude: list[str], weight=None) -> list:
|
||||
"""`per` random rows of `stmt`, the previous draw's (`exclude`) left out while the pool allows: a
|
||||
reshuffle over a small filtered pool kept dealing the same few titles. What the fresh rows leave
|
||||
short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail."""
|
||||
fresh = (
|
||||
db.execute(stmt.where(key_col.not_in(exclude)).order_by(func.random()).limit(per))
|
||||
.scalars()
|
||||
.all()
|
||||
if exclude
|
||||
else []
|
||||
)
|
||||
short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail.
|
||||
With a `weight` (`plex.affinity`) the draw leans on it, every third slot left blind (`app.draw`)."""
|
||||
|
||||
def pick(q, n: int) -> list:
|
||||
blind = db.execute(q.order_by(func.random()).limit(n)).scalars().all()
|
||||
if weight is None:
|
||||
return list(blind)
|
||||
key = draw.weighted_key(draw.random_u(), weight)
|
||||
leaning = db.execute(q.order_by(key).limit(n)).scalars().all()
|
||||
return draw.blend(list(leaning), list(blind), 1, n, key=lambda r: r.rating_key)
|
||||
|
||||
fresh = pick(stmt.where(key_col.not_in(exclude)), per) if exclude else []
|
||||
if len(fresh) == per:
|
||||
return list(fresh)
|
||||
return fresh
|
||||
taken = [key_col.not_in([r.rating_key for r in fresh])] if fresh else []
|
||||
rest = db.execute(stmt.where(*taken).order_by(func.random()).limit(per - len(fresh))).scalars()
|
||||
return [*fresh, *rest.all()]
|
||||
return [*fresh, *pick(stmt.where(*taken), per - len(fresh))]
|
||||
|
||||
|
||||
def _kept(
|
||||
@@ -474,6 +478,8 @@ def surprise(
|
||||
if keep:
|
||||
return {"scope": scope, "items": _kept(db, user.id, movie_ids, show_ids, keep, lang)}
|
||||
per = min(limit, SURPRISE_PER_KIND)
|
||||
# What the user's history leans towards; with none, every title weighs the same (a plain draw).
|
||||
leaning = affinity.for_user(db, user.id)
|
||||
movies = []
|
||||
if movie_ids:
|
||||
# Exclude movies this user has watched.
|
||||
@@ -501,6 +507,7 @@ def surprise(
|
||||
PlexItem.rating_key,
|
||||
per,
|
||||
exclude,
|
||||
leaning.weight(PlexItem) if leaning else None,
|
||||
)
|
||||
shows = []
|
||||
if show_ids:
|
||||
@@ -523,6 +530,7 @@ def surprise(
|
||||
PlexShow.rating_key,
|
||||
per,
|
||||
exclude,
|
||||
leaning.weight(PlexShow) if leaning else None,
|
||||
)
|
||||
mcards = _movie_cards(db, user.id, movies, lang)
|
||||
scards = _show_cards(db, user.id, shows, lang)
|
||||
|
||||
@@ -227,16 +227,17 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
|
||||
playable: Mapped[str | None] = mapped_column(String(12), index=True)
|
||||
probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# Transcode HW-capability facet (the playback-class filter). NULL for non-transcode items; for a
|
||||
# `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
|
||||
# AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
|
||||
# breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
|
||||
# set codec-only at sync (bit depth unknown from the cheap listing), then refined by the bit-depth
|
||||
# backfill (`_probe_media_batch`) which downgrades a 10-bit/HDR HEVC to False. The one filterable
|
||||
# home of the invariant `_want_hw`'s source gate computes at play time — kept in sync via the shared
|
||||
# classifier so the catalog facet and the runtime encoder choice never disagree by codec.
|
||||
# `playable="transcode"` item: True = the prod iGPU ENCODES it (a HW-decodable 8-bit SDR source, a
|
||||
# true-HDR one through the OpenCL tonemap, or an 8-bit source in a codec the CPU decodes for it —
|
||||
# VC-1, XviD, DivX), False = software libx264 (AV1, or a 10-bit SDR source — see plex/stream.py).
|
||||
# Derived by `plex.sync.transcode_hw_capable`: set codec-only at sync (bit depth unknown from the
|
||||
# cheap listing), then refined by the bit-depth backfill (`_probe_media_batch`), which downgrades a
|
||||
# 10-bit SDR HEVC to False. The filterable twin of the rule `_select_encode` applies at play time;
|
||||
# the two share their codec sets, not their code, so a change to one tree is a change to both.
|
||||
transcode_hw: Mapped[bool | None] = mapped_column(Boolean, index=True)
|
||||
# Video bit depth (8/10/12…) from the Plex media Stream, filled by the bit-depth backfill. NULL =
|
||||
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw`.
|
||||
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw` (a
|
||||
# 10-bit SDR source drops to software; an HDR one keeps the GPU through the tonemap).
|
||||
video_bit_depth: Mapped[int | None] = mapped_column(Integer)
|
||||
# Watermark for the bit-depth backfill: when this item's media Stream was last probed for
|
||||
# bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are
|
||||
|
||||
@@ -6,9 +6,9 @@ The rest of the mirror's writes stay with the sync (`plex.sync`): a collection's
|
||||
after a write-back, its local cleanup after a delete.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import CursorResult, delete, func, select, update
|
||||
from sqlalchemy import CursorResult, delete, func, or_, select, update
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -271,6 +271,10 @@ def reorder_playlist(db: Session, pl: PlexPlaylist, rating_keys: list[str]) -> N
|
||||
|
||||
# --- a user's watch state --------------------------------------------------------------------
|
||||
|
||||
# How long after a watched mark a mid-item position is read as the finishing player's late tick
|
||||
# (`checkpoint`): the tick is every 10 s, plus the request's own latency and a retry.
|
||||
_LATE_CHECKPOINT_S = 30
|
||||
|
||||
|
||||
def mark_finished(db: Session, user_id: int, item_id: int, now: datetime) -> bool:
|
||||
"""Mark the item watched (position cleared) and say whether THIS call made it so. Two near-end
|
||||
@@ -352,16 +356,33 @@ def checkpoint(
|
||||
now: datetime,
|
||||
*,
|
||||
dirty: bool,
|
||||
) -> PlexState:
|
||||
"""Record a resume position. `dirty` marks it not yet mirrored to Plex — set before a push, so
|
||||
the reconcile re-pushes it if the push fails and the push can settle it once Plex confirms."""
|
||||
) -> bool:
|
||||
"""Record a resume position, and say whether it was recorded. `dirty` marks it not yet mirrored
|
||||
to Plex — set before a push, so the reconcile re-pushes it if the push fails and the push can
|
||||
settle it once Plex confirms.
|
||||
|
||||
Not onto a row marked watched in the last `_LATE_CHECKPOINT_S`: that position is the player's
|
||||
tick sent just before its own finish (or a mark from another tab), and writing it made a watched
|
||||
item in progress again — on Plex too, through the resume push it carried. Decided by the write
|
||||
itself, since the mark can commit between any read here and the write. A rewatch is checkpointed
|
||||
as before once the mark is that far behind it."""
|
||||
st = state_row(db, user, item_id, st)
|
||||
st.position_seconds = position
|
||||
st.progress_updated_at = now
|
||||
values: dict[str, object] = {"position_seconds": position, "progress_updated_at": now}
|
||||
if dirty:
|
||||
st.synced_to_plex = False
|
||||
values["synced_to_plex"] = False
|
||||
not_just_finished = or_(
|
||||
PlexState.status != "watched",
|
||||
PlexState.watched_at.is_(None),
|
||||
PlexState.watched_at <= now - timedelta(seconds=_LATE_CHECKPOINT_S),
|
||||
)
|
||||
result = db.execute(
|
||||
update(PlexState)
|
||||
.where(PlexState.id == st.id, not_just_finished)
|
||||
.values(values)
|
||||
.execution_options(synchronize_session="fetch")
|
||||
)
|
||||
db.flush()
|
||||
return st
|
||||
return isinstance(result, CursorResult) and result.rowcount > 0
|
||||
|
||||
|
||||
def forget_state(db: Session, st: PlexState) -> None:
|
||||
|
||||
+265
-129
@@ -4,9 +4,10 @@ Playback is from the LOCAL physical file. Browser-compatible files (playable="di
|
||||
raw with HTTP range requests. Everything else that is h264 video (playable="remux") is remuxed on
|
||||
the fly to HLS with **video stream-copy** (cheap, I/O-bound — no video re-encode) and audio to AAC
|
||||
when needed. Non-h264 video (playable="transcode": HEVC/VP9/AV1/MPEG-2/XviD…) needs a full re-encode
|
||||
to H.264: on the prod host that runs on the Intel iGPU via VAAPI (`h264_vaapi`, near-zero CPU), with
|
||||
software libx264 as the automatic fallback where the render node is absent or the source codec isn't
|
||||
hardware-decodable. Same HLS session/seek machinery either way — only the video encoder differs.
|
||||
to H.264: on the prod host that is encoded on the Intel iGPU via VAAPI (`h264_vaapi`) — decoded there
|
||||
too when it can, else decoded on the CPU and uploaded (VC-1, XviD, DivX) — with software libx264 as
|
||||
the fallback where the render node is absent, for a source neither GPU route takes (AV1, 10-bit SDR),
|
||||
and when a GPU session produces nothing. Same HLS session/seek machinery either way.
|
||||
|
||||
Seek model (Jellyfin-style): one ffmpeg session per item, started at a given offset via `-ss`
|
||||
(fast keyframe seek). A seek beyond the generated region restarts the session at the new offset,
|
||||
@@ -35,7 +36,7 @@ from app.config import settings
|
||||
from app.downloads.edit import encode_thread_cap
|
||||
from app.plex.models import PlexItem
|
||||
from app.plex import paths
|
||||
from app.plex.sync import HDR_TRANSFERS, HW_DECODABLE_CODECS
|
||||
from app.plex.sync import CPU_DECODE_CODECS, HDR_TRANSFERS, HW_DECODABLE_CODECS
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
_HLS_ROOT = (
|
||||
@@ -66,7 +67,9 @@ _SESSION_IDLE_S = 600 # reap a session with no access for this long
|
||||
# restart leaves them behind like any other. `sweep_orphans` deletes ONLY names
|
||||
# matching one of those two shapes: `_HLS_ROOT` is admin-configurable (PLEX_HLS_DIR), so it may
|
||||
# legitimately point at a shared scratch path we do not own, and a blanket "delete every directory
|
||||
# under the root" would take the neighbours with it.
|
||||
# under the root" would take the neighbours with it. A torn-down session's directory is renamed to
|
||||
# its own name + `_TOMB` + 8 hex digits before it is deleted (see `_detach`), so a crash between the
|
||||
# two leaves a name this sweep still recognises.
|
||||
#
|
||||
# The KEY is anchored to digits (a Plex rating_key is a number) — that is what keeps foreign names
|
||||
# out. A `.+` there let `snapshot_2024_backup_+1.00` through, and pairing `.+` with an optional tail
|
||||
@@ -74,8 +77,10 @@ _SESSION_IDLE_S = 600 # reap a session with no access for this long
|
||||
# non-numeric rating_key would merely leave its directory unswept, which is the safe direction to
|
||||
# fail. Covered by test_plex_stream.py — this regex is the only thing standing between the sweep
|
||||
# and someone else's data.
|
||||
_TOMB = ".dead-"
|
||||
_SESSION_DIR_RE = re.compile(
|
||||
r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{2}$"
|
||||
r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$"
|
||||
rf"|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{{2}}({re.escape(_TOMB)}[0-9a-f]{{8}})?$"
|
||||
)
|
||||
# When this process started. `sweep_orphans` only deletes directories older than this: everything
|
||||
# newer belongs to a session THIS run created, and since `start_session` reuses a directory path
|
||||
@@ -93,7 +98,7 @@ _MTIME_GRACE_S = 2.0
|
||||
|
||||
_lock = threading.Lock()
|
||||
# Keyed by VIEWER and item, (user_id, rating_key). Keyed by the item alone, a second account opening
|
||||
# the same film `_kill`ed the first one's session: the displaced viewer got a 404, reclaimed at its
|
||||
# the same film tore down the first one's session: the displaced viewer got a 404, reclaimed at its
|
||||
# playhead, and so evicted the other in turn — two people could not watch one film at once. (One
|
||||
# account on two devices still shares a session; that case ends in the player's bounded reclaim.)
|
||||
_sessions: dict[tuple[int, str], "HlsSession"] = {}
|
||||
@@ -116,12 +121,13 @@ class HlsSession:
|
||||
hw: bool = False,
|
||||
tonemap: bool = False,
|
||||
user_id: int = 0,
|
||||
cpu_decode: bool = False,
|
||||
):
|
||||
# Unguessable per-session id, carried in the URLs the client is handed. A seek-restart
|
||||
# replaces the viewer's registered session for the item — and the old player's next request
|
||||
# for `seg_12.ts`, resolved by (viewer, rating_key) alone, was answered 200 with the NEW session's twelfth segment, produced from a different offset
|
||||
# under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it.
|
||||
# A request naming a session that is no longer registered is a 404,
|
||||
# A request naming a session that is no longer registered is a 410,
|
||||
# which the player recovers from by restarting at the playhead.
|
||||
self.id = secrets.token_urlsafe(9)
|
||||
self.key = key # the item's rating_key (what the logs name)
|
||||
@@ -131,6 +137,7 @@ class HlsSession:
|
||||
self.transcode = transcode # video re-encoded (non-h264 source), not stream-copied
|
||||
self.hw = hw # transcode ran on the VAAPI hardware encoder (else libx264 software)
|
||||
self.tonemap = tonemap # HW HDR/10-bit path: OpenCL tonemap between VAAPI decode and encode
|
||||
self.cpu_decode = cpu_decode # HW encode of frames the CPU decoded (a codec the GPU can't)
|
||||
self.start_s = start_s # the REQUESTED seek offset
|
||||
# The REAL media start: with `-ss X -c:v copy` ffmpeg can only cut at the nearest earlier
|
||||
# keyframe K<=X, and hls.js zero-bases the timeline to K (so video.currentTime=0 is K, not X).
|
||||
@@ -184,8 +191,11 @@ _HLS_TAIL = [
|
||||
# * VAAPI hardware (`h264_vaapi`) — the prod default once the iGPU is passed in. A full GPU pipeline
|
||||
# (decode + scale + encode all on-device) offloads the CPU almost entirely, so several concurrent
|
||||
# 1080p transcodes are fine and the API stays responsive.
|
||||
# * CPU decode → VAAPI encode (`format=nv12,hwupload` + `h264_vaapi`) — a codec the GPU cannot decode
|
||||
# but whose 8-bit picture it encodes (`CPU_DECODE_CODECS`: VC-1, XviD, DivX). The encode is the
|
||||
# expensive half, so this still takes most of the load off the CPU.
|
||||
# * libx264 software — the automatic fallback where `/dev/dri` is absent (dev box, or prod before the
|
||||
# iGPU enablement) or the source codec isn't hardware-decodable (AV1/XviD on Gen 9.5). CPU-bound, so
|
||||
# iGPU enablement) or the source fits neither GPU route (AV1, a 10-bit SDR source). CPU-bound, so
|
||||
# it reserves a core via `-threads` (see encode_thread_cap).
|
||||
# Everything else — the `-copyts`/`_HLS_TAIL`/segment/seek-restart model and the AAC audio path — is
|
||||
# unchanged, so hls.js, subtitles, multi-audio and seeking keep working exactly as for remux.
|
||||
@@ -194,10 +204,10 @@ _VAAPI_DEVICE = "/dev/dri/renderD128"
|
||||
# here; its presence is the precondition for the HW HDR→SDR tonemap graph (see _opencl_present).
|
||||
_OPENCL_ICD_DIR = Path("/etc/OpenCL/vendors")
|
||||
# Codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for the
|
||||
# full-GPU VAAPI pipeline. A source outside it (AV1, MPEG-4 ASP/XviD…) falls back to software even with
|
||||
# the render node present, because the GPU can't decode it to feed its own encoder. THE single source is
|
||||
# plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't disagree by
|
||||
# codec; aliased to the old name to keep the call sites below unchanged.
|
||||
# full-GPU VAAPI pipeline. A source outside it is decoded on the CPU when its codec is in
|
||||
# `CPU_DECODE_CODECS` (the GPU still encodes), else transcoded in software. THE single source of both
|
||||
# sets is plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't
|
||||
# disagree by codec; aliased to the old name to keep the call sites below unchanged.
|
||||
_HW_DECODABLE = HW_DECODABLE_CODECS
|
||||
# Color-transfer values that mark a true-HDR source (PQ / HLG). A HW-decodable HDR source can't take the
|
||||
# plain scale_vaapi=format=nv12 path (Gen 9.5 VPP refuses it) — it needs the OpenCL tonemap graph
|
||||
@@ -298,16 +308,22 @@ def _opencl_present() -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _hwaccel_in_opts(hw: bool, tonemap: bool = False) -> list[str]:
|
||||
def _hwaccel_in_opts(hw: bool, tonemap: bool = False, cpu_decode: bool = False) -> list[str]:
|
||||
"""VAAPI decode options, placed before `-i` so the source is decoded (and kept) on the GPU.
|
||||
`-hwaccel_output_format vaapi` keeps frames as GPU surfaces all the way to `h264_vaapi` — no
|
||||
round-trip to system memory. Empty for the software path.
|
||||
|
||||
The tonemap variant additionally derives an OpenCL device FROM the VAAPI one (`opencl=ocl@va`) so
|
||||
tonemap_opencl can map the decoded surfaces zero-copy (see _venc_opts); it needs named devices, so it
|
||||
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`."""
|
||||
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`.
|
||||
|
||||
The CPU-decode variant opens the VAAPI device for the FILTER graph only (`hwupload` needs one) and
|
||||
asks for no `-hwaccel`: the codec is one the GPU refuses to decode. This exact form is the one
|
||||
measured on prod with the HLS muxer and `-copyts` (and to the end of the file)."""
|
||||
if not hw:
|
||||
return []
|
||||
if cpu_decode:
|
||||
return ["-init_hw_device", f"vaapi=va:{_VAAPI_DEVICE}", "-filter_hw_device", "va"]
|
||||
if tonemap:
|
||||
return [
|
||||
"-init_hw_device",
|
||||
@@ -355,6 +371,7 @@ def _venc_opts(
|
||||
preset: str = "veryfast",
|
||||
crf: int = 20,
|
||||
max_height: int = 0,
|
||||
cpu_decode: bool = False,
|
||||
) -> list[str]:
|
||||
"""The video-encoder args that replace `-c:v copy` for a transcode session (returns the copy args
|
||||
when not transcoding). A re-encode must MINT its own keyframes at the segment boundaries — the copy
|
||||
@@ -381,6 +398,10 @@ def _venc_opts(
|
||||
"h264_vaapi",
|
||||
*force_kf,
|
||||
]
|
||||
if hw and cpu_decode:
|
||||
# The CPU decoded the frames: convert them to the encoder's 8-bit nv12 and upload them to the
|
||||
# device `-filter_hw_device` named (see _hwaccel_in_opts).
|
||||
return ["-vf", "format=nv12,hwupload", "-c:v", "h264_vaapi", *force_kf]
|
||||
if hw:
|
||||
# scale_vaapi with format=nv12 downconverts a 10-bit (p010) HEVC source to 8-bit nv12 so
|
||||
# h264_vaapi (8-bit High) can encode it; for an 8-bit source it's a cheap format pass-through.
|
||||
@@ -418,13 +439,18 @@ def _render_node_present() -> bool:
|
||||
return Path(_VAAPI_DEVICE).exists()
|
||||
|
||||
|
||||
def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> tuple[bool, bool]:
|
||||
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap)`:
|
||||
* `(False, False)` → software libx264 (the fallback: no render node, non-decodable source, or a
|
||||
10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
|
||||
* `(True, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR source.
|
||||
* `(True, True)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG) source, when
|
||||
the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
|
||||
def _select_encode(
|
||||
db: DbSession, item: PlexItem, transcode: bool, src: Path
|
||||
) -> tuple[bool, bool, bool]:
|
||||
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap, cpu_decode)`:
|
||||
* `(False, False, False)` → software libx264 (the fallback: no render node, a source neither GPU
|
||||
route takes, or a 10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
|
||||
* `(True, False, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR
|
||||
source the GPU decodes.
|
||||
* `(True, True, False)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG)
|
||||
source, when the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
|
||||
* `(True, False, True)` → CPU decode → `hwupload` → VAAPI encode — an 8-bit SDR source in a codec
|
||||
the GPU cannot decode (`CPU_DECODE_CODECS`).
|
||||
|
||||
A 10-bit *SDR* source is deliberately NOT a `(True, True)` case: it needs a plain 10→8-bit downconvert,
|
||||
not the HDR tone curve (which would visibly darken/flatten an SDR picture), so it drops to software.
|
||||
@@ -434,31 +460,40 @@ def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) ->
|
||||
backstops a session that produces nothing — but still routing a true-HDR source through the tonemap
|
||||
graph so it doesn't hit the scale_vaapi wall); "auto" (default) → HW only when the render node is
|
||||
present AND the iGPU can hardware-decode the source, with tonemap chosen only for a true-HDR (PQ/HLG)
|
||||
source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
|
||||
source that has an OpenCL runtime, while a 10-bit SDR source drops to software. In both "on" and
|
||||
"auto" a codec the GPU is known not to decode goes the CPU-decode route: asking the GPU to decode it
|
||||
only bought the fallback's wasted second before libx264."""
|
||||
sw = (False, False, False)
|
||||
if not transcode:
|
||||
return (False, False)
|
||||
return sw
|
||||
mode = sysconfig.get_str(db, "plex_transcode_hw")
|
||||
if mode == "off" or not _render_node_present():
|
||||
return (False, False)
|
||||
return sw
|
||||
codec = (item.codec_video or "").lower()
|
||||
if mode == "on":
|
||||
if codec in CPU_DECODE_CODECS:
|
||||
return (True, False, True)
|
||||
# Trust the admin: HW wherever the render node exists, skipping the decodability gate (the runtime
|
||||
# backstop catches a miss). Only a true-HDR (PQ/HLG) source needs the tonemap graph — a 10-bit SDR
|
||||
# source takes the plain scale path here, NOT the HDR tone curve. Probe only when OpenCL exists
|
||||
# (short-circuit), so "on" stays probe-free where tonemap can't apply anyway.
|
||||
tonemap = _opencl_present() and _probe_video_kind(src) == _VK_HDR
|
||||
return (True, tonemap)
|
||||
return (True, tonemap, False)
|
||||
# auto
|
||||
if (item.codec_video or "").lower() not in _HW_DECODABLE:
|
||||
return (False, False)
|
||||
if codec in CPU_DECODE_CODECS:
|
||||
# The upload converts to 8-bit nv12: only an 8-bit SDR picture comes through it unchanged.
|
||||
return (True, False, True) if _probe_video_kind(src) == _VK_HWSAFE else sw
|
||||
if codec not in _HW_DECODABLE:
|
||||
return sw
|
||||
kind = _probe_video_kind(src)
|
||||
if kind == _VK_HWSAFE:
|
||||
return (True, False) # 8-bit SDR → full-GPU scale path
|
||||
return (True, False, False) # 8-bit SDR → full-GPU scale path
|
||||
if kind == _VK_HDR:
|
||||
# PQ/HLG: HW OpenCL tonemap when the runtime is present, else software libx264.
|
||||
return (True, True) if _opencl_present() else (False, False)
|
||||
return (True, True, False) if _opencl_present() else sw
|
||||
# _VK_SDR10 — 10-bit SDR: a plain 10→8-bit downconvert, done exactly on the CPU. Never the HDR tone
|
||||
# curve (it would alter an SDR picture), and off the unverified scale_vaapi 10-bit path — software.
|
||||
return (False, False)
|
||||
return sw
|
||||
|
||||
|
||||
def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
|
||||
@@ -467,16 +502,17 @@ def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
|
||||
return _select_encode(db, item, transcode, src)[0]
|
||||
|
||||
|
||||
# CPU-nice increment for the SOFTWARE transcode child. The Plex ffmpeg is a Popen in the long-lived API
|
||||
# process (which must NOT itself be niced), so — unlike the download worker's _deprioritize (run once in
|
||||
# its own main()) — we lower priority per-child via preexec_fn in the fork. The GPU path needs none of
|
||||
# this. Reuses the download child's nice default so all co-located re-encoders share one handicap.
|
||||
# CPU-nice increment for a transcode child that works the CPU: a software encode, or a CPU decode
|
||||
# feeding the GPU encoder. The Plex ffmpeg is a Popen in the long-lived API process (which must NOT
|
||||
# itself be niced), so — unlike the download worker's _deprioritize (run once in its own main()) — we
|
||||
# lower priority per-child via preexec_fn in the fork. The full-GPU path needs none of this. Reuses
|
||||
# the download child's nice default so all co-located re-encoders share one handicap.
|
||||
_SW_TRANSCODE_NICE = settings.download_child_nice
|
||||
|
||||
|
||||
def _sw_child_preexec() -> None:
|
||||
"""Runs in the forked child (POSIX) before exec: renice this software-transcode ffmpeg down so it
|
||||
can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
|
||||
"""Runs in the forked child (POSIX) before exec: renice this CPU-bound transcode ffmpeg down so
|
||||
it can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
|
||||
playback."""
|
||||
try:
|
||||
os.nice(_SW_TRANSCODE_NICE)
|
||||
@@ -536,13 +572,15 @@ def _ffmpeg_cmd(
|
||||
crf: int = 20,
|
||||
max_height: int = 0,
|
||||
loudnorm: bool = False,
|
||||
cpu_decode: bool = False,
|
||||
) -> list[str]:
|
||||
# For a transcode session the video is re-encoded (VAAPI or libx264) instead of stream-copied; the
|
||||
# hwaccel decode options go before -i, the encoder args replace `-c:v copy` below. Everything else is
|
||||
# identical to remux. `tonemap` is the HDR/10-bit HW variant (OpenCL tonemap between VAAPI decode and
|
||||
# encode) — it needs matching decode-side (named devices) and encode-side (the OpenCL filter graph) opts.
|
||||
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height)
|
||||
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap)]
|
||||
# `cpu_decode` is the HW encode of CPU-decoded frames: a device for the filter graph, no -hwaccel.
|
||||
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height, cpu_decode)
|
||||
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap, cpu_decode)]
|
||||
# Keep ORIGINAL timestamps on the output so the first segment carries the true absolute PTS of the
|
||||
# keyframe ffmpeg actually seeked to. hls.js still zero-bases playback to that PTS; we read it back
|
||||
# (see _probe_first_pts) to learn the real content offset. Without -copyts the segment PTS is
|
||||
@@ -641,29 +679,82 @@ def _probe_first_pts(seg: Path) -> float | None:
|
||||
return None
|
||||
|
||||
|
||||
def _kill(s: HlsSession) -> None:
|
||||
# Teardown is split in two, because every segment request and checkpoint takes `_lock`: waiting up to
|
||||
# 3 s for an ffmpeg to exit and deleting a whole segmented film under it stalled every other
|
||||
# viewer's playback for one viewer's seek or one reap pass. `_detach` (under the lock) signals the
|
||||
# process and renames its directory aside — so the deletion can never hit a NEW session that reuses
|
||||
# the path — and `_finish` (after the lock is released) waits and deletes.
|
||||
#
|
||||
# What the rename cannot cover is the dying ffmpeg itself: it writes by path, so until it has exited
|
||||
# it can still put a segment, or its final ENDLIST playlist, into a new session's directory on the
|
||||
# same path. `_dying` remembers it by that path, and `_spawn` waits it out — under the lock, on
|
||||
# purpose — before it creates the directory again.
|
||||
_dying: dict[Path, subprocess.Popen] = {}
|
||||
|
||||
|
||||
def _await_exit(key: str, proc: subprocess.Popen) -> None:
|
||||
try:
|
||||
s.proc.terminate()
|
||||
try:
|
||||
s.proc.wait(timeout=3)
|
||||
proc.wait(timeout=3)
|
||||
# silent-ok: the escalation IS the handling. A transcode that ignores TERM for three
|
||||
# seconds gets KILL, which is exactly what this branch does.
|
||||
except subprocess.TimeoutExpired:
|
||||
s.proc.kill()
|
||||
proc.kill()
|
||||
except Exception as exc: # noqa: BLE001 — teardown finishes whatever the process does
|
||||
# Not nothing: what survives this is an ffmpeg nobody owns any more, holding a VAAPI
|
||||
# handle and a core, and the transcode cap that is supposed to bound them counts rows in
|
||||
# `_sessions` rather than processes. One line is the only way that ever surfaces.
|
||||
log.info(
|
||||
"could not stop the transcode process",
|
||||
extra=logs.event("plex.stream.kill_failed", key=key, reason=type(exc).__name__),
|
||||
)
|
||||
|
||||
|
||||
def _detach(s: HlsSession) -> tuple[HlsSession, Path | None]:
|
||||
"""Caller holds _lock and has taken `s` out of `_sessions`. Hand the result to `_finish` once the
|
||||
lock is released."""
|
||||
try:
|
||||
s.proc.terminate()
|
||||
except Exception as exc: # noqa: BLE001 — teardown finishes whatever the process does
|
||||
log.info(
|
||||
"could not stop the transcode process",
|
||||
extra=logs.event("plex.stream.kill_failed", key=s.key, reason=type(exc).__name__),
|
||||
)
|
||||
shutil.rmtree(s.dir, ignore_errors=True)
|
||||
_dying[s.dir] = s.proc
|
||||
tomb = s.dir.with_name(f"{s.dir.name}{_TOMB}{secrets.token_hex(4)}")
|
||||
try:
|
||||
s.dir.rename(tomb)
|
||||
# silent-ok: nothing on disk (ffmpeg died before writing, or a sweep took it) — nothing to delete.
|
||||
except FileNotFoundError:
|
||||
return s, None
|
||||
except OSError as exc:
|
||||
# Cannot be moved aside, so it cannot be deleted outside the lock either: the path may be a
|
||||
# new session's a moment from now. Deleted here instead, the old way.
|
||||
log.info(
|
||||
"a session directory could not be moved aside; deleted under the lock",
|
||||
extra=logs.event("plex.stream.tomb_failed", key=s.key, reason=type(exc).__name__),
|
||||
)
|
||||
shutil.rmtree(s.dir, ignore_errors=True)
|
||||
return s, None
|
||||
return s, tomb
|
||||
|
||||
|
||||
def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
|
||||
"""Caller holds _lock. Three bounds, applied after a session is registered, each dropping the
|
||||
least recently accessed sessions over it:
|
||||
def _finish(doomed: "list[tuple[HlsSession, Path | None]]") -> None:
|
||||
"""The slow half of a teardown, called WITHOUT `_lock`."""
|
||||
for s, tomb in doomed:
|
||||
_await_exit(s.key, s.proc)
|
||||
with _lock:
|
||||
if _dying.get(s.dir) is s.proc:
|
||||
del _dying[s.dir]
|
||||
if tomb is not None:
|
||||
shutil.rmtree(tomb, ignore_errors=True)
|
||||
|
||||
|
||||
def _enforce_caps(
|
||||
user_id: int, max_transcodes: int, max_sessions: int
|
||||
) -> "list[tuple[HlsSession, Path | None]]":
|
||||
"""Caller holds _lock, and `_finish`es what this returns once it has released it. Three bounds,
|
||||
applied after a session is registered, each dropping the least recently accessed sessions over it:
|
||||
|
||||
* this account's own sessions (`_MAX_SESSIONS_PER_USER`) — first, so a single account filling the
|
||||
box evicts itself before anyone else;
|
||||
@@ -672,10 +763,12 @@ def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
|
||||
whole segmented copy of the film under `_HLS_ROOT`, written faster than realtime. Counted only
|
||||
by the transcode cap, N distinct items were N ffmpegs and N film-sized scratch directories.
|
||||
|
||||
Somebody's playback stops at each drop. The route turns the missing session into a retryable 404
|
||||
Somebody's playback stops at each drop. The route turns the missing session into a 410
|
||||
and the player restarts at the playhead, so from the server nothing failed — and "my film keeps
|
||||
rebuffering" has no other explanation anywhere, hence the line."""
|
||||
|
||||
doomed: list[tuple[HlsSession, Path | None]] = []
|
||||
|
||||
def cull(pool: list[tuple[tuple[int, str], HlsSession]], cap: int, bound: str) -> None:
|
||||
if len(pool) <= cap:
|
||||
return
|
||||
@@ -690,12 +783,13 @@ def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
|
||||
sessions=len(pool),
|
||||
),
|
||||
)
|
||||
_kill(sess)
|
||||
_sessions.pop(reg, None)
|
||||
doomed.append(_detach(sess))
|
||||
|
||||
cull([kv for kv in _sessions.items() if kv[0][0] == user_id], _MAX_SESSIONS_PER_USER, "user")
|
||||
cull([kv for kv in _sessions.items() if kv[1].transcode], max_transcodes, "transcode")
|
||||
cull(list(_sessions.items()), max_sessions, "total")
|
||||
return doomed
|
||||
|
||||
|
||||
def start_session(
|
||||
@@ -740,7 +834,11 @@ def start_session(
|
||||
return directory / segment_name(i, _VIDEO_VARIANT if variants else None)
|
||||
|
||||
def _spawn(
|
||||
hw: bool, tonemap: bool = False, *, replace_only: "HlsSession | None" = None
|
||||
hw: bool,
|
||||
tonemap: bool = False,
|
||||
cpu_decode: bool = False,
|
||||
*,
|
||||
replace_only: "HlsSession | None" = None,
|
||||
) -> "HlsSession | None":
|
||||
# `replace_only` makes the supersede identity-aware for the runtime fallback: the whole
|
||||
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
|
||||
@@ -755,72 +853,88 @@ def start_session(
|
||||
preset = sysconfig.get_str(db, "plex_transcode_preset")
|
||||
crf = sysconfig.get_int(db, "plex_transcode_crf")
|
||||
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
|
||||
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
|
||||
# can't starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a
|
||||
# platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
|
||||
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
|
||||
with _lock:
|
||||
if replace_only is not None and _sessions.get(reg) is not replace_only:
|
||||
return None # superseded/culled while we waited out of lock — don't clobber the newer one
|
||||
old = _sessions.pop(reg, None)
|
||||
if old is not None:
|
||||
_kill(old)
|
||||
shutil.rmtree(directory, ignore_errors=True)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
proc = subprocess.Popen(
|
||||
_ffmpeg_cmd(
|
||||
src,
|
||||
start_s,
|
||||
# Nice the child that works the CPU — a SOFTWARE transcode, or a CPU decode feeding the GPU
|
||||
# encoder (not the full-GPU path, and not remux) — so it can't starve the co-located
|
||||
# API/playback. preexec_fn runs in the forked child (POSIX); on a platform without fork (the API
|
||||
# always runs in a Linux container in prod) it's simply omitted.
|
||||
cpu_bound = transcode and (not hw or cpu_decode)
|
||||
preexec = _sw_child_preexec if (cpu_bound and os.name == "posix") else None
|
||||
doomed: list[tuple[HlsSession, Path | None]] = []
|
||||
# Finished even when the new session fails to start (no space, no ffmpeg, a DB error): what
|
||||
# was moved aside under the lock is otherwise left for a restart's sweep.
|
||||
try:
|
||||
with _lock:
|
||||
if replace_only is not None and _sessions.get(reg) is not replace_only:
|
||||
return None # superseded/culled while we waited out of lock — don't clobber the newer one
|
||||
old = _sessions.pop(reg, None)
|
||||
if old is not None:
|
||||
doomed.append(_detach(old))
|
||||
# An ffmpeg still exiting from THIS path (the old session, or one stopped a moment ago)
|
||||
# would write into the directory about to be made; see `_dying`.
|
||||
prev = _dying.pop(directory, None)
|
||||
if prev is not None:
|
||||
_await_exit(key, prev)
|
||||
shutil.rmtree(directory, ignore_errors=True)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
proc = subprocess.Popen(
|
||||
_ffmpeg_cmd(
|
||||
src,
|
||||
start_s,
|
||||
directory,
|
||||
audio_ord,
|
||||
aoff,
|
||||
audio_langs if is_multi else None,
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
tonemap=tonemap,
|
||||
preset=preset,
|
||||
crf=crf,
|
||||
max_height=max_height,
|
||||
loudnorm=loudnorm,
|
||||
cpu_decode=cpu_decode,
|
||||
),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
preexec_fn=preexec,
|
||||
)
|
||||
s = HlsSession(
|
||||
key,
|
||||
directory,
|
||||
audio_ord,
|
||||
aoff,
|
||||
audio_langs if is_multi else None,
|
||||
proc,
|
||||
start_s,
|
||||
variants,
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
tonemap=tonemap,
|
||||
preset=preset,
|
||||
crf=crf,
|
||||
max_height=max_height,
|
||||
loudnorm=loudnorm,
|
||||
),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
preexec_fn=preexec,
|
||||
)
|
||||
s = HlsSession(
|
||||
key,
|
||||
directory,
|
||||
proc,
|
||||
start_s,
|
||||
variants,
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
tonemap=tonemap,
|
||||
user_id=user_id,
|
||||
)
|
||||
_sessions[reg] = s
|
||||
# The one line that says which pipeline a session actually got. Everything downstream
|
||||
# — the cap, the HW→SW fallback, the subtitle offset — is about this choice, and until
|
||||
# now it was only inferable from the ffmpeg command nobody keeps.
|
||||
log.info(
|
||||
"an HLS session started",
|
||||
extra=logs.event(
|
||||
"plex.stream.session_started",
|
||||
key=key,
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
tonemap=tonemap,
|
||||
variants=len(variants or ()),
|
||||
start_s=round(start_s, 1),
|
||||
),
|
||||
)
|
||||
# Admin-configurable caps (Configuration → Plex); at least 1 each so playback can't be
|
||||
# capped to zero.
|
||||
_enforce_caps(
|
||||
user_id,
|
||||
max(1, sysconfig.get_int(db, "plex_max_transcodes")),
|
||||
max(1, sysconfig.get_int(db, "plex_max_sessions")),
|
||||
)
|
||||
user_id=user_id,
|
||||
cpu_decode=cpu_decode,
|
||||
)
|
||||
_sessions[reg] = s
|
||||
# The one line that says which pipeline a session actually got. Everything downstream
|
||||
# — the cap, the HW→SW fallback, the subtitle offset — is about this choice, and until
|
||||
# now it was only inferable from the ffmpeg command nobody keeps.
|
||||
log.info(
|
||||
"an HLS session started",
|
||||
extra=logs.event(
|
||||
"plex.stream.session_started",
|
||||
key=key,
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
tonemap=tonemap,
|
||||
cpu_decode=cpu_decode,
|
||||
variants=len(variants or ()),
|
||||
start_s=round(start_s, 1),
|
||||
),
|
||||
)
|
||||
# Admin-configurable caps (Configuration → Plex); at least 1 each so playback can't be
|
||||
# capped to zero.
|
||||
doomed += _enforce_caps(
|
||||
user_id,
|
||||
max(1, sysconfig.get_int(db, "plex_max_transcodes")),
|
||||
max(1, sysconfig.get_int(db, "plex_max_sessions")),
|
||||
)
|
||||
finally:
|
||||
_finish(doomed)
|
||||
return s
|
||||
|
||||
s = _spawn(*_select_encode(db, item, transcode, src))
|
||||
@@ -847,7 +961,7 @@ def start_session(
|
||||
# stale and stops the session it names (`stop_session`), which stopped the playback that won.
|
||||
# Never None either: the POST /session route turns a None return into a "Media file not found
|
||||
# on disk" 404 — dishonest for a file that exists. A dead session instead degrades to a
|
||||
# retryable 404 ("No active playback session" / "Not available yet") the client re-requests.
|
||||
# 410 ("session_gone") on its URLs, and the client asks again.
|
||||
return s
|
||||
s = respawned
|
||||
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
|
||||
@@ -866,7 +980,8 @@ def start_session(
|
||||
if not wait_for(_seg(i), timeout=_WARMUP_SEG_TIMEOUT_S, proc=s.proc):
|
||||
break
|
||||
log.info(
|
||||
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s tonemap=%s",
|
||||
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s"
|
||||
" tonemap=%s cpu_decode=%s",
|
||||
key,
|
||||
start_s,
|
||||
s.media_start_s,
|
||||
@@ -875,6 +990,7 @@ def start_session(
|
||||
transcode,
|
||||
s.hw,
|
||||
s.tonemap,
|
||||
s.cpu_decode,
|
||||
)
|
||||
return s
|
||||
|
||||
@@ -974,6 +1090,11 @@ def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
|
||||
# session, absent for a single-audio one.
|
||||
if variant not in (s.variants or (None,)):
|
||||
return False
|
||||
# And the name must be the one ffmpeg SPELLS for that sequence: `int()` also reads `seg_007.ts`
|
||||
# and a non-ASCII digit run as a sequence the session may have reached, and neither file is
|
||||
# ever written — a yes there parked the request for the whole wait.
|
||||
if filename != segment_name(seq, variant):
|
||||
return False
|
||||
if seq == 0:
|
||||
return True
|
||||
try:
|
||||
@@ -1084,9 +1205,13 @@ def sweep_orphans(older_than: float | None = None) -> int:
|
||||
return dropped
|
||||
|
||||
|
||||
# How long a session whose ffmpeg has FINISHED may go unfetched before a new session request reaps
|
||||
# it (see reap_idle).
|
||||
_FINISHED_IDLE_S = 30
|
||||
# The slowest heartbeat a live player sends: a browser cuts the timers of a hidden page that makes no
|
||||
# sound to one a minute (measured on Chromium: a paused and a muted player, about a minute after
|
||||
# hiding; a hidden player with sound keeps its 10 s).
|
||||
_THROTTLED_HEARTBEAT_S = 60
|
||||
# How long a session whose ffmpeg has FINISHED may go without a signal before `reap_idle` takes it:
|
||||
# the beat before the reaper looks, one lost on the way, and the time the next one takes to arrive.
|
||||
_FINISHED_IDLE_S = 3 * _THROTTLED_HEARTBEAT_S
|
||||
|
||||
|
||||
def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
|
||||
@@ -1095,29 +1220,39 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
|
||||
|
||||
A finished session is also what a PAUSED viewer has: a remux runs far faster than realtime, so
|
||||
ffmpeg is done a minute or two in and writes ENDLIST; hls.js then stops asking for the playlist,
|
||||
and with its forward buffer full a paused player fetches nothing at all. With the 30 s rule that
|
||||
killed a one-minute pause, and playback died ~30 s after resuming.
|
||||
and with its forward buffer full a paused player fetches nothing at all.
|
||||
|
||||
What makes the short rule safe again is that fetching is no longer the only signal: a player
|
||||
parked MID-ITEM checkpoints its position every 10 s, and that route calls `touch_session`, so a
|
||||
viewer who is merely paused says so far more often than this rule asks. The heartbeat
|
||||
What makes the short rule safe is that fetching is not the only signal: a player parked MID-ITEM
|
||||
checkpoints its position and that route calls `touch_session`. That beat is every 10 s only while
|
||||
the page is visible or making sound; a hidden, silent one sends it once a minute, which a 30 s
|
||||
rule killed whenever the reaper happened to run more than 30 s after the beat (measured: a paused
|
||||
session gone that way, and one that lived on the same cadence because its phase was luckier).
|
||||
So the rule is sized to the throttled beat (`_THROTTLED_HEARTBEAT_S`). The heartbeat
|
||||
stops once the item is finished or before a position is known (`usePlexProgress.flush`) — both
|
||||
states where reaping is correct, and where the player's 404 → reclaim recovers a viewer who
|
||||
scrubs back in anyway. Do not restate this as "every 10 s whether or not it is playing":
|
||||
states where reaping is correct, and where the player's 410 → reclaim recovers a viewer who
|
||||
scrubs back in anyway. Do not restate this as "whether or not it is playing":
|
||||
a review found that claim in two comments and not in the code. An earlier blanket
|
||||
`reap_abandoned` bought the same safety by keeping every finished session for ten minutes — each
|
||||
one a whole segmented copy of the film on a volume shared with the download center."""
|
||||
now = time.time()
|
||||
dropped = 0
|
||||
doomed: list[tuple[HlsSession, Path | None]] = []
|
||||
reaped: list[tuple[str, str, int]] = []
|
||||
with _lock:
|
||||
for reg, s in list(_sessions.items()):
|
||||
done = s.proc.poll() is not None
|
||||
idle = now - s.last_access
|
||||
if idle > _SESSION_IDLE_S or (done and idle > finished_idle_s):
|
||||
_kill(s)
|
||||
_sessions.pop(reg, None)
|
||||
dropped += 1
|
||||
return dropped
|
||||
doomed.append(_detach(s))
|
||||
reaped.append((s.key, "finished_idle" if done else "idle", int(idle)))
|
||||
# The viewer of a session wrongly taken sees only a 410 later, so this line is the one record of why.
|
||||
for key, reason, idle_s in reaped:
|
||||
log.info(
|
||||
"an idle session was reaped",
|
||||
extra=logs.event("plex.stream.session_reaped", key=key, reason=reason, idle_s=idle_s),
|
||||
)
|
||||
_finish(doomed)
|
||||
return len(doomed)
|
||||
|
||||
|
||||
def stop_session(user_id: int, key: str, session_id: str) -> bool:
|
||||
@@ -1132,8 +1267,9 @@ def stop_session(user_id: int, key: str, session_id: str) -> bool:
|
||||
s = _sessions.get((user_id, key))
|
||||
if s is None or s.id != session_id:
|
||||
return False
|
||||
_kill(s)
|
||||
_sessions.pop((user_id, key), None)
|
||||
doomed = [_detach(s)]
|
||||
_finish(doomed)
|
||||
return True
|
||||
|
||||
|
||||
|
||||
+34
-21
@@ -60,14 +60,20 @@ def classify_playable(vcodec: str | None, acodec: str | None, container: str | N
|
||||
|
||||
# Video codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for
|
||||
# the full-GPU VAAPI transcode pipeline. THE single source of this set: plex.stream imports it here so
|
||||
# the runtime encoder choice (`_want_hw`) and this catalog facet can never disagree by codec. A source
|
||||
# outside it (AV1, MPEG-4 ASP/XviD…) is software-only even on the iGPU (it can't decode it to feed its
|
||||
# own encoder).
|
||||
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"})
|
||||
# the runtime encoder choice (`_select_encode`) and this catalog facet can never disagree by codec. VC-1
|
||||
# is NOT in it: measured on prod, the iHD driver answers "No support for codec vc1 profile 3" (VC-1
|
||||
# Advanced, what a Blu-ray remux carries), so every session that asked the GPU to decode one died and
|
||||
# fell back to libx264.
|
||||
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "mpeg2video"})
|
||||
# Codecs the iGPU cannot decode but whose picture it can still ENCODE: the CPU decodes, the frames go up
|
||||
# to the GPU as 8-bit nv12, and h264_vaapi encodes (plex.stream's `cpu_decode` path). Measured on prod:
|
||||
# three segments in ~5 s for a 1080p VC-1 remux (libx264: ~17 s) and ~1 s for an SD XviD or DivX
|
||||
# episode. Only codecs that are always 8-bit SDR, and only ones measured; any other codec stays software.
|
||||
CPU_DECODE_CODECS = frozenset({"vc1", "mpeg4", "msmpeg4v3"})
|
||||
# Color-transfer values marking an HDR source (PQ / HLG). THE single source of this set: plex.stream
|
||||
# (the tonemap choice) and downloads.clip (a clip's tonemap) import it here, so a transfer added for
|
||||
# one is tonemapped by all. A 10-bit/HDR source breaks the VAAPI scale=nv12 path on Gen 9.5 (see
|
||||
# plex.stream) → software transcode, so it counts as NOT HW-capable.
|
||||
# one is tonemapped by all. An HDR source breaks the plain VAAPI scale=nv12 path on Gen 9.5, so it takes
|
||||
# the OpenCL tonemap graph (see plex.stream) — still the GPU encoder.
|
||||
HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
|
||||
|
||||
|
||||
@@ -78,18 +84,24 @@ def transcode_hw_capable(
|
||||
hdr: bool = False,
|
||||
) -> bool | None:
|
||||
"""The stored `PlexItem.transcode_hw` facet: for a `playable="transcode"` item, whether the prod
|
||||
iGPU can HARDWARE-transcode it (else software libx264 only); None for a non-transcode item. HW needs
|
||||
a HW-decodable codec AND an 8-bit SDR source — a 10-bit/HDR source (bit_depth>8 or an HDR transfer)
|
||||
breaks the VAAPI nv12 pipeline and falls back to software. `bit_depth`/`hdr` are unknown at sync
|
||||
time (the cheap listing carries no Stream), so they default to 8-bit SDR (codec-only) and the
|
||||
bit-depth backfill refines a 10-bit/HDR HEVC to False later."""
|
||||
iGPU ENCODES it ("Transcoded on the hardware encoder"), else software libx264; None for a
|
||||
non-transcode item. The same answer plex.stream's `_select_encode` gives on prod (render node and
|
||||
OpenCL runtime present, policy "auto"):
|
||||
* a HW-decodable codec: 8-bit SDR → the full-GPU path; HDR → the OpenCL tonemap, also on the GPU;
|
||||
10-bit SDR → software (an exact downconvert, never the HDR tone curve);
|
||||
* a CPU-decode codec (`CPU_DECODE_CODECS`) at 8 bits → CPU decode, GPU encode;
|
||||
* any other codec → software.
|
||||
`bit_depth`/`hdr` are unknown at sync time (the cheap listing carries no Stream), so they default to
|
||||
8-bit SDR and the bit-depth backfill refines a 10-bit SDR HEVC to False later."""
|
||||
if playable != "transcode":
|
||||
return None
|
||||
if (vcodec or "").lower() not in HW_DECODABLE_CODECS:
|
||||
return False
|
||||
if bit_depth is not None and bit_depth > 8:
|
||||
return False
|
||||
return not hdr
|
||||
codec = (vcodec or "").lower()
|
||||
deep = bit_depth is not None and bit_depth > 8
|
||||
if codec in HW_DECODABLE_CODECS:
|
||||
return hdr or not deep
|
||||
if codec in CPU_DECODE_CODECS:
|
||||
return not deep and not hdr
|
||||
return False
|
||||
|
||||
|
||||
def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
|
||||
@@ -266,8 +278,9 @@ def sync(db: Session) -> dict:
|
||||
# a Plex shape change, a DB error or our own bug, and `%s` told them apart badly.
|
||||
log.exception("the Plex people-enrich batch failed")
|
||||
# Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
|
||||
# playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
|
||||
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
|
||||
# playback-class filter's HW/SW split is precise (a 10-bit SDR HEVC → software) over
|
||||
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are classed
|
||||
# by codec alone.
|
||||
try:
|
||||
stats["media_probed"] = _probe_media_batch(db, plex)
|
||||
except Exception: # best-effort; never fail the sync over a backfill
|
||||
@@ -294,15 +307,15 @@ def sync(db: Session) -> dict:
|
||||
# whole library is covered over successive runs (each fetched item is stamped people_enriched_at once).
|
||||
_ENRICH_BATCH = 60
|
||||
# Same bounded-per-run pattern for the bit-depth probe. Only HW-decodable-codec transcode items are
|
||||
# probed (typically a small minority — the bulk of transcodes are old codecs that are SW by codec), so
|
||||
# probed (typically a small minority — the bulk of transcodes are old codecs classed by codec), so
|
||||
# a modest batch covers the work-list in a run or two; a stuck item retries next run (watermark unset).
|
||||
_PROBE_BATCH = 60
|
||||
|
||||
|
||||
def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH) -> int:
|
||||
"""Rich-fetch up to `limit` not-yet-probed HW-decodable transcode items, read the video Stream's
|
||||
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit/HDR HEVC →
|
||||
False/software). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
|
||||
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit SDR HEVC →
|
||||
False/software; an HDR one stays on the GPU through the tonemap). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
|
||||
so an item is probed once and not re-fetched every sync; a transient fetch FAILURE leaves the
|
||||
watermark unset to retry. Oldest-id first. Returns how many items were probed this run."""
|
||||
rows = (
|
||||
|
||||
@@ -41,11 +41,25 @@ def video(db: Session, video_id: str) -> Video | None:
|
||||
# --- tags ------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int]]:
|
||||
"""The tags this user sees (the shared system ones and their own), each with how many channel
|
||||
links it has, by category then name."""
|
||||
def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int, int]]:
|
||||
"""The tags this user sees (the shared system ones and their own), by category then name, each
|
||||
with how many channel links it has and how many of the user's own links sit on a channel they
|
||||
subscribe to and have not hidden — the rows the Channels page's tag filter shows (a hidden
|
||||
channel has its own tab). A link outlives an unsubscribe and a hide, so the two differ."""
|
||||
subscribed = select(Subscription.channel_id).where(
|
||||
owned_by(Subscription.user_id, user), Subscription.hidden.is_(False)
|
||||
)
|
||||
on_subscribed = dict(
|
||||
db.execute(
|
||||
select(ChannelTag.tag_id, func.count(ChannelTag.id))
|
||||
.where(owned_by(ChannelTag.user_id, user), ChannelTag.channel_id.in_(subscribed))
|
||||
.group_by(ChannelTag.tag_id)
|
||||
)
|
||||
.tuples()
|
||||
.all()
|
||||
)
|
||||
return [
|
||||
(tag, count)
|
||||
(tag, count, on_subscribed.get(tag.id, 0))
|
||||
for tag, count in db.execute(
|
||||
select(Tag, func.count(ChannelTag.id))
|
||||
.outerjoin(ChannelTag, ChannelTag.tag_id == Tag.id)
|
||||
@@ -132,12 +146,17 @@ def item_counts(db: Session, ids: list[int]) -> dict[int, int]:
|
||||
)
|
||||
|
||||
|
||||
def durations(db: Session, ids: list[int]) -> dict[int, int]:
|
||||
"""Each playlist's running time: the sum of its videos' lengths."""
|
||||
def durations(db: Session, ids: list[int]) -> dict[int, tuple[int, int]]:
|
||||
"""Each playlist's running time — the sum of its videos' known lengths — and how many videos
|
||||
it left out for having none (`Video.without_length`)."""
|
||||
return {
|
||||
pid: int(total or 0)
|
||||
for pid, total in db.execute(
|
||||
select(PlaylistItem.playlist_id, func.coalesce(func.sum(Video.duration_seconds), 0))
|
||||
pid: (int(total or 0), int(left_out or 0))
|
||||
for pid, total, left_out in db.execute(
|
||||
select(
|
||||
PlaylistItem.playlist_id,
|
||||
func.coalesce(func.sum(Video.length_seconds), 0),
|
||||
func.count().filter(Video.without_length),
|
||||
)
|
||||
.join(Video, Video.id == PlaylistItem.video_id)
|
||||
.where(PlaylistItem.playlist_id.in_(ids))
|
||||
.group_by(PlaylistItem.playlist_id)
|
||||
@@ -177,11 +196,16 @@ def covers(db: Session, ids: list[int]) -> dict[int, str | None]:
|
||||
)
|
||||
|
||||
|
||||
def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None]:
|
||||
"""One playlist's item count, running time and cover. An outer join, so an empty list still
|
||||
answers (0, 0, None)."""
|
||||
count, total = db.execute(
|
||||
select(func.count(PlaylistItem.id), func.coalesce(func.sum(Video.duration_seconds), 0))
|
||||
def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, int, str | None]:
|
||||
"""One playlist's item count, running time, the videos the running time left out for having no
|
||||
length (`Video.without_length`) and cover. An outer join, so an empty list still answers
|
||||
(0, 0, 0, None)."""
|
||||
count, total, left_out = db.execute(
|
||||
select(
|
||||
func.count(PlaylistItem.id),
|
||||
func.coalesce(func.sum(Video.length_seconds), 0),
|
||||
func.count().filter(Video.without_length),
|
||||
)
|
||||
.select_from(PlaylistItem)
|
||||
.outerjoin(Video, Video.id == PlaylistItem.video_id)
|
||||
.where(PlaylistItem.playlist_id == playlist_id)
|
||||
@@ -193,7 +217,7 @@ def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None
|
||||
.order_by(PlaylistItem.position)
|
||||
.limit(1)
|
||||
)
|
||||
return count or 0, total or 0, cover
|
||||
return count or 0, total or 0, left_out or 0, cover
|
||||
|
||||
|
||||
def playlist_rows(db: Session, user: User, playlist_id: int) -> list:
|
||||
|
||||
+159
-65
@@ -9,7 +9,7 @@ unique key, a row lock taken before a presence check. The comfort shelf's writes
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from typing import cast
|
||||
from typing import Literal, cast
|
||||
|
||||
from sqlalchemy import Table, bindparam, delete, exists, func, select, text, update
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
@@ -255,17 +255,6 @@ def mark(db: Session, user: User, video_id: str, status: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def clear_state(db: Session, user: User, video_id: str) -> None:
|
||||
"""The video back to pristine for this user: status, resume point and `watched_at` all go.
|
||||
Playlist membership lives elsewhere and stays."""
|
||||
lock_takeout(db, user) # as `unmark`
|
||||
row = queries.video_state(db, user, video_id)
|
||||
if row is not None:
|
||||
db.delete(row)
|
||||
_remember_unmark(db, user, video_id)
|
||||
db.flush()
|
||||
|
||||
|
||||
def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
|
||||
"""The player's resume point. 0 clears it: a row holding nothing else goes, one with a status
|
||||
(watched, hidden) keeps it with the position zeroed. The status itself is never changed here.
|
||||
@@ -301,12 +290,17 @@ def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
|
||||
# --- playlists -------------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Every reason a move is refused; the UI names each one (`playlists.moveErrors.<reason>`, held by the
|
||||
# frontend's backendIds contract, which reads this Literal).
|
||||
MoveReason = Literal["same_playlist"]
|
||||
|
||||
|
||||
class MoveRefused(Exception):
|
||||
"""A move between playlists that cannot happen: `reason` is the code the UI translates."""
|
||||
|
||||
def __init__(self, reason: str):
|
||||
def __init__(self, reason: MoveReason):
|
||||
super().__init__(reason)
|
||||
self.reason = reason
|
||||
self.reason: MoveReason = reason
|
||||
|
||||
|
||||
def touch(pl: Playlist) -> None:
|
||||
@@ -412,30 +406,6 @@ def remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) -
|
||||
return True
|
||||
|
||||
|
||||
def place_item(db: Session, pl: Playlist, video_id: str, position: int) -> None:
|
||||
"""Put a video already in the list at `position` (clamped to the end), renumbering the whole
|
||||
list: positions can have gaps after removals, so "index N" is only meaningful over the
|
||||
ordered rows."""
|
||||
rows = queries.items(db, pl.id)
|
||||
new = next(r for r in rows if r.video_id == video_id)
|
||||
rows.remove(new)
|
||||
rows.insert(min(position, len(rows)), new)
|
||||
for pos, row in enumerate(rows):
|
||||
row.position = pos
|
||||
db.flush()
|
||||
|
||||
|
||||
def add_to_playlist(db: Session, pl: Playlist, video_id: str, position: int | None) -> None:
|
||||
"""Add a video to one of the user's lists, at `position` when given (the list page's "Undo" of
|
||||
a removal puts the row back where it was in ONE request), else at the end. A video already in
|
||||
the list stays where it is."""
|
||||
added = add_item(db, pl, video_id, mark_dirty=False)
|
||||
if added and position is not None:
|
||||
place_item(db, pl, video_id, position)
|
||||
recompute_dirty(db, pl)
|
||||
db.flush()
|
||||
|
||||
|
||||
def rename_playlist(db: Session, pl: Playlist, name: str) -> None:
|
||||
"""A new name moves `dirty`; the caller holds the playlist's lock (the scheduled mirror refresh
|
||||
reads `dirty` under it)."""
|
||||
@@ -453,39 +423,163 @@ def delete_playlist(db: Session, pl: Playlist) -> None:
|
||||
db.flush()
|
||||
|
||||
|
||||
def move_item(db: Session, src: Playlist, dst: Playlist, video_id: str) -> None:
|
||||
"""Move one video from `src` into `dst`, in the caller's one transaction.
|
||||
def remove_items(db: Session, pl: Playlist, video_ids: list[str]) -> list[str]:
|
||||
"""Remove these videos from a playlist, in one transaction; a video not in it any more is simply
|
||||
not among them. Says which went: an Undo must not bring back one this removal never took."""
|
||||
lock_playlists(db, pl.id)
|
||||
wanted = set(video_ids)
|
||||
rows = [it for it in queries.items(db, pl.id) if it.video_id in wanted]
|
||||
for row in rows:
|
||||
db.delete(row)
|
||||
if rows:
|
||||
recompute_dirty(db, pl)
|
||||
touch(pl)
|
||||
db.flush()
|
||||
return [row.video_id for row in rows]
|
||||
|
||||
The client-side shape — add to the target, then remove from the source — leaves the video in
|
||||
BOTH lists whenever the second call fails. Idempotent: a video already in the target is simply
|
||||
dropped from the source (moving into a list that has it is a removal, not a duplicate). Raises
|
||||
`MoveRefused` for the same list twice, or a video a concurrent edit already took out."""
|
||||
|
||||
def _insert_at(
|
||||
db: Session,
|
||||
pl: Playlist,
|
||||
placed: list[tuple[int, str]],
|
||||
leaving: dict[str, PlaylistItem] | None = None,
|
||||
staying_out: frozenset[str] = frozenset(),
|
||||
) -> int:
|
||||
"""Add videos at their indices (ascending, so each index is where the row stood when the rows
|
||||
before it were back) and renumber the list 0..n-1, as positions can have gaps. A video already in
|
||||
the list stays where it is, and one the catalog has lost or in `staying_out` is left out, the
|
||||
ones after it closing up. A video with a row in `leaving` brings that row along, for the reason
|
||||
`move_items` re-parents. Says how many were added; the caller holds the playlist's lock."""
|
||||
leaving = leaving or {}
|
||||
rows = queries.items(db, pl.id)
|
||||
present = {r.video_id for r in rows}
|
||||
known = queries.known_video_ids(db, [vid for _, vid in placed])
|
||||
added = 0
|
||||
left_out = 0
|
||||
for at, vid in sorted(placed, key=lambda p: p[0]):
|
||||
if vid in present or vid not in known or vid in staying_out:
|
||||
left_out += 1
|
||||
continue
|
||||
row = leaving.pop(vid, None)
|
||||
if row is None:
|
||||
row = PlaylistItem(playlist_id=pl.id, video_id=vid, position=0)
|
||||
db.add(row)
|
||||
else:
|
||||
row.playlist_id = pl.id
|
||||
rows.insert(min(at - left_out, len(rows)), row)
|
||||
present.add(vid)
|
||||
added += 1
|
||||
for pos, row in enumerate(rows):
|
||||
row.position = pos
|
||||
return added
|
||||
|
||||
|
||||
def restore_items(db: Session, pl: Playlist, video_ids: list[str], positions: list[int]) -> int:
|
||||
"""Put removed videos back at their places, in ONE request: an append plus a whole-list reorder
|
||||
leaves them at the end whenever the second call fails. Says how many came back."""
|
||||
lock_playlists(db, pl.id)
|
||||
added = _insert_at(db, pl, list(zip(positions, video_ids, strict=True)))
|
||||
if added:
|
||||
recompute_dirty(db, pl)
|
||||
touch(pl)
|
||||
db.flush()
|
||||
return added
|
||||
|
||||
|
||||
class ListGone(Exception):
|
||||
"""A list of a two-list write was deleted between the route's read and the lock."""
|
||||
|
||||
def __init__(self, *, source: bool):
|
||||
super().__init__("source" if source else "target")
|
||||
self.source = source
|
||||
|
||||
|
||||
def _lock_both(db: Session, src: Playlist, dst: Playlist) -> None:
|
||||
"""Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise. A list
|
||||
already gone is said here: a write into it would answer with its foreign key's 500."""
|
||||
found = lock_playlists(db, src.id, dst.id)
|
||||
if src.id not in found:
|
||||
raise ListGone(source=True)
|
||||
if dst.id not in found:
|
||||
raise ListGone(source=False)
|
||||
|
||||
|
||||
def move_items(
|
||||
db: Session, src: Playlist, dst: Playlist, video_ids: list[str]
|
||||
) -> tuple[list[str], list[str]]:
|
||||
"""Move these videos from `src` into `dst`, in the caller's one transaction, in the source's
|
||||
order whatever order they were picked in. Returns the ones that landed in the target and the ones
|
||||
it already held, which only left the source: the Undo needs to tell them apart. A video a
|
||||
concurrent edit already took out of the source is in neither. Raises `MoveRefused` for the same
|
||||
list twice."""
|
||||
if src.id == dst.id:
|
||||
raise MoveRefused("same_playlist")
|
||||
# Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise.
|
||||
lock_playlists(db, src.id, dst.id)
|
||||
row = queries.item(db, src.id, video_id)
|
||||
if row is None:
|
||||
raise MoveRefused("not_in_playlist")
|
||||
dst_changed = False
|
||||
if queries.item(db, dst.id, video_id) is None:
|
||||
# Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
|
||||
# survives the move, and there is no window where the video belongs to neither list. The
|
||||
# check above holds: both lists are locked, and an insert into the target (another tab's
|
||||
# add, a YouTube merge) waits for the lock — its foreign key needs the target row.
|
||||
row.playlist_id = dst.id
|
||||
row.position = next_item_position(db, dst.id)
|
||||
dst_changed = True
|
||||
if not dst_changed:
|
||||
db.delete(row)
|
||||
# The source always changed; the target only when the row landed in it. Touching a target
|
||||
# that already had the video promoted an untouched list in the rail's "recently changed" sort.
|
||||
recompute_dirty(db, src)
|
||||
touch(src)
|
||||
if dst_changed:
|
||||
_lock_both(db, src, dst)
|
||||
wanted = set(video_ids)
|
||||
rows = [it for it in queries.items(db, src.id) if it.video_id in wanted]
|
||||
held = queries.in_playlist(db, dst.id, [r.video_id for r in rows])
|
||||
position = next_item_position(db, dst.id)
|
||||
moved: list[str] = []
|
||||
already_there: list[str] = []
|
||||
for row in rows:
|
||||
if row.video_id in held:
|
||||
already_there.append(row.video_id)
|
||||
db.delete(row)
|
||||
else:
|
||||
moved.append(row.video_id)
|
||||
# Re-parented rather than deleted and inserted: `added_at` ("when I saved this") survives
|
||||
# the move, and there is no window where the video belongs to neither list. `held` stays
|
||||
# true: both lists are locked, and an insert into the target (another tab's add, a
|
||||
# YouTube merge) waits for the lock, as its foreign key needs the target row.
|
||||
row.playlist_id = dst.id
|
||||
row.position = position
|
||||
position += 1
|
||||
if rows:
|
||||
recompute_dirty(db, src)
|
||||
touch(src)
|
||||
# Only when a row landed in it: touching a target that already had the video promoted an untouched
|
||||
# list in the rail's "recently changed" sort.
|
||||
if moved:
|
||||
recompute_dirty(db, dst)
|
||||
touch(dst)
|
||||
db.flush()
|
||||
return moved, already_there
|
||||
|
||||
|
||||
def undo_move(
|
||||
db: Session,
|
||||
src: Playlist,
|
||||
dst: Playlist,
|
||||
moved: list[str],
|
||||
already_there: list[str],
|
||||
positions: list[int],
|
||||
) -> int:
|
||||
"""Take a move back in one transaction: the `moved` videos leave `dst` and return to their places
|
||||
in `src`, keeping their rows, as the move did. The `already_there` ones return too and stay in
|
||||
`dst`, which held them before. A moved video `dst` no longer holds has gone on from there, and
|
||||
bringing it back would leave it in lists the reader never put it in, so it stays out, as a
|
||||
move skips a video a concurrent edit took out. `positions` pairs with `moved` + `already_there`.
|
||||
Says how many came back. Raises `MoveRefused` for the same list twice, as a move does."""
|
||||
if src.id == dst.id:
|
||||
raise MoveRefused("same_playlist")
|
||||
_lock_both(db, src, dst)
|
||||
wanted = set(moved)
|
||||
rows = {it.video_id: it for it in queries.items(db, dst.id) if it.video_id in wanted}
|
||||
changed = bool(rows)
|
||||
placed = list(zip(positions, [*moved, *already_there], strict=True))
|
||||
back = _insert_at(db, src, placed, leaving=rows, staying_out=frozenset(wanted - rows.keys()))
|
||||
# What `src` holds again by now only leaves `dst`.
|
||||
for row in rows.values():
|
||||
db.delete(row)
|
||||
db.flush()
|
||||
if changed:
|
||||
recompute_dirty(db, dst)
|
||||
touch(dst)
|
||||
if back:
|
||||
recompute_dirty(db, src)
|
||||
touch(src)
|
||||
db.flush()
|
||||
return back
|
||||
|
||||
|
||||
def reorder_items(db: Session, pl: Playlist, order: list[str]) -> int:
|
||||
|
||||
+20
-10
@@ -1,29 +1,39 @@
|
||||
"""A request the app refuses, said in a way the UI can put in the reader's language.
|
||||
|
||||
Every refusal the API answers carries a structured `detail` — `{code, reason, …params}` — and the client
|
||||
translates it (`lib/api/core.ts` `localizeDetail`). A bare English string reached the Hungarian UI
|
||||
translates it (`lib/api/refusalText.ts` `localizeDetail`). A bare English string reached the Hungarian UI
|
||||
verbatim: the blocking error dialog, and every inline surface that printed `HttpError.detail`, showed
|
||||
whatever sentence the route happened to hand-write. `refused` is the generic family
|
||||
(`errors.refused.<reason>`, the params interpolated into the sentence); the older specific codes
|
||||
(`not_found`, `youtube`, `playlist`, the admin refusals, …) stay, since each is already translated.
|
||||
|
||||
A test (`test_http_details.py`) reads every `HTTPException` in the app and fails on a string detail,
|
||||
and the frontend's contract test requires every `refused(…, "<reason>")` to be named in both
|
||||
languages — so the rule is enforced, not described.
|
||||
A test (`test_http_details.py`) reads every place an `HTTPException` or a `JSONResponse` sends a detail
|
||||
and fails on one the client cannot say: a sentence, a dict without a `code`, a variable, none at all.
|
||||
`tests/refusal_sites.py` reads every `{"code": …}` detail from the syntax tree, written where it is sent
|
||||
(or returned by a factory that only ever is), with any `**` spread before its `code` and `reason`, and
|
||||
follows each reason along a closed list of shapes: a literal, a `Literal` type on the parameter or
|
||||
attribute it travels in, every call of a helper like `refused` below, a map's values; anything else is
|
||||
reported, never guessed. The frontend's contract test
|
||||
(`i18n/backendIds.contract.test.ts`) runs it and requires each code in one record of
|
||||
`lib/api/refusalCodes.ts`, and each reason's sentence in both languages; a code or reason it cannot
|
||||
follow fails it, unless no screen shows that code. So the rule is enforced, not described.
|
||||
"""
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app.sysconfig import InvalidSetting, SettingReason
|
||||
|
||||
|
||||
def refused(status: int, reason: str, **params: object) -> HTTPException:
|
||||
"""The refusal to raise: `raise refused(409, "view_name_taken", name=name)`. `params` travel in
|
||||
the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see."""
|
||||
return HTTPException(status_code=status, detail={"code": "refused", "reason": reason, **params})
|
||||
the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see.
|
||||
They are spread first, so a `code` or `reason` among them could never replace these two."""
|
||||
return HTTPException(status_code=status, detail={**params, "code": "refused", "reason": reason})
|
||||
|
||||
|
||||
# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the frontend's
|
||||
# contract test can read every reason this can produce.
|
||||
_SETTING_REASONS = {
|
||||
# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the refusal
|
||||
# scan reads every reason this can produce, and holds its keys to `SettingReason`.
|
||||
_SETTING_REASONS: dict[SettingReason, str] = {
|
||||
"not_a_number": "setting_not_a_number",
|
||||
"too_low": "setting_too_low",
|
||||
"too_high": "setting_too_high",
|
||||
@@ -32,6 +42,6 @@ _SETTING_REASONS = {
|
||||
}
|
||||
|
||||
|
||||
def setting_refused(exc) -> HTTPException:
|
||||
def setting_refused(exc: InvalidSetting) -> HTTPException:
|
||||
"""A value the admin entered that `sysconfig.set_value` refused (`exc`: its `InvalidSetting`)."""
|
||||
return refused(400, _SETTING_REASONS[exc.reason], key=exc.key, **exc.params)
|
||||
|
||||
@@ -6,9 +6,9 @@ and "fetch a row this user owns or 404" idioms have one definition, one status c
|
||||
one existence-leak policy.
|
||||
"""
|
||||
|
||||
from typing import Any, Protocol, TypeVar
|
||||
from typing import Annotated, Any, Protocol, TypeVar
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
from fastapi import HTTPException, Query, Request
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -21,6 +21,10 @@ from app.youtube.client import YouTubeError, refusal_reason
|
||||
|
||||
M = TypeVar("M", bound=Base)
|
||||
|
||||
# A shuffle / surprise seed off the query string. The client draws one under a billion; bounded so a
|
||||
# hand-typed 400-digit one is a 422, not an int no float or bigint can hold further down.
|
||||
Seed = Annotated[int, Query(ge=-(2**53), le=2**53)]
|
||||
|
||||
# Resume-position thresholds of a player that only keeps a resume point (the YouTube player, the
|
||||
# library's): positions below this are "didn't really start" and within this of the end are
|
||||
# "basically finished" — neither is worth storing, so the position is cleared instead (no progress
|
||||
|
||||
+19
-10
@@ -976,7 +976,16 @@ def _resend_verify_if_eligible(email: str) -> None:
|
||||
send_verification_link(db, user)
|
||||
|
||||
|
||||
def _login_refused(status: int, reason: str, user: User | None = None, **extra) -> HTTPException:
|
||||
# Every refusal a password sign-in answers with; the UI names each one (`welcome.auth.blocked.<reason>`,
|
||||
# held by the frontend's backendIds contract, which reads this Literal).
|
||||
LoginRefusal = Literal[
|
||||
"rate_limited", "invalid", "suspended", "email_unverified", "pending_approval"
|
||||
]
|
||||
|
||||
|
||||
def _login_refused(
|
||||
status: int, reason: LoginRefusal, user: User | None = None, **extra
|
||||
) -> HTTPException:
|
||||
"""Every refusal password_login can answer with, in the codebase's structured `{code, reason}`
|
||||
shape so the SPA can say it in the user's own language (the old bare English strings were
|
||||
unreadable under a bilingual UI, and unusable as anything but a one-line error).
|
||||
@@ -992,15 +1001,15 @@ def _login_refused(status: int, reason: str, user: User | None = None, **extra)
|
||||
`email_verified` is false, and vice versa — the SPA renders whichever is actually outstanding
|
||||
rather than assuming a sequence.
|
||||
"""
|
||||
detail: dict = {"code": "login", "reason": reason}
|
||||
if user is not None:
|
||||
detail |= {
|
||||
"email": user.email,
|
||||
"email_verified": user.email_verified,
|
||||
"approved": user.is_active,
|
||||
}
|
||||
detail |= {k: v for k, v in extra.items() if v is not None}
|
||||
return HTTPException(status_code=status, detail=detail)
|
||||
account = (
|
||||
{"email": user.email, "email_verified": user.email_verified, "approved": user.is_active}
|
||||
if user is not None
|
||||
else {}
|
||||
)
|
||||
given = {k: v for k, v in extra.items() if v is not None}
|
||||
return HTTPException(
|
||||
status_code=status, detail={**account, **given, "code": "login", "reason": reason}
|
||||
)
|
||||
|
||||
|
||||
@router.post("/password-login")
|
||||
|
||||
@@ -76,6 +76,10 @@ def list_channels(user: User = Depends(current_user), db: Session = Depends(get_
|
||||
"details_synced": ch.details_synced_at is not None,
|
||||
"recent_synced": ch.recent_synced_at is not None,
|
||||
"backfill_done": ch.backfill_done,
|
||||
# When the follow reached this list: written by the subscription import the run that
|
||||
# first saw it, or by a subscribe made here — not the channel row's own date, which an
|
||||
# explored or discovered channel had before it was followed.
|
||||
"added_at": iso(sub.created_at),
|
||||
}
|
||||
for sub, ch in rows
|
||||
]
|
||||
|
||||
@@ -23,7 +23,7 @@ from app.discover import pins, rails
|
||||
from app.discover.budget import Budget, DEFAULT_BUDGET
|
||||
from app.models import User
|
||||
from app.plex.cards import norm_lang
|
||||
from app.routes._common import not_found
|
||||
from app.routes._common import Seed, not_found
|
||||
|
||||
router = APIRouter(prefix="/api/discover", tags=["discover"])
|
||||
|
||||
@@ -44,7 +44,7 @@ class ReorderIn(BaseModel):
|
||||
@router.get("")
|
||||
def get_discover(
|
||||
budget: Budget = DEFAULT_BUDGET,
|
||||
seed: int = 0,
|
||||
seed: Seed = 0,
|
||||
lang: str = "en",
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
@@ -79,7 +79,7 @@ MAX_EXCLUDE = 32
|
||||
@router.get("/pick")
|
||||
def get_pick(
|
||||
budget: Budget = DEFAULT_BUDGET,
|
||||
seed: int = 0,
|
||||
seed: Seed = 0,
|
||||
exclude: str = "",
|
||||
start: int = 0,
|
||||
lang: str = "en",
|
||||
|
||||
@@ -208,7 +208,7 @@ def set_credential(
|
||||
try:
|
||||
cleaned = dlcreds.validate_cookies(body.cookies)
|
||||
except dlcreds.InvalidCookies as e:
|
||||
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)}) from e
|
||||
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": e.reason}) from e
|
||||
dlcreds.upsert_credential(db, user.id, host_key, cleaned)
|
||||
audit.record(
|
||||
db,
|
||||
@@ -318,7 +318,9 @@ def enqueue_edit(
|
||||
except quota.QuotaExceeded as e:
|
||||
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
|
||||
except service.EditError as e:
|
||||
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason}) from e
|
||||
raise HTTPException(
|
||||
status_code=400, detail={**e.params, "code": "edit", "reason": e.reason}
|
||||
) from e
|
||||
db.commit()
|
||||
return serialize_job(db, job)
|
||||
|
||||
@@ -341,6 +343,8 @@ class EnqueueClipIn(BaseModel):
|
||||
audio_track: int | None = None
|
||||
# A crop frame (downloads/clip): fractions of the box it was drawn in, and that box's aspect.
|
||||
crop: ClipCropIn | None = None
|
||||
# A GIF of this long side (downloads/gif) instead of a video; None is the video clip.
|
||||
gif_side: int | None = None
|
||||
display_name: str | None = None
|
||||
|
||||
|
||||
@@ -353,14 +357,15 @@ def enqueue_clip(
|
||||
try:
|
||||
crop = body.crop.model_dump() if body.crop else None
|
||||
spec = clipmod.normalize(
|
||||
body.source, body.ref, body.start_s, body.end_s, body.audio_track, crop
|
||||
body.source, body.ref, body.start_s, body.end_s, body.audio_track, crop, body.gif_side
|
||||
)
|
||||
display_name = display_text(body.display_name)
|
||||
job = service.enqueue_clip(db, user.id, spec, display_name=display_name)
|
||||
except clipmod.ClipError as e:
|
||||
status = 404 if e.reason == "not_found" else 422
|
||||
detail = {"code": "clip", "reason": e.reason, **e.params}
|
||||
raise HTTPException(status_code=status, detail=detail) from e
|
||||
raise HTTPException(
|
||||
status_code=status, detail={**e.params, "code": "clip", "reason": e.reason}
|
||||
) from e
|
||||
except quota.QuotaExceeded as e:
|
||||
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
|
||||
db.commit()
|
||||
@@ -610,7 +615,7 @@ def list_links(
|
||||
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
) -> list[dict]:
|
||||
own_job(db, user, job_id)
|
||||
return [linksmod.owner_view(link) for link in queries.links_to(db, job_id=job_id)]
|
||||
return [linksmod.owner_view(db, link) for link in queries.links_to(db, job_id=job_id)]
|
||||
|
||||
|
||||
class CreateLinkIn(BaseModel):
|
||||
@@ -652,7 +657,7 @@ def create_link(
|
||||
raise refused(409, "share_unfinished")
|
||||
link, token = _new_link(db, user, body, job_id=job_id)
|
||||
db.commit()
|
||||
return linksmod.owner_view(link, token=token)
|
||||
return linksmod.owner_view(db, link, token=token)
|
||||
|
||||
|
||||
def _shareable_job(db: Session, user: User, job_id: int) -> "DownloadJob":
|
||||
@@ -683,7 +688,7 @@ def list_folder_links(
|
||||
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
) -> list[dict]:
|
||||
own_folder(db, user, folder_id)
|
||||
return [linksmod.owner_view(link) for link in queries.links_to(db, folder_id=folder_id)]
|
||||
return [linksmod.owner_view(db, link) for link in queries.links_to(db, folder_id=folder_id)]
|
||||
|
||||
|
||||
@router.post("/folders/{folder_id}/links")
|
||||
@@ -696,7 +701,7 @@ def create_folder_link(
|
||||
_shareable_folder(db, user, folder_id)
|
||||
link, token = _new_link(db, user, body, folder_id=folder_id)
|
||||
db.commit()
|
||||
return linksmod.owner_view(link, token=token)
|
||||
return linksmod.owner_view(db, link, token=token)
|
||||
|
||||
|
||||
class UpdateLinkIn(BaseModel):
|
||||
@@ -723,7 +728,7 @@ def update_link(
|
||||
sent=frozenset(body.model_fields_set),
|
||||
)
|
||||
db.commit()
|
||||
return linksmod.owner_view(link)
|
||||
return linksmod.owner_view(db, link)
|
||||
|
||||
|
||||
@router.delete("/links/{link_id}")
|
||||
|
||||
+13
-12
@@ -10,6 +10,7 @@ from sqlalchemy.orm.exc import StaleDataError
|
||||
|
||||
from app import jobruns, quota, scheduler
|
||||
from app.catalog import queries as catalog_queries, service as catalog_service
|
||||
from app.catalog.models import known_length
|
||||
from app.auth import current_user
|
||||
from app.db import get_db, release_read_connection
|
||||
from app.mailtext import lang_for_user
|
||||
@@ -19,6 +20,7 @@ from app.models import User
|
||||
from app.reader import queries, service, takeout
|
||||
from app.routes._common import (
|
||||
ProgressIn,
|
||||
Seed,
|
||||
capped_body,
|
||||
drain_body,
|
||||
kept_position,
|
||||
@@ -88,7 +90,7 @@ def _feed_params(
|
||||
def get_feed(
|
||||
params: dict = Depends(_feed_params),
|
||||
sort: str = "newest",
|
||||
seed: int = 0,
|
||||
seed: Seed = 0,
|
||||
limit: int = Query(default=60, le=200),
|
||||
cursor: str | None = None,
|
||||
user: User = Depends(current_user),
|
||||
@@ -204,22 +206,21 @@ def set_video_state(
|
||||
return {"ok": True, "video_id": video_id, "status": status}
|
||||
|
||||
|
||||
@router.delete("/videos/{video_id}/state")
|
||||
def clear_video_state(
|
||||
@router.delete("/videos/{video_id}/progress")
|
||||
def clear_video_progress(
|
||||
video_id: str,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Reset a video to pristine for this user — drop the whole VideoState row (status,
|
||||
watch position and watched_at), as if it had never been opened. Saved/playlist membership
|
||||
lives elsewhere and is untouched."""
|
||||
"""Forget where the user left a video: its resume point goes, its status (watched, hidden) stays.
|
||||
Saved/playlist membership lives elsewhere and is untouched."""
|
||||
try:
|
||||
service.clear_state(db, user, video_id)
|
||||
service.set_progress(db, user, video_id, 0)
|
||||
db.commit()
|
||||
# silent-ok: the row went under the delete, which is the outcome the delete wanted.
|
||||
# silent-ok: the row went under the write, which is the outcome the write wanted.
|
||||
except StaleDataError:
|
||||
db.rollback()
|
||||
return {"ok": True, "video_id": video_id, "status": "new", "position_seconds": 0}
|
||||
return {"ok": True, "video_id": video_id, "position_seconds": 0}
|
||||
|
||||
|
||||
@router.post("/videos/{video_id}/progress")
|
||||
@@ -288,7 +289,7 @@ def get_video_detail(
|
||||
"channel_title": v.channel.title if v.channel else None,
|
||||
"published_at": iso(v.published_at),
|
||||
"view_count": v.view_count,
|
||||
"duration_seconds": v.duration_seconds,
|
||||
"duration_seconds": v.length_seconds,
|
||||
}
|
||||
|
||||
# The DB path above is free; the off-catalog YouTube lookup below spends a quota unit.
|
||||
@@ -316,8 +317,8 @@ def get_video_detail(
|
||||
"channel_title": snippet.get("channelTitle"),
|
||||
"published_at": snippet.get("publishedAt"),
|
||||
"view_count": int(views) if views is not None else None,
|
||||
"duration_seconds": parse_iso8601_duration(
|
||||
items[0].get("contentDetails", {}).get("duration")
|
||||
"duration_seconds": known_length(
|
||||
parse_iso8601_duration(items[0].get("contentDetails", {}).get("duration"))
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -139,7 +139,7 @@ def own_job_for_update(db: Session, user: User, job_id: int) -> "DownloadJob":
|
||||
|
||||
def quota_detail(e: quota.QuotaExceeded) -> dict:
|
||||
"""Structured error the trilingual client localizes (reason key + numbers), instead of a
|
||||
pre-baked English string with a dot-decimal GB value. See lib/api.ts localizeDetail()."""
|
||||
pre-baked English string with a dot-decimal GB value. See lib/api/refusalText.ts localizeDetail()."""
|
||||
return {"code": "quota", "reason": e.reason, "limit": e.limit, "current": e.current}
|
||||
|
||||
|
||||
@@ -715,13 +715,8 @@ def get_file(
|
||||
return response
|
||||
|
||||
|
||||
# A preview's own headers. `sandbox` leaves the answer no script and an opaque origin, whatever the
|
||||
# file turns out to hold; `frame-ancestors 'self'` lets the app's viewer frame a PDF (every other
|
||||
# answer keeps the app-wide 'none', main.py), with X-Frame-Options saying the same for old engines.
|
||||
PREVIEW_HEADERS = {
|
||||
"Content-Security-Policy": "sandbox; frame-ancestors 'self'",
|
||||
"X-Frame-Options": "SAMEORIGIN",
|
||||
}
|
||||
# A preview's own headers, shared with a share link's picture (`preview.SANDBOX_HEADERS`).
|
||||
PREVIEW_HEADERS = preview.SANDBOX_HEADERS
|
||||
# Set on a text preview cut at `preview.TEXT_PREVIEW_BYTES`, so the viewer can say it shows the start.
|
||||
TRUNCATED_HEADER = "X-Preview-Truncated"
|
||||
|
||||
|
||||
+177
-49
@@ -3,10 +3,10 @@ CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-w
|
||||
mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
|
||||
out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, model_validator
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import quota
|
||||
@@ -41,11 +41,16 @@ router = APIRouter(prefix="/api/playlists", tags=["playlists"])
|
||||
# It guards EVERY route that spends under QuotaAction.PLAYLISTS_PULL, not just the bulk one: a
|
||||
# scripted loop over the single-playlist re-pull drains the same shared budget, and limiting one of
|
||||
# two routes with the same quota action is the asymmetric fix this codebase keeps re-learning.
|
||||
# Deliberately NOT on the push side (`push-plan` / `push`): those run on every push attempt,
|
||||
# including the no-op "already up to date" path, so a 3-per-10-minute allowance would refuse
|
||||
# legitimate work. Their exposure is tracked as SEC-18 instead.
|
||||
_playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600)
|
||||
|
||||
# SEC-18, the push side: `push-plan` reads the live list (1 unit a page) and `push` reads and writes
|
||||
# it, both against the same instance-wide budget. They get a budget of their own, counted in calls
|
||||
# and far wider than the pull's: a press in the UI is one plan and, unless nothing changed, one push,
|
||||
# so no person reaches it, while a scripted loop stops at it. A big push is not this limit's job —
|
||||
# `push` already refuses one the day's remaining budget cannot cover.
|
||||
PUSH_ALLOWANCE = 20
|
||||
_playlist_push_limiter = RateLimiter(max_events=PUSH_ALLOWANCE, window_seconds=600)
|
||||
|
||||
|
||||
def _check_pull_cooldown(user: User) -> None:
|
||||
"""Shared gate so the two PLAYLISTS_PULL routes cannot drift apart again."""
|
||||
@@ -53,7 +58,23 @@ def _check_pull_cooldown(user: User) -> None:
|
||||
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
|
||||
|
||||
|
||||
def _refused(status: int, reason: str) -> HTTPException:
|
||||
# Every playlist action the server refuses; the UI names each one (`playlists.errors.<reason>`, held by
|
||||
# the frontend's backendIds contract, which reads this Literal).
|
||||
PlaylistRefusal = Literal[
|
||||
"name_required",
|
||||
"not_on_youtube",
|
||||
"push_cooldown",
|
||||
"quota_exceeded",
|
||||
"sync_all_busy",
|
||||
"sync_busy",
|
||||
"watch_later_not_pushable",
|
||||
"watch_later_undeletable",
|
||||
"youtube_editing_off",
|
||||
"youtube_read_scope",
|
||||
]
|
||||
|
||||
|
||||
def _refused(status: int, reason: PlaylistRefusal) -> HTTPException:
|
||||
"""A playlist action the server refuses, as a reason the UI translates
|
||||
(`playlists.errors.<reason>`) — a bare English detail reached the Hungarian UI verbatim."""
|
||||
return HTTPException(status_code=status, detail={"code": "playlist", "reason": reason})
|
||||
@@ -74,10 +95,13 @@ def _video_or_404(db: Session, video_id: str) -> None:
|
||||
def _summary(
|
||||
pl: "Playlist",
|
||||
count: int,
|
||||
total_duration: int = 0,
|
||||
duration: tuple[int, int] = (0, 0),
|
||||
has_video: bool | None = None,
|
||||
cover: str | None = None,
|
||||
) -> dict:
|
||||
"""`duration` is `queries.durations`'s pair: the known running time, and the videos it left out
|
||||
for having no length — the "+N without length" beside it."""
|
||||
total_duration, left_out = duration
|
||||
out = {
|
||||
"id": pl.id,
|
||||
"name": pl.name,
|
||||
@@ -87,6 +111,7 @@ def _summary(
|
||||
"dirty": pl.dirty,
|
||||
"item_count": count,
|
||||
"total_duration_seconds": total_duration,
|
||||
"without_length": left_out,
|
||||
"cover_thumbnail": cover,
|
||||
# When this list last CHANGED (rename, add/remove/move/reorder) — the rail's
|
||||
# "recently changed first" sort reads it. See `service.touch` for why it isn't automatic.
|
||||
@@ -117,7 +142,7 @@ def list_playlists(
|
||||
_summary(
|
||||
p,
|
||||
counts.get(p.id, 0),
|
||||
durations.get(p.id, 0),
|
||||
durations.get(p.id, (0, 0)),
|
||||
(p.id in member) if contains else None,
|
||||
covers.get(p.id),
|
||||
)
|
||||
@@ -229,9 +254,16 @@ def revert_youtube(
|
||||
raise youtube_refusal(exc, playlist=True) from exc
|
||||
|
||||
|
||||
def _check_push_cooldown(user: User) -> None:
|
||||
"""Shared gate of the two push routes, so they spend one allowance and cannot drift apart."""
|
||||
if not _playlist_push_limiter.allow(str(user.id)):
|
||||
raise _refused(429, "push_cooldown")
|
||||
|
||||
|
||||
def _pushable(db: Session, user: User, playlist_id: int) -> "Playlist":
|
||||
"""A local playlist the user may push to YouTube (create or update). The built-in
|
||||
Watch later and read-only YouTube mirrors are never pushable."""
|
||||
"""A playlist the user may push to YouTube: a local one (created there on its first push) or a
|
||||
mirror of their own YouTube list (its local edits merged back). Only the built-in Watch later is
|
||||
never pushable."""
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
if pl.kind == "watch_later":
|
||||
raise _refused(400, "watch_later_not_pushable")
|
||||
@@ -248,6 +280,7 @@ def push_plan(
|
||||
the quota estimate, so the UI can show a confirm. For a linked playlist this reads the live
|
||||
YouTube state (cheap, 1 unit/page). Write scope is required — syncing is a YouTube write."""
|
||||
pl = _pushable(db, user, playlist_id)
|
||||
_check_push_cooldown(user) # after the refusal: a list that cannot be pushed spends nothing
|
||||
try:
|
||||
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
|
||||
plan = plan_merge(db, user, pl)
|
||||
@@ -273,6 +306,8 @@ def push(
|
||||
with sync_claim(db, pl.id) as free:
|
||||
if not free:
|
||||
raise _refused(409, "sync_busy")
|
||||
# Spent only by a push that runs, as Revert's pull: a busy list or Watch later costs nothing.
|
||||
_check_push_cooldown(user)
|
||||
try:
|
||||
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
|
||||
# Read the live playlist ONCE (linked playlists only) and share it with both the
|
||||
@@ -334,8 +369,8 @@ def rename_playlist(
|
||||
raise _refused(400, "name_required")
|
||||
service.rename_playlist(db, pl, name)
|
||||
db.commit()
|
||||
count, total, cover = queries.playlist_totals(db, pl.id)
|
||||
return _summary(pl, count, total, cover=cover)
|
||||
count, total, left_out, cover = queries.playlist_totals(db, pl.id)
|
||||
return _summary(pl, count, (total, left_out), cover=cover)
|
||||
|
||||
|
||||
@router.delete("/{playlist_id}")
|
||||
@@ -371,24 +406,17 @@ def delete_playlist(
|
||||
return {"deleted": playlist_id}
|
||||
|
||||
|
||||
class AddItemIn(VideoRefIn):
|
||||
# Where the video lands among the list's items, in their stored order (0 = first). Omitted, it
|
||||
# is appended. The list page's "Undo" of a removal puts the row back where it was with this —
|
||||
# ONE request, because append-then-reorder leaves it at the end whenever the second call fails.
|
||||
position: int | None = Field(default=None, ge=0)
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items")
|
||||
def add_item(
|
||||
playlist_id: int,
|
||||
body: AddItemIn,
|
||||
body: VideoRefIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
pl = _own_playlist(db, user, playlist_id, lock=True)
|
||||
video_id = body.video_id
|
||||
_video_or_404(db, video_id)
|
||||
service.add_to_playlist(db, pl, video_id, body.position)
|
||||
service.add_item(db, pl, video_id, mark_dirty=True)
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
|
||||
|
||||
@@ -400,58 +428,158 @@ def remove_item(
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
pl = _own_playlist(db, user, playlist_id, lock=True)
|
||||
service.remove_item(db, pl, video_id, mark_dirty=True)
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
|
||||
|
||||
|
||||
class MoveItemIn(BaseModel):
|
||||
class VideoIdsIn(BaseModel):
|
||||
# Required: with a default, a misnamed key answered 200 having acted on no video.
|
||||
video_ids: list[str]
|
||||
|
||||
|
||||
def _paired(video_count: int, positions: list[int]) -> None:
|
||||
"""One place per video, none before the start and none twice: both undo bodies come back from
|
||||
the client, and the insert's arithmetic counts on places that only rise."""
|
||||
if len(positions) != video_count:
|
||||
raise ValueError("positions must pair one-to-one with the videos")
|
||||
if any(at < 0 for at in positions):
|
||||
raise ValueError("positions cannot be negative")
|
||||
if len(set(positions)) != len(positions):
|
||||
raise ValueError("positions cannot repeat")
|
||||
|
||||
|
||||
class RestoreItemsIn(VideoIdsIn):
|
||||
# Where each video goes back: an index over the ordered rows, one per video.
|
||||
positions: list[int] = []
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _one_position_per_video(self):
|
||||
_paired(len(self.video_ids), self.positions)
|
||||
return self
|
||||
|
||||
|
||||
class MoveItemsIn(VideoIdsIn):
|
||||
target_playlist_id: int
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/{video_id}/move")
|
||||
def move_item(
|
||||
class UndoMoveIn(BaseModel):
|
||||
from_playlist_id: int
|
||||
# Required, as `video_ids` is. An empty one is still a real body: the target held them all.
|
||||
moved: list[str]
|
||||
already_there: list[str] = []
|
||||
# One per video of `moved` + `already_there`, in that order.
|
||||
positions: list[int]
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _one_position_per_video(self):
|
||||
_paired(len(self.moved) + len(self.already_there), self.positions)
|
||||
return self
|
||||
|
||||
|
||||
def _list_gone(e: service.ListGone) -> HTTPException:
|
||||
"""Each list of a two-list write under the reason the route's own read gives it."""
|
||||
return not_found("playlist") if e.source else not_found("move_target")
|
||||
|
||||
|
||||
def _move_refused(e: service.MoveRefused) -> HTTPException:
|
||||
return HTTPException(status_code=400, detail={"code": "playlist_move", "reason": e.reason})
|
||||
|
||||
|
||||
def _other_list(db: Session, user: User, playlist_id: int) -> "Playlist":
|
||||
"""The other list of a move. Its own reason: "this playlist doesn't exist any more" about a list
|
||||
that is gone was read as the one on screen, which is still there."""
|
||||
other = queries.own_playlist(db, user, playlist_id)
|
||||
if other is None:
|
||||
raise not_found("move_target")
|
||||
return other
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/remove-bulk")
|
||||
def remove_items(
|
||||
playlist_id: int,
|
||||
video_id: str,
|
||||
body: MoveItemIn,
|
||||
body: VideoIdsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Move one video from a playlist into another of the caller's playlists, atomically.
|
||||
"""Remove many videos from a playlist at once. Returns the ones that went."""
|
||||
pl = _own_playlist(db, user, playlist_id, lock=True)
|
||||
removed = service.remove_items(db, pl, body.video_ids)
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": pl.id, "removed": removed}
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/restore")
|
||||
def restore_items(
|
||||
playlist_id: int,
|
||||
body: RestoreItemsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""The "Undo" of a removal: the videos back at their places. Returns how many came back."""
|
||||
pl = _own_playlist(db, user, playlist_id, lock=True)
|
||||
added = service.restore_items(db, pl, body.video_ids, body.positions)
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": pl.id, "added": added}
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/move-bulk")
|
||||
def move_items(
|
||||
playlist_id: int,
|
||||
body: MoveItemsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Move videos into another of the caller's playlists, atomically.
|
||||
|
||||
ONE transaction on purpose. The obvious client-side shape — add to the target, then remove
|
||||
from the source — leaves the video in BOTH lists whenever the second call fails, which is
|
||||
exactly the kind of half-applied state this codebase treats as a bug rather than an edge
|
||||
case. Idempotent: if the video is already in the target, the source row is simply dropped
|
||||
(moving into a list that already has it is a removal, not a duplicate)."""
|
||||
from the source — leaves a video in BOTH lists whenever the second call fails. A video the
|
||||
target already has only leaves the source, and one a concurrent edit (another tab, a YouTube
|
||||
re-pull) already took out is skipped. Says which landed there and which it already held, for
|
||||
the "Undo"."""
|
||||
src = _own_playlist(db, user, playlist_id)
|
||||
# Its own reason: "this playlist doesn't exist any more" about a TARGET that is gone was read as
|
||||
# the list on screen — which is still there.
|
||||
dst = queries.own_playlist(db, user, body.target_playlist_id)
|
||||
if dst is None:
|
||||
raise not_found("move_target")
|
||||
dst = _other_list(db, user, body.target_playlist_id)
|
||||
try:
|
||||
service.move_item(db, src, dst, video_id)
|
||||
moved, already_there = service.move_items(db, src, dst, body.video_ids)
|
||||
except service.MoveRefused as e:
|
||||
# A same-list move is a bad request; a video a concurrent edit (another tab, a YouTube
|
||||
# re-pull) already took out is gone.
|
||||
status = 400 if e.reason == "same_playlist" else 404
|
||||
raise HTTPException(
|
||||
status_code=status, detail={"code": "playlist_move", "reason": e.reason}
|
||||
) from None
|
||||
raise _move_refused(e) from None
|
||||
except service.ListGone as e:
|
||||
raise _list_gone(e) from None
|
||||
db.commit()
|
||||
return {"ok": True, "from_playlist_id": src.id, "to_playlist_id": dst.id, "video_id": video_id}
|
||||
return {
|
||||
"ok": True,
|
||||
"from_playlist_id": src.id,
|
||||
"to_playlist_id": dst.id,
|
||||
"moved": moved,
|
||||
"already_there": already_there,
|
||||
}
|
||||
|
||||
|
||||
class ReorderItemsIn(BaseModel):
|
||||
video_ids: list[str] = []
|
||||
@router.post("/{playlist_id}/items/undo-move")
|
||||
def undo_move(
|
||||
playlist_id: int,
|
||||
body: UndoMoveIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""The "Undo" of a move, back into this playlist in one transaction. Returns how many came back."""
|
||||
src = _own_playlist(db, user, playlist_id)
|
||||
dst = _other_list(db, user, body.from_playlist_id)
|
||||
try:
|
||||
added = service.undo_move(db, src, dst, body.moved, body.already_there, body.positions)
|
||||
except service.MoveRefused as e:
|
||||
raise _move_refused(e) from None
|
||||
except service.ListGone as e:
|
||||
raise _list_gone(e) from None
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": src.id, "added": added}
|
||||
|
||||
|
||||
@router.put("/{playlist_id}/order")
|
||||
def reorder_items(
|
||||
playlist_id: int,
|
||||
body: ReorderItemsIn,
|
||||
body: VideoIdsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
|
||||
@@ -432,6 +432,7 @@ def item_detail(
|
||||
"summary": (it.summary_hu or it.summary) if hu else it.summary,
|
||||
"year": it.year,
|
||||
"duration_seconds": it.duration_s,
|
||||
"size_bytes": it.size_bytes,
|
||||
"playable": it.playable,
|
||||
# Playback-class detail: HW/SW transcode split + the codec/bit-depth the info-page chip shows.
|
||||
"transcode_hw": it.transcode_hw,
|
||||
|
||||
@@ -234,6 +234,15 @@ def person_image(u: str, request: Request) -> Response:
|
||||
return imgcache.response(data, ct)
|
||||
|
||||
|
||||
def _supersede_resume_pushes(user_id: int, item_id: int, now: datetime) -> None:
|
||||
"""A watched mark supersedes every resume push still in flight for the item: stamping it as the
|
||||
latest push makes `_resume_push_superseded` drop them. Popping the entry instead (to keep the map
|
||||
small) removed the very guard, and an older resume push could then land at Plex AFTER the
|
||||
scrobble and revive a resume point on a watched item. Eviction bounds the map."""
|
||||
with _push_throttle_lock:
|
||||
_last_push_at[(user_id, item_id)] = now
|
||||
|
||||
|
||||
def _push_resume_guarded(
|
||||
user_id: int,
|
||||
item_id: int,
|
||||
@@ -314,13 +323,14 @@ def item_progress(
|
||||
`_should_push_resume`). `final` (sent on pause / seek / pagehide / unmount) pushes immediately and
|
||||
resets the throttle window, so Plex sees a settled position the instant the user pauses or seeks."""
|
||||
it = item_or_404(db, rating_key)
|
||||
# A player parked mid-item — playing or PAUSED — checkpoints every 10 s, so this is the signal
|
||||
# A player parked mid-item — playing or PAUSED — keeps checkpointing (every 10 s, once a minute
|
||||
# in a hidden silent page), so this is the signal
|
||||
# that such a viewer still wants their HLS session: the one thing `last_access` could not see,
|
||||
# since it moved only on a file fetch and a paused player fetches nothing (see
|
||||
# stream.reap_idle). It is NOT every open player: `usePlexProgress.flush` sends nothing once the
|
||||
# finish latch is up, or before the first position is known. Those are the states in which
|
||||
# reaping is the right answer anyway, and a viewer who scrubs back into a reaped session gets
|
||||
# the same 404 → reclaim the player already handles.
|
||||
# the same 410 → reclaim the player already handles.
|
||||
plex_stream.touch_session(user.id, str(rating_key))
|
||||
position = max(0, int(body.position_seconds))
|
||||
duration = int(body.duration_seconds)
|
||||
@@ -329,12 +339,7 @@ def item_progress(
|
||||
near_end = duration > 0 and position > duration - _FINISH_MARGIN_S
|
||||
if near_end:
|
||||
flipped = service.mark_finished(db, user.id, it.id, now)
|
||||
# The finish supersedes every resume push still in flight for this item: stamping it as the
|
||||
# latest push makes `_resume_push_superseded` drop them. Popping the entry instead (to keep
|
||||
# the map small) removed the very guard, and an older resume push could then land at Plex
|
||||
# AFTER the scrobble and revive a resume point on a watched item. Eviction bounds the map.
|
||||
with _push_throttle_lock:
|
||||
_last_push_at[(user.id, it.id)] = now
|
||||
_supersede_resume_pushes(user.id, it.id, now)
|
||||
db.commit()
|
||||
# Only the checkpoint that flipped the row scrobbles: a later checkpoint of the same finished
|
||||
# item — or a second one racing this one — would otherwise bump Plex's viewCount again.
|
||||
@@ -352,7 +357,11 @@ def item_progress(
|
||||
push_now = _should_push_resume(user.id, it.id, now, final)
|
||||
# Marked dirty before the (background) push so the reconcile knows to re-push if it fails, and so
|
||||
# `push_state_to_plex` can settle synced_to_plex once Plex confirms.
|
||||
service.checkpoint(db, user, it.id, st, position, now, dirty=push_now)
|
||||
if not service.checkpoint(db, user, it.id, st, position, now, dirty=push_now):
|
||||
# Marked watched a moment ago: this is the finishing player's late tick (see
|
||||
# `service.checkpoint`), and the answer is the same one a near-end checkpoint gets.
|
||||
db.commit()
|
||||
return {"ok": True, "status": "watched", "position_seconds": 0}
|
||||
db.commit()
|
||||
if push_now:
|
||||
_push_watch(
|
||||
@@ -408,6 +417,8 @@ def item_state(
|
||||
_push_watch(background, db, user.id, it.id, rating_key, "unwatched")
|
||||
return {"ok": True, "status": "new"}
|
||||
service.set_status(db, user, it.id, st, status)
|
||||
if status == "watched":
|
||||
_supersede_resume_pushes(user.id, it.id, datetime.now(timezone.utc))
|
||||
db.commit()
|
||||
if status == "watched" and prev_status != "watched":
|
||||
_push_watch(background, db, user.id, it.id, rating_key, "watched")
|
||||
@@ -427,6 +438,10 @@ def _bulk_state(
|
||||
always reads the committed marks."""
|
||||
pushable = plex_watch.link_for_push(db, user_id) is not None
|
||||
to_push = service.bulk_state(db, user_id, eps, watched)
|
||||
if watched:
|
||||
now = datetime.now(timezone.utc)
|
||||
for item_id, _rating_key, _action in to_push:
|
||||
_supersede_resume_pushes(user_id, item_id, now)
|
||||
if pushable and to_push:
|
||||
# One coalesced background task for the whole batch (reuses a single session + Plex client),
|
||||
# not one task per episode — a 200-ep show mark is 1 task, not 200.
|
||||
@@ -595,7 +610,9 @@ def stream_hls_file(
|
||||
is per (key, viewer, offset, tag) — so resolving by the key alone handed the old player the new
|
||||
session's `seg_12.ts`, produced from a different offset under `-copyts`: wrong
|
||||
bytes with a 200, which the player has no way to notice. A request for a
|
||||
session that is no longer the registered one is a 404, and the player reclaims at the playhead.
|
||||
session that is no longer the registered one is a 410, and the player reclaims at the playhead.
|
||||
410, not 404: a 404 here means "that segment is not there yet", worth hls.js's retries, and the
|
||||
player can only read the status (a gone session retried for half a minute behind that 404).
|
||||
|
||||
That reclaim is also why a request with no live session is no longer auto-started here: the
|
||||
auto-start could only ever begin at 0, silently throwing away the viewer's position."""
|
||||
@@ -611,7 +628,7 @@ def stream_hls_file(
|
||||
raise not_found("file")
|
||||
s = plex_stream.current_session(user.id, str(rating_key))
|
||||
if s is None or s.id != session_id:
|
||||
raise refused(404, "session_gone")
|
||||
raise refused(410, "session_gone")
|
||||
path = s.dir / filename
|
||||
# Only a name THIS session can actually produce is worth waiting for: its own playlists, and a
|
||||
# segment of one of its own renditions whose predecessor is already on disk (ffmpeg writes them
|
||||
@@ -632,14 +649,15 @@ def stream_hls_file(
|
||||
try:
|
||||
content = path.read_bytes()
|
||||
except OSError:
|
||||
# The session was torn down between the wait and the read — `_kill` rmtrees the whole
|
||||
# directory, and nothing holds `_lock` across those two steps. It has to present as the same
|
||||
# 404 a gone session does: the player's recovery is keyed on the STATUS (`code === 404`), so
|
||||
# The session was torn down between the wait and the read — teardown moves the whole
|
||||
# directory aside and deletes it, and nothing holds `_lock` across those two steps. It has
|
||||
# to present as the same 410 a gone session does: the player's recovery is keyed on the
|
||||
# STATUS, so
|
||||
# a 500 here skipped the reclaim and ended playback with "couldn't play" instead of
|
||||
# restarting at the playhead. Ordinary trigger: `loadSession` awaits the new session — which
|
||||
# kills and rmtrees the old one — before destroying the old hls instance, so the outgoing
|
||||
# instance's in-flight segment request races the rmtree on every seek-restart.
|
||||
raise refused(404, "session_gone") from None
|
||||
raise refused(410, "session_gone") from None
|
||||
return Response(
|
||||
content=content,
|
||||
media_type=_HLS_CT[ext],
|
||||
|
||||
@@ -183,7 +183,9 @@ def playlist_remove_item(
|
||||
|
||||
|
||||
class PlexItemKeysIn(BaseModel):
|
||||
item_rating_keys: list[str] = []
|
||||
# Required: with a default, a misnamed key answered 200 having acted on no item, and a reorder
|
||||
# took it for an empty order.
|
||||
item_rating_keys: list[str]
|
||||
|
||||
|
||||
class PlexItemAddBulkIn(PlexItemKeysIn):
|
||||
|
||||
@@ -16,7 +16,7 @@ from sqlalchemy.orm import Session
|
||||
from app.config import settings
|
||||
from app.db import get_db
|
||||
from app.downloads import links as linksmod
|
||||
from app.downloads import queries, storage
|
||||
from app.downloads import preview, queries, storage
|
||||
from app.downloads.status import JobStatus
|
||||
from app.ratelimit import RateLimiter, client_key
|
||||
from app.security import verify_password
|
||||
@@ -176,8 +176,15 @@ def watch_meta(token: str, db: Session = Depends(get_db)) -> dict:
|
||||
returns only `needs_password` until the viewer unlocks it (no title/thumbnail leak)."""
|
||||
link = _link_or_404(db, token)
|
||||
if link.password_hash:
|
||||
# Which kind it opens, for the password prompt's title — never what is in it.
|
||||
return {"needs_password": True, "kind": "folder" if link.folder_id is not None else "file"}
|
||||
# Which kind it opens, for the password prompt's words — never what is in it. `video` tells
|
||||
# a video (watched) from any other file (opened); the address is only a hint, every
|
||||
# prefix opens every link.
|
||||
opens = linksmod.opens(db, link)
|
||||
return {
|
||||
"needs_password": True,
|
||||
"kind": "folder" if opens == "folder" else "file",
|
||||
"video": opens == "video",
|
||||
}
|
||||
meta = _opened(db, link, token)
|
||||
db.commit()
|
||||
return meta
|
||||
@@ -219,8 +226,9 @@ def _granted(link: "DownloadLink", token: str, g: str | None) -> None:
|
||||
|
||||
|
||||
def _serve(link: "DownloadLink", asset: "MediaAsset") -> FileResponse:
|
||||
"""Range-aware stream of a shared file. Inline by default (plays in the page); an
|
||||
`allow_download` link serves it as an attachment."""
|
||||
"""Range-aware stream of a shared file. Inline by default (a video plays and a picture shows in
|
||||
the page, and a chat's link preview can fetch the picture); an `allow_download` link serves it
|
||||
as an attachment."""
|
||||
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
|
||||
if path is None:
|
||||
raise refused(404, "video_gone")
|
||||
@@ -234,6 +242,17 @@ def _serve(link: "DownloadLink", asset: "MediaAsset") -> FileResponse:
|
||||
# served with its real audio/video type, inline unless the link forces a download so it can play
|
||||
# in the watch page. Both are range-aware (FileResponse → 206) for seeking/resume.
|
||||
media_type = storage.served_media_type(asset.media_kind, asset.container)
|
||||
disposition = "attachment" if link.allow_download else "inline"
|
||||
if media_type is None and preview.kind(asset.media_kind, asset.container) == "image":
|
||||
# A picture (a GIF made from a clip, an uploaded photo) is shown as itself: its type from the
|
||||
# preview allow-list, under the preview's sandbox, so a mislabelled one still runs nothing.
|
||||
return FileResponse(
|
||||
path,
|
||||
filename=filename,
|
||||
media_type=preview.media_type("image", asset.container),
|
||||
content_disposition_type=disposition,
|
||||
headers=preview.SANDBOX_HEADERS,
|
||||
)
|
||||
if media_type is None:
|
||||
return FileResponse(
|
||||
path,
|
||||
@@ -241,7 +260,6 @@ def _serve(link: "DownloadLink", asset: "MediaAsset") -> FileResponse:
|
||||
media_type="application/octet-stream",
|
||||
content_disposition_type="attachment",
|
||||
)
|
||||
disposition = "attachment" if link.allow_download else "inline"
|
||||
return FileResponse(
|
||||
path, filename=filename, media_type=media_type, content_disposition_type=disposition
|
||||
)
|
||||
|
||||
@@ -128,7 +128,8 @@ def update_view(
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
raise refused(409, "view_name_taken", name=view.name) from None
|
||||
# The name asked for: the rollback has put the view's old name back on `view`.
|
||||
raise refused(409, "view_name_taken", name=name) from None
|
||||
return _serialize(view)
|
||||
|
||||
|
||||
|
||||
@@ -314,7 +314,8 @@ def search_youtube(
|
||||
|
||||
|
||||
class ClearSearchIn(BaseModel):
|
||||
video_ids: list[str] = []
|
||||
# Required: a default read a misnamed key as an empty list, a 200 that cleared nothing.
|
||||
video_ids: list[str]
|
||||
|
||||
|
||||
@router.post("/clear")
|
||||
|
||||
@@ -48,8 +48,9 @@ def list_tags(user: User = Depends(current_user), db: Session = Depends(get_db))
|
||||
"category": tag.category,
|
||||
"system": tag.user_id is None,
|
||||
"channel_count": count,
|
||||
"subscribed_count": subscribed,
|
||||
}
|
||||
for tag, count in queries.tags_with_counts(db, user)
|
||||
for tag, count, subscribed in queries.tags_with_counts(db, user)
|
||||
]
|
||||
|
||||
|
||||
@@ -107,7 +108,8 @@ def update_tag(
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
raise refused(409, "tag_name_taken", name=tag.name) from None
|
||||
# The name asked for: the rollback has put the tag's old name back on `tag`.
|
||||
raise refused(409, "tag_name_taken", name=name) from None
|
||||
return {
|
||||
"ok": True,
|
||||
"id": tag.id,
|
||||
|
||||
@@ -443,15 +443,20 @@ def _default(s: ConfigSpec):
|
||||
return getattr(settings, s.default_attr)
|
||||
|
||||
|
||||
# Every reason `set_value` refuses with. `refusal.setting_refused` looks each one up in its map, which
|
||||
# the refusal scan holds to exactly these keys: a reason the map lacks would answer a 500.
|
||||
SettingReason = Literal["not_a_number", "too_low", "too_high", "not_a_choice", "no_encryption_key"]
|
||||
|
||||
|
||||
class InvalidSetting(ValueError):
|
||||
"""A value `set_value` refuses, as a reason the UI translates (`errors.refused.setting_<reason>`)
|
||||
plus what the sentence needs — the English message is for logs only. A ValueError still, so a
|
||||
caller that only needs "refused or not" keeps working."""
|
||||
|
||||
def __init__(self, key: str, reason: str, **params: object):
|
||||
def __init__(self, key: str, reason: SettingReason, **params: object):
|
||||
super().__init__(f"{key}: {reason} {params}")
|
||||
self.key = key
|
||||
self.reason = reason
|
||||
self.reason: SettingReason = reason
|
||||
self.params = params
|
||||
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
# `pip install -r requirements.lock && pip check` must pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177); yt-dlp 2026.7.4 -> 2026.8.19
|
||||
# (it marks the clients that need a stream PO token, so the downloader stops fetching into a 403);
|
||||
# certifi 2026.7.22 -> 2026.10.19 (GHSA-6q48-6c7w-9gw2); brotli 1.2.0 -> 1.2.1 (PYSEC-2026-4322);
|
||||
# Authlib 1.7.2 -> 1.9.1 (CVE-2026-96760, CVE-2026-114616).
|
||||
# Authlib 1.7.2 -> 1.9.1 (CVE-2026-96760, CVE-2026-114616); joserfc 1.7.4 -> 1.7.6 (CVE-2026-119283, CVE-2026-119284).
|
||||
APScheduler==3.11.3
|
||||
Authlib==1.9.1
|
||||
Mako==1.4.1
|
||||
@@ -39,7 +39,7 @@ httptools==0.8.0
|
||||
httpx==0.28.1
|
||||
idna==3.18
|
||||
itsdangerous==2.2.0
|
||||
joserfc==1.7.4
|
||||
joserfc==1.7.6
|
||||
markdown-it-py==4.2.0
|
||||
mdurl==0.1.2
|
||||
mutagen==1.48.1
|
||||
|
||||
@@ -0,0 +1,738 @@
|
||||
"""Every refusal the backend can answer, read from its syntax tree: each `{"code": …, "reason": …}` detail,
|
||||
with every reason its code can be sent with.
|
||||
|
||||
The frontend's contract (`i18n/backendIds.contract.test.ts`) runs this and requires a sentence for each
|
||||
code and reason in both languages. A scan that tries to follow every shape opens a silent gap with each one
|
||||
it learns, and following how a value flows through Python (variables, scopes, aliases, a dict changed in
|
||||
place) never closes. So the CODE keeps to a narrow form, and this reads a CLOSED list of it, reporting
|
||||
everything else, never guessing:
|
||||
|
||||
- where a detail stands: where it is sent — `HTTPException`'s second argument or `detail=`, the `"detail"`
|
||||
value of a `JSONResponse`'s content dict — through the arms of `a if c else b`, or as what a module-level
|
||||
factory returns when every return of it is a detail and every reference to it is a call sent as one. A
|
||||
detail held in a variable is not followed;
|
||||
- the code: one literal. A `**` spread, or a key that is not a literal, stands BEFORE a literal `code` and
|
||||
`reason`, both of which the detail then carries: the later key wins, so no spread can change them;
|
||||
- a reason: a string literal (`None` sends none: the site counts as reasonless); both arms of
|
||||
`a if c else b`; a function's local bound only by plain `x = …`; a module constant reached as `mod.NAME`;
|
||||
a call's `Literal` return annotation;
|
||||
- a parameter typed with a `Literal` (basedpyright holds every value passed in), or a `str` one of a
|
||||
module-level helper (`refused`, `not_found`), followed to every call, each of which must pass it by
|
||||
position or by name; a parameter its body binds again is unread;
|
||||
- an attribute: `e.x` of the exception an `except C as e` catches, or of a parameter typed `C`, when a
|
||||
method of `C` declares `self.x: <Literal>`;
|
||||
- a map lookup (`_SETTING_REASONS[exc.reason]`): the map's values, and its keys must be exactly what the
|
||||
index can be.
|
||||
|
||||
A name bound twice at module level, inside a module-level `if`/`try`/loop, named by a `global`, or imported
|
||||
from two places is not looked up; nor is a local a lambda or a comprehension binds, or one a `nonlocal` or
|
||||
`global` names. A module constant or map that anything stores into, deletes from or (a map) uses but by
|
||||
`map[k]` or a read method is unread. A reference named like a followed helper that the scan cannot resolve
|
||||
(`import app.x`, a relative import, an app module that does not bind it) is unread. Anything else is UNREAD,
|
||||
or an unread code, and the contract fails on it unless no screen shows that code. Stdlib only: it runs on
|
||||
the host, beside the frontend's tests (`python -I backend/tests/refusal_sites.py` → JSON).
|
||||
"""
|
||||
|
||||
import ast
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
APP = pathlib.Path(__file__).resolve().parents[1] / "app"
|
||||
|
||||
# Files whose `"code"` dicts are data, not refusals: a subtitle track's status payload, TMDB country codes.
|
||||
NOT_REFUSALS = {
|
||||
"plex/subtitles.py": "a subtitle track's status payload (its own `code`)",
|
||||
"plex/tmdb.py": "a production country's ISO code",
|
||||
}
|
||||
|
||||
# What a map may be used by besides `map[k]`: a method that only reads it.
|
||||
READS = {"get", "items", "keys", "values"}
|
||||
|
||||
Node = ast.AST
|
||||
FUNCS = (ast.FunctionDef, ast.AsyncFunctionDef)
|
||||
SCOPES = (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda, ast.ClassDef)
|
||||
COMPREHENSIONS = (ast.ListComp, ast.SetComp, ast.DictComp, ast.GeneratorExp)
|
||||
COMPOUND = (
|
||||
ast.If,
|
||||
ast.Try,
|
||||
ast.TryStar,
|
||||
ast.For,
|
||||
ast.AsyncFor,
|
||||
ast.While,
|
||||
ast.With,
|
||||
ast.AsyncWith,
|
||||
ast.Match,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Res:
|
||||
values: set[str] = field(default_factory=set)
|
||||
unread: list[str] = field(default_factory=list)
|
||||
none: bool = False # an arm sends no reason
|
||||
|
||||
def __or__(self, other: "Res") -> "Res":
|
||||
return Res(self.values | other.values, self.unread + other.unread, self.none or other.none)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Module:
|
||||
path: str # relative to app/, posix
|
||||
name: str # dotted, `app.…`
|
||||
tree: ast.Module
|
||||
parents: dict[Node, Node]
|
||||
# local name → (module, symbol or None for the module)
|
||||
imports: dict[str, tuple[str, str | None]]
|
||||
top: dict[str, Node] # module-level bindings: def, class, or the assignment
|
||||
ambiguous: set[
|
||||
str
|
||||
] # bound more than once, conditionally, by a `global`, or imported from two places
|
||||
slots: set[int] # the ids of the nodes sent as a detail
|
||||
|
||||
|
||||
def _own_scope(node: Node) -> Iterator[Node]:
|
||||
"""`node` and what it holds in its own scope: a nested def, class or lambda is itself, not its body."""
|
||||
stack = [node]
|
||||
while stack:
|
||||
n = stack.pop()
|
||||
yield n
|
||||
if n is not node and isinstance(n, SCOPES):
|
||||
continue
|
||||
stack.extend(ast.iter_child_nodes(n))
|
||||
|
||||
|
||||
def _stored(target: Node) -> set[str]:
|
||||
"""The names an assignment target binds, not the ones it reads (`LABELS[k] = v` binds none)."""
|
||||
return {
|
||||
n.id for n in ast.walk(target) if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)
|
||||
}
|
||||
|
||||
|
||||
def _binds(node: Node) -> set[str]:
|
||||
"""The names one node binds in the scope it sits in."""
|
||||
if isinstance(node, ast.Assign):
|
||||
return set().union(*(_stored(t) for t in node.targets))
|
||||
if isinstance(node, (ast.AnnAssign, ast.AugAssign, ast.For, ast.AsyncFor, ast.NamedExpr)):
|
||||
return _stored(node.target)
|
||||
if isinstance(node, (ast.With, ast.AsyncWith)):
|
||||
return set().union(*(_stored(i.optional_vars) for i in node.items if i.optional_vars))
|
||||
if isinstance(node, (ast.Import, ast.ImportFrom)):
|
||||
return {(a.asname or a.name).split(".")[0] for a in node.names}
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
return {node.name}
|
||||
if isinstance(node, (ast.MatchAs, ast.MatchStar, ast.ExceptHandler)) and node.name:
|
||||
return {node.name}
|
||||
return set()
|
||||
|
||||
|
||||
def _sender(call: ast.Call) -> str | None:
|
||||
return getattr(call.func, "id", None) or getattr(call.func, "attr", None)
|
||||
|
||||
|
||||
def detail_slots(tree: Node) -> Iterator[tuple[ast.Call, Node | None]]:
|
||||
"""Every place a detail is sent from, with what it holds: `HTTPException`'s second argument or
|
||||
`detail=` (None: sent without one), and the `"detail"` value of a `JSONResponse`'s content dict (a
|
||||
content without one is an answer, not a refusal)."""
|
||||
for node in ast.walk(tree):
|
||||
if not isinstance(node, ast.Call):
|
||||
continue
|
||||
kw = {k.arg: k.value for k in node.keywords}
|
||||
if _sender(node) == "HTTPException":
|
||||
yield node, kw.get("detail", node.args[1] if len(node.args) > 1 else None)
|
||||
elif _sender(node) == "JSONResponse":
|
||||
content = kw.get("content", node.args[0] if node.args else None)
|
||||
if isinstance(content, ast.Dict):
|
||||
for k, v in zip(content.keys, content.values, strict=True):
|
||||
if isinstance(k, ast.Constant) and k.value == "detail":
|
||||
yield node, v
|
||||
|
||||
|
||||
def _code_dict(node: Node) -> bool:
|
||||
"""A `{"code": …}` dict, or a `dict(code=…)` call."""
|
||||
if isinstance(node, ast.Dict):
|
||||
return any(isinstance(k, ast.Constant) and k.value == "code" for k in node.keys)
|
||||
return (
|
||||
isinstance(node, ast.Call)
|
||||
and getattr(node.func, "id", None) == "dict"
|
||||
and any(k.arg == "code" for k in node.keywords)
|
||||
)
|
||||
|
||||
|
||||
def _module_name(path: str) -> str:
|
||||
parts = path.removesuffix(".py").split("/")
|
||||
if parts[-1] == "__init__":
|
||||
parts = parts[:-1]
|
||||
return ".".join(["app", *parts])
|
||||
|
||||
|
||||
def _load(sources: dict[str, str]) -> dict[str, Module]:
|
||||
mods: dict[str, Module] = {}
|
||||
for path, src in sources.items():
|
||||
tree = ast.parse(src)
|
||||
parents = {child: node for node in ast.walk(tree) for child in ast.iter_child_nodes(node)}
|
||||
origins: dict[str, set[tuple[str, str | None]]] = {}
|
||||
ambiguous: set[str] = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
|
||||
for a in node.names:
|
||||
origins.setdefault(a.asname or a.name, set()).add((node.module, a.name))
|
||||
elif isinstance(node, ast.Import):
|
||||
for a in node.names:
|
||||
if a.asname:
|
||||
origins.setdefault(a.asname, set()).add((a.name, None))
|
||||
elif isinstance(node, ast.Global):
|
||||
ambiguous |= set(node.names)
|
||||
top: dict[str, Node] = {}
|
||||
count: dict[str, int] = {}
|
||||
ambiguous |= {name for name, o in origins.items() if len(o) > 1}
|
||||
for stmt in tree.body:
|
||||
if isinstance(stmt, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
top[stmt.name] = stmt
|
||||
elif isinstance(stmt, ast.Assign):
|
||||
for t in stmt.targets:
|
||||
if isinstance(t, ast.Name):
|
||||
top[t.id] = stmt
|
||||
elif isinstance(stmt, ast.AnnAssign) and isinstance(stmt.target, ast.Name):
|
||||
top[stmt.target.id] = stmt
|
||||
for n in [stmt] if isinstance(stmt, SCOPES) else _own_scope(stmt):
|
||||
for name in _binds(n):
|
||||
count[name] = count.get(name, 0) + 1
|
||||
if isinstance(stmt, COMPOUND):
|
||||
ambiguous.add(name)
|
||||
ambiguous |= {name for name, c in count.items() if c > 1}
|
||||
imports = {name: next(iter(o)) for name, o in origins.items() if name not in ambiguous}
|
||||
slots = {id(d) for _, d in detail_slots(tree) if d is not None}
|
||||
name = _module_name(path)
|
||||
mods[name] = Module(path, name, tree, parents, imports, top, ambiguous, slots)
|
||||
return mods
|
||||
|
||||
|
||||
class Scan:
|
||||
def __init__(self, sources: dict[str, str]):
|
||||
self.mods = _load(sources)
|
||||
self.errors: list[str] = []
|
||||
self._following: set[tuple[str, str, str]] = set()
|
||||
self._followed: dict[tuple[str, str, str], Res] = {}
|
||||
self._resolving: set[int] = set()
|
||||
self._factory: dict[int, bool] = {}
|
||||
self._changed: dict[int, bool] = {}
|
||||
|
||||
# --- names ---------------------------------------------------------------------------------------
|
||||
|
||||
def lookup(
|
||||
self, m: Module, name: str, seen: frozenset[tuple[str, str]] = frozenset()
|
||||
) -> tuple[Module, Node] | tuple[str, None] | None:
|
||||
"""What a module-level name is: (its module, its binding), or (a module's name, None)."""
|
||||
if name in m.ambiguous:
|
||||
return None
|
||||
if name in m.top:
|
||||
return m, m.top[name]
|
||||
if name in m.imports:
|
||||
mod, sym = m.imports[name]
|
||||
if sym is None:
|
||||
return (mod, None) if mod in self.mods else None
|
||||
if f"{mod}.{sym}" in self.mods:
|
||||
return f"{mod}.{sym}", None
|
||||
if mod in self.mods and (mod, sym) not in seen:
|
||||
return self.lookup(self.mods[mod], sym, seen | {(mod, sym)})
|
||||
return None
|
||||
|
||||
def lookup_expr(self, m: Module, e: Node) -> tuple[Module, Node] | tuple[str, None] | None:
|
||||
if isinstance(e, ast.Name):
|
||||
return None if self.binder(m, e) is not None else self.lookup(m, e.id)
|
||||
if isinstance(e, ast.Attribute):
|
||||
base = self.lookup_expr(m, e.value)
|
||||
if base and base[1] is None and isinstance(base[0], str):
|
||||
return self.lookup(self.mods[base[0]], e.attr)
|
||||
return None
|
||||
|
||||
def binder(self, m: Module, e: ast.Name) -> Node | None:
|
||||
"""The innermost scope `e` is bound in: a function (a parameter, or a binding in its own body), a
|
||||
lambda (a parameter), a comprehension (its target); None: a module-level name."""
|
||||
node = m.parents.get(e)
|
||||
while node is not None:
|
||||
if isinstance(node, (*FUNCS, ast.Lambda)):
|
||||
a = node.args
|
||||
params = [*a.posonlyargs, *a.args, *a.kwonlyargs, a.vararg, a.kwarg]
|
||||
if e.id in {p.arg for p in params if p}:
|
||||
return node
|
||||
if isinstance(node, FUNCS) and self.assignments(node, e.id):
|
||||
return node
|
||||
elif isinstance(node, COMPREHENSIONS) and any(
|
||||
e.id in _stored(g.target) for g in node.generators
|
||||
):
|
||||
return node
|
||||
node = m.parents.get(node)
|
||||
return None
|
||||
|
||||
def at(self, m: Module, e: Node) -> str:
|
||||
return f"{m.path}:{getattr(e, 'lineno', '?')}: {ast.unparse(e)}"
|
||||
|
||||
def enclosing(self, m: Module, e: Node, kind: type | tuple[type, ...]) -> Node | None:
|
||||
node = m.parents.get(e)
|
||||
while node is not None and not isinstance(node, kind):
|
||||
node = m.parents.get(node)
|
||||
return node
|
||||
|
||||
def assignments(self, fn: Node, name: str) -> list[Node]:
|
||||
"""What binds `name` in this function's own body (not in a nested one)."""
|
||||
return [n for n in _own_scope(fn) if n is not fn and name in _binds(n)]
|
||||
|
||||
def caught_as(self, m: Module, e: Node, name: str) -> ast.ExceptHandler | None:
|
||||
node = m.parents.get(e)
|
||||
while node is not None and not isinstance(node, FUNCS):
|
||||
if isinstance(node, ast.ExceptHandler) and node.name == name:
|
||||
return node
|
||||
node = m.parents.get(node)
|
||||
return None
|
||||
|
||||
def references(
|
||||
self, m: Module, target: Node
|
||||
) -> Iterator[tuple[Module, Node, ast.Call | None, bool]]:
|
||||
"""Every reference to a module-level def or class: the call it makes (None: not a call), and
|
||||
whether it resolved. A reference named like it that the scan cannot resolve is yielded too."""
|
||||
name = getattr(target, "name", None)
|
||||
for cm in self.mods.values():
|
||||
for node in ast.walk(cm.tree):
|
||||
if not isinstance(node, (ast.Name, ast.Attribute)) or not isinstance(
|
||||
node.ctx, ast.Load
|
||||
):
|
||||
continue
|
||||
if (node.id if isinstance(node, ast.Name) else node.attr) != name:
|
||||
continue
|
||||
parent = cm.parents.get(node)
|
||||
if isinstance(parent, ast.Attribute) and parent.value is node:
|
||||
continue
|
||||
found = self.lookup_expr(cm, node)
|
||||
if found:
|
||||
if not (found[0] is m and found[1] is target):
|
||||
continue
|
||||
elif not self.unresolved(cm, node):
|
||||
continue
|
||||
call = parent if isinstance(parent, ast.Call) and parent.func is node else None
|
||||
yield cm, node, call, found is not None
|
||||
|
||||
def unresolved(self, m: Module, node: Node) -> bool:
|
||||
"""A name, or a dotted path, the app could mean that the scan cannot resolve: not a local, nor an
|
||||
outside module's (an attribute of an object the function holds is not one)."""
|
||||
root = node
|
||||
while isinstance(root, ast.Attribute):
|
||||
root = root.value
|
||||
if not isinstance(root, ast.Name) or self.binder(m, root) is not None:
|
||||
return False
|
||||
if root.id in m.ambiguous or (root.id not in m.top and root.id not in m.imports):
|
||||
return True
|
||||
if root.id in m.top:
|
||||
return False
|
||||
if root is node: # imported from an app module that does not bind it where the scan can see
|
||||
mod = m.imports[root.id][0]
|
||||
return (mod == "app" or mod.startswith("app.")) and self.lookup(m, root.id) is None
|
||||
found = self.lookup(m, root.id)
|
||||
return (
|
||||
found is not None and found[1] is None
|
||||
) # an app module whose dotted path went nowhere
|
||||
|
||||
def changed(self, m: Module, stmt: Node, name: str, *, only_read: bool) -> bool:
|
||||
"""Something in the app stores into, deletes or rebinds the module-level `name` that `stmt` binds,
|
||||
or (`only_read`, a map) uses it but by `map[k]` or a method that only reads it."""
|
||||
key = id(stmt) * 2 + only_read
|
||||
if key in self._changed:
|
||||
return self._changed[key]
|
||||
own = stmt.targets if isinstance(stmt, ast.Assign) else [getattr(stmt, "target", None)]
|
||||
out = False
|
||||
for cm in self.mods.values():
|
||||
for node in ast.walk(cm.tree):
|
||||
if out:
|
||||
break
|
||||
if not (
|
||||
isinstance(node, ast.Name)
|
||||
and node.id == name
|
||||
or isinstance(node, ast.Attribute)
|
||||
and node.attr == name
|
||||
) or (cm is m and node in own):
|
||||
continue
|
||||
found = self.lookup_expr(cm, node)
|
||||
if not (found and found[0] is m and found[1] is stmt):
|
||||
continue
|
||||
if not isinstance(node.ctx, ast.Load):
|
||||
out = True
|
||||
elif only_read:
|
||||
p = cm.parents.get(node)
|
||||
out = not (
|
||||
isinstance(p, ast.Subscript)
|
||||
and p.value is node
|
||||
and isinstance(p.ctx, ast.Load)
|
||||
or isinstance(p, ast.Attribute)
|
||||
and p.attr in READS
|
||||
)
|
||||
self._changed[key] = out
|
||||
return out
|
||||
|
||||
# --- types ---------------------------------------------------------------------------------------
|
||||
|
||||
def ann_values(self, m: Module, ann: Node | None, depth: int = 0) -> set[str] | None:
|
||||
"""The values a `Literal[…]` annotation, or an alias of one, allows; None when it is not one."""
|
||||
if isinstance(ann, ast.Subscript) and getattr(ann.value, "id", None) == "Literal":
|
||||
items = ann.slice.elts if isinstance(ann.slice, ast.Tuple) else [ann.slice]
|
||||
if all(isinstance(i, ast.Constant) and isinstance(i.value, str) for i in items):
|
||||
return {i.value for i in items}
|
||||
return None
|
||||
if isinstance(ann, (ast.Name, ast.Attribute)) and depth < 8:
|
||||
found = self.lookup_expr(m, ann)
|
||||
if found and isinstance(found[0], Module) and isinstance(found[1], ast.Assign):
|
||||
return self.ann_values(found[0], found[1].value, depth + 1)
|
||||
return None
|
||||
|
||||
def classes(self, m: Module, e: Node | None) -> list[tuple[Module, ast.ClassDef]] | None:
|
||||
found = self.lookup_expr(m, e) if e is not None else None
|
||||
if found and isinstance(found[0], Module) and isinstance(found[1], ast.ClassDef):
|
||||
return [(found[0], found[1])]
|
||||
return None
|
||||
|
||||
def typed(self, m: Module, at: Node, name: str) -> list[tuple[Module, ast.ClassDef]] | None:
|
||||
"""The class `name` holds where `at` sits: the one an `except C as name` catches, or a parameter's
|
||||
annotation (a parameter its body binds again holds no known class)."""
|
||||
handler = self.caught_as(m, at, name)
|
||||
if handler is not None:
|
||||
return self.classes(m, handler.type)
|
||||
fn = self.enclosing(m, at, FUNCS)
|
||||
if fn is None:
|
||||
return None
|
||||
params = {a.arg: a for a in [*fn.args.posonlyargs, *fn.args.args, *fn.args.kwonlyargs]}
|
||||
if name in params and not self.assignments(fn, name):
|
||||
return self.classes(m, params[name].annotation)
|
||||
return None
|
||||
|
||||
def attr_values(self, cm: Module, cls: ast.ClassDef, attr: str) -> set[str] | None:
|
||||
"""An attribute's `Literal` values, from `self.attr: T` in a method of the class."""
|
||||
for node in ast.walk(cls):
|
||||
if (
|
||||
isinstance(node, ast.AnnAssign)
|
||||
and isinstance(node.target, ast.Attribute)
|
||||
and node.target.attr == attr
|
||||
and getattr(node.target.value, "id", None) == "self"
|
||||
):
|
||||
return self.ann_values(cm, node.annotation)
|
||||
return None
|
||||
|
||||
# --- reasons -------------------------------------------------------------------------------------
|
||||
|
||||
def resolve(self, m: Module, e: Node) -> Res:
|
||||
if isinstance(e, ast.Constant):
|
||||
if e.value is None:
|
||||
return Res(none=True)
|
||||
if isinstance(e.value, str):
|
||||
return Res({e.value})
|
||||
if isinstance(e, ast.IfExp):
|
||||
return self.resolve(m, e.body) | self.resolve(m, e.orelse)
|
||||
if isinstance(e, ast.Name):
|
||||
return self.resolve_name(m, e)
|
||||
if isinstance(e, ast.Attribute):
|
||||
return self.resolve_attr(m, e)
|
||||
if isinstance(e, ast.Call):
|
||||
found = self.lookup_expr(m, e.func)
|
||||
if found and isinstance(found[0], Module) and isinstance(found[1], FUNCS):
|
||||
v = self.ann_values(found[0], found[1].returns)
|
||||
if v is not None:
|
||||
return Res(v)
|
||||
if isinstance(e, ast.Subscript):
|
||||
return self.resolve_lookup(m, e)
|
||||
return Res(unread=[self.at(m, e)])
|
||||
|
||||
def resolve_name(self, m: Module, e: ast.Name) -> Res:
|
||||
if self.caught_as(m, e, e.id) is not None:
|
||||
return Res(unread=[self.at(m, e)]) # the exception itself, not a reason
|
||||
fn = self.binder(m, e)
|
||||
if fn is None:
|
||||
return Res(unread=[self.at(m, e)])
|
||||
if not isinstance(fn, FUNCS):
|
||||
return Res(unread=[f"{self.at(m, e)} (bound by a lambda or a comprehension)"])
|
||||
if any(isinstance(n, (ast.Nonlocal, ast.Global)) and e.id in n.names for n in ast.walk(fn)):
|
||||
return Res(unread=[f"{self.at(m, e)} (named by a nonlocal or a global)"])
|
||||
a = fn.args
|
||||
params = {p.arg: p for p in [*a.posonlyargs, *a.args, *a.kwonlyargs]}
|
||||
assigned = self.assignments(fn, e.id)
|
||||
if e.id in params:
|
||||
if assigned:
|
||||
return Res(unread=[f"{self.at(m, e)} (a parameter its body binds again)"])
|
||||
return self.resolve_param(m, fn, params[e.id], e)
|
||||
out = Res(unread=[] if assigned else [self.at(m, e)]) # `*args`, `**kwargs`
|
||||
for s in assigned:
|
||||
if not (isinstance(s, ast.Assign) and all(isinstance(t, ast.Name) for t in s.targets)):
|
||||
out = out | Res(unread=[self.at(m, e)])
|
||||
elif id(s) in self._resolving:
|
||||
out = out | Res(unread=[f"{self.at(m, e)} (reached again while it is resolved)"])
|
||||
else:
|
||||
self._resolving.add(id(s))
|
||||
out = out | self.resolve(m, s.value)
|
||||
self._resolving.discard(id(s))
|
||||
return out
|
||||
|
||||
def resolve_param(self, m: Module, fn: Node, param: ast.arg, e: Node) -> Res:
|
||||
v = self.ann_values(m, param.annotation)
|
||||
if v is not None:
|
||||
return Res(v)
|
||||
if param.annotation is not None and getattr(param.annotation, "id", None) != "str":
|
||||
return Res(unread=[self.at(m, e)])
|
||||
return self.follow(m, fn, param)
|
||||
|
||||
def follow(self, m: Module, fn: Node, param: ast.arg) -> Res:
|
||||
"""A `str` parameter of a helper: what every call of the helper passes it."""
|
||||
assert isinstance(fn, FUNCS)
|
||||
key = (m.name, fn.name, param.arg)
|
||||
if key in self._followed:
|
||||
return self._followed[key]
|
||||
where = f"{m.path}:{fn.lineno}: {fn.name}({param.arg})"
|
||||
if key in self._following:
|
||||
return Res(unread=[f"{where} reached again while it is followed"])
|
||||
if m.top.get(fn.name) is not fn or fn.name in m.ambiguous:
|
||||
return Res(unread=[f"{where} is not a module-level helper"])
|
||||
self._following.add(key)
|
||||
positional = [a.arg for a in [*fn.args.posonlyargs, *fn.args.args]]
|
||||
by_name = param not in fn.args.posonlyargs
|
||||
out = Res()
|
||||
for cm, node, call, resolved in self.references(m, fn):
|
||||
if not resolved:
|
||||
out = out | Res(
|
||||
unread=[f"{self.at(cm, node)} (a reference the scan cannot resolve)"]
|
||||
)
|
||||
continue
|
||||
if call is None:
|
||||
out = out | Res(unread=[f"{self.at(cm, node)} (the helper used as a value)"])
|
||||
continue
|
||||
kw = {k.arg: k.value for k in call.keywords}
|
||||
given = param.arg in positional and positional.index(param.arg) < len(call.args)
|
||||
if by_name and param.arg in kw:
|
||||
out = out | self.resolve(cm, kw[param.arg])
|
||||
elif any(isinstance(a, ast.Starred) for a in call.args):
|
||||
out = out | Res(unread=[self.at(cm, call)])
|
||||
elif given:
|
||||
out = out | self.resolve(cm, call.args[positional.index(param.arg)])
|
||||
else:
|
||||
out = out | Res(unread=[f"{self.at(cm, call)} (does not pass {param.arg})"])
|
||||
self._followed[key] = out # once followed, its key answers from here: never reached again
|
||||
return out
|
||||
|
||||
def resolve_attr(self, m: Module, e: ast.Attribute) -> Res:
|
||||
base = e.value
|
||||
if isinstance(base, ast.Name):
|
||||
typed = self.typed(m, e, base.id)
|
||||
if typed is not None:
|
||||
out = Res()
|
||||
for cm, cls in typed:
|
||||
v = self.attr_values(cm, cls, e.attr)
|
||||
out = out | (
|
||||
Res(v)
|
||||
if v is not None
|
||||
else Res(
|
||||
unread=[f"{self.at(m, e)} ({cls.name}.{e.attr} has no Literal type)"]
|
||||
)
|
||||
)
|
||||
return out
|
||||
found = self.lookup_expr(m, e)
|
||||
if (
|
||||
found
|
||||
and isinstance(found[0], Module)
|
||||
and isinstance(found[1], (ast.Assign, ast.AnnAssign))
|
||||
and isinstance(found[1].value, ast.Constant)
|
||||
):
|
||||
if self.changed(found[0], found[1], e.attr, only_read=False):
|
||||
return Res(unread=[f"{self.at(m, e)} (bound again or deleted elsewhere)"])
|
||||
return self.resolve(found[0], found[1].value)
|
||||
return Res(unread=[self.at(m, e)])
|
||||
|
||||
def resolve_lookup(self, m: Module, e: ast.Subscript) -> Res:
|
||||
found = self.lookup_expr(m, e.value)
|
||||
if not (
|
||||
found
|
||||
and isinstance(found[0], Module)
|
||||
and isinstance(found[1], (ast.Assign, ast.AnnAssign))
|
||||
):
|
||||
return Res(unread=[self.at(m, e)])
|
||||
table = found[1].value
|
||||
if not isinstance(table, ast.Dict) or not all(
|
||||
isinstance(k, ast.Constant) and isinstance(k.value, str) for k in table.keys
|
||||
):
|
||||
return Res(unread=[self.at(m, e)])
|
||||
name = e.value.attr if isinstance(e.value, ast.Attribute) else getattr(e.value, "id", "")
|
||||
if self.changed(found[0], found[1], name, only_read=True):
|
||||
return Res(unread=[f"{self.at(m, e)} (the map is changed after it is built)"])
|
||||
out = Res()
|
||||
for v in table.values:
|
||||
out = out | self.resolve(found[0], v)
|
||||
index = self.resolve(m, e.slice)
|
||||
keys = {k.value for k in table.keys if isinstance(k, ast.Constant)}
|
||||
if index.unread or index.none or index.values != keys:
|
||||
self.errors.append(
|
||||
f"{self.at(m, e)}: the map's keys {sorted(keys)} are not what its index can be "
|
||||
f"{sorted(index.values)}{' + unread ' + str(index.unread) if index.unread else ''}"
|
||||
)
|
||||
return out
|
||||
|
||||
# --- sites ---------------------------------------------------------------------------------------
|
||||
|
||||
def is_detail(self, node: Node | None) -> bool:
|
||||
if isinstance(node, ast.IfExp):
|
||||
return self.is_detail(node.body) and self.is_detail(node.orelse)
|
||||
return _code_dict(node)
|
||||
|
||||
def factory(self, m: Module, fn: Node) -> bool:
|
||||
"""A module-level function each return of which is a detail, and each reference to which is a
|
||||
call sent as one (`quota_detail`)."""
|
||||
assert isinstance(fn, FUNCS)
|
||||
if id(fn) in self._factory:
|
||||
return self._factory[id(fn)]
|
||||
returns = [n for n in _own_scope(fn) if isinstance(n, ast.Return)]
|
||||
ok = (
|
||||
m.top.get(fn.name) is fn # a name bound twice: its calls are unresolved below
|
||||
and bool(returns)
|
||||
and all(self.is_detail(r.value) for r in returns)
|
||||
)
|
||||
for cm, _, call, resolved in self.references(m, fn) if ok else []:
|
||||
ok = resolved and id(call) in cm.slots # a bare reference is no call: never sent
|
||||
if not ok:
|
||||
break
|
||||
self._factory[id(fn)] = ok
|
||||
return ok
|
||||
|
||||
def placed(self, m: Module, d: Node) -> bool:
|
||||
"""`d` is sent as a detail: from a sender's slot, through the arms of `a if c else b`, or as what a
|
||||
factory returns."""
|
||||
holder, p = d, m.parents.get(d)
|
||||
while isinstance(p, ast.IfExp) and holder is not p.test:
|
||||
holder, p = p, m.parents.get(p)
|
||||
if id(holder) in m.slots:
|
||||
return True
|
||||
fn = self.enclosing(m, p, FUNCS) if isinstance(p, ast.Return) else None
|
||||
return fn is not None and self.factory(m, fn)
|
||||
|
||||
def sayable(self, m: Module, d: Node | None) -> bool:
|
||||
"""What a slot holds is a detail the client can say: a code dict, or a factory's call."""
|
||||
if isinstance(d, ast.IfExp):
|
||||
return self.sayable(m, d.body) and self.sayable(m, d.orelse)
|
||||
if _code_dict(d):
|
||||
return True
|
||||
found = self.lookup_expr(m, d.func) if isinstance(d, ast.Call) else None
|
||||
return bool(
|
||||
found
|
||||
and isinstance(found[0], Module)
|
||||
and isinstance(found[1], FUNCS)
|
||||
and self.factory(found[0], found[1])
|
||||
)
|
||||
|
||||
def entries(self, d: Node) -> list[tuple[str | None, Node]]:
|
||||
"""A dict's (key, value) pairs: None for a `**` spread, "?" for a key that is not a literal."""
|
||||
if isinstance(d, ast.Dict):
|
||||
return [
|
||||
(
|
||||
k.value
|
||||
if isinstance(k, ast.Constant) and isinstance(k.value, str)
|
||||
else None
|
||||
if k is None
|
||||
else "?",
|
||||
v,
|
||||
)
|
||||
for k, v in zip(d.keys, d.values, strict=True)
|
||||
]
|
||||
assert isinstance(d, ast.Call)
|
||||
return [(None, a) for a in d.args] + [(k.arg, k.value) for k in d.keywords]
|
||||
|
||||
def site(self, m: Module, d: Node, out: dict) -> None:
|
||||
items = self.entries(d)
|
||||
code = Res()
|
||||
for v in (v for k, v in items if k == "code"):
|
||||
code = code | self.resolve(m, v)
|
||||
problems = [
|
||||
f"{self.at(m, v)} (a spread in dict())"
|
||||
for k, v in items
|
||||
if k is None and isinstance(d, ast.Call)
|
||||
] + [
|
||||
f"{self.at(m, v)} (a spread or a computed key the code and the reason do not follow)"
|
||||
for i, (k, v) in enumerate(items)
|
||||
if k in (None, "?") and not {"code", "reason"} <= {k2 for k2, _ in items[i + 1 :]}
|
||||
]
|
||||
if not self.placed(m, d):
|
||||
problems.append(f"{self.at(m, d)} (a detail put where the scan cannot follow it)")
|
||||
if code.unread or code.none or len(code.values) != 1 or problems:
|
||||
out["unreadCodes"] += problems or [self.at(m, d)]
|
||||
return
|
||||
entry = out["codes"].setdefault(
|
||||
next(iter(code.values)), {"reasons": set(), "unread": [], "sites": 0, "reasonless": []}
|
||||
)
|
||||
entry["sites"] += 1
|
||||
res, has_reason = Res(), False
|
||||
for k, v in items:
|
||||
if k == "reason":
|
||||
res, has_reason = res | self.resolve(m, v), True
|
||||
if not has_reason or res.none:
|
||||
entry["reasonless"].append(f"{m.path}:{d.lineno}")
|
||||
entry["reasons"] |= res.values
|
||||
entry["unread"] += res.unread
|
||||
|
||||
def other_details(self) -> list[tuple[str, int, str | None]]:
|
||||
"""Each slot that sends what the client cannot say — nothing, or anything but a code dict or a
|
||||
factory's call: (its file, its line, the function it is sent from)."""
|
||||
out = []
|
||||
for m in self.mods.values():
|
||||
for call, d in detail_slots(m.tree):
|
||||
if self.sayable(m, d):
|
||||
continue
|
||||
fn = self.enclosing(m, call, FUNCS)
|
||||
out.append((m.path, call.lineno, getattr(fn, "name", None)))
|
||||
return sorted(out)
|
||||
|
||||
def run(self) -> dict:
|
||||
out: dict = {
|
||||
"codes": {},
|
||||
"unreadCodes": [],
|
||||
"errors": self.errors,
|
||||
"excluded": {},
|
||||
"aliases": {},
|
||||
"notRefusals": sorted(NOT_REFUSALS),
|
||||
}
|
||||
for m in self.mods.values():
|
||||
for name, stmt in m.top.items():
|
||||
if (
|
||||
isinstance(stmt, ast.Assign)
|
||||
and isinstance(stmt.value, ast.Subscript)
|
||||
and (v := self.ann_values(m, stmt.value))
|
||||
):
|
||||
out["aliases"].setdefault(m.path, {})[name] = sorted(v)
|
||||
for node in ast.walk(m.tree):
|
||||
if not _code_dict(node):
|
||||
continue
|
||||
if m.path in NOT_REFUSALS:
|
||||
out["excluded"][m.path] = out["excluded"].get(m.path, 0) + 1
|
||||
continue
|
||||
self.site(m, node, out)
|
||||
for entry in out["codes"].values():
|
||||
entry["reasons"] = sorted(entry["reasons"])
|
||||
return out
|
||||
|
||||
|
||||
def scan(sources: dict[str, str]) -> dict:
|
||||
return Scan(sources).run()
|
||||
|
||||
|
||||
def app_sources(app: pathlib.Path = APP) -> dict[str, str]:
|
||||
return {
|
||||
p.relative_to(app).as_posix(): p.read_text(encoding="utf-8")
|
||||
for p in sorted(app.rglob("*.py"))
|
||||
if "__pycache__" not in p.parts
|
||||
}
|
||||
|
||||
|
||||
def scan_app(app: pathlib.Path = APP) -> dict:
|
||||
return scan(app_sources(app))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
json.dump(
|
||||
scan_app(pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else APP),
|
||||
sys.stdout,
|
||||
indent=1,
|
||||
sort_keys=True,
|
||||
)
|
||||
@@ -125,3 +125,109 @@ def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_u
|
||||
r = c.post("/api/playlists/sync-youtube")
|
||||
assert r.status_code == 429
|
||||
assert r.json()["detail"] == {"code": "sync_cooldown"}
|
||||
|
||||
|
||||
def _pushable_list(db, make_user, monkeypatch, plroutes):
|
||||
"""A local (never exported) list and a user with the write scope; the merge itself is stubbed —
|
||||
the limiter is under test, not the YouTube round trip."""
|
||||
from app import auth
|
||||
from app.reader.models import Playlist
|
||||
|
||||
user = make_user()
|
||||
pl = Playlist(user_id=user.id, name="Local", source="local")
|
||||
db.add(pl)
|
||||
db.commit()
|
||||
monkeypatch.setattr(auth, "has_write_scope", lambda u: True)
|
||||
monkeypatch.setattr(plroutes, "plan_merge", lambda d, u, p, live=None: {"units_estimate": 0})
|
||||
monkeypatch.setattr(plroutes, "merge_playlist", lambda d, u, p, live=None: {"added": 0})
|
||||
return user, pl
|
||||
|
||||
|
||||
def test_the_push_side_goes_on_cooldown_after_its_allowance(db, as_user, make_user, monkeypatch):
|
||||
# SEC-18: push-plan reads the live list and push writes it, both against the instance-wide
|
||||
# YouTube budget. A press in the UI is one plan + one push, so the allowance is generous — but a
|
||||
# scripted loop stops at it, with a reason the UI can say.
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
c = as_user(user)
|
||||
allowance = plroutes.PUSH_ALLOWANCE
|
||||
codes = [c.get(f"/api/playlists/{pl.id}/push-plan").status_code for _ in range(allowance)]
|
||||
assert codes == [200] * allowance
|
||||
r = c.get(f"/api/playlists/{pl.id}/push-plan")
|
||||
assert r.status_code == 429
|
||||
assert r.json()["detail"] == {"code": "playlist", "reason": "push_cooldown"}
|
||||
|
||||
|
||||
def test_plan_and_push_draw_on_one_allowance(db, as_user, make_user, monkeypatch):
|
||||
# The asymmetric fix again, avoided up front: a limit on one of the two routes leaves the same
|
||||
# drain open next door.
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
c = as_user(user)
|
||||
for i in range(plroutes.PUSH_ALLOWANCE):
|
||||
if i % 2:
|
||||
r = c.get(f"/api/playlists/{pl.id}/push-plan")
|
||||
else:
|
||||
r = c.post(f"/api/playlists/{pl.id}/push")
|
||||
assert r.status_code == 200, (i, r.json())
|
||||
r = c.post(f"/api/playlists/{pl.id}/push")
|
||||
assert r.status_code == 429
|
||||
assert r.json()["detail"] == {"code": "playlist", "reason": "push_cooldown"}
|
||||
|
||||
|
||||
def test_a_refused_push_spends_no_allowance(db, as_user, make_user, monkeypatch):
|
||||
# The allowance counts pushes that run: Watch later (never pushable) and a list another sync
|
||||
# holds are refused before any YouTube call, so they must not leave the next real press at 429.
|
||||
from contextlib import contextmanager
|
||||
|
||||
from app.reader.models import Playlist
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
wl = Playlist(user_id=user.id, name="Watch later", kind="watch_later")
|
||||
db.add(wl)
|
||||
db.commit()
|
||||
c = as_user(user)
|
||||
over = plroutes.PUSH_ALLOWANCE + 1
|
||||
assert {c.get(f"/api/playlists/{wl.id}/push-plan").status_code for _ in range(over)} == {400}
|
||||
assert {c.post(f"/api/playlists/{wl.id}/push").status_code for _ in range(over)} == {400}
|
||||
|
||||
@contextmanager
|
||||
def busy(_db, _id):
|
||||
yield False
|
||||
|
||||
with monkeypatch.context() as m:
|
||||
m.setattr(plroutes, "sync_claim", busy)
|
||||
assert {c.post(f"/api/playlists/{pl.id}/push").status_code for _ in range(over)} == {409}
|
||||
assert c.post(f"/api/playlists/{pl.id}/push").status_code == 200
|
||||
|
||||
|
||||
def test_a_mirrored_youtube_list_is_pushable(db, as_user, make_user, monkeypatch):
|
||||
# A mirror of the user's own YouTube list takes local edits (`dirty`) and syncs them back; only
|
||||
# Watch later is refused.
|
||||
from app.reader.models import Playlist
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user, _ = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
mirror = Playlist(
|
||||
user_id=user.id, name="Mine", source="youtube", yt_playlist_id="PLx", dirty=True
|
||||
)
|
||||
db.add(mirror)
|
||||
db.commit()
|
||||
assert as_user(user).get(f"/api/playlists/{mirror.id}/push-plan").status_code == 200
|
||||
|
||||
|
||||
def test_the_push_cooldown_is_per_user(db, as_user, make_user, monkeypatch):
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
a, pa = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
b, pb = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
ca = as_user(a)
|
||||
codes = [
|
||||
ca.get(f"/api/playlists/{pa.id}/push-plan").status_code
|
||||
for _ in range(plroutes.PUSH_ALLOWANCE + 1)
|
||||
]
|
||||
assert codes == [200] * plroutes.PUSH_ALLOWANCE + [429]
|
||||
assert as_user(b).get(f"/api/playlists/{pb.id}/push-plan").status_code == 200
|
||||
|
||||
@@ -72,3 +72,23 @@ def test_the_sync_status_counts_the_callers_subscriptions(db, make_user, as_user
|
||||
assert (body["subscriptions"], body["channels_total"]) == (1, 1)
|
||||
mine = as_user(user).get("/api/sync/my-status").json()
|
||||
assert (mine["channels_total"], mine["channels_recent_pending"]) == (1, 1)
|
||||
|
||||
|
||||
def test_a_channel_lists_when_it_entered_this_users_list_not_when_the_channel_did(
|
||||
db, make_user, as_user
|
||||
):
|
||||
from datetime import datetime, timezone
|
||||
|
||||
user = make_user()
|
||||
db.add(Channel(id="UCa", title="A", created_at=datetime(2026, 6, 10, tzinfo=timezone.utc)))
|
||||
db.flush()
|
||||
db.add(
|
||||
Subscription(
|
||||
user_id=user.id,
|
||||
channel_id="UCa",
|
||||
created_at=datetime(2026, 10, 12, 8, 30, tzinfo=timezone.utc),
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
listed = as_user(user).get("/api/channels").json()
|
||||
assert [(c["id"], c["added_at"]) for c in listed] == [("UCa", "2026-10-12T08:30:00+00:00")]
|
||||
|
||||
@@ -531,6 +531,9 @@ def test_a_youtube_clip_hands_the_range_to_yt_dlp(monkeypatch, tmp_path):
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *_a):
|
||||
return False
|
||||
|
||||
@@ -707,6 +710,160 @@ def test_a_clip_larger_than_what_its_owner_has_left_is_refused_and_not_placed(
|
||||
)
|
||||
|
||||
|
||||
# --- a GIF clip ------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_gif_clip_spec_names_its_size_and_is_another_clip_than_the_video():
|
||||
mp4 = clipmod.normalize("youtube", "abcdefghijk", 10, 20)
|
||||
gif = clipmod.normalize("youtube", "abcdefghijk", 10, 20, gif_side=320)
|
||||
assert gif == {**mp4, "gif": {"side": 320}}
|
||||
# A video clip's spec (and so its cache key) is the one it always was.
|
||||
assert "gif" not in mp4
|
||||
assert clipmod.sig(gif) != clipmod.sig(mp4)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("kw", "reason", "params"),
|
||||
[
|
||||
({"end_s": 40.001, "gif_side": 480}, "gif_too_long", {"limit_s": 30}),
|
||||
({"end_s": 20, "gif_side": 500}, "bad_gif", {}),
|
||||
({"end_s": 20, "gif_side": True}, "bad_gif", {}),
|
||||
],
|
||||
)
|
||||
def test_a_gif_clip_it_cannot_make_is_refused_with_its_reason(kw, reason, params):
|
||||
with pytest.raises(clipmod.ClipError) as e:
|
||||
clipmod.normalize("youtube", "abcdefghijk", 10, **kw)
|
||||
assert (e.value.reason, e.value.params) == (reason, params)
|
||||
|
||||
|
||||
def test_the_thirty_second_gif_cap_itself_is_allowed():
|
||||
assert clipmod.duration(clipmod.normalize("plex", "42", 0, 30, gif_side=480)) == 30
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("start", "end", "kw"),
|
||||
[
|
||||
# Each a range exactly at a cap whose float difference lands a hair past it.
|
||||
(30.529, 60.529, {"gif_side": 480}),
|
||||
(424.005, 1024.005, {}),
|
||||
(0.001, 1.001, {}),
|
||||
],
|
||||
)
|
||||
def test_a_range_exactly_at_a_cap_is_judged_in_milliseconds(start, end, kw):
|
||||
assert end - start not in (1, 30, 600) # the float difference is off the cap itself
|
||||
assert round((end - start) * 1000) in (1000, 30000, 600000)
|
||||
clipmod.normalize("youtube", "abcdefghijk", start, end, **kw)
|
||||
|
||||
|
||||
def test_a_gif_clip_keeps_no_audio_track_so_the_track_does_not_make_another_gif():
|
||||
first = clipmod.normalize("plex", "42", 10, 20, audio_track=0, gif_side=480)
|
||||
other = clipmod.normalize("plex", "42", 10, 20, audio_track=1, gif_side=480)
|
||||
assert "audio_track" not in first
|
||||
assert clipmod.sig(first) == clipmod.sig(other)
|
||||
# A video clip keeps the track it was heard on.
|
||||
assert clipmod.normalize("plex", "42", 10, 20, audio_track=1)["audio_track"] == 1
|
||||
|
||||
|
||||
def test_a_gif_clip_is_queued_as_a_file_and_too_long_says_its_own_cap(
|
||||
db, make_user, as_user, yt_video
|
||||
):
|
||||
c = as_user(make_user())
|
||||
r = _clip(c, source="youtube", ref=yt_video, gif_side=480)
|
||||
assert r.status_code == 200, r.text
|
||||
job = r.json()
|
||||
assert job["edit_spec"]["gif"] == {"side": 480}
|
||||
# A GIF is a picture: no player, no editor; it previews as an image.
|
||||
assert db.get(MediaAsset, job["asset"]["id"]).media_kind == "file"
|
||||
r = _clip(c, source="youtube", ref=yt_video, gif_side=480, end_s=45)
|
||||
assert r.status_code == 422
|
||||
assert r.json()["detail"] == {"code": "clip", "reason": "gif_too_long", "limit_s": 30}
|
||||
|
||||
|
||||
def _gif_bytes(w: int, h: int, n: int = 0) -> bytes:
|
||||
return b"GIF89a" + w.to_bytes(2, "little") + h.to_bytes(2, "little") + b"\x00" * (1 + n)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gif_worker(worker, monkeypatch):
|
||||
"""The worker stand-in, writing a GIF header where a GIF is asked for; the still is a sidecar."""
|
||||
seen = worker
|
||||
seen["gif_extra"] = 0
|
||||
|
||||
def ffmpeg(cmd, *_a, **_k):
|
||||
seen["cmd"] = cmd
|
||||
out = Path(cmd[-1])
|
||||
out.write_bytes(_gif_bytes(270, 480, seen["gif_extra"]) if out.suffix == ".gif" else b"x")
|
||||
|
||||
def still(path):
|
||||
dest = path.with_suffix(".jpg")
|
||||
dest.write_bytes(b"\xff\xd8\xff")
|
||||
return dest
|
||||
|
||||
def phone_safe(*_a, **_k):
|
||||
raise AssertionError("a GIF is not a video: no phone-safe pass")
|
||||
|
||||
monkeypatch.setattr(pipeline.editmod, "run_ffmpeg", ffmpeg)
|
||||
monkeypatch.setattr(pipeline.gifmod, "ensure_still", still)
|
||||
monkeypatch.setattr(pipeline, "_ensure_phone_safe", phone_safe)
|
||||
monkeypatch.setattr(probemod, "probe_media", lambda _p: (None, None, None, None, None))
|
||||
return seen
|
||||
|
||||
|
||||
def test_a_gif_clip_is_cut_through_the_gif_command_and_stored_as_an_image_file(
|
||||
db, make_user, as_user, plex_episode, gif_worker
|
||||
):
|
||||
gif_worker["probe"] = {
|
||||
"streams": [{"codec_type": "video", "color_transfer": "smpte2084"}, {"codec_type": "audio"}]
|
||||
}
|
||||
job = _clip(as_user(make_user()), source="plex", ref="777", gif_side=480, end_s=25).json()
|
||||
job, asset = gif_worker["run"](job)
|
||||
assert (job.status, asset.status, asset.container, asset.media_kind) == (
|
||||
"done",
|
||||
"ready",
|
||||
"gif",
|
||||
"file",
|
||||
)
|
||||
assert asset.rel_path.endswith(".gif")
|
||||
assert (asset.width, asset.height) == (270, 480)
|
||||
assert asset.poster_path == str(Path(asset.rel_path).with_suffix(".jpg"))
|
||||
cmd = gif_worker["cmd"]
|
||||
graph = cmd[cmd.index("-filter_complex") + 1]
|
||||
# The HDR source is tonemapped before the palette is made from it; no sound rides along.
|
||||
assert graph.index("tonemap") < graph.index("palettegen")
|
||||
assert "-an" in cmd and "-map" not in cmd
|
||||
assert cmd[cmd.index("-t") + 1] == "15.000"
|
||||
|
||||
|
||||
def test_a_gif_clip_is_never_refused_for_an_audio_track(
|
||||
db, make_user, as_user, plex_episode, gif_worker
|
||||
):
|
||||
# A GIF queued with a track (as one made before GIFs dropped it was): the file has one track,
|
||||
# the spec names a fourth, and a GIF, which has no sound, is made all the same.
|
||||
job = _clip(as_user(make_user()), source="plex", ref="777", gif_side=480, end_s=25).json()
|
||||
row = db.get(DownloadJob, job["id"])
|
||||
row.edit_spec = {**row.edit_spec, "audio_track": 3}
|
||||
db.commit()
|
||||
job, asset = gif_worker["run"](job)
|
||||
assert (job.status, asset.status) == ("done", "ready")
|
||||
|
||||
|
||||
def test_a_gif_over_its_size_cap_is_not_kept_and_says_why(
|
||||
db, make_user, as_user, plex_episode, gif_worker, monkeypatch
|
||||
):
|
||||
from app.downloads import gif as gifmod
|
||||
|
||||
monkeypatch.setattr(gifmod, "MAX_BYTES", 20)
|
||||
gif_worker["gif_extra"] = 50
|
||||
job = _clip(as_user(make_user()), source="plex", ref="777", gif_side=640).json()
|
||||
job, asset = gif_worker["run"](job)
|
||||
assert (job.status, job.error_code, job.error, asset.rel_path) == (
|
||||
"error",
|
||||
"gif_too_big",
|
||||
dlerrors.GIF_TOO_BIG_MESSAGE,
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
def test_a_youtube_clip_range_yt_dlp_refuses_for_its_cap_says_it_is_too_big(
|
||||
db, make_user, as_user, yt_video, worker, monkeypatch
|
||||
):
|
||||
|
||||
@@ -75,6 +75,8 @@ def test_our_own_worker_refusals_classify_as_themselves():
|
||||
assert classify(dlerrors.CLIP_TOO_BIG_MESSAGE) == "quota_exceeded"
|
||||
assert classify(dlerrors.CONVERTED_TOO_BIG_MESSAGE) == "quota_exceeded"
|
||||
assert classify(dlerrors.DOWNLOAD_TOO_BIG_MESSAGE) == "quota_exceeded"
|
||||
# A GIF over its own cap is not the storage: its own category, its own sentence.
|
||||
assert classify(dlerrors.GIF_TOO_BIG_MESSAGE) == "gif_too_big"
|
||||
|
||||
|
||||
def test_the_redacted_network_text_classifies_back_to_network():
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
"""The editor's crop lands on the part of the picture the frame was drawn on, also when a video's
|
||||
pixels are not square. The browser reports such a video by its shown size (320 stored, 640 shown),
|
||||
and ffmpeg's `crop` counts stored pixels: measured on a real clip, a frame over the middle half kept
|
||||
half the frame's red surroundings. So a crop carries the size it was measured on.
|
||||
|
||||
Real ffmpeg: the pixels of the output are the oracle.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import edit, pipeline, service, storage
|
||||
from app.downloads import upload as uploadmod
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
|
||||
pytestmark = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
|
||||
|
||||
|
||||
def _source(path: Path, size: str, box: str, sar: int, rotate: int = 0) -> None:
|
||||
"""Red, with a green box: a crop of the box holds no red. `rotate` tags the file to be shown
|
||||
turned by that many degrees counter-clockwise, as a phone records a portrait video."""
|
||||
tag = ["-display_rotation:v:0", str(rotate)] if rotate else []
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", f"color=c=red:s={size}:d=2:r=10",
|
||||
"-f", "lavfi", "-i", "sine=duration=2",
|
||||
"-vf", f"drawbox={box}:c=lime:t=fill,setsar={sar}",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", str(path),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
if tag:
|
||||
tagged = path.with_name("tagged-" + path.name)
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-loglevel", "error", *tag, "-i", str(path), "-c", "copy", str(tagged)],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
tagged.replace(path)
|
||||
|
||||
|
||||
def _red_and_green(path: Path) -> tuple[int, int]:
|
||||
raw = subprocess.run(
|
||||
["ffmpeg", "-loglevel", "error", "-i", str(path), "-frames:v", "1",
|
||||
"-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
).stdout # fmt: skip
|
||||
red = green = 0
|
||||
for i in range(0, len(raw) - 2, 3):
|
||||
if raw[i] > 180 and raw[i + 1] < 80:
|
||||
red += 1
|
||||
elif raw[i + 1] > 180 and raw[i] < 80:
|
||||
green += 1
|
||||
return red, green
|
||||
|
||||
|
||||
def _cut_asset(db, make_user, src: Path, spec: dict, then=lambda: None) -> MediaAsset:
|
||||
user = make_user()
|
||||
source = uploadmod.ingest_upload(
|
||||
db, user.id, filename="clip.mp4", src_path=src, size=src.stat().st_size
|
||||
)
|
||||
assert source.status == "done"
|
||||
then()
|
||||
job = service.enqueue_edit(db, user.id, source, spec)
|
||||
asset = db.get(MediaAsset, job.asset_id)
|
||||
job.status, asset.status = "running", "downloading" # the worker's claim
|
||||
db.commit()
|
||||
|
||||
pipeline._process_edit(job.id, asset.id)
|
||||
|
||||
db.expire_all()
|
||||
job, asset = db.get(DownloadJob, job.id), db.get(MediaAsset, job.asset_id)
|
||||
assert (job.status, job.error, asset.status) == ("done", None, "ready")
|
||||
return asset
|
||||
|
||||
|
||||
def _cut(db, make_user, tmp_path, src: Path, crop: dict, **spec) -> Path:
|
||||
asset = _cut_asset(db, make_user, src, {"crop": crop, **spec})
|
||||
return storage.abs_path(str(tmp_path), asset.rel_path)
|
||||
|
||||
|
||||
def test_a_crop_of_a_video_with_wide_pixels_keeps_what_the_frame_showed(
|
||||
db, make_user, tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
|
||||
|
||||
# What the editor sends for the middle half of this video: the browser shows it at 640 x 240.
|
||||
final = _cut(
|
||||
db, make_user, tmp_path, src,
|
||||
{"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240},
|
||||
) # fmt: skip
|
||||
|
||||
red, green = _red_and_green(final)
|
||||
assert (red, green) == (0, 160 * 120)
|
||||
|
||||
|
||||
def test_a_crop_of_square_pixels_is_cut_where_it_was_drawn(db, make_user, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "480x640", "x=120:y=160:w=240:h=320", sar=1)
|
||||
|
||||
final = _cut(
|
||||
db, make_user, tmp_path, src,
|
||||
{"x": 120, "y": 160, "w": 240, "h": 320, "frame_w": 480, "frame_h": 640},
|
||||
) # fmt: skip
|
||||
|
||||
assert _red_and_green(final) == (0, 240 * 320)
|
||||
|
||||
|
||||
def test_a_crop_stored_before_it_carried_its_frame_is_cut_as_it_always_was():
|
||||
spec = edit.normalize_edit_spec({"crop": {"x": 4, "y": 6, "w": 100, "h": 50}})
|
||||
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
|
||||
cmd = edit.build_edit_cmd(Path("in.mp4"), Path("out.mp4"), spec, "mp4")
|
||||
assert cmd[cmd.index("-vf") + 1].startswith("crop=100:50:4:6,")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("frame", [{"frame_w": 0, "frame_h": 240}, {"frame_w": "x", "frame_h": 2}])
|
||||
def test_a_frame_that_cannot_be_read_is_not_divided_by(frame):
|
||||
spec = edit.normalize_edit_spec({"crop": {"x": 4, "y": 6, "w": 100, "h": 50, **frame}})
|
||||
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
|
||||
|
||||
|
||||
def test_a_crop_of_a_join_keeps_what_the_frame_showed(db, make_user, tmp_path, monkeypatch):
|
||||
"""A cut-list joins through `filter_complex`, a crop path of its own."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
|
||||
|
||||
final = _cut(
|
||||
db, make_user, tmp_path, src,
|
||||
{"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240},
|
||||
segments=[{"start_s": 0, "end_s": 0.5}, {"start_s": 1, "end_s": 1.5}], accurate=True,
|
||||
) # fmt: skip
|
||||
|
||||
assert _red_and_green(final) == (0, 160 * 120)
|
||||
|
||||
|
||||
def test_a_crop_of_a_rotated_video_is_cut_where_it_was_drawn(db, make_user, tmp_path, monkeypatch):
|
||||
"""The browser shows a rotated file upright, so the frame was drawn on the upright picture."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
# Stored 320 x 240, green on its left half; shown turned a quarter, 240 x 320.
|
||||
_source(src, "320x240", "x=0:y=0:w=160:h=240", sar=1, rotate=90)
|
||||
|
||||
# Turned counter-clockwise, the stored left half is the shown bottom half.
|
||||
final = _cut(
|
||||
db, make_user, tmp_path, src,
|
||||
{"x": 0, "y": 160, "w": 240, "h": 160, "frame_w": 240, "frame_h": 320},
|
||||
) # fmt: skip
|
||||
|
||||
assert _red_and_green(final) == (0, 240 * 160)
|
||||
|
||||
|
||||
def test_a_clip_ffprobe_cannot_size_gets_the_size_its_crop_cut(
|
||||
db, make_user, tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
|
||||
|
||||
def probe_fails():
|
||||
monkeypatch.setattr(pipeline.probe, "probe_media", lambda _p: (None,) * 5)
|
||||
|
||||
asset = _cut_asset(
|
||||
db, make_user, src,
|
||||
{"crop": {"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240}},
|
||||
then=probe_fails,
|
||||
) # fmt: skip
|
||||
|
||||
# Half the shown width of a 320-wide file: 160 stored, not the 320 the frame counted.
|
||||
assert (asset.width, asset.height) == (160, 120)
|
||||
|
||||
|
||||
def test_a_rotated_clip_ffprobe_cannot_size_gets_the_upright_size_its_crop_cut(
|
||||
db, make_user, tmp_path, monkeypatch
|
||||
):
|
||||
"""The frame was drawn on the upright picture, which is the stored one turned: so is its cut."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "320x240", "x=0:y=0:w=160:h=240", sar=1, rotate=90)
|
||||
|
||||
def probe_fails():
|
||||
monkeypatch.setattr(pipeline.probe, "probe_media", lambda _p: (None,) * 5)
|
||||
|
||||
asset = _cut_asset(
|
||||
db, make_user, src,
|
||||
{"crop": {"x": 0, "y": 160, "w": 240, "h": 160, "frame_w": 240, "frame_h": 320}},
|
||||
then=probe_fails,
|
||||
) # fmt: skip
|
||||
|
||||
# What ffmpeg cut (the rotated test above): 240 x 160, not the stored 320 x 120.
|
||||
assert (asset.width, asset.height) == (240, 160)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key", ["frame_w", "x"])
|
||||
def test_an_infinite_number_in_a_crop_is_dropped_not_a_crash(key):
|
||||
crop = {"x": 4, "y": 6, "w": 100, "h": 50, "frame_w": 640, "frame_h": 240}
|
||||
spec = edit.normalize_edit_spec({"crop": {**crop, key: float("inf")}})
|
||||
if key == "frame_w":
|
||||
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
|
||||
else:
|
||||
assert "crop" not in spec
|
||||
@@ -0,0 +1,197 @@
|
||||
"""The editor's GIF output: one marked range of a library video made into a GIF (downloads/gif), held
|
||||
to the GIF's limits at enqueue, and made by the worker with real ffmpeg."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import clip as clipmod
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import pipeline, service
|
||||
from app.downloads import probe as probemod
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import upload as uploadmod
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
|
||||
needs_ffmpeg = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
|
||||
|
||||
|
||||
def test_a_gif_edit_spec_keeps_its_size_and_is_another_edit_than_the_video():
|
||||
video = editmod.normalize_edit_spec({"trim": {"start_s": 1, "end_s": 5}})
|
||||
gif = editmod.normalize_edit_spec({"trim": {"start_s": 1, "end_s": 5}, "gif": {"side": 320}})
|
||||
assert gif["gif"] == {"side": 320} and "gif" not in video
|
||||
assert editmod.edit_sig(gif) != editmod.edit_sig(video)
|
||||
# A GIF is always made, never stream-copied.
|
||||
assert editmod.needs_reencode(gif)
|
||||
|
||||
|
||||
def _source(db, user, tmp_path, seconds=40):
|
||||
src = tmp_path / "in.mp4"
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", f"testsrc2=size=640x360:rate=10:duration={seconds}",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", str(src),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
return uploadmod.ingest_upload(
|
||||
db, user.id, filename="in.mp4", src_path=src, size=src.stat().st_size
|
||||
)
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
@pytest.mark.parametrize(
|
||||
("spec", "reason"),
|
||||
[
|
||||
# A GIF is one range with an end: no join, no open end.
|
||||
({"segments": [{"start_s": 0, "end_s": 2}, {"start_s": 4, "end_s": 6}]}, "gif_range"),
|
||||
({"trim": {"start_s": 3}}, "gif_range"),
|
||||
({"trim": {"start_s": 0, "end_s": 30.5}}, "gif_too_long"),
|
||||
({"trim": {"start_s": 0, "end_s": 5}, "gif_side": 999}, "bad_gif"),
|
||||
],
|
||||
)
|
||||
def test_a_gif_edit_it_cannot_make_is_refused_with_its_reason(
|
||||
db, make_user, tmp_path, monkeypatch, spec, reason
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
source = _source(db, u, tmp_path)
|
||||
side = spec.pop("gif_side", 480)
|
||||
with pytest.raises(service.EditError) as e:
|
||||
service.enqueue_edit(db, u.id, source, {**spec, "gif": {"side": side}})
|
||||
assert e.value.reason == reason
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_the_edit_route_says_a_gif_too_long_with_its_cap(
|
||||
db, make_user, as_user, tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
c = as_user(u)
|
||||
source = _source(db, u, tmp_path)
|
||||
r = c.post(
|
||||
"/api/downloads/edit",
|
||||
json={
|
||||
"source_job_id": source.id,
|
||||
"edit_spec": {"trim": {"start_s": 0, "end_s": 30.5}, "gif": {"side": 480}},
|
||||
},
|
||||
)
|
||||
assert r.status_code == 400, r.text
|
||||
# The cap goes along: the refusal's sentence says it ("at most 30 seconds").
|
||||
assert r.json()["detail"] == {"code": "edit", "reason": "gif_too_long", "limit_s": 30}
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_gif_edit_of_exactly_thirty_seconds_is_allowed(db, make_user, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
source = _source(db, u, tmp_path)
|
||||
# Ends kept to the millisecond whose float difference lands a hair over 30 s.
|
||||
assert 32.02 - 2.02 > gifmod.MAX_S
|
||||
job = service.enqueue_edit(
|
||||
db, u.id, source, {"trim": {"start_s": 2.02, "end_s": 32.02}, "gif": {"side": 480}}
|
||||
)
|
||||
assert job.edit_spec["gif"] == {"side": 480}
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_gif_is_made_of_a_video_never_of_audio(db, make_user, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
src = tmp_path / "in.m4a"
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "sine=frequency=440:duration=10", "-c:a", "aac", str(src),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
source = uploadmod.ingest_upload(
|
||||
db, u.id, filename="in.m4a", src_path=src, size=src.stat().st_size
|
||||
)
|
||||
assert db.get(MediaAsset, source.asset_id).media_kind == "audio"
|
||||
spec = {"trim": {"start_s": 1, "end_s": 5}}
|
||||
with pytest.raises(service.EditError) as e:
|
||||
service.enqueue_edit(db, u.id, source, {**spec, "gif": {"side": 480}})
|
||||
assert e.value.reason == "gif_not_video"
|
||||
# The same range as audio is an edit as it always was.
|
||||
assert service.enqueue_edit(db, u.id, source, spec).edit_spec["trim"]["end_s"] == 5
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_an_hdr_source_is_tonemapped_before_its_gif_palette(db, make_user, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
src = tmp_path / "hlg.mp4"
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "testsrc2=size=640x360:rate=10:duration=4",
|
||||
"-vf", "setparams=color_trc=arib-std-b67:color_primaries=bt2020:colorspace=bt2020nc",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", str(src),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
source = uploadmod.ingest_upload(
|
||||
db, u.id, filename="hlg.mp4", src_path=src, size=src.stat().st_size
|
||||
)
|
||||
# The upload keeps its HLG tag, as an HDR phone video's does.
|
||||
placed = Path(settings.files_root) / db.get(MediaAsset, source.asset_id).rel_path
|
||||
assert clipmod.is_hdr(probemod.probe_json(placed) or {})
|
||||
job = service.enqueue_edit(
|
||||
db, u.id, source, {"trim": {"start_s": 1, "end_s": 3}, "gif": {"side": 320}}
|
||||
)
|
||||
asset = db.get(MediaAsset, job.asset_id)
|
||||
job.status, asset.status = "running", "downloading"
|
||||
db.commit()
|
||||
seen: dict = {}
|
||||
build = gifmod.build_cmd
|
||||
|
||||
def spy(*a, **k):
|
||||
seen["pre"] = k["pre"]
|
||||
return build(*a, **k)
|
||||
|
||||
monkeypatch.setattr(pipeline.gifmod, "build_cmd", spy)
|
||||
pipeline._process_edit(job.id, asset.id)
|
||||
|
||||
assert "tonemap" in seen["pre"]
|
||||
db.expire_all()
|
||||
assert db.get(DownloadJob, job.id).status == "done"
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_gif_edit_is_made_by_the_worker_and_kept_as_an_image_file(
|
||||
db, make_user, tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
source = _source(db, u, tmp_path, seconds=6)
|
||||
spec = {"trim": {"start_s": 1, "end_s": 3}, "crop": {"x": 0, "y": 0, "w": 320, "h": 360}}
|
||||
job = service.enqueue_edit(db, u.id, source, {**spec, "gif": {"side": 320}})
|
||||
asset = db.get(MediaAsset, job.asset_id)
|
||||
assert asset.media_kind == "file"
|
||||
job.status, asset.status = "running", "downloading" # the worker's claim
|
||||
db.commit()
|
||||
|
||||
pipeline._process_edit(job.id, asset.id)
|
||||
|
||||
db.expire_all()
|
||||
job, asset = db.get(DownloadJob, job.id), db.get(MediaAsset, job.asset_id)
|
||||
assert (job.status, asset.status, asset.container, asset.media_kind) == (
|
||||
"done",
|
||||
"ready",
|
||||
"gif",
|
||||
"file",
|
||||
)
|
||||
placed = tmp_path / "root" / asset.rel_path
|
||||
assert placed.read_bytes()[:6] == b"GIF89a"
|
||||
# The crop's 320x360 has its long side (360) fitted into 320.
|
||||
assert (asset.width, asset.height) == gifmod.size(placed) == (284, 320)
|
||||
assert asset.poster_path and (tmp_path / "root" / asset.poster_path).read_bytes()[:2] == (
|
||||
b"\xff\xd8"
|
||||
)
|
||||
@@ -0,0 +1,412 @@
|
||||
"""SEC-17: the egress gate judges the address a guarded download actually dials.
|
||||
|
||||
Two loopback addresses stand in for the two sides: 127.0.0.2 plays a public host, 127.0.0.1 our own
|
||||
network. The gate under test refuses exactly 127.0.0.1, so every assertion is about the judgement and
|
||||
the wiring, with real sockets, a real yt-dlp and a real ffmpeg — no mocked transport.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import http.server
|
||||
import ipaddress
|
||||
import socket
|
||||
import socketserver
|
||||
import struct
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yt_dlp
|
||||
from yt_dlp.utils import download_range_func
|
||||
|
||||
from app.downloads import adapters, egress, errors, formats
|
||||
|
||||
PUBLIC = "127.0.0.2"
|
||||
INTERNAL = "127.0.0.1"
|
||||
|
||||
|
||||
def _forbid_internal(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
return str(ip) == INTERNAL
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gate():
|
||||
g = egress.start_gate(forbidden=_forbid_internal)
|
||||
yield g
|
||||
g.shutdown()
|
||||
g.server_close()
|
||||
|
||||
|
||||
class _Listener:
|
||||
"""A TCP listener that only counts who connected — the internal service under attack."""
|
||||
|
||||
def __init__(self, host: str = INTERNAL, port: int = 0):
|
||||
self.hits = 0
|
||||
self.sock = socket.socket()
|
||||
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
self.sock.bind((host, port))
|
||||
self.sock.listen()
|
||||
self.port = self.sock.getsockname()[1]
|
||||
threading.Thread(target=self._serve, daemon=True).start()
|
||||
|
||||
def _serve(self):
|
||||
while True:
|
||||
try:
|
||||
conn, _ = self.sock.accept()
|
||||
except OSError:
|
||||
return
|
||||
self.hits += 1
|
||||
with contextlib.suppress(OSError):
|
||||
conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
|
||||
conn.close()
|
||||
|
||||
def close(self):
|
||||
self.sock.close()
|
||||
|
||||
|
||||
def _public_server(routes: dict, port: int = 0, directory: Path | None = None):
|
||||
"""An HTTP server on the 'public' address: `routes` maps a path to (status, headers); anything
|
||||
else is served from `directory`."""
|
||||
|
||||
class H(http.server.SimpleHTTPRequestHandler):
|
||||
def __init__(self, *a, **k):
|
||||
super().__init__(*a, directory=str(directory or "."), **k)
|
||||
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
if self.path in routes:
|
||||
status, headers, body = routes[self.path]
|
||||
self.send_response(status)
|
||||
for k, v in headers.items():
|
||||
self.send_header(k, v)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
return
|
||||
super().do_GET()
|
||||
|
||||
srv = socketserver.ThreadingTCPServer((PUBLIC, port), H)
|
||||
srv.daemon_threads = True
|
||||
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||
return srv
|
||||
|
||||
|
||||
def _guarded_opts(tmp_path: Path, gate_url: str, **extra) -> dict:
|
||||
opts = formats.build_ydl_opts({}, str(tmp_path / "%(id)s.%(ext)s"), lambda d: None, egress=True)
|
||||
opts.update(retries=0, fragment_retries=0, extractor_retries=0, socket_timeout=5)
|
||||
opts["proxy"] = gate_url # the test's gate (it lets 127.0.0.2 through), not the worker's
|
||||
opts.update(extra)
|
||||
return opts
|
||||
|
||||
|
||||
def _download_error(opts: dict, url: str) -> str:
|
||||
with pytest.raises(yt_dlp.utils.DownloadError) as exc, egress.open_ydl(opts) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
return str(exc.value)
|
||||
|
||||
|
||||
# --- the judgement ---------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_name_with_any_forbidden_answer_is_refused():
|
||||
def resolve(answers):
|
||||
return lambda host, port, *a: [(2, 1, 6, "", (ip, port)) for ip in answers]
|
||||
|
||||
assert egress.vetted_address(
|
||||
"x", 80, resolve=resolve([PUBLIC]), forbidden=_forbid_internal
|
||||
) == (
|
||||
PUBLIC,
|
||||
80,
|
||||
)
|
||||
assert (
|
||||
egress.vetted_address(
|
||||
"x", 80, resolve=resolve([PUBLIC, INTERNAL]), forbidden=_forbid_internal
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
def no_answer(*a):
|
||||
raise socket.gaierror("no such name")
|
||||
|
||||
assert egress.vetted_address("x", 80, resolve=no_answer, forbidden=_forbid_internal) is None
|
||||
|
||||
|
||||
def test_the_real_judgement_refuses_our_own_networks():
|
||||
# The worker's gate uses urlguard's judgement unchanged — the same one the enqueue check uses.
|
||||
for addr in ("127.0.0.1", "10.1.2.3", "172.18.0.5", "192.168.1.1", "169.254.169.254"):
|
||||
assert egress.vetted_address(addr, 80, forbidden=egress.urlguard.is_forbidden) is None, addr
|
||||
assert egress.vetted_address("93.184.215.14", 80, forbidden=egress.urlguard.is_forbidden)
|
||||
|
||||
|
||||
def test_the_gate_dials_the_address_it_vetted_not_a_second_answer(gate):
|
||||
# DNS rebinding: a name that answers public for the check and private for the fetch. The gate
|
||||
# resolves once per connection and dials that answer.
|
||||
public = _public_server({"/": (200, {}, b"public")})
|
||||
internal = _Listener(INTERNAL, public.server_address[1])
|
||||
calls = []
|
||||
|
||||
def rebinding(host, port, *a):
|
||||
calls.append(host)
|
||||
ip = PUBLIC if len(calls) == 1 else INTERNAL
|
||||
return [(2, 1, 6, "", (ip, port))]
|
||||
|
||||
gate.resolve = rebinding
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", gate.server_address[1])) as s:
|
||||
s.sendall(
|
||||
f"GET http://rebind.test:{public.server_address[1]}/ HTTP/1.1\r\n"
|
||||
f"Host: rebind.test\r\n\r\n".encode()
|
||||
)
|
||||
answer = b"".join(iter(lambda: s.recv(4096), b""))
|
||||
assert answer.endswith(b"public")
|
||||
assert calls == ["rebind.test"]
|
||||
assert internal.hits == 0
|
||||
finally:
|
||||
public.shutdown()
|
||||
internal.close()
|
||||
|
||||
|
||||
def test_a_tunnel_into_the_network_is_refused_before_it_is_dialled(gate):
|
||||
internal = _Listener()
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", gate.server_address[1])) as s:
|
||||
s.sendall(f"CONNECT {INTERNAL}:{internal.port} HTTP/1.1\r\n\r\n".encode())
|
||||
answer = s.recv(4096)
|
||||
assert answer.startswith(b"HTTP/1.1 403")
|
||||
assert internal.hits == 0
|
||||
finally:
|
||||
internal.close()
|
||||
|
||||
|
||||
def test_a_client_that_hangs_up_mid_transfer_is_not_an_error(gate, monkeypatch):
|
||||
# yt-dlp closes a connection once it has read what it needed; seen live as a traceback the gate
|
||||
# printed for every such download.
|
||||
errors_seen = []
|
||||
monkeypatch.setattr(gate, "handle_error", lambda *a: errors_seen.append(a))
|
||||
public = _public_server({"/big": (200, {}, b"x" * 4_000_000)})
|
||||
try:
|
||||
s = socket.create_connection(("127.0.0.1", gate.server_address[1]))
|
||||
s.sendall(f"GET http://{PUBLIC}:{public.server_address[1]}/big HTTP/1.1\r\n\r\n".encode())
|
||||
s.recv(1024)
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
|
||||
s.close() # a reset, not a polite close
|
||||
time.sleep(0.5)
|
||||
assert errors_seen == []
|
||||
finally:
|
||||
public.shutdown()
|
||||
|
||||
|
||||
# --- a real yt-dlp through it ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_redirect_into_the_network_reaches_it_unguarded_and_not_through_the_gate(tmp_path, gate):
|
||||
internal = _Listener()
|
||||
public = _public_server(
|
||||
{"/v.mp4": (302, {"Location": f"http://{INTERNAL}:{internal.port}/v.mp4"}, b"")}
|
||||
)
|
||||
url = f"http://{PUBLIC}:{public.server_address[1]}/v.mp4"
|
||||
try:
|
||||
# The hole, as it was: yt-dlp follows the 3xx into our network.
|
||||
bare = {"outtmpl": str(tmp_path / "bare" / "%(id)s.%(ext)s"), "quiet": True, "retries": 0}
|
||||
with contextlib.suppress(yt_dlp.utils.DownloadError), yt_dlp.YoutubeDL(bare) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
assert internal.hits >= 1
|
||||
before = internal.hits
|
||||
|
||||
_download_error(_guarded_opts(tmp_path, gate.url), url)
|
||||
assert internal.hits == before
|
||||
finally:
|
||||
public.shutdown()
|
||||
internal.close()
|
||||
|
||||
|
||||
def test_a_live_and_a_dead_internal_port_leave_the_same_text(tmp_path, gate):
|
||||
live = _Listener()
|
||||
dead = _Listener()
|
||||
dead_port = dead.port
|
||||
dead.close()
|
||||
public = _public_server(
|
||||
{
|
||||
"/live.mp4": (302, {"Location": f"http://{INTERNAL}:{live.port}/x"}, b""),
|
||||
"/dead.mp4": (302, {"Location": f"http://{INTERNAL}:{dead_port}/x"}, b""),
|
||||
}
|
||||
)
|
||||
base = f"http://{PUBLIC}:{public.server_address[1]}"
|
||||
try:
|
||||
stored = []
|
||||
for name in ("live", "dead"):
|
||||
msg = _download_error(_guarded_opts(tmp_path / name, gate.url), f"{base}/{name}.mp4")
|
||||
stored.append(errors.user_error_text("url", msg, errors.classify(msg)))
|
||||
assert stored[0] == stored[1]
|
||||
assert live.hits == 0
|
||||
finally:
|
||||
public.shutdown()
|
||||
live.close()
|
||||
|
||||
|
||||
def test_a_format_fetched_past_the_gate_is_never_chosen(tmp_path):
|
||||
# ffmpeg speaks rtmp, rtsp, tcp… directly, where no HTTP proxy is asked. A guarded download keeps
|
||||
# only the formats fetched over http(s).
|
||||
opts = _guarded_opts(tmp_path, "http://127.0.0.1:9")
|
||||
info = {
|
||||
"id": "x",
|
||||
"title": "x",
|
||||
"extractor": "generic",
|
||||
"extractor_key": "Generic",
|
||||
"webpage_url": f"http://{PUBLIC}/page",
|
||||
"formats": [
|
||||
{"format_id": "rtmp", "url": f"rtmp://{INTERNAL}/live", "ext": "flv", "height": 1080},
|
||||
{"format_id": "web", "url": f"http://{PUBLIC}/a.mp4", "ext": "mp4", "height": 360},
|
||||
],
|
||||
}
|
||||
with egress.open_ydl(opts) as ydl:
|
||||
chosen = ydl.process_ie_result(dict(info), download=False)
|
||||
assert chosen["format_id"] == "web"
|
||||
|
||||
only_rtmp = dict(info, formats=[info["formats"][0]])
|
||||
with (
|
||||
pytest.raises(yt_dlp.utils.YoutubeDLError, match="Requested format is not available"),
|
||||
egress.open_ydl(opts) as ydl,
|
||||
):
|
||||
ydl.process_ie_result(only_rtmp, download=False)
|
||||
|
||||
|
||||
def test_a_format_yt_dlp_fetches_itself_is_kept():
|
||||
# f4m, ism and the DASH generator fetch through yt-dlp's own urlopen; a bare m3u8 goes to the
|
||||
# native HLS fetcher under hls_prefer_native (set for every guarded download). All ask the gate.
|
||||
kept = [
|
||||
{"format_id": p, "url": f"https://{PUBLIC}/{p}", "protocol": p}
|
||||
for p in ("f4m", "ism", "m3u8", "m3u8_native", "http_dash_segments_generator", "https")
|
||||
]
|
||||
raw = {"format_id": "rtmp", "url": f"rtmp://{PUBLIC}/live", "protocol": "rtmp"}
|
||||
_, info = egress.HttpOnlyFormats().run({"formats": [*kept, raw]})
|
||||
assert [f["format_id"] for f in info["formats"]] == [f["format_id"] for f in kept]
|
||||
|
||||
|
||||
def test_a_single_url_result_past_the_gate_loses_its_url():
|
||||
_, info = egress.HttpOnlyFormats().run({"url": f"rtmp://{INTERNAL}/live"})
|
||||
assert info["url"] is None
|
||||
_, info = egress.HttpOnlyFormats().run({"url": f"https://{PUBLIC}/a.mp4"})
|
||||
assert info["url"] == f"https://{PUBLIC}/a.mp4"
|
||||
|
||||
|
||||
def test_the_format_filter_says_no_phase_of_its_own(tmp_path):
|
||||
# It is a post-processor only to run before the format choice; the job's postprocessor hook
|
||||
# must not see it (it painted every guarded job "processing" before a byte was fetched).
|
||||
calls: list[dict] = []
|
||||
opts = _guarded_opts(tmp_path, "http://127.0.0.1:9", postprocessor_hooks=[calls.append])
|
||||
with egress.open_ydl(opts) as ydl:
|
||||
ydl.run_all_pps("pre_process", {"id": "x", "formats": []})
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_an_allowed_tunnel_carries_bytes_both_ways(gate):
|
||||
# The CONNECT path a site served over https takes: a 200, then the bytes the client sent after
|
||||
# the head (in the same packet here) reach the far end, and its answer comes back.
|
||||
far = socket.socket()
|
||||
far.bind((PUBLIC, 0))
|
||||
far.listen()
|
||||
|
||||
def echo():
|
||||
conn, _ = far.accept()
|
||||
with conn:
|
||||
conn.sendall(b"PONG:" + conn.recv(64))
|
||||
|
||||
threading.Thread(target=echo, daemon=True).start()
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", gate.server_address[1]), timeout=5) as s:
|
||||
s.sendall(f"CONNECT {PUBLIC}:{far.getsockname()[1]} HTTP/1.1\r\n\r\nPING".encode())
|
||||
got = b""
|
||||
while b"PONG:PING" not in got:
|
||||
chunk = s.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
got += chunk
|
||||
assert got.startswith(b"HTTP/1.1 200")
|
||||
assert got.endswith(b"PONG:PING")
|
||||
finally:
|
||||
far.close()
|
||||
|
||||
|
||||
def test_the_workers_own_gate_refuses_our_network():
|
||||
# Every other gate here is built with a test rule; this is the one `proxy_url()` starts.
|
||||
internal = _Listener()
|
||||
try:
|
||||
port = int(egress.proxy_url().rsplit(":", 1)[1])
|
||||
with socket.create_connection(("127.0.0.1", port), timeout=5) as s:
|
||||
s.sendall(f"CONNECT {INTERNAL}:{internal.port} HTTP/1.1\r\n\r\n".encode())
|
||||
assert s.recv(4096).startswith(b"HTTP/1.1 403")
|
||||
assert internal.hits == 0
|
||||
finally:
|
||||
internal.close()
|
||||
|
||||
|
||||
def _clip_hits(tmp_path: Path, segment: str, proxy: str) -> int:
|
||||
"""How many connections the internal listener takes while a clip (`download_ranges`, which
|
||||
yt-dlp hands to ffmpeg even for an HLS format) is cut from a playlist whose one segment is
|
||||
`segment` — formatted with the listener's address."""
|
||||
internal = _Listener()
|
||||
playlist = (
|
||||
"#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-TARGETDURATION:2\n#EXTINF:2.0,\n"
|
||||
+ segment.format(host=INTERNAL, port=internal.port)
|
||||
+ "\n#EXT-X-ENDLIST\n"
|
||||
).encode()
|
||||
public = _public_server(
|
||||
{"/p.m3u8": (200, {"Content-Type": "application/vnd.apple.mpegurl"}, playlist)}
|
||||
)
|
||||
try:
|
||||
opts = _guarded_opts(tmp_path, proxy, download_ranges=download_range_func([], [(0, 1)]))
|
||||
with contextlib.suppress(yt_dlp.utils.DownloadError), egress.open_ydl(opts) as ydl:
|
||||
ydl.extract_info(f"http://{PUBLIC}:{public.server_address[1]}/p.m3u8", download=True)
|
||||
return internal.hits
|
||||
finally:
|
||||
public.shutdown()
|
||||
internal.close()
|
||||
|
||||
|
||||
def test_a_clip_cut_by_ffmpeg_reaches_a_segment_only_through_the_gate(tmp_path, gate):
|
||||
# Without the gate, ffmpeg fetches a hostile playlist's segment from inside our network — the
|
||||
# control that shows this path is real. Through the gate it is never dialled.
|
||||
segment = "http://{host}:{port}/s.ts"
|
||||
assert _clip_hits(tmp_path / "bare", segment, proxy="") >= 1
|
||||
assert _clip_hits(tmp_path / "gated", segment, proxy=gate.url) == 0
|
||||
|
||||
|
||||
def test_ffmpeg_itself_refuses_a_raw_tcp_segment(tmp_path):
|
||||
# A boundary, not our code: the HTTP proxy cannot see a tcp:// segment, and measured, ffmpeg's
|
||||
# HLS reader does not open one from an http playlist. This fails if an ffmpeg upgrade starts to.
|
||||
assert _clip_hits(tmp_path, "tcp://{host}:{port}/", proxy="") == 0
|
||||
|
||||
|
||||
# --- the wiring ------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_only_youtube_downloads_skip_the_gate():
|
||||
assert adapters.SourceAdapter.egress_guarded is True
|
||||
assert adapters.resolve_adapter("url", "https://www.facebook.com/x").egress_guarded is True
|
||||
assert adapters.resolve_adapter("url", "https://vimeo.com/1").egress_guarded is True
|
||||
assert adapters.resolve_adapter("youtube", "dQw4w9WgXcQ").egress_guarded is False
|
||||
|
||||
|
||||
def test_a_guarded_download_goes_through_the_gate_whatever_its_adapter_adds():
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, extra={"proxy": "", "hls_prefer_native": False}, egress=True
|
||||
)
|
||||
assert opts["proxy"] == egress.proxy_url()
|
||||
assert opts["hls_prefer_native"] is True
|
||||
assert opts[egress.GUARDED] is True
|
||||
assert "proxy" not in formats.build_ydl_opts({}, "/tmp/x", lambda d: None, egress=False)
|
||||
|
||||
|
||||
def test_every_yt_dlp_this_app_opens_goes_through_open_ydl():
|
||||
app = Path(egress.__file__).resolve().parents[1]
|
||||
offenders = [
|
||||
f"{p.relative_to(app)}:{n}"
|
||||
for p in sorted(app.rglob("*.py"))
|
||||
if p.name != "egress.py"
|
||||
for n, line in enumerate(p.read_text(encoding="utf-8").splitlines(), 1)
|
||||
if "YoutubeDL(" in line
|
||||
]
|
||||
assert offenders == []
|
||||
@@ -18,7 +18,7 @@ import yt_dlp
|
||||
from yt_dlp.networking.exceptions import TransportError
|
||||
from yt_dlp.networking.impersonate import ImpersonateTarget
|
||||
|
||||
from app.downloads import adapters, pipeline
|
||||
from app.downloads import adapters, egress, pipeline
|
||||
from app.downloads import errors as dlerrors
|
||||
|
||||
_URL = "https://www.facebook.com/reel/806379462330494/?fs=e&fs=e"
|
||||
@@ -73,6 +73,9 @@ def ydl(monkeypatch):
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
@@ -157,6 +160,16 @@ def test_a_cookie_group_post_downloads_the_posts_own_video(ydl, tmp_path):
|
||||
# Never the post link: that is where the feed video comes from.
|
||||
assert web["extracted"] == [watch, watch]
|
||||
assert [o.get("impersonate") for o in seen[1:]] == [None, _CHROME]
|
||||
# The post page read and both attempts all go through the egress gate.
|
||||
assert all(o.get(egress.GUARDED) and o["proxy"] == egress.proxy_url() for o in seen), seen
|
||||
|
||||
|
||||
def test_a_youtube_download_skips_the_gate(ydl, tmp_path):
|
||||
seen, outcomes, _ = ydl
|
||||
outcomes += [{"id": "dQw4w9WgXcQ"}]
|
||||
yt = adapters.resolve_adapter("youtube", "dQw4w9WgXcQ")
|
||||
pipeline._extract_with_fallback(1, 1, "https://youtu.be/dQw4w9WgXcQ", {}, tmp_path, yt)
|
||||
assert seen and not any(o.get(egress.GUARDED) or o.get("proxy") for o in seen)
|
||||
|
||||
|
||||
def test_a_group_post_without_a_single_video_fails_saying_so(ydl, tmp_path):
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
"""A Facebook download's title when the signed-in read gives only a placeholder.
|
||||
|
||||
Measured on prod (2026-11-01): every Facebook download since 2026-10-28 is titled "Facebook" — the
|
||||
signed-in read (the owner's cookie) now gives that placeholder and NO description, so the adapter had
|
||||
no post text to fall back on, and the download name copied the placeholder too. A signed-out read of
|
||||
the same reel (yt-dlp 2026.08.19, as on prod) still carries the post's text in its description.
|
||||
"""
|
||||
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.downloads import adapters, backfill, egress, pipeline
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
|
||||
_URL = "https://www.facebook.com/reel/1860682468422402/"
|
||||
# The page yt-dlp says it read for it (`webpage_url`); a group post's link resolves the same way.
|
||||
_PAGE = "https://www.facebook.com/watch/?v=1860682468422402"
|
||||
# The measured signed-out read of that reel.
|
||||
_PUBLIC = {
|
||||
"title": "163K reactions · 14K shares | sorry not helping #workhumor #officelife | Work dot exe",
|
||||
"description": "sorry not helping\n\n#workhumor #officelife",
|
||||
"uploader": "Work dot exe",
|
||||
}
|
||||
|
||||
|
||||
def _fake_ydl(monkeypatch, outcome):
|
||||
"""yt-dlp faked at its `YoutubeDL` boundary: records each read (its options and its `url`),
|
||||
answers `outcome`."""
|
||||
seen: list[dict] = []
|
||||
|
||||
class FakeYDL:
|
||||
def __init__(self, opts):
|
||||
self.opts = opts
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
def extract_info(self, url, download):
|
||||
seen.append({**self.opts, "url": url})
|
||||
return outcome
|
||||
|
||||
monkeypatch.setattr("yt_dlp.YoutubeDL", FakeYDL)
|
||||
return seen
|
||||
|
||||
|
||||
def test_a_placeholder_with_nothing_behind_it_takes_the_signed_out_pages_text(monkeypatch):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
fb = adapters.FacebookAdapter()
|
||||
|
||||
info = backfill.public_title_info({"title": "Facebook", "description": None}, _URL, fb, 7)
|
||||
|
||||
assert fb.title(info) == "sorry not helping"
|
||||
# One metadata-only read, and signed out: no cookie goes with it.
|
||||
assert len(seen) == 1
|
||||
assert seen[0]["skip_download"] is True
|
||||
assert not seen[0].get("cookiefile")
|
||||
assert seen[0][egress.GUARDED] is True and seen[0]["proxy"] == egress.proxy_url()
|
||||
|
||||
|
||||
def test_a_real_title_reads_nothing_more(monkeypatch):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
info = {"title": "Funny Dark Humor Dad Joke!", "description": None}
|
||||
|
||||
assert backfill.public_title_info(info, _URL, adapters.FacebookAdapter(), 7) is info
|
||||
assert seen == []
|
||||
|
||||
|
||||
def test_a_placeholder_with_the_posts_text_behind_it_reads_nothing_more(monkeypatch):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
info = {"title": "Video", "description": "Office jokes, part two"}
|
||||
|
||||
assert backfill.public_title_info(info, _URL, adapters.FacebookAdapter(), 7) is info
|
||||
assert seen == []
|
||||
|
||||
|
||||
def test_a_placeholder_the_signed_out_page_cannot_better_stays(monkeypatch):
|
||||
_fake_ydl(monkeypatch, {"title": "Facebook", "description": None})
|
||||
fb = adapters.FacebookAdapter()
|
||||
|
||||
info = backfill.public_title_info({"title": "Facebook", "description": None}, _URL, fb, 7)
|
||||
|
||||
assert fb.title(info) == "Facebook"
|
||||
|
||||
|
||||
def test_another_source_is_never_read_again(monkeypatch):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
info = {"title": "Video", "description": None}
|
||||
|
||||
assert backfill.public_title_info(info, _URL, adapters.SourceAdapter(), 7) is info
|
||||
assert seen == []
|
||||
|
||||
|
||||
def test_the_backfill_retitles_a_placeholder_download_and_the_names_that_repeated_it(
|
||||
db, make_user, monkeypatch
|
||||
):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
owner, other = make_user(), make_user()
|
||||
asset = MediaAsset(
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
format_sig="fb",
|
||||
status="ready",
|
||||
rel_path="facebook/x.mp4",
|
||||
poster_path="facebook/x.jpg",
|
||||
uploader_url="https://www.facebook.com/1",
|
||||
title="Facebook",
|
||||
ref_count=2,
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
copied = DownloadJob(
|
||||
user_id=owner.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="done",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
display_name="Facebook",
|
||||
)
|
||||
renamed = DownloadJob(
|
||||
user_id=other.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="done",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
display_name="Office jokes",
|
||||
)
|
||||
db.add_all([copied, renamed])
|
||||
db.commit()
|
||||
|
||||
assert backfill.run()["retitled"] == 1
|
||||
assert len(seen) == 1
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(MediaAsset, asset.id).title == "sorry not helping"
|
||||
assert db.get(DownloadJob, copied.id).display_name == "sorry not helping"
|
||||
assert db.get(DownloadJob, renamed.id).display_name == "Office jokes"
|
||||
# Run again: nothing left to retitle, and no page read for it.
|
||||
assert backfill.run()["retitled"] == 0
|
||||
assert len(seen) == 1
|
||||
|
||||
|
||||
def test_a_name_given_while_the_backfill_runs_stays(db, make_user, monkeypatch):
|
||||
# The owner renames the download while the backfill reads the page: the owner's name wins, it is
|
||||
# not overwritten by what was read before it.
|
||||
_fake_ydl(monkeypatch, _PUBLIC)
|
||||
owner = make_user()
|
||||
asset = MediaAsset(
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
format_sig="fb",
|
||||
status="ready",
|
||||
rel_path="facebook/x.mp4",
|
||||
uploader_url="https://www.facebook.com/1",
|
||||
title="Facebook",
|
||||
ref_count=1,
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
job = DownloadJob(
|
||||
user_id=owner.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="done",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
display_name="Facebook",
|
||||
)
|
||||
db.add(job)
|
||||
db.commit()
|
||||
|
||||
read_page = backfill.public_title_info
|
||||
|
||||
def renamed_while_reading(*args):
|
||||
with SessionLocal() as other:
|
||||
other.execute(
|
||||
update(DownloadJob)
|
||||
.where(DownloadJob.id == job.id)
|
||||
.values(display_name="Office jokes")
|
||||
)
|
||||
other.commit()
|
||||
return read_page(*args)
|
||||
|
||||
monkeypatch.setattr(backfill, "public_title_info", renamed_while_reading)
|
||||
monkeypatch.setattr(backfill.editmod, "ensure_poster", lambda a, root: None)
|
||||
|
||||
assert backfill.run()["retitled"] == 1
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(DownloadJob, job.id).display_name == "Office jokes"
|
||||
|
||||
|
||||
def test_the_poster_step_holds_no_row_lock(db, make_user, monkeypatch):
|
||||
# It can cut a frame for 30 s. A delete locks the asset and then the job: with the job held here
|
||||
# and the asset still to write, that is a deadlock, and a rename waits out the whole cut.
|
||||
_fake_ydl(monkeypatch, _PUBLIC)
|
||||
owner = make_user()
|
||||
asset = MediaAsset(
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
format_sig="fb",
|
||||
status="ready",
|
||||
rel_path="facebook/x.mp4",
|
||||
uploader_url="https://www.facebook.com/1",
|
||||
title="Facebook",
|
||||
ref_count=1,
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
job = DownloadJob(
|
||||
user_id=owner.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="done",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
display_name="Facebook",
|
||||
)
|
||||
db.add(job)
|
||||
db.commit()
|
||||
asset_id, job_id = asset.id, job.id
|
||||
|
||||
def poster_finds_rows_free(a, root):
|
||||
with SessionLocal() as other:
|
||||
other.execute(
|
||||
select(MediaAsset).where(MediaAsset.id == asset_id).with_for_update(nowait=True)
|
||||
)
|
||||
other.execute(
|
||||
select(DownloadJob).where(DownloadJob.id == job_id).with_for_update(nowait=True)
|
||||
)
|
||||
other.rollback()
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(backfill.editmod, "ensure_poster", poster_finds_rows_free)
|
||||
|
||||
assert backfill.run()["retitled"] == 1
|
||||
|
||||
|
||||
def test_a_download_titled_facebook_is_named_from_the_signed_out_page(
|
||||
db, make_user, monkeypatch, tmp_path
|
||||
):
|
||||
# The whole download path with yt-dlp and the media tools faked: the signed-in read gives only
|
||||
# the placeholder, and the file and the download's name get the post's text.
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
|
||||
monkeypatch.setattr(pipeline, "_ensure_phone_safe", lambda _j, produced, *_a: produced)
|
||||
monkeypatch.setattr(
|
||||
pipeline.probe, "resolve_media_facts", lambda produced, info: ("h264", "aac", 720, 1280, 9)
|
||||
)
|
||||
monkeypatch.setattr(pipeline.editmod, "thumbnail_to_jpg", lambda *a: None)
|
||||
monkeypatch.setattr(pipeline.editmod, "ensure_poster", lambda *a: None)
|
||||
owner = make_user()
|
||||
asset = MediaAsset(
|
||||
source_kind="url", source_ref=_URL, format_sig="fb", status="downloading", ref_count=1
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
job = DownloadJob(
|
||||
user_id=owner.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="running",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
)
|
||||
db.add(job)
|
||||
db.commit()
|
||||
|
||||
def signed_in_read(job_id, asset_id, url, spec, staging, adapter, **kw):
|
||||
f = staging / "1860682468422402.mp4"
|
||||
f.write_bytes(b"v" * 100)
|
||||
return {
|
||||
"id": "1860682468422402",
|
||||
"title": "Facebook",
|
||||
"uploader": "Work dot exe",
|
||||
"webpage_url": _PAGE,
|
||||
"requested_downloads": [{"filepath": str(f)}],
|
||||
}
|
||||
|
||||
monkeypatch.setattr(pipeline, "_extract_with_fallback", signed_in_read)
|
||||
pipeline._download(job.id, asset.id, "url", _URL, {}, owner.id)
|
||||
|
||||
# The signed-out read goes to the page yt-dlp read, not the link queued: a group post's link
|
||||
# resolves to its video, and signed out the post itself is the login wall.
|
||||
assert [r["url"] for r in seen] == [_PAGE]
|
||||
db.expire_all()
|
||||
assert db.get(DownloadJob, job.id).status == "done"
|
||||
assert db.get(MediaAsset, asset.id).title == "sorry not helping"
|
||||
assert db.get(DownloadJob, job.id).display_name == "sorry not helping"
|
||||
@@ -359,6 +359,9 @@ def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkey
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
@@ -367,8 +370,8 @@ def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkey
|
||||
f"ERROR: [youtube] dQw4w9WgXcQ: Video unavailable ({target})"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
|
||||
assert backfill._extract_meta(url, adapters.YouTubeAdapter(), 11) is None
|
||||
monkeypatch.setattr("yt_dlp.YoutubeDL", _Refusing)
|
||||
assert backfill.extract_meta(url, adapters.YouTubeAdapter(), 11) is None
|
||||
|
||||
lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
|
||||
assert lines and all("dQw4w9WgXcQ" not in json.dumps(o) for o in lines)
|
||||
@@ -386,14 +389,17 @@ def test_the_metadata_backfill_takes_out_a_generic_name_whatever_it_holds(sink,
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
def extract_info(self, target, download):
|
||||
raise yt_dlp.utils.DownloadError("ERROR: [generic] Our Trip: Day 1: Unable to download")
|
||||
|
||||
monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
|
||||
assert backfill._extract_meta(url, adapters.SourceAdapter(), 11) is None
|
||||
monkeypatch.setattr("yt_dlp.YoutubeDL", _Refusing)
|
||||
assert backfill.extract_meta(url, adapters.SourceAdapter(), 11) is None
|
||||
|
||||
lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
|
||||
assert lines and all("Day 1" not in json.dumps(o) for o in lines)
|
||||
@@ -449,6 +455,9 @@ def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, t
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
@@ -463,6 +472,7 @@ def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, t
|
||||
resolve_url=lambda target, opts: target,
|
||||
ydl_opts_extra=lambda clients: {},
|
||||
retriable=lambda text: True,
|
||||
egress_guarded=False,
|
||||
)
|
||||
monkeypatch.setattr(pipeline.yt_dlp, "YoutubeDL", _YDL)
|
||||
monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: None)
|
||||
|
||||
@@ -150,6 +150,7 @@ def test_a_password_folder_link_shows_nothing_until_unlocked(shelf, client):
|
||||
assert client.get(f"/api/public/watch/{token}").json() == {
|
||||
"needs_password": True,
|
||||
"kind": "folder",
|
||||
"video": False,
|
||||
}
|
||||
bare = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/file"
|
||||
assert client.get(bare).status_code == 403
|
||||
@@ -312,7 +313,7 @@ def test_a_password_file_link_says_it_opens_a_file(shelf, client):
|
||||
.json()
|
||||
)
|
||||
meta = client.get(f"/api/public/watch/{made['token']}").json()
|
||||
assert meta == {"needs_password": True, "kind": "file"}
|
||||
assert meta == {"needs_password": True, "kind": "file", "video": True}
|
||||
|
||||
|
||||
def test_a_file_whose_bytes_are_gone_is_not_offered(shelf, client, db):
|
||||
|
||||
@@ -15,16 +15,20 @@ from app.downloads import formats
|
||||
|
||||
|
||||
def _opts(spec):
|
||||
return formats.build_ydl_opts(spec, "/tmp/%(id)s.%(ext)s", lambda d: None)
|
||||
return formats.build_ydl_opts(spec, "/tmp/%(id)s.%(ext)s", lambda d: None, egress=False)
|
||||
|
||||
|
||||
def test_adapter_extra_opts_are_merged_in():
|
||||
# The SourceAdapter contributes source-specific opts (e.g. YouTube's extractor_args) via `extra`.
|
||||
extra = {"extractor_args": {"youtube": {"player_client": ["web"]}}}
|
||||
opts = formats.build_ydl_opts({}, "/tmp/%(id)s.%(ext)s", lambda d: None, extra=extra)
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "/tmp/%(id)s.%(ext)s", lambda d: None, extra=extra, egress=False
|
||||
)
|
||||
assert opts["extractor_args"]["youtube"]["player_client"] == ["web"]
|
||||
# No extra → no extractor_args (a generic source contributes nothing).
|
||||
assert "extractor_args" not in formats.build_ydl_opts({}, "/tmp/x", lambda d: None)
|
||||
assert "extractor_args" not in formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, egress=False
|
||||
)
|
||||
|
||||
|
||||
def test_mp4_default_selects_avc1_with_progressive_fallback():
|
||||
@@ -113,7 +117,7 @@ def test_a_playlist_url_can_only_yield_one_item():
|
||||
|
||||
|
||||
def test_the_remaining_quota_becomes_a_hard_size_ceiling():
|
||||
opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000)
|
||||
opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000, egress=False)
|
||||
assert opts["max_filesize"] == 5_000_000
|
||||
|
||||
|
||||
@@ -122,7 +126,7 @@ def test_no_budget_means_no_size_filter():
|
||||
# enqueue, not an impossible yt-dlp filter to express here.
|
||||
for budget in (None, 0, -1):
|
||||
assert "max_filesize" not in formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, max_bytes=budget
|
||||
{}, "/tmp/x", lambda d: None, max_bytes=budget, egress=False
|
||||
)
|
||||
|
||||
|
||||
@@ -130,7 +134,12 @@ def test_a_source_adapter_cannot_raise_the_size_ceiling():
|
||||
# The adapter merge happens after max_filesize is set, so this pins that no adapter option can
|
||||
# widen a storage cap — the mirror of the cookiefile rule, in the opposite direction.
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, extra={"max_filesize": 999_999_999}, max_bytes=5_000_000
|
||||
{},
|
||||
"/tmp/x",
|
||||
lambda d: None,
|
||||
extra={"max_filesize": 999_999_999},
|
||||
max_bytes=5_000_000,
|
||||
egress=False,
|
||||
)
|
||||
assert opts["max_filesize"] == 5_000_000
|
||||
|
||||
@@ -197,7 +206,7 @@ def test_yt_dlp_refuses_a_file_over_the_cap_without_raising_and_the_watch_sees_i
|
||||
for cap, refused in ((1_000, True), (10_000_000, False)):
|
||||
out = tmp_path / f"out-{cap}"
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, str(out / "%(id)s.%(ext)s"), lambda d: None, max_bytes=cap
|
||||
{}, str(out / "%(id)s.%(ext)s"), lambda d: None, max_bytes=cap, egress=False
|
||||
)
|
||||
with formats.watching_size_cap() as over, yt_dlp.YoutubeDL(opts) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
"""The GIF output (downloads/gif): its limits, the one ffmpeg command every GIF goes through, and what
|
||||
that command really makes — run with real ffmpeg, since the size, the palette and the frame are the
|
||||
encoder's, not the argv's."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import mediakind
|
||||
from app.downloads import probe as probemod
|
||||
from app.downloads.edit import INPUT_GUARD
|
||||
|
||||
needs_ffmpeg = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
|
||||
|
||||
|
||||
def test_a_side_is_one_of_the_offered_sizes_and_anything_else_is_refused():
|
||||
assert gifmod.SIDES == (320, 480, 640) and gifmod.DEFAULT_SIDE == 480
|
||||
for side in gifmod.SIDES:
|
||||
assert gifmod.normalize_side(side) == side
|
||||
for bad in (0, 100, 481, 1080, -480, True, "480", None, 480.5):
|
||||
with pytest.raises(ValueError):
|
||||
gifmod.normalize_side(bad)
|
||||
|
||||
|
||||
def test_the_limits_are_the_measured_ones():
|
||||
# 30 s at most; a link preview moves in Teams up to 15 MB (measured: 15 moved, 22 did not); a GIF
|
||||
# over 30 MB is not kept.
|
||||
assert gifmod.MAX_S == 30
|
||||
assert gifmod.PREVIEW_MAX_BYTES == 15 * 1024 * 1024
|
||||
assert gifmod.MAX_BYTES == 30 * 1024 * 1024
|
||||
assert gifmod.FPS == 12
|
||||
|
||||
|
||||
def _cmd(**kw):
|
||||
args = {"pre": "", "side": 480, **kw}
|
||||
return gifmod.build_cmd(Path("/m/in.mkv"), Path("/s/out.gif"), 61.5, 12.0, **args)
|
||||
|
||||
|
||||
def test_the_seek_is_on_the_input_and_the_library_read_is_guarded():
|
||||
cmd = _cmd()
|
||||
i = cmd.index("-i")
|
||||
assert cmd[cmd.index("-ss") + 1] == "61.500" and cmd.index("-ss") < i
|
||||
assert cmd[i + 1] == "/m/in.mkv"
|
||||
assert cmd[cmd.index("-t") + 1] == "12.000" and cmd.index("-t") > i
|
||||
g = cmd.index(INPUT_GUARD[0])
|
||||
assert cmd[g : g + len(INPUT_GUARD)] == INPUT_GUARD and g < i
|
||||
assert cmd[-1] == "/s/out.gif"
|
||||
|
||||
|
||||
def test_a_gif_has_no_sound_and_one_palette_of_its_own():
|
||||
cmd = _cmd()
|
||||
assert "-an" in cmd and "-sn" in cmd and "-dn" in cmd
|
||||
graph = cmd[cmd.index("-filter_complex") + 1]
|
||||
assert f"fps={gifmod.FPS}" in graph
|
||||
assert "palettegen=stats_mode=diff" in graph
|
||||
assert "paletteuse=dither=bayer:bayer_scale=5:diff_mode=rectangle" in graph
|
||||
assert "flags=lanczos" in graph
|
||||
|
||||
|
||||
def test_a_crop_or_a_tonemap_comes_before_the_scale_and_the_palette():
|
||||
cmd = _cmd(pre="crop=100:100:0:0,format=yuv420p")
|
||||
graph = cmd[cmd.index("-filter_complex") + 1]
|
||||
assert graph.index("crop=100:100:0:0") < graph.index("scale=") < graph.index("palettegen")
|
||||
|
||||
|
||||
def test_the_length_cap_counts_whole_milliseconds():
|
||||
# Two ends kept to the millisecond whose float difference lands a hair over 30 s.
|
||||
assert 60.529 - 30.529 > gifmod.MAX_S
|
||||
assert not gifmod.too_long(30.529, 60.529)
|
||||
assert not gifmod.too_long(0, 30)
|
||||
assert gifmod.too_long(0, 30.001)
|
||||
|
||||
|
||||
def test_the_pixels_are_squared_after_the_crop_and_before_the_size():
|
||||
graph = gifmod.filter_graph("crop=100:100:0:0", 480)
|
||||
assert graph.index("crop=") < graph.index("setsar=1") < graph.index("flags=lanczos")
|
||||
|
||||
|
||||
def _video(path: Path, size: str, seconds: int = 2, sar: str = "1") -> Path:
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", f"testsrc2=size={size}:rate=25:duration={seconds}",
|
||||
"-vf", f"setsar={sar}",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", str(path),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
return path
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
@pytest.mark.parametrize(
|
||||
("size", "sar", "expected"),
|
||||
[
|
||||
# A DVD stored 720x480 and shown at 16:9: a GIF shows its stored pixels as they are, so
|
||||
# they are squared first, or the picture comes out 480x320 (3:2), squeezed.
|
||||
("720x480", "32/27", (480, 270)),
|
||||
# HDV: 1440x1080 shown at 16:9.
|
||||
("1440x1080", "4/3", (480, 270)),
|
||||
],
|
||||
)
|
||||
def test_a_source_with_wide_pixels_keeps_its_shown_shape(tmp_path, size, sar, expected):
|
||||
src = _video(tmp_path / "in.mp4", size, sar=sar)
|
||||
out = tmp_path / "out.gif"
|
||||
subprocess.run(gifmod.build_cmd(src, out, 0.0, 1.0, pre="", side=480), check=True)
|
||||
assert gifmod.size(out) == expected
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
@pytest.mark.parametrize(
|
||||
("size", "side", "expected"),
|
||||
[
|
||||
# The long side is capped, so a phone's portrait video is as small as a landscape one.
|
||||
("1280x720", 480, (480, 270)),
|
||||
("720x1280", 480, (270, 480)),
|
||||
("1280x720", 320, (320, 180)),
|
||||
# A picture already smaller than the size is never blown up.
|
||||
("160x120", 640, (160, 120)),
|
||||
],
|
||||
)
|
||||
def test_the_long_side_is_capped_and_a_small_picture_is_kept(tmp_path, size, side, expected):
|
||||
src = _video(tmp_path / "in.mp4", size)
|
||||
out = tmp_path / "out.gif"
|
||||
subprocess.run(gifmod.build_cmd(src, out, 0.5, 1.0, pre="", side=side), check=True)
|
||||
assert out.read_bytes()[:6] == b"GIF89a"
|
||||
assert gifmod.size(out) == expected
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_made_gif_is_a_library_file_that_previews_as_an_image(tmp_path):
|
||||
src = _video(tmp_path / "in.mp4", "320x240")
|
||||
out = tmp_path / "out.gif"
|
||||
subprocess.run(gifmod.build_cmd(src, out, 0.0, 1.0, pre="", side=320), check=True)
|
||||
vcodec, acodec, *_ = probemod.probe_media(out)
|
||||
assert mediakind.derive(vcodec, acodec) == mediakind.FILE
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_gif_gets_a_jpeg_still_for_link_previews(tmp_path):
|
||||
src = _video(tmp_path / "in.mp4", "320x240")
|
||||
out = tmp_path / "out.gif"
|
||||
subprocess.run(gifmod.build_cmd(src, out, 0.0, 1.0, pre="", side=320), check=True)
|
||||
still = gifmod.ensure_still(out)
|
||||
assert still == out.with_suffix(".jpg")
|
||||
assert still.read_bytes()[:3] == b"\xff\xd8\xff"
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_the_still_is_cut_only_from_a_real_gif(tmp_path):
|
||||
# The still's read is held to the GIF demuxer: a "gif" that is something else gives no still.
|
||||
fake = tmp_path / "out.gif"
|
||||
fake.write_bytes(b"ffconcat version 1.0\nfile '/etc/passwd'\n")
|
||||
assert gifmod.ensure_still(fake) is None
|
||||
|
||||
|
||||
def test_the_size_is_read_from_the_gif_header_and_nothing_else_has_one(tmp_path):
|
||||
g = tmp_path / "a.gif"
|
||||
g.write_bytes(b"GIF89a" + (300).to_bytes(2, "little") + (170).to_bytes(2, "little") + b"\x00")
|
||||
assert gifmod.size(g) == (300, 170)
|
||||
other = tmp_path / "b.gif"
|
||||
other.write_bytes(b"\x89PNG\r\n\x1a\n")
|
||||
assert gifmod.size(other) is None
|
||||
assert gifmod.size(tmp_path / "missing.gif") is None
|
||||
@@ -0,0 +1,137 @@
|
||||
"""A shared GIF (or any shared picture): served inline as itself on the public page, and its link
|
||||
preview card. Teams moves a GIF `og:image` up to 15 MB; Messenger never showed one and showed a JPEG
|
||||
(measured 2026-11-09/11), so Facebook's crawler and a GIF too big to move get the still. DB-lane."""
|
||||
|
||||
import pytest
|
||||
|
||||
from app import main
|
||||
from app.config import settings
|
||||
from app.downloads import og
|
||||
from app.downloads import probe as probemod
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
|
||||
_PAGE = """<!doctype html><html><head>
|
||||
<!--OG:START--><meta property="og:title" content="Siftlode" /><!--OG:END-->
|
||||
</head><body><script src="/assets/watch.js"></script></body></html>"""
|
||||
|
||||
_GIF_PROBE = {
|
||||
"streams": [{"codec_type": "video", "codec_name": "gif", "width": 300, "height": 170}],
|
||||
"format": {"format_name": "gif"},
|
||||
}
|
||||
_GIF = b"GIF89a" + (300).to_bytes(2, "little") + (170).to_bytes(2, "little") + b"\x00" * 16
|
||||
|
||||
_TEAMS = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) SkypeUriPreview Preview/0.5 skype-url-preview@microsoft.com"
|
||||
_FACEBOOK = "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pages(tmp_path, monkeypatch):
|
||||
watch = tmp_path / "watch.html"
|
||||
watch.write_text(_PAGE, encoding="utf-8")
|
||||
monkeypatch.setattr(main, "WATCH_HTML", watch)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def shared(as_user, make_user, tmp_path, monkeypatch, upload_headers):
|
||||
"""Uploads `content` as `name` (ffprobe reading it as `data`) and shares it."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "media"))
|
||||
owner = as_user(make_user())
|
||||
|
||||
def share(name: str, content: bytes, data: dict | None, **link: object) -> dict:
|
||||
monkeypatch.setattr(probemod, "probe_json", lambda _p: data)
|
||||
job = owner.post(
|
||||
"/api/files/upload", content=content, headers=upload_headers(name, display_name=name)
|
||||
)
|
||||
assert job.status_code == 200, job.text
|
||||
made = owner.post(f"/api/downloads/{job.json()['id']}/links", json=link)
|
||||
assert made.status_code == 200, made.text
|
||||
return {**made.json(), "job_id": job.json()["id"]}
|
||||
|
||||
return share
|
||||
|
||||
|
||||
def _asset(db, made) -> MediaAsset:
|
||||
return db.get(MediaAsset, db.get(DownloadJob, made["job_id"]).asset_id)
|
||||
|
||||
|
||||
def test_a_shared_gif_is_served_inline_as_a_gif_and_can_run_nothing(shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE, allow_download=False)
|
||||
r = client.get(f"/api/public/watch/{made['token']}/file")
|
||||
assert r.status_code == 200
|
||||
assert r.headers["content-type"] == "image/gif"
|
||||
assert r.headers["content-disposition"].startswith("inline")
|
||||
# A "gif" whose bytes are a page must not run on this login-free origin.
|
||||
assert r.headers["x-content-type-options"] == "nosniff"
|
||||
assert "sandbox" in r.headers["content-security-policy"]
|
||||
assert r.content == _GIF
|
||||
|
||||
|
||||
def test_a_gif_link_that_allows_download_hands_it_over_as_a_file(shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE, allow_download=True)
|
||||
r = client.get(f"/api/public/watch/{made['token']}/file")
|
||||
assert r.headers["content-type"] == "image/gif"
|
||||
assert r.headers["content-disposition"].startswith("attachment")
|
||||
|
||||
|
||||
def test_any_other_file_is_still_only_a_download(shared, client):
|
||||
made = shared("notes.html", b"<script>alert(1)</script>", None)
|
||||
r = client.get(f"/api/public/watch/{made['token']}/file")
|
||||
assert r.headers["content-type"] == "application/octet-stream"
|
||||
assert r.headers["content-disposition"].startswith("attachment")
|
||||
|
||||
|
||||
def test_the_page_is_told_a_picture_is_a_picture(shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
meta = client.get(f"/api/public/watch/{made['token']}").json()
|
||||
assert meta["preview"] == "image"
|
||||
other = shared("backup.tar.gz", b"x", None)
|
||||
assert client.get(f"/api/public/watch/{other['token']}").json()["preview"] is None
|
||||
|
||||
|
||||
def _og(text: str, prop: str) -> str | None:
|
||||
marker = f'property="{prop}" content="'
|
||||
if marker not in text:
|
||||
return None
|
||||
return text.split(marker, 1)[1].split('"', 1)[0]
|
||||
|
||||
|
||||
def test_a_gif_link_previews_as_the_moving_gif_itself(pages, shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
page = client.get(f"/file/{made['token']}", headers={"User-Agent": _TEAMS}).text
|
||||
assert _og(page, "og:image") == (f"{settings.app_base}/api/public/watch/{made['token']}/file")
|
||||
assert _og(page, "og:image:type") == "image/gif"
|
||||
# Its size from its own header: a card wants it up front.
|
||||
assert (_og(page, "og:image:width"), _og(page, "og:image:height")) == ("300", "170")
|
||||
|
||||
|
||||
def test_facebooks_crawler_gets_the_still_of_a_gif(pages, shared, client, db):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
asset = _asset(db, made)
|
||||
asset.poster_path = "wave.jpg"
|
||||
db.commit()
|
||||
page = client.get(f"/file/{made['token']}", headers={"User-Agent": _FACEBOOK}).text
|
||||
assert _og(page, "og:image") == (
|
||||
f"{settings.app_base}/api/public/watch/{made['token']}/poster.jpg"
|
||||
)
|
||||
assert _og(page, "og:image:type") == "image/jpeg"
|
||||
|
||||
|
||||
def test_a_gif_too_big_to_move_previews_as_its_still(pages, shared, client, db):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
asset = _asset(db, made)
|
||||
asset.poster_path = "wave.jpg"
|
||||
asset.size_bytes = 15 * 1024 * 1024 + 1
|
||||
db.commit()
|
||||
page = client.get(f"/file/{made['token']}", headers={"User-Agent": _TEAMS}).text
|
||||
assert _og(page, "og:image").endswith("/poster.jpg")
|
||||
|
||||
|
||||
def test_a_password_gif_link_shows_nothing_of_it(pages, shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE, password="pw")
|
||||
page = client.get(f"/file/{made['token']}", headers={"User-Agent": _TEAMS}).text
|
||||
assert _og(page, "og:image") is None
|
||||
|
||||
|
||||
def test_the_card_builder_takes_no_agent_as_a_browser(shared, db):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
assert 'content="image/gif"' in og._watch_tags(made["token"], db)
|
||||
@@ -5,11 +5,11 @@ POST /session: after the last playback, an abandoned session (its ffmpeg still w
|
||||
film, its directory a whole segmented copy of it) lived until somebody pressed play again.
|
||||
|
||||
And it must not take a PAUSED viewer's session: a remux finishes long before the film does, and a
|
||||
paused player with a full buffer fetches nothing, so "finished and unfetched for 30 s" does not mean
|
||||
paused player with a full buffer fetches nothing, so "finished and unfetched for a while" does not mean
|
||||
"abandoned". An earlier fix bought that safety by keeping EVERY finished session for the full
|
||||
ten-minute idle window — one segmented copy of a film per item, on the volume the download center
|
||||
shares. Its successor replaced the blanket retention with the signal that already existed: a player
|
||||
parked mid-item checkpoints its position every 10 s, paused or not, and that route says "still
|
||||
parked mid-item keeps checkpointing its position, paused or not, and that route says "still
|
||||
here" (it stops once the item is finished, where reaping is correct).
|
||||
|
||||
Both call sites are pinned here, because the bug each time was that only one of them moved: the
|
||||
@@ -81,8 +81,12 @@ class _Proc:
|
||||
return 0
|
||||
|
||||
|
||||
_PAST_THE_RULE = stream._FINISHED_IDLE_S + 5
|
||||
|
||||
|
||||
class _Session:
|
||||
def __init__(self, tmp_path, name: str, finished: bool, idle_s: float):
|
||||
self.key = name
|
||||
self.dir = tmp_path / name
|
||||
self.dir.mkdir()
|
||||
self.proc = _Proc(finished)
|
||||
@@ -97,11 +101,29 @@ def sessions(monkeypatch):
|
||||
|
||||
|
||||
def test_the_periodic_pass_reaps_a_finished_session_nobody_has_claimed(tmp_path, sessions):
|
||||
sessions["12345"] = _Session(tmp_path, "done", finished=True, idle_s=45)
|
||||
sessions["12345"] = _Session(tmp_path, "done", finished=True, idle_s=_PAST_THE_RULE)
|
||||
assert stream.reap_idle() == 1
|
||||
assert sessions == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("idle_s", [65, 125])
|
||||
def test_a_heartbeat_throttled_to_once_a_minute_keeps_a_finished_session(
|
||||
tmp_path, sessions, idle_s
|
||||
):
|
||||
"""A hidden page that makes no sound gets its timers cut to one a minute (measured on a paused and
|
||||
on a muted player, about a minute after hiding), so that is the viewer's heartbeat there: the
|
||||
reaper can see a beat over a minute old, and two if one is lost on the way."""
|
||||
s = _Session(tmp_path, "hidden", finished=True, idle_s=idle_s)
|
||||
sessions[(1, "12345")] = s
|
||||
assert stream.reap_idle() == 0
|
||||
assert (1, "12345") in sessions and s.dir.exists() and not s.proc.killed
|
||||
|
||||
|
||||
def test_an_abandoned_finished_session_is_still_gone_within_minutes(tmp_path, sessions):
|
||||
sessions[(1, "12345")] = _Session(tmp_path, "gone", finished=True, idle_s=185)
|
||||
assert stream.reap_idle() == 1
|
||||
|
||||
|
||||
def test_the_periodic_pass_still_reaps_what_nobody_has_touched_for_the_idle_limit(
|
||||
tmp_path, sessions
|
||||
):
|
||||
@@ -115,8 +137,32 @@ def test_the_periodic_pass_still_reaps_what_nobody_has_touched_for_the_idle_limi
|
||||
assert sessions == {}
|
||||
|
||||
|
||||
def test_every_reap_says_which_rule_took_the_session(tmp_path, sessions, caplog):
|
||||
"""A session that vanishes leaves the viewer a 410 and nothing else: with no line here, a live
|
||||
stall could only ever be blamed on the reaper by inference."""
|
||||
sessions["1"] = _Session(
|
||||
tmp_path, "gone_done", finished=True, idle_s=stream._SESSION_IDLE_S - 5
|
||||
)
|
||||
sessions["2"] = _Session(
|
||||
tmp_path, "gone_live", finished=False, idle_s=stream._SESSION_IDLE_S + 5
|
||||
)
|
||||
sessions["3"] = _Session(tmp_path, "kept", finished=False, idle_s=5)
|
||||
with caplog.at_level("INFO"):
|
||||
assert stream.reap_idle() == 2
|
||||
reaped = {
|
||||
r.attrs["key"]: r.attrs
|
||||
for r in caplog.records
|
||||
if getattr(r, "event", "") == "plex.stream.session_reaped"
|
||||
}
|
||||
assert set(reaped) == {"gone_done", "gone_live"}
|
||||
assert reaped["gone_done"]["reason"] == "finished_idle"
|
||||
assert reaped["gone_live"]["reason"] == "idle"
|
||||
assert reaped["gone_done"]["idle_s"] == pytest.approx(stream._SESSION_IDLE_S - 5, abs=2)
|
||||
assert reaped["gone_live"]["idle_s"] == pytest.approx(stream._SESSION_IDLE_S + 5, abs=2)
|
||||
|
||||
|
||||
def test_touching_a_session_keeps_it_through_the_short_rule(tmp_path, sessions):
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
|
||||
sessions[(1, "12345")] = s
|
||||
stream.touch_session(1, "12345")
|
||||
assert stream.reap_idle() == 0
|
||||
@@ -159,7 +205,7 @@ def test_a_paused_viewers_checkpoint_keeps_their_session_alive(
|
||||
user = User(email="paused@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
|
||||
sessions[(user.id, "paused1")] = s
|
||||
resp = _client(db, user, override_dep).post(
|
||||
f"/api/plex/item/{it.rating_key}/progress",
|
||||
@@ -174,7 +220,7 @@ def test_starting_a_session_leaves_another_viewers_paused_one_alone(
|
||||
db, monkeypatch, tmp_path, sessions, override_dep
|
||||
):
|
||||
"""The opportunistic sweep this route used to run was not scoped to the key being started: it
|
||||
walked the WHOLE registry with the 30 s finished rule, so pressing play on any item killed a
|
||||
walked the WHOLE registry with the finished rule, so pressing play on any item killed a
|
||||
paused viewer's session — the very case the retention fix set out to protect."""
|
||||
monkeypatch.setattr(state, "_configured_cache", True)
|
||||
monkeypatch.setattr(plex_paths, "local_media_path", lambda db_, fp: "/local/a.mkv")
|
||||
@@ -182,7 +228,7 @@ def test_starting_a_session_leaves_another_viewers_paused_one_alone(
|
||||
user = User(email="starter@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
paused = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
paused = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
|
||||
sessions[(999, "someone-else")] = paused
|
||||
|
||||
def fake_start(db_, item, user_id, start, audio=None, aoff=0.0, multi=False, transcode=False):
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
"""Tearing a session down does not stall everybody else's playback.
|
||||
|
||||
Every segment request (`current_session`) and every checkpoint (`touch_session`) takes the one
|
||||
registry `_lock`. Stopping a session waits up to 3 s for its ffmpeg to exit and then deletes a whole
|
||||
segmented film, ~1200 files — and that used to happen under the same lock, so one viewer's seek, a
|
||||
cap cull or a reap pass held up every other viewer's HLS requests for seconds, each one a parked
|
||||
threadpool worker and its pooled DB connection.
|
||||
|
||||
What still has to be atomic is pinned too: a session directory is reused for the same item, offset
|
||||
and tag, so a new session on that path must not start before the old ffmpeg — which writes by path
|
||||
— has exited.
|
||||
"""
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from app.plex import stream
|
||||
|
||||
|
||||
class _SlowProc:
|
||||
"""An ffmpeg that takes a while to exit after TERM: `wait` blocks until `exited` is set."""
|
||||
|
||||
def __init__(self, order: list[str], name: str):
|
||||
self.order = order
|
||||
self.name = name
|
||||
self.waiting = threading.Event()
|
||||
self.exited = threading.Event()
|
||||
|
||||
def poll(self):
|
||||
return 0 if self.exited.is_set() else None
|
||||
|
||||
def terminate(self):
|
||||
pass
|
||||
|
||||
def kill(self):
|
||||
self.exited.set()
|
||||
|
||||
def wait(self, timeout=None):
|
||||
self.waiting.set()
|
||||
if not self.exited.wait(timeout):
|
||||
raise stream.subprocess.TimeoutExpired("ffmpeg", timeout)
|
||||
self.order.append(f"{self.name} exited")
|
||||
return 0
|
||||
|
||||
|
||||
class _Session:
|
||||
def __init__(self, directory, proc, key: str):
|
||||
self.id = f"sid-{key}"
|
||||
self.key = key
|
||||
self.dir = directory
|
||||
self.proc = proc
|
||||
self.last_access = time.time()
|
||||
self.transcode = False
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sessions(monkeypatch):
|
||||
live: dict = {}
|
||||
monkeypatch.setattr(stream, "_sessions", live)
|
||||
monkeypatch.setattr(stream, "_dying", {}, raising=False)
|
||||
return live
|
||||
|
||||
|
||||
def test_stopping_a_session_does_not_hold_up_another_viewers_requests(tmp_path, sessions):
|
||||
order: list[str] = []
|
||||
slow = _SlowProc(order, "a")
|
||||
d = tmp_path / "1_u1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
(d / "seg_0.ts").write_bytes(b"x")
|
||||
sessions[(1, "a")] = _Session(d, slow, "a")
|
||||
sessions[(2, "b")] = _Session(tmp_path / "2_u2_0_0_+0.00", _SlowProc(order, "b"), "b")
|
||||
|
||||
stopper = threading.Thread(target=stream.stop_session, args=(1, "a", "sid-a"))
|
||||
stopper.start()
|
||||
try:
|
||||
assert slow.waiting.wait(2.0), "the stop never waited for its ffmpeg"
|
||||
got: list = []
|
||||
reader = threading.Thread(target=lambda: got.append(stream.current_session(2, "b")))
|
||||
reader.start()
|
||||
reader.join(1.0)
|
||||
assert not reader.is_alive(), "a segment request waited on another viewer's teardown"
|
||||
assert got and got[0] is sessions[(2, "b")]
|
||||
finally:
|
||||
slow.exited.set()
|
||||
stopper.join(5.0)
|
||||
# The teardown still finishes: the session is gone and nothing of its directory is left behind.
|
||||
assert (1, "a") not in sessions
|
||||
assert sorted(p.name for p in tmp_path.iterdir()) == []
|
||||
|
||||
|
||||
def test_a_reap_pass_does_not_hold_up_another_viewers_requests(tmp_path, sessions):
|
||||
order: list[str] = []
|
||||
slow = _SlowProc(order, "a")
|
||||
d = tmp_path / "1_u1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
gone = _Session(d, slow, "a")
|
||||
gone.last_access = time.time() - stream._SESSION_IDLE_S - 5
|
||||
sessions[(1, "a")] = gone
|
||||
sessions[(2, "b")] = _Session(tmp_path / "2_u2_0_0_+0.00", _SlowProc(order, "b"), "b")
|
||||
|
||||
reaper = threading.Thread(target=stream.reap_idle)
|
||||
reaper.start()
|
||||
try:
|
||||
assert slow.waiting.wait(2.0), "the reap never waited for its ffmpeg"
|
||||
toucher = threading.Thread(target=stream.touch_session, args=(2, "b"))
|
||||
toucher.start()
|
||||
toucher.join(1.0)
|
||||
assert not toucher.is_alive(), "a checkpoint waited on another viewer's teardown"
|
||||
finally:
|
||||
slow.exited.set()
|
||||
reaper.join(5.0)
|
||||
assert list(sessions) == [(2, "b")]
|
||||
assert not d.exists()
|
||||
|
||||
|
||||
class _FakeItem:
|
||||
rating_key = "12345"
|
||||
file_path = "/media/movie.mkv"
|
||||
playable = "remux"
|
||||
codec_video = "h264"
|
||||
|
||||
|
||||
def test_a_new_session_on_a_dying_ones_path_starts_only_after_it_exited(
|
||||
tmp_path, sessions, monkeypatch
|
||||
):
|
||||
"""The same item at the same second gets the same directory. The old ffmpeg writes by PATH, so
|
||||
while it is still winding down after TERM it can write a segment — or, on a clean exit, a final
|
||||
playlist with ENDLIST — into the NEW session's directory."""
|
||||
order: list[str] = []
|
||||
procs = iter([_SlowProc(order, "old"), _SlowProc(order, "new")])
|
||||
|
||||
def popen(*a, **k):
|
||||
order.append("popen")
|
||||
return next(procs)
|
||||
|
||||
monkeypatch.setattr(stream, "_HLS_ROOT", tmp_path)
|
||||
src = tmp_path / "movie.mkv"
|
||||
src.write_bytes(b"x")
|
||||
monkeypatch.setattr(stream.paths, "local_media_path", lambda db, fp: src)
|
||||
monkeypatch.setattr(stream.sysconfig, "get_int", lambda db, k: 2)
|
||||
monkeypatch.setattr(stream.sysconfig, "get_bool", lambda db, k: False)
|
||||
monkeypatch.setattr(stream.subprocess, "Popen", popen)
|
||||
monkeypatch.setattr(stream, "wait_for", lambda *a, **k: True)
|
||||
|
||||
first = stream.start_session(db=None, user_id=7, item=_FakeItem(), start_s=0.0)
|
||||
assert first is not None
|
||||
stopper = threading.Thread(target=stream.stop_session, args=(7, "12345", first.id))
|
||||
stopper.start()
|
||||
assert first.proc.waiting.wait(2.0)
|
||||
threading.Timer(0.3, first.proc.exited.set).start()
|
||||
second = stream.start_session(db=None, user_id=7, item=_FakeItem(), start_s=0.0)
|
||||
stopper.join(5.0)
|
||||
assert second is not None and second.dir == first.dir
|
||||
assert order.index("old exited") < order.index("popen", 1)
|
||||
assert second.dir.is_dir(), "the old teardown deleted the new session's directory"
|
||||
second.proc.exited.set()
|
||||
@@ -1,53 +1,88 @@
|
||||
"""No refusal answers in a hand-written English sentence.
|
||||
"""No refusal answers in a way the client cannot say.
|
||||
|
||||
The UI is bilingual and translates a structured `detail` (`{code, reason, …}`), but a string detail it
|
||||
can only print as it came — so ~150 of them reached the Hungarian UI verbatim, in the blocking error
|
||||
dialog and on every inline surface that shows `HttpError.detail`. `app/refusal.py` is the family they
|
||||
became; this test is what keeps a new one from being written the old way. Pure: it reads the sources.
|
||||
became; this test is what keeps a new one from being written the old way. It reads the same slots the
|
||||
refusal scan does (`refusal_sites.detail_slots`), so the two cannot disagree on where a detail is sent.
|
||||
Pure: it reads the sources.
|
||||
"""
|
||||
|
||||
import ast
|
||||
import pathlib
|
||||
import textwrap
|
||||
|
||||
APP = pathlib.Path(__file__).resolve().parents[1] / "app"
|
||||
from tests.refusal_sites import Scan, app_sources
|
||||
|
||||
# Answers that are no refusal of the app's, by the function they are sent from: the client shows a
|
||||
# detail only when it is an object (`lib/api/core.ts`), so these read as its generic line, and no screen
|
||||
# needs more from them.
|
||||
NO_REFUSAL = {
|
||||
("main.py", "setup_gate"): "the 503 before setup: the setup screen is what the user sees",
|
||||
("main.py", "spa_fallback"): "a bare 404 for an api path no route matches: no screen calls one",
|
||||
("main.py", "_validation_error"): "FastAPI's own 422 list: a body the client itself got wrong",
|
||||
("routes/diagnostics.py", "_parsed_events"): "the client-events beacon's 422: nothing shows it",
|
||||
}
|
||||
|
||||
|
||||
def _is_text(node: ast.AST) -> bool:
|
||||
"""A detail that is a sentence: a literal, an f-string, `str(exc)`, or `"…" + x`."""
|
||||
if isinstance(node, ast.Constant) and isinstance(node.value, str):
|
||||
return True
|
||||
if isinstance(node, ast.JoinedStr):
|
||||
return True
|
||||
if isinstance(node, ast.Call) and getattr(node.func, "id", "") == "str":
|
||||
return True
|
||||
if isinstance(node, ast.BoolOp):
|
||||
return any(_is_text(v) for v in node.values)
|
||||
if isinstance(node, ast.BinOp):
|
||||
return _is_text(node.left) or _is_text(node.right)
|
||||
return False
|
||||
def _offenders(sources: dict[str, str]) -> list[str]:
|
||||
return [
|
||||
f"{path}:{line}"
|
||||
for path, line, fn in Scan(sources).other_details()
|
||||
if (path, fn) not in NO_REFUSAL
|
||||
]
|
||||
|
||||
|
||||
def test_no_http_exception_carries_a_sentence():
|
||||
offenders = []
|
||||
for path in sorted(APP.rglob("*.py")):
|
||||
for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))):
|
||||
if not (
|
||||
isinstance(node, ast.Call)
|
||||
and getattr(node.func, "id", getattr(node.func, "attr", "")) == "HTTPException"
|
||||
):
|
||||
continue
|
||||
kw = {k.arg: k.value for k in node.keywords}
|
||||
detail = kw.get("detail", node.args[1] if len(node.args) > 1 else None)
|
||||
if detail is not None and _is_text(detail):
|
||||
offenders.append(f"{path.relative_to(APP).as_posix()}:{node.lineno}")
|
||||
def test_every_detail_is_one_the_client_can_say():
|
||||
offenders = _offenders(app_sources())
|
||||
assert offenders == [], (
|
||||
"a refusal with an English sentence for a detail — raise `refused(status, reason)` "
|
||||
f"(app/refusal.py) and name the reason in both locales: {offenders}"
|
||||
"a refusal the client can only answer with its generic line — send `{code, reason}` "
|
||||
f"(`refused(status, reason)`), or list it in NO_REFUSAL with why: {offenders}"
|
||||
)
|
||||
|
||||
|
||||
def test_the_scan_sees_a_sentence_when_there_is_one():
|
||||
def test_the_detail_scan_sees_each_shape_the_client_cannot_say():
|
||||
"""A guard that matches nothing is green over nothing."""
|
||||
sample = ast.parse('HTTPException(status_code=400, detail=f"bad {x}")').body[0].value
|
||||
assert _is_text(sample.keywords[1].value)
|
||||
assert not _is_text(ast.parse('{"code": "refused", "reason": "x"}').body[0].value)
|
||||
src = textwrap.dedent(
|
||||
"""
|
||||
def f(x):
|
||||
raise HTTPException(409, detail={"reason": "view_name_taken"})
|
||||
def g(x):
|
||||
raise HTTPException(409, dict(reason="taken"))
|
||||
def h(x):
|
||||
raise HTTPException(status_code=404)
|
||||
def i(x):
|
||||
return JSONResponse({"detail": "This instance isn't set up yet."}, status_code=503)
|
||||
def j(x):
|
||||
return JSONResponse(content={"detail": {"reason": "x"}}, status_code=409)
|
||||
def k(x):
|
||||
raise HTTPException(400, detail=f"bad {x}")
|
||||
def l(x):
|
||||
raise HTTPException(404, None)
|
||||
def n(x, msg):
|
||||
raise HTTPException(400, detail=msg)
|
||||
def o(x):
|
||||
raise HTTPException(400, detail=["a sentence"])
|
||||
def p(x):
|
||||
raise HTTPException(400, detail=describe(x))
|
||||
def q(x):
|
||||
raise HTTPException(400, detail={"code": "a"} if x else "text")
|
||||
def r(x):
|
||||
raise HTTPException(400, detail=nothing(x))
|
||||
def nothing(x):
|
||||
pass
|
||||
def ok(x):
|
||||
raise HTTPException(409, {"code": "refused", "reason": "x"})
|
||||
raise HTTPException(409, detail=quota(x))
|
||||
raise HTTPException(409, detail={"code": "a"} if x else {"code": "b"})
|
||||
return JSONResponse({"detail": {"code": "too_large"}}, status_code=413)
|
||||
return JSONResponse({"ok": True})
|
||||
def quota(x):
|
||||
return {"code": "quota", "reason": "files"}
|
||||
def setup_gate(x):
|
||||
return JSONResponse({"detail": "listed"}, status_code=503)
|
||||
"""
|
||||
)
|
||||
lines = (3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25)
|
||||
assert _offenders({"main.py": src}) == [f"main.py:{n}" for n in lines]
|
||||
# A file whose code dicts are data still answers its refusals in a way the client can say.
|
||||
data = 'def f():\n raise HTTPException(400, "text")\n'
|
||||
assert _offenders({"plex/tmdb.py": data}) == ["plex/tmdb.py:2"]
|
||||
|
||||
@@ -45,6 +45,9 @@ def test_an_interval_override_moves_into_sysconfig_and_back(db):
|
||||
}
|
||||
assert conn.execute(text("SELECT to_regclass('scheduler_settings')")).scalar() is None
|
||||
assert scheduler.load_intervals(db)["rss_poll"] == 25
|
||||
# The way down rebuilds an index CONCURRENTLY (0117), which waits for every open transaction —
|
||||
# the read above left this session's open.
|
||||
db.rollback()
|
||||
|
||||
command.downgrade(cfg, "0109_job_runs")
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Migration 0119's data step, DB-lane: the stored `transcode_hw` facet follows the rule that it says
|
||||
whether the GPU does the encode, and a 10-bit decodable item is probed again so its HDR flag decides.
|
||||
|
||||
The schema half is checked by test_migrations.py; a facet value left stale is invisible there.
|
||||
"""
|
||||
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db import engine
|
||||
|
||||
|
||||
def test_the_stored_facet_is_rewritten_by_the_gpu_encode_rule(db):
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0118_videos_shuffle_key")
|
||||
try:
|
||||
with engine.begin() as conn:
|
||||
lib = conn.execute(
|
||||
text(
|
||||
"INSERT INTO plex_libraries (plex_key, title, kind, enabled)"
|
||||
" VALUES ('m1', 'Movies', 'movie', true) RETURNING id"
|
||||
)
|
||||
).scalar_one()
|
||||
|
||||
def item(key, playable, codec, hw, depth=None, probed=False):
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO plex_items (rating_key, library_id, kind, title, playable,"
|
||||
" codec_video, transcode_hw, video_bit_depth, media_probed_at)"
|
||||
" VALUES (:k, :l, 'movie', :k, :p, :c, :hw, :d,"
|
||||
" CASE WHEN :pr THEN now() END)"
|
||||
),
|
||||
{
|
||||
"k": key,
|
||||
"l": lib,
|
||||
"p": playable,
|
||||
"c": codec,
|
||||
"hw": hw,
|
||||
"d": depth,
|
||||
"pr": probed,
|
||||
},
|
||||
)
|
||||
|
||||
item("vc1", "transcode", "VC1", True, 8, probed=True) # the old list said decodable
|
||||
item("xvid", "transcode", "mpeg4", False)
|
||||
item("divx", "transcode", "msmpeg4v3", False)
|
||||
item("av1", "transcode", "av1", False)
|
||||
item("h8", "transcode", "hevc", True, 8, probed=True)
|
||||
item("h10", "transcode", "hevc", False, 10, probed=True) # SDR or HDR: unknown here
|
||||
item("remux", "remux", "h264", None)
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
with engine.connect() as conn:
|
||||
rows = {
|
||||
k: (hw, probed)
|
||||
for k, hw, probed in conn.execute(
|
||||
text("SELECT rating_key, transcode_hw, media_probed_at IS NOT NULL FROM plex_items")
|
||||
).tuples()
|
||||
}
|
||||
assert rows == {
|
||||
"vc1": (True, True),
|
||||
"xvid": (True, False),
|
||||
"divx": (True, False),
|
||||
"av1": (False, False),
|
||||
"h8": (True, True),
|
||||
"h10": (False, False), # re-queued: the probe reads its HDR flag again
|
||||
"remux": (None, False),
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
"""A rename onto a name already taken says WHICH name is taken: the one asked for, not the row's
|
||||
old name (the rollback that follows the conflict puts the old name back on the row). DB lane.
|
||||
"""
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import app.state as state
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.models import User
|
||||
|
||||
|
||||
def _client(db, monkeypatch, override_dep):
|
||||
u = User(email="taken@example.com")
|
||||
db.add(u)
|
||||
db.commit()
|
||||
monkeypatch.setattr(
|
||||
state, "_configured_cache", True
|
||||
) # the request gate 503s an unconfigured app
|
||||
override_dep(get_db, db.info["request"])
|
||||
override_dep(current_user, lambda: u)
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_renaming_a_view_onto_a_taken_name_names_the_taken_one(db, monkeypatch, override_dep):
|
||||
c = _client(db, monkeypatch, override_dep)
|
||||
gaming = c.post("/api/saved-views", json={"name": "gaming", "filters": {}, "source": "feed"})
|
||||
assert gaming.status_code == 200
|
||||
hh = c.post("/api/saved-views", json={"name": "hh", "filters": {}, "source": "feed"}).json()
|
||||
|
||||
r = c.patch(f"/api/saved-views/{hh['id']}", json={"name": "gaming"})
|
||||
|
||||
assert r.status_code == 409
|
||||
assert r.json()["detail"] == {"code": "refused", "reason": "view_name_taken", "name": "gaming"}
|
||||
|
||||
|
||||
def test_renaming_a_tag_onto_a_taken_name_names_the_taken_one(db, monkeypatch, override_dep):
|
||||
c = _client(db, monkeypatch, override_dep)
|
||||
assert c.post("/api/tags", json={"name": "music"}).status_code == 200
|
||||
news = c.post("/api/tags", json={"name": "news"}).json()
|
||||
|
||||
r = c.patch(f"/api/tags/{news['id']}", json={"name": "music"})
|
||||
|
||||
assert r.status_code == 409
|
||||
assert r.json()["detail"] == {"code": "refused", "reason": "tag_name_taken", "name": "music"}
|
||||
@@ -181,7 +181,7 @@ def test_clearing_a_video_leaves_another_users_state(db, make_user, as_user):
|
||||
_videos(db, "v1")
|
||||
db.add(VideoState(user_id=other.id, video_id="v1", status="hidden", position_seconds=90))
|
||||
db.commit()
|
||||
assert as_user(me).delete("/api/videos/v1/state").status_code == 200
|
||||
assert as_user(me).delete("/api/videos/v1/progress").status_code == 200
|
||||
assert _video_state(db, other) == ("hidden", 90)
|
||||
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Adding a video AT a position: the list page's "Undo" of a removal.
|
||||
"""Putting a video back AT a position: the list page's "Undo" of a removal.
|
||||
|
||||
The row has to come back where it was in ONE request. The two-call shape (append, then send the
|
||||
whole order) leaves it at the end of the list whenever the second call fails, and the reorder would
|
||||
@@ -38,14 +38,17 @@ def _ids(db, pl_id):
|
||||
]
|
||||
|
||||
|
||||
def _add(client, pl_id, body):
|
||||
return client.post(f"/api/playlists/{pl_id}/items", json=body)
|
||||
def _restore(client, pl_id, video_id, position):
|
||||
return client.post(
|
||||
f"/api/playlists/{pl_id}/items/restore",
|
||||
json={"video_ids": [video_id], "positions": [position]},
|
||||
)
|
||||
|
||||
|
||||
def test_a_position_puts_the_video_back_in_its_place(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b", "c"])
|
||||
r = _add(as_user(user), pl.id, {"video_id": "vX", "position": 1})
|
||||
r = _restore(as_user(user), pl.id, "vX", 1)
|
||||
assert r.status_code == 200
|
||||
assert _ids(db, pl.id) == ["a", "vX", "b", "c"]
|
||||
|
||||
@@ -54,36 +57,37 @@ def test_the_position_counts_rows_not_stored_numbers(db, make_user, as_user):
|
||||
# A removal leaves a gap (0, 5, 9): index 2 is before the THIRD row, not "position 2".
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b", "c"], positions=[0, 5, 9])
|
||||
_add(as_user(user), pl.id, {"video_id": "vX", "position": 2})
|
||||
_restore(as_user(user), pl.id, "vX", 2)
|
||||
assert _ids(db, pl.id) == ["a", "b", "vX", "c"]
|
||||
|
||||
|
||||
def test_a_position_past_the_end_appends(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b"])
|
||||
_add(as_user(user), pl.id, {"video_id": "vX", "position": 40})
|
||||
_restore(as_user(user), pl.id, "vX", 40)
|
||||
assert _ids(db, pl.id) == ["a", "b", "vX"]
|
||||
|
||||
|
||||
def test_no_position_still_appends(db, make_user, as_user):
|
||||
def test_an_add_appends(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b"])
|
||||
_add(as_user(user), pl.id, {"video_id": "vX"})
|
||||
r = as_user(user).post(f"/api/playlists/{pl.id}/items", json={"video_id": "vX"})
|
||||
assert r.status_code == 200
|
||||
assert _ids(db, pl.id) == ["a", "b", "vX"]
|
||||
|
||||
|
||||
def test_a_video_already_in_the_list_does_not_move(db, make_user, as_user):
|
||||
# The add is idempotent; a position must not turn it into a reorder.
|
||||
# A restore is idempotent; a position must not turn it into a reorder.
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b", "c"])
|
||||
_add(as_user(user), pl.id, {"video_id": "c", "position": 0})
|
||||
_restore(as_user(user), pl.id, "c", 0)
|
||||
assert _ids(db, pl.id) == ["a", "b", "c"]
|
||||
|
||||
|
||||
def test_a_negative_position_is_refused(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a"])
|
||||
r = _add(as_user(user), pl.id, {"video_id": "vX", "position": -1})
|
||||
r = _restore(as_user(user), pl.id, "vX", -1)
|
||||
assert r.status_code == 422
|
||||
assert _ids(db, pl.id) == ["a"]
|
||||
|
||||
@@ -101,6 +105,6 @@ def test_putting_a_removed_video_back_leaves_a_linked_list_clean(db, make_user,
|
||||
assert client.delete(f"/api/playlists/{pl.id}/items/b").status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).dirty is True
|
||||
_add(client, pl.id, {"video_id": "b", "position": 1})
|
||||
_restore(client, pl.id, "b", 1)
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).dirty is False
|
||||
|
||||
@@ -0,0 +1,640 @@
|
||||
"""Many videos at once: remove, move, and the Undo of each. Every test reads both lists after the call,
|
||||
as `test_playlist_move` does: a half-applied move is the failure these routes exist to rule out."""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.catalog.models import Channel, Video
|
||||
from app.reader import queries, service
|
||||
from app.reader.models import Playlist, PlaylistItem
|
||||
from app.sync.playlists import current_fingerprint
|
||||
|
||||
|
||||
def _seed_videos(db, ids):
|
||||
db.add(Channel(id="UCtest", title="Test"))
|
||||
now = datetime.now(timezone.utc)
|
||||
for v in ids:
|
||||
db.add(Video(id=v, channel_id="UCtest", published_at=now))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _playlist(db, user, name, video_ids, **kw):
|
||||
pl = Playlist(user_id=user.id, name=name, **kw)
|
||||
db.add(pl)
|
||||
db.flush()
|
||||
for i, vid in enumerate(video_ids):
|
||||
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=i))
|
||||
db.commit()
|
||||
return pl
|
||||
|
||||
|
||||
def _ids(db, pl_id):
|
||||
db.expire_all()
|
||||
return list(
|
||||
db.execute(
|
||||
select(PlaylistItem.video_id)
|
||||
.where(PlaylistItem.playlist_id == pl_id)
|
||||
.order_by(PlaylistItem.position)
|
||||
).scalars()
|
||||
)
|
||||
|
||||
|
||||
def _positions(db, pl_id):
|
||||
db.expire_all()
|
||||
return list(
|
||||
db.execute(
|
||||
select(PlaylistItem.position)
|
||||
.where(PlaylistItem.playlist_id == pl_id)
|
||||
.order_by(PlaylistItem.position)
|
||||
).scalars()
|
||||
)
|
||||
|
||||
|
||||
def test_remove_bulk_takes_out_the_named_videos_and_counts_what_was_there(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "d"])
|
||||
pl = _playlist(db, user, "L", ["a", "b", "c", "d"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["b", "d", "gone"]}
|
||||
)
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {"ok": True, "playlist_id": pl.id, "removed": ["b", "d"]}
|
||||
assert _ids(db, pl.id) == ["a", "c"]
|
||||
|
||||
|
||||
def test_remove_bulk_of_nothing_leaves_the_list_untouched(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
pl = _playlist(db, user, "L", ["a"])
|
||||
stamp = datetime.now(timezone.utc) - timedelta(days=3)
|
||||
pl.updated_at = stamp
|
||||
db.commit()
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["gone"]}
|
||||
)
|
||||
|
||||
assert r.json()["removed"] == []
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).updated_at == stamp
|
||||
|
||||
|
||||
def test_restore_puts_the_videos_back_at_their_places(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "d"])
|
||||
pl = _playlist(db, user, "L", ["b", "d"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/restore",
|
||||
json={"video_ids": ["c", "a"], "positions": [2, 0]},
|
||||
)
|
||||
|
||||
assert r.json() == {"ok": True, "playlist_id": pl.id, "added": 2}
|
||||
assert _ids(db, pl.id) == ["a", "b", "c", "d"]
|
||||
assert _positions(db, pl.id) == [0, 1, 2, 3]
|
||||
|
||||
|
||||
def test_restore_leaves_a_video_already_back_where_it_is_and_skips_one_the_catalog_lost(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
pl = _playlist(db, user, "L", ["b", "a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/restore",
|
||||
json={"video_ids": ["a", "lost"], "positions": [0, 1]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 0
|
||||
assert _ids(db, pl.id) == ["b", "a"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"body",
|
||||
[
|
||||
{"video_ids": ["a"], "positions": []},
|
||||
{"video_ids": ["a"], "positions": [0, 1]},
|
||||
{"video_ids": ["a"], "positions": [-1]},
|
||||
],
|
||||
)
|
||||
def test_restore_refuses_positions_that_do_not_pair(db, make_user, as_user, body):
|
||||
user = make_user()
|
||||
pl = _playlist(db, user, "L", [])
|
||||
|
||||
assert as_user(user).post(f"/api/playlists/{pl.id}/items/restore", json=body).status_code == 422
|
||||
|
||||
|
||||
def test_move_bulk_lands_in_the_sources_order_whatever_order_they_were_picked_in(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "x"])
|
||||
src = _playlist(db, user, "Source", ["a", "b", "c"])
|
||||
dst = _playlist(db, user, "Target", ["x"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["c", "a"], "target_playlist_id": dst.id},
|
||||
)
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {
|
||||
"ok": True,
|
||||
"from_playlist_id": src.id,
|
||||
"to_playlist_id": dst.id,
|
||||
"moved": ["a", "c"],
|
||||
"already_there": [],
|
||||
}
|
||||
assert _ids(db, src.id) == ["b"]
|
||||
assert _ids(db, dst.id) == ["x", "a", "c"]
|
||||
|
||||
|
||||
def test_move_bulk_says_which_the_target_already_held_and_skips_one_already_gone(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c"])
|
||||
src = _playlist(db, user, "Source", ["a", "b"])
|
||||
dst = _playlist(db, user, "Target", ["b"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a", "b", "c"], "target_playlist_id": dst.id},
|
||||
)
|
||||
|
||||
assert r.json()["moved"] == ["a"]
|
||||
assert r.json()["already_there"] == ["b"]
|
||||
assert _ids(db, src.id) == []
|
||||
assert _ids(db, dst.id) == ["b", "a"]
|
||||
|
||||
|
||||
def test_move_bulk_keeps_when_each_video_was_saved(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"])
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
saved = datetime(2020, 1, 2, tzinfo=timezone.utc)
|
||||
row = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == src.id))
|
||||
row.added_at = saved
|
||||
db.commit()
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": dst.id},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
db.expire_all()
|
||||
moved = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == dst.id))
|
||||
assert moved.added_at == saved
|
||||
|
||||
|
||||
def test_a_target_that_gained_nothing_does_not_rise_in_the_recently_changed_sort(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
stamp = datetime.now(timezone.utc) - timedelta(days=3)
|
||||
dst.updated_at = stamp
|
||||
db.commit()
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": dst.id},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, dst.id).updated_at == stamp
|
||||
|
||||
|
||||
def test_move_bulk_into_the_same_list_is_refused(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
pl = _playlist(db, user, "L", ["a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": pl.id},
|
||||
)
|
||||
|
||||
assert r.status_code == 400
|
||||
assert r.json()["detail"] == {"code": "playlist_move", "reason": "same_playlist"}
|
||||
assert _ids(db, pl.id) == ["a"]
|
||||
|
||||
|
||||
def test_move_bulk_marks_a_linked_list_on_both_sides_as_unsynced(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"], yt_playlist_id="PLs")
|
||||
dst = _playlist(db, user, "Target", [], yt_playlist_id="PLd")
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": dst.id},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, src.id).dirty is True
|
||||
assert db.get(Playlist, dst.id).dirty is True
|
||||
|
||||
|
||||
def test_undo_move_restores_both_lists_as_they_stood(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "d"])
|
||||
src = _playlist(db, user, "Source", ["a", "b", "c", "d"])
|
||||
dst = _playlist(db, user, "Target", ["c"])
|
||||
client = as_user(user)
|
||||
|
||||
moved = client.post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a", "c"], "target_playlist_id": dst.id},
|
||||
).json()
|
||||
assert _ids(db, src.id) == ["b", "d"]
|
||||
assert _ids(db, dst.id) == ["c", "a"]
|
||||
|
||||
r = client.post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={
|
||||
"from_playlist_id": dst.id,
|
||||
"moved": moved["moved"],
|
||||
"already_there": moved["already_there"],
|
||||
"positions": [0, 2],
|
||||
},
|
||||
)
|
||||
|
||||
assert r.json() == {"ok": True, "playlist_id": src.id, "added": 2}
|
||||
assert _ids(db, src.id) == ["a", "b", "c", "d"]
|
||||
# The one the target held before the move stays there.
|
||||
assert _ids(db, dst.id) == ["c"]
|
||||
|
||||
|
||||
def test_undo_move_keeps_when_each_video_was_saved(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
saved = datetime(2020, 1, 2, tzinfo=timezone.utc)
|
||||
row = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == dst.id))
|
||||
row.added_at = saved
|
||||
db.commit()
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
db.expire_all()
|
||||
back = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == src.id))
|
||||
assert back.added_at == saved
|
||||
|
||||
|
||||
def test_undo_move_leaves_a_video_that_moved_on_where_it_is(db, make_user, as_user):
|
||||
"""A to B, then B to C, then the first Undo: the video is not in the list the Undo takes it from,
|
||||
and coming back would leave it in two lists the reader never put it in."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", ["b"])
|
||||
third = _playlist(db, user, "Third", ["a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a", "b"], "positions": [0, 1]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
assert _ids(db, src.id) == ["b"]
|
||||
assert _ids(db, dst.id) == []
|
||||
assert _ids(db, third.id) == ["a"]
|
||||
|
||||
|
||||
def test_an_undo_closes_up_the_places_of_a_video_that_stayed_out(db, make_user, as_user):
|
||||
"""Each index counts the rows put back before it, so one that stays out moves the rest up."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "d"])
|
||||
src = _playlist(db, user, "Source", ["c", "d"])
|
||||
dst = _playlist(db, user, "Target", ["b"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a", "b"], "positions": [0, 1]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
assert _ids(db, src.id) == ["b", "c", "d"]
|
||||
|
||||
|
||||
def test_a_restore_closes_up_the_places_of_a_video_the_catalog_lost(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["b", "c", "d"])
|
||||
pl = _playlist(db, user, "L", ["c", "d"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/restore",
|
||||
json={"video_ids": ["lost", "b"], "positions": [0, 1]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
assert _ids(db, pl.id) == ["b", "c", "d"]
|
||||
|
||||
|
||||
def test_undo_move_takes_a_video_out_of_the_target_when_the_source_has_it_again(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 0
|
||||
assert _ids(db, src.id) == ["a"]
|
||||
assert _ids(db, dst.id) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "body", "gone", "reason"),
|
||||
[
|
||||
("move-bulk", {"video_ids": ["a"], "target_playlist_id": "dst"}, "dst", "move_target"),
|
||||
("move-bulk", {"video_ids": ["a"], "target_playlist_id": "dst"}, "src", "playlist"),
|
||||
(
|
||||
"undo-move",
|
||||
{"from_playlist_id": "dst", "moved": ["a"], "positions": [0]},
|
||||
"src",
|
||||
"playlist",
|
||||
),
|
||||
(
|
||||
"undo-move",
|
||||
{"from_playlist_id": "dst", "moved": ["a"], "positions": [0]},
|
||||
"dst",
|
||||
"move_target",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_a_list_deleted_before_the_lock_is_not_found(
|
||||
db, make_user, as_user, monkeypatch, path, body, gone, reason
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
ids = {"src": src.id, "dst": dst.id}
|
||||
body = {k: (dst.id if v == "dst" else v) for k, v in body.items()}
|
||||
# Another tab's delete committed between the route's read and this lock.
|
||||
monkeypatch.setattr(service, "lock_playlists", lambda _db, *got: set(got) - {ids[gone]})
|
||||
|
||||
r = as_user(user).post(f"/api/playlists/{src.id}/items/{path}", json=body)
|
||||
|
||||
assert r.status_code == 404
|
||||
assert reason in str(r.json()["detail"])
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path", "body"),
|
||||
[
|
||||
("post", "items", {"video_id": "a"}),
|
||||
("delete", "items/a", None),
|
||||
("post", "items/remove-bulk", {"video_ids": ["a"]}),
|
||||
("post", "items/restore", {"video_ids": ["a"], "positions": [0]}),
|
||||
],
|
||||
)
|
||||
def test_a_one_list_write_finds_its_list_under_the_lock(
|
||||
db, make_user, as_user, monkeypatch, method, path, body
|
||||
):
|
||||
"""Read with its lock, so a delete that won the row answers 404 rather than a foreign key's 500."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
pl = _playlist(db, user, "L", [])
|
||||
real = queries.own_playlist
|
||||
|
||||
def locked_read_finds_nothing(db_, user_, playlist_id, *, lock=False):
|
||||
return None if lock else real(db_, user_, playlist_id)
|
||||
|
||||
monkeypatch.setattr(queries, "own_playlist", locked_read_finds_nothing)
|
||||
kw = {"json": body} if body is not None else {}
|
||||
|
||||
r = getattr(as_user(user), method)(f"/api/playlists/{pl.id}/{path}", **kw)
|
||||
|
||||
assert r.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "body"),
|
||||
[
|
||||
("items/remove-bulk", {"ids": ["a"]}),
|
||||
("items/move-bulk", {"ids": ["a"], "target_playlist_id": 1}),
|
||||
("items/undo-move", {"ids": ["a"], "from_playlist_id": 1}),
|
||||
],
|
||||
)
|
||||
def test_a_body_without_its_videos_is_refused_rather_than_acting_on_none(
|
||||
db, make_user, as_user, path, body
|
||||
):
|
||||
user = make_user()
|
||||
pl = _playlist(db, user, "L", [])
|
||||
|
||||
assert as_user(user).post(f"/api/playlists/{pl.id}/{path}", json=body).status_code == 422
|
||||
|
||||
|
||||
def test_an_undo_of_only_videos_the_target_held_names_no_moved_ones(db, make_user, as_user):
|
||||
"""An empty `moved` is a real body: every video of the move was already in the target."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": [], "already_there": ["a"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
assert _ids(db, src.id) == ["a"]
|
||||
assert _ids(db, dst.id) == ["a"]
|
||||
|
||||
|
||||
def test_an_undo_within_one_list_is_refused_and_keeps_its_videos(db, make_user, as_user):
|
||||
"""A move refuses the same list twice, so no Undo of one exists; a row must not go for asking."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
pl = _playlist(db, user, "L", ["a", "b"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/undo-move",
|
||||
json={"from_playlist_id": pl.id, "moved": ["a"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.status_code == 400
|
||||
assert r.json()["detail"] == {"code": "playlist_move", "reason": "same_playlist"}
|
||||
assert _ids(db, pl.id) == ["a", "b"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "key", "extra"),
|
||||
[
|
||||
("items/restore", "video_ids", {}),
|
||||
("items/undo-move", "moved", {"from_playlist_id": None}),
|
||||
],
|
||||
)
|
||||
def test_two_videos_cannot_ask_for_one_place(db, make_user, as_user, path, key, extra):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c"])
|
||||
pl = _playlist(db, user, "L", ["c"])
|
||||
other = _playlist(db, user, "Other", ["a", "b"])
|
||||
body = {key: ["a", "b"], "positions": [0, 0]}
|
||||
body.update({k: other.id for k in extra})
|
||||
|
||||
r = as_user(user).post(f"/api/playlists/{pl.id}/{path}", json=body)
|
||||
|
||||
assert r.status_code == 422
|
||||
assert _ids(db, pl.id) == ["c"]
|
||||
|
||||
|
||||
def test_a_reorder_without_its_videos_is_refused(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _playlist(db, user, "L", [])
|
||||
|
||||
assert as_user(user).put(f"/api/playlists/{pl.id}/order", json={"ids": []}).status_code == 422
|
||||
|
||||
|
||||
def _synced(db, pl):
|
||||
pl.synced_fingerprint = current_fingerprint(db, pl)
|
||||
pl.dirty = False
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_a_bulk_remove_and_its_undo_leave_a_linked_list_as_it_was(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
pl = _playlist(db, user, "L", ["a", "b"], yt_playlist_id="PLx")
|
||||
_synced(db, pl)
|
||||
client = as_user(user)
|
||||
|
||||
r = client.post(f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["a"]})
|
||||
assert r.status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).dirty is True
|
||||
|
||||
r = client.post(
|
||||
f"/api/playlists/{pl.id}/items/restore", json={"video_ids": ["a"], "positions": [0]}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).dirty is False
|
||||
|
||||
|
||||
def test_a_move_and_its_undo_leave_both_linked_lists_as_they_were(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
src = _playlist(db, user, "Source", ["a", "b"], yt_playlist_id="PLs")
|
||||
dst = _playlist(db, user, "Target", ["b"], yt_playlist_id="PLd")
|
||||
_synced(db, src)
|
||||
_synced(db, dst)
|
||||
client = as_user(user)
|
||||
|
||||
moved = client.post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": dst.id},
|
||||
).json()
|
||||
assert moved["moved"] == ["a"]
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, src.id).dirty is True
|
||||
assert db.get(Playlist, dst.id).dirty is True
|
||||
|
||||
r = client.post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, src.id).dirty is False
|
||||
assert db.get(Playlist, dst.id).dirty is False
|
||||
|
||||
|
||||
def test_undo_move_is_one_transaction(db, make_user, as_user, monkeypatch):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
|
||||
def boom(*a, **kw):
|
||||
raise RuntimeError("second step failed")
|
||||
|
||||
monkeypatch.setattr(service, "_insert_at", boom)
|
||||
client = as_user(user)
|
||||
body = {"from_playlist_id": dst.id, "moved": ["a"], "already_there": [], "positions": [0]}
|
||||
with pytest.raises(RuntimeError):
|
||||
_ = client.post(f"/api/playlists/{src.id}/items/undo-move", json=body)
|
||||
|
||||
db.rollback()
|
||||
assert _ids(db, dst.id) == ["a"]
|
||||
assert _ids(db, src.id) == []
|
||||
|
||||
|
||||
def test_undo_move_refuses_positions_that_do_not_pair(db, make_user, as_user):
|
||||
user = make_user()
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a"], "already_there": ["b"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "body"),
|
||||
[
|
||||
("remove-bulk", {"video_ids": ["a"]}),
|
||||
("restore", {"video_ids": ["a"], "positions": [0]}),
|
||||
("move-bulk", {"video_ids": ["a"], "target_playlist_id": 0}),
|
||||
("undo-move", {"from_playlist_id": 0, "moved": ["a"], "positions": [0]}),
|
||||
],
|
||||
)
|
||||
def test_another_accounts_list_is_not_found(db, make_user, as_user, path, body):
|
||||
owner, other = make_user(), make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
theirs = _playlist(db, owner, "Theirs", ["a"])
|
||||
mine = _playlist(db, other, "Mine", [])
|
||||
body = {
|
||||
k: (mine.id if k in ("target_playlist_id", "from_playlist_id") else v)
|
||||
for k, v in body.items()
|
||||
}
|
||||
|
||||
r = as_user(other).post(f"/api/playlists/{theirs.id}/items/{path}", json=body)
|
||||
|
||||
assert r.status_code == 404
|
||||
assert _ids(db, theirs.id) == ["a"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "key"), [("move-bulk", "target_playlist_id"), ("undo-move", "from_playlist_id")]
|
||||
)
|
||||
def test_the_other_list_of_a_move_must_be_the_callers_too(db, make_user, as_user, path, key):
|
||||
owner, other = make_user(), make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
theirs = _playlist(db, owner, "Theirs", ["a"])
|
||||
mine = _playlist(db, other, "Mine", ["a"])
|
||||
body = {"video_ids": ["a"], "moved": ["a"], "positions": [0], key: theirs.id}
|
||||
|
||||
r = as_user(other).post(f"/api/playlists/{mine.id}/items/{path}", json=body)
|
||||
|
||||
assert r.status_code == 404
|
||||
assert "move_target" in str(r.json()["detail"])
|
||||
assert _ids(db, theirs.id) == ["a"]
|
||||
assert _ids(db, mine.id) == ["a"]
|
||||
@@ -51,7 +51,8 @@ def _ids(db, pl_id):
|
||||
|
||||
def _move(client, src, video_id, dst):
|
||||
return client.post(
|
||||
f"/api/playlists/{src}/items/{video_id}/move", json={"target_playlist_id": dst}
|
||||
f"/api/playlists/{src}/items/move-bulk",
|
||||
json={"video_ids": [video_id], "target_playlist_id": dst},
|
||||
)
|
||||
|
||||
|
||||
@@ -67,7 +68,8 @@ def test_move_leaves_the_video_in_exactly_one_list(db, make_user, as_user):
|
||||
"ok": True,
|
||||
"from_playlist_id": src.id,
|
||||
"to_playlist_id": dst.id,
|
||||
"video_id": "v1",
|
||||
"moved": ["v1"],
|
||||
"already_there": [],
|
||||
}
|
||||
db.expire_all()
|
||||
assert _ids(db, src.id) == ["v2"]
|
||||
@@ -116,8 +118,9 @@ def test_move_of_a_video_that_left_the_source_reports_it(db, make_user, as_user)
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
|
||||
r = _move(as_user(user), src.id, "v1", dst.id)
|
||||
assert r.status_code == 404
|
||||
assert r.json()["detail"] == {"code": "playlist_move", "reason": "not_in_playlist"}
|
||||
assert r.status_code == 200
|
||||
assert (r.json()["moved"], r.json()["already_there"]) == ([], [])
|
||||
assert _ids(db, dst.id) == []
|
||||
|
||||
|
||||
def test_move_into_someone_elses_list_404s_and_changes_nothing(db, make_user, as_user):
|
||||
|
||||
@@ -102,16 +102,24 @@ PLEX = PlexPlaylist.__table__.name
|
||||
"call",
|
||||
[
|
||||
lambda c, pid, other: c.post(f"/api/playlists/{pid}/items", json={"video_id": "v2"}),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items", json={"video_id": "v2", "position": 0}
|
||||
),
|
||||
lambda c, pid, other: c.post("/api/playlists/watch-later", json={"video_id": "v2"}),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
|
||||
f"/api/playlists/{pid}/items/move-bulk",
|
||||
json={"video_ids": ["v1"], "target_playlist_id": other},
|
||||
),
|
||||
lambda c, pid, other: c.put(f"/api/playlists/{pid}/order", json={"video_ids": ["v1"]}),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/remove-bulk", json={"video_ids": ["v1"]}
|
||||
),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/restore", json={"video_ids": ["v2"], "positions": [0]}
|
||||
),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/undo-move",
|
||||
json={"from_playlist_id": other, "moved": ["v2"], "positions": [0]},
|
||||
),
|
||||
],
|
||||
ids=["add", "add-at", "watch-later", "move", "reorder"],
|
||||
ids=["add", "watch-later", "move", "reorder", "remove-bulk", "restore", "undo-move"],
|
||||
)
|
||||
def test_a_youtube_playlist_write_locks_the_playlist(db, make_user, as_user, locks, call):
|
||||
user = make_user()
|
||||
@@ -168,18 +176,35 @@ PLEX_ITEMS = PlexPlaylistItem.__table__.name
|
||||
"call",
|
||||
[
|
||||
lambda c, pid, other: c.post(f"/api/playlists/{pid}/items", json={"video_id": "v1"}),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items", json={"video_id": "v2", "position": 0}
|
||||
),
|
||||
lambda c, pid, other: c.delete(f"/api/playlists/{pid}/items/v1"),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
|
||||
f"/api/playlists/{pid}/items/move-bulk",
|
||||
json={"video_ids": ["v1"], "target_playlist_id": other},
|
||||
),
|
||||
lambda c, pid, other: c.put(f"/api/playlists/{pid}/order", json={"video_ids": ["v1"]}),
|
||||
# the ORM cascade deletes the items before the list row
|
||||
lambda c, pid, other: c.delete(f"/api/playlists/{pid}"),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/remove-bulk", json={"video_ids": ["v1"]}
|
||||
),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/restore", json={"video_ids": ["v2"], "positions": [0]}
|
||||
),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/undo-move",
|
||||
json={"from_playlist_id": other, "moved": ["v2"], "positions": [0]},
|
||||
),
|
||||
],
|
||||
ids=[
|
||||
"add-present",
|
||||
"remove",
|
||||
"move",
|
||||
"reorder",
|
||||
"delete",
|
||||
"remove-bulk",
|
||||
"restore",
|
||||
"undo-move",
|
||||
],
|
||||
ids=["add-present", "add-at", "remove", "move", "reorder", "delete"],
|
||||
)
|
||||
def test_a_youtube_item_write_locks_the_playlist_before_any_item(
|
||||
db, make_user, as_user, statements, call
|
||||
@@ -193,14 +218,23 @@ def test_a_youtube_item_write_locks_the_playlist_before_any_item(
|
||||
assert _locked_before_items(statements, YT, YT_ITEMS), statements
|
||||
|
||||
|
||||
def test_a_move_locks_both_lists_in_one_ordered_statement(db, make_user, as_user, statements):
|
||||
"""Two moves A→B and B→A each locking "its" list first would lock crosswise."""
|
||||
@pytest.mark.parametrize(
|
||||
("path", "body"),
|
||||
[
|
||||
("move-bulk", lambda other: {"video_ids": ["v1"], "target_playlist_id": other}),
|
||||
("undo-move", lambda other: {"from_playlist_id": other, "moved": ["v2"], "positions": [0]}),
|
||||
],
|
||||
ids=["move", "undo"],
|
||||
)
|
||||
def test_a_move_locks_both_lists_in_one_ordered_statement(
|
||||
db, make_user, as_user, statements, path, body
|
||||
):
|
||||
"""Two moves A→B and B→A each locking "its" list first would lock crosswise, and an Undo runs the
|
||||
other way round from a move by its nature."""
|
||||
user = make_user()
|
||||
pid, other = _yt(db, user)
|
||||
statements.clear()
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
|
||||
)
|
||||
r = as_user(user).post(f"/api/playlists/{pid}/items/{path}", json=body(other))
|
||||
assert r.status_code == 200, r.text
|
||||
lock = _locked_before_items(statements, YT, YT_ITEMS)
|
||||
assert lock and "ORDER BY" in lock, lock
|
||||
|
||||
@@ -219,6 +219,7 @@ def test_item_detail_movie_serves_from_row_despite_live_outage(db, monkeypatch,
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="The Godfather",
|
||||
size_bytes=15_000_000_000,
|
||||
genres=["Crime", "Drama"],
|
||||
directors=["Francis Ford Coppola"],
|
||||
studio="Paramount",
|
||||
@@ -238,6 +239,7 @@ def test_item_detail_movie_serves_from_row_despite_live_outage(db, monkeypatch,
|
||||
resp = _get(db, user, monkeypatch, override_dep, "/api/plex/item/m1")
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["size_bytes"] == 15_000_000_000
|
||||
assert body["imdb_rating"] == 8.7
|
||||
assert body["imdb_id"] == "tt0068646"
|
||||
assert body["imdb_url"] == "https://www.imdb.com/title/tt0068646/"
|
||||
|
||||
@@ -13,7 +13,7 @@ from datetime import datetime, timedelta, timezone
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.plex.models import PlexItem, PlexLibrary, PlexState
|
||||
from app.plex.models import PlexItem, PlexLibrary, PlexSeason, PlexShow, PlexState
|
||||
from app.plex import service as plex_service
|
||||
from app.routes.plex import playback
|
||||
|
||||
@@ -81,3 +81,150 @@ def test_the_finish_supersedes_a_resume_push_still_in_flight(db, as_user, make_u
|
||||
)
|
||||
assert r.json() == {"ok": True, "status": "watched", "position_seconds": 0}
|
||||
assert playback._resume_push_superseded(user.id, item.id, earlier) is True
|
||||
|
||||
|
||||
def test_a_checkpoint_sent_before_the_finish_does_not_land_on_the_watched_row(
|
||||
db, as_user, make_user, monkeypatch
|
||||
):
|
||||
"""The 10 s tick posted at 2699 of 3000 s (not near the end) races the credits auto-skip's
|
||||
"watched". When the mark commits first, the late checkpoint used to write its position onto the
|
||||
watched row and schedule a resume push after the scrobble: Plex showed the item in progress, and
|
||||
a Plex-authoritative pull imported it back."""
|
||||
user = make_user()
|
||||
item = _item(db)
|
||||
monkeypatch.setattr(playback.plex_stream, "touch_session", lambda uid, key: None)
|
||||
client = as_user(user)
|
||||
assert client.post("/api/plex/item/m1/state", json={"status": "watched"}).status_code == 200
|
||||
|
||||
r = client.post(
|
||||
"/api/plex/item/m1/progress", json={"position_seconds": 2699, "duration_seconds": 3000}
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json() == {"ok": True, "status": "watched", "position_seconds": 0}
|
||||
db.expire_all()
|
||||
st = db.execute(select(PlexState).where(PlexState.item_id == item.id)).scalars().one()
|
||||
assert (st.status, st.position_seconds) == ("watched", 0)
|
||||
|
||||
|
||||
def test_a_rewatch_records_its_position_once_the_finish_is_behind_it(
|
||||
db, as_user, make_user, monkeypatch
|
||||
):
|
||||
"""The other half: a watched item played again is still checkpointed — only a position arriving
|
||||
right on the heels of the mark is read as that mark's late tick."""
|
||||
user = make_user()
|
||||
item = _item(db)
|
||||
long_ago = datetime.now(timezone.utc) - timedelta(days=3)
|
||||
db.add(PlexState(user_id=user.id, item_id=item.id, status="watched", watched_at=long_ago))
|
||||
db.commit()
|
||||
monkeypatch.setattr(playback.plex_stream, "touch_session", lambda uid, key: None)
|
||||
r = as_user(user).post(
|
||||
"/api/plex/item/m1/progress", json={"position_seconds": 600, "duration_seconds": 3000}
|
||||
)
|
||||
assert r.json() == {"ok": True, "position_seconds": 600}
|
||||
db.expire_all()
|
||||
st = db.execute(select(PlexState).where(PlexState.item_id == item.id)).scalars().one()
|
||||
assert (st.status, st.position_seconds) == ("watched", 600)
|
||||
|
||||
|
||||
def test_marking_watched_supersedes_a_resume_push_still_in_flight(
|
||||
db, as_user, make_user, monkeypatch
|
||||
):
|
||||
"""The same guard the near-end finish has: a resume push scheduled before the mark must not reach
|
||||
Plex after its scrobble."""
|
||||
user = make_user()
|
||||
item = _item(db)
|
||||
earlier = datetime.now(timezone.utc) - timedelta(seconds=5)
|
||||
assert playback._should_push_resume(user.id, item.id, earlier, final=True)
|
||||
r = as_user(user).post("/api/plex/item/m1/state", json={"status": "watched"})
|
||||
assert r.status_code == 200, r.text
|
||||
assert playback._resume_push_superseded(user.id, item.id, earlier) is True
|
||||
|
||||
|
||||
def _state(db, user, item):
|
||||
db.expire_all()
|
||||
st = (
|
||||
db.execute(
|
||||
select(PlexState).where(PlexState.user_id == user.id, PlexState.item_id == item.id)
|
||||
)
|
||||
.scalars()
|
||||
.one()
|
||||
)
|
||||
return st.status, st.position_seconds
|
||||
|
||||
|
||||
def test_a_mark_committed_after_the_checkpoints_read_still_refuses_it(db, make_user):
|
||||
"""The race itself, not its sequential shadow: the route read the row while it was still in
|
||||
progress, and the finish committed from another session before the checkpoint wrote. Only a write
|
||||
conditional on the row as it is NOW refuses that position."""
|
||||
user = make_user()
|
||||
item = _item(db)
|
||||
db.add(PlexState(user_id=user.id, item_id=item.id, status="new", position_seconds=2690))
|
||||
db.commit()
|
||||
now = datetime.now(timezone.utc)
|
||||
read = plex_service.state_row(db, user, item.id) # what the checkpoint's route read: "new"
|
||||
assert read.status == "new"
|
||||
|
||||
other = SessionLocal()
|
||||
try:
|
||||
assert plex_service.mark_finished(other, user.id, item.id, now) is True
|
||||
other.commit()
|
||||
finally:
|
||||
other.close()
|
||||
|
||||
assert plex_service.checkpoint(db, user, item.id, read, 2699, now, dirty=True) is False
|
||||
db.commit()
|
||||
assert _state(db, user, item) == ("watched", 0)
|
||||
|
||||
|
||||
def test_a_tick_seconds_behind_the_mark_is_refused_a_rewatch_a_minute_later_is_not(db, make_user):
|
||||
"""A position right behind the mark is that mark's late tick; one a minute later is a rewatch.
|
||||
Literal times on purpose: a window widened to an hour must fail here, not move with it."""
|
||||
user = make_user()
|
||||
item = _item(db)
|
||||
now = datetime.now(timezone.utc)
|
||||
st = PlexState(user_id=user.id, item_id=item.id, status="watched", watched_at=now)
|
||||
db.add(st)
|
||||
db.commit()
|
||||
|
||||
st.watched_at = now - timedelta(seconds=20)
|
||||
db.commit()
|
||||
assert plex_service.checkpoint(db, user, item.id, st, 600, now, dirty=True) is False
|
||||
db.commit()
|
||||
st.watched_at = now - timedelta(seconds=60)
|
||||
db.commit()
|
||||
assert plex_service.checkpoint(db, user, item.id, st, 600, now, dirty=True) is True
|
||||
db.commit()
|
||||
assert _state(db, user, item) == ("watched", 600)
|
||||
|
||||
|
||||
def test_marking_a_season_watched_supersedes_its_episodes_resume_pushes(db, as_user, make_user):
|
||||
"""The bulk mark has the same guard as one item's: an episode's resume push scheduled before
|
||||
the season was marked must not reach Plex after the scrobble."""
|
||||
user = make_user()
|
||||
lib = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
sh = PlexShow(rating_key="sh1", library_id=lib.id, title="Show")
|
||||
db.add(sh)
|
||||
db.flush()
|
||||
se = PlexSeason(rating_key="se1", show_id=sh.id, season_number=1)
|
||||
db.add(se)
|
||||
db.flush()
|
||||
ep = PlexItem(
|
||||
rating_key="e1",
|
||||
library_id=lib.id,
|
||||
kind="episode",
|
||||
title="E1",
|
||||
show_id=sh.id,
|
||||
season_id=se.id,
|
||||
season_number=1,
|
||||
episode_number=1,
|
||||
)
|
||||
db.add(ep)
|
||||
db.commit()
|
||||
earlier = datetime.now(timezone.utc) - timedelta(seconds=5)
|
||||
assert playback._should_push_resume(user.id, ep.id, earlier, final=True)
|
||||
|
||||
r = as_user(user).post("/api/plex/season/se1/state", json={"watched": True})
|
||||
assert r.json() == {"ok": True, "changed": 1, "watched": True}
|
||||
assert playback._resume_push_superseded(user.id, ep.id, earlier) is True
|
||||
|
||||
@@ -5,7 +5,7 @@ seek-restart — or a second viewer of the same item — replaces the registered
|
||||
viewer's next request for `seg_12.ts`, resolved by rating_key, was answered 200 with the NEW
|
||||
session's twelfth segment: bytes produced from a different `-ss` offset under `-copyts`, i.e. a
|
||||
timeline the client is not on. Wrong content with a success status is the one failure a player cannot
|
||||
notice, and the client's reclaim-on-404 could never fire on it.
|
||||
notice, and the client's reclaim could never fire on it.
|
||||
|
||||
No `db` fixture: the route reads nothing from the database (its own `Depends(get_db)` went with the
|
||||
removed auto-start, and `current_user` is overridden wholesale), so these must not sit in the DB
|
||||
@@ -88,10 +88,10 @@ def test_another_viewers_session_of_the_same_item_is_not_this_viewers(client, tm
|
||||
(d / "index.m3u8").write_bytes(b"#EXTM3U")
|
||||
sessions[(7, "m1")] = _Session("sid-a", d)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/index.m3u8")
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.status_code == 410, r.text
|
||||
|
||||
|
||||
def test_a_segment_of_a_replaced_session_is_a_404_not_the_new_sessions_bytes(
|
||||
def test_a_segment_of_a_replaced_session_is_gone_not_the_new_sessions_bytes(
|
||||
client, tmp_path, sessions
|
||||
):
|
||||
"""Viewer A is at 0:00; viewer B seeks to 25:00 and their session takes the registry slot. A's
|
||||
@@ -102,15 +102,15 @@ def test_a_segment_of_a_replaced_session_is_a_404_not_the_new_sessions_bytes(
|
||||
(theirs / "seg_12.ts").write_bytes(b"theirs")
|
||||
sessions[(VIEWER, "m1")] = _Session("sid-b", theirs)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.status_code == 410, r.text
|
||||
assert b"theirs" not in r.content
|
||||
|
||||
|
||||
def test_a_playlist_with_no_live_session_is_a_404_not_a_silent_restart_at_zero(
|
||||
def test_a_playlist_with_no_live_session_is_gone_not_a_silent_restart_at_zero(
|
||||
client, monkeypatch, sessions
|
||||
):
|
||||
"""The entry playlist used to auto-start a session when none was registered — always at 0.0,
|
||||
silently throwing away the viewer's position. A 404 sends the player through its own reclaim,
|
||||
silently throwing away the viewer's position. A 410 sends the player through its own reclaim,
|
||||
which restarts at the playhead."""
|
||||
started: list[float] = []
|
||||
monkeypatch.setattr(
|
||||
@@ -119,7 +119,7 @@ def test_a_playlist_with_no_live_session_is_a_404_not_a_silent_restart_at_zero(
|
||||
lambda *a, **k: started.append(a[3] if len(a) > 3 else 0.0),
|
||||
)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-gone/index.m3u8")
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.status_code == 410, r.text
|
||||
assert started == []
|
||||
|
||||
|
||||
@@ -198,6 +198,31 @@ def test_a_name_the_session_cannot_produce_fails_at_once(client, tmp_path, sessi
|
||||
assert elapsed < 2.0, f"waited {elapsed:.1f}s for a playlist this session never writes"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "audio_langs"),
|
||||
[
|
||||
("seg_00.ts", None), # `int()` reads it as segment 0, which needs no predecessor
|
||||
("seg_007.ts", None), # …as segment 7, whose predecessor seg_6.ts is on disk
|
||||
("seg_٧.ts", None), # an Arabic-Indic 7: `\d` and `int()` both take it
|
||||
("seg_a0_07.ts", ["hu", "en"]),
|
||||
("seg_0_00.ts", ["hu", "en"]),
|
||||
],
|
||||
)
|
||||
def test_a_segment_name_ffmpeg_never_spells_is_refused(tmp_path, name, audio_langs):
|
||||
"""ffmpeg writes `seg_%d.ts`: no padding, ASCII digits. A name that only PARSES to a sequence the
|
||||
session has reached is still a file that will never appear, and saying yes to it parked the
|
||||
request for the full 20 s wait — the parking this guard exists to end."""
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
for v in plex_stream.session_variants(audio_langs) or (None,):
|
||||
(d / plex_stream.segment_name(6, v)).write_bytes(b"x")
|
||||
sess = _Session("sid-a", d, running=True, audio_langs=audio_langs)
|
||||
assert not plex_stream.artifact_may_arrive(sess, name)
|
||||
# The canonical spelling of the same next segment is still accepted.
|
||||
v = (plex_stream.session_variants(audio_langs) or (None,))[0]
|
||||
assert plex_stream.artifact_may_arrive(sess, plex_stream.segment_name(7, v))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("digits", [100, 5000])
|
||||
def test_an_absurd_segment_index_is_a_404_not_a_500(client, tmp_path, sessions, digits):
|
||||
"""`int()` on an unbounded digit run raises ValueError past 4300 digits, and only OSError was
|
||||
@@ -210,12 +235,12 @@ def test_an_absurd_segment_index_is_a_404_not_a_500(client, tmp_path, sessions,
|
||||
assert r.status_code == 404, r.text
|
||||
|
||||
|
||||
def test_a_session_torn_down_during_the_read_is_a_404_not_a_500(
|
||||
def test_a_session_torn_down_during_the_read_is_gone_not_a_500(
|
||||
client, monkeypatch, tmp_path, sessions
|
||||
):
|
||||
"""`_kill` rmtrees the whole directory and nothing holds `_lock` across the wait and the read,
|
||||
"""Teardown moves the whole directory aside and nothing holds `_lock` across the wait and the read,
|
||||
so the file can vanish in between — on every seek-restart, since `loadSession` awaits the new
|
||||
session before destroying the old hls instance. The player's recovery is keyed on the 404
|
||||
session before destroying the old hls instance. The player's recovery is keyed on the
|
||||
STATUS, so a 500 here skipped the reclaim and ended playback with "couldn't play"."""
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
@@ -224,7 +249,7 @@ def test_a_session_torn_down_during_the_read_is_a_404_not_a_500(
|
||||
# The file existed when the wait checked, and is gone by the time the route reads it.
|
||||
monkeypatch.setattr(plex_stream, "wait_for", lambda *a, **k: True)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.status_code == 410, r.text
|
||||
|
||||
|
||||
def test_a_dead_producer_is_not_waited_out(client, tmp_path, sessions):
|
||||
|
||||
@@ -32,15 +32,25 @@ class TestTranscodeHwCapable:
|
||||
assert sync.transcode_hw_capable("transcode", "hevc") is True
|
||||
assert sync.transcode_hw_capable("transcode", "vp9", bit_depth=8) is True
|
||||
|
||||
def test_non_decodable_codec_is_sw(self):
|
||||
# AV1 / MPEG-4 ASP (XviD) can't be HW-decoded on Gen 9.5 → software regardless of bit depth.
|
||||
def test_a_codec_neither_path_knows_is_sw(self):
|
||||
assert sync.transcode_hw_capable("transcode", "av1") is False
|
||||
assert sync.transcode_hw_capable("transcode", "mpeg4", bit_depth=8) is False
|
||||
assert sync.transcode_hw_capable("transcode", "av1", bit_depth=8) is False
|
||||
|
||||
def test_10bit_or_hdr_hevc_is_sw(self):
|
||||
# A HW-decodable codec that is 10-bit or HDR still breaks the VAAPI nv12 pipeline → software.
|
||||
def test_a_cpu_decoded_8bit_codec_is_encoded_on_the_gpu(self):
|
||||
# VC-1 (the iGPU has no VC-1 Advanced decode), XviD, DivX: the CPU decodes, the GPU encodes.
|
||||
for codec in ("vc1", "mpeg4", "msmpeg4v3"):
|
||||
assert sync.transcode_hw_capable("transcode", codec) is True
|
||||
assert sync.transcode_hw_capable("transcode", "mpeg4", bit_depth=10) is False
|
||||
|
||||
def test_10bit_sdr_hevc_is_sw(self):
|
||||
# 10-bit SDR: an exact downconvert on the CPU (libx264), never the HDR tone curve.
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=10) is False
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=8, hdr=True) is False
|
||||
|
||||
def test_hdr_hevc_is_encoded_on_the_gpu(self):
|
||||
# HDR goes through the OpenCL tonemap between VAAPI decode and encode (the prod image ships the
|
||||
# runtime); the facet said software for it while every such session ran on the GPU.
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=10, hdr=True) is True
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=8, hdr=True) is True
|
||||
|
||||
|
||||
class TestVideoStreamProps:
|
||||
@@ -87,6 +97,9 @@ class TestSyncDerivation:
|
||||
row2 = PlexItem(rating_key="y", kind="movie")
|
||||
sync._apply_item(row2, lib_id=1, kind="movie", meta=self._meta("av1"))
|
||||
assert row2.transcode_hw is False # av1 not HW-decodable
|
||||
row3 = PlexItem(rating_key="z", kind="movie")
|
||||
sync._apply_item(row3, lib_id=1, kind="movie", meta=self._meta("vc1"))
|
||||
assert row3.transcode_hw is True # CPU decode, GPU encode
|
||||
|
||||
def test_probed_value_survives_resync_when_codec_unchanged(self):
|
||||
row = PlexItem(rating_key="x", kind="movie")
|
||||
@@ -124,21 +137,24 @@ class TestSyncDerivation:
|
||||
assert row.video_bit_depth is None and row.media_probed_at is None # re-probe scheduled
|
||||
|
||||
|
||||
def test_migration_codec_set_matches_hw_decodable_codecs():
|
||||
# Migration 0077 hardcodes the HW-decodable codec set in raw SQL and its comment mandates it stay
|
||||
# identical to plex.sync.HW_DECODABLE_CODECS. A comment isn't enforcement (house rule) — pin the
|
||||
# equality so adding/removing a codec from the frozenset without editing the migration fails here
|
||||
# (else already-migrated prod rows keep a wrong transcode_hw the runtime derivation wouldn't catch).
|
||||
def test_migration_codec_sets_match_the_sync_sets():
|
||||
# The newest migration that rewrites `transcode_hw` hardcodes both codec sets in raw SQL. Pin them
|
||||
# to plex.sync, so a codec added to or removed from a frozenset without a migration that rewrites
|
||||
# the stored rows fails here (else prod rows keep a facet the runtime choice disagrees with).
|
||||
mig = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "alembic"
|
||||
/ "versions"
|
||||
/ "0077_plex_playback_class.py"
|
||||
/ "0119_plex_transcode_gpu.py"
|
||||
)
|
||||
m = re.search(r"lower\(codec_video\)\s+IN\s+\(([^)]*)\)", mig.read_text())
|
||||
assert m, "could not find the codec IN(...) list in migration 0077"
|
||||
codecs = {c.strip().strip("'\"").lower() for c in m.group(1).split(",")}
|
||||
assert codecs == {c.lower() for c in sync.HW_DECODABLE_CODECS}
|
||||
src = mig.read_text()
|
||||
sets = {}
|
||||
for name in ("HW_DECODABLE_CODECS", "CPU_DECODE_CODECS"):
|
||||
m = re.search(name + r"\s*=\s*\(([^)]*)\)", src)
|
||||
assert m, f"could not find {name} in migration 0119"
|
||||
sets[name] = {c.strip().strip("'\"").lower() for c in m.group(1).split(",") if c.strip()}
|
||||
assert sets["HW_DECODABLE_CODECS"] == {c.lower() for c in sync.HW_DECODABLE_CODECS}
|
||||
assert sets["CPU_DECODE_CODECS"] == {c.lower() for c in sync.CPU_DECODE_CODECS}
|
||||
|
||||
|
||||
class _FakePlex:
|
||||
@@ -182,22 +198,42 @@ def test_probe_media_batch_refines_hw_to_sw_for_10bit(db):
|
||||
codec_video="mpeg4",
|
||||
transcode_hw=False,
|
||||
)
|
||||
db.add_all([hevc8, hevc10, xvid])
|
||||
hdr10 = PlexItem(
|
||||
rating_key="hdr",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="hdr",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
)
|
||||
db.add_all([hevc8, hevc10, xvid, hdr10])
|
||||
db.commit()
|
||||
plex = _FakePlex(
|
||||
{
|
||||
"h8": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]},
|
||||
"h10": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 10}]}]}]},
|
||||
"hdr": {
|
||||
"Media": [
|
||||
{
|
||||
"Part": [
|
||||
{"Stream": [{"streamType": 1, "bitDepth": 10, "colorTrc": "smpte2084"}]}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
)
|
||||
probed = sync._probe_media_batch(db, plex)
|
||||
db.refresh(hevc8)
|
||||
db.refresh(hevc10)
|
||||
db.refresh(xvid)
|
||||
# Only the two HW-decodable HEVC items are probed; the XviD is SW-by-codec and never fetched.
|
||||
assert probed == 2
|
||||
db.refresh(hdr10)
|
||||
# Only the HW-decodable HEVC items are probed; the XviD's class follows from its codec alone.
|
||||
assert probed == 3
|
||||
assert hevc8.video_bit_depth == 8 and hevc8.transcode_hw is True and hevc8.media_probed_at
|
||||
assert hevc10.video_bit_depth == 10 and hevc10.transcode_hw is False # downgraded to software
|
||||
assert hevc10.video_bit_depth == 10 and hevc10.transcode_hw is False # 10-bit SDR: software
|
||||
assert hdr10.video_bit_depth == 10 and hdr10.transcode_hw is True # HDR: the GPU tonemap
|
||||
assert xvid.media_probed_at is None # not in the work-list
|
||||
# A second run has nothing left to do (watermarks set).
|
||||
assert sync._probe_media_batch(db, plex) == 0
|
||||
|
||||
@@ -6,6 +6,8 @@ rows, since positions have gaps after a removal. One request: an append plus a s
|
||||
leaves the rows at the end whenever that call fails, and would overwrite another tab's changes.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from app.plex.models import PlexItem, PlexLibrary, PlexPlaylist, PlexPlaylistItem
|
||||
|
||||
|
||||
@@ -101,6 +103,22 @@ def test_positions_must_pair_with_the_keys(db, make_user, as_user):
|
||||
assert _order(db, pl.id)[0] == ["a"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path"),
|
||||
[("post", "items/bulk"), ("post", "items/remove-bulk"), ("put", "order")],
|
||||
)
|
||||
def test_a_body_without_its_keys_is_refused_rather_than_acting_on_none(
|
||||
db, make_user, as_user, method, path
|
||||
):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a"], extra=())
|
||||
|
||||
r = getattr(as_user(user), method)(f"/api/plex/playlists/{pl.id}/{path}", json={"keys": ["a"]})
|
||||
|
||||
assert r.status_code == 422
|
||||
assert _order(db, pl.id)[0] == ["a"]
|
||||
|
||||
|
||||
def test_remove_then_put_back_is_no_change(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b", "c", "d"], extra=())
|
||||
|
||||
@@ -348,7 +348,8 @@ def test_an_early_checkpoint_drops_a_bare_state_and_zeroes_a_marked_one(db, make
|
||||
user = make_user()
|
||||
lib = _lib(db)
|
||||
bare, hid = _movie(db, lib, "bare"), _movie(db, lib, "hid")
|
||||
st = service.checkpoint(db, user, bare.id, None, 120, NOW, dirty=True)
|
||||
assert service.checkpoint(db, user, bare.id, None, 120, NOW, dirty=True)
|
||||
st = queries.state(db, user, bare.id)
|
||||
assert (st.position_seconds, st.synced_to_plex) == (120, False)
|
||||
_state(db, user, hid, status="hidden", position_seconds=40)
|
||||
service.clear_progress(db, queries.state(db, user, bare.id), NOW)
|
||||
@@ -429,7 +430,8 @@ def test_a_first_write_that_lost_the_race_updates_the_winners_row(db, make_user)
|
||||
user = make_user()
|
||||
it = _movie(db, _lib(db), "m-race")
|
||||
_state(db, user, it, position_seconds=10) # the request that won, committed
|
||||
st = service.checkpoint(db, user, it.id, None, 42, NOW, dirty=True) # read "none" before it
|
||||
assert service.checkpoint(db, user, it.id, None, 42, NOW, dirty=True) # read "none" before it
|
||||
st = queries.state(db, user, it.id)
|
||||
assert (st.position_seconds, st.synced_to_plex) == (42, False)
|
||||
st = service.set_status(db, user, it.id, None, "hidden")
|
||||
assert st.status == "hidden"
|
||||
|
||||
@@ -39,6 +39,7 @@ class _Proc:
|
||||
class _Session:
|
||||
def __init__(self, session_id: str, directory):
|
||||
self.id = session_id
|
||||
self.key = "m1"
|
||||
self.dir = directory
|
||||
self.proc = _Proc()
|
||||
self.last_access = time.time()
|
||||
|
||||
@@ -26,6 +26,9 @@ from app.plex.stream import _SESSION_DIR_RE
|
||||
"12345_0_2_+0.00", # current shape, explicit audio_ord
|
||||
"12345_842_multi_-1.25", # multi-audio, negative offset
|
||||
"12345_842_multi_+12.50",
|
||||
# a torn-down session's directory, moved aside to be deleted outside the lock — left behind
|
||||
# only if the process died between the rename and the delete
|
||||
"12345_u7_842_multi_+12.50.dead-0123abcd",
|
||||
],
|
||||
)
|
||||
def test_our_session_directories_are_swept(name):
|
||||
@@ -47,6 +50,9 @@ def test_our_session_directories_are_swept(name):
|
||||
"snapshot_2024_backup_+1.00", # session-ish tail, but the key isn't a rating_key
|
||||
"backup_2024_multi_+0.00",
|
||||
".config",
|
||||
"12345_u7_0_0_+0.00.dead-", # a tombstone suffix with no id
|
||||
"12345_u7_0_0_+0.00.dead-notours!",
|
||||
"backup_u7_0_0_+0.00.dead-0123abcd",
|
||||
],
|
||||
)
|
||||
def test_foreign_directories_are_left_alone(name):
|
||||
@@ -159,7 +165,7 @@ class TestSweepOrphans:
|
||||
|
||||
|
||||
class _FakeProc:
|
||||
"""Stand-in for the ffmpeg Popen — start_session only stores it (and _kill/reap poll it later)."""
|
||||
"""Stand-in for the ffmpeg Popen — start_session only stores it (and teardown/reap poll it later)."""
|
||||
|
||||
def poll(self):
|
||||
return None
|
||||
@@ -278,6 +284,24 @@ class TestTranscodeCommand:
|
||||
)
|
||||
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # not a software scale
|
||||
|
||||
def test_cpu_decode_uploads_the_frames_to_the_gpu_encoder(self):
|
||||
# A source the GPU cannot decode (VC-1 Advanced on Gen 9.5): the CPU decodes, the frames go to
|
||||
# nv12 and up to the GPU, and h264_vaapi encodes. No -hwaccel: asking the GPU to decode is what
|
||||
# failed.
|
||||
cmd = stream._ffmpeg_cmd(
|
||||
self.SRC, 0.0, self.OUT, None, transcode=True, hw=True, cpu_decode=True
|
||||
)
|
||||
assert "-hwaccel" not in cmd
|
||||
assert cmd[cmd.index("-init_hw_device") + 1] == f"vaapi=va:{stream._VAAPI_DEVICE}"
|
||||
assert cmd[cmd.index("-filter_hw_device") + 1] == "va"
|
||||
assert cmd.index("-init_hw_device") < cmd.index("-i")
|
||||
assert cmd[cmd.index("-vf") + 1] == "format=nv12,hwupload"
|
||||
assert _venc(cmd) == "h264_vaapi"
|
||||
assert "libx264" not in cmd
|
||||
assert (
|
||||
cmd[cmd.index("-force_key_frames") + 1] == f"expr:gte(t,n_forced*{stream._SEG_SECONDS})"
|
||||
)
|
||||
|
||||
def test_hdr_tonemap_uses_the_opencl_graph(self):
|
||||
# The HDR/10-bit HW path: VAAPI decode → OpenCL tonemap → VAAPI encode. Gen 9.5 can't tonemap in
|
||||
# the VAAPI VPP, so this is the only HW route for a 10-bit/HDR source.
|
||||
@@ -359,19 +383,48 @@ class TestWantHw:
|
||||
is True
|
||||
)
|
||||
|
||||
def test_auto_falls_back_to_software_for_a_non_decodable_source(self, monkeypatch):
|
||||
# AV1 / MPEG-4 ASP (XviD) aren't decodable on Gen 9.5 → the GPU can't feed its own encoder.
|
||||
def test_auto_falls_back_to_software_for_a_codec_neither_path_knows(self, monkeypatch):
|
||||
# AV1 is neither GPU-decodable on Gen 9.5 nor on the CPU-decode list (it can be 10-bit HDR).
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._hwsafe(monkeypatch, True) # decodability gate must fire BEFORE the probe
|
||||
self._hwsafe(monkeypatch, True) # the codec gate must fire BEFORE the probe
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="mpeg4"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("codec", ["vc1", "mpeg4", "msmpeg4v3"])
|
||||
def test_auto_cpu_decodes_what_the_gpu_cannot_and_still_encodes_on_it(self, monkeypatch, codec):
|
||||
# Measured on prod (UHD 630, iHD 25.2.3): "No support for codec vc1 profile 3", so every VC-1
|
||||
# HW session died and fell back to libx264 (~19 s to start). Decoding on the CPU and encoding on
|
||||
# the GPU produced three segments in ~5 s (VC-1 1080p) and ~1 s (XviD, DivX SD).
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._kind(monkeypatch, stream._VK_HWSAFE)
|
||||
self._opencl(monkeypatch, True)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video=codec), transcode=True, src=self.SRC
|
||||
) == (True, False, True)
|
||||
|
||||
def test_auto_a_cpu_decoded_codec_that_is_not_8bit_sdr_stays_software(self, monkeypatch):
|
||||
# The upload path converts to nv12 for the encoder; only an 8-bit SDR picture is known to come
|
||||
# through it unchanged.
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._kind(monkeypatch, stream._VK_SDR10)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="mpeg4"), transcode=True, src=self.SRC
|
||||
) == (False, False, False)
|
||||
|
||||
def test_on_cpu_decodes_a_codec_the_gpu_cannot(self, monkeypatch):
|
||||
# "on" trusts the GPU, but asking it to DECODE a codec it is known not to decode only buys the
|
||||
# fallback's wasted second; the upload path is the GPU route for such a source.
|
||||
self._policy(monkeypatch, "on")
|
||||
self._opencl(monkeypatch, False)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="vc1"), transcode=True, src=self.SRC
|
||||
) == (True, False, True)
|
||||
|
||||
def test_auto_hdr_source_falls_back_to_software_without_opencl(self, monkeypatch):
|
||||
# A decodable HEVC that is true-HDR (PQ/HLG) can't use scale_vaapi=format=nv12 (Gen 9.5), and with
|
||||
@@ -382,7 +435,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (False, False)
|
||||
) == (False, False, False)
|
||||
|
||||
def test_auto_hdr_source_uses_hw_tonemap_with_opencl(self, monkeypatch):
|
||||
# With the OpenCL runtime present, a true-HDR HEVC tone-maps on the GPU (VAAPI decode → OpenCL
|
||||
@@ -393,7 +446,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, True)
|
||||
) == (True, True, False)
|
||||
|
||||
def test_auto_10bit_SDR_source_uses_software_not_tonemap(self, monkeypatch):
|
||||
# F1 regression: a 10-bit *SDR* (bt709) source is NOT HDR, so it must NOT get the hable HDR tone
|
||||
@@ -405,7 +458,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (False, False)
|
||||
) == (False, False, False)
|
||||
|
||||
def test_auto_8bit_source_uses_plain_hw_not_tonemap(self, monkeypatch):
|
||||
# An 8-bit SDR source takes the plain full-GPU scale path — never the tonemap graph — even where
|
||||
@@ -416,7 +469,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
) == (True, False, False)
|
||||
|
||||
def test_off_forces_software_even_for_a_decodable_source(self, monkeypatch):
|
||||
self._policy(monkeypatch, "off")
|
||||
@@ -454,7 +507,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, True)
|
||||
) == (True, True, False)
|
||||
|
||||
def test_on_10bit_SDR_uses_plain_hw_not_tonemap(self, monkeypatch):
|
||||
# F1 regression under "on": a 10-bit SDR source is NOT HDR, so even with OpenCL present "on" must
|
||||
@@ -465,7 +518,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
) == (True, False, False)
|
||||
|
||||
def test_on_without_opencl_is_plain_hw(self, monkeypatch):
|
||||
# "on" with no OpenCL runtime → plain HW, tonemap=False, regardless of source class (the runtime
|
||||
@@ -477,7 +530,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
) == (True, False, False)
|
||||
|
||||
|
||||
class TestColdStartWarmup:
|
||||
@@ -614,6 +667,30 @@ class TestTranscodeStartSession:
|
||||
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # no SW cap
|
||||
assert kwargs.get("preexec_fn") is None # GPU encode is cheap — never niced
|
||||
|
||||
def test_a_cpu_decode_session_encodes_on_the_gpu_and_falls_back_to_software(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (True, False, True)
|
||||
)
|
||||
seen = {"n": 0}
|
||||
|
||||
def flaky_wait(path, timeout=20.0, proc=None):
|
||||
seen["n"] += 1
|
||||
return seen["n"] > 1 # the upload session yields nothing; the software respawn does
|
||||
|
||||
monkeypatch.setattr(stream, "wait_for", flaky_wait)
|
||||
s = stream.start_session(
|
||||
db=None, user_id=7, item=_FakeItem(codec_video="vc1"), start_s=0.0, transcode=True
|
||||
)
|
||||
(up_cmd, *_), up_kwargs = _CapturingProc.calls[0]
|
||||
assert "format=nv12,hwupload" in up_cmd and _venc(up_cmd) == "h264_vaapi"
|
||||
if os.name == "posix": # the decode is CPU work: niced like a software transcode
|
||||
assert up_kwargs.get("preexec_fn") is stream._sw_child_preexec
|
||||
(sw_cmd, *_), _ = _CapturingProc.calls[1]
|
||||
assert _venc(sw_cmd) == "libx264" and "format=nv12,hwupload" not in sw_cmd
|
||||
assert s is not None and s.hw is False and s.cpu_decode is False
|
||||
|
||||
def test_hdr_tonemap_transcode_threads_the_opencl_graph_and_flags_the_session(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
@@ -1052,3 +1129,63 @@ class TestSessionsPerViewer:
|
||||
self._start(monkeypatch, user_id=uid)
|
||||
assert len(stream._sessions) == 3 and (1, "12345") not in stream._sessions
|
||||
assert oldest.proc.killed and not oldest.transcode
|
||||
|
||||
|
||||
class TestTeardownFinishes:
|
||||
"""A session taken down by a cap or replaced by its own viewer is only renamed aside under the
|
||||
lock; it is deleted after. The rename alone already moves the directory and `terminate()` alone
|
||||
already marks the proc killed — so what is pinned here is the tomb: a whole segmented copy of the
|
||||
film that only a restart's sweep would otherwise reclaim."""
|
||||
|
||||
capturing_env = TestTranscodeStartSession.capturing_env
|
||||
|
||||
@staticmethod
|
||||
def _left_behind(root):
|
||||
return sorted(p.name for p in root.iterdir() if stream._TOMB in p.name) + sorted(
|
||||
str(p) for p in stream._dying
|
||||
)
|
||||
|
||||
def test_a_culled_session_is_deleted_not_only_moved_aside(self, capturing_env, monkeypatch):
|
||||
monkeypatch.setattr(stream, "_dying", {})
|
||||
first = TestSessionsPerViewer._start(monkeypatch, user_id=1, rating_key="1")
|
||||
first.last_access -= 60
|
||||
TestSessionsPerViewer._start(monkeypatch, user_id=1, rating_key="2")
|
||||
TestSessionsPerViewer._start(monkeypatch, user_id=1, rating_key="3")
|
||||
assert first.proc.killed and not first.dir.exists()
|
||||
assert self._left_behind(capturing_env) == []
|
||||
|
||||
def test_a_viewers_restart_deletes_the_session_it_replaces(self, capturing_env, monkeypatch):
|
||||
monkeypatch.setattr(stream, "_dying", {})
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
stream, "_probe_first_pts", lambda seg: None
|
||||
) # no real segment to probe
|
||||
old = stream.start_session(db=None, user_id=1, item=_FakeItem(), start_s=0.0)
|
||||
new = stream.start_session(db=None, user_id=1, item=_FakeItem(), start_s=600.0)
|
||||
assert old.proc.killed and new.dir.exists() and not old.dir.exists()
|
||||
assert self._left_behind(capturing_env) == []
|
||||
|
||||
def test_a_restart_that_fails_to_start_still_deletes_what_it_replaced(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
# The new ffmpeg cannot start (a full disk, no ffmpeg, no memory): the old session is already
|
||||
# moved aside, and nothing else in this process would ever delete it.
|
||||
monkeypatch.setattr(stream, "_dying", {})
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
stream, "_probe_first_pts", lambda seg: None
|
||||
) # no real segment to probe
|
||||
old = stream.start_session(db=None, user_id=1, item=_FakeItem(), start_s=0.0)
|
||||
|
||||
def no_ffmpeg(*args, **kwargs):
|
||||
raise OSError(28, "No space left on device")
|
||||
|
||||
monkeypatch.setattr(stream.subprocess, "Popen", no_ffmpeg)
|
||||
with pytest.raises(OSError):
|
||||
stream.start_session(db=None, user_id=1, item=_FakeItem(), start_s=600.0)
|
||||
assert old.proc.killed and not old.dir.exists()
|
||||
assert self._left_behind(capturing_env) == []
|
||||
|
||||
@@ -14,7 +14,7 @@ from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from app.discover import pins
|
||||
from app.catalog.models import Channel, Video
|
||||
from app.catalog.models import Channel, Subscription, Video
|
||||
from app.reader import queries, service
|
||||
from app.reader.models import ChannelTag, ComfortPin, Playlist, PlaylistItem, Tag, VideoState
|
||||
|
||||
@@ -60,12 +60,44 @@ def test_a_system_tag_is_nobodys_to_edit(db, make_user):
|
||||
service.create_tag(db, other, "Theirs", "other", None) # another user's tag stays theirs
|
||||
db.commit()
|
||||
assert queries.own_tag(db, user, system.id) is None
|
||||
assert [t.id for t, _ in queries.tags_with_counts(db, user)] == [system.id]
|
||||
assert [t.id for t, *_ in queries.tags_with_counts(db, user)] == [system.id]
|
||||
|
||||
|
||||
# --- tags ------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _music_on_four(db, make_user):
|
||||
"""A tag on four channels: two followed, one followed but hidden, one no longer followed."""
|
||||
user, other = make_user(), make_user()
|
||||
tag = service.create_tag(db, user, "Music", "other", None)
|
||||
for cid in ("UCa", "UCb", "UChid", "UCgone"):
|
||||
db.add(Channel(id=cid, title=cid))
|
||||
db.flush()
|
||||
for cid in ("UCa", "UCb"):
|
||||
db.add(Subscription(user_id=user.id, channel_id=cid))
|
||||
db.add(Subscription(user_id=user.id, channel_id="UChid", hidden=True))
|
||||
db.add(Subscription(user_id=other.id, channel_id="UCgone"))
|
||||
for cid in ("UCa", "UCb", "UChid", "UCgone"):
|
||||
db.add(ChannelTag(channel_id=cid, tag_id=tag.id, user_id=user.id, source="user"))
|
||||
db.commit()
|
||||
return user, tag
|
||||
|
||||
|
||||
def test_a_tag_counts_apart_the_links_on_channels_the_user_still_follows(db, make_user):
|
||||
"""An unsubscribe and a hide keep the channel's tag links; the Channels page's filter lists only
|
||||
the followed channels not hidden, so its count must not hold the others."""
|
||||
user, tag = _music_on_four(db, make_user)
|
||||
[(_, count, subscribed)] = [r for r in queries.tags_with_counts(db, user) if r[0].id == tag.id]
|
||||
assert (count, subscribed) == (4, 2)
|
||||
|
||||
|
||||
def test_the_tag_list_sends_each_count_under_its_own_name(db, make_user, as_user):
|
||||
"""Both counts are ints, so a swap would type-check: the answer itself pins which is which."""
|
||||
user, tag = _music_on_four(db, make_user)
|
||||
[row] = [t for t in as_user(user).get("/api/tags").json() if t["id"] == tag.id]
|
||||
assert (row["channel_count"], row["subscribed_count"]) == (4, 2)
|
||||
|
||||
|
||||
def test_a_tag_is_made_once_per_name_and_an_unknown_category_is_other(db, make_user):
|
||||
user = make_user()
|
||||
tag = service.create_tag(db, user, "Later", "nonsense", "#fff")
|
||||
@@ -190,9 +222,6 @@ def test_a_cleared_resume_point_drops_a_bare_row_and_zeroes_one_with_a_status(db
|
||||
db.commit()
|
||||
assert _state(db, user, "v1") is None
|
||||
assert (_state(db, user, "v2").status, _state(db, user, "v2").position_seconds) == ("hidden", 0)
|
||||
service.clear_state(db, user, "v2")
|
||||
db.commit()
|
||||
assert _state(db, user, "v2") is None
|
||||
|
||||
|
||||
# --- playlists -------------------------------------------------------------------------------
|
||||
@@ -213,9 +242,8 @@ def test_an_add_is_idempotent_and_lands_where_asked(db, make_user):
|
||||
pl = service.create_playlist(db, user, "L")
|
||||
assert service.add_item(db, pl, "a", mark_dirty=False) is True
|
||||
assert service.add_item(db, pl, "a", mark_dirty=False) is False
|
||||
service.add_to_playlist(db, pl, "b", None)
|
||||
service.add_to_playlist(db, pl, "c", 0)
|
||||
service.add_to_playlist(db, pl, "a", 2) # already there: stays put
|
||||
assert service.add_item(db, pl, "b", mark_dirty=True) is True
|
||||
assert service.restore_items(db, pl, ["c", "a"], [0, 2]) == 1 # "a" is there: it stays put
|
||||
assert _uncommitted(db)
|
||||
db.commit()
|
||||
assert _ids(db, pl) == ["c", "a", "b"]
|
||||
@@ -224,7 +252,7 @@ def test_an_add_is_idempotent_and_lands_where_asked(db, make_user):
|
||||
db.commit()
|
||||
assert _ids(db, pl) == ["c", "b"]
|
||||
assert queries.item_counts(db, [pl.id]) == {pl.id: 2}
|
||||
assert queries.durations(db, [pl.id]) == {pl.id: 120}
|
||||
assert queries.durations(db, [pl.id]) == {pl.id: (120, 0)}
|
||||
assert queries.containing(db, [pl.id], "b") == {pl.id}
|
||||
|
||||
|
||||
@@ -238,14 +266,13 @@ def test_a_move_never_leaves_a_video_in_two_lists(db, make_user):
|
||||
service.add_item(db, dst, "b", mark_dirty=False)
|
||||
db.commit()
|
||||
with pytest.raises(service.MoveRefused, match="same_playlist"):
|
||||
service.move_item(db, src, src, "a")
|
||||
service.move_item(db, src, dst, "a")
|
||||
service.move_item(db, src, dst, "b") # already in the target: dropped from the source
|
||||
service.move_items(db, src, src, ["a"])
|
||||
assert service.move_items(db, src, dst, ["a"]) == (["a"], [])
|
||||
# already in the target: dropped from the source
|
||||
assert service.move_items(db, src, dst, ["b"]) == ([], ["b"])
|
||||
db.commit()
|
||||
assert (_ids(db, src), _ids(db, dst)) == ([], ["b", "a"])
|
||||
with pytest.raises(service.MoveRefused, match="not_in_playlist"):
|
||||
service.move_item(db, src, dst, "a")
|
||||
db.rollback()
|
||||
assert service.move_items(db, src, dst, ["a"]) == ([], [])
|
||||
|
||||
|
||||
def test_a_reorder_keeps_what_it_left_out_after_the_listed(db, make_user):
|
||||
@@ -267,7 +294,7 @@ def test_a_rename_and_a_delete(db, make_user):
|
||||
service.add_item(db, pl, "a", mark_dirty=False)
|
||||
service.rename_playlist(db, pl, "New")
|
||||
db.commit()
|
||||
assert queries.playlist_totals(db, pl.id) == (1, 60, None)
|
||||
assert queries.playlist_totals(db, pl.id) == (1, 60, 0, None)
|
||||
assert pl.name == "New"
|
||||
service.delete_playlist(db, pl)
|
||||
db.commit()
|
||||
|
||||
@@ -0,0 +1,770 @@
|
||||
"""The refusal scan (`refusal_sites.py`) on small sources: each shape on its closed list is followed to its
|
||||
values, and each exit from that list is reported, unread or as an unread code. A scan that silently drops a
|
||||
shape is a guard green over nothing, so every exit is pinned here, not only the paths the app takes. Pure:
|
||||
no app import, no database."""
|
||||
|
||||
import textwrap
|
||||
|
||||
from tests.refusal_sites import scan, scan_app
|
||||
|
||||
|
||||
def _scan(**files: str) -> dict:
|
||||
return scan(
|
||||
{f"{name.replace('__', '/')}.py": textwrap.dedent(src) for name, src in files.items()}
|
||||
)
|
||||
|
||||
|
||||
def _code(result: dict, code: str) -> dict:
|
||||
return result["codes"][code]
|
||||
|
||||
|
||||
def _unread(result: dict, code: str) -> list[str]:
|
||||
return [u.split(": ", 1)[1] for u in _code(result, code)["unread"]]
|
||||
|
||||
|
||||
REFUSAL = """
|
||||
from fastapi import HTTPException
|
||||
|
||||
def refused(status: int, reason: str, **params: object) -> HTTPException:
|
||||
return HTTPException(status_code=status, detail={**params, "code": "refused", "reason": reason})
|
||||
"""
|
||||
|
||||
|
||||
def test_a_literal_wherever_it_sits_in_the_dict():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(x):
|
||||
raise HTTPException(409, {"code": "clip", "limit_s": 5, "reason": "too_long"})
|
||||
def g(x):
|
||||
raise HTTPException(409, {"reason": "before", "code": "clip"})
|
||||
def h(x):
|
||||
raise HTTPException(409, {
|
||||
"code": "clip", # a comment between
|
||||
"reason": "after_comment",
|
||||
})
|
||||
def i(x):
|
||||
raise HTTPException(409, detail=dict(code="clip", reason="by_call"))
|
||||
"""
|
||||
)
|
||||
assert _code(r, "clip")["reasons"] == ["after_comment", "before", "by_call", "too_long"]
|
||||
assert _code(r, "clip")["unread"] == []
|
||||
|
||||
|
||||
def test_both_arms_of_a_conditional_and_a_none_arm_makes_the_site_reasonless():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(x):
|
||||
raise HTTPException(409, {"code": "edit", "reason": "a" if x else "b"})
|
||||
def g(x):
|
||||
raise HTTPException(409, {"code": "playlist", "reason": "lit" if x else None})
|
||||
def h(x):
|
||||
raise HTTPException(409, {"code": "busy", "reason": None})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "edit")["reasons"] == ["a", "b"] and _code(r, "edit")["reasonless"] == []
|
||||
assert _code(r, "playlist")["reasons"] == ["lit"]
|
||||
assert _code(r, "playlist")["reasonless"] == ["a.py:5"]
|
||||
assert _code(r, "busy")["reasonless"] == ["a.py:7"]
|
||||
|
||||
|
||||
def test_what_the_list_does_not_hold_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(x, svc):
|
||||
raise HTTPException(409, {"code": "edit", "reason": x.why or "c"})
|
||||
def g(svc):
|
||||
raise HTTPException(409, {"code": "edit", "reason": svc.reason()})
|
||||
def h():
|
||||
raise HTTPException(409, {"code": "edit", "reason": 5})
|
||||
def i(e):
|
||||
raise HTTPException(409, {"code": "edit", "reason": f"x{e}"})
|
||||
"""
|
||||
)
|
||||
assert _unread(r, "edit") == ["x.why or 'c'", "svc.reason()", "5", "f'x{e}'"]
|
||||
assert _code(r, "edit")["reasons"] == []
|
||||
|
||||
|
||||
def test_a_helper_is_followed_to_every_call_however_it_is_written():
|
||||
r = _scan(
|
||||
refusal=REFUSAL,
|
||||
routes__a="""
|
||||
from http import HTTPStatus
|
||||
from app import refusal
|
||||
from app.refusal import refused
|
||||
|
||||
def f(status):
|
||||
raise refused(status, "by_name_status")
|
||||
def g():
|
||||
raise refused(HTTPStatus.CONFLICT, reason="by_keyword")
|
||||
def h():
|
||||
raise refusal.refused(409, "qualified")
|
||||
def i(x):
|
||||
raise refused(409, "a" if x else "second_arm")
|
||||
""",
|
||||
routes__b="""
|
||||
from other.lib import refused
|
||||
|
||||
def f():
|
||||
refused("an outside library's, not the app's")
|
||||
""",
|
||||
)
|
||||
assert _code(r, "refused")["reasons"] == [
|
||||
"a",
|
||||
"by_keyword",
|
||||
"by_name_status",
|
||||
"qualified",
|
||||
"second_arm",
|
||||
]
|
||||
assert _code(r, "refused")["unread"] == []
|
||||
|
||||
|
||||
def test_a_call_the_helper_cannot_be_followed_through_is_unread():
|
||||
r = _scan(
|
||||
refusal=REFUSAL,
|
||||
routes__a="""
|
||||
from app.refusal import refused
|
||||
|
||||
def f(args, kw):
|
||||
pick = refused
|
||||
raise refused(*args)
|
||||
def g(kw):
|
||||
raise refused(409, **kw)
|
||||
def h():
|
||||
raise refused(409)
|
||||
""",
|
||||
routes__b="""
|
||||
import app.refusal
|
||||
|
||||
def f():
|
||||
raise app.refusal.refused(409, "dotted")
|
||||
""",
|
||||
routes__c="""
|
||||
from ..refusal import refused
|
||||
|
||||
def f():
|
||||
raise refused(409, "relative")
|
||||
""",
|
||||
)
|
||||
unread = _unread(r, "refused")
|
||||
assert len(unread) == 6
|
||||
assert sum("the helper used as a value" in u for u in unread) == 1
|
||||
assert sum("a reference the scan cannot resolve" in u for u in unread) == 2
|
||||
assert sum("does not pass reason" in u for u in unread) == 2 # `**kw`, and nothing at all
|
||||
assert any(u == "refused(*args)" for u in unread)
|
||||
assert _code(r, "refused")["reasons"] == []
|
||||
|
||||
|
||||
def test_a_default_is_not_followed_a_call_must_pass_the_reason():
|
||||
r = _scan(
|
||||
routes___common="""
|
||||
def not_found(what: str = "generic"):
|
||||
return HTTPException(404, {"code": "not_found", "reason": what})
|
||||
|
||||
def get_or_404(db, id_, *, what: str = "generic"):
|
||||
raise not_found(what)
|
||||
""",
|
||||
routes__a="""
|
||||
from app.routes._common import get_or_404, not_found
|
||||
|
||||
def f(db):
|
||||
get_or_404(db, 1, what="user")
|
||||
get_or_404(db, 2)
|
||||
raise not_found("debug_session")
|
||||
def g():
|
||||
raise not_found()
|
||||
""",
|
||||
)
|
||||
assert _code(r, "not_found")["reasons"] == ["debug_session", "user"]
|
||||
assert [u.endswith("(does not pass what)") for u in _unread(r, "not_found")] == [True, True]
|
||||
|
||||
|
||||
def test_a_positional_only_reason_passed_by_name_is_not_bound_to_it():
|
||||
r = _scan(
|
||||
a="""
|
||||
def deny(reason: str, /, **params):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
|
||||
def f():
|
||||
raise deny("by_position")
|
||||
def g():
|
||||
raise deny(reason="lands_in_params")
|
||||
"""
|
||||
)
|
||||
assert _code(r, "playlist")["reasons"] == ["by_position"]
|
||||
assert _unread(r, "playlist") == ["deny(reason='lands_in_params') (does not pass reason)"]
|
||||
|
||||
|
||||
def test_a_helper_reaching_two_sites_gives_both_its_reasons_and_one_reached_again_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
def deny(reason: str, admin):
|
||||
if admin:
|
||||
return HTTPException(409, {"code": "activate", "reason": reason})
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
|
||||
def again(reason: str, n):
|
||||
if n:
|
||||
return again(reason, n - 1)
|
||||
return HTTPException(409, {"code": "edit", "reason": reason})
|
||||
|
||||
def f():
|
||||
raise deny("foo", True)
|
||||
def g():
|
||||
raise again("bar", 3)
|
||||
"""
|
||||
)
|
||||
assert _code(r, "activate")["reasons"] == ["foo"]
|
||||
assert _code(r, "playlist")["reasons"] == ["foo"]
|
||||
assert _code(r, "edit")["reasons"] == ["bar"]
|
||||
assert any("reached again while it is followed" in u for u in _unread(r, "edit"))
|
||||
|
||||
|
||||
def test_a_parameter_its_body_binds_again_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
from typing import Literal
|
||||
|
||||
def refused(status, reason: str = ""):
|
||||
reason = reason or "fallback"
|
||||
return HTTPException(status, {"code": "refused", "reason": reason})
|
||||
|
||||
def typed(reason: Literal["a"]):
|
||||
if not reason:
|
||||
reason = "b"
|
||||
return HTTPException(409, {"code": "edit", "reason": reason})
|
||||
|
||||
def f():
|
||||
raise refused(409, "x")
|
||||
"""
|
||||
)
|
||||
assert _code(r, "refused")["reasons"] == []
|
||||
assert _unread(r, "refused") == ["reason (a parameter its body binds again)"]
|
||||
assert _unread(r, "edit") == ["reason (a parameter its body binds again)"]
|
||||
|
||||
|
||||
def test_a_local_is_followed_only_through_plain_assignments():
|
||||
r = _scan(
|
||||
a="""
|
||||
from typing import Literal
|
||||
|
||||
def f(x):
|
||||
why = "first"
|
||||
if x:
|
||||
why = "second" if x > 1 else "third"
|
||||
raise HTTPException(409, {"code": "edit", "reason": why})
|
||||
def g(svc):
|
||||
hit, why = svc.fetch()
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
def h(items):
|
||||
for why in items:
|
||||
pass
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
def i():
|
||||
why: Literal["typed"] = pick()
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
def j(items):
|
||||
LABELS[why] = "NotAReason"
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
def k(*why):
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "edit")["reasons"] == ["first", "second", "third"]
|
||||
assert _code(r, "playlist")["reasons"] == []
|
||||
assert _unread(r, "playlist") == [
|
||||
"why",
|
||||
"why",
|
||||
"why",
|
||||
"why",
|
||||
"why",
|
||||
] # unpacked, a loop, annotated, unbound, `*args`
|
||||
|
||||
|
||||
def test_a_call_answers_with_its_literal_return_type():
|
||||
r = _scan(
|
||||
a="""
|
||||
from typing import Literal
|
||||
|
||||
def why(x) -> Literal["quota", "gone"]:
|
||||
return "quota"
|
||||
def how(x) -> str:
|
||||
return "x"
|
||||
|
||||
def f(x):
|
||||
raise HTTPException(409, {"code": "youtube", "reason": why(x)})
|
||||
def g(x):
|
||||
raise HTTPException(409, {"code": "youtube", "reason": how(x)})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "youtube")["reasons"] == ["gone", "quota"]
|
||||
assert _unread(r, "youtube") == ["how(x)"]
|
||||
|
||||
|
||||
def test_a_module_constant_is_read_only_as_mod_name_and_only_when_bound_once():
|
||||
r = _scan(
|
||||
urlguard="""
|
||||
REASON_MALFORMED = "malformed"
|
||||
TWICE = "a"
|
||||
TWICE = "b"
|
||||
if FLAG:
|
||||
LATER = "c"
|
||||
""",
|
||||
routes__a="""
|
||||
from app import urlguard
|
||||
|
||||
BARE = "bare"
|
||||
|
||||
def f():
|
||||
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.REASON_MALFORMED})
|
||||
def g():
|
||||
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.TWICE})
|
||||
def h():
|
||||
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.LATER})
|
||||
def i():
|
||||
raise HTTPException(409, {"code": "url_blocked", "reason": BARE})
|
||||
""",
|
||||
)
|
||||
assert _code(r, "url_blocked")["reasons"] == ["malformed"]
|
||||
assert _unread(r, "url_blocked") == ["urlguard.TWICE", "urlguard.LATER", "BARE"]
|
||||
|
||||
|
||||
def test_a_name_imported_from_two_places_or_under_an_if_is_not_looked_up():
|
||||
r = _scan(
|
||||
m1='R = Literal["one"]',
|
||||
m2='R = Literal["two"]',
|
||||
a="""
|
||||
def f():
|
||||
from app.m1 import R
|
||||
def g():
|
||||
from app.m2 import R
|
||||
def h(reason: R):
|
||||
return HTTPException(409, {"code": "edit", "reason": reason})
|
||||
""",
|
||||
b="""
|
||||
if FLAG:
|
||||
from app.m1 import R
|
||||
def h(reason: R):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
""",
|
||||
c="""
|
||||
from app.m1 import R
|
||||
def h(reason: R):
|
||||
return HTTPException(409, {"code": "clip", "reason": reason})
|
||||
""",
|
||||
)
|
||||
assert _code(r, "edit")["reasons"] == [] and _unread(r, "edit") == ["reason"]
|
||||
assert _code(r, "playlist")["reasons"] == [] and _unread(r, "playlist") == ["reason"]
|
||||
assert _code(r, "clip")["reasons"] == ["one"]
|
||||
|
||||
|
||||
def test_a_caught_exception_answers_with_the_literal_its_class_declares_on_self():
|
||||
r = _scan(
|
||||
errors="""
|
||||
from typing import Literal
|
||||
ClipReason = Literal["bad_range", "too_short"]
|
||||
EditReason = Literal["not_media"]
|
||||
|
||||
class ClipError(Exception):
|
||||
def __init__(self, reason: ClipReason, **params: float):
|
||||
self.reason: ClipReason = reason
|
||||
self.params = params
|
||||
|
||||
class EditError(Exception):
|
||||
reason: EditReason
|
||||
|
||||
class Untyped(Exception):
|
||||
def __init__(self, reason: str):
|
||||
self.reason = reason
|
||||
|
||||
class Narrower(ClipError):
|
||||
pass
|
||||
""",
|
||||
routes__a="""
|
||||
from app import errors
|
||||
|
||||
def f():
|
||||
try:
|
||||
pass
|
||||
except errors.ClipError as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except (errors.ClipError, errors.EditError) as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except errors.EditError as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except errors.Untyped as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except errors.Narrower as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except errors.ClipError as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e})
|
||||
""",
|
||||
)
|
||||
assert _code(r, "clip")["reasons"] == ["bad_range", "too_short"]
|
||||
assert _unread(r, "clip") == [
|
||||
"e.reason", # a tuple of classes
|
||||
"e.reason (EditError.reason has no Literal type)", # declared on the class body only
|
||||
"e.reason (Untyped.reason has no Literal type)",
|
||||
"e.reason (Narrower.reason has no Literal type)", # a subclass: not looked up the bases
|
||||
"e", # the exception itself
|
||||
]
|
||||
|
||||
|
||||
def test_a_parameter_answers_with_its_literal_or_its_class_and_any_other_type_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
from typing import Literal
|
||||
Why = Literal["one", "two"]
|
||||
|
||||
class Err(Exception):
|
||||
def __init__(self, reason: Literal["three"]):
|
||||
self.reason: Literal["three"] = reason
|
||||
|
||||
def typed(reason: Why):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
def inline(reason: Literal["four"]):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
def by_class(exc: Err):
|
||||
return HTTPException(409, {"code": "playlist", "reason": exc.reason})
|
||||
def rebound(exc: Err):
|
||||
exc = other()
|
||||
return HTTPException(409, {"code": "playlist", "reason": exc.reason})
|
||||
def optional(reason: Literal["five"] | None = None):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
def quoted(reason: "Why"):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
def number(reason: int):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
|
||||
class C:
|
||||
def method(self, reason: str):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "playlist")["reasons"] == ["four", "one", "three", "two"]
|
||||
unread = _unread(r, "playlist")
|
||||
# a parameter bound again, `| None`, a quoted annotation, `int`
|
||||
assert unread[:4] == ["exc.reason", "reason", "reason", "reason"]
|
||||
assert "method(reason) is not a module-level helper" in unread[4]
|
||||
assert len(unread) == 5
|
||||
|
||||
|
||||
def test_a_str_helper_nobody_calls_sends_nothing():
|
||||
r = _scan(
|
||||
a="""
|
||||
def helper(reason: str):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "playlist") == {"reasons": [], "unread": [], "sites": 1, "reasonless": []}
|
||||
|
||||
|
||||
def test_a_map_lookup_is_its_values_and_its_keys_must_be_exactly_what_the_index_can_be():
|
||||
src = """
|
||||
from typing import Literal
|
||||
SettingReason = Literal["too_low", "too_high"]
|
||||
|
||||
class InvalidSetting(ValueError):
|
||||
def __init__(self, key: str, reason: SettingReason, **params: object):
|
||||
self.reason: SettingReason = reason
|
||||
self.params = params
|
||||
|
||||
_MAP: dict[SettingReason, str] = {{{keys}}}
|
||||
|
||||
def refused(status: int, reason: str, **params: object):
|
||||
return HTTPException(status, {{**params, "code": "refused", "reason": reason}})
|
||||
|
||||
def setting_refused(exc: InvalidSetting):
|
||||
return refused(400, _MAP[{index}], **exc.params)
|
||||
"""
|
||||
both = '"too_low": "setting_too_low", "too_high": "setting_too_high"'
|
||||
whole = _scan(a=src.format(keys=both, index="exc.reason"))
|
||||
assert _code(whole, "refused")["reasons"] == ["setting_too_high", "setting_too_low"]
|
||||
assert whole["errors"] == [] and whole["unreadCodes"] == []
|
||||
for keys, index in [
|
||||
('"too_low": "setting_too_low"', "exc.reason"), # a key short
|
||||
(both + ', "gone": "setting_gone"', "exc.reason"), # a key no index can be
|
||||
(both, "exc.key"), # an index the scan cannot read
|
||||
]:
|
||||
r = _scan(a=src.format(keys=keys, index=index))
|
||||
assert len(r["errors"]) == 1 and "not what its index can be" in r["errors"][0]
|
||||
|
||||
|
||||
def test_a_lookup_in_anything_but_a_module_dict_of_literal_keys_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
BUILT = make()
|
||||
NUMBERED = {1: "one"}
|
||||
|
||||
def f(table, k):
|
||||
raise HTTPException(409, {"code": "edit", "reason": table[k]})
|
||||
def g(k):
|
||||
raise HTTPException(409, {"code": "edit", "reason": BUILT[k]})
|
||||
def h(k):
|
||||
raise HTTPException(409, {"code": "edit", "reason": NUMBERED[k]})
|
||||
"""
|
||||
)
|
||||
assert _unread(r, "edit") == ["table[k]", "BUILT[k]", "NUMBERED[k]"]
|
||||
assert r["errors"] == []
|
||||
|
||||
|
||||
def test_a_code_must_be_one_literal():
|
||||
r = _scan(
|
||||
a="""
|
||||
BUSY = "busy"
|
||||
def f(code):
|
||||
raise HTTPException(409, {"code": "busy"})
|
||||
def g(code):
|
||||
raise HTTPException(409, {"code": code})
|
||||
def h():
|
||||
raise HTTPException(409, {"code": BUSY})
|
||||
def i(x):
|
||||
raise HTTPException(409, {"code": "a" if x else "b"})
|
||||
def j(x):
|
||||
raise HTTPException(409, {"code": "a" if x else None})
|
||||
def k(key):
|
||||
raise HTTPException(409, {"code": "edit", key: "late"})
|
||||
"""
|
||||
)
|
||||
assert r["codes"]["busy"]["reasonless"] == ["a.py:4"]
|
||||
assert set(r["codes"]) == {"busy"}
|
||||
assert len(r["unreadCodes"]) == 5
|
||||
assert r["unreadCodes"][-1].endswith(
|
||||
"(a spread or a computed key the code and the reason do not follow)"
|
||||
)
|
||||
|
||||
|
||||
def test_a_spread_or_a_computed_key_stands_before_both_the_code_and_the_reason():
|
||||
def unread_codes(detail: str) -> list[str]:
|
||||
return _scan(a=f"def f(x, more, key):\n raise HTTPException(409, {detail})\n")[
|
||||
"unreadCodes"
|
||||
]
|
||||
|
||||
assert unread_codes('{**more, "code": "refused", "reason": "a"}') == []
|
||||
assert unread_codes('{key: 1, **more, "reason": "a", "code": "refused"}') == []
|
||||
for late in [
|
||||
'{"code": "refused", "reason": "a", **more}', # the spread may replace both
|
||||
'{"code": "refused", **more, "reason": "a"}', # … the code
|
||||
'{"reason": "a", **more, "code": "refused"}', # … the reason
|
||||
'{**more, "code": "refused"}', # no reason after it: the spread may carry one
|
||||
'{"code": "refused", "reason": "a", key: "x"}', # a key that may be either
|
||||
'dict(**more, code="refused", reason="a")', # dict() raises on a repeated key
|
||||
'dict(more, code="refused", reason="a")',
|
||||
]:
|
||||
assert unread_codes(late), late
|
||||
|
||||
|
||||
def test_a_detail_stands_only_where_it_is_sent():
|
||||
def result(body: str, extra: str = "") -> dict:
|
||||
return _scan(a=f"def f(x, cached, fix):\n{textwrap.indent(body, ' ')}\n{extra}")
|
||||
|
||||
for body in [
|
||||
'detail = {"code": "login", "reason": "a"}\nraise HTTPException(401, detail)',
|
||||
'if (detail := {"code": "login", "reason": "a"}):\n raise HTTPException(401, detail)',
|
||||
'raise HTTPException(401, {"code": "login", "reason": "a"}.copy())',
|
||||
'raise HTTPException(401, fix({"code": "login", "reason": "a"}))',
|
||||
'raise HTTPException(401, cached or {"code": "login", "reason": "a"})',
|
||||
'return JSONResponse({"code": "login", "reason": "a"}, status_code=401)', # the whole body
|
||||
'raise HTTPException(401, None, {"code": "login", "reason": "a"})', # the headers
|
||||
'raise HTTPException(401, headers={"code": "login", "reason": "a"})',
|
||||
'raise HTTPException(401, "a" if {"code": "login", "reason": "a"} else "b")',
|
||||
]:
|
||||
r = result(body)
|
||||
assert r["unreadCodes"] and "login" not in r["codes"], body
|
||||
assert "(a detail put where the scan cannot follow it)" in r["unreadCodes"][0], body
|
||||
held = _scan(a='class C:\n detail = {"code": "login"}\n')
|
||||
assert "(a detail put where the scan cannot follow it)" in held["unreadCodes"][0]
|
||||
|
||||
sent = result(
|
||||
'raise HTTPException(401, {"code": "login", "reason": "a"} if x else {"code": "login"})\n'
|
||||
'raise HTTPException(401, detail=dict(code="login", reason="b"))\n'
|
||||
'return JSONResponse(content={"detail": {"code": "login", "reason": "c"}})'
|
||||
)
|
||||
assert sent["unreadCodes"] == [] and _code(sent, "login")["reasons"] == ["a", "b", "c"]
|
||||
|
||||
|
||||
def test_a_factory_is_a_detail_only_when_each_return_is_one_and_each_call_is_sent():
|
||||
factory = """
|
||||
def quota(e) -> dict:
|
||||
return {"code": "quota", "reason": "files"}
|
||||
"""
|
||||
sent = "def f(e):\n raise HTTPException(422, detail=quota(e))\n"
|
||||
assert _scan(a=factory + sent)["codes"]["quota"]["reasons"] == ["files"]
|
||||
for change in [
|
||||
"def g(e):\n d = quota(e)\n d['code'] = 'other'\n raise HTTPException(422, d)\n",
|
||||
"def g(e):\n raise HTTPException(422, detail=quota(e).copy())\n",
|
||||
"pick = quota\n",
|
||||
]:
|
||||
r = _scan(a=factory + sent + change)
|
||||
assert r["unreadCodes"] and "quota" not in r["codes"], change
|
||||
twice = _scan(a="quota = None\n" + factory + sent) # the def is the last of two bindings
|
||||
assert twice["unreadCodes"] and "quota" not in twice["codes"]
|
||||
for other in [
|
||||
" if e:\n return None\n",
|
||||
" if e:\n return cached\n",
|
||||
" if e:\n return {'code': 'quota', 'reason': 'x'} if e > 1 else None\n",
|
||||
" log({'code': 'quota', 'reason': 'logged'})\n", # a detail it does not return
|
||||
]:
|
||||
bent = factory.replace("-> dict:\n", "-> dict:\n" + other)
|
||||
assert _scan(a=bent + sent)["unreadCodes"], other
|
||||
method = "class C:\n def quota(self):\n return {'code': 'quota', 'reason': 'm'}\n"
|
||||
assert _scan(a=method)["unreadCodes"]
|
||||
dotted = _scan(
|
||||
a=factory,
|
||||
routes__b="import app.a\n\ndef g(e):\n raise HTTPException(422, detail=app.a.quota(e))\n",
|
||||
)
|
||||
assert dotted["unreadCodes"] and "quota" not in dotted["codes"]
|
||||
|
||||
|
||||
def test_a_name_a_lambda_a_comprehension_a_nonlocal_or_a_global_binds_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(svc):
|
||||
why = "invalid"
|
||||
make = lambda why: HTTPException(401, {"code": "login", "reason": why})
|
||||
raise make(svc.state)
|
||||
def g(svc):
|
||||
why = "invalid"
|
||||
raise [HTTPException(401, {"code": "login", "reason": why}) for why in svc.all][0]
|
||||
def h(svc):
|
||||
why = "invalid"
|
||||
def inner():
|
||||
nonlocal why
|
||||
why = svc.reason
|
||||
inner()
|
||||
raise HTTPException(401, {"code": "login", "reason": why})
|
||||
def i(svc):
|
||||
global why
|
||||
why = "invalid"
|
||||
raise HTTPException(401, {"code": "login", "reason": why})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "login")["reasons"] == []
|
||||
assert sorted(_unread(r, "login")) == [
|
||||
"why (bound by a lambda or a comprehension)",
|
||||
"why (bound by a lambda or a comprehension)",
|
||||
"why (named by a nonlocal or a global)",
|
||||
"why (named by a nonlocal or a global)",
|
||||
]
|
||||
|
||||
|
||||
def test_a_local_that_reaches_itself_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(x):
|
||||
why = "a"
|
||||
if x:
|
||||
why = why if x > 1 else "b"
|
||||
raise HTTPException(409, {"code": "edit", "reason": why})
|
||||
def g(x):
|
||||
a = "z"
|
||||
b = a
|
||||
a = b
|
||||
raise HTTPException(409, {"code": "playlist", "reason": a})
|
||||
def h(x):
|
||||
why = "c"
|
||||
if x:
|
||||
raise HTTPException(409, {"code": "busy", "reason": why})
|
||||
raise HTTPException(409, {"code": "busy", "reason": why})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "busy")["reasons"] == ["c"] and _code(r, "busy")["unread"] == []
|
||||
assert _code(r, "edit")["reasons"] == ["a", "b"]
|
||||
assert _unread(r, "edit") == ["why (reached again while it is resolved)"]
|
||||
assert _code(r, "playlist")["reasons"] == ["z"]
|
||||
assert any("reached again while it is resolved" in u for u in _unread(r, "playlist"))
|
||||
|
||||
|
||||
def test_a_module_constant_or_a_map_changed_elsewhere_is_unread():
|
||||
def reasons(urlguard: str, route: str = "", **more: str) -> tuple[list[str], list[str]]:
|
||||
r = _scan(
|
||||
**more,
|
||||
urlguard="REASON = 'malformed'\n_MAP = {'a': 'map_a'}\n" + urlguard,
|
||||
routes__a="from app import urlguard\n"
|
||||
+ route
|
||||
+ "\ndef f(k: Literal['a']):\n"
|
||||
+ " raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard.REASON})\n"
|
||||
+ " raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard._MAP[k]})\n",
|
||||
)
|
||||
return _code(r, "url_blocked")["reasons"], _unread(r, "url_blocked")
|
||||
|
||||
assert reasons("def get(k):\n return _MAP.get(k)\n") == (["malformed", "map_a"], [])
|
||||
other = "from app import other\ndef g():\n other.REASON = 'x'\n" # another module's REASON
|
||||
assert reasons("", other, other="REASON = 'y'\n") == (["malformed", "map_a"], [])
|
||||
shadow = (
|
||||
"def h(*urlguard):\n"
|
||||
" raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard.REASON})\n"
|
||||
)
|
||||
assert reasons("", shadow)[1] == ["urlguard.REASON"]
|
||||
for urlguard, route in [
|
||||
("def reset():\n global REASON\n REASON = 'x'\n", ""),
|
||||
("", "def g():\n urlguard.REASON = 'x'\n"),
|
||||
("", "def g():\n del urlguard.REASON\n"),
|
||||
]:
|
||||
assert reasons(urlguard, route)[0] == ["map_a"], (urlguard, route)
|
||||
for urlguard in [
|
||||
"_MAP['a'] = compute()\n",
|
||||
"def init(x):\n _MAP['a'] = x\n",
|
||||
"_MAP.update(a='other')\n",
|
||||
"def drop():\n del _MAP['a']\n",
|
||||
"register(_MAP)\n",
|
||||
]:
|
||||
got, unread = reasons(urlguard)
|
||||
assert got == ["malformed"] and "(the map is changed after it is built)" in unread[0]
|
||||
|
||||
|
||||
def test_a_helper_bound_twice_is_not_followed():
|
||||
r = _scan(
|
||||
refusal="refused = None\n" + textwrap.dedent(REFUSAL),
|
||||
routes__a="from app.refusal import refused\n\ndef f():\n raise refused(409, 'lost')\n",
|
||||
)
|
||||
assert _code(r, "refused")["reasons"] == []
|
||||
assert _unread(r, "refused")[0].endswith("refused(reason) is not a module-level helper")
|
||||
|
||||
|
||||
def test_a_dotted_path_through_an_app_module_that_goes_nowhere_is_unread():
|
||||
r = scan(
|
||||
{
|
||||
"__init__.py": "",
|
||||
"refusal.py": textwrap.dedent(REFUSAL),
|
||||
"routes/a.py": "import app as a\n\ndef f():\n raise a.refusal.refused(409, 'dotted')\n",
|
||||
}
|
||||
)
|
||||
assert _unread(r, "refused") == ["a.refusal.refused (a reference the scan cannot resolve)"]
|
||||
|
||||
|
||||
def test_a_helper_reached_through_an_app_module_that_does_not_bind_it_is_unread():
|
||||
for shim in [
|
||||
"from app.refusal import *\n",
|
||||
"try:\n from app.refusal import refused\nexcept ImportError:\n refused = None\n",
|
||||
]:
|
||||
r = _scan(
|
||||
refusal=REFUSAL,
|
||||
shim=shim,
|
||||
routes__a="from app.shim import refused\n\ndef f():\n raise refused(409, 'hidden')\n",
|
||||
)
|
||||
assert _unread(r, "refused") == ["refused (a reference the scan cannot resolve)"], shim
|
||||
package = scan(
|
||||
{
|
||||
"pkg/refusal.py": textwrap.dedent(REFUSAL),
|
||||
"pkg/__init__.py": "from .refusal import refused\n",
|
||||
"routes/a.py": "from app.pkg import refused\n\ndef f():\n raise refused(409, 'hidden')\n",
|
||||
}
|
||||
)
|
||||
assert _unread(package, "refused") == ["refused (a reference the scan cannot resolve)"]
|
||||
|
||||
|
||||
def test_the_data_files_are_excluded_by_name_only():
|
||||
r = _scan(
|
||||
plex__tmdb='def f():\n return {"code": "HU", "name": "Hungary"}',
|
||||
plex__other='def f():\n return {"code": "HU"}',
|
||||
)
|
||||
assert r["excluded"] == {"plex/tmdb.py": 1}
|
||||
assert "HU" in r["codes"]
|
||||
|
||||
|
||||
def test_the_app_scans():
|
||||
"""The real tree, on the interpreter the backend runs: every refusal it holds is read (the
|
||||
frontend's contract judges what is read)."""
|
||||
result = scan_app()
|
||||
assert {"refused", "not_found", "login", "clip"} <= set(result["codes"])
|
||||
assert result["errors"] == [] and result["unreadCodes"] == []
|
||||
@@ -101,8 +101,10 @@ def test_a_video_state_cleared_through_the_route_is_gone(db, make_user, as_user)
|
||||
_video(db)
|
||||
db.add(VideoState(user_id=user.id, video_id="v1", status="hidden", position_seconds=90))
|
||||
db.commit()
|
||||
assert as_user(user).delete("/api/videos/v1/state").status_code == 200
|
||||
assert _state(db, user) is None
|
||||
assert as_user(user).delete("/api/videos/v1/progress").status_code == 200
|
||||
db.expire_all()
|
||||
row = _state(db, user)
|
||||
assert (row.status, row.position_seconds) == ("hidden", 0)
|
||||
|
||||
|
||||
# --- saved views ------------------------------------------------------------------------------
|
||||
@@ -308,6 +310,14 @@ def test_clearing_leaves_a_video_the_caller_never_found(db, make_user, as_user):
|
||||
assert r.json() == {"ok": True, "removed": 0}
|
||||
|
||||
|
||||
def test_clearing_refuses_a_body_without_its_list(db, make_user, as_user):
|
||||
# A misnamed key read as an empty list: 200, and nothing cleared.
|
||||
user = make_user()
|
||||
r = as_user(user).post("/api/search/clear", json={"ids": ["s1"]})
|
||||
assert r.status_code == 422
|
||||
assert as_user(user).post("/api/search/clear", json={"video_ids": []}).status_code == 200
|
||||
|
||||
|
||||
# --- Plex show and season marks ---------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -22,6 +22,31 @@ from app.runstatus import RunStatus, RunTrigger
|
||||
_JOB = "unit_test_claim" # not a real job id — exercises the primitive without touching live jobs
|
||||
|
||||
|
||||
class _StrictlyLater(datetime):
|
||||
"""A clock that never repeats or goes back: each reading at least a microsecond after the last."""
|
||||
|
||||
last: datetime | None = None
|
||||
clock: type[datetime] = datetime
|
||||
|
||||
@classmethod
|
||||
def now(cls, tz=None):
|
||||
t = cls.clock.now(tz)
|
||||
if cls.last is not None and t <= cls.last:
|
||||
t = cls.last + timedelta(microseconds=1)
|
||||
cls.last = t
|
||||
return t
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _runs_start_in_order(monkeypatch):
|
||||
"""Runs started in a row start in that order. These tests open several runs within microseconds
|
||||
and read them back newest first, by start time; the wall clock of a container on a VM can step
|
||||
back between two readings (seen: the third of three runs sorted before the second)."""
|
||||
_StrictlyLater.last = None
|
||||
_StrictlyLater.clock = jobruns.datetime
|
||||
monkeypatch.setattr(jobruns, "datetime", _StrictlyLater)
|
||||
|
||||
|
||||
def _release():
|
||||
scheduler._record(_JOB, running=False)
|
||||
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
"""A share link's address says what it opens: /watch/ a video, /file/ anything else, /folder/ a folder.
|
||||
|
||||
Only a word for the reader: the token is the key, so every path opens every link, and the links
|
||||
already sent out on /watch/ keep working for good. DB-lane.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from cryptography.fernet import Fernet
|
||||
|
||||
from app import main, security
|
||||
from app.config import settings
|
||||
from app.downloads import og
|
||||
from app.downloads import probe as probemod
|
||||
|
||||
_PAGE = """<!doctype html><html><head>
|
||||
<!--OG:START--><meta property="og:title" content="Siftlode" /><!--OG:END-->
|
||||
</head><body><script src="/assets/{entry}.js"></script></body></html>"""
|
||||
|
||||
_VIDEO = {
|
||||
"streams": [
|
||||
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
|
||||
{"codec_type": "audio", "codec_name": "aac"},
|
||||
],
|
||||
"format": {"format_name": "mov,mp4", "duration": "5"},
|
||||
}
|
||||
_AUDIO = {
|
||||
"streams": [{"codec_type": "audio", "codec_name": "aac"}],
|
||||
"format": {"format_name": "mov,mp4", "duration": "5"},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pages(tmp_path, monkeypatch):
|
||||
index, watch = tmp_path / "index.html", tmp_path / "watch.html"
|
||||
index.write_text(_PAGE.format(entry="app"), encoding="utf-8")
|
||||
watch.write_text(_PAGE.format(entry="watch"), encoding="utf-8")
|
||||
monkeypatch.setattr(main, "INDEX_HTML", index)
|
||||
monkeypatch.setattr(main, "WATCH_HTML", watch)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def key(monkeypatch):
|
||||
"""A link's address is read back from its encrypted token."""
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def shared(as_user, make_user, tmp_path, monkeypatch, upload_headers):
|
||||
"""Uploads a file ffprobe reads as `data` (None: it cannot read it) and shares it."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "media"))
|
||||
owner = as_user(make_user())
|
||||
|
||||
def share(name: str, data: dict | None, **link: object) -> dict:
|
||||
monkeypatch.setattr(probemod, "probe_json", lambda _p: data)
|
||||
job = owner.post(
|
||||
"/api/files/upload",
|
||||
content=b"some bytes",
|
||||
headers=upload_headers(name, display_name=name),
|
||||
)
|
||||
assert job.status_code == 200, job.text
|
||||
made = owner.post(f"/api/downloads/{job.json()['id']}/links", json=link)
|
||||
assert made.status_code == 200, made.text
|
||||
return made.json()
|
||||
|
||||
share.owner = owner # type: ignore[attr-defined]
|
||||
return share
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "data", "prefix"),
|
||||
[
|
||||
("clip.mp4", _VIDEO, "/watch/"),
|
||||
("song.m4a", _AUDIO, "/file/"),
|
||||
("backup.tar.gz", None, "/file/"),
|
||||
],
|
||||
)
|
||||
def test_a_link_is_given_out_on_the_path_of_what_it_opens(shared, name, data, prefix):
|
||||
made = shared(name, data)
|
||||
assert made["url"] == f"{prefix}{made['token']}"
|
||||
|
||||
|
||||
def test_a_folder_link_is_given_out_on_folder(shared, key):
|
||||
owner = shared.owner
|
||||
folder = owner.post("/api/files/folders", json={"name": "Friends"}).json()
|
||||
made = owner.post(f"/api/downloads/folders/{folder['id']}/links", json={}).json()
|
||||
assert made["url"] == f"/folder/{made['token']}"
|
||||
listed = owner.get(f"/api/downloads/folders/{folder['id']}/links").json()
|
||||
assert [entry["url"] for entry in listed] == [made["url"]]
|
||||
|
||||
|
||||
def test_the_list_gives_the_same_address_as_the_making(shared, key):
|
||||
made = shared("backup.tar.gz", None)
|
||||
owner = shared.owner
|
||||
files = owner.get("/api/files").json()
|
||||
job_id = next(f["id"] for f in files if f["display_name"] == "backup.tar.gz")
|
||||
listed = owner.get(f"/api/downloads/{job_id}/links").json()
|
||||
assert [entry["url"] for entry in listed] == [made["url"]]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["watch", "file", "folder"])
|
||||
@pytest.mark.parametrize(
|
||||
("name", "data"), [("clip.mp4", _VIDEO), ("backup.tar.gz", None)], ids=["video", "file"]
|
||||
)
|
||||
def test_every_path_opens_every_link(pages, shared, client, path, name, data):
|
||||
"""The links already sent on /watch/ open whatever they open, for good."""
|
||||
token = shared(name, data)["token"]
|
||||
r = client.get(f"/{path}/{token}")
|
||||
assert r.status_code == 200
|
||||
assert "/assets/watch.js" in r.text and "/assets/app.js" not in r.text
|
||||
assert f'content="{name}"' in r.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["/file/no-such-token", "/folder/no-such-token", "/file/a/b"])
|
||||
def test_a_new_path_without_a_live_link_still_gets_the_player(pages, client, path):
|
||||
r = client.get(path)
|
||||
assert r.status_code == 200
|
||||
assert "/assets/watch.js" in r.text and "/assets/app.js" not in r.text
|
||||
|
||||
|
||||
def test_a_card_names_a_file_as_a_file(shared, db):
|
||||
made = shared("backup.tar.gz", None)
|
||||
tags = og._watch_tags(made["token"], db)
|
||||
assert f'{made["url"]}"' in tags
|
||||
assert "video.other" not in tags
|
||||
|
||||
locked = shared("notes.tar.gz", None, password="pw")
|
||||
hidden = og._watch_tags(locked["token"], db)
|
||||
assert "Password-protected file" in hidden and "notes" not in hidden
|
||||
assert "video.other" not in hidden
|
||||
|
||||
|
||||
def test_a_card_of_a_video_stays_a_video(shared, db):
|
||||
made = shared("clip.mp4", _VIDEO)
|
||||
tags = og._watch_tags(made["token"], db)
|
||||
assert f"/watch/{made['token']}" in tags and "video.other" in tags
|
||||
locked = shared("other.mp4", _VIDEO, password="pw")
|
||||
assert "Password-protected video" in og._watch_tags(locked["token"], db)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "data", "video"),
|
||||
[("clip.mp4", _VIDEO, True), ("song.m4a", _AUDIO, False), ("backup.tar.gz", None, False)],
|
||||
)
|
||||
def test_a_password_link_says_whether_it_opens_a_video(shared, client, name, data, video):
|
||||
"""The lock screen asks to watch a video and to open anything else, whichever path it is on."""
|
||||
token = shared(name, data, password="pw")["token"]
|
||||
meta = client.get(f"/api/public/watch/{token}").json()
|
||||
assert meta == {"needs_password": True, "kind": "file", "video": video}
|
||||
@@ -0,0 +1,223 @@
|
||||
"""A seeded shuffle walks a stored random key instead of sorting the whole scope. DB lane.
|
||||
|
||||
Every video carries a random number drawn once (`Video.shuffle_key`). A seed picks a point on that
|
||||
ring; the feed's Shuffle sort pages along the ring from there and wraps, and the Home surprise reads
|
||||
a window of it and shuffles that window by the seed (`discover.queries.yt_surprise`). Ordering the
|
||||
scope by md5(id || seed) cost ~350 ms per call on prod and grew with every Shuffle page.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import random
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import event, insert, text
|
||||
|
||||
from app import feedquery
|
||||
from app.catalog.models import Channel, Subscription, Video
|
||||
from app.discover import queries
|
||||
from app.discover.queries import _FEED_BASE
|
||||
|
||||
NOW = datetime.now(timezone.utc)
|
||||
KEYS = {"k05": 0.05, "k20": 0.20, "k40": 0.40, "k60": 0.60, "k80": 0.80, "k95": 0.95}
|
||||
|
||||
|
||||
def _seed(db, make_user, keys=KEYS):
|
||||
user = make_user()
|
||||
db.add(Channel(id="UCring", title="Ring"))
|
||||
db.flush()
|
||||
db.add(Subscription(user_id=user.id, channel_id="UCring"))
|
||||
db.add_all(
|
||||
[
|
||||
Video(
|
||||
id=vid,
|
||||
channel_id="UCring",
|
||||
title=vid,
|
||||
duration_seconds=20 * 60,
|
||||
published_at=NOW - timedelta(days=i + 1),
|
||||
shuffle_key=key,
|
||||
)
|
||||
for i, (vid, key) in enumerate(keys.items())
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
return user
|
||||
|
||||
|
||||
def _ring(seed, keys=KEYS):
|
||||
"""The ring order from the seed's point: the keys at or after it ascending, then the rest."""
|
||||
r = feedquery.shuffle_start(seed)
|
||||
return sorted(keys, key=lambda v: (keys[v] < r, keys[v]))
|
||||
|
||||
|
||||
def _seed_landing_between(lo, hi):
|
||||
return next(s for s in range(1000) if lo < feedquery.shuffle_start(s) < hi)
|
||||
|
||||
|
||||
def test_the_walk_starts_at_the_seeds_point_and_wraps(db, make_user):
|
||||
user = _seed(db, make_user)
|
||||
query, _ = feedquery.filtered_query(db, user, **_FEED_BASE)
|
||||
seed = _seed_landing_between(0.45, 0.55) # starts at k60, wraps to k05 after k95
|
||||
got = [r.id for r in feedquery.shuffle_walk(db, query, seed, 6)]
|
||||
assert got == ["k60", "k80", "k95", "k05", "k20", "k40"]
|
||||
assert [r.id for r in feedquery.shuffle_walk(db, query, seed, 4)] == got[:4]
|
||||
|
||||
|
||||
def test_the_feed_shuffle_pages_along_the_ring(db, make_user, as_user):
|
||||
user = _seed(db, make_user)
|
||||
client = as_user(user)
|
||||
for seed in (_seed_landing_between(0.1, 0.15), _seed_landing_between(0.85, 0.9)):
|
||||
walked, cursor = [], None
|
||||
for _ in range(5):
|
||||
url = f"/api/feed?sort=shuffle&seed={seed}&limit=2" + (
|
||||
f"&cursor={cursor}" if cursor else ""
|
||||
)
|
||||
body = client.get(url).json()
|
||||
walked += [it["id"] for it in body["items"]]
|
||||
cursor = body["next_cursor"]
|
||||
if not body["has_more"]:
|
||||
break
|
||||
assert walked == _ring(seed), seed
|
||||
|
||||
|
||||
def test_a_new_seed_opens_the_shuffle_elsewhere(db, make_user, as_user):
|
||||
user = _seed(db, make_user)
|
||||
client = as_user(user)
|
||||
first = {
|
||||
seed: client.get(f"/api/feed?sort=shuffle&seed={seed}&limit=2").json()["items"][0]["id"]
|
||||
for seed in (_seed_landing_between(0.1, 0.15), _seed_landing_between(0.65, 0.7))
|
||||
}
|
||||
assert sorted(first.values()) == ["k20", "k80"]
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _statements(db):
|
||||
seen: list[str] = []
|
||||
|
||||
def grab(conn, cursor, statement, *args):
|
||||
seen.append(statement)
|
||||
|
||||
engine = db.get_bind()
|
||||
event.listen(engine, "before_cursor_execute", grab)
|
||||
try:
|
||||
yield seen
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", grab)
|
||||
|
||||
|
||||
def _draws_of_videos(seen):
|
||||
return [s for s in seen if "FROM videos" in s and "LIMIT" in s]
|
||||
|
||||
|
||||
def test_neither_draw_orders_the_scope_by_a_hash(db, make_user):
|
||||
# Each draw reads the videos as an ascending range of the key (what ix_videos_shuffle_key serves),
|
||||
# never ordered by a hash or by the ring's CASE (which no index can serve).
|
||||
user = _seed(db, make_user)
|
||||
with _statements(db) as surprise:
|
||||
queries.yt_surprise(db, user, "any", 7, 3)
|
||||
with _statements(db) as page:
|
||||
feedquery.feed_page(
|
||||
db, user, dict(_FEED_BASE), sort="shuffle", seed=7, limit=2, cursor=None, lang="en"
|
||||
)
|
||||
for seen in (surprise, page):
|
||||
draws = _draws_of_videos(seen)
|
||||
assert draws
|
||||
for s in draws:
|
||||
assert "md5(" not in s.lower()
|
||||
assert "ORDER BY videos.shuffle_key, videos.id" in s.replace("\n", " "), s
|
||||
|
||||
|
||||
def test_the_walk_reads_the_ring_index_and_sorts_nothing(db, make_user):
|
||||
# The text of the query is not the plan: each arc must be served by ix_videos_shuffle_key in key
|
||||
# order (an index on another shape, or none, leaves a Sort of the whole scope). A scope of a few
|
||||
# thousand videos, analysed, so the planner weighs it as it does a real one.
|
||||
user = _seed(db, make_user)
|
||||
rng = random.Random(7)
|
||||
db.execute(
|
||||
insert(Video),
|
||||
[
|
||||
{
|
||||
"id": f"bulk{i:05}",
|
||||
"channel_id": "UCring",
|
||||
"title": "b",
|
||||
"duration_seconds": 600,
|
||||
"published_at": NOW - timedelta(minutes=i),
|
||||
"shuffle_key": rng.random(),
|
||||
}
|
||||
for i in range(4000)
|
||||
],
|
||||
)
|
||||
db.commit()
|
||||
db.execute(text("ANALYZE videos"))
|
||||
query, _ = feedquery.filtered_query(db, user, **_FEED_BASE)
|
||||
captured: list[tuple[str, object]] = []
|
||||
|
||||
def grab(conn, cursor, statement, params, *args):
|
||||
if "FROM videos" in statement and "LIMIT" in statement:
|
||||
captured.append((statement, params))
|
||||
|
||||
engine = db.get_bind()
|
||||
event.listen(engine, "before_cursor_execute", grab)
|
||||
try:
|
||||
seed = _seed_landing_between(0.45, 0.55)
|
||||
feedquery.shuffle_walk(db, query, seed, 6) # the first arc, from the seed's point
|
||||
feedquery.shuffle_walk(db, query, seed, 6, (1, 0.3, "bulk00001")) # the wrap, resumed
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", grab)
|
||||
assert len(captured) == 2
|
||||
conn = db.connection()
|
||||
for statement, params in captured:
|
||||
plan = "\n".join(r[0] for r in conn.exec_driver_sql("EXPLAIN " + statement, params))
|
||||
assert "ix_videos_shuffle_key" in plan, plan
|
||||
assert "Sort Key: videos.shuffle_key" not in plan, plan
|
||||
|
||||
|
||||
def test_the_surprise_reads_a_window_of_the_ring(db, make_user, monkeypatch):
|
||||
user = _seed(db, make_user)
|
||||
monkeypatch.setattr(queries, "SURPRISE_WINDOW", 3)
|
||||
for seed in range(20):
|
||||
drawn = {r["id"] for r in queries.yt_surprise(db, user, "any", seed, 3)}
|
||||
assert drawn == set(_ring(seed)[:3]), seed
|
||||
|
||||
|
||||
def test_a_wide_gap_on_the_ring_does_not_make_a_favourite(db, make_user):
|
||||
# One video after an empty half of the ring, twenty packed into the other half. Taking the first
|
||||
# video after the seed's point would deal the lonely one about half the time; the window is
|
||||
# shuffled by the seed, so it leads about as often as any other.
|
||||
keys = {"lonely": 0.5} | {f"p{i:02}": 0.51 + i * 0.02 for i in range(20)}
|
||||
user = _seed(db, make_user, keys)
|
||||
leads = [queries.yt_surprise(db, user, "any", s, 1)[0]["id"] for s in range(90)]
|
||||
assert leads.count("lonely") < 15, leads.count("lonely")
|
||||
|
||||
|
||||
def test_any_seed_lands_on_the_ring():
|
||||
# An int too large for a float overflowed the multiplication: a 500 for a hand-typed seed.
|
||||
for seed in (10**400, -(10**400), 2**53 + 1, -7):
|
||||
assert 0 <= feedquery.shuffle_start(seed) < 1, seed
|
||||
|
||||
|
||||
def test_a_seed_out_of_range_is_refused_not_a_500(db, make_user, as_user):
|
||||
client = as_user(_seed(db, make_user))
|
||||
huge = "9" * 400
|
||||
for url in (
|
||||
f"/api/feed?sort=shuffle&seed={huge}",
|
||||
f"/api/discover?seed={huge}",
|
||||
f"/api/discover/pick?seed={huge}",
|
||||
):
|
||||
assert client.get(url).status_code == 422, url
|
||||
|
||||
|
||||
def test_a_shuffle_cursor_past_a_float_is_a_bad_cursor(db, make_user, as_user):
|
||||
client = as_user(_seed(db, make_user))
|
||||
for k in ("[Infinity, 0.5]", f"[0, {'9' * 400}]"):
|
||||
raw = '{"k": %s, "id": "k20"}' % k
|
||||
cursor = base64.urlsafe_b64encode(raw.encode()).decode()
|
||||
assert client.get(f"/api/feed?sort=shuffle&seed=1&cursor={cursor}").status_code == 400, k
|
||||
|
||||
|
||||
def test_a_new_video_gets_its_own_key(db, make_user):
|
||||
_seed(db, make_user, {})
|
||||
db.add_all([Video(id=f"n{i}", channel_id="UCring", title="n") for i in range(3)])
|
||||
db.commit()
|
||||
keys = [db.get(Video, f"n{i}").shuffle_key for i in range(3)]
|
||||
assert len(set(keys)) == 3 and all(0 <= k < 1 for k in keys)
|
||||
@@ -0,0 +1,349 @@
|
||||
"""Surprise me leans on what the user watches, and stays a surprise.
|
||||
|
||||
The Home hero's surprise tier and the Plex Discover rail were uniform draws over the unseen. They are
|
||||
now weighted draws (`app.draw`): a title's weight is 1 + its affinity — for Plex the genres of the
|
||||
films and shows the user watched (recent ones weigh more) and the TMDB recommendations stored on them;
|
||||
for YouTube the channels the user watches. One slot of three stays a blind draw, and with no watch
|
||||
history the order is exactly the old one. DB lane.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app import draw
|
||||
from app.catalog.models import Channel, Subscription, Video
|
||||
from app.discover import queries
|
||||
from app.plex import affinity, discover as plex_discover
|
||||
from app.plex.models import PlexItem, PlexLibrary, PlexShow, PlexState
|
||||
from app.plex.query import LibraryFilters
|
||||
from app.reader.models import VideoState
|
||||
|
||||
NOW = datetime.now(timezone.utc)
|
||||
SEEDS = range(90)
|
||||
|
||||
|
||||
# ── the draw itself ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_every_third_slot_is_blind_and_nothing_repeats():
|
||||
weighted = ["a", "b", "c", "d", "e", "f"]
|
||||
blind = ["f", "e", "d", "c", "b", "a"]
|
||||
# offset 0: slots 0 and 3 are blind
|
||||
assert draw.blend(weighted, blind, 0, 6) == ["f", "a", "b", "e", "c", "d"]
|
||||
# offset 1: slot 2 and 5
|
||||
assert draw.blend(weighted, blind, 1, 6) == ["a", "b", "f", "c", "d", "e"]
|
||||
assert draw.blend(weighted, blind, 2, 3) == ["a", "f", "b"]
|
||||
|
||||
|
||||
def test_a_short_side_is_topped_up_from_the_other():
|
||||
assert draw.blend(["a"], ["a", "b", "c"], 0, 3) == ["a", "b", "c"]
|
||||
assert draw.blend(["a", "b", "c"], [], 0, 3) == ["a", "b", "c"]
|
||||
|
||||
|
||||
def test_a_watch_weighs_by_its_age_an_undated_one_two_half_lives_and_a_future_one_now():
|
||||
# Both affinities weigh every watch through this one rule; an old import has no date, and a
|
||||
# clock ahead of the server's must not weigh more than a watch made now.
|
||||
assert draw.recency(NOW, NOW, 30) == 1.0
|
||||
assert draw.recency(NOW - timedelta(days=30), NOW, 30) == 0.5
|
||||
assert draw.recency(None, NOW, 30) == 0.25
|
||||
assert draw.recency(NOW + timedelta(days=5), NOW, 30) == 1.0
|
||||
|
||||
|
||||
def test_scores_are_scaled_to_the_strongest_and_nothing_scored_is_empty():
|
||||
assert draw.scaled({"a": 4.0, "b": 1.0}) == {"a": 1.0, "b": 0.25}
|
||||
assert draw.scaled({}) == {}
|
||||
|
||||
|
||||
# ── YouTube: the channels the user watches ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _md5_order(ids, seed):
|
||||
return sorted(ids, key=lambda i: hashlib.md5(f"{i}{seed}".encode()).hexdigest())
|
||||
|
||||
|
||||
def _seed_yt(db, make_user, *, watched_in_fav: int):
|
||||
user = make_user()
|
||||
fav, other = Channel(id="UCfav", title="Fav"), Channel(id="UCother", title="Other")
|
||||
db.add_all([fav, other])
|
||||
db.flush()
|
||||
db.add_all(
|
||||
[
|
||||
Subscription(user_id=user.id, channel_id="UCfav"),
|
||||
Subscription(user_id=user.id, channel_id="UCother"),
|
||||
]
|
||||
)
|
||||
|
||||
def video(vid, ch, days):
|
||||
return Video(
|
||||
id=vid,
|
||||
channel_id=ch,
|
||||
title=vid,
|
||||
duration_seconds=20 * 60,
|
||||
published_at=NOW - timedelta(days=days),
|
||||
)
|
||||
|
||||
# One unseen video per channel, plus the user's watched history on the favourite.
|
||||
db.add_all([video("fav_new", "UCfav", 1), video("other_new", "UCother", 1)])
|
||||
for i in range(watched_in_fav):
|
||||
db.add(video(f"fav_seen{i}", "UCfav", 10 + i))
|
||||
db.flush()
|
||||
for i in range(watched_in_fav):
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id=f"fav_seen{i}",
|
||||
status="watched",
|
||||
watched_at=NOW - timedelta(days=i),
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
return user
|
||||
|
||||
|
||||
def _first(rows):
|
||||
return rows[0]["id"] if rows else None
|
||||
|
||||
|
||||
def test_a_watched_channel_wins_most_draws_but_not_all(db, make_user):
|
||||
user = _seed_yt(db, make_user, watched_in_fav=12)
|
||||
leads = [_first(queries.yt_surprise(db, user, "any", s, 2)) for s in SEEDS]
|
||||
fav = leads.count("fav_new") / len(leads)
|
||||
# weight 1 + 3 against 1: 0.8 in a weighted slot, 0.5 in the blind one (a third of the seeds)
|
||||
assert 0.6 < fav < 0.9, fav
|
||||
|
||||
|
||||
def test_the_blind_slot_deals_the_plain_shuffle_however_strong_the_lean(db, make_user):
|
||||
user = _seed_yt(db, make_user, watched_in_fav=12)
|
||||
for s in range(0, 30, 3): # (0 + seed) % 3 == 0: the first slot is the blind one
|
||||
got = _first(queries.yt_surprise(db, user, "any", s, 2))
|
||||
assert got == _md5_order(["fav_new", "other_new"], s)[0]
|
||||
|
||||
|
||||
def test_with_no_history_the_youtube_order_is_the_old_shuffle(db, make_user):
|
||||
user = _seed_yt(db, make_user, watched_in_fav=0)
|
||||
for s in range(12):
|
||||
got = [r["id"] for r in queries.yt_surprise(db, user, "any", s, 2)]
|
||||
assert got == _md5_order(["fav_new", "other_new"], s)
|
||||
|
||||
|
||||
def test_a_watched_video_is_never_drawn(db, make_user):
|
||||
user = _seed_yt(db, make_user, watched_in_fav=12)
|
||||
drawn = {r["id"] for s in SEEDS for r in queries.yt_surprise(db, user, "any", s, 5)}
|
||||
assert drawn == {"fav_new", "other_new"}
|
||||
|
||||
|
||||
# ── Plex: genres and recommendations of what the user watched ────────────────────────────────────
|
||||
|
||||
|
||||
def _seed_plex(db, make_user, *, history: bool):
|
||||
user = make_user()
|
||||
lib = PlexLibrary(plex_key="77", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
tmdb_ids = iter(range(5000, 6000))
|
||||
|
||||
def movie(key, genres, recs=None):
|
||||
m = PlexItem(
|
||||
rating_key=key,
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title=key,
|
||||
duration_s=100 * 60,
|
||||
genres=genres,
|
||||
guid=f"tmdb:{next(tmdb_ids)}",
|
||||
recommendations=recs,
|
||||
)
|
||||
db.add(m)
|
||||
return m
|
||||
|
||||
horror = movie("horror_new", ["Horror"])
|
||||
comedy = movie("comedy_new", ["Comedy"])
|
||||
tipped = movie("tipped_new", ["Western"])
|
||||
db.flush()
|
||||
tip = [{"tmdb_id": int(tipped.guid.split(":")[1]), "media_type": "movie", "title": "t"}]
|
||||
seen = [movie(f"horror_seen{i}", ["Horror"], tip if i == 0 else None) for i in range(3)]
|
||||
db.flush()
|
||||
if history:
|
||||
for i, m in enumerate(seen):
|
||||
db.add(
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=m.id,
|
||||
status="watched",
|
||||
watched_at=NOW - timedelta(days=i),
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
from app import sysconfig
|
||||
|
||||
sysconfig.set_value(db, "plex_enabled", True)
|
||||
db.commit()
|
||||
return user, horror, comedy, tipped
|
||||
|
||||
|
||||
def test_the_genre_and_the_recommendation_of_what_was_watched_win_most_draws(db, make_user):
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
leads = [_first(queries.plex_surprise(db, user.id, "any", s, 3, lang="en")) for s in SEEDS]
|
||||
# Weights: Horror 1 + 2 (its genre), the recommended Western 1 + 3, the Comedy nobody chose 1. In
|
||||
# a weighted slot the Comedy leads 1 time in 8, in the blind third 1 in 3: about 0.19 against
|
||||
# 0.81 for the other two together (a uniform draw: 1 in 3 each). It still leads now and then —
|
||||
# a surprise, not a ranking.
|
||||
comedy = leads.count("comedy_new")
|
||||
assert comedy > 0
|
||||
assert leads.count("horror_new") > 1.5 * comedy and leads.count("tipped_new") > 1.5 * comedy
|
||||
assert leads.count("horror_new") + leads.count("tipped_new") > 3 * comedy
|
||||
|
||||
|
||||
def test_with_no_history_the_plex_order_is_the_old_shuffle(db, make_user):
|
||||
user, *_ = _seed_plex(db, make_user, history=False)
|
||||
keys = ["horror_new", "comedy_new", "tipped_new"] + [f"horror_seen{i}" for i in range(3)]
|
||||
for s in range(12):
|
||||
got = [c["id"] for c in queries.plex_surprise(db, user.id, "any", s, 6, lang="en")]
|
||||
assert got == _md5_order(keys, s)
|
||||
|
||||
|
||||
def test_a_watched_film_is_never_drawn(db, make_user):
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
drawn = {
|
||||
c["id"] for s in SEEDS for c in queries.plex_surprise(db, user.id, "any", s, 6, lang="en")
|
||||
}
|
||||
assert drawn == {"horror_new", "comedy_new", "tipped_new"}
|
||||
|
||||
|
||||
def test_the_plex_blind_slot_deals_the_plain_shuffle(db, make_user):
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
unseen = ["horror_new", "comedy_new", "tipped_new"]
|
||||
for s in range(0, 30, 3):
|
||||
got = _first(queries.plex_surprise(db, user.id, "any", s, 3, lang="en"))
|
||||
assert got == _md5_order(unseen, s)[0]
|
||||
|
||||
|
||||
def test_a_binged_show_counts_once_and_recent_watches_weigh_more(db, make_user):
|
||||
user = make_user()
|
||||
lib = PlexLibrary(plex_key="78", title="Shows", kind="show", enabled=True)
|
||||
mlib = PlexLibrary(plex_key="79", title="Movies", kind="movie", enabled=True)
|
||||
db.add_all([lib, mlib])
|
||||
db.flush()
|
||||
show = PlexShow(rating_key="sitcom", library_id=lib.id, title="Sitcom", genres=["Comedy"])
|
||||
db.add(show)
|
||||
db.flush()
|
||||
eps = [
|
||||
PlexItem(
|
||||
rating_key=f"ep{i}",
|
||||
library_id=lib.id,
|
||||
kind="episode",
|
||||
title=f"e{i}",
|
||||
show_id=show.id,
|
||||
season_number=1,
|
||||
episode_number=i,
|
||||
)
|
||||
for i in range(10)
|
||||
]
|
||||
films = [
|
||||
PlexItem(
|
||||
rating_key=f"h{i}", library_id=mlib.id, kind="movie", title=f"h{i}", genres=["Horror"]
|
||||
)
|
||||
for i in range(2)
|
||||
]
|
||||
old = PlexItem(
|
||||
rating_key="w0", library_id=mlib.id, kind="movie", title="w0", genres=["Western"]
|
||||
)
|
||||
db.add_all([*eps, *films, old])
|
||||
db.flush()
|
||||
for it in [*eps, *films]:
|
||||
db.add(PlexState(user_id=user.id, item_id=it.id, status="watched", watched_at=NOW))
|
||||
db.add(
|
||||
PlexState(
|
||||
user_id=user.id, item_id=old.id, status="watched", watched_at=NOW - timedelta(days=720)
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
genres = affinity.for_user(db, user.id).genres
|
||||
# Ten episodes of one show are one title: two horror films outweigh it.
|
||||
assert genres["Horror"] == 1.0
|
||||
assert genres["Comedy"] == 0.5
|
||||
# Four half-lives ago weighs a sixteenth of today.
|
||||
assert abs(genres["Western"] - 0.5 / 16) < 0.01
|
||||
|
||||
|
||||
def test_the_discover_rail_leans_the_same_way(db, make_user):
|
||||
# The rail draws with random(), not a seed; setseed makes this session's draws repeatable.
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
db.execute(text("SELECT setseed(0.42)"))
|
||||
leads = [
|
||||
plex_discover.surprise(db, user, "movie", "en", 3, [], [], LibraryFilters())["items"][0][
|
||||
"id"
|
||||
]
|
||||
for _ in SEEDS
|
||||
]
|
||||
assert leads.count("comedy_new") > 0
|
||||
assert leads.count("horror_new") + leads.count("tipped_new") > 3 * leads.count("comedy_new")
|
||||
|
||||
|
||||
# ── whose history: only the user's own ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_another_users_youtube_history_does_not_lean_my_draw(db, make_user):
|
||||
watcher = _seed_yt(db, make_user, watched_in_fav=12)
|
||||
me = make_user()
|
||||
db.add_all(
|
||||
[
|
||||
Subscription(user_id=me.id, channel_id="UCfav"),
|
||||
Subscription(user_id=me.id, channel_id="UCother"),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
assert queries._channel_affinity(db, watcher)
|
||||
assert queries._channel_affinity(db, me) == {}
|
||||
|
||||
|
||||
def test_another_users_plex_history_does_not_lean_my_draw(db, make_user):
|
||||
_seed_plex(db, make_user, history=True)
|
||||
me = make_user()
|
||||
assert not affinity.for_user(db, me.id)
|
||||
keys = ["horror_new", "comedy_new", "tipped_new"] + [f"horror_seen{i}" for i in range(3)]
|
||||
for s in range(12):
|
||||
got = [c["id"] for c in queries.plex_surprise(db, me.id, "any", s, 6, lang="en")]
|
||||
assert got == _md5_order(keys, s)
|
||||
|
||||
|
||||
# ── the rail: its blind slot, and a show's own recommendations ───────────────────────────────────
|
||||
|
||||
|
||||
def test_the_rails_third_slot_is_the_blind_draw(db, make_user, monkeypatch):
|
||||
# The rail draws with random(), so the blend is watched rather than its odds: every leaning draw
|
||||
# goes through it, with the blind order beside the weighted one and the third slot blind.
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
calls = []
|
||||
real = draw.blend
|
||||
|
||||
def spy(weighted, blind, offset, n, key):
|
||||
calls.append((offset, n, sorted(map(key, weighted)), sorted(map(key, blind))))
|
||||
return real(weighted, blind, offset, n, key=key)
|
||||
|
||||
monkeypatch.setattr(draw, "blend", spy)
|
||||
got = plex_discover.surprise(db, user, "movie", "en", 3, [], [], LibraryFilters())["items"]
|
||||
assert calls and all(offset == 1 and n == 3 for offset, n, *_ in calls)
|
||||
pool = sorted(["horror_new", "comedy_new", "tipped_new"])
|
||||
assert calls[0][2] == pool and calls[0][3] == pool
|
||||
assert len(got) == 3
|
||||
|
||||
|
||||
def test_a_show_leans_on_the_shows_recommended_not_the_films(db, make_user):
|
||||
lib = PlexLibrary(plex_key="80", title="Shows", kind="show", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
tipped = PlexShow(rating_key="tv1", library_id=lib.id, title="t", guid="tmdb:1", genres=[])
|
||||
db.add(tipped)
|
||||
db.commit()
|
||||
shows = affinity.Affinity(show_recs={"tmdb:1": 1.0})
|
||||
films = affinity.Affinity(movie_recs={"tmdb:1": 1.0})
|
||||
|
||||
def weight(aff):
|
||||
return db.execute(select(aff.weight(PlexShow)).where(PlexShow.id == tipped.id)).scalar_one()
|
||||
|
||||
assert weight(shows) == 1 + affinity.REC_WEIGHT
|
||||
assert weight(films) == 1
|
||||
@@ -292,17 +292,16 @@ def test_a_known_video_is_marked_watched_at_its_takeout_time_and_the_rest_wait_o
|
||||
assert (states["older000000"].status, states["older000000"].position_seconds) == ("watched", 40)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("undo", ["unmark", "clear_state"])
|
||||
def test_a_video_the_user_un_marked_here_is_not_marked_again_by_an_older_view(db, make_user, undo):
|
||||
"""Un-marking (or resetting) a video leaves no state row, and the import inserted a watched one
|
||||
over the user's own choice. A view on YouTube after the un-mark is a new view, and counts."""
|
||||
user = make_user(f"{undo}@example.com")
|
||||
def test_a_video_the_user_un_marked_here_is_not_marked_again_by_an_older_view(db, make_user):
|
||||
"""Un-marking a video leaves no state row, and the import inserted a watched one over the user's
|
||||
own choice. A view on YouTube after the un-mark is a new view, and counts."""
|
||||
user = make_user("unmark@example.com")
|
||||
for vid in ("undone00000", "again000000"):
|
||||
_video(db, vid)
|
||||
db.commit()
|
||||
for vid in ("undone00000", "again000000"):
|
||||
service.mark(db, user, vid, "watched")
|
||||
getattr(service, undo)(db, user, vid)
|
||||
service.unmark(db, user, vid)
|
||||
db.commit()
|
||||
assert _states(db, user) == {}
|
||||
later = datetime.now(timezone.utc) + timedelta(minutes=5)
|
||||
@@ -603,7 +602,6 @@ def test_staging_and_settling_take_the_users_one_lock(db, make_user):
|
||||
),
|
||||
# Between the import's read of the un-marks and its marking, one would be overwritten.
|
||||
"unmark": lambda s, u: service.unmark(s, u, "aaaaaaaaaaa"),
|
||||
"reset": lambda s, u: service.clear_state(s, u, "aaaaaaaaaaa"),
|
||||
}
|
||||
with SessionLocal() as holder:
|
||||
service.lock_takeout(holder, holder.get(User, user.id))
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user